Why Erlang ?
Why we chose Erlang over Python, Ruby, PHP family for our back-end.
The short answer :
Because it’s a completely different approach from the OO programming languages (even from the pure functional ones like Lisp or Haskell), it’s concurrent, stateless, has been battle tested in real large-scale industrial products, has an active web app centric community and finally, because it fits perfectly with our view of a light back-end delivery service engine.
Quick links featured in this post :
- Erlang - Software for a concurrent world, Joe Armstrong
- Introduction to Erlang, Bob Ippolito (SimpleJson, Mochiweb)
- Mochiweb, the web application server we chose
- Yariv Sadan’s blog, creator of Erlyweb, an Erlang web application framework
- Yaws, a web server written in Erlang
- Server Apache vs. Yaws, a graphical benchmark
- Programming Erlang, the book from Joe Armstrong
When we were looking for our main back-end language back in April, Google had just launched a micro-bomb named GAE, the Google App Engine. We know all about Google’s killer web apps, services and brains reservoir, and so it was difficult not to have a closer look at the potential of their brand new SDK. In brief, GAE is the Google answer to the Amazon S3 in the cloud computing arena. It simply enables you to benefit the same scalable systems than those that power Google applications.
We decided that the server part of our web app should remain light, flexible and be able to deliver chunks of JSON as fast as possible. The client (browser) would be in charge, among other things, of processing the data and building the final XHTML.
At that time, I was comfortable with Ruby and was following the progresses of the Merb framework project. But after all, why not change and learn something new ? I love trying out new things, so why not Python, keeping in mind that we could benefit from GAE in the future? Don’t misunderstand me, I’m still following the development of Merb with a lot of interest :).
So, I started to look at WebPy, Pylons and was getting excited to enter this new community. Coming from Ruby, learning Python, and making a small working prototype using WebPy (soon to be transferred to the GAE SDK) was quite easy. The goal isn’t to enter in a deep comparison of those two languages, I’m not mastering Python enough anyway. It wasn’t to start a flame war either, it should have been just the opposite (Ruby2Python or even Python2PHP, …). ALL languages have their pros and cons and those might even change depending on the context.
I was confident in the potential of Python, mainly because I retrieved quite easily all the libraries I was looking for. But in the end, I became quite disenchanted. All in all, it wasn’t so different from what I was doing before. Perhaps I reached a point were the OO programming style bored me a little. I realized that starting a long term project with that kind of languages was not going to be more diversified than a walk in the park.
In the meantime, Mic tried to open my eyes to the beauty of Lambda, a kind of God traveling the globe wearing trainers signed with a ‘Y’ reversed. Experienced in Lisp, he wanted to give that language a chance in developing our web app. Hum … after wandering around the net a bit, reading that story about Reddit, and some more …, I rapidly realized that Lisp didn’t have a strong ‘web dev planet’.
A few tutorials with Lisp later and still looking at the functional languages, I landed on the Erlang OTP site. Great!
I passed back the message to Mic and soon we discovered a lot of web focused Erlang projects : Erlyweb, Mochiweb, Yaws, Ejabberd, CouchDB, SimpleDB, the Facebook chat engine … and so on. We were hooked!
It took me a little bit more than a month to learn that language and build the first prototype of our back-end service delivery system.
Erlang is really matching what we were looking for. By its modularity nature, our client architecture is potentially capable of sending a bunch of small AJAX service requests to the server. Using Mochiweb as our web server (proxied through Nginx), our engine is able to handle this kind of load, while keeping a small footprint in memory. It’s even possible to modify/update the engine without taking it out of service!
The crucial properties of Erlang are that it has been build from the ground up to be both a concurrent and a fault tolerant oriented language. Erlang supports hundreds of thousands of lightweight processes in a single virtual machine.
Not only does our engine natively benefit from multi-core architectures (the more processors, the more speed) but each of our application processes is also really independent from each other. They can die without making the whole request processing crash.
It also looks like it’s flexible enough to communicate with the other languages without too much hassle. We could for example benefit from external libraries (Python, Ruby, C, Java) for specific jobs : i.e. edit and compose bitmap images, pdf conversion, …
Erlang is certainly not perfect, you have to learn a -sometimes- scary syntax, but it’s really worth it in the end, it’s just a different animal.
Instead of writing what others have written a dozen times, I’d rather give you some key links in order to properly introduce you to Erlang.
The following material contributed a lot to our final decision :
http://www.infoq.com/presentations/erlang-software-for-a-concurrent-world
Where you learn what is a paradigm shift by Joe Armstrong one of the creators of Erlang.
http://bob.pythonmac.org/archives/2008/03/08/exploring-erlang-c41-video
Let me introduce you to Bob Ippolito. He is involved in the development of SimpleJson and Mochiweb, to name just a few.
http://code.google.com/p/mochiweb/
The web app server we chose.
http://www.trapexit.org/
The Erlang community.
http://yarivsblog.com/
The guy behind Erlyweb, the Erlang web application framework !
http://yaws.hyber.org/
A full featured web server.
http://www.sics.se/~joe/apachevsyaws.html
A small graphical benchmark 
http://www.pragprog.com/titles/jaerlang/programming-erlang
I strongly encourage you to buy this really helpful Joe Armstrong book if you seriously want to start learning Erlang.
In my next post, I’ll be talking about Mnesia, the distributed DataBase Management System …
Stay tuned.
I’d like a clarification about this point.
I’m learning erlang and i’m loving it!, but this issue is what scares me for software in “real projects”. I haven’t found any way to communicate with another languages besides jinterface. am i missing something?
if the syntax was just more pleasing…
Hey Blaxter,
Indeed, this sentence might sound a bit enthusiastic. There is an entire chapter (12) dedicated to interfacing Erlang with other languages in the book mentioned in the post. To be sincere, you’ll have to get your hands dirty in order to interface with specific libraries. But still, it’s feasible.
Anyway, in our case, this is not a priority. We will certainly need at some point to benefit from some functionalities offered by other languages, but it’s not going to happen often. Either way I could use the os:cmd() to call OS executables (ImageMagick, Ruby script, …) or/and, most importantly, from the browser call another service source.
As we mentioned in this post (http://beebole.com/blog/2008/07/07/true-aggregation-the-browser-as-a-platform/) we think that the browser should play an important role in integrating and aggregating various services. Although connecting natively to other languages may have some utility, we believe that most of the services, at least to build our application, can go this way.
One thing that always interests me as well when I see people using Lisp/Haskell/Erlang for developing what is a business application like this with a rich domain model is what you are using for persistence. In the Java and C# world you have Hibernate and NHibernate and other ORM tools. In Ruby and other scriptiong languages you have some implementation of the active object pattern. But I’m curious what you use in Erlang. I’ve looked at stuff for Haskell and been somewhat let down that there wasn’t anything that seemed production ready. Is it a different story for Erlang?
Probably the biggest thing that I’d need to wrap my brain around in any case is how, since all your variables/objects/etc. are immutable, do you go about updating the database (or whatever you use as a persistent storage mechanism)? I’ve never seen issues like this discussed in much detail, which is unfortunate since most of the web apps I’ve seen out there at least start off with nothing more than slightly glorified CRUD functionality.
@SecretNeko
Regarding the syntax, there are certainly some areas of improvement. Becoming more comfortable with the language, I consider this less and less as a real problem.
Overall, the code looks clear and well structured. Most importantly, I was really surprised to see how easy it was to re-enter my code after 1 month off. Even with my short experience with the language I came back in quite smoothly to enhance my prototype (yes, we jumped on the road to the Prod highway ;).
One of the key point is to make the -pattern matching feature- a second nature.
@Rich
I’m going to dedicate an entire post on that matter. I’ll explain how we use Mnesia ( http://www.erlang.org/doc/apps/mnesia/part_frame.html ) and why we are keeping an eye on the CouchDB project (as mentioned in the post, we are already massively using Json as our ‘communication protocol’ between the browser and the server).
In brief, in order to keep as much flexibility as possible in our model and to benefit from the native Erlang data structure, we decided to go without a traditional relational model.
What? No objects, no relational DB? Are you crazy?
As things goes, I’ll explain why it’s not as crazy as it may sound when it comes to play with DB documents and not just tables linked to each other.
@Blaxter
I have done some interfacing of Erlang and Ruby to make it possible for http://github.com/AlainODea/erlang_code_completion/tree/master to get code completion results to TextMate. I originally used TCP, but found it slow and cumbersome compared to erl_call.
I want to make this point very clearly, erl_call is solid gold. Become acquainted with it if you intend to interface with Erlang easily. It is very fast and provides a good way of interacting with Erlang from other languages lacking a formal integration like JInterface or Erl_Interface.
A Ruby wrapper for erl_call would be very useful. I might bring myself to write such a beast and post it up to GitHub if I get a chance.
Having erl_call in the main bin directory of the Erlang distribution instead of buried inside the erl_interface library would also be useful. You can have erl_call start an Erlang node on demand or execute code from the command-line or STDIN on an existing node using RPC. It is superbly useful. Check out my usage of it in erlang_code_completion to see if it helps solve your problem.
[...] 3000 is the number of unique visitors, in two days, who read our last post: Why Erlang ? [...]
Very interesting Guys! You probably know that I was a fan of Microsoft… BUT I did also some researches to find a good programming language for my personnel Web server. I’ve play a lot with many different languages and technologies. For me winner is (I never thought say that one day)… WebDynPro Abap. Why? Because of the MVC implementation, data binding, tight integration to the DB and easy language. OK, this is not realist to use a WAS server for my web server
… so I tried to find a solution based on true MVC architecture (where you can handle different views in the same page, link actions to views graphically), a solution that has a very simple data binding…I find no miracle solution. The nearest thing is for me: RUBY + RAILS. Erlang seems made for massive applications, I thought you’ve chosen a framework/lge more oriented for fast development (easy syntax/ORM/OO…), reliable code, clear separation between business logic and layout (mvc). I’m hungry to heard your next posts (how you deal with the data storage), so I will stay tuned! Good luck for your project ! -> very interesting blog! à la prochaine party OpenHR 
Hey Pierre,
Just a quick comment to stress out that Erlang is definitely as well appropriated for rapid prototyping than for production code. It’s a dynamic language, you can build a server really easily and change it’s behavior on the fly. The functional and stateless nature of the language opens new levels of possibilities. Turning the ideas floating in my head into real code remains a fast process (the last one : merging non-destructively two tree structures took me a little bit more than an hour - still, I had to re-read the ‘lists’ and ‘proplists’ modules in order to use some of their useful functions ;).
[...] public links >> mochiweb Why Erlang ? Saved by rottingtreesandgrass on Fri 22-8-2008 Multipart Post with Erlang and Mochiweb Saved by [...]
[...] our post Why Erlang ? for more [...]
[...] We started by choosing a light, flexible and really fast back-end as mentioned in our post: Why Erlang ? [...]
[...] say you want to give a try to Erlang (Discover our post about Why Erlang?) for your next web development project and you want to be up and running as quickly as [...]