Why we chose Erlang over Python, Ruby, PHP family for our back-end.
The short answer :
Because it’s a completely different approach from the OO programming languages (even from the pure functional ones like Lisp or Haskell), it’s concurrent, stateless, has been battle tested in real large-scale industrial products, has an active web app centric community and finally, because it fits perfectly with our view of a light back-end delivery service engine.
Quick links featured in this post :
- Erlang – Software for a concurrent world, Joe Armstrong
- Introduction to Erlang, Bob Ippolito (SimpleJson, Mochiweb)
- Mochiweb, the web application server we chose
- Yariv Sadan’s blog, creator of Erlyweb, an Erlang web application framework
- Yaws, a web server written in Erlang
- Server Apache vs. Yaws, a graphical benchmark
- Programming Erlang, the book from Joe Armstrong
When we were looking for our main back-end language back in April, Google had just launched a micro-bomb named GAE, the Google App Engine. We know all about Google’s killer web apps, services and brains reservoir, and so it was difficult not to have a closer look at the potential of their brand new SDK. In brief, GAE is the Google answer to the Amazon S3 in the cloud computing arena. It simply enables you to benefit the same scalable systems than those that power Google applications.
We decided that the server part of our web app should remain light, flexible and be able to deliver chunks of JSON as fast as possible. The client (browser) would be in charge, among other things, of processing the data and building the final XHTML.
At that time, I was comfortable with Ruby and was following the progresses of the Merb framework project. But after all, why not change and learn something new ? I love trying out new things, so why not Python, keeping in mind that we could benefit from GAE in the future? Don’t misunderstand me, I’m still following the development of Merb with a lot of interest :).
So, I started to look at WebPy, Pylons and was getting excited to enter this new community. Coming from Ruby, learning Python, and making a small working prototype using WebPy (soon to be transferred to the GAE SDK) was quite easy. The goal isn’t to enter in a deep comparison of those two languages, I’m not mastering Python enough anyway. It wasn’t to start a flame war either, it should have been just the opposite (Ruby2Python or even Python2PHP, …). ALL languages have their pros and cons and those might even change depending on the context.
I was confident in the potential of Python, mainly because I retrieved quite easily all the libraries I was looking for. But in the end, I became quite disenchanted. All in all, it wasn’t so different from what I was doing before. Perhaps I reached a point were the OO programming style bored me a little. I realized that starting a long term project with that kind of languages was not going to be more diversified than a walk in the park.
In the meantime, Mic tried to open my eyes to the beauty of Lambda, a kind of God traveling the globe wearing trainers signed with a ‘Y’ reversed. Experienced in Lisp, he wanted to give that language a chance in developing our web app. Hum … after wandering around the net a bit, reading that story about Reddit, and some more …, I rapidly realized that Lisp didn’t have a strong ‘web dev planet’.
A few tutorials with Lisp later and still looking at the functional languages, I landed on the Erlang OTP site. Great!
I passed back the message to Mic and soon we discovered a lot of web focused Erlang projects : Erlyweb, Mochiweb, Yaws, Ejabberd, CouchDB, SimpleDB, the Facebook chat engine … and so on. We were hooked!
It took me a little bit more than a month to learn that language and build the first prototype of our back-end service delivery system.
Erlang is really matching what we were looking for. By its modularity nature, our client architecture is potentially capable of sending a bunch of small AJAX service requests to the server. Using Mochiweb as our web server (proxied through Nginx), our engine is able to handle this kind of load, while keeping a small footprint in memory. It’s even possible to modify/update the engine without taking it out of service!
The crucial properties of Erlang are that it has been build from the ground up to be both a concurrent and a fault tolerant oriented language. Erlang supports hundreds of thousands of lightweight processes in a single virtual machine.
Not only does our engine natively benefit from multi-core architectures (the more processors, the more speed) but each of our application processes is also really independent from each other. They can die without making the whole request processing crash.
It also looks like it’s flexible enough to communicate with the other languages without too much hassle. We could for example benefit from external libraries (Python, Ruby, C, Java) for specific jobs : i.e. edit and compose bitmap images, pdf conversion, …
Erlang is certainly not perfect, you have to learn a -sometimes- scary syntax, but it’s really worth it in the end, it’s just a different animal.
Instead of writing what others have written a dozen times, I’d rather give you some key links in order to properly introduce you to Erlang.
The following material contributed a lot to our final decision :
Where you learn what is a paradigm shift by Joe Armstrong one of the creators of Erlang.
Let me introduce you to Bob Ippolito. He is involved in the development of SimpleJson and Mochiweb, to name just a few.
The web app server we chose.
The Erlang community.
The guy behind Erlyweb, the Erlang web application framework !
I strongly encourage you to buy this really helpful Joe Armstrong book if you seriously want to start learning Erlang.
In my next post, I’ll be talking about Mnesia, the distributed DataBase Management System …