Hello folks! I have already told you that I am currently working on a set of interviews with some of the speakers you will find at Erlang Factory SF Bay Area 2013. Today I have the pleasure to interview Knut Nesheim. Knut is widely known for some applications he created while working at Wooga (e.g., Elli and Locker). In this interview we will ask him something more about his Erlang experience, his famous Erlang applications and the talk he will give at Erlang Factory SF Bay.
Rock ‘n Roll? No, lock and rule!
Paolo – Hi Knut! Thanks for making yourself avaiable for an interview. Please, describe yourself in few words.
Knut – Hey Paolo, thanks for having me! I’ve been developing software for the last five years, in Norway where I’m originally from, Sweden and now Germany. I’ve always had an urge to understand how things work and computers provide endless opportunities for tinkering, hacking and exploring.
Paolo – You started as with an educational background in music and then moved to computers right? How come did you switch?
Knut – After computers, music is my second big interest in life. Music obeys “laws”, many quite intricate. It’s interesting to take apart a piece of music and understand how the parts work together to form something that can create emotion, give energy and inspire in people without them maybe even noticing, like in film. I find music for film to be very interesting.
It became apparent pretty quickly that I was much better at computers than music. I still play guitar and I regularly visit the Berlin Philharmonie. I just bought an electric piano and I want to start taking lessons again. It’s a nice hobby.
Paolo – I do believe that your main language before Erlang was Python. In your opinion what is the most difficult concept of Erlang for a developer coming from Python?
Knut – I was using Python quite a lot, yes. I was doing smaller web applications with lots of business logic.
The hardest part about learning Erlang for me was immutable state. I was used to thinking of data being stored in places and having different parts of the program storing new values in that place. Understanding that after I modify something, there are now two versions of it took some time. It’s a bit sad that it’s mostly inside of our programs we can have the immutable state. I find Datomic, the database from Rich Hickey to be very interesting.
Paolo – You are the author of Elli, an Erlang application I like very much. Can you tell us somethink about it?
Knut – Good to hear someone likes it! Elli is a special purpose webserver you can run inside your Erlang application to let your code speak HTTP with your clients. It’s not general purpose in that it lacks features you would take for granted if you’re for example making a web application. It’s a mashup of ideas from different projects, both within the Erlang community and outside, specifically it takes a lot of inspiration from Rack.
It has been a fun project to work on and I have received 34 pull requests to date, which isn’t much compared to other bigger projects, but I think it’s great to see so users contributing their improvements. It’s used in production in Wooga. Upcoming games are using it in very interesting ways. The traffic is absolutely crazy, with each online user making a request every 1-5 seconds.
Paolo – Why should we take a look at Elli? I mean, in the Erlang community we have already many Web Servers; when should we prefer Elli to them?
Knut – It’s true that there’s a bunch of projects already there, overlapping in quite big areas resulting in duplicated efforts.
I wrote Elli for the needs of Wooga, which was difficult to meet with the existing projects. Now, that’s not because they are bad in any way, lack features or are buggy. They are quite successful at making developers more productive.
When the volumes of traffic grow, so does the amount of weird input and amount of unexpected interactions in the implementation of the webserver. If you have an error that statistically speaking happens every 10,000 requests, at Wooga that error happens more than once a second. We found that developer productivity wasn’t all that important for us, it’s more important to focus on performance, robustness and operations.
A webserver that did less, we thought, would be less likely to have features interact in unexpected ways and would be faster from less code (and fewer processes!). Using the Rack-style of request-response where the handler function returns the response (as opposed to sending it directly on the socket), allows us to create middlewares. In the “web world”, middlewares are a common way to plug in third-party functionality, like an authentication middleware inspecting and injecting headers, a middleware implementing cookies or a middleware collecting statistics and exposing it over HTTP by overriding a url. It doesn’t fit everything, you can’t for example postprocess a streamed response.
If you are able to judge the trade-offs made by the different projects and you find Elli fits your needs better, I would encourage you to use it. I’m happy to help and committed to maintaining the project. If you’re not sure which one to pick, go for Yaws.
Paolo – During your career you have been working in many companies as for example Wooga and Klarna. How does it feel to work in some of the biggest Erlang companies out there?
Knut – Pretty cool! I really enjoyed my time at Wooga. I joined January 2011 as a “gun for hire” to help out with a experimental Erlang project, started by Paolo Negri. Together we built the first backend using Erlang, paving the way for adoption within the company. They now have 14 developers using Erlang with almost all working on upcoming games.
Now I’m heading up a very interesting project at Unity, the makers of the 3d game engine. It’s too early to talk publicly about the projects we are building, but I’m very excited about this opportunity. The team is starting to form and by the end of the year, hopefully we have something really cool to show.
Paolo – During the Erlang Factory in San Francisco you will give the following talk “Locker: Consistent Distributed Locking”. Would you like to tell us something about it?
Knut – Locker is a distributed locking service, implemented in Erlang and uses the Erlang cluster for messaging. It’s a multi-master key-value in-memory database, where a key expires if the lease is not renewed periodically. In terms of CAP, it will sacrifice availability when a quorum can not be made
It’s a hack, in the sense that it’s not an implementation of a “proper” algorithm from a paper. For example, coordination of writes happens in two phases, where if a quorum can be made in the first phase, the second phase writes the value on all masters. Some people properly educated in distributed systems tell me it’s good stuff, others tell me it ignores too many hard problems to be useful, like group membership. I’m looking forward to hopefully getting more feedback.
Paolo – How come Wooga needed something as Locker? What are the main benefits one may experience using it?
Knut – Just like with Elli, eredis and statman, locker started out as a frustration with what was currently available to us. In the stateful application servers we were building, we needed a way of ensuring that each online user has only one process. This can be solved in many different ways and we found ourselves constantly thinking about it. Our first version was just a central point which serialized all updates, but we wanted it to be available in case of partitions or more common to us, node failure, software crash and operator error.
We thought about this challenge for more than a year before we decided to tackle it head on. A recurring question was “what if we could do the perfect solution to our problem, rather than trying to shoe-horn existing solutions with complex hacks for scaling?” closely followed by “how hard could it be?” Eventually, I decided to just try it. I’m very happy with the result.
Paolo – How long did it take to have a working version of Locker? I find interesting the fact that such a piece of software if available for free on GitHub, don’t you agree?
Knut – From I decided to try building it to having a working version took around three weeks, out of which one afternoon was spent coding. The rest of the time I spent thinking, drawing and reading. Writing tests, async replication, various optimizations, etc, maybe a month of total effort has been spent over the last year, with contributions from others at Wooga. It’s just 330 lines of code, but coming up with those exact lines was very hard.
From developers to CTO and CEO, Wooga is pushing to open-source what can be open-sourced to hopefully contribute something back to the community. It took some time to convince everybody that locker was something Wooga should give away for free, even to competitors. I was the one holding back, but was eventually convinced by Jesper Richter-Reichhelm, head of engineering. It’s always scary to put something out there for everybody to see.
Paolo – Who should follow your talk and why? What are the basic skills a wannabe attendant to your talk should have?
Knut – If you’re interested in distributed systems, it might be interesting to see how locker was built and why it was built that way. If you’re an expert, you could come and help me improve locker by providing feedback on why it’s good or why you think it’s total crap. If you’re interested in the cultural parts of a successful game studio and how we build amazing things in a very short time, you could get a glimpse inside the inner workings. Knowing Erlang isn’t really necessary. The challenging parts of understanding locker is not in the code.
Hello everybody! Thanks to Erlang Solutions I have been able to interview Evan Miller, one of the speakers you can meet at the upcoming Erlang Factory which will be held in San Francisco (21-22 March) . Evan is mostly known as the author of Chicago Boss. Let’s know him better with this interview! Hope you will enjoy it!
Meet the boss!
Paolo – Hi Evan, it’s really nice to have you here. Can you briefly introduce yourself to our readers?
Evan – It’s great to be here, and I’m glad to do it. I’m a graduate student in economics living in Chicago who has a passion for programming. People in the Erlang world know me as the creator of Chicago Boss, the
maintainer of ErlyDTL, and a loudmouthed defender of unpopular language features. Before grad school I briefly worked as an ops guy for Amazon.com and as a developer for IMVU, both out in Silicon Valley.
Paolo – How come a person with a degree in physics gets in touch with Erlang? Was it “love at first sight” or did you take a while in order to grasp it?
Evan – I stumbled across Erlang in a rather roundabout way. While I was involved in search operations at Amazon.com, I discovered that the source of almost all pain and suffering in the ops world can be traced
directly to blocking, threaded socket code. This led me to tinker with Nginx — a non-blocking server — but although I love programming in C (really!), I couldn’t stand the callback-based API. Shortly thereafter, a friend of mine at IMVU introduced me to Erlang, which is non-blocking… but without the callbacks! I instantly saw through Erlang’s syntax warts and fell in love with “what’s on the inside” — the absolute best VM for doing any kind of network programming.
As for the physics connection, I agree with Joe Armstrong — the creator of Erlang and who incidentally is an ex-physics person — when he says that Erlang’s process model comes naturally to anyone who has
studied relativity. Each process is its own “frame of reference”, in a manner of speaking — no process knows what’s on the inside of other processes, and heck, process clocks may be out of sync, but that’s OK!
You learn how to program around it and end up with much more reliable systems than ones that assume there is a “true frame of reference” of some sort. There can’t be, because you have to assume that *any*
server might fail or become unreachable in a networked environment.
Paolo – You are going to be one of the speakers at the next Erlang Factory held in San Francisco Bay area on March, what will be topic of your talk?
Evan – I am! The topic’s title is “Chicago Boss: a Web Framework Built For Comfort (And Speed)”. I’ll be giving a high-level overview of my pet framework and talk about some down and dirty details of why it’s designed the way it is.
Paolo – Who should attend your talk? Do they need to be expert Erlang or Web programmers?
Evan – Everyone should attend! I’ll try to make it interesting for both experts and novices. For example, I’ll explain why Chicago Boss is so darned good at fast server-side template rendering, which is something
that most languages have a lot of trouble with. I’ll also show some code samples to make things concrete.
Paolo – As the author of Chicago Boss would you like to introduce it briefly to our users?
Evan – Sure thing. Chicago Boss is a web framework that basically attempts to steal all of the good ideas from Ruby on Rails, and fix everything that’s broken: namely, slowwwwww templates, rampant RAM usage, lack of real-time messaging (WebSockets), and all this DevOps nonsense. So Chicago Boss combines fast templates with a clean API and a sophisticated “ORM”. I put that last bit in quotation marks because Erlang doesn’t really have objects, so we use a sort of fake object called a parameterized module that everyone on the Erlang mailing list hates with a passion, but which we love because it’s perfect for modeling data.
Anyway the framework is still in beta, but a lot of people are using it in production and we’re hoping to have a stable release soon!
Paolo – In your opinion, what are the main reasons that should make a web programmer switch from other platforms written in different languages as Rails, Symfony or Django to Chigago Boss?
Evan – Well, the first advantage is getting to work with a nice functional language like Erlang, even if CB sprinkles in some OO-style conventions here and there. Once you get the hang of it — which may take a while! — it’s generally easier to write correct code in Erlang than in non-functional languages.
But I think the real benefit actually comes down to operational costs — Chicago Boss runs smoothly on relatively little hardware and doesn’t need to be constantly restarted like most Ruby and Python servers. Erlang truly has a first-rate VM that is perfect for serving web pages. We’re really excited about running CB applications on ARM-based servers, which could really up-end the economics of server applications in the next few years.
Paolo – Chicago Boss in not the only option when talking about Erlang and web applications, what are the strength of Boss with respect to the others? Is there any feature you would like to “steal” from them?
Evan – CB is more of a full-stack framework than the others out there — in particular, it’s the only one that ships with any kind of database abstraction, letting you write your code first and choose from one of
six SQL and NoSQL database options later. There’s also an email server baked in, and an internationalization system — we’ve really tried to provide a complete package, whereas most of the other Erlang servers tend to be more bare-bones toolkits.
Don’t tell my mother, but I never hesitate to steal! But most of what I’ve stolen has actually come from the world outside Erlang. For example, the default template language is a tag-for-tag reimplementation of Django’s template language, and there’s now experimental support for the Jade template language.
Paolo – What kind of data storage support may we have with Chicago Boss? Do you recommend any of them, and if so why?
Evan – As I mentioned there are many options — in the SQL camp there’s PostgreSQL and MySQL; in the NoSQL camp there’s MongoDB, Riak, and Tokyo Tyrant, and then there’s an Mnesia adapter which isn’t SQL per se but it is schema’d. I don’t want to start a religious war, so I’ll stay quiet about my own preferences — but it’s really easy to add support for more databases — usually just requires a couple hundred lines of code.
Paolo – At the beginning Boss was implemented around Misultin, but later you moved to Cowboy. Was it difficult to switch the architecture? How many code did you rewrite in that sense?
Evan – For a long time, Chicago Boss supported only Mochiweb and Misultin; as we were pondering how to support Cowboy’s alien-looking API, someone had the brilliant idea to write a Mochiweb-compatible wrapper around Cowboy! It’s called “mochicow” and lets us adapt CB to use Cowboy in rather short order. Anyway, most of the hard work here is actually done by the folks over at Nitrogen who created SimpleBridge, which is an abstraction layer over I think six different web servers. It’s really a nice little library and we’ve contributed code back to it.
Paolo – One strength of Chicago Boss consists in my opinion in the good amount and quality of its documentation. Would you like to address your favourite online resources to a developer willing to learn more about Chicago Boss? Is there any “real” book about Chicago Boss?
Evan – As much as I love killing trees, there’s no book yet! The closest thing to it is a PDF tutorial on the website, and people are always adding new code samples to the Wiki hosted on GitHub. The best way to get started is to work through the tutorial, peruse the API documents, and hop on the mailing list. It’s an active, friendly place where we welcome in newcomers all the time.