Archive
An interview with Steve Vinoski (@stevevinoski)
Today you can read my interview to Steve Vinoski, a famous Erlang developer/speaker and distributed systems expert. Steve will give the talk “Addressing Network Congestion in Riak Clusters” at Erlang User Conference 2013.
Some questions, some answers
Paolo – Hi Steve! It’s really good to have one of the most famous Erlangers here in my blog. Would you mind to introduce yourself to our readers in a few words?
Steve - I’m Steve Vinoski, a member of the architecture group at Basho Technologies, the makers of Riak and RiakCS. I have a background in middleware and distributed systems, and have been an Erlang user since 2006.
Paolo – I know you are expert in several programming languages. How did you end up using Erlang? Did you have any previous experience with functional languages?
Steve - As far as functional languages go, I’ve played with them on and off for decades, but never used one in production until I found Erlang.
I worked in middleware from 1991 to 2007, and in 2004 at IONA Technologies I started looking into innovative ways of expanding our product line and reducing the cost of product development. IONA’s products were written in C++, which I’ve used since 1988 and so I am well aware of its complexity, and Java, which frankly I’ve never liked (I like the JVM but don’t like the Java language). Neither language lends itself to rapid development or easy maintenance. I built a prototype that layered Ruby over one of our C++ products that allowed for an order of magnitude decrease in the number of lines of code required to write client applications, and built another prototype that provided a JavaScript layer for writing server applications, but customers didn’t seem interested, and both approaches only increased development and maintenance costs.
Then I found Erlang/OTP. I grew more and more intrigued as I discovered that it already provided numerous features that we had spent years developing and maintaining in our middleware systems, things like internode messaging, node monitoring, naming and discovery, portability across multiple network stacks, logging, tracing, etc. Not only did it provide all the features we needed, but its features were much more powerful and elegant. I put together a proposal for the IONA executive team that suggested we rebuild all of our product servers in Erlang so we could reduce maintenance costs, but the proposal was rejected because, as I later learned, they were trying to sell the company so it didn’t make sense to make such large changes to the code. I left IONA and joined Verivue, where we built video delivery hardware, and there I trained seven or eight other developers in Erlang and we used it to great advantage. After Verivue, I wanted to continue working with Erlang, which is part of the reason I joined Basho.
Paolo – In your blog you state that Erlang is your favourite programming language. Why?
Steve - To me Erlang/OTP is the type of system my middleware colleagues and I spent years trying to create. It’s got so many things a distributed systems developer wants: easy access to networking, libraries and utilities to make
interacting with distributed nodes straightforward, wonderful concurrency support, all the upgrading and reliability capabilities, and the Erlang language itself is sort of a “distributed systems DSL” where its elegance and small size make it easy to learn and easy to use to quickly become productive building distributed applications. And as if that’s not enough, the Erlang community is great, pleasantly supporting each other and newcomers while avoiding pointless arguments and rivalries you find in other communities. My use of other programming languages has actually decreased in recent years due primarily to my continued satisfaction with Erlang/OTP — it’s not great for every problem, but it’s fantastic for the types of problems I tend to work on.
Paolo – I know that in a previous working experience you had to deal with multimedia systems, a field where Erlang has still a minor impact with respect to languages like C++. Do you think Erlang will be able to find its place in this field as well? Can you give reasons for your answer?
Steve - Erlang/OTP is excellent for server systems in general, including multimedia servers. The Verivue system I worked on a few years ago had special TCP offload hardware for video delivery, so we didn’t need Erlang for that. Rather, we used Erlang for the control plane, which for example handled incoming client requests, looked up subscriber details in databases, and interacted with the hardware to set up multimedia data flows. Multimedia systems also have to integrate with billing systems, monitoring systems, and hardware from other vendors, and Erlang shines there as well, especially when it comes to finding bugs in the other systems and hot-loading code to compensate for those bugs. Customers tend to love you when you can quickly turn around fixes like that.
Another Erlang developer, Max Lapshin, built and supports erlyvideo, which seems to work well. I’ve never met Max but I know he faced some challenges along the way, as we did at Verivue, but I think he’s generally happy with how erlyvideo has turned out.
Paolo – Currently you are working at Basho, a very important company in the Erlang world. Do you mind telling our readers something more about your job?
Steve - At Basho I work in CTO Justin Sheehy’s architecture group. It’s a broad role with a lot of freedom to speak at and attend conferences and meetups, and I also work on research projects and pick up development tasks and projects from our Engineering team and Professional Services team when they need my help.
Paolo – At Erlang User Conference 2013 you will give a talk about Riak, its behaviour under extreme loads and the issues we may face when we want to scale it. Can you tell us something more about the topic?
Steve - At Basho we’re fortunate to have customers who continually push the boundaries of Riak’s comfort zone. Network traffic in Riak all goes over TCP — client requests, intracluster messages, and distributed Erlang communication. When clusters are extremely busy with client requests and transfer of data and messages between nodes, under certain conditions network throughput can drop significantly and messages can be lost, including messages intended for client applications. I am currently investigating the use of alternative network protocols to see if they can help prioritize different kinds of network traffic. This work is not yet finished, so my talk will give an overview of the problems along with the current status of the solution I’m investigating.
Paolo – I heard that you will also introduce during the talk a new Erlang network driver that should tackle some of this issues. Is this correct? Can you give us an insight?
Steve - Yes, I have been working on a new network driver. It implements an alternative UDP-based protocol for data transfer that can utilize full bandwidth when available but can also watch for congestion on network switches and quickly back off when detected. It also yields to TCP traffic under congestion conditions, preventing background data transfer tasks from shutting out more important messages like client requests and responses.
Paolo – Who should be interested in this talk? What are the minimum requisites needed in order to fully understand the topics of the talk?
Steve - Attendees should have a high-level understanding of Erlang’s architecture, what drivers are, and how they fit into the system. Other than that, my talk will explain in detail the problems I’m trying to address as well as the solution I’ve been investigating, so neither deep networking expertise nor deep understanding of Erlang internals is required.
Paolo – I can say without doubts that you are an expert in middleware and distributed computing systems. Can you suggest to our readers interested in those topics some books or internet resources?
Steve - The nice thing about distributed systems is that they never seem to get any easier, so there have been interesting research and development in this area for decades. The downside of that is that there are an enormous number of papers I could point to. In no particular order, here are some interesting papers and articles, most of which are currently sitting open in my browser tabs:
“Eventual Consistency Today: Limitations, Extensions, and Beyond”, Peter Bailis, Ali Ghodsi. This article provides an excellent description of eventual consistency and
recent work on eventually consistent systems.
“A comprehensive study of Convergent and Commutative Replicated Data Types”, M. Shapiro, N. Preguiça, C. Baquero, M. Zawirski. This paper explores and details data types that work well for applications built on eventually consistent systems.
“Notes on Distributed Systems for Young Bloods”, J. Hodges. This excellent blog post succinctly summarizes the past few decades of
distributed systems research and discoveries, and also explains some implementation concerns we’ve learned along the way to keep in mind when build distributed applications.
“Impossibility of Distributed Consensus with One Faulty Process”, M.Fischer, N. Lynch, M. Paterson. This paper is nearly 30 years old but is critical to understanding fundamental properties of distributed systems.
“Dynamo: Amazon’s Highly Available Key-value Store”, G. DeCandia, et al. A classic paper detailing trade-offs for high availability distributed systems.
Paolo – Day-by-day Erlang becomes more popular. In your opinion what can we expect from Erlang in the future? What are the next goals the Erlang community should try to reach?
Steve - Under the guidance of Ericsson’s OTP team and with valuable input from the open source community, Erlang/OTP continues to evolve gracefully to address production systems. I expect Erlang will continue to improve as a language
and platform for building large-scale systems that perform well and are relatively easy to understand, reason about, and maintain without requiring an army of developers. In particular I’m looking forward to the OTP team’s
continued work on optimizing multicore Erlang process scheduling. The Erlang community is very good at proving how good Erlang/OTP is through the results of the systems they build, so they need to keep doing that to broaden Erlang’s appeal. If you’re a developer building practical open source or commercial software, the presentations given by community members at events like the Erlang User Conference and the Erlang Factory conferences are amazing sources of knowledge and wisdom for what works well for Erlang/OTP applications and what can be problematic.
Erlang Camp 2013 is coming!
Amsterdam: beautiful city of bicycles, canals and….. Erlang!
Nothing to do on Aug 30-31, 2013? What about travelling to the lovely city of Amsterdam and attend the Erlang Camp 2013?
If you have been following my blog for a while you should already know what Erlang Camp is: “an intensive two day learning experience focused on getting you up to speed on creating large scale, fault tolerant distributed applications in Erlang“.
In particular, during the Erlang Camp 2013 which is exceptionally sponsored by the amazing company SpilGames you will get in touch with several Erlang topics as:
- Erlang basic stuff
- Erlang OTP
- How to ship your Erlang code using applications and releases
- Erlang Distribution
More information on the Erlang Camp schedule may be found in this web page.
Erlang Camp is a pretty good way to learn Erlang language and to get in touch with some of the best Erlang teachers and developers outh there. Knowing that only 100 seats are available and that they will go quickly I suggest you to hurry and register for the event!
How to handle configuration in init/1 function without slowing down your erlang supervisor startup
Many times if you work with Erlang and follow the OTP design principles in your implementation, you may end up having one or more supervisors spawning a set of processes that can be either other supervisors or workers.
Most likely the child processes implementing the workers will be based on gen_server behaviour and whether you’re working on your small side project or in some big company project they will need some sort of initialization during the start-up phase: in fact creating an ets or mnesia table, reading a configuration file or accepting connections on a socket are pretty common operations that you want to be executed before the worker handles other messages and executes the operations connected to such messages.
According to the relative documentation, the gen_server process calls Module:init/1 to initialize and therefore the first strategy you may think to employ consists in doing the operations listed above within this function. What I mean here is something like:
start_link() ->
gen_server:start_link({local, ?SERVER}, ?MODULE, [], []).
init([]) ->
%% Some configuration operation here (e.g. handle ets/mnesia table)
{ok, #state{}}.
This kind of approach is pretty common when the operations we want to take during initialization are cheap in terms of time, but what happens if the initialisation is expected to take a long time?
Suppose you have a supervisor that spawns many children and each child has some long time taking configuration. The supervisor will probably call the function start_link/3,4 of each child in sequence and will not be able to return until Module:init/1 of the child it is starting has returned. This means that the supervisor won’t be able to start the next children on the fly and this will somehow slow down the whole supervisor startup phase.
How can we solve this issue? Well, there are a couple of different ways to do it, but all of them are based on splitting the gen_server initialisation into two phases, a first phase implemented in the init/1 function during which we trigger some internal message for a future configuration and that returns immediately to the supervisor and a second phase in which the configuration actually takes place. In such a way we can free the supervisor startup from the time burden of all the children configurations.
Let’s see with some code what are the most common ways to achieve this results. My favourite technique consists into triggering the future configuration using a gen_server cast inside the init/1 function as follows:
start_link() ->
gen_server:start_link({local, ?SERVER}, ?MODULE, [], []).
init([]) ->
gen_server:cast(self(), startup),
{ok, state{}}.
As you can see within the init/1 function we trigger a cast message to our process and immediately return. At this point we just need to handle the cast in the function handle_cast/2 and perform the needed configuration. This can be done in this way:
handle_cast(startup, State) ->
%% Do your configuration here
{noreply, State}.
A different way to achieve the same “two phases” result can be implemented as follows:
start_link() ->
gen_server:start_link({local, ?SERVER}, ?MODULE, [], []).
init([]) ->
self() ! startup,
{ok, #state{}}.
This time we first send the atom ‘startup’ to the gen_server and then we return. Of course we need to handle that message within our gen_server as follows:
handle_info(startup, State) ->
%% Do your configuration here
{noreply, State}.
As you can see the logic is pretty much the same here so I won’t go into further details.
The last way to achieve our result can be implemented by taking advantage of a timeout message. In practice in the init/1 function, instead of returning the tuple {ok, #state{}} we return the tuple {ok, #state{}, Timeout}. By including the value Timeout in the last tuple we specify that a ‘timeout’ atom will be sent to our gen_server unless a request or a message is received within Timeout milliseconds.
The ‘timeout’ atom should be handled by the handle_info/2 callback function. By setting the value of Timeout to 0 and adapting handle_info we can implement once again in an easy way our “two phases” configuration. Let’s see how this can be obtained:
start_link() ->
gen_server:start_link({local, ?SERVER}, ?MODULE, [], []).
init([]) ->
{ok, #state{}, 0}.
And the ‘timeout’ message can be handled as:
handle_info(timeout, State) ->
%% Do your configuration here
{noreply, State}.
Personally I don’t like the last approach because the atom ‘timeout’ is not so meaningful and it can lead to some misunderstanding. By the way the real problem here is that this approach is implemented taking advantage of an internal timer that should be evaluted: we can’t be sure that the message will be sent immediately, we just know that the message will be sent after at least 0 milliseconds.
Some reader here may say that this “two phases” approach is risky, because no one assures us that the configuration message will be the first message handled either in handl_cast/2 or handle_info/2. Actually this is not completely true.
We can’t be sure 100% that a message sent in init/1 using either a cast of the operator ! will be the first one in the process queue of our gen_server, but considering that start_link/3,4 is synchronous there are really few chances for another process to send a message to our gen_server before we send the configuration message.
Final consideration: there is a more elegant way to achieve the same result that consists in the combination of the functions start_link/3 and init_ack/1 of the module proc_lib. For those of you interested in the topic I suggest the user guide of ranch.
A mock APN server in #erlang
The number of apps people install on their smartphones is increasing day by day. One of the most interesting technologies a developer may employ when developing his own app consists in using push notifications.
Nowadays the biggest companies out there are providing some service in this sense: Apple has Apple Push Notification, Google has Google Cloud Messaging, Nokia has Nokia Notifications and Microsoft has Push Notification for Windows Phone.
Some time ago I had to work with Apple Push Notification, which does not provide a REST interface to the service as the other platforms but instead provides a protocol for exchanging streams between client and server.
Most of the stuff I had to deal with when working with APN was fairly easy and this is true specially because I was working with Erlang: I do believe that Erlang and in particular binary sintax can be really helpful when you have to build a packet following the Apple specifications, and in fact a lot of resources are available online in this sense as for example the tutorial you may find in this blog.
During my initial implementation and trobuleshooting I didn’t want to test my own software on the real APN server. Yes I know, you may use the sandbox for testing purposes, but if you are running a test with hundred thousands of notifications in order to stress your backend it is not a good idea to spam APN, moreover APN can be tricky some times so I did prefer to test my software on a controlled environment.
For this reason I implemented mockapn a mock server that provides very basic functionalities but that may be very helpful when you want to test your implementation of an APN client.
mockapn was built using the strategy proposed by Fred in the fantastic book LYSE: the supervisor (mock_apn_sup) starts a set of acceptors and whenever a connection is handled a new acceptor is spawned so that we have always n acceptors ready. Of course mockapn is built using ssl module and not gen_tcp, but still the implementation is almost the same, you have only to remember to change in the supervisor the values for certicate, key and password according to your needs.
Any incoming connection to mockapn is handled using a module (mock_apn_server) implemented on top of the gen_server behaviour: this module parses the incoming binaries looking for APN packets built using simple or enhanced formats and prints on screen the token and the JSON received.
Only for enhanced format mock_apn_server checks whether the token is invalid (invalid tokens are specified in the macro INVALID_TOKENS) and if so sends back an error message to the client and closes the connection. Since in my experience this is the most common error I did’t implement any other kind of error check in mockapn, but you are free to fork and adapt the code according to your needs. You may want to see the README file for more information on how to compile and run mockapn.
Now you may ask: “Paolo, is APN so easy as you say?“. No. some parts of APN are indeed tricky.
For example it is obvious that you can’t start an SSL connection to APN only to send a single notification and then close the connection. You can’t do it because:
- if you do it very often you may bother APN servers that may see you as a DOS attacker
- most of the times you want to send multiple push notification as a bulk over the same SSL connection.
If all your notifications in the aforesaid bulk are correct then no problem, APN will forward them to the clients and everybody will be happy.
But what happens if one of the notifications in the middle of the bulk is wrong? Well, for what I have experienced I may say that if one of the notifications is not valid (e.g., it has an invalid token) the connection will be closed and you will receive an error message (if you are using the enhanced format) but all the notifications sent on the same socket after the wrong one are lost in a limbo, and you will not know whether they were right or wrong. In Erlang this means that if you send in a fast way your notifications you will get an ok message all the notifications and eventually at some point you will get a message representing the fact that the socket was closed by the peer.
I suggest you to keep this in mind when you deal with APN, because only few websites cited this issue as for example this stackoverflow question, and this google project.
mockapn behaves in a similar way and you may test it using the file mock_apn_test.erl (which you have to compile using the old good erlc). In mock_apn_test I do send three notifications to mockapn (you have to change the tokens according to you needs), the second one should have a wrong token (i.e., one of the tokens you speficied previously in the list INVALID_TOKENS in mock_apn_server).
Running this code you will receive to the client an error message from mockapn for the second packet, but the third push will be never delivered and will be lost even though it was correct.
Ok, let me know if you need any more specific question!
Last thing! I want to add this nice youtube video I foud in an online tutorial (sadly I lost the link to it). In the tutorial the author states that coding a bullet proof APN interface is easy as the action you will see in the video 
An interview with Eugene Fooksman #erlang
. Today you can read my interview to Eugene Fooksman, a software developer at WhatsApp. As always I hope you will like it!WhatsApp Eugene?
nitely start with deep understanding of OTP principles and concepts. Supervision models can be a little tricky, but it’s worth spending time and effort figuring it out.
An interview with Knut Nesheim (@knutin)
Hello folks! I have already told you that I am currently working on a set of interviews with some of the speakers you will find at Erlang Factory SF Bay Area 2013. Today I have the pleasure to interview Knut Nesheim. Knut is widely known for some applications he created while working at Wooga (e.g., Elli and Locker). In this interview we will ask him something more about his Erlang experience, his famous Erlang applications and the talk he will give at Erlang Factory SF Bay.
Rock ‘n Roll? No, lock and rule!
Paolo – Hi Knut! Thanks for making yourself avaiable for an interview. Please, describe yourself in few words.
Knut - Hey Paolo, thanks for having me! I’ve been developing software for the last five years, in Norway where I’m originally from, Sweden and now Germany. I’ve always had an urge to understand how things work and computers provide endless opportunities for tinkering, hacking and exploring.
Paolo – You started as with an educational background in music and then moved to computers right? How come did you switch?
Knut - After computers, music is my second big interest in life. Music obeys “laws”, many quite intricate. It’s interesting to take apart a piece of music and understand how the parts work together to form something that can create emotion, give energy and inspire in people without them maybe even noticing, like in film. I find music for film to be very interesting.
It became apparent pretty quickly that I was much better at computers than music. I still play guitar and I regularly visit the Berlin Philharmonie. I just bought an electric piano and I want to start taking lessons again. It’s a nice hobby.
Paolo – I do believe that your main language before Erlang was Python. In your opinion what is the most difficult concept of Erlang for a developer coming from Python?
Knut - I was using Python quite a lot, yes. I was doing smaller web applications with lots of business logic.
The hardest part about learning Erlang for me was immutable state. I was used to thinking of data being stored in places and having different parts of the program storing new values in that place. Understanding that after I modify something, there are now two versions of it took some time. It’s a bit sad that it’s mostly inside of our programs we can have the immutable state. I find Datomic, the database from Rich Hickey to be very interesting.
Paolo – You are the author of Elli, an Erlang application I like very much. Can you tell us somethink about it?
Knut - Good to hear someone likes it! Elli is a special purpose webserver you can run inside your Erlang application to let your code speak HTTP with your clients. It’s not general purpose in that it lacks features you would take for granted if you’re for example making a web application. It’s a mashup of ideas from different projects, both within the Erlang community and outside, specifically it takes a lot of inspiration from Rack.
It has been a fun project to work on and I have received 34 pull requests to date, which isn’t much compared to other bigger projects, but I think it’s great to see so users contributing their improvements. It’s used in production in Wooga. Upcoming games are using it in very interesting ways. The traffic is absolutely crazy, with each online user making a request every 1-5 seconds.
Paolo – Why should we take a look at Elli? I mean, in the Erlang community we have already many Web Servers; when should we prefer Elli to them?
Knut - It’s true that there’s a bunch of projects already there, overlapping in quite big areas resulting in duplicated efforts.
I wrote Elli for the needs of Wooga, which was difficult to meet with the existing projects. Now, that’s not because they are bad in any way, lack features or are buggy. They are quite successful at making developers more productive.
When the volumes of traffic grow, so does the amount of weird input and amount of unexpected interactions in the implementation of the webserver. If you have an error that statistically speaking happens every 10,000 requests, at Wooga that error happens more than once a second. We found that developer productivity wasn’t all that important for us, it’s more important to focus on performance, robustness and operations.
A webserver that did less, we thought, would be less likely to have features interact in unexpected ways and would be faster from less code (and fewer processes!). Using the Rack-style of request-response where the handler function returns the response (as opposed to sending it directly on the socket), allows us to create middlewares. In the “web world”, middlewares are a common way to plug in third-party functionality, like an authentication middleware inspecting and injecting headers, a middleware implementing cookies or a middleware collecting statistics and exposing it over HTTP by overriding a url. It doesn’t fit everything, you can’t for example postprocess a streamed response.
If you are able to judge the trade-offs made by the different projects and you find Elli fits your needs better, I would encourage you to use it. I’m happy to help and committed to maintaining the project. If you’re not sure which one to pick, go for Yaws.
Paolo – During your career you have been working in many companies as for example Wooga and Klarna. How does it feel to work in some of the biggest Erlang companies out there?
Knut – Pretty cool! I really enjoyed my time at Wooga. I joined January 2011 as a “gun for hire” to help out with a experimental Erlang project, started by Paolo Negri. Together we built the first backend using Erlang, paving the way for adoption within the company. They now have 14 developers using Erlang with almost all working on upcoming games.
Now I’m heading up a very interesting project at Unity, the makers of the 3d game engine. It’s too early to talk publicly about the projects we are building, but I’m very excited about this opportunity. The team is starting to form and by the end of the year, hopefully we have something really cool to show.
Paolo – During the Erlang Factory in San Francisco you will give the following talk “Locker: Consistent Distributed Locking”. Would you like to tell us something about it?
Knut - Locker is a distributed locking service, implemented in Erlang and uses the Erlang cluster for messaging. It’s a multi-master key-value in-memory database, where a key expires if the lease is not renewed periodically. In terms of CAP, it will sacrifice availability when a quorum can not be made
It’s a hack, in the sense that it’s not an implementation of a “proper” algorithm from a paper. For example, coordination of writes happens in two phases, where if a quorum can be made in the first phase, the second phase writes the value on all masters. Some people properly educated in distributed systems tell me it’s good stuff, others tell me it ignores too many hard problems to be useful, like group membership. I’m looking forward to hopefully getting more feedback.
Paolo – How come Wooga needed something as Locker? What are the main benefits one may experience using it?
Knut - Just like with Elli, eredis and statman, locker started out as a frustration with what was currently available to us. In the stateful application servers we were building, we needed a way of ensuring that each online user has only one process. This can be solved in many different ways and we found ourselves constantly thinking about it. Our first version was just a central point which serialized all updates, but we wanted it to be available in case of partitions or more common to us, node failure, software crash and operator error.
We thought about this challenge for more than a year before we decided to tackle it head on. A recurring question was “what if we could do the perfect solution to our problem, rather than trying to shoe-horn existing solutions with complex hacks for scaling?” closely followed by “how hard could it be?” Eventually, I decided to just try it. I’m very happy with the result.
Paolo – How long did it take to have a working version of Locker? I find interesting the fact that such a piece of software if available for free on GitHub, don’t you agree?
Knut - From I decided to try building it to having a working version took around three weeks, out of which one afternoon was spent coding. The rest of the time I spent thinking, drawing and reading. Writing tests, async replication, various optimizations, etc, maybe a month of total effort has been spent over the last year, with contributions from others at Wooga. It’s just 330 lines of code, but coming up with those exact lines was very hard.
From developers to CTO and CEO, Wooga is pushing to open-source what can be open-sourced to hopefully contribute something back to the community. It took some time to convince everybody that locker was something Wooga should give away for free, even to competitors. I was the one holding back, but was eventually convinced by Jesper Richter-Reichhelm, head of engineering. It’s always scary to put something out there for everybody to see.
Paolo – Who should follow your talk and why? What are the basic skills a wannabe attendant to your talk should have?
Knut - If you’re interested in distributed systems, it might be interesting to see how locker was built and why it was built that way. If you’re an expert, you could come and help me improve locker by providing feedback on why it’s good or why you think it’s total crap. If you’re interested in the cultural parts of a successful game studio and how we build amazing things in a very short time, you could get a glimpse inside the inner workings. Knowing Erlang isn’t really necessary. The challenging parts of understanding locker is not in the code.
An interview with Evan Miller about Chicago Boss
Hello everybody! Thanks to Erlang Solutions I have been able to interview Evan Miller, one of the speakers you can meet at the upcoming Erlang Factory which will be held in San Francisco (21-22 March) . Evan is mostly known as the author of Chicago Boss. Let’s know him better with this interview! Hope you will enjoy it!
Meet the boss!
Paolo – Hi Evan, it’s really nice to have you here. Can you briefly introduce yourself to our readers?
Evan - It’s great to be here, and I’m glad to do it. I’m a graduate student in economics living in Chicago who has a passion for programming. People in the Erlang world know me as the creator of Chicago Boss, the
maintainer of ErlyDTL, and a loudmouthed defender of unpopular language features. Before grad school I briefly worked as an ops guy for Amazon.com and as a developer for IMVU, both out in Silicon Valley.
Paolo – How come a person with a degree in physics gets in touch with Erlang? Was it “love at first sight” or did you take a while in order to grasp it?
Evan - I stumbled across Erlang in a rather roundabout way. While I was involved in search operations at Amazon.com, I discovered that the source of almost all pain and suffering in the ops world can be traced
directly to blocking, threaded socket code. This led me to tinker with Nginx — a non-blocking server — but although I love programming in C (really!), I couldn’t stand the callback-based API. Shortly thereafter, a friend of mine at IMVU introduced me to Erlang, which is non-blocking… but without the callbacks! I instantly saw through Erlang’s syntax warts and fell in love with “what’s on the inside” – the absolute best VM for doing any kind of network programming.
As for the physics connection, I agree with Joe Armstrong — the creator of Erlang and who incidentally is an ex-physics person — when he says that Erlang’s process model comes naturally to anyone who has
studied relativity. Each process is its own “frame of reference”, in a manner of speaking — no process knows what’s on the inside of other processes, and heck, process clocks may be out of sync, but that’s OK!
You learn how to program around it and end up with much more reliable systems than ones that assume there is a “true frame of reference” of some sort. There can’t be, because you have to assume that *any*
server might fail or become unreachable in a networked environment.
Paolo – You are going to be one of the speakers at the next Erlang Factory held in San Francisco Bay area on March, what will be topic of your talk?
Evan - I am! The topic’s title is “Chicago Boss: a Web Framework Built For Comfort (And Speed)”. I’ll be giving a high-level overview of my pet framework and talk about some down and dirty details of why it’s designed the way it is.
Paolo – Who should attend your talk? Do they need to be expert Erlang or Web programmers?
Evan - Everyone should attend! I’ll try to make it interesting for both experts and novices. For example, I’ll explain why Chicago Boss is so darned good at fast server-side template rendering, which is something
that most languages have a lot of trouble with. I’ll also show some code samples to make things concrete.
Paolo – As the author of Chicago Boss would you like to introduce it briefly to our users?
Evan - Sure thing. Chicago Boss is a web framework that basically attempts to steal all of the good ideas from Ruby on Rails, and fix everything that’s broken: namely, slowwwwww templates, rampant RAM usage, lack of real-time messaging (WebSockets), and all this DevOps nonsense. So Chicago Boss combines fast templates with a clean API and a sophisticated “ORM”. I put that last bit in quotation marks because Erlang doesn’t really have objects, so we use a sort of fake object called a parameterized module that everyone on the Erlang mailing list hates with a passion, but which we love because it’s perfect for modeling data.
Anyway the framework is still in beta, but a lot of people are using it in production and we’re hoping to have a stable release soon!
Paolo – In your opinion, what are the main reasons that should make a web programmer switch from other platforms written in different languages as Rails, Symfony or Django to Chigago Boss?
Evan - Well, the first advantage is getting to work with a nice functional language like Erlang, even if CB sprinkles in some OO-style conventions here and there. Once you get the hang of it — which may take a while! — it’s generally easier to write correct code in Erlang than in non-functional languages.
But I think the real benefit actually comes down to operational costs — Chicago Boss runs smoothly on relatively little hardware and doesn’t need to be constantly restarted like most Ruby and Python servers. Erlang truly has a first-rate VM that is perfect for serving web pages. We’re really excited about running CB applications on ARM-based servers, which could really up-end the economics of server applications in the next few years.
Paolo – Chicago Boss in not the only option when talking about Erlang and web applications, what are the strength of Boss with respect to the others? Is there any feature you would like to “steal” from them?
Evan - CB is more of a full-stack framework than the others out there — in particular, it’s the only one that ships with any kind of database abstraction, letting you write your code first and choose from one of
six SQL and NoSQL database options later. There’s also an email server baked in, and an internationalization system — we’ve really tried to provide a complete package, whereas most of the other Erlang servers tend to be more bare-bones toolkits.
Don’t tell my mother, but I never hesitate to steal! But most of what I’ve stolen has actually come from the world outside Erlang. For example, the default template language is a tag-for-tag reimplementation of Django’s template language, and there’s now experimental support for the Jade template language.
Paolo – What kind of data storage support may we have with Chicago Boss? Do you recommend any of them, and if so why?
Evan - As I mentioned there are many options — in the SQL camp there’s PostgreSQL and MySQL; in the NoSQL camp there’s MongoDB, Riak, and Tokyo Tyrant, and then there’s an Mnesia adapter which isn’t SQL per se but it is schema’d. I don’t want to start a religious war, so I’ll stay quiet about my own preferences — but it’s really easy to add support for more databases — usually just requires a couple hundred lines of code.
Paolo – At the beginning Boss was implemented around Misultin, but later you moved to Cowboy. Was it difficult to switch the architecture? How many code did you rewrite in that sense?
Evan - For a long time, Chicago Boss supported only Mochiweb and Misultin; as we were pondering how to support Cowboy’s alien-looking API, someone had the brilliant idea to write a Mochiweb-compatible wrapper around Cowboy! It’s called “mochicow” and lets us adapt CB to use Cowboy in rather short order. Anyway, most of the hard work here is actually done by the folks over at Nitrogen who created SimpleBridge, which is an abstraction layer over I think six different web servers. It’s really a nice little library and we’ve contributed code back to it.
Paolo – One strength of Chicago Boss consists in my opinion in the good amount and quality of its documentation. Would you like to address your favourite online resources to a developer willing to learn more about Chicago Boss? Is there any “real” book about Chicago Boss?
Evan - As much as I love killing trees, there’s no book yet! The closest thing to it is a PDF tutorial on the website, and people are always adding new code samples to the Wiki hosted on GitHub. The best way to get started is to work through the tutorial, peruse the API documents, and hop on the mailing list. It’s an active, friendly place where we welcome in newcomers all the time.
My impressions about Learn You Some Erlang as a real book
Thanks to Fred Hébert and No Starch Press I had the chance to get a review copy of “Learn You Some Erlang for Great Good!” so I guess that somehow it is my duty to write in this Erlang blog my opinion about it.
When I started with Erlang some years ago it was not easy. I am a huge Erlang fan (I guess you noticed that) but for a guy coming from Java and C++ with no knowledge at all about functional programming, the beginning was not trivial. After reading a couple of good books (some of them are on the right panel of this page) I felt better: my Erlang coding improved but I still felt that something was missing…then I found *it*.
I remember the first time I landed to “Learn You Some Erlang”: I was desperate with some strange behaviour with supervisors and asking online I got a really good answer that pointed me to a new free online book about Erlang, so I gave it a try and clicked the link. I must admit that at the beginning I expected a totally different book…I mean, most of the free books you find online are usually incomplete and boring, so I guess you can understand how I felt when I saw the magnificent work Fred did.
For me, it was such a pleasure to read his work about supervisors that I switched back to the introduction and read in a few hours all the previous chapters…if you had the chance to get through Fred’s work you know what I am saying: the book is technical (of course!), but in it you can find not only the usual explanation of the language, you can find much more! The book contains a bunch of useful real life examples, a lot of theory that can be applied to any distributed systems and a thousand funny pictures that make me smile and enjoy the reading even more. You may think that since the book is available for free online (and for this I guess we should thank No Starch Press) there is no reason to buy the solid book for $49.95: well, you couldn’t be more wrong about it.
I can’t describe what I felt opening the envelope and turning page after page looking for my favourite parts of the book….it just felt good, like meeting an old friend I haven’t seen for a while. The book layout has slightly changed and a set of small arrangement have been done to Fred’s book, so the visual impact is even better and the reading is actually easier.
“Learn You Some Erlang” comes as paperback, but keeping it in your hands you can feel that it is built in a good way so that you don’t have to worry about chasing in the future some flying pages. You shouldn’t be afraid of its size as well: even though the book has more or less 600 pages, it is
light enough and you can keep it easily in one hand. Some random guy over there could say the images in the book are in grey level and not coloured as in the online version. They could also say that nowadays books are prehistory (I don’t agree) and that almost every developer has a tablet; the latter could be true but you should know that the book comes with a free Ebook version (with the solid book) that has colours, so if you want you can have it in your PC, tablet of smartphone.
Now, about the contents of the book there is much to say!
In its first chapters Fred helps you with the first steps in the Erlang world as for example its syntax and the basic principles of functional development. Further in the book, more interesting topics are covered, here you will learn about Erlang concurrency easyness and you will start understand why Erlang is so good for distributed and fault tolerant systems. But Fred’s work is not only for Erlang beginners: diving more into the book you will see that that are a lot of topics which are interesting also for experienced Erlang programmers as Releases, Common Tests and Distributed OTP Applications.
I guess I made myself clear within this posts: “Learn You Some Erlang” is a must-read-book every developer out there should add to his Erlang shelf. I recommend it without any doubt and I take this post as good chance to thank Fred Hébert once more for his great job and for the help he gave me without even knowing. I would also like to thank Fred and No Starch press for giving me the chance to review this amazing book.
So what are you waiting? Click here and buy your own “Learn You Some Erlang for Great Good” copy!
Talking again about Erlang with Francesco Cesarini
In 2011 I interviewed Francesco Cesarini for my blog. That was my second interview, a giant leap for my blogging experience. Today I am talking again with Francesco, mostly about his 2012 and an upcoming event organized by Erlang Solutions for all the erlanger in the USA: the Evening School of Erlang.
A 2013 full of Erlang!
Paolo – Hi Francesco! It’s good to have you here once again talking about Erlang. How was your Erlang year?
Francesco - 2012 has been a really good year. I spent a good part of it in the US, visiting companies using or looking to adopt Erlang to better understand their problems and needs. I taught the Concurrency Oriented Programming course at Oxford University both in the spring and the fall, alongside celebrating the 10th anniversary teaching at the IT University of Gothenburg. 600 served and counting. The best thing is seeing those who chose to work with Erlang after they graduated. Two EU funded research projects, Release and Paraphrase are sharing in excess of seven million euro. They have made excellent progress improving Erlang’s viability on multi-core architectures (we are talking tens of thousands of cores). A third project got funding in March and started in November. It is the Prowess project, taking property based testing to the next level. And Erlang Solutions is also doing very well. We expanded the offices in Krakow, moved to larger offices in London. There has been a huge interest for MongooseIM (our IM platform), our Erlang Management Console alongside lots of interesting courses and projects, all topped up and managed by great and dedicated staff. Record numbers at the San Francisco Erlang Factory and the Erlang User Conference, a lot more Erlang Factory Lites compared to 2011. It is clear that the adoption of Erlang is growing faster than ever before. I can’t wait for 2013.
Paolo – During the year many Erlang related events are held all around the world. In your opinion is this the best way to spread the Erlang word? How can we all help?
Francesco - The whole community is doing a fantastic job, so more of everything which builds the community is probably the right answer. We need more bloggers sharing their experiences, more people speaking at conferences and writing articles about their experiences. We need people to run user groups, self study circles, coding dojos and katas. We need better tools catering for the 21st century. Think cloud, embedded, massive scalability, with tens of thousands of nodes all interacting with each other. We need more research, more universities using Erlang as a tool to teach aspects of computer science such as distribution, concurrency and multi-core programming. And most important, we need to start speaking (and learn how to speak) to the CXO level. They are the decision makers who sit on the budgets and have a say on how and if Erlang is used out there. It is a very different language than that spoken by engineers. Instead of talking about cool debugging techniques or getting embroiled in the frames debate, we should be taking about financial benefits Erlang brings. How one reduces development, operations and hardware costs, making the software future proof towards future multi-core architectures.
Paolo – If you ask around you may find that many companies are afraid about using Erlang, mostly because they have a lot of know-how only about mainstream languages and fear the switch. How can we tackle this problem?
Francesco - I recently sat in a meeting where they were about to start a project with 100 Java developers. We suggested running a project in parallel, where we said we could do the work in half the time with 10 Erlang developers. It was a switching problem which had to handle peaks of traffic; Erlang was the perfect fit. The answer we got was, “If you are right, what do we do with our Java team”! This is fear, not common sense.
One has to introduce Erlang (or any emerging technology) starting small. Do a prototype or a proof of concept, show the advantages, and build on success. Mike Williams, co inventor of Erlang, include design by prototyping: In is not good enough to have ideas, you must be able to implement them and show they work. And make mistakes on a small scale, not in a full production project. When moving away from mainstream, at least in large companies, the conservative naysayers will be circling over you like vultures, waiting for the slightest hint of trouble. If you are 10% above budget, it does not matter if your budget is 1/10 of similar projects using mainstream languages. You are still over budget! Don’t give them the opportunity. But I am not answering your question, am I? My advise is to find your Erlang champion within the organization, make sure you are using the right tool for the job, get management buy in and start small.
Paolo – As many other Erlang developers, I am a member of the Erlang Questions mailing list; lately I spotted there a message from Erlang Solutions about an event called “Evening School of Erlang”. Can you tell us something more about it (e.g. locations and prices)?
Francesco - One of the biggest hurdles to adoption is lack of programmers. That was a clear concern when visiting companies on the US West coast. There are few experienced Erlang programmers commanding extremely high salaries and often changing jobs when new opportunities arise. To jump start the Erlang job ecosystem in the US, we decided to try out the Evening School of Erlang. Two nights a week, for four weeks, local trainers will be giving Erlang classes to developers looking for a career change. A few weeks after the course, those attending will be offered to take the certification exam. These classes will be heavily subsidized by corporate sponsors looking at recruiting/placing Erlang developers and will cost 199 USD. In exchange, your contact details will be shared with these sponsors.
In the new year, we are starting the courses on the 8th of Jan in San Francisco, close to Union Square, followed by Pasadena (Feb 19th) and San Jose (Feb 20th). Mountain View and Santa Monica will be announced soon. The goal for 2013 is to train 400 developers while expanding the network to cover Chicago, Seattle, Boston, New York, DC, and any other city where companies recruiting are willing to sponsor.
For those not interested in changing jobs but interested in learning Erlang, there is a corporate rate available which reflects the actual cost of the class. In this case, the details of the attendee will not be shared.
Paolo – Who may want to attend this kind of Erlang course? What are the topics and what is the Erlang level one must have in order to fully grasp them?
Francesco - Everyone who is interested in a career change and believes Erlang is the way forward. While we are hoping to attract experienced developers, students are just as welcome. Topics include:
- Introduction
- Basic Erlang
- Sequential Programming
- Concurrent Programming
- Process Design Patterns
- Process Error Handling
- Code Updating
- Records and Funs
- ETS tables
- Distributed Programming
- Advanced Constructs
- Ports and Sockets
- Style and Efficiency
Paolo – In the aforesaid Erlang mailing list message, there was some reference to a “Certification Exam”. Can you tell our readers what it is? Will the exam price be included in the course price? I guess Erlang Solutions will the entity that grants the certification, am I right?
Francesco - The Certification in Erlang provides validation of Erlang/OTP knowledge. We provide two exams, one for Erlang and one for OTP. At a corporate level this validation is important both for employer and employee. It allows for employers to identify potential employees who are suitably skilled, and provides an on-going upgrade path to suit an organisation’s staff development plans and career aspirations.
It is possible to sit the examination for certification directly following the class, but we recommend that candidates undertake some self-study and obtain work in addition to the course, prior to sitting the exam. It is not easy to pass.
The Exam for the San Francisco class will take place in conjunction with the Erlang Factory in late March. The cost is included in the cost of the class and is subsidized by the sponsors.
Paolo – Do you think this kind of course may be helpful in order to get an Erlang related job? How can one developer attending to course to get in touch with these companies?
Francesco - The Evening School of Erlang is being sponsored by companies looking to recruit Erlang developers. At the end of every class, sponsors will give a 5 minute presentation on what they do with Erlang and pay for a round of drinks. These companies will get in touch with you.
Paolo – How can company become a supporter for this kind of event? How to get more information about it?
Francesco - For this and other classes, we are looking for support in the following areas:
- Help in spreading the word to friends, colleagues and communities you believe might be interested in attending
- Sponsors interested in recruiting Erlang developers willing to help out cover the costs of the trainer and material
- Trainers interested in teaching the course
- Companies with training facilities all across the US seating 10 – 20 delegates
You are welcome to contact me via our website, where you can also find more information and register:
https://www.erlang-solutions.com/services/evening-school-erlang
Paolo – What about European Erlang developers? As far as I know you already started some courses in Krakow. How long should we wait for something like this to happen all around the Europe (maybe Italy )?
Francesco - This is entirely driven by market needs. Europe has been teaching Erlang in its universities for over 20 years! Thousands of developers have been trained, so the situation is not as acute as in the US. But if companies come to us and say they are having a hard time recruiting, we will certainly consider it.
You mention Krakow, where Erlang Solutions had to recruit. We tried out the evening school of Erlang back in 2008. The first time, 7 people registered. The second time, the number went up to 20, and the following year, we had 70 applicants. Word got around. The universities there started teaching it. They started doing research. And the yearly Erlang Factory Lite has for the last three years attracted in excess of a hundred attendees.
Paolo – I do believe 2012 was a good year for Erlang: a lot of events, courses and some new good book. Our world proved to be fault-tolerant to 21st december 2012, so What should be expect from 2013?
Francesco - Massive concurrent expansion of the language adoption. Lots of books are being written. I am co-authoring the follow up of “Erlang Programming”, tentatively called “Designing for Scalability with Erlang/OTP”. It will be published by O’Reilly. Joe Armstrong is working on the second edition of his book. On top of that, there are four other books in the writing which I am aware of but am unsure if I am allowed to speak about. My wish of having a whole bookshelf with books on Erlang will be realized this year. We are planing at least an Erlang Factory Lite per month, break the previous attendance records at the Erlang Factory in San Francisco and Stockholm and continue our work on educating and helping people. We will soon be launching a new community site aimed at replacing Trapexit. On top of that, Erlang Solutions will continue its international expansion and help grow the community. So all in all, if 2012 was an excellent year, we are really looking forward to working with the community in making 2013 an even better one!
Distance between two points on Earth in Erlang: an implementation of Haversine function
During my daily routine at Univesity of Trento I deal with different tasks in different programming languages. Lately I had to find a good way to compute the distance between two points on Earth which were identified by their lat, long coordinates. After a while, I decided to solve this task by using the haversine function.
The haversine (as known as the haversed sine), is a trigonometric function mostly employed in order to compute the great-circle distances between two points on a sphere, where the two points are identified by their longitude and latitude.
This method is strongly recommended in the literature for calculating short distances on Earth and in that sense it works great, but when we use it we have to keep in mind that it has a major drawback: the formula approximates the shape of the Earth to a sphere (actually it is an ellipsoidal), therefore it introduces some overestimation errors for trans-polar distances and underestimation errors for trans-equatorial distances.
If you don’t know how to implement the haversine function you will probably use your google-fu. The first web page you should check out is the wikipedia page for haversine, and the second one is this webpage. In the second link you may find a bunch of different implementations in several languages (e.g. Haskell, Python, C, Clojure etc) of the haversine problem…but wait! There isn’t an Erlang implementation!
Ok, let’s solve this problem right now! Here is my implementation of the haversine function for two points distance:
-module(haversine).
-export([distance/4]).
distance(Lng1, Lat1, Lng2, Lat2) ->
Deg2rad = fun(Deg) -> math:pi()*Deg/180 end,
[RLng1, RLat1, RLng2, RLat2] = [Deg2rad(Deg) || Deg <- [Lng1, Lat1, Lng2, Lat2]],
DLon = RLng2 - RLng1,
DLat = RLat2 - RLat1,
A = math:pow(math:sin(DLat/2), 2) + math:cos(RLat1) * math:cos(RLat2) * math:pow(math:sin(DLon/2), 2),
C = 2 * math:asin(math:sqrt(A)),
%% suppose radius of Earth is 6372.8 km
Km = 6372.8 * C,
Km.
As you may notice the implementation is fairly easy: first of all we use a list comprehension to convert our latitudes and longitues to radians. Once we get the values expressed in radians, we just need to extract the desired distance by using the haversine function.
Let’s try the code with the webpage example! Suppose you want to compute the distance between Nashville, TN, USA: (36.12, -86.67) and Los Angeles International Airport (LAX) in Los Angeles, CA, USA: (33.94, -118.40):
1> haversine:distance(-86.67,36.12,-118.40,33.94). 2887.2599506071087
Good! Seems working! You may find my implementation here (pick the right git branch), just if you don’t want to cut and paste the code from this web page!
Hope this will be helpful to you as it was for me!
