An interview with Fred Hebert (@mononcqc)
Hi there! Today you can read my interview to Fred Hebert. For those of you living on the moon for the last couple of years, he is the author of “Learn You Some Erlang for great good!”. In the interview we talked about Erlang, his book and distributed systems in general. Hope you will enjoy it as much as I did.
Answer you some questions for our good!
Paolo – Hi Fred! Let’s start with something easy: how would you introduce yourself to our readers?
Fred – I’m Fred Hebert. I’m a Québécois/French Canadian, and I’m a developer/programmer by accident, mostly self-taught. I’m mostly known in the Erlang community for having written Learn You Some Erlang for great good! I’ve spoken in a few conferences, spent a year training people in Erlang and writing course material. More recently, I’ve won Erlang User of the Year 2012, which I’m very happy about.
Other than that, I’m mostly unknown and plain. I’ve played bass guitar for years, although my work with LYSE has made me give it up in a large part the last two years. I’ve done improv, love to read, and so on. I’m rather cynical and verbose, I enjoy science fiction, music, documentaries, and watching Hockey. I’m studying part time in a local University (computer science, the equivalent of a minor) while working full time for BLOOM Digital Platforms, doing Real Time Bidding with Erlang. See http://ferd.ca/rtb-where-erlang-blooms.html for an explanation.
Paolo – What is your IT background and when did you start using Erlang?
Fred – This is a bit of a long story. I’ve never really known what I wanted to do in life. I mostly am driven by whatever I find interesting, in some kind of balancing act. At different parts of my life I’ve wanted to be a radio announcer, comedian, writer, musician, scientist (whatever kids would perceive as science), and so on. I decided to go in Multimedia, given it appeared to be a good way to be creative; I’d do web design and get paid for it, opposed to my views of radio, comedy, or music, where I’d work, but not really get paid for it.
The multimedia course introduced me to programming with basic JScript and PHP classes. That was an entirely new world to me, and I loved it. I thus started focusing on that, learning at night, until I landed an intership at a local dating site, the largest one in Quebec, with millions of members and thousands of them active at any point in time. I learned a lot there related to business processes, dealing with people, maintaining older software, refactoring, some scaling principles, and a lot more.
I had a good attitute, loved to learn and explore, try new languages, and this landed me a place in a tiny experimental Research and Development department. This was a small team with 3 people, one of which was on the company’s board. Because he was on the team, we were on the same level as the full executive board in the company’s hierarchy. The board member gave me the assignment to look into Erlang given Facebook were using it for their chat system and we were looking to rebuild one. Then, the board member left the company for unrelated reasons, and the other guy left on parental leave.
That left me alone as one-man department for 3 months, with nobody in the company able to tell me what to do because I was in an organizational black hole. I took these three months and a few books to teach myself Erlang. I was hooked on the concurrency model; it just fit in very nicely with the way I think. Eventually the board figured out my job was absurd, and demoted me back to integrator. I got bored and wanted out. I had no qualifications whatsoever, no paper to prove I was competent to be a real programmer, so I did two things: I started going to university part time, and I started writing LYSE.
I had a few chapters done and my name started to get out, so one day I sent Francesco Cesarini a message on twitter asking him if he needed anybody from Canada. He didn’t really need one, but they gave me a spot as the training course developer at Erlang Solutions Ltd. I was also to train people with the material I wrote for them. I did it for a year, enjoyed it a whole lot, but missed developing stuff. That’s when I got an offer to work at BLOOM Digital Platforms, and now I’ve been there for almost a year.
It’s been a large mix of luck and lots of hard work. That’s my background.
Paolo – You are some kind of celebrity among Erlang developers, mostly because of your book (which I find very inspiring). Why did you decide to write it?
Fred – As mentionned earlier, part of it was that I wanted to fix a few issues I had with Erlang while learning it, namely not really having a good online resource for free for it.
I began growing the idea of writing this after reading Miran Lipovača’s Learn You a Haskell for great Good! (LYAH) tutorial; I thought he did a great job with it. As I already knew him from online forums, I asked him how he felt about me writing an Erlang counterpart to his book. He liked the idea, being somewhat interested in Erlang.
A more complete list of reasons would be that I wanted to prove I knew stuff to get better jobs, I wanted to force myself to learn Erlang better, I wanted to know what writing a book would be like (I wrote plenty of bad tutorials before), needed some kind of programming-related project, wanted to help the Erlang userbase grow (so I could get jobs in it) and who knows, maybe I could get a dead tree book out of it, which sounded incredibly cool to me. Now it’s somewhat more normal, but things tend to look impressive and impossible up until you actually try to do them. Then it tends to become simpler as you learn how it’s done.
Paolo – How come did you write a free internet book instead of a “normal” book? I actually bet most of the erlangers out there would be willing to pay you some beers for this🙂
Fred – Because one of the reasons I wanted to write LYSE was to make it less painful to learn Erlang (no need to pay), it has been imperative for me to keep an online version for free. Writing the website version first would give me more leverage on that point if a publisher ever wanted to ask me to turn it in a book, as they did for LYAH.
My original knowledge of Erlang was limited, and I’m also rather bad at keeping focus on a single task for a long time. Having a public, incremental development process allowed me to get to get more reviews, and it kept me from giving up as it would make me look dumb.
That was a lot of pressure. Last of all, I don’t think I could get a book deal with my experience at the time without going public. Making things online for free and in a format that was alright for books just gave me a chance that at some point, someone would think “you know what? let’s get this printed” and hand me a contract. Which pretty much happened. It was a safe way out for me.
Regarding beers, I do not drink. I will however recommend that people offer beers to people who helped me a lot — reviewers and proof-readers. They’re in the FAQ (http://learnyousomeerlang.com/faq) for the site. I benefited a lot from LYSE: job offers, reputation, traveling, and friends. The reviewers are doing a very thankless and selfless work compared to me, and they deserve many thanks and beers.
Paolo – Which chapters were the most difficult to write for you as an Erlang expert? And which ones do you think are the most difficult to understand for an Erlang newbie?
Fred – I didn’t really consider me an Erlang expert until people started calling me that way. I always hated calling myself an expert because I invariably become full of myself, and I risk turning into a jerk, someone I don’t want to be. For most of the chapters, I had to do a big part of discovery on my own, although I did have a lot of help from the community at times.
There were 3 chapters I found particularly challenging to write. The first one is the one on exceptions — the one where I explain the difference between errors, exits, and throws. The challenge there comes from the fact the semantics for these 3 types of exceptions are loosely defined. I asked Robert Virding (who helped create Erlang) and Richard Calrsson (who added try … catch to the language) what the meanings were, and they didn’t agree. I figured out the semantics weren’t very clear, so I tried to hit a middle ground, extract what I read from the Erlang libraries, and push for a clearly defined difference in the chapter.
The second and third ones (the most difficult ones) were the chapters on releases and relups. That stuff is somewhat nasty. There are two concurrent systems, with very few people understanding them very well (this is getting better though). Their documentation seems to be coming straight from the halls of Ericsson and are hard to get into. People have a tendency to use rebar and blindly take templates without really understanding the systems behind it. I had a lot of decyphering, a lot of experimenting, and a lot of banging-my-head-against-a-wall to do about them. I’m glad I did it, though, I think it needed to be done.
The most difficult to understand for newbies are probably the Common Test and Mnesia chapters. They’re going to be harder to grasp and they’re long pieces of work. Oh and the FSM one, with the asynchronous trading system. The protocol is hard to test, it’s rather complex, and it’s another long chapter.
Otherwise, I’d guess all the chapters on concurrency are challenging because I fully expect any newcomer to skip over all the basics and jump in there, then get confused with all the weird features I assume they know by that point. They ignore them until I add more and more and finally it no longer makes sense. It’s a funny thing, and I don’t think it can be avoided. I’d probably do the same.
Paolo – I like the way you made it easy for people to understand a topic using different skeletons of “real life applications” for each chapter. How come did you follow this approach for your book? Was it difficult to link chapters to application?
Fred – It’s the one that always felt the best for me. As I said before, I’m mostly self-taught, and that means I can get to bang my head on the wall a lot when it comes to taking the examples in books and applying them in real life. I also hate “exercises left to the reader”. I tried to improve things by doing something I felt would have been better and that’s what it gave as a result. In the end, it’s nothing more than solving problems the way they would please me. It just happens that lot of other people have similar demands.
Of course, even the book examples are not enough to get by in real production setting — I’m constantly reminded of that by reader questions. It’s somewhat unavoidable unless you’re ready to put even more time in it. At some point you get a bit bored and want to move on, so you leave holes here and there.
Then there will always be people who hate my way of writing or teaching. That’s not avoidable. Fortunately there’s a whole lot of other books to help them. The more variety we have, the easier it is for everyone to learn in the way that’s most comfortable to them.
Fred – For web servers, I tend to go with cowboy.
Generally that’s a tough question, though. I started with web stuff, experimented with Erlang for the web a bit, but I do very little web development with it. I prefer to sit in the dark rooms of backend development, where my customer is another developer rather than a non-technical user. They act like a buffer between me and non-technical users, for whom what I do tends to look like black magic and I’m finding myself isolated from spec changes, design by committee, etc. far more than others. It’s easier to build on stable ground.
Being a front-end dev, or a dev that works close to the front-end is a thankless, difficult job where you juggle countless pieces of technology, bad protocols, people (who are always full of good or bad surprises), users with all kinds of software and weird behaviours. I try to stay away from it because it’s easier on my morale and keeps me from going entirely cynical. Props to people who love it and thrive in that environment, they’re impressive.
I’m not a fan of most web frameworks in Erlang right now, especially for larger websites. They’re almost all taking the monolythic approach of Ruby on Rails or Django where you develop things inside the framework.
The way I think web development should go is to write your entire application in Erlang, testable and independent from the web, and then plug it in a web front-end, or a framework that is almost just middleware for your app. That should make testing it easier, and extending it for more interactions also easier (say, mobile apps or non-HTTP front-ends).
We’ve got this rich framework in place called OTP, and a language that’s fantastic to handle everything server-side. We’ve got a piece of technology that should be great for these kinds of tasks. Then we take it all and shove it inside a framework inspired by what is done in other languages without the capabilities of Erlang for server-side stuff. I feel like using a monolythic framework that forces me to shove my design in there is restrictive. OTP is my framework. Web frameworks are tools to help me put my OTP stuff online.
I think we’ve got it kind of backwards, but I have no data to prove my ideal way would be better. It’s all speculation.
Paolo – You where among the partecipants of Spawfest 2012. Can you tell us something more about your project?
Fred – Yeah, the project is The Erl Next Door. I’ve worked on it with Malcolm Matalka. We wanted The Erl Next Door (tend) to be easy as hell to use for tutorial writers and people who want to demonstrate their libraries and applications on their own website.
We were tired of people having to explain how to install stuff. People always want a gem-like system for Erlang and we tried plenty of times as a community, but nothing stuck. We decided to explore things the way PLT Scheme (now Racket) did it with PLaneT (http://docs.racket-lang.org/planet/Using_PLaneT.html), something you can use inside your code and fetch applications dynamically.
We did make a couple of changes to how PLaneT works — we wanted something for tutorial writers and users first. We made it so any .erl or .zip file can be used to download a module or an OTP application, and get them built automatically no matter what tool they use. Then they’re loaded in memory and can be reused across sessions, although you can specify to dump the files in /tmp/ or something like that to make them impermanent.
Tutorial writers can just link to a zip or erl file anywhere in their code by adding the rel=”erlang-tend” attribute to hyperlinks to such files, or a tag in the header of the page. TEND can then scrape the page, find all code mentioned, and install it in one shot. I could then make Learn You Some Erlang have all its code downloaded in a function call (tend:load(“http://learnyousomeerlang.com/”)), or temas that have a given product could let you try it by doing tend:load(“path-to-my-tutorial’s-code”). You could make yourself a “web development bundle” for example, just by listing libraries in a webpage and sending TEND there.
A funny side-effect of making it loosely-specified like that is that it becomes rather trivial to write a repository system (like PLaneT) using this. For example, github and bitbucket both allow to download .zip files of any tags or the master/default branch by default. You could make your own repository while hosting no files, just linking to different sources.
In fact, we’ve made two demo sites for it during the contest: http://functional-orbitz.blogspot.ca/2012/07/erl-next-door.html for a blog post that uses it, and http://ferd.ca/tend/ for a tiny site that behaves as a repository.
As soon as I’m done with LYSE, I’ll be putting a bit of time into making a full blown repository for Erlang apps with a wrapper to make it friendly, entering the already overcrowded arena of dependency management in Erlang. Maybe this time it can stick. Worst case scenario is we have another bit of code nobody adopts. The idea of bundles have been left somewhat unexplored so far. Maybe that’s where I can make it fit in.
Paolo – What books, blogs and so on would you recommend to a young developer who wants to learn more about distributed systems architecture and development?
Fred – Good question. There aren’t many I know of. I stole a copy of the old “Concurrent Programming in ERLANG” (second edition) from my first job; the books are worth a couple hundreds of dollars now and they didn’t know they had it. The first part is free on erlang.org, but the best part is the second one. It shows a lot of the guiding principles behind Erlang’s distributed libraries, and it’s a great source for people who know programming, understand a bit of Erlang, but don’t know much about distributed programming.
Understanding the CAP theorem is paramount. I’d recommend the following blog post as a good introduction to it: http://www.julianbrowne.com/article/viewer/brewers-cap-theorem, and then I’d tell people to go read “You can’t sacrifice partition tolerance” (http://codahale.com/you-cant-sacrifice-partition-tolerance/) after that.
I would then recommend Amazon’s Dynamo paper. It’s an incredibly good paper to condense a lot of material and compromise in very few pages. It shows a crapload of great stuff.
Reading on “The Fallacies of Distributed Computing” is also a great idea.
I tried to synthesize all of that content in my Distribunomicon to make it easier to understand, but getting an intuitive understanding of it in any way at all is good.
Then, I’d direct people to try and understand vector clocks/Lamport clocks. They’re a great way to think about how time is relative on different machines or processes, and it helps gain a more intuitive understanding of things.
Once that is somewhat well understood (CAP is a theorem, not something you can ‘beat’, there are gradients of consistency, it’s not either CA or CP with no middle ground), looking into PACELC is interesting. It’s basically the cap theorem, extended. PAC is “during a (P)artition, do you pick (A)vailability or (C)onsistency”, and “(E)lse, do you pick (L)atency or (C)onsistency”. For this one, http://www.slideshare.net/abadid/cap-pacelc-and-determinism and http://dbmsmusings.blogspot.ca/2010/04/problems-with-cap-and-yahoos-little.html are decent introductions. Exploring PACELC is getting into more and more obscure parts of distributed system theory and you’ll find yourself having more trouble discussing it with people, but it always helps to know about it.
Paolo – Anything else?
Fred – Yeah, I’d like to announce that Learn You Some Erlang is now available for preorder at http://nostarch.com/erlang and on sites like O’Reilly’s store or Amazon. The paper book is in black and white, and the ebooks are in color.
No Starch Press have been pretty flexible with most of my demands and they’ve been fine with allowing a copy of LYSE to remain online. They’re very open people and understand new technology better than many publishers, in my opinion.
Also, if you’re ever in Quebec, get some Poutine.