Hello! Today in this blog you can read my interview to Gianfranco Alongi. Gianfranco is a Senior Software Developer at Ericsson with a strong passion for Erlang, Extreme Programming and Agile Methodologies. I am sure you will enjoy his answers
Erlang, TDD, XP, agile….what else?
Paolo – Hi Gianfranco! It’s a pleasure to have you here today. Can you please introduce yourself to our readers?
Gianfranco - Hi, and thank you! I’m honored to be interesting enough to be mentioned in this blog!
My name is obviously Gianfranco, but my full name is Gianfranco Franco Alongi, the middle name is a relic from the laws regulating concatenation of first names.
I have always felt a deep satisfaction in programming and problem solving involving abstraction through rule sets. I get a feeling of emotional purification when crystalizing a concept, or even a crude feeling of something into a formalization that can be analyzed, executed and mechanized. I can only imagine this is what Vulcan ’enlightenment’ feels like, when you see a clear path through reality, a sense of very vivid focus and understanding.
I am a Senior Software Developer at Ericsson, doing Erlang and other things, such as lecturing about good Design, Clean Code, TDD and Software Craftsmanship. I get paid to do Erlang, but my cognitive breath mint equivalent is APL, Haskell or CLISP. I have lately got totally hooked on APL, and will travel to Florida the 19th to hold a workshop on TDD and present a Testing Library I developed for the APL community.
To me, programming is a hobby, and I get paid to do this hobby during the day, but do my hobby without pay during the rest of the day (don’t tell my managers: but I would probably do the same thing even if I was not paid ; ))
My mind is to intellectual challenges what a husky is to physical exercise, I need it every day, and a lot of it. If I go without it for too long, I get restless and depressed.
Paolo - Would you like to tell us something more about your personal experience with programming and especially with Erlang? How did you start using it?
Gianfranco - Ah, yes. I got into contact with programming through my father. He worked as an IBM mainframe developer for many years, and we always had lots of books and stuff at home. He had a whole shelf with RPG, Assembly and instruction sets for different architectures. However, none of this was interesting to me. I got going with Quick Basic in 7th grade (I must have been 12 at the time), and wrote some useless stuff like text adventures.
From there, it moved on quite steadily, and in high school (Gymnasiet in swedish) – I had a math teacher who handed me a (gasp!) Pirated Copy of a Redhat enterprise edition. This is where I got serious with C++ for a while. I was now 15 and got a lot of help by this man. I knew that this was what I wanted to do! This felt so right. The programming.
When I later moved to Gothenburg, and studied at Chalmers, I met John Hughes, a very influential man with a keen sense for Functional Programming.
During this time I did a lot of Haskell, it was the main weapon of choice, but in one of John’s courses, we just happened to glance at Erlang and this was enough to catch my interest. I got hooked. And since that day, I’ve been doing it. For some reason, I have never felt that C, C++, Java, or C# has the right ‘feel’.
My cognitive mind has chemistry issues when it comes to those languages, but Haskell, Erlang, CLISP and now APL has been like hanging out with a really good friend. It’s a silent understanding between two friends. It is beautiful.
Leaving Chalmers, I was contacted by Ulf Wiger, and I started working for Erlang Solutions.
Paolo - You had the opportunity of being an Erlang developer in two of the biggest Erlang companies out there: Ericsson and ESL. Can you tell us something more about these two experiences?
Gianfranco - Erlang Solutions is a company with a lot of young talents and a lot of opportunities. You can be sent to Mexico, work with people in the USA, and basically see and do anything, as long as you are willing and able. This was a fun and interesting time, moving to London, leaving Sweden and meeting a lot of new friends. As ESL is mainly a contractor company, you get to work on a lot of different projects, this suited me well, and I loved the high pace. It is an environment where the fast movers and strong fighters can grow quickly, accumulating a strong CV with various technologies.
Ericsson is different from this. It is a corporation, thus, by default, the pace is lower. Much lower. Ericsson does not attract the passionate developers as well as smaller companies like ESL, and there can be a lot of bureaucracy which prevents the fast movements I was used to. However, Ericsson, being a large corporation, has the money to support and lift the passionate developers on its strong shoulders, and it can be a very different experience based on different managers. All my managers have been great, and value skill and craftsmanship, giving me the opportunity to act with a lot of freedom – and the accompanying responsibility to go with that freedom.
At the end of the day, it is us, the developers who form the culture we live in, the wrong people can make gold turn to lead even in the best places. If we are willing to change ourselves, we can do anything.
Paolo - In your current position at Ericsson you are also teaching Extreme Programming values and principles. Even though I know Ericsson has a solid story on agile methodologies I have to ask: is it difficult sometimes to advocate agile principles in such a big company?
Gianfranco - Yes.
Like most large corporations, the ‘Agile Transformation’ is first hand an adoption of ceremonies and not really about the governing values and principles. What should be a common understanding and discussion on values and goals, becomes quickly an act of doing the right things, instead of reflecting on why we do things, and how we do them. My belief is that part to blame for this, is that Agile is being pushed down from the top, and not being something perceived necessary from the largest part of the organization.
I have yet to meet a developer who is positive about Agile. Few have even heard the term Agile Software Development, and even fewer know about XP. We have a large legacy of Telecom experts, and a dwindling few who are Software Developer Experts. As a result of this, few care how things are done, the majority cares for doing things. And, as you know – the price is paid in sweat, bugs, and blood.
The only way to go fast – is to go clean.
Paolo - In the company where I work, Kent Beck’s book on XP is a must-read book. Beck states that XP helps developers to deliver better software. What do you think about this sentence?
Gianfranco - I agree.
My only remark is that we must never get stuck in one particular school of thought and shall always explore, educate ourselves and invest in our own career as Software Craftsmen. It is not only a matter of professionalism, but also a matter of survival. By learning new practices and principles from others, if I can reduce my own cognitive load by using a certain technique, and therefore spend more time optimizing for maintainability, it is definitely worth the time and effort to learn.
We all leave a legacy behind us, daily. If I take an extra hour refactoring my code, after passing all the tests and removing
duplication, it may seem like a large cost, but I know that in my unit, with ~200 developers, if I can reduce the time they spend
reading to comprehend it, there is an economy of scale that makes a good case to optimize for maintainability.
To me, XP is a lot about being a good developer with great practices, delivering value and being a pleasant man to work with.
The ‘team’ is very important to me, and I believe the best place to work at, is a place where you feel that you go to work with friends, the kind of people you feel would have your back in a Zombie Apocalypse.
Paolo - Another great book by Beck is Test Driven Development By Example. When I read it I really appreciated the part about eliminating the fear using tests. I know you also appreciate this approach, so how would you encourage a developer to start adopting TDD?
Gianfranco - My best tip to learn anything, is to do it during dedicated practice time. Children are fearless learners, they just do, and fail, and do, and fail. Eventually the feedback mechanisms allow us to tune ourselves, and we start succeeding more than failing.
With dedicated practice time, there is absolutely no pressure to succeed, and so, we loose the fear. Fear is the mind killer that puts the mind into an accelerated mode of anger and self pity, instead of the relaxed state necessary to tune properly on the feedback.
- Do a Code Dojo together with team mates
- or just book a 2 hour slot alone, at home.
Start by reading about TDD, and then choose a small practice problem, maybe a Kata or any other problem you can think of.
The main point is that it is not the problem in itself that is important, it is the practice.
It does not matter that you do not finish anything, is it the journey, and the act of noticing what you notice, that is the important bit.
Once you feel that you have a grip on TDD, then you can apply it on production code.
Because if you have to get stuck, if you have to fail, do it in a safe place, don’t do it during work time.
Paolo - I appreciated very much your blog posts about TDD and Erlang. What are in your opinion the best tools an Erlang developer can use for testing his code? EUnit? Common Tests? Something else?
Gianfranco - I do not have a favorite, to me they are tools which apply well for their respective tasks. The only thing I am very careful about, is the feedback time the tool gives me.
I always use EUnit for Unit Testing.
Then I can use Common Test for Function Testing, but if the specific test case would take 20 minutes to get into place properly with CT, and I can do the same thing in 2 minutes using a bash script with netcat, then I go for the speed choice.
In my unit, we have a different tool which is based on top of CT, but with it’s own syntax. Mostly I try to avoid it, and the straw that broke the camels back, was when I could test a certain use-case by manual configuration and some bash’ing, in just under 10 minutes, and we had tried achieving the same thing using this tool, for 3 days.
Paolo - What do you think about continuous integration and continuous development? Do you recommend any specific framework or service for Erlang code?
Gianfranco - I think they are necessary practices that give a lot of value and make life easier. By exposing complexity early, and dealing with stress up front, we remove the pressure of doing integration and delivery later in the project.
No recommendation, sorry.
It is the doing of the work that exposes what needs to be done.
Paolo - Last question: emacs, vim or what else? :D
Gianfranco - Emacs, and vim when there is no emacs. I use this together with Tmux, and the no-mouse environment I use, enables a very high bandwith between my brain and the computer.
 Star Trek Reference
 such as TDD
 Yes, I did it, I stole the quote ; )
Hello! September 2013 has come and here I am once more to give you some information about another year of Erlang blogging
Would you like to know something more about what I posted in this blog during 2013? Are you sure? Well, just keep on reading!
I can’t recall every single post I wrote this year, so I have to ad lib during this summary.
Since I noticed that you readers appreciate my interviews to famous Erlangers I kept on writing this kind of posts. I think nobody will be offended if I say that two of my favourite interviews this year were the one to Kenji Rikitake and the one to Steve Vinoski. I obviously want to thank all the others Erlangers that accepted my request for an interview and I hope that you had the chance to read at least some of their inspiring ideas and thoughts!
As promised during 2012 I tried to transform this blog into a space where Erlangers can get useful information about Erlang projects, events and books. In this sense I had the opportunity to write my impressions about the book “Learn You Some Erlang” (thanks No Starch Press) and my thoughts about “Erlang by example with Cesarini and Thompson” (thanks to O’Reilly).
Some posts were as usually more technical, in this sense I enjoyed very much writing the one about TDD and kata, the one about the cost in terms of time you have in your supervisor when spawning multiple gen_servers. I also liked sharing with you the most interesting Erlang articles I read online lately in my recent post.
That all! As always I want to thank all you guys for your support and patience. Hope you will keep on following my blog during these last months of 2013 and during 2014! Have a nice Erlang year!
Even though I haven’t been writing in this blog very much during the last moths, I had the chance to read many new (and old) interesting blog posts, articles and tutorial about Erlang.
Some of the stuff I had the chance to read was really well done and inspiring, therefore I would like to share these contents with you. In this blog post of mine I am going to make a list of the 4 articles that I loved most, with a brief description for each of team. Let’s start then!
Interesting Erlang stuff you should take a look at!
Here is my brief list; as I did in one of my previous blog posts I would like to remember that the order of the links is totally random!
- Create and Deploy Your Erlang/Cowboy Application on Heroku in 30 Minutes: In this blog post, Roberto Aloi explains how to create and deploy easily a web application written using Erlang and Cowboy. I liked this article very much: Roberto is one of the most famous Erlangers out there and even though the application he presents is really easy it’s always a pleasure to read some of his code. Moreover the part related to deploy focuses on Heroku the well known cloud platform, showing how easy it is to deploy our Erlang application there.
- Continuous Integration for Erlang With Travis-CI: continuous integration (CI) is a powerful tool. While I was searching for a good CI service to be used with one of my Erlang projects I ended up reading this nice blog post by Ward Bekker. In his tutorial Ward explains briefly what are the main tool we have in Erlang to automate our tests and teaches how to connect a public GitHub repository to Travis-CI. I really liked this blog post, and if you are new to CI I suggest you to read as well this article by Martin Fowler: Continuous Integration.
- Meck and Eunit Best Practices: TDD and testing in general are one of the things I am trying to learn more in these months. Even though this post by David Reid is not very recent (2011) I suggest you to read it if you want to learn more about Meck, the most famous Erlang mocking library written by Adam Lindberg. Once more, if you are new to mocking and so on, I suggest you once more to read a great article by Martin Fowler: Mocks Aren’t Stubs.
- A Week with Elixir: That’s true, Elixir is not Erlang, but I think we all should read what the great Joe Armstrong thinks about this new interesting language written by José Valim. In his blog post Joe gives his impressions about Elixir and its syntax, providing code samples and really wonderful insights. If you want to know more about Elixir you can go to the official web page or consider buying this book by Dave Thomas.
And that’s all folks! I hope you will enjoy these stuff as much as I did!
Hello there! Today I want to introduce you my interview to John Koening. John is a PhD student at University of Minnesota in the fields of distributed and real time simulations. John is also working in the game studio he founded. Currently they are developing the game The Electric Adventures of Watt which has some Erlang in it.
Learning something more about Ymir
Paolo – Hello John and welcome to my blog! Can you please introduce yourself to our readers please?
John - Hi Paolo, thanks for having me. My name is John Koenig and I am a PhD student at the University of Minnesota (UMN) studying distributed, real-time simulation. I am going into the third year of my PhD program preparing for my written and oral defenses. I have been a regular Erlang-user for about 6 years.
Prior to, and inter-mixed with my time at UMN I worked at Cray Inc. Most recently I was contracted under Cray’s Chapel team where I worked on several language improvements in the area of portability. A majority of my time at Cray was spent as part of their Custom Engineering initiative. Together, we engineered unique super-computing platforms and software stacks for various customers.
In 2010, I founded a game studio, Called Shot LLC, with my good friends Gabriel Brockman and William Block. We are currently in the first round of funding for our flagship game title: The Electric Adventures of Watt.
Paolo – This is a common question I ask during my interviews: how did you start using Erlang? What are the features of Erlang that made you learn it?
John - I was first introduced to Erlang while pursuing my undergraduate degree at University of Wisconsin – Eau Claire (UWEC), I think it was around 2006. As part of a Programming Languages course we were tasked with picking a new language and implementing a solution to a sufficiently interesting problem which applied to the language’s domain. At the time, I was big into Plan 9 and distributed software in general so I chose Erlang and implemented a distributed prime number sieve.
Being more of an applied school, UWEC had me spending most of my time programming C and C++ and I remember being really impressed with how Erlang modeled processes and inter-process communication directly in the language. Once I got past Erlang’s syntax learning curve and my newness to functional programming, I found myself able to express distributed solutions very naturally in Erlang. After that, I was hooked. I picked up Joe’s book, Programming Erlang, and started keeping up with the Erlang community online.
I first started using Erlang professionally at Basho in early 2008 when I was brought on as a Reliability Engineer. I had thought, coming out of UWEC, that I knew Erlang fairly well, but I grew considerably during my six months at Basho. Justin and his development team are incredibly talented and being around that level of skill and enthusiasm was highly contagious. I remember that time fondly.
Paolo – Would you like to introduce and describe in a few lines what Ymir is? Where will Ymir be used?
John - Ymir is an open-source (GPL), cross-platform, distributed 3D game engine written in Erlang.
With the number of cores available to gamers on the rise, it is Ymir’s purpose to break games out of the traditional, single-core-dominate game-loop and, in doing so, achieve faster, larger simulations which grow in proportion to the number of available cores.
Paolo – Why did you decide to use Erlang for Ymir? Was there any other candidate language at the beginning of the project?
John - Ymir grew out of a desire to create a multi-player RPG that got away from the traditional client/server model. Myself and a few friends enjoyed online RPGs but didn’t enjoy the MMORPG scene. We were interested the approach of Neverwinter Nights 2, however, which featured smaller worlds developed and hosted by members of the community. Hosting of these worlds could get terribly expensive, as the worlds were hosted on a single server and, as their player base grew, admins of these worlds would be required to either co-locate their server or pay for expensive home internet access with sufficient upload speed. I set out to change this, wanting instead to see a game capable of simulating a world in a more peer-to-peer fashion. Namely, a game engine capable of utilizing the additional computational power and bandwidth present as players login to enjoy the simulated world.
I didn’t consider anything other than Erlang for this task. Along with OTP, Erlang is still the best language for distributed development as it allows me to focus more on the high-level challenges of distributed real-time simulation and less on the gritty details of implementing my own task-queues, inter-process communication, etc. This choice was further cemented when I proved that communication to port drivers, with minimal trickery, was sufficiently fast to support online rendering.
Paolo – Reading your paper I spotted many words often used among Erlang developers: scalable, soft-thread, message passing and minimal amount of synchronization. Would you like to discuss the meaning of each term with respect to Erlang and Ymir?
John - Game engines are traditionally frame-centric. Their primary goal is to compute and render frames as quickly as possible. Two aspects make this approach difficult when scaling over multiple cores: first, computing a frame is recursively dependent on the frames which came before it and, second, traditional spatial data structures used in collision detection require all game entities to be synchronized in order to function.
Ymir takes an object-centric approach and aggregates frames as quickly as possible. Game objects (entities) are represented as Erlang processes (soft-threads) and each entity is responsible for simulating itself locally. Discrete events (e.g. collisions, user-input) are modeled as messages which occur at a specific point in simulation time. As an optimization we allow entities to exist at various points in simulation time and to resolve events in the recent past by application of timewarp. Entities proceed through their local simulations, streaming updates to their physical state to relevant renderers. To enforce fairness, a sense of global time is defined as the minimum of all entity simulation times. This introduces a small amount of global synchronization as entities “vote”on the value of this global time through various shared ETS-based counters.
To break away from traditional spatial data-structures, Ymir applies map/reduce to spatial reasoning in order to achieve scalable collision detection. When simulating forward in time, entities volumetrically hash their physical extents against a fixed cube to various buckets (also soft-threads) and aggregate contacts which result from writing their latest physical states into each selected bucket. The act of mapping to spatial buckets is analogous to selecting nearest-neighbors (broadphase) and the buckets themselves compute points of contact for each pair of entities overlapping within its given volume of simulation space. In short, Ymir serializes only those objects which are sufficiently close together while permitting objects sufficiently separated in space to simulate unimpeded.
Using map/reduce in this fashion allows Ymir to scale-out over many cores very well. Currently, we are able to realize ~11x speedup in overall simulation time on 16 cores and sustained frame rates of ~500 fps. Ymir’s performance is dependent on many factors, however, chief among these is the degrees of freedom between entities. As entities are serialized based on spatial proximity, scenes where all entities exist in persistent contact are currently unable to obtain such lofty speedups. I am currently expanding our methods to better model persistent contact which will help Ymir obtain better speedups in these scenarios. Furthermore, using map/reduce to compute contacts works well locally or over low latency networks but as we scale up to many machines connected with higher latency other approaches will be needed. I am currently investigating network overlays between entities which capitalize on spatial assumptions present in game simulations.
Paolo – Do you have any partial results about Ymir our readers can take a look at? What kind of tests do you do on Ymir?
John - We are actively maintaining performance results on Ymir’s indiedb page, and as time permits I will be documenting Ymir more completely on our development blog.
This video showcases the three testing scenarios we used to gather our preliminary results. All three scenes are rendered offline using Ymir’s built-in support for Mitsuba. Parallel is rather boring to watch, but provides a best case performance. Cylinders features a stack of spheres falling onto an static array of cylinders and is more representative of the types of rigid body interactions one might see in an interactive game. Last, is Bounce in which spheres move randomly within fixed scene boundaries. Bounce is currently being used to measure Ymir’s performance as it relates to scene density.
Paolo – Is there any way our readers can contribute to the development of Ymir? Is there any fund-raising? Can other developers join the project?
John - Glad you asked, yes! We are currently on indiegogo seeking funding for The Electric Adventures of Watt which will be powered by Ymir. In supporting The Electric Adventures of Watt, contributors will be directly helping us mature Ymir into its first public release.
We will be advertising the public repositories for Ymir concurrently with its first official release. In the meantime, if developers are interested in working on Ymir, please, don’t hesitate to get in touch with me: john calledshot.org.
Paolo – You are a PhD student at the University of Minnesota and your studies are mainly focused on parallel and distributed real time simulation. Do you think Erlang could be widely used in these fields?
John - Without a doubt, that is what is what Erlang was designed for. There is even a precedent for using Erlang server-side for games: MuchDifferent and SMASH. I feel that as CPUs continue to have more cores and affordable CPU accelerators (i.e. parallela) become available, game developers will turn to solutions like Ymir to grow their games. Scalable, real-time simulation is not an easy undertaking and savvy developers will be looking for the right tools for the job.
As many Erlang enthusiasts know, there are at times substantial resistance to using languages that are outside of other developer’s comfort zones. This is especially true for academia. I can’t even count the number of times I have had to defend Erlang to my lab mates at UMN. That said, we are not expecting that all developers wishing to use Ymir will embrace Erlang, and we have on our roadmap to develop front-ends for languages most game developers will find familiar: C/C++/Lua.
Paolo – Did you find any help in the Erlang community? Did any Erlang developer give you feedback or support online during your development?
John - Several times I got frustrated with the performance of Mnesia and ETS for Ymir’s collision detection and I turned to the Erlang IRC channel for support and guidance. The Erlang community has been nothing but insightful and supportive every time I have turned to them. Although for the life of me I cannot remember that handles of those who offered help, I owe the Erlang community thanks.
We also owe special thanks to you, Paolo, for featuring Ymir on your blog, Peer Stritzinger for helping us reach out to the Erlang community, and to your readers.
Hello there! In this post you can read my interview to Kenji Rikitake. Kenji is a famous Erlang developer and security expert. I really loved this interview because Kenji provided some really interesting anectodes connected to his personal life and many insights on the IT in Japan.
The Erlanger from Japan
Paolo – Hello Kenji! It’s great to have you here! Please, can you describe yourself to our readers.
Kenji - My name is Kenji Rikitake. I am a relatively new user and programmer of Erlang; my experience is only about five years.
I’ve been working on various aspects of internet and distributed computing for 25 years. I started as a intern of VAX/VMS sysadmin in 1987. A couple of years later, I became a VAX/VMS Asian screen management library programmer and the product tester in 1990 at Digital Equipment Corporation Japan.
After leaving Digital in 1992, I decided to start my career as an internet sysadmin, or “devops” in the latest trendy word, and a volunteer evangelist of explaining how internet would change the world. I worked for a systems integration company called TDI, and co-designed and implemented a corporate firewall with BSD/OS systems and dedicated routers, including a simple fault tolerance. The firewall system was running until 2000 when I left the company. I’ve also written two books about internet engineering and technologies in Japanese.
From 2001 to 2005, I was a researcher at KDDI R&D Labs, about network security on intrusion systems, DNS protocol, and teleworking. During the period, I also conducted a joint research with Osaka University as a PhD student. My PhD thesis was about DNS reliability and security.
From 2005 to 2010, I was a researcher for National Institute of Communications and Information Technology (NICT), a research body of the Telecom Ministry of Japan. I involved in the preliminary design of a network intrusion early warning and analysis system called “nicter”, and later I pursued the DNS reliability research especially on the behavior of DNS packet fragments. I also worked in IPv6 and NGN security.
After I met Erlang/OTP in 2008, my research interests have shifted into the concurrency programming and the various related issues, including security, efficiency, and the robustness. Distributed database design is my latest research topic, for the obvious reason that I am currently working on building Riak. I’ve presented four talks at Erlang Factory SF Bay Area from 2010 to 2013, one for each year.
From 2010 to 2012, I was a Kyoto University full professor, though my primary role there was to implement and supervise the campus network security policies and procedures. I worked on two Mersenne-Twister random number implementations for Erlang, called SFMT and TinyMT, which are published in ACM Erlang Workshop 2011 and 2012. I also organized the 2011 Workshop held in Tokyo, as the Workshop General Chair.
I’m currently working for Basho Japan, a Japanese subsidiary of Basho Technologies.
I’m an electronic geek, and my Twitter handle @jj1bdx is derived from my primary ham radio call sign in Japan, which I’ve been assigned since 1976. Morse Code on the shortwave is one of my favorites on the radio, though from 1986 to 1990 I also involved in the packet radio activities
based on TCP/IP. Music is another thing that makes me happy.
Paolo – First real question: how did you meet the functional programming world?
Kenji - I first read a Lisp book in the early 1980s when I was a teenager. I was not that interested in the S-expression though, because I didn’t have an execution environment then. It was even before the C language for the personal computers; I was playing around with my Apple II, mostly in the assembly language, and two tiny programming languages called GAME and TL/1. I even wrote a GAME compiler for 6502 running on Apple II.
Before starting my real career, I was a lab member of Professor Eiiti Wada from 1988 to 1990, at the University of Tokyo. Prof. Wada and his lab members created a Lisp implementation called UtiLisp, and the lab was the most advanced place in the campus networking. I was also learned some basic ideas on the functional and even logic programming, because of the nationwide buzzword called The Fifth Generation Computers. Some of the Wada lab alumni were the key designers and implementers of the language called Guarded Horn Clauses, which has surprisingly similar design philosophy to Erlang, although it is a logic programming language.
My problem about understanding functional/logic programming was, however, that I couldn’t really grasp the core reasons why those programming paradigms were effective and even required for a large-scale system design. I failed on a Prolog course in 1989 either because I didn’t find the unification principle was anything meaningful. So I was a very bad student. I wish I could have learned it in the Erlang way of the pattern matching then!
And unfortunately my mind in the late 1980s was too focused on how to run UUCP and email systems in an inexpensive way without UNIX, so any functional or logical programming paradigms seemed redundant to me, because they were so slow. I didn’t like regular commuting from my home to the university, so I wanted a way to discover a way of working from home. At that time my main target of code hacking at home was MS-DOS then; I had to wait until 1993 when I could use BSD/OS at home for experiencing the real UNIX at home. I later moved into FreeBSD in 1997. And I’ve been running Erlang/OTP mostly on FreeBSD since 2008.
Paolo – And when did you first hear about Erlang?
Kenji - I first saw a Japanese translation of Joe Armstrong’s “Programming Erlang”, published by Ohmsha in November 2007, at a bookstore I visited in Tokyo downtown in February 2008, on my way back home from Tokyo to Osaka. I instinctively found out this was the once I had to learn and go for, so I immediately bought it and started discovering the world of Erlang since then.
Paolo – You told me that you had some bad times during your experiences as developer and University Professor, but also that Erlang and functional programming helped you to overcome your difficulties. Can you tell our readers something about that?
Kenji - Let me start from my programming middle-age crisis first.
I was looking for something completely new and innovative for a programming system to learn, after I thought working only on C was no longer sufficient to keep myself up as a modern programmer. I was sick and tired of understanding and modifying the BIND 9 DNS server code, written mostly in C, for a DNS research paper I was writing then. I don’t blame the BIND 9 programmers because it does really complex magic things, and I admire ISC people especially Paul Vixie, one of my mentors in Digital Equipment and the father of BIND. Nevertheless, having to read hundreds of header macro lines to reach the actual code looked no longer practical to me at that time. And I thought I would have lost my competitiveness as a programming person then, if I stuck into the old way of C programming. So eventually I become a polyglot programmer; I use C, awk, Python, Perl, and Erlang.
I knew multi-core or massive-parallel computing hardware is coming and I wanted to learn something very much different from the past sequential and inherently procedural programming languages and systems. While Erlang is *not* specifically designed for a massive-parallel execution environment, Erlang does have a lot of practical constraints for modern computing hardware requirements embedded in the language, for example the single-assignment variable principle, and the OTP system themselves, for example the gen_server behaviour [sic] framework, to solicit the programmers to do the least wrong things. This is something which other languages cannot emulate or mimic.
Next about the University Professor life crisis.
During my Kyoto University career, most of the things I had been doing there was talking, negotiating, and dealing with people, not with computers. The university is a very large organization, and keeping the campus network secure is something practically impossible without the university member’s help, namely from the administrative, education, research staff, and of course from the students. I am an introvert person and most of the university people are not geeks although many are excellent researchers, so the human communication tasks were the toughest thing to do in my life. Also the long-time commuting from my house to the office, spending four hours in total every day, literally killed me.
Fortunately I was allowed to do the CS research activity, however, during the Kyoto University career. And I was eligible to run a large batch jobs on a large Linux supercomputer cluster. So I decided to run some Erlang code and do the fun things over there. One good thing about Erlang is that it is mostly OS independent, so I did the prototyping on my home FreeBSD machines, and let the huge multi-core jobs run on the cluster. I’ve put the research result into GitHub. So I didn’t have to throw away the possibility of my career as a CS researcher
Paolo – You are widely respected not only for your knowledge on Erlang/OTP but also for your expertise on distributed system security. What is the intersection between these two fields?
Kenji - Erlang/OTP is a very good candidate for making a reliable system. This means it would be a prospective candidate even for a secure system, if properlydesigned. In other words, an unreliable system could *never* be secure. And every system is not 100% reliable.
The word “security” has a lot of implications in many different aspects, and is widely misused in many contexts, even if I exclude the militaristic and socialistic implications, which may be out of scope of this interview, though very serious issues themselves indeed.
I believe that the foundation of a secure system is a reliable and fault-tolerant system. This has been frequently ignored even by many “security” experts; for many of them, security is only about cryptography, or about restricting the user’s behavior in a system, or just about analyzing the behavior of the pieces of malware. I do not deny those aspects, they are very important, and the outcomes of those research activities are surely essential for making a better computer system, but those aspects are not only *the* security. A very broad perspective is needed for a computer security expert.
Also, I have to stress that security is mostly about people and how people behave. People want convenient systems; at many circumstances, security and convenience do not coexist. For example, if you really want a secure system, do not connect it to the internet. But such a special system, which could enable you to provide sufficient communication capability within the system while rejecting all the attack thwarts and zero attack vector, is virtually impossible to make, from the financial point of view. See the Stuxnet case? Consider what if the power plant were using Erlang/OTP as the core and the end-point controllers.
I wish Erlang/OTP developers always think about making a reliable software. It’s not that difficult; thinking carefully when programming will solve most of the cases.
Paolo – What is the best way to “secure” an Erlang distributed systems?
Kenji – Traditionally, putting the whole systems in a protected network, is the only solution. And unfortunately it has still been so.
This is a very good question to answer, because in the current Distributed Erlang (disterl) system on the OTP, the security model is very weak if any existed. TLS-based disterl (with the ssl and crypto modules) will be a good solution to protect the communication between BEAMs, but the problem is that the communication between BEAMs and the port-mapper daemons are plain text and it’s not trivial to incorporate necessary authentication and cryptographic features.
Erlang/OTP has been depending on the assumption that the whole disterl cluster is in a protected network without any attack vectors. In other words, the disterl cluster itself was considered a system without protection. Opening the communication ports to the internet, however, makes this assumption rather unrealistic; the Erlang/OTP devops must think about all the possible attack vectors for the disterl cluster as a whole system.
One possibility on protecting BEAM-to-BEAM communication is to establish cryptographically authenticated links between the BEAMs and let the links be used persistently, with proper periodic re-keying, without using any port-mapper daemon. I believe incorporating such a facility into Erlang is not that difficult, though the rendez-vous problem between the multiple BEAMs should be solved in another way.
Paolo – During your experience as Professor at Kyoto University, you did also research activity using Erlang and OTP. You worked in particular on SFMT and TinyMT. Would you like to introduce these two projects to our readers?
Kenji – Mersenne Twister (MT), a BSD-licensed innovative long-period (typically 2^19937 – 1) non-cryptographic pseudo random number generator (PRNG) by Profs. Makoto Matsumoto and Takuji Nishimura, has become the de facto standard on popular programming languages such as Python and R. SIMD-oriented Fast MT (SFMT) and TinyMT are the improved algorithms, by Profs. Makoto Matsumoto and Mutsuo Saito. The MT algorithms have all a very high order of equidistribution, which fits very well on a large-scale simulation, including the software testing.
SFMT is an improved version of the original MT, which is even faster than the MT, and has a tunable characteristics of the generation period and the sequence generation. TinyMT is another variant of MT, which has a much shorter generation period (2^127 – 1) and smaller memory footprint, but is still suitable for most simulation use. The algorithm of TinyMT is much compact than SFMT or MT, and can generate a massive number (~ 2^56) of independent orthogonal number sequences, which is suitable for massive-parallel asynchronous PRNG.
For the further details on SFMT and TinyMT, please take a look at: http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/SFMT/index.html
I am not a mathematician so I cannot mathematically prove how MT and the derivatives are better than the other algorithms. But I have to emphasize very much that Erlang/OTP’s random module is still using an archaic old algorithm invented in 1980s which has a significantly shorter generation period (~ 2^43), and that has already become an indirect source of security vulnerability once (CVE-2011-0766, discovered by Geoff Cant). SFMT and TinyMT have much better characteristics than the random module, and I strongly suggest you to try them out if you really need a better non-cryptographic PRNG.
Recently I have put 256M (= 2^28) precomputed keys of TinyMT 32-bit and 64-bit generation parameters. This archive is huge (~82GB), but if you would like to use TinyMT for a serious simulation, it is worth taking a look for. The archive is at: https://github.com/jj1bdx/tinymtdc-longbatch/
Paolo – Currently you are working at Basho Japan. Can I ask you what is like to work in one of the most acknowledged Erlang companies? How much Erlang code do you see in your working daily routine?
Kenji – Basho developers are all superb and are very energetic on making Riak and recently-open-sourced Riak CS even more better products. Working with such talented engineers and keeping yourself up with them is very very tough, but if you are capable to point bugs and propose contributions which have proven to work correctly to Basho’s open-sourced projects, you will surely be welcome.
I would also like to emphasize that Basho is not just an Erlang company. You need to know every programming languages and the computer science elements, from C, C++, Java, Python, Ruby, to the gory details of distributed database, including how the vector clocks work and commutative/conflict-free replicated data types (CRDTs). Riak, Riak CS, and rebar, include a lot of their by-products. See the deps/ directory under Riak and you will be astonished. On the other hand, there might be many ways to contribute your skills.
I would also like to emphasize that Basho’s client service engineers, sales and marketing people including the documentation experts, and all the other staff members, are closely working together with the developers and maintain the high standard of delivering the quality service and products.
I can only answer that the amount of Erlang code I have to see is *enormous*.
Paolo – I am very interested about Erlang and Japan. Is Erlang a niche programming language there as well or is it spreading fast as in the US and north Europe?
Kenji - I would rather want to ask the question back: is Erlang a popular language in anywhere in the world? I think the answer is probably no, comparing to the popularity of Java or C++. Looking at the TIOBE index will prove this. And I’d rather say nobody cares about that, because whether a language is spreading fast or not has already become irrelevant, comparing to the jobs or tasks what you want to get done with the language.
I do understand Erlang has gained a larger momentum in Sweden, where the language is from. And I see many people solving problems with Erlang in Europe. And in the USA and Canada (hi Fred!). And in Japan too, especially for the server-side programming solutions. So I feel the developers in Japan are slowly but surely showing more interests.
Getting back to the situation in Japan: I think not many people are interested in whatever the new paradigm of programming, except for relatively small number of communities. Fortunately, those communities surely exist. And some visionaries have discovered some languages, such as Haskell, OCaml, or Erlang, to solve *their* problems and helping others solving the problems. But for the majority of programmers, most of the details are “not really something to be carefully taken care of and to be blindly delegated to the experts”, also called the *omakase* attitude in Japan. So most programmers just do the omakase to the Rails, or to Java libraries, or to the pre-built C++ libraries. And that irresponsible attitude towards their profession, though not necessarily only of their sole responsibilities, cause a lot of sometimes lethal or disastrous bugs in the production systems. Unfortunately, many of programmers in Japan are not well-educated as the software engineers, and their supervisors are sometimes even worse. Their mindset of dumping the risks (or *doing the marunage*) for every difficult problem makes things even worse.
I think programming is not something for omakase and the quality of code will not be sufficient so long as major users of computers are doing the marunage to the developers in Japan. And I believe Erlang/OTP is not for the people who are not willing to take the risk of their own computer systems. On the other hand, for those who want to maintain the system by themselves or at least to eagerly, deliberately, and willingly take the responsibility of running the system without major outage, Erlang/OTP will become a great tool because the system provides the critical and essential functions such as non-stopping module replacement.
Paolo – As many other Erlang gurus out there, you are very active not only when it comes to promote new Erlang applications but also when Erlang newbies ask for support or suggestions. In your opinion what are the factors that make the Erlang community so nice?
Kenji – I was pretty much impressed by the friendly environment of the erlang-questions mailing list and the modest attitude of the experienced Erlang community-driving people there, when I first asked some questions. I just read and read and read all the things in the Erlang-related mailing lists as much as I could. Erlang Workshop papers were also a set of excellent source of information. And now we’ve been full of good code in GitHub, including the OTP itself. So we’ve got many many more things ready to learn now for free!
I’ve heard that one of the old sayings in the Erlang community is “no prima donna allowed”. This is so important for maintaining a community. I understand everybody wants to get grumpy sometimes, and quite often flame wars occur, but many people just endure and keep silent. I respect this rather European or even Swedish way of getting rid of chaos
Paolo – I think that the Erlang community is growing fast: many applications, conferences and new books, still most of the developers out there don’t know that behind many of the tools they use every day there is a piece of Erlang. How would you explain that?
Kenji – I think this is in fact a very good thing. People want to solve their own problems in whatever tools they have to use, or they think suitable to use. Erlang has flexible package release tools which can minimize the users of the package to think about the installation of Erlang/OTP itself. In many popular applications, the Erlang virtual machine and the necessary libraries are silently built-in and being there; and most people don’t care whether it uses Erlang/OTP or not so long as the software works OK. Erlang/OTP has become a part of the infrastructural ecosystem.
Of course, there is a strong negative side of this trend, too; developers are doing the marunage with the omakase attitude to the developer of those infrastructural tools with no knowledge about the tools. I try not to fall in this trap by building all user-land programs, kernels, and the Port programs of my FreeBSD development servers, at least for the past ten years. You have to think about the bugs if you have to build your own tools; this is a very good way to learn a new thing. You need to forcefully do so frequently.
Paolo – OK, Kenji. Many thanks for the interview!
Kenji – You’re welcome!
In my previous post about TDD and Erlang I listed some ways to improve both coding proficiency and Erlang knowledge. This week I would like to write here my opinion about the series of videos gently provided to me by O’Reilly: “Erlang by Example with Cesarini and Thompson”.
The series is composed by 8 chapters (even though I would prefer calling them “lectures”):
- Basic Erlang
- Sequential Programming
- Concurrent Programming I
- Concurrent Programming II
- Process Error Handling
- Mobile Frequency Server I
- Mobile Frequency Server II
I guess that many of you (especially the non Erlangers ones) are now wondering: “What are the topics in detail? Who is the target audience? Should I buy the videos instead of a normal book? Are these videos really so good?” Well, let me answer that in the rest of this post.
What are the topics in details? – As you may notice from the list above the video lectures starts with the basics (data types, variables, pattern matching, etc etc.). After that you will learn things mostly related to sequential programming and concurrency. A good point of these videos is that you will end up with something real: a simple client-server application handling mobile frequencies. Notice that in the list above there is no reference to OTP: in fact you won’t learn about OTP here, but I believe many Erlangers are right when they say: “Learn with ‘normal’ Erlang and code your application using OTP”.
Who is the target audience? – Good question. Are you new to Erlang development? If so buy these videos. You will learn much and in a fast way. On the other hand I think that many experienced Erlang developer should take a look at these lessons, not only to review the basic concepts but also to hear the considerations and the suggestions of two of the most respected Erlangers out there. Sure, if you know Erlang very well you will skip some stuff, but still you will enjoy the lectures in their whole.
Should I buy the videos instead of a normal book? – No. Don’t do it! I have to be clear here: you won’t learn Erlang just by watching these videos. I believe these videos must be considered as a wonderful integration of what you read on a real book. As I wrote above you will benefit from the talking between the authors, but I must say that nothing beats the good old detailed books (especially the paper ones). So my advice is to select one of the many Erlang books out there, read it and complete the study chapter by chapter using these videos.
Are the videos really so good? – Yes they are, both for content quality and video quality. I must admit that every time I see some content authored by Francesco Cesarini and Simon Thompson I feel at ease. My first Erlang Book ever was “Erlang Programming” and since then Francesco and Simon never let me down. I believe this is mostly related to their great experience in teaching and consulting: they know what to say, when to say it and how to say it. The quality of the videos is great too! The videos are 1280 × 720 and last on average ~15 minutes and this time amount is perfect because you never got bored or tired while watching them. I would like to point out that I read in some reviews people complaining about codec problems, but before writing this post I tried them in Ubuntu, Mac, Windows 7 and iPad and didn’t notice any kind of problem.
That’s all folks! Now it’s up to you: are you going to buy these videos???