An interview with Peer Stritzinger about #erlang and #rtems
Hello! Today you can read in my blog my interview to Peer Stritzinger. Peer is a famous entrepreneur and Erlang developer who deals with several technologies connected to embedded electronic systems. Peer will give the talk Full Metal Erlang at the upcoming Erlang User Conference 2013. Enjoy!
Erlang? Let’s do some embedded systems with it!
Paolo – Hi Peer! Thanks for accepting my interview. Would you like to describe youself in a few words to our readers?
Peer – Hi Paolo! Thank you for your interest, I’m honored to be considered for one of your interviews — I have been enjoying your blog since your first interview. I work as software and product developer in my small company. Since I’m currently my companies sole employee I also have to double for all other necessary roles necessary. In the Munich area where I’m located we have a network of companies who join forces for larger projects however.
For example for my products for Automotive Embedded Control Unit flashing, the custom developed hardware was built by a partner company. Besides developing my own products I work as contractor for a wide spectrum of customers. It happens that many contracts are involved in one way or another with Embedded-Systems development where I have the advantage that I have experience with the very close to the hardware stuff and higher level system design and implementation.
Paolo – As many other Erlangers you started with a MSc. in Physics. How did you end up programming?
Peer – Well I can’t say I “ended up” in programming. Writing code fascinated me since my teens, and so did Mathematics and Physics. When I had to choose which track to take starting University I thought it would be easier to take Physics and do some Computer-Science classes on the side than the other way round. And it worked out pretty well, I cherry picked the interesting stuff from Computer-Science and did my MSc. (was actually called Diplom back then in Germany) in Physics. I did take a code centric approach to Physics also, writing simulations and dabbling with Reduce (a early computer algebra system). I added a bit of Biology to the mix, specializing in Biophysics (theoretical and experimental). Back then artificial Neural-Networks were being researched a lot and that meshed nicely with my interests.
Paolo – Part of your work focuses on Embedded Systems, a field mostly ‘ruled’ by C programmers. How come you started using Erlang?
Peer – Well I’m a C programmer also, but always had a high interest in all kinds of functional and symbolic programming languages. The kinds of Embedded Systems I usually built were often on the more complex and dynamic side. I didn’t do much eight bit controller programming contracts but more the 32bit end implementing more complex architectures with many protocols. Often there was some Hard-Realtime stuff in them but only about 10% of the code had to deal with it.
My first system I wrote in Erlang was however running on an embedded motherboard with a UNIX (first OpenBSD currently FreeBSD) class OS on it. It was the second version of my Automotive ECU Flashing system Hydraprog. The first version was written completely in C and I didn’t want to do the second version in C again — I was seeing that a more dynamic language with GC would be helpful (Java was out because I already had to use it for a past contracting job and didn’t like it very much). At the same time I planned to split up the big-box with many parallel channels into a networked boxes with two channels each, to make it possible to scale capacity more flexibly.
The distribution made me look at Erlang and the fault-tolerance and bit-pattern matching sold me on it. Since this was a real product with a real deadline I gave myself 4 weeks into the development which was basically the point of no-return where I still could have switched back to C. After these 4 weeks I had made so much progress, although I was still learning Erlang. There was no way going back.
Paolo – What are the benefits and the drawbacks one can experience by using Erlang instead of C for Embedded Systems?
Peer – The benefits are many, you get much more done in the same time. This translates directly into faster time to market and lower development cost. That Erlang/OTP has industrial strength implementation, well most of the Embedded Systems I work with are used in industrial use cases. These is are exactly the use cases where the phrase “industrial strength” comes from.
Then there are the whole Erlang/OTP fault tolerance and fault isolation features. You can imagine that this fault tolerance comes in handy if you are building systems to be used on the assembly line. We already use hardware watchdogs to restart systems that are stuck, but there you can only restart the whole system. It’s basically as if Erlang only had the Heartbeat Monitoring (http://erlang.org/doc/man/heart.html) and nothing else.
Plenty of OTP’s infrastructure features are needed in Embedded Systems, it’s like it was built for them. Which it actually was because the phone switches Erlang was invented for are very similar to todays Embedded Systems.
Also maintainability: Embedded Systems usually have to be maintained for 10-15 years, sometimes even longer. A system written in Erlang is much more maintainable than a system of same complexity that is written in C.
A lot of current Embedded Systems software only manages to deal with all this by being very simplistic, they have a hard time with the modern connected world. There is a lot of talking of M2M (Machine to Machine) communication and Industry 4.0, Internet of Things and all these buzzwords. For an Erlang developer these are fun things to do in Erlang. Many companies are not ready for this with their inflexible systems only written only in C, many are sitting on the side waiting for their world to get easy again. The handful of companies using Erlang for Embedded System will eat their lunch.
You also asked about drawbacks. You can in theory write a faster system that uses less CPU power and less memory in C. So if you have to fit your application in 1Mb on CPU Flash and only have 120 kB of on CPU Flash, then Erlang is hardly able to run on this system. Systems like these are used for high piece count applications like car electronics. There are many Embedded Systems that have much lower piece count being built, they often have external Flash and RAM on their boards. If you switch from the until now often used NOR Flashes to a NAND Flash, then suddenly you have so much space for the same price that you can easily fit a complete OTP release on it. In these systems the advantage of lowering the development costs is also worth most.
As for speed, a small piece of code in a micro benchmark is a lot faster in C than in Erlang. But many experienced Erlang developers found out that somehow the performance of whole systems is often quite good or even superior. I had just suspected my Erlang code for my Hydraprog flashing system of being responsible for slower than wanted flash time. But when I profiled my Erlang code (which is very easy, a luxury not many Embedded Systems developers have) I found out that the whole runtime of the Erlang code was only 6 seconds in a Flash Session that takes about 1000 seconds. Now I’m looking at the C code in the system, which somehow manages to produce substantial latency.
And finally it’s: don’t use C or Erlang, use both. Write as much code as possible in Erlang and write everything that really needs to be in C in C.
Paolo – At EUC 2013 you will give a talk about your port of Erlang to RTEMS. First of all what is RTEMS and in which devices is it used?
Peer – RTEMS is a Open-Source Hard-Realtime Embedded OS for smaller Embedded-Systems. It runs on basically every architecture used in Embedded that has a 32bit core (there is some experimental work to even make it run on AVR’s). The needed code size and RAM to run can be scaled to run on smaller SoC’s which have Flash and RAM on chip. It runs on devices used in manufacturing and in automotive control units. It is used by NASA and ESA on a variety of satellites and planetary missions. So you can see it is used like Erlang for applications where reliability is the top priority (for more details see http://www.rtems.org)
Paolo – Was it difficult to port thr Erlang VM to RTEMS? What are the features of the VM and Erlang you had to sacrifice if any?
Peer – One main difference to Unix like OS is that RTEMS doesn’t have processes in the Unix sense with separate memory. So it looks like one Unix process running multiple threads. So it kind of makes no sense to run multiple Erlang nodes on one RTEMS OS. When Erlang is started on “normal” OS with distribution enabled it needs the epmd daemon running as separate process. But we don’t have such a separate process on RTEMS. So one way would be to hack the empd C code to run alongside with the Erlang VM or implement a restricted epmd in Erlang and make it start before the distribution support comes up. I’m implementing the later approach (epmd in Erlang).
One more hurdle was that until very recently RTEMS had no support for dynamic linking. For using NIF’s dynamic linking is quite essential. First I thought I could do without NIF’s since linked in drivers don’t have this problem and can be linked statically. But then I can’t use built in stuff that uses NIF’s like e.g. crypto which would mean no easy SSL/TLS support for protocols which we need. I think I have to go back a step and tell you how a RTEMS application is built. With RTEMS the OS comes as a bunch of libraries that gets linked together with the application code to form one executive image that is booted on the hardware (either via a boot loader or right from the reset vector). What I did initially was to bake the Erlang runtime together with the RTEMS libs into one executable. This is a bit hard to maintain and use since there is no real place for the C part of the application. Normally there will be C parts for Hard-Realtime parts (we can do Hard and Soft Realtime mixed on a RTEMS+Erlang system) or drivers. Drivers in RTEMS are often just a C module that accesses the hardware registers directly. Until now I have a special make that copies the application C files into OTP’s source tree an patches the statically linked in driver list, then it builds OTP with RTEMS build tools and creates a RTEMS executable.
Just recently however some dynamic linking support is being added to RTEMS. So instead of faking dynamic linking for NIF’s I’ll use the new support and I also get a nice way to build Erlang and RTEMS as separate objects which makes it easier to distribute a prebuilt Erlang runtime for each architecture that will be supported and the device specific parts (called Board Support Package = BSP) and C application parts can just be linked to it.
Besides that RTEMS has a pretty good POSIX API support and Erlang has nice cross building support built in so that was a nice fit.
Paolo – Who should attend your talk? What will your audience know at the end of it?
Peer – My talk is intended for Erlang developers or Erlang developers in spe who are interested in Embedded and Realtime applications of Erlang. Also everyone who finds it interesting to run Erlang practically right on the bare hardware. RTEMS is getting SMP support at the moment since small embedded systems are beginning to use these. So what about running Erlang just with a minimal RTEMS layer without file-system and network support configured (all parts of RTEMS can be configured during application build time) and write a small OS in Erlang. Might have interesting latency properties when no conventional OS (and no visualization layer which stands in for a OS) gets in the way between Erlang and the hardware. Whoever thinks this might be fun should come to the talk.
While it can’t be a tutorial how to use RTEMS and Erlang I will say something how to get started with using it. I will tell something how the port was done and how it works and some first experiences with it and where it is headed.
Paolo – Erlang folks is more and more pushing on Embedded Systems, still most of Embedded Systems people are not aware of Erlang. How can we spread to them the Erlang word?
Peer – Build successful Embedded applications with Erlang and talk about it. But I’m skeptical that the Embedded Systems developers will pick up Erlang. I think rather than that Erlang developers with enough expertise of the low-level close to the metal stuff will have a huge advantage to build the next generation of intelligent Embedded-Systems. Personally I’m quite happy how it is at the moment, “Erlang as a trade secret” has also its advantages.
Paolo – In your opinion will Erlang be able to take its place in the Embedded Systems world? What are the future applications written in Erlang you see in the future in this sense?
Peer – Well, we’ll see. I will certainly build some more future systems with Erlang in the embedded space. I will either build these systems for my customers as contractors or as entrepreneur for my own products. Also I don’t think Erlang needs to “take” its place in Embedded Systems, its more like it might expand its area of application from Telecommunication embedded to other areas. All the buzz around Industry 4.0 and machines communicating directly with other machines along the assembly line will benefit Erlang’s use in Embedded-Systems.
I see applications with distributed AI planning algorithms built right into the embedded systems at the assembly line, the devices that directly control the motor and mechanics and RFID antennas of the transport system talking to each other and reacting flexibly to all kinds of influences. These applications can’t be built with current PLC technology that is used in this space at the moment. One needs powerful symbolic computation for these planning algorithms. Also it is necessary to be fast and flexible to adapt the software in these devices. Transparent distribution support will also be an advantage in this space. I’m looking forward to wield Erlang’s powers in this area.
- An interview with Michał Ślaski about TDD and testing in Erlang
- An interview with Alvaro Videla (@old_sound ) about RabbitMQ and Erlang
- Talking about Erlang, Riak and Vector Clocks with Christopher Meiklejohn (@cmeik)
- Talking about Erlang with Fernando Benavides
- Talking about Elixir and the Erlang VM with José Valim