Hello! Today you can read in my blog my interview to Peer Stritzinger. Peer is a famous entrepreneur and Erlang developer who deals with several technologies connected to embedded electronic systems. Peer will give the talk Full Metal Erlang at the upcoming Erlang User Conference 2013. Enjoy!
Erlang? Let’s do some embedded systems with it!
Paolo – Hi Peer! Thanks for accepting my interview. Would you like to describe youself in a few words to our readers?
Peer – Hi Paolo! Thank you for your interest, I’m honored to be considered for one of your interviews — I have been enjoying your blog since your first interview. I work as software and product developer in my small company. Since I’m currently my companies sole employee I also have to double for all other necessary roles necessary. In the Munich area where I’m located we have a network of companies who join forces for larger projects however.
For example for my products for Automotive Embedded Control Unit flashing, the custom developed hardware was built by a partner company. Besides developing my own products I work as contractor for a wide spectrum of customers. It happens that many contracts are involved in one way or another with Embedded-Systems development where I have the advantage that I have experience with the very close to the hardware stuff and higher level system design and implementation.
Paolo – As many other Erlangers you started with a MSc. in Physics. How did you end up programming?
Peer – Well I can’t say I “ended up” in programming. Writing code fascinated me since my teens, and so did Mathematics and Physics. When I had to choose which track to take starting University I thought it would be easier to take Physics and do some Computer-Science classes on the side than the other way round. And it worked out pretty well, I cherry picked the interesting stuff from Computer-Science and did my MSc. (was actually called Diplom back then in Germany) in Physics. I did take a code centric approach to Physics also, writing simulations and dabbling with Reduce (a early computer algebra system). I added a bit of Biology to the mix, specializing in Biophysics (theoretical and experimental). Back then artificial Neural-Networks were being researched a lot and that meshed nicely with my interests.
Paolo – Part of your work focuses on Embedded Systems, a field mostly ‘ruled’ by C programmers. How come you started using Erlang?
Peer – Well I’m a C programmer also, but always had a high interest in all kinds of functional and symbolic programming languages. The kinds of Embedded Systems I usually built were often on the more complex and dynamic side. I didn’t do much eight bit controller programming contracts but more the 32bit end implementing more complex architectures with many protocols. Often there was some Hard-Realtime stuff in them but only about 10% of the code had to deal with it.
My first system I wrote in Erlang was however running on an embedded motherboard with a UNIX (first OpenBSD currently FreeBSD) class OS on it. It was the second version of my Automotive ECU Flashing system Hydraprog. The first version was written completely in C and I didn’t want to do the second version in C again — I was seeing that a more dynamic language with GC would be helpful (Java was out because I already had to use it for a past contracting job and didn’t like it very much). At the same time I planned to split up the big-box with many parallel channels into a networked boxes with two channels each, to make it possible to scale capacity more flexibly.
The distribution made me look at Erlang and the fault-tolerance and bit-pattern matching sold me on it. Since this was a real product with a real deadline I gave myself 4 weeks into the development which was basically the point of no-return where I still could have switched back to C. After these 4 weeks I had made so much progress, although I was still learning Erlang. There was no way going back.
Paolo – What are the benefits and the drawbacks one can experience by using Erlang instead of C for Embedded Systems?
Peer – The benefits are many, you get much more done in the same time. This translates directly into faster time to market and lower development cost. That Erlang/OTP has industrial strength implementation, well most of the Embedded Systems I work with are used in industrial use cases. These is are exactly the use cases where the phrase “industrial strength” comes from.
Then there are the whole Erlang/OTP fault tolerance and fault isolation features. You can imagine that this fault tolerance comes in handy if you are building systems to be used on the assembly line. We already use hardware watchdogs to restart systems that are stuck, but there you can only restart the whole system. It’s basically as if Erlang only had the Heartbeat Monitoring (http://erlang.org/doc/man/heart.html) and nothing else.
Plenty of OTP’s infrastructure features are needed in Embedded Systems, it’s like it was built for them. Which it actually was because the phone switches Erlang was invented for are very similar to todays Embedded Systems.
Also maintainability: Embedded Systems usually have to be maintained for 10-15 years, sometimes even longer. A system written in Erlang is much more maintainable than a system of same complexity that is written in C.
A lot of current Embedded Systems software only manages to deal with all this by being very simplistic, they have a hard time with the modern connected world. There is a lot of talking of M2M (Machine to Machine) communication and Industry 4.0, Internet of Things and all these buzzwords. For an Erlang developer these are fun things to do in Erlang. Many companies are not ready for this with their inflexible systems only written only in C, many are sitting on the side waiting for their world to get easy again. The handful of companies using Erlang for Embedded System will eat their lunch.
You also asked about drawbacks. You can in theory write a faster system that uses less CPU power and less memory in C. So if you have to fit your application in 1Mb on CPU Flash and only have 120 kB of on CPU Flash, then Erlang is hardly able to run on this system. Systems like these are used for high piece count applications like car electronics. There are many Embedded Systems that have much lower piece count being built, they often have external Flash and RAM on their boards. If you switch from the until now often used NOR Flashes to a NAND Flash, then suddenly you have so much space for the same price that you can easily fit a complete OTP release on it. In these systems the advantage of lowering the development costs is also worth most.
As for speed, a small piece of code in a micro benchmark is a lot faster in C than in Erlang. But many experienced Erlang developers found out that somehow the performance of whole systems is often quite good or even superior. I had just suspected my Erlang code for my Hydraprog flashing system of being responsible for slower than wanted flash time. But when I profiled my Erlang code (which is very easy, a luxury not many Embedded Systems developers have) I found out that the whole runtime of the Erlang code was only 6 seconds in a Flash Session that takes about 1000 seconds. Now I’m looking at the C code in the system, which somehow manages to produce substantial latency.
And finally it’s: don’t use C or Erlang, use both. Write as much code as possible in Erlang and write everything that really needs to be in C in C.
Paolo – At EUC 2013 you will give a talk about your port of Erlang to RTEMS. First of all what is RTEMS and in which devices is it used?
Peer – RTEMS is a Open-Source Hard-Realtime Embedded OS for smaller Embedded-Systems. It runs on basically every architecture used in Embedded that has a 32bit core (there is some experimental work to even make it run on AVR’s). The needed code size and RAM to run can be scaled to run on smaller SoC’s which have Flash and RAM on chip. It runs on devices used in manufacturing and in automotive control units. It is used by NASA and ESA on a variety of satellites and planetary missions. So you can see it is used like Erlang for applications where reliability is the top priority (for more details see http://www.rtems.org)
Paolo – Was it difficult to port thr Erlang VM to RTEMS? What are the features of the VM and Erlang you had to sacrifice if any?
Peer – One main difference to Unix like OS is that RTEMS doesn’t have processes in the Unix sense with separate memory. So it looks like one Unix process running multiple threads. So it kind of makes no sense to run multiple Erlang nodes on one RTEMS OS. When Erlang is started on “normal” OS with distribution enabled it needs the epmd daemon running as separate process. But we don’t have such a separate process on RTEMS. So one way would be to hack the empd C code to run alongside with the Erlang VM or implement a restricted epmd in Erlang and make it start before the distribution support comes up. I’m implementing the later approach (epmd in Erlang).
One more hurdle was that until very recently RTEMS had no support for dynamic linking. For using NIF’s dynamic linking is quite essential. First I thought I could do without NIF’s since linked in drivers don’t have this problem and can be linked statically. But then I can’t use built in stuff that uses NIF’s like e.g. crypto which would mean no easy SSL/TLS support for protocols which we need. I think I have to go back a step and tell you how a RTEMS application is built. With RTEMS the OS comes as a bunch of libraries that gets linked together with the application code to form one executive image that is booted on the hardware (either via a boot loader or right from the reset vector). What I did initially was to bake the Erlang runtime together with the RTEMS libs into one executable. This is a bit hard to maintain and use since there is no real place for the C part of the application. Normally there will be C parts for Hard-Realtime parts (we can do Hard and Soft Realtime mixed on a RTEMS+Erlang system) or drivers. Drivers in RTEMS are often just a C module that accesses the hardware registers directly. Until now I have a special make that copies the application C files into OTP’s source tree an patches the statically linked in driver list, then it builds OTP with RTEMS build tools and creates a RTEMS executable.
Just recently however some dynamic linking support is being added to RTEMS. So instead of faking dynamic linking for NIF’s I’ll use the new support and I also get a nice way to build Erlang and RTEMS as separate objects which makes it easier to distribute a prebuilt Erlang runtime for each architecture that will be supported and the device specific parts (called Board Support Package = BSP) and C application parts can just be linked to it.
Besides that RTEMS has a pretty good POSIX API support and Erlang has nice cross building support built in so that was a nice fit.
Paolo – Who should attend your talk? What will your audience know at the end of it?
Peer – My talk is intended for Erlang developers or Erlang developers in spe who are interested in Embedded and Realtime applications of Erlang. Also everyone who finds it interesting to run Erlang practically right on the bare hardware. RTEMS is getting SMP support at the moment since small embedded systems are beginning to use these. So what about running Erlang just with a minimal RTEMS layer without file-system and network support configured (all parts of RTEMS can be configured during application build time) and write a small OS in Erlang. Might have interesting latency properties when no conventional OS (and no visualization layer which stands in for a OS) gets in the way between Erlang and the hardware. Whoever thinks this might be fun should come to the talk.
While it can’t be a tutorial how to use RTEMS and Erlang I will say something how to get started with using it. I will tell something how the port was done and how it works and some first experiences with it and where it is headed.
Paolo – Erlang folks is more and more pushing on Embedded Systems, still most of Embedded Systems people are not aware of Erlang. How can we spread to them the Erlang word?
Peer – Build successful Embedded applications with Erlang and talk about it. But I’m skeptical that the Embedded Systems developers will pick up Erlang. I think rather than that Erlang developers with enough expertise of the low-level close to the metal stuff will have a huge advantage to build the next generation of intelligent Embedded-Systems. Personally I’m quite happy how it is at the moment, “Erlang as a trade secret” has also its advantages.
Paolo – In your opinion will Erlang be able to take its place in the Embedded Systems world? What are the future applications written in Erlang you see in the future in this sense?
Peer – Well, we’ll see. I will certainly build some more future systems with Erlang in the embedded space. I will either build these systems for my customers as contractors or as entrepreneur for my own products. Also I don’t think Erlang needs to “take” its place in Embedded Systems, its more like it might expand its area of application from Telecommunication embedded to other areas. All the buzz around Industry 4.0 and machines communicating directly with other machines along the assembly line will benefit Erlang’s use in Embedded-Systems.
I see applications with distributed AI planning algorithms built right into the embedded systems at the assembly line, the devices that directly control the motor and mechanics and RFID antennas of the transport system talking to each other and reacting flexibly to all kinds of influences. These applications can’t be built with current PLC technology that is used in this space at the moment. One needs powerful symbolic computation for these planning algorithms. Also it is necessary to be fast and flexible to adapt the software in these devices. Transparent distribution support will also be an advantage in this space. I’m looking forward to wield Erlang’s powers in this area.
Hello there! Today you may read my interview to Omer Kilic, a very famous guy you may know if interested in Erlang and Embedded Electronic Systems.
I think you will find his answers very inspiring….it is time to get a Raspberry PI!
Code some Erlang while you are having some raspberries!
Paolo – Hi Omer! Thank you for making yourself available for one of my interviews. Would you like to tell our readers something about you?
Sure, I’m an Embedded Systems Engineer currently working on Erlang Embedded, a Knowledge Transfer Partnership between Erlang Solutions and University of Kent. The aim of this KTP project is to bring the benefits of concurrent systems development using Erlang to the field of embedded systems; through investigation, analysis, software development and evaluation.
Paolo – Why should we want to have Erlang on embedded devices?
Erlang has features that are highly relevant to solving problems we face today with modern embedded systems such as dealing with connectivity, distribution and concurrency. We also have a different take on error recovery and supervision hierarchies which reduces the overall complexity of the system and leads to less code being written, which in turn increases testability.
These features make Erlang a suitable tool to use as the orchestrator and the systems language of these complex embedded applications we are developing for the future.
While Erlang is currently being used for very large scale distributed applications, it has roots in Embedded Systems. It was originally conceived to deal with the issues, such as the fault-tolerance, concurrency and the distributed nature of telephony hardware at Ericsson.
Paolo – What are the devices that currently can run Erlang?
The rule of thumb is: if you can run Linux on your device you should be able to run Erlang on it too. This covers most of the current ARM and MIPS based platforms. There have been ports to other operating systems, such as VxWorks, in the past as well. We currently don’t support a ‘bare-metal’ setup.
The VM doesn’t actually require a lot of resources to run (this obviously is application dependent) and you can slim down the size of the installation using the packaging tools included, which will create a tailored distribution that will only contain the runtime system, required libraries and your application.
We have chosen Raspberry Pi as our reference platform and we provide packages for the Raspbian distribution so you can get started with Erlang quite easily if you have a Raspberry Pi board. We will also be releasing packages for other distributions and platform in the near future.
Paolo – Why did you choose Raspberry PI and not one among the other famous platforms (e.g. Beagle Board)?
Raspberry Pi is an exciting project with a very enthusiastic community around it. I think they have successfully removed the hurdles in getting started with Embedded Linux by providing an affordable, simple and well documented hardware platform with a thriving hardware and software development body attached to it.
Another reason why we chose the Pi is the educational side of it. If we can introduce concurrency/parallelism into the ICT curriculum, young people will learn about these computer science concepts way before they start their further education.
Paolo – I think Raspberry PI is very good for learning, but do you think it is mature enough for industrial purposes?
We see it as a great educational tool but also acknowledge that it may not be the best platform to integrate into industrial applications. As such, we work with other Embedded platforms such as Beagleboard and Gumstix for commercial applications.
With the layered hardware abstraction model that we are working on at the moment, the platform choice doesn’t really change the way you architect your system. Learning/prototyping on the Pi and deploying it to another platform is perfectly viable as long as the low level drivers are in place.
Paolo – Was it difficult to port Erlang on Raspberry PI? Did you change some stuff related to the VM?
Other than a few minor fixes it wasn’t difficult at all, in the end we actually ditched our cross-compilation setup and compiled natively on the Pi itself!
Paolo – Most of the video published on you tube are simple examples of applications which turn on/off a led or exchange Erlang messages. Are you actually working on something more complex? And if so can you tell us something about it?
We wanted some simple examples for those videos as gentle introductions to some of the features of Erlang.
Our current project is the hardware abstraction library that I’ve mentioned previously, which maps hardware to the Erlang domain doing some interesting transformations along the way to facilitate concurrency. We will be releasing the first version of our library in the coming weeks.
Paolo – I believe you are not controlling the sensors connected to Raspberry PI with Erlang. Am I wrong? How do you deal with this? Maybe with a C port?
We are controlling the sensors using the peripherals on the Raspberry Pi, using a collection of NIFs and ports.
Most digital sensors can be controlled by the Pi as it has peripherals such as SPI and I2C to deal with the interfacing, analog sensors require some additional circuitry though as the Pi does not have an Analog-to-Digital converter.
We abstract away the lowest level functionality into C ‘drivers’ and interface them to the Erlang domain using ports and NIFs. These facilities are documented in the Interoperability User’s Guide.
Paolo – Nowadays, smart grids and smart buildings are on everyone’s lips. Both are related to distributed systems where efficiency, reliability and low power consumption are features needed. Do you think Erlang could help in this sense?
Certainly, we see Erlang as a highly relevant tool to tackle these challenges.
The Actor model of concurrency we have in Erlang lends itself well to distributed systems. The runtime is fully network aware and passing messages between different physical nodes is as simple as local message passing.
The OTP framework has battle-tested supervision hierarchies that can be employed to deal with error recovery/containment and provide high reliability. Quoting Joe Armstrong, the father of Erlang:
The Erlang flagship project (built by Ericsson, the Swedish telecoms company) is the AXD301. This has over 2 million lines of Erlang.
The AXD301 has achieved a NINE nines reliability (yes, you read that right, 99.9999999%). Let’s put this in context: 5 nines is reckoned to be good (5.2 minutes of downtime/year). 7 nines almost unachievable … but we did 9.
Power efficiency is an area we haven’t looked at yet but it is definitely on our very exciting list of future explorations.
Paolo – How can we know more about Erlang embedded? Is there any technical document or blog about it?
More information about the Erlang Embedded project can be found on our blog and you can also follow us on the @ErlangEmbedded twitter account. Please feel free to get in touch if you have any specific questions!
- RT @dhh: For starters, the good news is that Silicon Valley VCs and echo chamber has never been more optional. Reconsider. https://t.co/f2x… 2 hours ago
- RT @joeerl: Experiment : walk around your workplace. Are your devs discussing problems with their tools or the problems that the tools shou… 2 hours ago
- RT @MarcusBiel: Crafting perfect #Java Docker build flow - Hacker Noon nzzl.us/7gHBVnI 1 day ago
- An interview with Michał Ślaski about TDD and testing in Erlang
- An interview with Alvaro Videla (@old_sound ) about RabbitMQ and Erlang
- Talking about Erlang, Riak and Vector Clocks with Christopher Meiklejohn (@cmeik)
- Talking about Erlang with Fernando Benavides
- Talking about Elixir and the Erlang VM with José Valim