FrOSCon - Robust Linux embedded platform

2012-09-04 20:05
The second talk I went to at FrOSCon was given by Thilo Fromm on Building a robust embedded Linux platform. For more information on the underlying project see also projec HidaV on github. Slides of the talk Building a robust Linux embedded platform are already online.

Inspired by a presentation on safe upgrade prodedures in embedded devices by Arnaut Vandecappelle in the Embedded Dev Room FOSDEM earlier this year Thilo extended the scope of the presentation a bit to cover safe kernel upgrades as well as package updates in embedded systems.

The main goal of the design he presented was to allow for developing embedded systems that are robust - both in normal operation but also when upgrading to a new firmware version or a set of new packages - the design included support for upgrading and rolling back to a known working state in an atomic way. Having systems deployed somewhere in the wild to power a wind turbine, inside of busses and trains or even within satellites pretty much forbids relying on an admin to press the "reset button".



Original image xkcd.com/705

The reason for putting that much energy into making these systems robust also lies in the ways they are deployed. Failure vectors include not only your usual software bugs, power failures or configuration incompatibilities. Transmission errors, storage corruption, temperature, humidity add their share to increase the probability of failure.

Achieving these goals by building a custom system isn't too trivial. Building a platform that is versatile enough to be used by others building embedded systems adds to the challenges: Suddenly having easy to use build and debug tools, support for software life-cycle management and extend-ability are no longer nice-to-have features.

Thilo presented two main points to address the requirements: The first is to avoid trying to cater every use case. Setting requirements for a platform in terms of performance, un-brickability (see also urban dictionary, third entry as of this writing). Even setting a requirement for dual boot support or to the internal storage technology used. As a result designing the platform can become a lot less painful.

The second step is to harden the platform itself. Here that means that upgrading the system (both firmware and packages) is atomic, can be rolled-back atomically and thus no longer carries the danger of taking the device down for longer than intended: A device that does no longer perform it's task in the embedded world usually is considered broken and shipped back to the producer. As a result upgrading may be necessary but should never render the device useless.

One way to deal with that is to store boot configurations in a round robin manner - for each configuration a "was booted" (set by the bootloader on boot) and a "is healthy" (set by the system after either a certain time of stability or after running self tests) flags are needed. This way at each boot it is clear what the last healthy configuration was.

To do the same with your favourite package management system is slightly more complicated: Imagine running something like apt-get upgrade with the option to switch back to the previous state in an atomic way if anything goes wrong. One option to deal with that presented is to work with transparant overlay filesystems that allow for having a read-only base layer - and a "transparent" r/w layer on top. If a file does not exist in the transparent layer, the filesystem will return the original r/o version. If it does exist it will return the version in the transparent overlay. In addition there's also an option to mark files as deleted in the overlay.

With that upgrading becomes as easy as installing the upgraded versions into some directory in your filesystem and mounting said directory as transparent overlay. With that roll-back as well as snapshots are easy to do.

The third ingredient to achieving a re-usable platform presented was to use Open Embedded. Including an easy to extend layer-based concept, support for often recent software versions, versioning and dependency modelling, some BSP layers officially supported by hardware manufacturers building a platform on top of Open Embedded is one option to make it easily re-useable by others.

If you want to know more on the concepts described join HiDaV platform project - many of the concepts described are already - or soon to be - implemented.

Clojure Berlin - March 2012

2012-03-07 22:37
In today's Clojure meetup Stefan Hübner gave an introduction to Cascalog - a Clojure library based on Cascading for large scale data processing on Apache Hadoop without hassle.

After a brief overview of what he is using the tool for to do log processing at his day job for http://maps.nokia.com Stefan went into some more detail on why he chose Cascalog over other project that provide abstraction layers on top of Hadoop's plain map/reduce library: Both Pig and Hive provide easy to learn SQL-like languages to quickly write analysis jobs. The major disadvantage however comes when in need for domain specific operators - in particular when these turn out to be needed just once: Developers end up switching back and forth between e.g. Pig Latin and Java code to accomplish their analysis need. These kinds of one-off analysis tasks are exactly where Cascalog shines: No need to leave the Clojure context, just program your map/reduce jobs on a very high level (Cascalog itself is quite similar to datalog in syntax which makes it easy to read and simple to forget about all the nitty-gritty details of writing map/reduce jobs).

Writing a join to compute persons' age and gender from a trivial data model is as simple as typing:


;; Persons' age and gender
(? [?person ?age ?gender]
(age ?person ?age)
(gender ?person ?gender)


Multiple sorts of input generators are implemented already: Reading text files, using files in HDFS as input are both common use cases. Of course it is possible to provide your own implementation for that as well to integrate any type of data input in addition to what is available already.

In my view Cascalog combines the speed of development that was brought by Pig and Hive with the flexibility of being able to seemlessly switch to a powerful programming language for anything custom. If you yourself have been using or even contributing to either Cascalog or Cascading: I'd love to see your submission to Berlin Buzzwords - remember, the submission deadline is this week on Sunday *MEZ*.

HowTo: Meetups in Berlin

2012-02-14 20:23
I get that question once in a while - and need the list below myself every now and then: How to actually setup a meetup in Germany. Essentially it all boils down to three questions: Which channels to use for PR? Where to do the meeting? What other benefits to offer to attendees?

When it comes to PR there are several options:


  • Announce the meetup on relevant mailing lists
  • Use social networking sites relevant to your project - in Germany Xing works best, Twitter, Facebook, Linked.In and Google+ are other options
  • Ask anyone you know personally for help with spreading the word
  • If you have one post information on your personal blog


Where to go for the meetup:

The venue usually is the biggest question mark. After deciding on how big you'd like to shoot for initially you can start looking for a location. For your first meetup don't rent a room - with a bit of creativity there are lots of options that are free of charge.

  • If you are a student or have active relations to any university going there usually is the cheapest and least complicated version.
  • Another option is to just book a table in a restaurant that has a reasonably large room. Simply choose your favourite one - knowing the owner helps in getting extra space.
  • Third option is to go to any co-working space that also has a meeting area. In general they are very open to hosting community events - co-up Berlin, Betahaus are just two options.
  • If you are planning a less formal event, your local hacker spaces might be an option: c-base Berlin, in Berlin e.V. are two Berlin examples. Hackers Dojo and Noisebridge are two Bay Area examples.
  • Last but not least look out for local startups that are currently hiring new people: They tend to be very open to hosting events. See Berlin Buzzwords Hackathon providers list for some examples.


What else?


  • Make sure attendees can register themselves - xing works for that, so do Google forms
  • Setup a mailing list or some other notification service to help people track future events (Google Groups works, so does a dedicated Twitter Account)
  • Provide some background online - meetup.com works but does charge a small fee. Setting up a blog on wordpress or blogger works as well, though it is not quite as interactive as the meetup.com site.
  • Get in touch with attendees and local companies - usually they are quite happy to provide some financial support to your meetup for free drinks or videos.
  • If you want videos: Recording audio is trivial, putting it online is extremely simple if you use soundcloud's app. Recording video also is rather simple but can be time consuming. Finding sponsors to pay for them if you offer to brand the videos is reasonably simple. For the Hadoop Get Together we usually hire Martin Schmidt. Sites to put videos online: Vimeo works but has rather low upload limits, blip.tv is a bit better in this respect.
  • Sponsoring in general: Companies looking for developers related to the meetup's technology as well as those providing consulting for that technology tend to be open to supporting local events. What works best is to contact people you already know there - they will know best who to ask internally.


One final note: Being the organiser of such a meetup puts you at the center of a local community. Over time people will start remembering your face and name. Make sure you do the same - you should at least be able to remember faces, affiliations and names of your regular attendees.

Clojure in Berlin

2012-02-02 00:01
Though I had the chance to tinker with some Clojure code only briefly it's programming model and the resulting compact programs do fascinate me. As the resulting code runs on a JVM and does integrate well with existing Java libraries migration is comparably cheap and easy.

Today I finally managed to attend the local Berlin Clojure meetup, co-organised by Stefan Hübner and Fronx. Timing couldn't have been much better: In this evenings event Philip Potter from Thoughtworks introduced Overtone - a library for making music with Clojure.

After installing and configuring jack for sound output, supercollider, and overtone outputting your first tone is as simple as registering the overtone library and typing

(definst foo [] (saw 220))
(foo)

To stop it type (stop).

Other types of waves of course are supported as well, so is playing different waves simultaneously and modifying them at runtime. Also expressing sounds as notes (c, d, e, f, g) that may have a certain length is possible of course – which makes it so much easier to design music than having to thing in frequencies.

A sample of what can easily be done with Overtone:


Original sound way better - this sample was taken with a mobile phone, compressed, re-coded and then put online. Checkout Overtone project for the real thing - and don't even try to listen to the sample with low-end laptop speakers ;)

Overall a well organised meetup (Thanks to Soundcloud for hosting it, to the organisers for putting it together and to the speaker for a really well done introduction to Overtone) and an interesting way to get started with Clojure with very fast (audio) feedback.

Teddy in Chicago

2012-01-22 21:56
Last week I spent several days in Chicago mainly to attend a few meetings at the local Nokia/Navteq office. Though the schedule was pretty packed, a few hours remained to explore the then frosty and windy city:



Top three images: Some impressions of the city. Bottom left: Teddy's new friend. Bottom right: Situation at ORD when flying out - fortunately both, the airport as well as the airline (Swiss) have quite some experience with challenging weather conditions so that we could leave without too much delay.


As usual I wondered whether there are any Apache people close by. So before flying in I checked our committers map. As there were a few people in that general area I sent a brief heads-up to the greatly under-advertised, private, non-archived, committers only list party@apache.org. In case you've never heard about it: The main use case of that list is to provide a means for committers to arrange for meeting up with fellow Apache people and share travel details.

As a result I received a brief list of things to do in Chicago and got to attend a small but really nice meetup. Having a means to get in touch with locals can make such a difference - thanks for the warm welcome! Hopefully next time I'm there weather is as warm - would love to explore the (at least according to my travel guide book) beautiful nature of the great lakes.

#28c3

2011-12-30 02:07

Restate my assumptions.

One: Mathematics is the language of nature.

Two: Everything around us can be represented and understood through numbers.

Three: If you graph the numbers of any system, patterns emerge. Therefore, there are patterns everywhere in nature.


The above is a quote from today's "Hackers in movies" talk at 28c3 - which amongst others also showed a brief snippet of the movie Pi. For several years I stayed well away from that one famous Hackers' conference in Berlin that takes place annually between Christmas and New Year. 23C3 was the last congress I attended until today. Though there were several fantastic talks and mean presentation quality was pretty good the standard deviation of talk quality was just too high for my taste. In addition due to limited room sizes with 4 tracks there were quite a few space issues.



In recent years much of that has changed: The maximum number of tickets is strictly enforced, there is an additional lounge area in a large tent next to the entrance, for the sake of having larger rooms the number of tracks was reduced to three. Streaming works for the most part making it possible for those who did not get one of the 3000 full conference tickets to follow the program from their preferred hacker space. In addition fem does an amazing job of recording, mastering, encoding and pushing videos online: Hacker Jeopardy - a show that wasn't over until early Thursday morning (about 3a.m.?) - was up on Youtube at least on Thursday at 7a.m if not earlier.

Several nice geeks got me talked into joining the crowd briefly this evening for a the last three talks in "Saal 1" depicted above: You cannot be in Berlin during 28c3 and not see the so-called "fnord Jahresrückblick" by Fefe and Frank Rieger, creators of the Alternativlos podcast.



Overall it is amazing to watch BCC being invaded by a large group of hackers. It's fun to see quite a few of them on Alexanderplatz, watch people have fun with a TV B Gone in front of large electronics stores. It's great to get to watch highly technical but also several political talks 4 days in a row from 11 a.m. until at least 2p.m. the following day that are being given by people who are very passionate about what they do and the projects they spend their time on.

If you are into tinkering, hacking, trying out sorting algorithms and generally having fun with technology make sure you check out the 28c3 Youtube channel. If you want to learn more on Hacker culture, mark the days between Christmas and New Year next year and attend 29c3 - don't worry if you do not speak German - the majority of talks is in English, most of the ones that aren't are being translated on the fly by volunteers. If you are good at translations, feel free to volunteer yourself for that task. Speaking of volunteering: My respect to all angels (helping hands), heralds (those introducing speakers), noc (network operating center), poc (phone operating center), the organisation team and anyone who helps keep make that event as enjoyable to attendees as it is.

Update: Thank you to the geeks who after staying in our apartment for #28c3 helped get it back to a clean state - actually cleaner than it was before. You rock!

Berlin Tech Meetups

2011-12-09 22:32
Berlin currently is of growing interest for software engineers, has a very active startup scene and as a result several community organised meetups. Listed below is a short, "highly objective" selection of local user groups - showing just the breadth of topics discussed.


If you want to discover new meetups: It helps attending one that is closest to your interest as usually people follow several user groups. In addition watching the scheduled event at co-working and hacker spaces like co-up Berlin, betahaus, c-base can help.

GoTo Con AMS - Day 2

2011-10-23 10:47
Day two of GoTo Con Amsterdam started with a keynote by former Squeak developer Dan Ingalls. He introduced the Lively kernel - a component architecture for HTML5 that runs in any browser and allows easy composition, sharing and programming of items. Having seen Squeak years ago and being thrilled by its concepts even back then it was amazing to see what you can do with Lively kernel in a browser. If you are a designer and have some spare minutes to spend, consider taking a closer look at this project and dedicating some of your time to help them get better graphics and shapes for their system.

After the keynote I had a hard time deciding on whether to watch Ross Gardler's introduction to the Apache way or Friso van Vollenhoven's talk on building three Hadoop clusters in a year - too many interesting stuff in parallel that day. In the end I went for the Hadoop talk - listening to presentations on what Hadoop is actually being used for is always interesting - especially if it involves institutions like RIPE who have the data to analyze the internet downtime in Egypt.

Frise gave a great overview of Hadoop and how you can even use it for personal purposes: Apache Whirr makes it easy to use Hadoop in the cloud by enabling simple EC2 deployment, the Twitter API is a never ending source for more data to analyze (if you don't have any yourself).

After Jim Webber's presentation on the graph database neo4j I joined the talk on HBase use cases by Michael Stack. He introduced a set of HBase usages, problems people ran into and lessons learned. HBase in itself is built on top of HDFS - and as such inherits its advantages, strengths and some of its weaknesses.

It is great for handling 100s of GB up to PB of data in an online, random access but strong consistency kind of model. It provides a ruby based shell, comes with a java api, map reduce connectivity, pig, hive and cascading integration, provides metrics through the hadoop metrics subsystem that are exposed via JMX and through Ganglia, provides server side filters and co-processors, hadoop security, versioning, replication and more.

Stumbleupon

Stumbleupon deals with 1B stumbles a month, has 20M users (growing), users spend approx 7h a month stumbling. For them HBase is the de-factor storage engine. It's now 2.5years in production and enabled a "throw nothing away" culture, streamlined development. Analysis is done on a separate HBase cluster from the online version. Their lessons learnt: Educate engineering on how it works, study production numbers (small changes can make a for big payoff), over provisioning makes your life easier and gets your weekends back.

OpenTSDB

... is a distributed, scalable time series database that collects, stores and serves metrics on the fly. For stumbleupon it is their ears and eyes into the system that quickly replaced the usual mix of ganglia, munin and cacti.

Facebook

As announced earlier this year, Facebook's messaging system is based on HBase. Also facebook metrics and analytics are stored in HBase. The full story is available in a SIGMOD paper by Facebook.

In short - for Facebook Messaging HBase has to deal with 500M users, millions of messages and billions of instant messages per day. Most interesting piece of the system here was their migration path that by running both systems in parallel made switching over really smooth albeit still technologically challenging.

Their lessons learnt include the need to study production and adjust accordingly, to iterate on the schema to get it right. They also made the experience that there were still some pretty gnarly bugs - however with the help of the HBase community those could be sorted out bit by bit. They also concentrated on building a system that allows for locality - inter rack communication can kill you.

Yahoo

They keep their version of the bing webcrawl in HBase. They have high data ingest volumns (up to multiple TB/hour) from multiple streams. Atop their application also has a wide spectrum of access patterns (from scans down to single cell access). Yahoo right now runs the single larges known HBase cluster on top of 980 2.4 GHz nodes with 16 cores and 24GB Ram each in addition to 6x2TB of disk. Their biggest table has 50B documents, most of the data is loaded in bulk though.

YFrog

... uses HBase as backend for their image hosting service. In contrast to the above HBase users they don't have a dedicated dev team but are highly motivated and skilled ops. Being cost senstive and with a little bit of bad luck with them really everything went bad that could go bad - from crashing JVMs, bad RAM crashes, bad glibc with a race condition, etc. Their lessons learnt include that it's better to run more smaller nodes than less big nodes. In addition lots of RAM is always great to avoid swapping.

The final talk I attended that day was on tackling the folklore around high performance computing. The speakers re-visited common wisdom that is generally known in the Java Community and re-evaluated it's applicability to recent hardware architectures. Make sure to check out their slides for details on common mis-conceptions when it comes to optimization patterns. Basic take away from this talk is to know not only your programming language and framework but also the VM you are implementing your code for, the system your application will run on and the hardware your stuff will be deployed to: Hardware vendors have gone to great length optimizing their systems, but software developers have been amazing at cancelling out those optimizations quicker then they were put in.

All in all a great conference with lots of inspiring input. Thanks to the organizers for their hard work. Looking forward to seeing some of you over in Vancouver for Apache Con NA 2011.

GoTo Con AMS - Day 1

2011-10-22 20:44
Last week GoTo Con took place in Amsterdam. Being a sister conference to GoTo in Aarhus the Amsterdam event focused on the broad topics of agile development, architectural challenges, backend and frontend development, platforms like the JVM and .NET. In addition the Amsterdam event featured a special Apache track tailored towards presentations focusing on the development model at Apache and the technologies developed at Apache.

Keynote: Dart



The first day started with the keynote by Kasper Lund who introduced Google's new language Dart. Kasper was involved with developing V8 at Google. Based on his (and other project members') experiences with large JavaScript projects the idea to create a new language for the browser was born. The goal was to build a language that had less pitfalls than JavaScript, makes it easier to provide tool support for and makes reasoning about code easier. Dart comes with class based single inheritance, lexical scoping, optional typing. It is by design single threaded. The concept of isolates cleanly introduces the concept of isolated workers that communicate through message passing only and thus can be run in parallel by the VM. One concept that seemed particularly interesting for an interpreted language was that of snapshots: An application can be serialized after it has loaded and initialized, the result can even be transferred, shortening load time substantially.

So far Dart is just a tech preview - on the agenda of the development team we find items such as better support for REST arguments, enums, reflection, pattern matching, tooling for test coverage and profiling. All code is freely available, also the language specification and tutorials are open. The developers would love to get more feedback from external teams.

Twitter JVM tuning best practices



In his presentation on JVM tuning Attila Szegedi went into quite some detail on what kind of measures Twitter usually takes when it comes to optimizing code that run on the JVM and exhibits performance issues. Broadly speaking there are three dimensions along which the usual culprits for bad performance hide:

  • Memory footprint of the applciation.
  • Latency of requests.
  • Thread coordination issues.


Memory footprint reduction

A first step always should be to verify that memory is actually responsible for the issues seen. Running the JVM with verbosegc turned on helps identify how often and how effective full GC cycles happen on the machine. Next step is to take into account the simple solution: Evaluate whether the application can simply be given more memory. If that does not help or is impossible start thinking about how to shrink memory requirements: Use caching to avoid having to load all data im memory at once, trim down the data representation used in your implementation, when looking into what to trim know exactly what amount of memory various objects need and how many of these object you actually keep in memory - this analysis should also go into detail when using code generated from frameworks like thrift.

Latency fights

When taking a simple view latency optimization boils down to making a tradeoff between memory usage and time. A little less naive view is to understand that actually it is a set of three goals to optimize:



Tuning an application means to take the product of the three, shift focus but keep the product stable. Optimization is assumed to increase the resulting product.

Biggest thread to latency are full gc cycles. Things to keep in mind when tuning and optimizing: Though the type of gc to run is configurable, this configuration does not apply to cleanup of eden space - Eden is always cleaned up with a stop-the-world gc. In general this is not too grave, as cleaning up objects that are no longer referenced is very cheap. However it can turn into a problem when there are too many surviving objects.

When it comes to selecting GC implementations: Optimize for throughput by delaying GC for as long as possible. This is especially handy for bulk jobs. When optimizing for responsiveness use low pause collectors - they incur a somewhat constant penalty however those avoid having single requests with extremely large response time. This is most handy for online jobs.

Other options to look into: Use adaptivesizepolicy and maxgcholdmillis to allow the jvm to size heap on its own based on your target characteristics. Use the printheapatgc option to view gc heap collection statistics - especially watch out for fromspace being less than 100%, use printtenuredistribution to keep an eye on number of ages, size distribution. In general, give an app as much memory as possible - when using concurrent mark and sweep implementation make sure to over-provision by about 25 to 30% to give the app a gc cushion for operation. If you can spare one cpu, set initiateoccupationfraction to 0 and let gc run all the time.

Thread coordination

The last issue in general causing delays are thread coordination issues. The facilities for multi-threaded programming in Java are still pretty low level - even worse, developers generally hardly know about synchronized - not so much about the atomic data types that are available - let alone other features of the concurrent package.

Make sure you check out the speaker's slides they certainly contain valuable information for developers that want to scale their Java applications.

Akka


Another talk that was pretty interesting to me was the introduction of Akka - a project I had only heard about before but did not have any deep technical background knowledge on. The goals when building it were fault tolerance, scalability and concurrency. Basically an easy way to scale up and out. Built in Scala, Akka also comes with Java bindings.

Akka is built around the actor model for easier distribution. Actors are isolated, communicate only via messages and have no shared memory - making it easy to run them in a distributed way without having to worry about synchronization. Distribution across machines is currently based on protocol buffers and NIO. However the resulting network topology is still hard wired during development time.

The goal of new Akka developments is to make roll-out dynamic and adaptive. For that they came up with a zookeeper based virtual address resolution, configurable load balancing strategies and the option for reconfiguration during runtime.

Concluding remarks



The first day was filled with lots of technical talks - so several remained more on the overview/introductory level - which is a good thing to learn about new technologies. In addition there were a few presentations on new features of upcoming and past releases for instance for Java 7 and Sprint 3.1 - it's always nice to learn about the rational behind changes and improvements.

As for the agile talks - most of them propagated pretty innovative ideas that need a lot of courage to put into practice. However in several cases I could not help but get the feeling that either the processes presented were very specific to the environment they were established in and would not survive sudden stress - be it decline in revenue or team issues. In addition quite a few ideas that were introduced as novelties were already inherent in existing processes: Trust and natural communication really is the goal when establishing things like Scrum. In the end, the meetings are just the tool to get there. Clarity wrt to vision and even business value is core to prioritizing work to be done. Understanding and finding suitable metrics to measure and monitor business value of a product should be at the heart of any development project.

Overall the first day brought together a good crowd of talented people exchanging interesting ideas, news on current projects and technical details of battle-field-stories. Being still rather small, the Amsterdam edition of GoTo con certainly made it easy to get in touch with speakers as well as other attendees over a cup of coffee and discuss the presented issues. Huge thanks to the organizers for putting together an interesting schedule, booking a really tasty meal and having a friendly answer to any question from confused attendees.

Night Trains

2011-10-21 05:40
It was several years ago that a frequent traveller told me about it being more comfortable and time saving to travel mid-sized distances (e.g. from Berlin to Amsterdam) by train - night train that is - instead of flying. Back then without decent knowledge of which combination of booking early and discount card make prizes of trips by train somewhat comparable to those offered by airlines that wasn't really an option for me.



It took me years and a conference in Vienna to re-visit to his proposal: Trying to find ways to reduce the amount of times I fly I looked for alternatives. Going by car clearly is not an option as it's more time consuming and on such long distances also more stressful. Going by regular train seemed like a waste of time as well. So I re-checked the offers for night trains. The idea of going to bed in Berlin and waking up at my destination just seemed too good.

Currently sitting in a CNL to Amsterdam I have to admit that for now this is the most relaxing way to travel I've found so far: Get on board, sleep heavenly (though not as comfortable as in your preferred hotel, beds are still ok), get your breakfast brought to bed in the morning. Mix in meeting friendly people (so far maybe I was just lucky) that are either discovering Europe as backpackers (met one from Australia and one from Canada yesterday) or happen to be taking that train as part of their weekly commute.

I think at least for most European cities I'm converted now :)