On being aggressivly public

2014-02-01 19:06
If it didn't happen on the mailing list it didn't happen at all.
... with all it's implications this seems to be the hardest lesson for newcomers to the Apache way of development to learn.

In the minimal sense it means that any decision a project takes has to be taken publicly, preferably in some archived, searchable medium. In a wider sense it's usually interpreted as:

  • ... don't ask people getting started questions privately by mail, rather go to the user mailing list.
  • ... don't coordinate and take decisions in video conferences, rather take the final decision on the dev mailing list.
  • ... don't ask people privately on IM, rather go to the user mailing list.


There are even canned answers that can get sent out to those asking privately.

This concept seems to be very hard to grasp for people: As a student, developers tend to be afraid to ask "stupid questions" - it's easy to say that "there are no stupid questions, only stupid answers" - truly feeling this to be true can be hard though in particular for people who are just beginning.

However (assuming the question hasn't been answered before or the answer wasn't easy to find with a search engine of your choice) asking these questions is a valuable contribution: Chances are, there are a couple other people who would have the same question - and are too shy to ask. Chances are even bigger that those developing the project would never think of documenting the answer anywhere simply because they do not see it as a question due to their background and knowledge.

For researchers in particular on their way to getting a PhD. the problem usually is the habit of publishing facts only after they are checked, cleaned and well presentable. When it comes to developing software what this approach boils down to developing a huge amount of code hidden from others. The disadvantages are clear: You get no feedback on whether or not there would be simpler/faster approaches to solving parts of your problem. Developers of the upstream project won't take the needs for this extension into account when rewriting current modules or adding new code. As a result general advise is to share and discuss your ideas before even starting to code.

Even for seasoned software engineers this approach tends to be a problem when not used to it: In a corporate environment feedback on code often is perceived as humiliating. As much as we preach that a developer isn't their code so feedback on coding issues is not to be taken seriously it still takes a lot of getting used to when suddenly people give that feedback - and give it in the open for everyone to see.

Also often decisions are taken privately before communicating publicly in corporations. As a result having heated design discussions tends to feel like a major marketing problem. Instead what is achieved is making the decision process publicly visible to others so they can follow the arguments without repeating them over and over. It also means that others can chime in and help out if they are affected (after all the second mantra is that those who do the work are the ones who ultimately take the decisions).

Again another reason could be for people new to the project or new to some role in the project that asking trivial questions looks intimidating. Again - even if you've contributed to a project for years noone expects you to know anything. If any piece isn't documented in sufficient depth asking the question provides a track record for others to later refer to. It also means you tend to get your answer faster because there are more people who can answer your question. Oh - and if you don't get an answer chances are, everyone else is in the dark as well but was just not asking.

One final word: One fear many people seem to share when asking is to make their own knowledge gaps and flaws public - not only to peers but also to current or future superiors. There's three parts to the advise that I tend to give: Everyone once started small. If your potential future manager or team mates don't understand this, than learning and making mistakes obviously aren't welcome in the environment that you are aspiring to join - is it really worth joining such an environment? Even so you start small, if you continue to contribute on a regular basis the amount of information that you produce online will grow. Guess how much time people spend on evaluating CVs - it's very unlikely they will read and follow each discussion that you contributed to. And as a third point - do assume that by default everything you do online will be visible and searchable forever. However chances are that ten or fifteen years from now what you are writing today will no longer be accessible: Services go away, projects and their infrastructure die, discussion media change - anyone still remember using NNTP on a regular basis?

As an exercise to the reader: Before writing this post I spend a couple of hours trying to dig up the mails I had sent I believe in 2000 to the I think KURT and RTLinux discussion lists in order to prepare some presentation about the state of realtime Linux back at university. I search the web, search the archives I could find, searched my own backups from back then - but couldn't find anything. If you do find some trace of that, I owe you a beer next time we meet at FOSDEM ;)

Scientific debugging - take 2

2014-01-28 20:13
Back in - OMG was that really back in 2010? - 2010 I wrote a post on scientific debugging. Today I was reminded of this post as I actually had the pleasure of watching this principle carried out - except this was for a medical "bug" instead of one in a piece of software.

To quote the book Why programs fail the method of scientific debugging consists of 5 easy to follow steps:
  1. Observe a failure (i.e., as described in the problem description).
  2. Invent a hypothesis as to the failure cause that is consistent with the observations.
  3. Use the hypothesis to make predictions.
  4. Test the hypothesis by experiments and further observations:
    • If the experiment satisfies the predictions, refine the hypothesis.
    • If the experiment does not satisfy the predictions, create alternate hypothesis.
  5. Repeat steps 3 and 4 until the hypothesis can no longer be refined.
In the past I've often seen developers jump immediately from a problem description in a user reported bug to implementing what looks like the obvious solution only to find out later that the actual problem underlying the problem description was something completely different.

I know that making sure one has found the true problem can take time. However today I was reminded a) just how long that time can be and b) just how important each step in that search can be:

So to start with the problem description (step 1 above): "On Saturday evening on my way into the bath tub some pretty bad pain hit me and forced me to lie down flat on the floor. After a few minutes the pain was gone, what remained until today's Tuesday is a pain in my back that gets worse when I walk or lie down on my left side. As a side-note: I'm expecting a baby - not sure if that has anything to do with the above."*

husband urged me to get to our general practitioner. So off I went - told him the story above. He couldn't do a whole lot for me but to him it seemed like the typical sciatic pain syndrome (step 2 above) - so off he sent me to the orthopaedic (he needed an expert's opinion to check his hypothesis apparently).

I called the one doctor I was recommended, was told to go there immediately, went there. Some 15 minutes later I told the doctor the story above (step 1). She followed the hypothesis - except for one tiny little problem: Given the location of the pain and my condition she knew of at least one more hypothesis that would also fit the observation (step 2). She made a manual check (step 3). Her observation was still consistent with the hypothesis (step 4). Now in order to reject said hypothesis she would have needed to do one more check - except she couldn't do that one herself. So she sent me off to my gynecologist to get an expert's opinion (and gave me an appointment for late this week/early next week in case really only the original hypothesis is valid).

Off I went, called the third doctor. I told her the whole story (step 1), she continued where the other doctor had left off and made the missing check (step 3) - and rejected the alternative hypotheses right away (step 4).

So off she went to step 5: Sometimes labour pains start in the back (step 2) - so she put me on a CTG (step 3) - after several minutes that hypothesis was rejected as well (step 4). So off she went to step 5:

The last non-trivial hypothesis would have been that something's wrong with kidneys (step 2). So she checked with her medical ultrasonics (step 3) - she wasn't 100% sure though it seemed to look ok so she send me off to one last expert for a second opinion.

So on my way home I got to tell the story above the fourth time (step 1). Again she continued with where the other doctor had left off - namely step 3: One more medical ultrasonic and a blood test later the last non trivial hypothesis was finally rejected altogether (step 4) So what was left was the initial hunch of there being some blockade in my back. Finally we can step from observing to acting, namely keeping warm (hot water bottle), relaxing and to keep moving regularly.

As annoying as the day for the system to be debugged was there's at least two valuable lessons learnt in it:

Being absolutely certain about a certain problem cause can be expensive (in this case in particular time consuming for me, rest is something the doctors involved have to discuss with my public health insurance). However jumping directly from problem description to bugfix can turn out to be much more expensive if said bugfix takes a long time to implement (and show effects) but is treating the wrong problem.

Proving one hypothesis right sometimes involves ruling out options that are easy to check but have bad consequences if left untreated. Translated back - before jumping from problem description to bugfix it may make sense to stop for a second and think whether the particular problem you see might actually be caused by a bug far worse than what you anticipated.

Now off to search for my warm little elephant.



* Yeah, that's why I didn't make it to the Berlin Open Source meetup on Sunday that I organised - at least my husband had fun together with Lennart and others trying to explain to the non German speaker what on earth is the English translation for the term Hexenschuß.

Children tinkering

2014-01-05 02:07
Years ago I decided that in case I got the same question for at least three times I would write down the answer and put it somewhere online in a more or less public location that I can link to. The latest question I got once too often came from daddies (mostly, sorry - not even a handful of moms around me, let alone moms who are into tech) looking for ways to get there children in touch with technology.

Of course every recommendation depends heavily on the age and interest of the little one in question. However most recommendations are based on using a child's love for games - see also a story on how a father accidentally turned his daughter into a dungeons and dragons fan for a bit more background on what I mean.

There are several obvious paths, including Lego Mindstorms, the programming kits by Fischertechnik, several electronics kits you get at your favourite shop, fun stuff like Makey, makey kits that can turn a banana into a controller. Also many games come with their own level designers (think Little Big Planet, though the older children might remember that even Far Cry, Doom and friends came with level designers).

In addition by now there are quite a few courses and hacking events that kids are invited to go to - like the FrogLabs co-located with FrosCon, the Chaos macht Schule initiative by CCC, meetups like the ones hosted by Open Tech School, Jugend Hackt. In addition quite a few universities collaborate with schools to bring pupils in touch with research (and oftentimes tech) - like e.g. at HU Berlin.

In addition there are a three more less typical recommendations:

  • As a child I loved programming a turtle (well, a white dot really) to move across the screen forward or backwards, to turn east, south, west or north, to paint or to stop painting. The slightly more advanced (both in a graphical as well as in an interactive sense of the word) version of that would be to go for Squeak (all smalltalk, first heard about it about a decade ago at the Chemnitzer Linuxtage) or Scratch (a geek dad kindly showed that to me years ago).
  • When it comes to hardware hacking one recommendation I can give from personal experience is to take part in one of the soldering courses by Mitch Altman - you know, the guy who invented the "TV-B-Gone". Really simple circuits, you solder yourself (no worries, the parts are large and robust enough that breaking them is really, really, really hard). What you end up with tends to be blinking and in some cases is programmable. As an aside: Those courses really aren't only interesting for little ones - I've seen adults attend, including people who are pretty deep into Java programming and barely ever touch circuits in their daily work.
  • If you are more into board games: Years ago one of my friends invited me to a RoboRally session. Essentially every player programs their little robot to move across the board.


When it comes to books one piece I can highly recommend (didn't know something like that existed until my colleagues came up with it) would be the book "Geek Mom" - there's also an edition called "Geek Dad". Be warned though, this is not tech only.

If you know of any other events, meetups, books or games that you think should really go on that list, let me know.