Part 4: Constant evaluation and improvement: Finding sources for feedback.

2010-10-24 08:13
In recent years demand for shorter feedback cycles especially in software development has increased. Agile development, lean management and even Scrum are all for short feedback cycles: Coming from the dark ages when software projects would last for months or even years before any results could be delivered to customers we are transforming development into a process that integrates the customer in the design and evolution of his own product. Developers have learned that planning ahead for years does not work: It's not only customers changing their mind so fast, it's requirements changing quickly. The only achievement from months-long cycles is getting input on your work later, being able to hide deficiencies longer.

However not only for planning and management does it make sense to install fast feedback loops. A few days ago I finished reading the book "Apprenticeship patterns". A book that gives an overview of various patterns that help improve software development skills.

One major theme was about getting fast feedback constantly. On the (agile) development side, automated tests (unit and integration) and continuous integration systems are technical aids that can help. Pair programming and code review take the idea of fast feedback one step further by having humans give you feedback on what cannot possibly be evaluated automatically.

There is just one minor glitch: With really short feedback loops any mistake you make gets revealed instantly. This is not particularly special to agile development. Another area with these kinds of fast feedback loops are projects developing free software - see also the last paragraph in Bertrand's blog on This is how we work at Apache.

There are developers who have a hard time living with exposing their work to a wider audience quickly. However it has the huge benefit of revealing any errors as early as possible - ideally at a point in time where fixing them is still cheap. Examples for options spotting mistakes early in various environments are listed below:


  • Open source development: Mistakes are revealed during code review of patches (or checkins).
  • Scrum: Speeding up error spotting can be implemented by closely integrating the product owner during each sprint. As soon as a feature is done - according to the developer team - it gets reviewed by the product owner this way reducing risk of features getting rejected during the sprint review.
  • In the team: Get each change set either mailed to all developers allowing for fast review of incoming code.


These are all good ways for feedback, however what about non-coding areas? Are there ways to build fast feedback into tasks that do not involve coding? I'll pick just one example to highlight some ways that facilitate fast feedback in a non-coding environment.

From time to time even hard-core coders have to meet face-to-face to discuss new designs, learn more about customers' requirements. This may involve going to conferences, giving talks, organising internal workshops, public meetups or even conferences.

Usually people doing the organisation are too busy to "watch what happens": They already drown in tasks to do and have no time (and are in no good position) to objectively evaluate the conference's quality.

Still there are ways to build feedback loops even into this kind of setup. Most of them have to do with communication:

  • Ask people to give you feedback in an official feedback form: Don't expect more than a fraction of the actual attendees to answer that form. Still it can be a source for honest feedback when done correctly. Include a few open questions, don't ask people to rate each and every task - they'll never find the time to do that anyway. Read the free-form comments - usually they provide way more insight than any rating question anyway.
  • Talk to people, ask for proposals on what you should do differently next time.
  • Watch what people are telling on the net - however keep in mind that those statements usually are a bit biased showing only the extreme ends of the spectrum.


The same applies to people giving presentations: Talk to people from the audience after your presentation is over. If your talk was video-taped, you are in a really great situation, as now you can judge for yourself what should be improved and where there are glitches in your arguments.

According to my experience people very rarely give explicit feedback - except when being really disappointed or extremely positively surprised. However when asked for precise feedback on specific aspects people are usually more than willing to share their experience, tell you where to improve and what to change. Usually it turns out to be a good idea to actively seek out people for evaluation of your projects to get better at what you do, to encourage peers to tell you what you do wrong or even where you could get slightly better.

Salsa steps

2010-10-14 20:50
If you always wanted to start learning Salsa and never really had the time to do so: Today a new beginners' course starts at Salsa Con Corazon in Berlin Schöneberg. The course is still open for registration teaching the very basics in four sessions - one per week.

A Get Together Checklist

2010-10-06 19:38
Still on the list of potentially interesting books: The Checklist Manifesto - explaining why checklists can still be valuable - especially for complex problems and tasks.

Though not very complex, I chose to come up with a checklist for running a Hadoop Get Together in Berlin as an exercise. I'm trying to stick with advise provided by the Checklist for Checklists.

Parties involved


  • Find two to three speakers two months in advance.
  • Find a sponsor for the videos.

Gathering information


  • Double check time and date with all speakers and newthinking store.
  • Get name, title, abstract from the speakers.
  • Get logo and exact conditions from sponsor.

Spreading the word


  • Put together an announcement text including thanks to video and venue sponsors.
  • Publish the event on Upcoming.
  • Publish the event on Xing.
  • Augment the announcement text by the Xing event and Upcoming links.
  • Send a newsletter to the Meetup Xing group.
  • Send the text to the Get Together mailing list, and if appropriate to the Hadoop, HBase, katta, Lucene, Solr and Mahout mailing lists.
  • On event day send a reminder to the Get Together mailing list
  • Create meetup intro slides including thanks for the sponsors, schedule, announcements of future events.

During the meetup


  • Mention newthinking bar during introduction.
  • Self-introduction of all participants.
  • Get mail addresses of future mailing list subscribers.
  • Keep presentations at 30 to 40 minutes.
  • Get speakers' slides.

After the event


  • Publish talks' slides.
  • Publish links to videos.


The more meetups you have run the larger the chance of the main organiser getting sick the day the meetup takes place. To avoid having to re-schedule the event make sure there are people that are capable and willing to take over moderation.

Are devs contributing to OSS happier?

2010-09-24 20:18
When talking to fellow developers or meeting with students it happens from time to time that I get the question of why on earth I spent my freetime working on an open source project? Why do I spend weekends at developers' conferences like FOSDEM? Why do spent afternoons organising meetups? Why is it that I am reviewing and writing code after work for free?

Usually I point people to a post by Shalin explaining some of his reasons to contribute to open source. The post quite nicely summarises most reasons that match well with why I contribute back.

On the Apache Community mailing list Grant Ingersoll asked the question about whether devs who work on or use open source are happier in their employment.

In his response Mike posted a link to a video on what motivates people that adds another piece of information to the question of why work on open source software can be perceived as very rewarding though no money is involved: With people doing cognitively challenging tasks, motivation via payment can get you only so far. There are other motivational factors that might play an equal if not larger role in getting people to perform well on their day-to-day work:


  • Autonomy: If people are supposed to be engaged with their project they need time and freedom to chose how to solve their tasks. Many large engineering driven companies like Google or Atlassian have gone even further by introducing the concept of giving people a day a week to work on what they want how they want provided they share their results. These so-called 20% projects have shown to have high potential of turning into new, creative project ideas but also even into bugs or problems getting fixed.
  • Mastery: Great developers strive to get better at what they do - simply because realizing that you actually learn something and get better at what you do can be very satisfying. One way of achieving that goal is to work together with peers on common projects. The larger the pool of peers to draw from, the higher the probability of you finding mentors to help you out and to point out mistakes you make.

    There is one more factor why working on open source increases your coding level that should not be underestimated. Grant Ingersoll nicely described it in the thread mentioned above: "I was just talking with a friend yesterday, and fellow committer, who said he is a much better programmer since contributing. Of course, it makes sense. If your underwear is on display for all to see, you sure better make sure it is clean!"
  • Purpose: People like to work on projects for a purpose. Be it to make all information accessible to the world or to turn earth into a better place by making cheap calls available to everyone. As a counter example deploying some software only for the purpose of selling a license and not make life of your client better by recommending the best solution to help solve his problem may not be half as satisfying.


There is quite some documentation out there on what drives people who contribute to open source projects. The video shared by Mike nicely summarizes some of the motivations of people that are independent of open source work but are closely related to it.

Part 3: A polite way to say no - and why there are times when it doesn’t work.

2010-09-07 23:05
After having shared my thoughts on how to improve focus and how to track tasks eating up time this post will explain how to keep time invested at a more or less constant level. The goal of this exercise is to keep obligations at a reasonable level - be it at work or during ones spare time.

In recent time I have collected a small set of techniques to reduce what gets to my desk - I don't claim this list to be exhaustive. However some of it did help me organise conference and still have a life besides that.

Sharing and delegating tasks



Sharing and delegating are actually two different ways of integrating other people: Sharing for me means working together on a topic. That could be as easy as setting up a CMS or it could be more involved as in publishing articles on Lucene in some magazine. The advantage is that both of you can contribute to the task, possible even learn from each other: When I was doing the article series on Lucene together with Uwe it also was a great learning experience for me to have someone take the time to explain to me - well, not only to me - what flexible indexing, local search and numeric range queries are really all about, as in technically implemented. So it was not only an enormous time-saver for me, as the alternative would have been me reading through documentation, code and mailing lists to get up to date. But it also gave me the unique opportunity to learn from the very developers of these features about how they work and how they are meant to be used.

The disadvantage of sharing is that part of the work still remains on your desk. That's where delegation helps: Take the task, find someone who is capable and willing to solve it and give it to them. There are two problems here: First you have to trust people to actually work on the task. Second you probably cannot avoid checking back from time to time to see if there is progress, if there are any impediments etc. So it means less work than with sharing. But there is more risk in not getting your results and more work to be done for co-ordination. However it is a very powerful technique if applied correctly to scale what can be achieved: Telling people what you need help with and letting them take over some of that work does scale way better than micro-managing people or even trying to be part of every piece of a project. It means giving up some of your control, in return you can turn to other potentially more involved tasks. Note to self: Need to build up more trust in that area.

Both concepts however are not actually about saying no but about being able to say yes even if you already have just very few time left.

Prioritisation



Prioritising tasks can be done on a scale from zero to any arbitrarily large number. Obviously it helps with deciding whom to say no to: It's going to be those projects rated very low. That is those you could easily do without That's the simplest case as it is easiest to explain. The strategy I usually use is to be honest with people: If there are conflicting conferences, it's easy to reject invitations. If some publication does not pay for you, it's easiest to be open and honest with people and tell them. Usually they will understand.

A second reason for a rating of zero is that the task is one of those "Does not belong on my desk" tasks. My advice for those would be to get rid of them as quickly as possible: They draw away your energy without giving back any value. This issue plays nicely with the "patches welcome" theme from open source: People working on open source projects are most successful if they are driven by their own needs. So if you want something implemented, either implement and submit it yourself - or find someone you can pay to do so. People will not work for you. You can jump up and down, complain on the mailing lists - but if the feature you would like to see is something that no-one else in the existing community needs, it won't get done until someone needs it.

Introduce barriers



A nice way of rejecting favours that works at least sometimes is to raise the barrier. The example here would be getting an invitation to give an introductory talk for a closed audience. So what I tried was to raise the bar by asking for funding for travel and accommodation.

Keep in mind though that there is the risk that the one inviting you actually accepts your conditions - no matter how high you think you have set them. Especially the example given above has the problem of being too low a bar in most cases. So be prepared to have to keep your promise. As a result the conditions you set really should lead to the task turning into something that is fun to do.

Cut off early



Imaging you have committed to some task. Later on you realise you won't actually make it: You have no time, priorities have changed, the task is too involved or any other reason you could potentially imaging.

The important way to reduce the load on your desk is to communicate this issue as early as possible. It's clear that people will be more disappointed the later they learn that something they probably depend on won't arrive in time or will never happen: They'll never be extremely happy, however the sooner they learn the more time they have on their part to react. And actually, most people don't react that disappointed at all, simply because they may have counted some risk into the equation when giving you the task - which is not to say you should lower the reliability of your commitments, simply because no-one is expecting you to meet your goals anyway. However usually the amount of trouble expected is way higher than what actually happens. Second note to self: Don't forget about this option.

Patches welcome



At least in open source: If it's nothing that helps make your world better - there are other people out there to help out. Patches being welcome may seem obvious. However in some areas it really is not: If someone asks the project member to be present at some conference, he may himself not consider himself capable of representing the project or even just making an impact by talking to people about it. That is the point where to encourage people that any input is welcome - not only code, but also documentation, communication and marketing work.

Of course as with any Pattern there are boundaries when not to apply it or when applying it would mean too much effort or loss. If that is the case and you have committed and cannot step back, than you should think about what could be a great reward if you went through the tasks: What would it take to make you happily comply and still gain energy through what you are doing? Basically it isn't about doing what you like but about loving what you do (L. Tolstoi).

There is also valuable advice on managing ones energy from the Apache Software Foundation that is specially targeted at new committers. If you have not done so yet take the time to read it.

Part 2: Tracking tasks, or - Where the hack did my time go to last week?

2010-09-03 18:22
After summarising some strategies for not loosing track of tasks, meetings and conferences in the last post, this one is going to focus on the retrospect on achievements. If at some point in time you have asked yourself "Where the hack did time go to?" - maybe after two busy weeks of work this article might have a few techniques for you.

Usually when that happens to me it's either a sign that I've been on vacation (where that is totally fine) or that too many different, sometimes small but numerous tasks have sneaked into my schedule.

Together with Thilo I have found a few techniques helpful in dealing with these kind of problems. The goals in applying them (at least for me) have been:

  • Configure the planned work load to a manageable amount.
  • Make transparent and trackable (to oneself and others) which and how many tasks have been finished.
  • Track over time any changes in number of tasks accomplished per time slot.


After hearing about Scrum and its way of planning tasks I thought about using it not only for software development but for task planning in general. Scrum comprises some techniques that help achieving the goals described above:


  1. In Scrum, development is split into sprints: Iterations of focussed software development that are confined to a fixed length. Each sprint is filled up with tasks. The number of tasks put into one sprint is defined by the so-called velocity of the team.
  2. Tasks are ordered by priority by the product owner. Priority here is influenced by factors like risk (riskier tasks should be attacked earlier than safe ones), ROI (those tasks that promise to increase ROI most should of course be done and launched first) and a few more. After priorisation, tasks are estimated in order - that way those tasks most important to the product owner are guaranteed to have an estimated complexity defined even if there was not enough time to estimate all items.
  3. Complexity here does not mean "amount of time to implement a feature" - it's more like how much time do we need, how much communication overhead is involved, how complex is the feature. A workable way to come up with reasonably sensible numbers is to chose one base item, assing complexity of one to it and estimate all coming items in terms of "is as complex as the base item", "has double the complexity" - and so on - according to the fibonacci series. Fibonacci is well suited for that task as do not increase linearly - similarly humans are better at estimating small things (be it distances or complexities). As soon as items get too big, estimation also tends to be way off the real number.
  4. To come up with a reasonable estimate of what can be done in any week, I usually just look back to past weeks and use that as an estimate. That technique is close enough to the real number to be a working approach.


To track what got done during the past week, we use a whiteboard as Scrum Board. Putting tasks into the known categories of todo, checked out and done. That way when resetting the board after one week and adding tasks for the following week it is pretty obvious which actions ate up most of the time. The amount of work that goes onto the board is restricted to not be larger than what got accomplished during the past week.

So what goes onto the whiteboard? Basically anything that we cannot track as working hours: The Hadoop Get Together can be found just next to doing the laundry. Writing and sending out the long deferred e-mail is put right next to going out for dinner with potential sponsors for free software courses at university.

Now that weekly time tracking is set-up - is there a way to also come up with a nice longer term measure? Turns out, there are actually three:

First and most obviously the whiteboard itself provides an easy measure: By tracking weekly velocity and plotting that against time it is easy to identify weeks with more or less freetime. As a second source of information a quick look into ones calendar quickly shows how many meetings and conferences one attended over the course of a year. Last but not least it helps to track talks given on a separate webpage.

It helps to look back from time to time: To evaluate the benefit of the respective activities, to not loose track of the tasks accomplished, to prioritise and maybe re-order stuff on the ToDo list. Would be great if you'd share some of your techniques of tracking and tracing time and tasks - either in the comments or as a separate blog post.

Apache Dinner DUS

2010-08-17 19:10
the evening after FrOSCon - that is on August 22nd 2010 at 7:30p.m. CEST - a combined "FSFE Fellowship meetup/ Apache dinner*" takes place in Tigges in Düsseldorf (Brunnenstraße 1, at Bilker S-Bahnhof). Given it doesn't rain, we'll be sitting outside.

Would be great to meet you there for tasty food, interesting discussions on Apache in general, as well as projects like Lucene, Hadoop or Tomcat in particular. Anyone interested in either the FSFE or Apache is welcome to join us.

One personal request: Somehow, Rainer (Kersten, FSFE) talked me into preparing a talk on what the ASF is all about - would be really great to have more people around share their experience.


See you in Düsseldorf

NoSQL summer Berlin - this evening

2010-08-11 06:38
This evening at Volkspark Friedrichshain, Café Schoenbrunn the next NoSQL summer Berlin (organised by Tim Lossen) is meeting to discuss the paper on Amazon's Dynamo "Dynamo: Amazon's Highly Available Key-value Store". The group is planning to meet at 19:30 for some beer and discussions on the publication.

Part 1: Travelling minds

2010-08-03 06:00
In the last post I promised to share some more information on techniques I came across and found useful under an increasing work load. Instead of taking a close look at my professional calendar I decided to use my private one as an example - first because spare time is even more precious then working hours, simply because there is so few of it and secondly because I am free to publicly scrutinize not only the methods for keeping it in good shape but also the entries in it.

I am planning to split the article in four pieces as follows as keeping all information in one article would lead to a text longer then I could possibly expect to be read from beginning to end:



  1. Part 1: Traveling minds - how to stay focussed in an always-on community.
  2. Part 2: Tracking tasks, or: Where the hack did my time go to last week?
  3. Part 3: A polite way to say no - and why there are times when it doesn't work.
  4. Part 4: Constant evaluation and improvement: Finding sources for feedback.
  5. Part 5: A final word on vacation.



Several years ago, I had no problem with tasks like going out reading a book for hours, working on code for hours, answering mails only from time to time, thinking about one particular problem for days. As the number of projects and tasks grew, these tasks became increasingly hard to accomplish: Writing code, my mind would wander off to the mailing list; when reviewing patches my mind would start actually thinking about that one implementation that was still lingering on my hard disk.

There are a few techniques for getting back to that state of thinking about just one thing at a time. One article I found very insightful was an essay by Paul Graham. He gave a pretty good analysis of thoughts that can bind your attention and draw them away from what should actually be the thing you are thinking about. According to his analysis a pretty reliable way to discover ideas that steal your attention is to observe what thoughts your mind wanders to when you are taking a shower (I would add cycling to work here, basically anything that lets your mind free to dream and think): If it is not in line with what you would like to think about, it might be a good time to think about the need to change.

There are a few ways to force your mind to stay "on-topic". Some very easy ones are explained in a recent blog post on attention span (Thanks to Thilo for the link):

  • Organising your virtual desktops such that applications are sorted according to tasks (one for communication, one for coding project x, another one for working on project y) helps to switch off distraction that would otherwise hide in plain sight. Who wants to work on code if TweetDeck is blinking at you next to your editor? In contrast to the original author I would not go so far to switch off multiple monitors: Its great to have your editor, some terminals, documentation in the browser open all at the same time in one workspace. However I do try to keep everything that has do with communication separate from coding etc.
  • Train to work for longer and longer periods of time on one task and one task only: The world does not fall apart, if people have to wait for an answer to your mail for longer than 30min - at least they'll get used to it. You do not need to take your phone to meetings: If anything is starting to melt down there will be people who know where you are and who will drag you out of the meeting room in no time. Anything else can well wait for another 60min.
  • When working with tabbed browsing: Don't open more tabs then you can easily scan. You won't read those interesting blog post you found four weeks ago anyway. In modern browsers it is possible to detach tabs. That way you can follow the first hint of keeping even the web pages sorted on desktops according to activity: You do not need your time tracking application next to your editor. Having only documentation and testing application open there does help.
  • Keep your environment friendly and supportive. Who has ever shared an office (or a lecture at university back when I was a student) with me knows that close to my desk the probability of finding sweets, cookies, drinks and snacks approaches one. Being hungry when trying to fix a bug does not help, believe me.


One additional trick that helps staying just focussed enough for debugging complex problems is to make use of systematic debugging by Andreas Zeller (also explained in Zen and the Art of Motorcycle Maintenance). The trick is to explicitly track you thoughts on paper: Write down your hypothesis of what causes the problem. Then identify an experiment to test the hypothesis - you should know how to use your debugger, when to use print statements, which unit tests to write and when to simply take a very close look at the code and potentially make it simpler for that. Only when your experiment confirms that you have found the cause of the problem you really have identified what you need to fix.

There are a few other techniques for getting things off of your head that are just there to distract you: If you ever have read the book "Getting things done" or seen the Inbox zero presentations you may already have an idea of what I am hinting at.

By now I have a calendar application that works like a charm: It reminds me of meetings ahead of time, it warns me in case of conflicts, it accepts notes, it has an amazing life span of one year and is always available (provided I do not forget it at home):
- got mine here ;) That's for organising meetings, going to conferences, getting articles done in time and not forgetting about family birthdays.

For week to week planning we tend to use Scrum including a scrum board. However that is not only for planning as anyone using Scrum may have expected already.

For my inbox the rule is to filter any mailing list into its own folder. Second rule is to keep the number of messages in my inbox to something that fits into a window with less than 15 lines: Anything I need for further reference (conference instructions, contacts, addresses that did not yet go into my little blue book, phone numbers not yet stored in my mobile phone) goes into its own folder. Anything that needs a reply is not allowed to stay in the inbox for longer than half a week. For larger projects mail gets sorted into their own project folders. Anything else simply goes to an archive: There are search indexes available, even Linux supports desktop search, search is even integrated in most mail clients. Oh and did I mention that I managed to search for one specific mail for an hour just recently, though it was filed into its own perfectly logical folder - simply because I had forgotten which folder it was?

To get rid of things I have to do "some time in the near future but not now" I keep a list in my notebook - just so my mind knows the note is there for me to review and it knows I don't forget about it. So to some extend my notebook is my personal swap space. One thing I learnt at Google was to not use loose paper for these kinds of notes - a bound book is way better in that it keeps all notes in one place. In addition you do not get into danger of throwing notes away too early or mis-place them.

The only thing missing is a real product backlog that keeps track of larger things to do and projects to accomplish - something like "I really do need to find a weekend to drive these >250km north to the eastbaltic sea (Thanks to Astro for pointing out the typo to me - hey, that means there is at least one guy who actually did read that blog post from beginning to end - wow!) and relax" :)

Series: Getting things done

2010-07-30 07:07
Probably not too unusual for people working on free software mostly (though no longer exclusively) in their spare time, the number of items that appear in my private calendar have increased steadily in the past months and years:

  • Every three months I am organising the Apache Hadoop Get Together in Berlin.
  • I have been asked (and accepted the offer) to publish articles on Hadoop and Lucene in magazines.
  • There are various conferences I attend - either as speaker or simply as participant: FOSDEM, Froscon, Apache Con NA, Devoxx, Chemnitzer Linuxtag - to name just a few.
  • For Berlin Buzzwords I did get quite a bit of time for organisation, still some issues leaked over to what others would call free time.
  • I am mentoring one of Mahout's GSoC students which is a lot of fun.
  • At least I try to spend as much time as possible on the Mahout mailing lists keeping up with what is developed and discussed there.


There are various techniques to cope with increased work load and still find enough time to relax. Some of them involve simply remembering what to do at the right time, some involve prioritization, others deal with measuring and planning what to do. In this tiny series I'll explain the techniques I employ - or at least try to - in the hope of getting your feedback, and comments on how to improve the system. After all, the most important task is to constantly improve ones own processes.