Inductive Bias

The upcoming Apache Hadoop Get-Together is scheduled for 22. February, 6 p.m. - taking place at Axel Springer, Axel-Springer-Str. 65, 10888 Berlin. Thanks to Springer for sponsoring the location!

Note: It is important to indicate attendance. Due to security restrictions at the venue only registered visitors will be permitted. Get your ticket here: https://www.xing.com/events/hadoop-22-02-859807

Talks scheduled thus far:

Markus Andrezak : “Queue Management in Product Development with Kanban - enabling flow and fast feedback along the value chain” - It’s a truism today that fast feedback from your market is a key advantage. This talk is about how you can deliver smallest product increments or MVPs (minimal viable products) quickly to your market to get fastest possible feedback on cause and effect of your product changes. To achieve that, it helps to provide a continuous deployment infrastructure as well as all you need for A/B testing and other feedback instruments. To make the most of these achievements, Kanban helps to limit work in progress, thus manage queues and speed up lead times (time from order to delivery or concept to cash). This helps us speed through the OODA Loop, i.e. Eric Ries’ (The Lean Startup) Model -> Build -> Code -> Measure -> Data -> Validate -> Model. The more we can go through the loop, the more we have a chance to fine tune and validate our model of the business and finally make the right decisions.

Markus is one of Germany’s leading Kanban practitioners - writing and presenting talks about it in numerous publications and conferences. He will provide a brief view into how he is achieving fast feedback in diverse contexts.
Currently he is Head of mobile commerce at mobile.de.

Martin Scholl : “On Firehoses and Storms: Event Thinking, Event Processing” - The SQL doctrine is still in full effect and still fundamentally affects the way software is designed, the state it is stored in as well as the system architecture. With the NoSQL movement people have started to realize that the manner in which data is stored affects the full stack — and that reduction of impedance mismatch is a good thing(TM). “Thinking in events” follows this tradition of questioning what is state-of-the-art. Modeling a system not in mutable entities (as with data stores) but as a stream of immutable events that incrementally modify state, yields results that will exceed your expectations. This talk will be about event thinking, event software modeling and how Twitter’s Storm can help you process events at large.

Martin Scholl is interested in data management systems. He is also a Founder of infinipool GmbH.

Fabian Hüske : “Large-Scale Data Analysis Beyond Map/Reduce” - Stratosphere is a joint project by TU Berlin, HU Berlin, and HPI Potsdam and researches “Information Management on the Cloud”. In the course of the project, a massively parallel data processing system is built. The current version of the system consists of the parallel PACT programming model, a database inspired optimizer, and the parallel dataflow processing engine, Nephele. Stratosphere has been released as open source. This talk will focus on the PACT programming model, which is a generalization of Map/Reduce, and show how PACT eases the specification of complex data analysis tasks. At the end of the talk, an overview of Stratosphere’s upcoming release will be given.

Fabian has been a research associate at the Database Systems and Information Management (DIMA) group at the Technische Universität Berlin since June 2008. He is working in the Stratosphere research project, focusing on parallel programming models, parallel data processing, and query optimization. Fabian started his studies at the University of Cooperative Education, Stuttgart, in cooperation with IBM Germany in 2003. During that course, he visited the IBM Almaden Research Center in San Jose, USA, twice and finished in 2006. Fabian undertook his studies at Universität Ulm and earned a master’s degree in 2008. His research interests include distributed information management, query processing, and query optimization.

A big Thank You goes to Axel Springer for providing the venue at no cost for our event and for paying for videos to be taped of the presentations. A huge thanks also to David Obermann for organising the event.

Looking forward to seeing you in Berlin.

Get Together Berlin Apache Hadoop, Hadoop, Kanban, Scrum

c-base - 8p.m. on a Monday evening - the room is packed (and pretty cloudy as well): Time for Dorkbot, a short series of talks on “People doing strange things with electricity” hosted by Frank Rieger.

First talk up on stage was Gismo on Raumfahrtagentur - a Berlin maker-space located in Wedding. Originating from the presenter’s interest in electrical bikes a group of ten people interested in hardware hacking got together. Projects include but are not limited to 3D printing, 3D scanning, textile hacking, a collaborative podcast. Essentially the idea is to provide room and infrastructure to be used collaboratively by a group of members. From an organisational point of view the group is incorporated as a GmbH - however none of the projects is mainly targeted to commercialization: It’s main target group are hobbyists, researchers and open hardware/software people. If interested: Each Monday evening there is a “Sunday of the Kosmonauts” where externals are invited to come visit.

Second talk was on the project Drinkenlights (Klackerlaken) - a way for children to learn the basics of electronics without any soldering (hardware available for three Euros max). Experiences made with giving the ingredients for creating these toys to children of varying ages were interesting: From kids of about five years playing around up to ten/eleven year olds that when in school seemingly had to re-learn being creative without being given much direction or instruction on the task at hand.

In the third talk Martin Kaltenbrunner introduced his Tworse Key - a nice symbiosis of old technology (a morse key) and new media (Twitter). Essentially built on top of an Arduino Ethernet board it made it possible to turn morse messages into Tweets. Martin also gave a brief overview of related art projects and briefly touched upon the changes that open source and open hardware bring to art: There are projects that open all design and source code to the public to benefit from a wider distribution channel (without having to actually produce anything), working on designs in a collaborative way and get improvements back to the original project. All of these form a stark contrast to the existing idea of having one single author whose contribution is to build a physical object that is then presented in exhibitions - providing both, new possibilities and new challenges to artists.

In the last presentation Milosch introduced his new project ETIB whose goal it is to bring hardware hacking geeks together with textile geeks to work on integrating circuits into clothes.

If you are interested in hacking spaces in general and what is happening in that direction in Berlin, mark this Friday in your calendar: c-base will be hosting a Hackerspace meetup - so if you want to know how hackerspaces work or want to create one yourself, this event might be interesting to you.

Hacking c-base, Hacking

After quite some time off I went to the Scrumtisch Berlin. The event was incredibly well visited - roughly 50 people filled the upper floor at Cafe Hundertwasser. Today’s event was organised such that participants first collected discussion topics, prioritised them together and then discussed the top three items in a timebox of 15 minutes each.

Topics collected were:

• Best tricks to make teams self organised (20 votes)
• What is QA doing fulltime in a team (13 votes)
• Ops and planning in a team (15 votes)
• PO disappears and takes backlog and vision with him - what now? (7 votes)
• Working with non-Software teams (17 votes)
• Pimp up my retrospective (12 votes)
• Multiple teams on one projects and vice versa (4 votes)
• IBM doing Scrum/ massive Scrum (9 votes)
• Feature knowledge vs market knowledge - what is more important in a PO if you have to choose due to people constraints (9 votes)
• How to convince a team to do more (3 votes)
• Why is agile good (10 votes)

Compared to previous meetings quite some topics repeat. About half of the attendees were there for the first time - so it seems there is a common set of questions people usually run into when rolling out Scrum.

Self organising teams

Seems like this is one of the most common questions run into when rolling out Scrum - how to really get to self organising teams. The question can be answered from two positions: What are the pre-requisites it takes to enable teams to become self organising? How to actually transform teams that are used to a command and control structure and are reluctant to transform?

The discussion, mainly led by Andrea Provaglio, CSM trainer focussed mainly on the first part of the question. Even when limiting discussion that way, the answer will still depend heavily on the organisation structure, number of management levels, team sizes.

Marion made the topic a bit more concrete: Given the flexible vacation planning approach of Netflix, her question was whether that sort of loose approach could work in a typical German company (after all we have 30 instead of <20 vacation days, we have fixed holidays, she as a C?O of course wants to avoid customers being left alone when the whole team is on vacation.) Andrea re-phrased agility a bit here. His proposal was to not allow people to take their time off just anytime but to give them the freedom to figure out when to take time off. He identified five principles for leadership:

• Clearly setting a goal (in that case: Everyone needs to have a vacation plan at a given date.)
• Provide the team with all resources, information and with the environment they need to accomplish their task.
• Define constraints (”there must be at least one guy in the office on any given working day”)
• Check back regularly
• Make yourself available to answer any questions

The discussion on teams reluctant to adopt self organisation was separated out and deferred. His point was mainly about enabling and encouraging self organisation. Enforcing self organisation however is not possible.

Scrum in non-Software teams

Though phrased very broadly the topic quickly turned into a “how to do Scrum for hardware” discussion. Main problem here is that the further down you go the longer design generally takes. Even just routing lines on one decently sized circuit board can take several weeks. Mainly three possible ways out of the problem emerged from the discussion:

• Loosening the definition of done - “potentially shipable” may not mean sellable or even really shipable. I don’t think one should go down that slippery path. Only by actually shipping can I get the feedback I need to improve my design. So instead of loosening the definition of done we should instead start thinking about ways to get faster feedback, reduce risk and introduce shorter iterations.
• Another way is to look for ways to reduce iteration length, even though we might not get down to software release cycles, and align releases such that integration can happen earlier.
• The third way out could be to realize that maybe Scrum does not quite fit that way of working and use a different process instead that still provides the transparency and fast feedback that is needed (think Kanban).

Overall the most important result of the discussion was that within 15min discussion the issues cannot be solved. After all the solution will depend on what exactly you are working on, who your suppliers are and what your team looks like. Most important is to recognize that there is a problem and to work on removing that impediment - most important is to identify issues and to improve your process.

Operations and planning in Scrum

The last question discussed involved operations and Scrum planning: Given a team that does software development but is interrupted frequently with production issues - how should they work in a Scrum environment.

There are multiple facets to that problem: When it comes to deciding whether to deal with something immediately or not it makes sense to weigh size of the issue against amount of work it takes to resolve it. “Getting things done” states that the minimum size of an issue to deal with instantly is 2min of work. Issue with that is that the assumption of GTD was that issues flow into and inbox that is dealt is when there is time. In production environments however these issues usually trickle in instantly interrupting developers over and over again incurring a huge cost due to task switching.

One way out might be to have an event queue and assign developers (on a rotating basis) to deal with the issues and leave time for others to work in a focused way. Make sure to rotate frequently instead of by sprint - otherwise you run into the problem of making the team unstable thus delivering no stable amount of business value each sprint.

Another obvious way is to account for frequent interruptions and include a buffer for those in your plan. The most important benefit of that approach is to make the cost of this working mode clearly visible to management - leaving the decision how to deal with it up to them.

Other simple fixes include introducing some level of indirection between the actual developer and the customer raising the issue, documenting solutions as well as incoming issues for better visibility, introducing a single point of contact capable of prioritising.

Coming back to vanilla Scrum however there is one interesting observation to be made: The main contract with iterations is for developers to be able to work in a focused way. Instead of having their tasks switched each day they are promised a fixed period of time to solve a given set of stories. In the end a sprint is a compromise between what management may need (change their mind on what is important frequently) and what developers need (working on a set of defined tasks not interrupted by re-priorisation). If the assumption of focus does not longer hold true, Scrum might be the wrong model. If what needs to be done changes daily, Kanban again might be the better option. Still making sure that the cost of task switching is visible is vitally important.

To sum up a very interesting Scrumtisch - in particular as agile methods really seem to become more and more common also in Berlin. Speaking of challenges: As user groups grow sometimes their character changes as well, in particular when built around participation and discussion. It will be interesting to watch Scrumtisch deal with that growth. Maybe at some point splitting the audience and having separate breakout sessions might make sense. Admittedly I’d also love to know more on the background of the audience: How many are actually using Scrum in the trenches vs. teaching Scrum as coaches? How long have they been using Scrum? In what kind of organisation? Maybe a topic for next time.

Scrum Scrum, scrumtisch

Last week I spent several days in Chicago mainly to attend a few meetings at the local Nokia/Navteq office. Though the schedule was pretty packed, a few hours remained to explore the then frosty and windy city:

Top three images: Some impressions of the city. Bottom left: Teddy’s new friend. Bottom right: Situation at ORD when flying out - fortunately both, the airport as well as the airline (Swiss) have quite some experience with challenging weather conditions so that we could leave without too much delay.

As usual I wondered whether there are any Apache people close by. So before flying in I checked our committers map. As there were a few people in that general area I sent a brief heads-up to the greatly under-advertised, private, non-archived, committers only list party@apache.org. In case you’ve never heard about it: The main use case of that list is to provide a means for committers to arrange for meeting up with fellow Apache people and share travel details.

As a result I received a brief list of things to do in Chicago and got to attend a small but really nice meetup. Having a means to get in touch with locals can make such a difference - thanks for the warm welcome! Hopefully next time I’m there weather is as warm - would love to explore the (at least according to my travel guide book) beautiful nature of the great lakes.

General chicago, teddy

“Agile and Lean have a single purpose: to continually challenge the status quo. If you’re not doing that, you’re probably an impediment to it.”

agile42.com

Judging from the way some people become overly careful when discussing agile in general and Scrum in particular in my presence I seem to slowly have built up a reputation for being a strong proponent of these methods. Given the large number of flaky implementations as well as misunderstandings it seems to have become fashionable to blame Scrum for all badness and dismiss it altogether - up to the point where developers are proud to finally having abandoned Scrum completely - so that now they

• can work in iterations,
• accept new tasks only for upcoming but not for the current iteration,
• develop in a test-driven way,
• have daily sync meetings,
• mark tasks done only when they are delivered to and accepted by the customer,
• have regular “how to improve our work” meetings,
• estimate tasks in story points and only plan for as much work per iteration as was done in the past iteration

… my personal take on that: Add in regular releases and you end up with a pretty decent scrum/agile implementation, no matter what your preferred name for it may be. Just for clarification: Though very often I write about what I call Scrum, I don’t use that particular method just because it is the latest fashion. It simply is a tool that has served me well on multiple occasions and given me working guidelines when I had no idea at all what software development in a professional setting should look like.

So where does all that friction with anything Scrum, agile, lean or whatever you call it come from? Recently I came across a blog post that jillesvangurp.com nicely identified some grave issues with current Scrum adoption. Unfortunately the blog post only lists the failures without going into a deeper analysis of the actual defects causing those failures.

First of all, lets assume as working hypothesis that Scrum in itself does not solve any issues in your organisation but is a great tool to uncover deficiencies. The natural conclusion should be to use it as a tool to discover problems, but search for solutions for these problems elsewhere.

With that hypothesis, lets discern the the issues discussed in the post above and assign them to one of three defect categories.

Category one: Issues with the team

Problem: You have a team of all-junior developers, or of all-mediocre developers.

Goal: Turn them into a high performing team.

Solution: Imagine you were not using Scrum at all, what would be the ideal solution? Well the obvious route probably is to re-adjust the team, add several seniors so that you end up with the right mix of people that have experience and share a vision - juniors than can learn and adapt what works from them.

Comparing that to our hypothesis: Scrum is all for short delivery cycles. You will uncover teams that perform badly much faster than in methods with longer iteration periods. So it should be reasonably simple to figure out teams that have a dysfunctional configuration. Changing that configuration however no longer is dictated by Scrum.

Category two: Bugs introduced during Scrum roll-out

The failures discussed in the blog post include people following Scrum mechanically: Only because your developers are moving post-it notes from left to right does not mean they are doing anything agile. It’s perfectly possible to do waterfall in Scrum. Whether that helps solve any of your issues is a different matter.

Instead of mechanically going through the process what is more important is to understand the reasons and goals of each of the pieces that form Scrum. To make a rather far fetched comparison:

When introducing Scrum without a deep understanding of why each piece is done, what you end up with is people following that process without understanding the meaning of each step. They end up mimicking behaviour without knowing the real world: To some extend seeing only the shadows of good development patterns without understanding the real items producing these shadows.

As a general rule: Understand why there is a retrospective meeting, remember why you need estimations, think about why there are daily stand-ups (instead of weekly meetings, instead of daily sit-togethers, instead of hourly stand-ups). Figure out why there is a product owner, what the role of a scrum master does. Pro-Tip: As soon as you really have understood Scrum, you don’t need a checklist of all meetings to hold for a successful iteration - they will just fit in naturally. If they don’t, you are probably missing an important piece of the puzzle - rather than rely on a pre-fabricated checklist, go bug your trainer or coach with questions to better understand the purpose of all the different bits and pieces.

One very grave bug on roll-out is the general believe that Scrum is equal to a little bit of fairy dust you spread over your teams and all problems will automatically be solved afterwards. It is not - it’s not a silver bullet, it’s not fairy dust, it’s no magic - such things exist in fairy tales but have been seen nowhere in the real world. According to our working hypothesis above however Scrum does something really magical: By shortening delivery cycles it introduces short feedback loops which make it very easy to uncover problems in your organisation way faster than people are able to cover them up and hide them. Finding a solution on the other hand is still up to you.

The last roll-out issue mentioned is that of crappy certification - current certification programs are designed such that the naive organisation may believe that after two days of training their employers will magically turn into super heroes. Guess what - as with any certification training is just the very first step. Actual understanding comes with experience. Compare that to learning to drive: Only because you managed to get a drivers license does not turn you into a formula one winner. Instead that requires a whole lot of training. Same applies for any Scrum Master or Product Owner.

Category three: Organisation specifics

All other issues with Scrum mentioned in the blog post are either specific to the broken structures in the organisation under investigation or due to general Scrum mis-conceptions. Leaving these aside here.

To sum up: Scrum to me is nothing but a term to summarize good, proven development practices. I don’t care how you name them - however having any one name that is well defined makes it way easier to communicate. Scrum is not silver bullet - it does not solve all the issues your organisation may have. However it is a very effective debugger uncovering the issues employees and managers are trying to cover up. If you know all those issues very precisely already or you are certain that you don’t have any, chances are you don’t need Scrum.

Scrum agile, plato, Scrum

I’ve heard of several people who are not quite sure yet whether they should visit Berlin Buzzwords or not - in particular when having to travel far and cross 9 time zones to attend. My general recommendation is to plan to spend some more days in Europe. The conference is conveniently scheduled on Monday and Tuesday which gives you one weekend before to explore the city and the whole week afterwards to go and see more either in the city or around.

In case you are wondering whether the city is a worthy destination when travelling with children - below is a list of things to do and places to go I sent to someone recently. Hope it helps with your decision as well. In general the city is pretty green, there are several locations specially amenable to a visit with kids - so treat the list below as what it is: An incomplete listing of some of the most obvious locations that might be of interest collected by someone who knows a few parents and their children. Also in case you speak German make sure to check out one of the many guide books for Berlin with children available in local book stores - Dussmann and Hugendubel generally have the largest selection though Chatwins is my preferred one for anything about travelling.

In the city

In case of good weather:

• Tierpark Berlin - make sure you visit Tierpark (not Zoo) - it’s much larger and friendlier. See also images taken by Berlin Buzzwords fotographer Philipp, general images.

• There is a huge park in walking distance of Brandenburger Tor: Tiergarten for recreation after sight seeing.
• For swimming head over to either Wannsee or Schlachtensee
• For exploring a NSA listening station head west to Teufels-berg
• On warm evenings plan for some time at Maybachufer

For bad weather:

• If your kids like tech go to Technik Museum (it features one of the first computer (the one built by Zuse that is))

• If you kids like nature go to Naturkunde Museum
• If you are interested in science - make sure to be here for the long night of science (web page may need google translate unless you speak German.)
• For a city tour check out the following scribbles - they also include some interesting parts of the bus line 100 and 200

Close to the city:

If you have some more time to spend make sure you also explore the closer surroundings:

• 80km north: rent a canoo and explore Mecklenburg
• 200km north: visit Rügen, spend some time swimming, some time to see the amazing chalk cliffs, some time to see the isle by bike
• 250km south: go hiking or rafting in Elbsandsteingebirge
• 80km south: rent a canoo and explore the canals in Spreewald

Recommendations from friends

• Dawid Weiss: Badeschiff - a pool-on-the-river thing. It’s not something you get in any ordinary city
• Steve Loughran: My son’s favourite part of a trip to berlin (age 9) was actually the Bauspielplatz: Smaller kids get a play area where they can use the sand + water to build streams, dam them and generally make a mess, while the 8+ get a playground where they actually help build it under adult supervision. They also run a good open air waffle/pancake/coffee shop. They’re open in the afternoons.

Hope to see you in Berlin in June. If you need more information or recommendations don’t hesitate to ask.

Berlin Buzzwords, Relocating to Berlin Berlin, Berlin Buzzwords, ostsee, paddeln, ruegen, Spreewald, vacation

The countdown started several weeks ago - finally in the past days the date for Berlin Buzzwords was announced, the call for submissions published. It’s exciting to see that the first talk is in already. Looking forward to yours.

Compared to last year there are two changes:

1. Submissions are no longer evaluated by Jan, Simon and myself only. Due to the large number of talks submitted last year we reached out for help to be able to split the task of reviewing talks.
2. Also the conference itself grew quite a bit in the past two years. As a result it now takes several full time positions to handle not only ticketing, hosting and software development, sponsorships, venue management, travel support, but also external communication and marketing. The team of newthinking grew quite a bit and is helping substantially with tasks that before were handled by Jan, Simon and myself exclusively to keep some of our time reserved for the fun part of schedule curation. Please make sure to include info@berlinbuzzwords.de if you have questions that need a quick answer.

We are looking forward to a successful community conference on all things scalable - be it search, NoSQL or data analytics. Don’t be afraid to submit highly technical talks - Berlin Buzzwords always has been a place for developers to discuss new technologies, algorithms and implementations.

If your community need more than just a day to meet - please do talk to us. We will be providing room for meetups on Wednesday after the conference. Those are handed out on a first come first serve basis.

If you are a local Berlin company and want to get Berlin Buzzwords into your offices, please talk to us - we are more than happy to get you in touch with one of the meetup organisers.

If you would like to co-locate trainings with Berlin Buzzwords - we are happy to co-promote you event. Talk to us to be included in our official schedule. In case you need any help organising your training, newthinking will be more than happy to provide their services for your event.

Looking forward to June: It’s amazing how large that event grew in the past two years - and almost scary to return back online after a flu and see how things unfolded magically.

Berlin Buzzwords Berlin Buzzwords

Hacking arduino, Hacking

Yesterday Thilo invited me to attend an “Electronics 101″ workshop including an introduction to soldering that was scheduled to start at 7p.m. this evening at the offices of IN-Berlin e.V.. As part of my studies back in university I do have a little bit of background in Electronics, but never before had tried any serious soldering (apart from fixing one of our audio cables) so I thought, why not.

The workshop turned out to be a lot of fun: The organisers Mitch Altman and Jimmie Rodgers had brought several pre-packaged kits for people to work on. Quite a few of them based on Arduino so after putting them together you can actually continue having fun with writing little programs. After giving a brief but very well done, easy to understand introduction to digital electronics Mitch showed attendees how to use a soldering iron (make sure to check out his comic “soldering is easy” if you want to know more) and got everyone started. Both Jimmie and Mitch did a great job answering any upcoming questions, fixing issues and generally helping out with any problems. Even those that never used a soldering iron before quickly got up to speed and in the end went home with that nice experience of having built something that you cannot only program but can touch and hold in your hands.

I got myself a LoL shield (still to be done), and a Diavolino. Still missing is the FTDI TTL-232R cable for getting the device hooked up to our laptops and be able to re-program it (though most likely that will be easier to find than a >1G Ohm resistor Thilo is looking for to be able to calibrate his Geiger counter).

Results of my first session are below:

The board	First pins attached	Last pins attached

Also thanks to Sven Guckes organising and announcing this workshop on short notice. And thanks to Thilo for talking me into that.

Update: Images of the event are available online.

Hacking Hacking, hardware

Back in November I gave a talk at Apache Con NA in Vancouver on talking friends and colleagues into contributing patches to open source projects. The intended audience for this talk were experienced committers to Apache projects, the goal was to learn more on their tricks for talking people into patching. First of all thanks for an interesting discussion on the topic - it was great to get into the room with barely enough slides to fill 10 min and still have a lively discussion 45min later.

For the impatient - the written feedback is available as Google Doc. Most common advise I heard involved patience, teaching, explaining, fast feedback and reward.

One warning before going into more detail on the talk: All assumptions and observations stated are highly subjective, influenced by my personal experience or by whatever the experience of the audience was. Do not expect an objective, balanced, well research analysis of the problems in general. That said, lets start with the talk itself. Before the talk I decided to limit scope to getting people in that have limited experience with open source. That intentionally excluded anyone downstream projects depending on one’s code. Though in particular interaction with common Linux distributions and their package maintainers is vital, that issue warrants for a separate talk and discussion.

I divided those inexperienced with open source into three groups to keep discussion somewhat focused:

• Students learning about open source projects during their education and have neither background in software engineering nor in open source but are generally very eager to lean and open to new ideas.
• Researchers learning about the concept as part of a research grant who have some software engineering experience, some experience with open source - in particular with using it - but in general do not have writing open source software as their main objective, but have to participate as part of their research grant.
• Software engineers having experience with software engineering, some experience in particular with using open source and in general both strong opinions on what the right way of doing things is and who have a strong position in their team that helps them in no way when starting to contribute.

One very common way

To understand some of the issues below let me first highlight what seems to be the most common way to become involved with any Apache project: Usually it starts with using one of their software packages. After some time what is shipped does no longer fit your needs, reveals bugs that stop you from reaching your goals or is missing one particular feature - even if that is just one particular method being protected instead of private.

People fix those issues. As the best software developers are utterly lazy the contribute stuff back to the project to avoid the work of having to maintain their private fork just for some simple modification. The more features of a project are being used, the more likely it gets that also larger contributions become possible. Overall this way of selecting issues to fix has a lot to do with scratching your own itch. In the end this kind of issue prioritisation also influences the general direction of a project: Whatever is most important to those actively contributing is driving the design and development. So the only way to change a project’s direction to better fit your needs is to start getting active yourself: Those that do are the ones that decide.

Students

Lets take a closer look at students aspiring to work on an open source project. They are very keen on contributing new stuff, learning the process and open to new ways of doing things. However for the most part they are no active users of the projects they selected so they do not directly see what is important to fix. In addition they have only limited software development experience - at least when looking at German universities, bug trackers, source version control, build systems, release management, maintaining backwards compatibility, unit test frameworks are on no schedule - and most likely shouldn’t be neither. So your average student has to learn to deal with checking out code, compiling it, getting it into their favourite editor, adding tests and making them pass.

Apart from teaching, giving even simple feedback it helps to provide the right links to literature at the right times, and generally mentor students actively. In addition it can be helpful to leave non-critical, easy to fix issues open and mark them as “beginner level” to make it easier for new-comers to get started. One last advise: Get students to publish what they do as early and as often as possible. Back in the days I used to do projects at TU Berlin with the goal of getting students to contribute to Mahout. In the first semester I left the decision on when to open up the code to the students - they never went public. In the second semester I forced them to publish progress on a weekly basis (and made that part of how their final evaluation was done) - suddenly what was developed turned into a patch committed to the code base.

Researchers

A second group of people that has an increasing interest in open source projects are researchers. In particular for EU project research grant the promise of providing results and software developed with the help of European tax-payers money under and open source license has become an important plus when asking for project grants.

However before becoming all too optimistic it might make sense to take a closer look: Even though there is an open source check box on your average research grant that by no means leads to highly motivated, well educated new contributors for your project: With software development only being a means to reach the ultimate goal of influential publications researchers usually do not have the time and motivation to polish software to the level needed for a successful and useful contribution. In addition the concept of maintaining your contribution for a longer time usually does not fit the timeline and timeframe of a research project.

Apart from teaching and mentoring projects themselves should start asking for the motivation of the contribution. There are a few popular arguments to contribute patches back. However not all of them really work for the research use case: The cost of maintaining a fork is close to zero if you intend to never upgrade to a new version and do not need security fixes. Another common argument is an improved visibility of your work and an improved reputation of yourself as software developer. If software development for you is just a means to reach a much higher goal those arguments may not mean much to you. A third common argument is that of improving code quality by having more than one pair of eyes review it - and where would you get a better review than in the project bringing together the original code authors? However if ultimate stability, security and flexibility is not your goal than also that may not mean much to you.

Key is to find out where the interest for working on open source comes from and build up arguments from there.

Software engineers

The third group I identified was professional software developers - as clarified after a question from the audience: Yes, I consider people who are unable to create, read, apply patches as professional software developers. If I would exclude these people there would be noone left who earns his living with software development and does not already work on open source projects.

In contrast to the above groups these people have extensive software development experience. However that also means that after having seen a lot of stuff that works and that does not work they do have a strong position in their teams. Usually those fixing issues in libraries they use re the ones that have established work-flows that work for them very well and who are used to being pretty influential. When going into an open source community however no-one knows them. In general they are only judged based on their patch. They get open feedback - in the context of that project. Projects tend to have established coding guidelines, best practices, build systems - that may differ from what you are used to in your corporate environment.

Getting up to speed in such an environment can be intimidating at best in particular if everything you do is public, searchable and findable by definition. All the more it is important to get involved and get feedback early by even putting online early sketches of what your plan is.

However with everything being open there is also one major positive side to motivating contributors: Give credit where credit is due - add praise to the issue tracker by assigning issues to the one providing he patch, add the name of the contributor to your release notes. When substantial, mention the contribution with name in talks, presentations and publications.

Another important issue here is the influence of deadlines: If it takes half a year to get feedback on your particular improvement the reason why you made it may no longer exist - the project may have been cancelled, the developer moved to a different team, the patch applied internally as is fixing the existing issues. Fast feedback on new patches, in particular if they are clean and come with tests is vital. One positive example for providing feedback on formal issues quickly is the automated review bot at Apache Hadoop: It checks stuff like style, addition of tests, checks against existing tests and the like quickly after the patch is submitted in an automated way. Just one nitpick from the audience: The output of that bot could be either marked more clearly as “this is automated” or the text formulated a bit friendlier - if a human had done the review it would have mentioned the positive things first before criticising what is wrong.

Last but not least (applies to researchers as well), there may be legal issues lurking: Most if not all contracts entail that at least what you do during working hours belongs to your employer - so it’s up to them what gets open sourced and what doesn’t. Suddenly your very technical new contributor has to convince management, deal with legal departments and work his way through the employers processes - most likely without deep prior knowledge on open source licenses - let alone contributor agreements (or did you know what the Apache CCLA entails, let alone being able to explain it to others before really getting active?)

General advise

To briefly summarise the most important points:

• Give feedback fast - projects only run for so long, interest only lasts for so long. The faster a contributor is told what is not too great about his patch, the more likely those issues are fixed as part of the contribution. (Inspired by Avro and Zookeeper who were amazingly fast in providing feedback, committing and in the case of Avro even releasing a fixed version).
• When it comes to new contributors be patient, remain friendly even when faced with seemingly stupid mistakes.
• Give credit where credit is due - or could be due. Mention contributors in publications, press releases, release notes, the bug tracker. Let them know that you do. (Inspired by Drools, Tomcat, Zookeeper, Avro). Pro-tip: Make sure to have no typo in people’s names even if checking takes one extra minute. (Learned from Otis).
• Use any chance you get to teach the uninitiated about the whole patch process. I know that this seems trivial to those who work with open source on a daily basis. However when getting dependencies through Maven it may already be cumbersome to figure out where to get the source from. When used to git in the daily workflow it may be a hurdle to remember how to checkout stuff from svn Back in June we had a Hadoop Hackathon in Berlin that was well attended - mostly by non-committers. Jakob Homan proposed a rather unusual but very well received format: In the Hadoop bug tracker there are several issues marked as trivial (typos in documentation and the like). Attendees were asked to choose one of these issues, checkout the source, create a patch and contribute it back to the project. Optionally they got explained how the process continues from there on the committer side of things. It may seem trivial to mechanically go through the patch process, however it help lower the bar in case you have a real issue to fix to first get accustomed to just how it works. If instead of contributing to Apache you are more into working on the Linux kernel I’d like to advise you to watch Greg Kroah Hartman on writing and submitting your first Linux kernel patch (FOSDEM).
• Last but not least make sure to lower the bar for contribution - do not require people to jump through numerous loops, in general even just getting a patch ready is complicated enough. Provide a how to contribute page (e.g. see how to contribute and how to become a committer pages in the Apache Mahout wiki.
• In particular when your project is still very young lower the bar by turning contributors into committers quickly - even if they are “just” contributing documentation fixes - in my view one of the most important contribution there is as only users spot areas for documentation improvement.

In case you yourself are thinking about contributing and need some additional advice as to why and for what purposes: Dr Dobbs has more information on reasons why developers tend to start to contribute to Apache software, Shalin explains why he contributes to open source, on the Mahout mailing list we hade a discussion on why also students should consider contributing, on the Apache community mailing list there was an interesting discussion on whether developers working on open source are happier than those that don’t.

Apache Con Apache, contributing, Open Source