Inductive Bias

Mid November Apache hosts its famous yearly conference - this time in Vancouver/Canada. They kindly accepted my presentations on Apache Mahout for intelligent data analysis (mostly focused on introducing the project to new comers and showing what happened within the project in the past year - if you have any wish concerning topics you would like to see covered in particular, please let me know) as well as a more committer focused one on Talking people into creating patches (with the goal of highlighting some of the issues new-comers to free software projects that want to contribute run into and initiating a discussion on what helps to convince them to keep up the momentum and over come and obstacles).

Looking forward to seeing you in Vancouver for Apache Con NA.

Apache Con, Free Software, Mahout ApacheConNA, Vancouver

This evening Mary Poppendieck gave a presentation on Design Thinking in Berlin. “What does the US Military have in common with Apple?” In her talk Mary went into more detail about how increasing complexity calls for a need to rethink design. Slides are available online. Thanks for sharing them.

Mary started with a question for the audience on who in your company decides what is to be developed: Is it the PO? Is it a “them” or is it “us”? In a world of growing complexities and faster innovation cycles, thinking of them doing the design more and more becomes a huge business risk. It’s time to start thinking of development teams as groups of people solving a business need - forgetting about the “that’s none of my business” attitude.

Lets first dig deeper into the three steps of design (according to the book The design of design) the three steps of design are:

• Understanding the problem: Actually finding out what is the problem, what is to be designed is the hardest part. It’s best done in a small team as opposed to by a single person.
• Design the solution: This step involves uncovering specific requirements and identifying alternative solutions - before actually being able to judge whether an alternative is a viable way of solving a given problem, the harder part is identifying possible alternative solutions.
• Implementing the design: This step must involve getting users and developers of the resulting system work closely together.

To come up with a truly successful design these three steps most likely need to be iterated as what we learn in later steps has to influence earlier steps.

When it comes to design, at traditional corporations there are various job titles for people involved with design. The graph distinguishes teams developing products vs. supporting internal processes. It makes a difference between software and hardware development. In addition there is a difference between people deciding on what to design (denoted in red) and those deciding how to develop (denoted in green). When looking at Scrum all of these roles are combined in the role of the product owner. When taking a closer look this seems to be a large over-simplification - especially when taking into account that the product owner in general is realized not as a role that has to be provided for but as a job description of one single person. It seems like this is too much work and too many hats for one single person.

In contrast Mary advocates a far more hollistic definition of the designers role: In current teams design is mostly being done up-front, in a very detailed way with results in the form of small user stories being handed over to the team afterwards. In contrast to that developers should be integrated in the design process itself - they need to participate, provide their input to come up with really innovative solutions.

Step one: Understanding the problem

The presentation went into more detail on three different approaches to solving the first step in design - that is understanding the problem. The first way was coined as the military approach. In the The operations process a chapter on design was added to cope with complex situations that do not match the expected configuration.

In such situations a combination of collaboration, dialog, critical thinking and creative thinking seems more important than blindly following command. According to the US military’s procedures design again is an iterative three step process: From framing the problem, to experimenting and building prototypical solutions, to making sense of the situation and possibly going back to earlier steps based on what we have learned. One important feature of successful design teams is that the are composed of people with diverse backgrounds to not miss details about the situation.

The second approach to design is the Ethnographic approach. Mary showed a video introducing design company IDEO. The interesting take away for me was that again teams need to be diverse to see all details, that designers need not only find statistics on existing products but also go out to the customer using the product and listen to those “experts”. The need to work under time constraints, work in a very focused manner, build prototypes - not to deploy them but to learn from them, merge them and throw away what does not work early in the process.

Coming from a data driven, Hadoop-y background, the third approach was the one most familiar to me: The data based approach focuses on finding success metrics and optimising those. When going back to the “Mythical man month” example of IBM building the IBM PCjr - the team spending 2 years developing a product that was a horrible failure, a product that was judged very badly in common it magazines: What could they have changed to do better? Well the solution almost is in the question: If there are people writing articles to judge products, people that have criteria for judging product success - why not optimise for their criteria? Another example would be Toyota developing the Lexus: Early in the process they decided to optimise for a top score on each of the car magazine reviews. Turned out though hard it was, it was not impossible. And lead to huge success on the sales side.

So instead of being caught in our little development bubble maybe it’s time to think bigger - to think not only about what makes our project successful but to think about what makes business successful. And while at it - why not measure success or failure of a piece of software by how successful it is monetarily? This leads to a whole new way of thinking about development:

• Instead of a product roadmap we start thinking about a business model canvas: What metric gives us information on the business success of a new feature? How can we measure that? This can be as easy as click conversion rate. It’s closely tied to the way you make money.
• Instead of a product vision we turn to measuring a product to market fit.
• Instead of a release plan we start thinking about the minimal viable product: What is the least you can do to generate more revenue?
• Instead of an on-site customer we start having a on-site developer. This is about getting developers out of the building, get them to talk to your users and get this experience back in. Note: To some extend this is even possible by looking over the user’s shoulder by analysing your log files. Ever thought of taking a closer look at your search engines queries that return no results? Maybe instead of cursing the users that are too stupid to use your interface it’s time to re-think your interface design or add more features to your search engine.
• Instead of iterations we have a loop of building - measuring - learning about our product.
• Instead of an iteration backlog at the end of each loop we have to decide about whether to preserve the current development direction or pivot it a bit.
• Instead of a backlog of things to do we have a list of things to learn about our users.
• Instead of detailed tiny user stories we have hypothesis of how our product is used that are validated or falsified by actual data.
• Instead of continuous integration you have continuous deployment that is much more interesting.
• Instead of acceptance tests we may have split tests, A/B tests or similar.
• Instead of a definition of done including only development it suddenly also comprises validating the data - did we actually convert more users?
• Instead of costumer feedback we are having a cohort-based metric, we start watching our users to learn - they are rarely willing to give explicit feedback, but their actions are very telling.
• Instead of a product owner we have entrepreneurs that have the business as a whole in mind.

Design a solution

Good industrial design is made up of ten principles: It’s innovative, makes the product useful, understandable, aesthetic, unobtrusive, honest (does not fool users), long lasting, thorough through, environmentally friendly - and most important of all there is as few design in there as possible.

As one example Mary mentioned the flight booking site Hipmunk.com that gives users the chance to sort flights by least agony. It optimises for the best flying experience - not the best buying experience.

In the end it’s all about building what the user needs - not selling what your build. It’s about watching your users and learning about their needs.

Implementing design

One grave warning: Do not divorce design from development: Ford was a driver and racer himself, making him a great car designer. Wright flew the aircrafts he designed. Tom Edison thought through the whole process. Also in the tech industry the companies that are nowadays most successful were founded by people who are or at least were themselves very actively using the stuff they are building - think Bill Gates, Steve Wozniak and Steve Jobs, Sergey Brin and Larry Page. Takeaway is to do the job you are trying to automate, to have incremental development and delivery.

In the end it all boils down to how fast a learner you are: Not the most intelligent nor the strongest will survive. It’s the one that is fastest to adapt to change that will be successful. One commonly used analogy for that is the so-called OODA cycle coming from John Boyd: The tighter your cycle of observing, orienting, deciding and acting is, the likelier you are to be successful among your competitors.

Speaking of release cycles Mary gave a very nice graph of various release cycle lengths.

In her experience teams that have deployment cycles of six months and above are likely to spend one third of their time hardening their software. If you are down to a quarter it’s likely that the bottleneck is your business model. Maybe it is weird contracts or business partners expecting support for the exact version you shipped years ago. When you are down to a month per release you likely are running a software as a service model. As clarified later after a question from the audience for Mary this does include setups where customers basically buy a software including all future updates for a set period of time - including an automated roll-out. If you are down to daily or even just weekly releases you are likely running the product yourself. Mose likely then everyone is part of the team.

After her talk Mary answered various questions from the audience: According to her experience Scrum defines the team way too narrow by focusing on development only - with Scrum we are possibly getting perfect at doing the wrong thing very well - forgetting all about design and business.

One very general advise that leads to an interesting conclusion is to not waste your life, to not spend years of your life working on projects that are not needed by anyone. In the end this leads to realizing that failed projects are also caused by the developers not raising concerns and issues.

Mary is a huge proponent of trusting your team: Do not hand them detailed lists of user stories. You have hired great, senior people (or haven’t you??) - give them an idea what you want and let them figure out the details by talking to the right people.

When asked for tools to support the lean startup process in a startup Mary emphasized the need for communication. Tools are likely to only slow you down: Get people to communicate, to sit at one table. Have a weekly or bi-weekly meeting to get everyone in sync. Otherwise let developers figure out their way to collaborate.

The talk was kindly hosted by Immobilien Scout, organised by agiliero with some tickets handed over to the Berlin Scrumtisch. Thanks guys.

Scrum agile, lean, Scrum, scrumtisch

Together with Sebastian Schelter I held a Nokia sponsored (Thank you!) lecture on large scale data analysis and data mining during the past few months.

After supervising a few successful university projects based on Apache Mahout the goal of this lecture was to introduce students to some of the basic concepts and problems encountered today in a world where huge datasets are generally available and are easy to process with Apache Hadoop. As such the course is targeted at an entry level audience - thorough treatment of the mathematical background of latest machine learning technology is left to the machine learning research groups in Potsdam, at TU Berlin and the neural information processing group at TU.

Slides and exercises are available online via git. Please let me know if you want to re-use them in your lecture.

The very first problem that users of machine learning algorithms usually come across is mapping their application problem to one of the various machine learning problems. In 2010 Michael Brückner gave a lecture on Intelligent Data Analysis with Matlab (slides and videos in German) including a simple taxonomy of algorithms:

According to

• the types of input data an algorithm can handle (either independent instances, also called examples, sequences or graphs of instances)
• the type of training data available (e.g. instances with assigned nominal target attribute, no labels at all, a partial sorting of sets of instances)
• and the learning goal

algorithms can be nicely partitioned by the learning problem that they solve. Based on that very first step of identifying exactly what the problem setting is, deciding which algorithm to use becomes much easier. Based on that taxonomy I came up with the above graphic giving a first overview of which tasks can be solved with machine learning:

Boxes in dark blue are what in general is called supervised learning, yellow unsupervised and light blue semi supervised - based on the amount of labeled training data available. Red boxes indicate settings with the goal of knowledge discovery. Green are any ranking problems.

Science Apache Mahout, Machine Learning, Theory

Last year Sebastian Schelter from Berlin was added to the list of committers for Apache Mahout. With two committers in town the idea was born to meet some day, work on Mahout. So why not just announce that meeting publicly and invite others who might be interested in learning more about the framework? I got in touch with c-base - a hacker space in Berlin well suited to host a Hackathon - and quickly got their ok for the event.

As a result the first Apache Mahout Hackathon took place at c-base in Berlin last weekend. We had about eight attendees - arriving at varying times: I guess 11a.m. simply is way too early to get up for your average software developer on a Saturday. I got a few people surprised by the venue - especially those who were attending a Hackathon for the very first time and had expected c-base to be some IT company

We started the day with a brief collection of ideas that everyone wanted to work on: Some needed help to use Mahout - topics included:

• How to use Apache Mahout collaborative filtering with complex models.
• How to use Apache Mahout via a web application?
• How to use classification (mostly focussed on using Naive Bayes from within web applications).
• Is HBase a solution for scalable graph mining algorithms?
• Is there a frequent itemset algorithm that respects temporal changes in patterns?

Those more into Mahout development proposed a slightly different set of topics:

• PLSI and Map/Reduce?
• Build customisable sampling strategies for distributed recommendations.
• Come up with a more Java API friendly configuration scheme for Mahout clusterings.
• Complete the distributed SVD recommender.

Quickly teams of two to three (and more) people formed. First several user side questions could be addressed by mixing more experienced Mahout developers with newbie users. Apart from Mahout specifics also more basic questions of getting involved even by simply contributing to the online documentation, answering questions on the mailing lists or just providing structured access to existing material that users generally have trouble finding.

Another topic that is being overlooked all too when asking users to contribute to the project is the process of creating, submitting, applying and reviewing patches itself: Being deeply involved with free software projects dealing with patches, integration of issue tracker and svn with the project mailing lists all seems very obvious. However even this seemingly basic setup sometimes looks confusing and complex to regular users - that is very common but not limited to people who are just starting to work as software developers.

In the evening people finally started hacking more sophisticated tasks - working on the first project patches. On Sunday only the really hard core developers remained - leading to a rather focussed work on Mahout improvements which in the end led to first patches sent in from the Mahout Hackathon.

Hacking, Mahout Apache Mahout, c-base, Hackathon, Mahout

Putting the information here to make retrieving it a bit easier next time.

When working with svn and some random IDE I’d really love to avoid checking in any files that are IDE specific (project configuration, classpath, etc.). The command to do that:

svn propedit svn:ignore $directory_to_edit


After issuing this command you’ll be prompted to enter file patterns for files to ignore or the directory names.

More detailed information in the official documentation on svn:ignore.

Hacking cheat sheet, svn, trivia

The February Scrumtisch Berlin featured a talk by Lyssa Adkins, famously known for her publications on Coaching Agile teams. A mixture of fifty developers, scrum masters, coaches and product owner as well as one project manager followed Marion Eikmann’s invitation. Thanks for organising the event, as well as thank you to Hypoport for providing the venue.

In her one-hour presentation she mainly focussed on two core topics: On the roles agile coaches have to fullfill as well as on the skill set needed by agile coaches. Being an agile coach (as well as being a scrum master, really), entails four very basic roles:

• Being a bulldozer - that is getting impediments out of the way. That can be as easy (or hard) as getting appropriate equipment for your developers or teaming up with the product owner to communicate with anyone trying to break the sprint.
• Being a servant leader - for a coach that means to work for the team, to ask the right questions and enable the team, to listen during dailies - instead of asking for additional reports: Most tools are already within the Scrum toolbox. The hard task is to identify them and use them correctly and efficiently.
• Being a shepherd - that may be as easy as getting people to attend the dailies, or as complex as communicating common values, a common goal.
• Being the guard of quality and performance - as a coach that means making degrading quality visible - and letting the Scrum team take the decision on how to deal with it.

However in reality each team is different, each sprint is different. So coaching really is similar to bing the river guide: Each trip is different. It is your task to make the team succeed and adapt to differing situations. To get to team to high performance - over and over again.

When talking about coaching what people need is a very specific skill set:

• To become a successful agile coach it helps to have coaching skills to be able to understand the client, to see their impediments and help the team become better by listening and asking powerful questions.
• Be a facilitator who organises sessions, meetings, conversations and may mediate in meetings.
• Have business domain expertise - that is to know process designs, figure out a companies strategy options.
• Be a great teacher to help people understand the basic concepts. That entails knowledge on course design, most likely design of exercises.
• Have mentoring skills to guide a team in the process.
• Know about lean and agile processes and stay up to date.
• Have the technical skills on what makes development successful, know the extreme programming techniques to help team excel.
• Have transformational skills - that is be familiar with change management, transformation leadership, knowing how to change organisations.

However being a coach remember that people are influenced not only by what you teach - but mostly by what you do. Most of what being a good coach means is invisible to the uninitiated outsider. It’s about living the process you teach, staying true to the principles you want others to implement.

To get there it helps to get inspired, to talk with others in local meetups (just as with any coding practice: Try to find a common forum to exchange ideas and get fresh input). It may help to keep a value journal - not only to track your accomplishments and be able to prove them to higher management, but also to track where to improve yourself.

The talk provided several interesting starting points for exploring further. However with an added exercise sixty minutes covered only the very basic ideas. Especially for the skill sets needed to successfully coach teams it would be very interesting to learn more on what books to read or what courses to attend to learn more on each topic. Ultimately not only agile coaches benefit from being great teachers, mentors or facilitators.

Scrum agile, coaching, lean, Scrum

Devoxx featured several interesting case studies of how HBase and Hadoop can be used to scale data analysis back ends as well as data serving front ends.

Twitter

Dmitry Ryaboy from Twitter explained how to scale high load and large data systems using Cassandra. Looking at the sheer amount of tweets generated each day it becomes obvious that with a system like MySQL alone this site cannot be run.

Twitter has released several of their internal tools under a free software license for others to re-use – some of them being rather straight forward, others more involved. At Twitter each Tweet is annotated by a user_id, a time stamp (ok if skewed by a few minutes) as well as a unique tweet_id. In order to come up with a solution for generating the latter one they built a library called snowflake. Though rather simple algorithm even works in a cross data-centre set-up: The first bits are composed of the current time stamp, the following bits encode the data-centre, after that there is room for a counter. The tweet_ids are globally ordered by time and distinct across data-centres without the need for global synchronisation.

With gizzard Twitter released a rather general sharding implementation that is used internally to run distributed versions of Lucene, MySQL as well as Redis (to be introduced for caching tweet timelines due to its explicit support for lists as data structures for values that are not available in memcached).

FlockDB for large scale social graph storage and analysis. Rainbird for time series analysis, though with OpenTSDB there is something comparable available for HBase. Haplocheirus for message vector caching (currently based on memcached, soon to be migrated to Redis for its richer data structures). The queries available through the front-end are rather limited thus making it easy to provide pre-computed, optimised version in the back-end. As with the caching problem a tradeoff between hit rate on the pool of pre-computed items vs. storage cost can be made based on the observed query distribution.

In the back-end of Twitter various statistical and data mining analysis are run on top of Hadoop HBase To compute potentially interesting followers for users, to extract potentially interesting products etc.
The final take-home message here: Go from requirements to final solution. In the space of storage systems there is not such thing as a silver bullet. Instead you have to carefully evaluate features and properties of each solutions as your data and load increase.

Facebook

When implementing Facebook Messaging (a new feature that was announced this week) Facebook decided to go for HBase instead of Cassandra. The requirements of the feature included massive scale, long-tail write access to the database (which more or less ruled out MySQL and comparable solutions) and a need for strict ordering of messages (which ruled out any eventually consistent system. The decision was made to use HBase.

A team of 15 developers (including operations and frontend) was working on the system for one year before it was finally released. The feature supports for integration of facebook messaging, IM, SMS and mail into one single system making it possible to group all messages by conversation no matter which device was used to send the message originally. That way each user’s inbox turns into a social inbox.

Adobe

Cosmin Lehene presented four use cases of Hadoop at Adobe. The first one dealt with creating and evaluating profiles of the Adobe Media Player. Users would be associated with a vector giving more information on what types of genre the meda they consumed belonged to. These vectors would then be used to generate recommendations for additional content to view in order to increase consumption rate. Adobe built a clustering system that would interface Mahout’s canopy- and k-means implementations with their HBase backend for user grouping. Thanks Cosmin for including that information in your presentation!

A second use case focussed on finding out more on the usage of flash on the internet. Using Google to search for flash content was no good as only the first 2000 results could be viewed thus resulting in a highly skewed sample. Instead they used a mixture of nutch and HBase for storage to retrieve the content. Analysis was done with respect to various features of flash movies, such as frame rates. The analysis revealed a large gap between the perceived typical usage and the actual usage of flash on the internet.

The third use case involves analysis of images and usage patterns on the Photoshop-in-a-browser edition of Photoshop.com. The forth use case dealt with scaling the infrastructure that powers businesscatalyst – a turn-key online business platform solution including analysis, campaigning and more. When purchased by Adobe the system was very successful business-wise. However the infrastructure was by no means able to put up with the load it had to accommodate. Changing to a back-end based on HBase led to better performance, faster report generation.

General, Hacking, Mahout adobe, Devoxx, facebook, hbase, Mahout, NOSQL, twitter

Day two started with a really good talk on caching architectures by Greg Luck. He first motivated why caching works: Even with SSIDs being available now there is still a huge performance gap between RAM access times and having to go to disk. The issue is even worse in systems that are architected in a distributed way making frequent calls to remote systems.

When sizing systems for typical load, what is oftentimes forgotten is that there is no such thing as typical load: Usually the load distribution observed over one day for a service used mainly in one time zone has the shape of an elephant – most queries are issued during lunch time (head of the elephant) with another but smaller peak during the afternoon. This pattern repeats when looking at the weekly distribution, repeats again when looking at the yearly distribution. When looking at the peak time of the year, at the peak day, at the peak time your lead may be increased by several orders of magnitude compared to average load.

Although query volume may be high in most applications that reach out for caching, these queries usually exhibit a power law distribution. This means that there are just a few queries being issued very frequently, however many queries are pretty seldom. This pattern allows for high cache hit rates thus reducing load substantially even during very busy times.

The speaker went into some more detail concerning different architectures: Usually projects start with one cache located directly on the frontend server. When scaling horizontally and adding more and more frontends this leads to an ever increasing load on the database during one period of lifetime for one cached item. The first idea employed to remedy this setup is to link the different caches to each other increasing cache hit rates. Problem here are updates racing to the various caches when the same query is issued to the backend by more than one frontend. The usual next step is to go for a distributed remote cache such as memcache. Of course this has the draw-back of now having to do a network call for each cache access slowing down response times by several milliseconds. Another problem with distributed caching systems is a theorem well known to people building distributed NoSQL databases: CAP says that you can get only two of the three desired properties consistency, availability and partition-tolerance. Ehcache with a terracotta back end lets you configure where your priority lies.

Hacking caching, Devoxx, Java

The first day at Devoxx featured several tutorials – most interesting to me was the pragramatic programmer. The speaker also is the author of the equally named book at O’Reilly. The book was the result of the observation that developers today are more and more IDE bound, no longer able to use the command line effectively. The result are developers that are unnecessarily slow when creating software. The goal was to bring usage patterns of productive software development to juniors how grew up in a GUI only environment. However half-way through the book, it became apparent that a book on command line wizardry only is barely interesting at all. So the focus was shifted and now includes more general productivity patterns.
The goal was to accelerate development – mostly by avoiding time consuming usage patterns (minimise mouse usage) and automation of repetitive tasks (computers are good at doing dull, repetitive tasks – that’s what they are made for.
Second goal was increasing focus. Two main ingredients to that are switching off anything that disturbs the development flow: No more pop-ups, not more mail notifications, no more flashing side windows. If you have ever had the effect of thinking “So late already?” when your colleagues were going out for lunch – then you know what is meant by being in the flow. It takes up to 20min to get into this mode – but just the fraction of a second to be thrown out. With developers being significantly more productive in this state it makes sense to reduce the risk of being thrown out.
Third goal was about canonicality, fourth one on automation.
During the morning I hopped on and off the Hadoop talk as well – the tutorial was great to get into the system, Tom White went into detail also explaining several of the most common advanced patterns. Of course not that much new stuff if you sort-of know the system already

General, Hacking antwerp, Devoxx, hbase, productive programmer

Today the last Scrumtisch Berlin in 2010 took place in Friedrichshain. Thanks to Marion Eickmann and Andrea Tomasini for organising the Scrum user group regularly for the past years.

Though no presentation had been scheduled ahead of time the Scrumtisch was well attended by over twenty people, mostly from companies based in Berlin who are either using Scrum already or are currently in a transition phase.

We went straight into collecting and voting on topics for discussion. In total we ended up having eight potential topics listed, including but not limited to

• Scrum and non-feature teams, does it work - and if, how?
• Transitioning to Scrum - what are the stake holders that must be convinced first in a company?
• Scrum with teams made of people buying into Scrum and those that don’t - does that work?
• Can Scrum be combined with people who want to telecommute?
• Scrum and platform development - how does that get combined?
• Scrum in systems engineering, embedded development - how to setup teams?

After voting we had the two clear winners discussing Scrum in teams that don’t buy into the method completely as well as telecommuting with Scrum teams.

Scrum with broken teams

The situation described: The attendee proposing the topic has the problem of being Scrum master at a team that does not completely buy into Scrum. There are a few developers who like being self-organising, who love short feedback cycles. However there are a few others who would rather stick with their technological niche, get tasks assigned to them and avoid taking over tasks from others.

During discussion we found out that in this company Scrum had been introduced as a grass-roots movement little over a year ago. The introduction of the method led to success clearly visible in the company. In turn the method was tried on a larger team as well. However at the moment the team is at a point where it is about to break apart: Into developers happy with change, flexible enough to adapt to shift in technology and a second half that would rather continue developing the old way.

One very important point was raised by one of the attendees: With Scrum getting introduced so fast, compared to the length in time the company had been living before, it may well be time to slow down a bit. To sit down with the team in a relaxed environment and find out more on how everyone is assessing the current situation. Find out more on what people like about the new approach, and about what should be changed and still needs improvement. In the end it’s not a process issue but a people problem - there is a need to get the team on-board.

Team-building activities might help as well - let the team experience what it means to be able to rely on each other. What does it mean to learn new things in short time, to co-operate to solve tasks so far untackled?

If team-members start ignoring the sprint backlog working on other tasks instead there is a question about whether there is enough trust in the product-owner’s decisions. On the other hand with pressure resting on the team’s shoulders there might be a need to stop the train, fix all open issues and continue only after the project is back in shape. However also this needs all team members working towards a common goal - with everyone willing to take up any open task.

Scrum and telecommuting

Basically the question was whether it works at all (clear yes from the audience) and if, which best practices to use. To be more precise: Does Scrum still work if some of the team members work from home a few days a week but are in the office all other time. The risk of course lies in loosing information, in the team building common knowledge. And thus becoming less productive.

There are technical tools that can help the process: electronic scrum boards (such as Greenhopper for JIRA or Agilo) as well as tele-conferencing systems, wikis, social networking tools, screen sharing for easier pair programming. All tools used must entail less overhead then the provide in benefit to the team however. Communication will become more costly - however if and to what extend this translates to a loss in productivity varies greatly.

There must be a clear commitment from both sides - the telecommuter as well as the team on-site - to keep the remote person in the loop. Actually it is easier with teams that are completely remote. This experience is sort of familiar from any open source project: With people working in different time zones it comes naturally to take any decision on a mailing list. However with some people having the chance to communicate face-to-face suddenly decisions become way less transparent. At Apache we even go as far as telling people that any decision that is not taken on the mailing list, never really was taken at all. A good insight into how distributed teams at Apache work has been given earlier by Bertrand Delacrétaz.

For team building reasons it may make sense to start out with a co-located team and split off people interested in working from home later on. That way people have a chance to get to know each other face-to-face which makes later digital-only communication way easier.

Thanks again to Marion and Andrea for organising today’s Scrumtisch. If you are using Scrum and happen to be in Berlin - send an e-mail to Marion to let her know you are interested in the event, or simply join us at the published date.

Scrum Scrum, Self organising teams, Software Foundation, team, Team Motivation