Inductive Bias

Last week I had the honour of giving a keynote at Adobe's Open Source Summit EU in Basel. Among many interesting talks they hosted a panel to answer questions around all things open source, strategy, inner source, open development. One of the questions I found intersting is how inner source/ open development and agile are related.

To get everyone on the same page, what do I mean when talking about inner source/ open development? I first encountered the concept at ApacheCon EU Amsterdam in 2008: Bertrand Delacretaz, then Day Software, now Adobe was talking about how to apply open source development principles in a corporate environment. A while ago I ran across the term http://innersourcecommons.org that essentially wants to achieve the same thing. What I find intersting about it is the idea of applying values like transparency, openess to contributions as well as concepts like asynchronous communication and decision making at a wider context.

What does that have to do with Agile? Looking at the mere mechanics of Agile like being co-located, talking face to face frequently, syncing daily in in person meetings this seems to be pretty much opposite of what open source teams do.

Starting with the values and promises of Agile though I think the two can compliment each other quite nicely:

"Individuals and interactions over processes and tools" - there's a saying in many successful popular projects that the most important ingredient to a successful open source project is to integrate people into a team working towards a shared vision. Apache calls this Community over code, ZeroMQ's C4 calls it "People before code", I'm sure there are many other incarnations of the same concept.

Much like in agile, this doesn't mean that tools are neglected alltogether. Git was created because there was nothing quite like this tool before. However at the core of it's design was the desire for ideal support for the way the Linux community works together.

"Working software over comprehensive documentation" - While people strive for good documentation I'd be so bold as to suggest that for the open source projects I'm familiar with the most pressing motivation to provide good documentation is to lower the number of user questions as well as the amount of questions people involved with the project have to answer.

For projects like Apache Lucene or Apache httpd I find the documentation that comes with the project to be exceptionally good. In addition both projects are supported with dead tree documentation even. My guess would be that working on geographically distributed teams who's members work on differing schedules is one trigger for that: Given teams that are geographically distributed over multiple time zones means that most communication will happen in writing anyway. So instead of re-typing the same answers over and over, it's less cost intensive to type them up in a clean way and post them to a place that has a stable URL and is easy to discoveri. So yes, while working software clearly is still in focus, the lack for frequent face to face communication can have a positive influence on the availability of documentation.

"Customer collaboration over contract negotiation" - this is something open source teams take to the extreme: Anybody is invited to participate. Answers to feature requests like "patches welcome" typically are honest requests for support and collaboration, usually if taken up resulting in productive work relationships.

"Responding to change over following a plan" - in my opinion this is another value that's taken to the extreme in open source projects. Without control over your participants, no ability to order people to do certain tasks and no influence over when and how much time someone else can dedicate to a certain task reaciting to changes is core to each open source project cosisting of more than one maintaining entity (be it single human being or simply one company paying for all currently active developers).

One could look further, digging deeper into how the mechanics of specific Agile frameworks like Scrum match against what's being done in at least some open source projects (clearly I cannot speak for each and ever project out there given that I only know a limited set of them), but that's a topic for a another post.

The idea of applying open source development practices within corporations has been hanging in the air for over 15 years, driven by different individuals over time. I think now that people are gathering to collect these ideas in one place and role them out in more corporations discussions around the topic are going to become interesting: Each open source community has their own specifics of working together, even though all seem to follow pretty much the same style when looked at from very high above. I'm curious to what common values and patterns can be derived from that which work across organisations.

“Agile and Lean have a single purpose: to continually challenge the status quo. If you’re not doing that, you’re probably an impediment to it.”

agile42.com

Judging from the way some people become overly careful when discussing agile in general and Scrum in particular in my presence I seem to slowly have built up a reputation for being a strong proponent of these methods. Given the large number of flaky implementations as well as misunderstandings it seems to have become fashionable to blame Scrum for all badness and dismiss it altogether - up to the point where developers are proud to finally having abandoned Scrum completely - so that now they

• can work in iterations,
  
• accept new tasks only for upcoming but not for the current iteration,
  
• develop in a test-driven way,
  
• have daily sync meetings,
  
• mark tasks done only when they are delivered to and accepted by the customer,
  
• have regular “how to improve our work” meetings,
  
• estimate tasks in story points and only plan for as much work per iteration as was done in the past iteration
  

… my personal take on that: Add in regular releases and you end up with a pretty decent scrum/agile implementation, no matter what your preferred name for it may be. Just for clarification: Though very often I write about what I call Scrum, I don’t use that particular method just because it is the latest fashion. It simply is a tool that has served me well on multiple occasions and given me working guidelines when I had no idea at all what software development in a professional setting should look like.

So where does all that friction with anything Scrum, agile, lean or whatever you call it come from? Recently I came across a blog post that jillesvangurp.com nicely identified some grave issues with current Scrum adoption. Unfortunately the blog post only lists the failures without going into a deeper analysis of the actual defects causing those failures.

First of all, lets assume as working hypothesis that Scrum in itself does not solve any issues in your organisation but is a great tool to uncover deficiencies. The natural conclusion should be to use it as a tool to discover problems, but search for solutions for these problems elsewhere.

With that hypothesis, lets discern the the issues discussed in the post above and assign them to one of three defect categories.

Category one: Issues with the team

Problem: You have a team of all-junior developers, or of all-mediocre developers.

Goal: Turn them into a high performing team.

Solution: Imagine you were not using Scrum at all, what would be the ideal solution? Well the obvious route probably is to re-adjust the team, add several seniors so that you end up with the right mix of people that have experience and share a vision - juniors than can learn and adapt what works from them.

Comparing that to our hypothesis: Scrum is all for short delivery cycles. You will uncover teams that perform badly much faster than in methods with longer iteration periods. So it should be reasonably simple to figure out teams that have a dysfunctional configuration. Changing that configuration however no longer is dictated by Scrum.

Category two: Bugs introduced during Scrum roll-out

The failures discussed in the blog post include people following Scrum mechanically: Only because your developers are moving post-it notes from left to right does not mean they are doing anything agile. It’s perfectly possible to do waterfall in Scrum. Whether that helps solve any of your issues is a different matter.

Instead of mechanically going through the process what is more important is to understand the reasons and goals of each of the pieces that form Scrum. To make a rather far fetched comparison:



When introducing Scrum without a deep understanding of why each piece is done, what you end up with is people following that process without understanding the meaning of each step. They end up mimicking behaviour without knowing the real world: To some extend seeing only the shadows of good development patterns without understanding the real items producing these shadows.

As a general rule: Understand why there is a retrospective meeting, remember why you need estimations, think about why there are daily stand-ups (instead of weekly meetings, instead of daily sit-togethers, instead of hourly stand-ups). Figure out why there is a product owner, what the role of a scrum master does. Pro-Tip: As soon as you really have understood Scrum, you don’t need a checklist of all meetings to hold for a successful iteration - they will just fit in naturally. If they don’t, you are probably missing an important piece of the puzzle - rather than rely on a pre-fabricated checklist, go bug your trainer or coach with questions to better understand the purpose of all the different bits and pieces.

One very grave bug on roll-out is the general believe that Scrum is equal to a little bit of fairy dust you spread over your teams and all problems will automatically be solved afterwards. It is not - it’s not a silver bullet, it’s not fairy dust, it’s no magic - such things exist in fairy tales but have been seen nowhere in the real world. According to our working hypothesis above however Scrum does something really magical: By shortening delivery cycles it introduces short feedback loops which make it very easy to uncover problems in your organisation way faster than people are able to cover them up and hide them. Finding a solution on the other hand is still up to you.

The last roll-out issue mentioned is that of crappy certification - current certification programs are designed such that the naive organisation may believe that after two days of training their employers will magically turn into super heroes. Guess what - as with any certification training is just the very first step. Actual understanding comes with experience. Compare that to learning to drive: Only because you managed to get a drivers license does not turn you into a formula one winner. Instead that requires a whole lot of training. Same applies for any Scrum Master or Product Owner.

Category three: Organisation specifics

All other issues with Scrum mentioned in the blog post are either specific to the broken structures in the organisation under investigation or due to general Scrum mis-conceptions. Leaving these aside here.

To sum up: Scrum to me is nothing but a term to summarize good, proven development practices. I don’t care how you name them - however having any one name that is well defined makes it way easier to communicate. Scrum is not silver bullet - it does not solve all the issues your organisation may have. However it is a very effective debugger uncovering the issues employees and managers are trying to cover up. If you know all those issues very precisely already or you are certain that you don't have any, chances are you don't need Scrum.

This evening Mary Poppendieck gave a presentation on Design Thinking in Berlin. "What does the US Military have in common with Apple?" In her talk Mary went into more detail about how increasing complexity calls for a need to rethink design. Slides are available online. Thanks for sharing them.

Mary started with a question for the audience on who in your company decides what is to be developed: Is it the PO? Is it a "them" or is it "us"? In a world of growing complexities and faster innovation cycles, thinking of them doing the design more and more becomes a huge business risk. It's time to start thinking of development teams as groups of people solving a business need - forgetting about the "that's none of my business" attitude.

Lets first dig deeper into the three steps of design (according to the book The design of design) the three steps of design are:

• Understanding the problem: Actually finding out what is the problem, what is to be designed is the hardest part. It's best done in a small team as opposed to by a single person.
  
• Design the solution: This step involves uncovering specific requirements and identifying alternative solutions - before actually being able to judge whether an alternative is a viable way of solving a given problem, the harder part is identifying possible alternative solutions.
  
• Implementing the design: This step must involve getting users and developers of the resulting system work closely together.
  

To come up with a truly successful design these three steps most likely need to be iterated as what we learn in later steps has to influence earlier steps.

When it comes to design, at traditional corporations there are various job titles for people involved with design. The graph distinguishes teams developing products vs. supporting internal processes. It makes a difference between software and hardware development. In addition there is a difference between people deciding on what to design (denoted in red) and those deciding how to develop (denoted in green). When looking at Scrum all of these roles are combined in the role of the product owner. When taking a closer look this seems to be a large over-simplification - especially when taking into account that the product owner in general is realized not as a role that has to be provided for but as a job description of one single person. It seems like this is too much work and too many hats for one single person.

In contrast Mary advocates a far more hollistic definition of the designers role: In current teams design is mostly being done up-front, in a very detailed way with results in the form of small user stories being handed over to the team afterwards. In contrast to that developers should be integrated in the design process itself - they need to participate, provide their input to come up with really innovative solutions.

Step one: Understanding the problem

The presentation went into more detail on three different approaches to solving the first step in design - that is understanding the problem. The first way was coined as the military approach. In the The operations process a chapter on design was added to cope with complex situations that do not match the expected configuration.



In such situations a combination of collaboration, dialog, critical thinking and creative thinking seems more important than blindly following command. According to the US military's procedures design again is an iterative three step process: From framing the problem, to experimenting and building prototypical solutions, to making sense of the situation and possibly going back to earlier steps based on what we have learned. One important feature of successful design teams is that the are composed of people with diverse backgrounds to not miss details about the situation.

The second approach to design is the Ethnographic approach. Mary showed a video introducing design company IDEO. The interesting take away for me was that again teams need to be diverse to see all details, that designers need not only find statistics on existing products but also go out to the customer using the product and listen to those "experts". The need to work under time constraints, work in a very focused manner, build prototypes - not to deploy them but to learn from them, merge them and throw away what does not work early in the process.

Coming from a data driven, Hadoop-y background, the third approach was the one most familiar to me: The data based approach focuses on finding success metrics and optimising those. When going back to the "Mythical man month" example of IBM building the IBM PCjr - the team spending 2 years developing a product that was a horrible failure, a product that was judged very badly in common it magazines: What could they have changed to do better? Well the solution almost is in the question: If there are people writing articles to judge products, people that have criteria for judging product success - why not optimise for their criteria? Another example would be Toyota developing the Lexus: Early in the process they decided to optimise for a top score on each of the car magazine reviews. Turned out though hard it was, it was not impossible. And lead to huge success on the sales side.

So instead of being caught in our little development bubble maybe it's time to think bigger - to think not only about what makes our project successful but to think about what makes business successful. And while at it - why not measure success or failure of a piece of software by how successful it is monetarily? This leads to a whole new way of thinking about development:

• Instead of a product roadmap we start thinking about a business model canvas: What metric gives us information on the business success of a new feature? How can we measure that? This can be as easy as click conversion rate. It's closely tied to the way you make money.
  
• Instead of a product vision we turn to measuring a product to market fit.
  
• Instead of a release plan we start thinking about the minimal viable product: What is the least you can do to generate more revenue?
  
• Instead of an on-site customer we start having a on-site developer. This is about getting developers out of the building, get them to talk to your users and get this experience back in. Note: To some extend this is even possible by looking over the user's shoulder by analysing your log files. Ever thought of taking a closer look at your search engines queries that return no results? Maybe instead of cursing the users that are too stupid to use your interface it's time to re-think your interface design or add more features to your search engine.
  
• Instead of iterations we have a loop of building - measuring - learning about our product.
  
• Instead of an iteration backlog at the end of each loop we have to decide about whether to preserve the current development direction or pivot it a bit.
  
• Instead of a backlog of things to do we have a list of things to learn about our users.
  
• Instead of detailed tiny user stories we have hypothesis of how our product is used that are validated or falsified by actual data.
  
• Instead of continuous integration you have continuous deployment that is much more interesting.
  
• Instead of acceptance tests we may have split tests, A/B tests or similar.
  
• Instead of a definition of done including only development it suddenly also comprises validating the data - did we actually convert more users?
  
• Instead of costumer feedback we are having a cohort-based metric, we start watching our users to learn - they are rarely willing to give explicit feedback, but their actions are very telling.
  
• Instead of a product owner we have entrepreneurs that have the business as a whole in mind.
  

Design a solution

Good industrial design is made up of ten principles: It's innovative, makes the product useful, understandable, aesthetic, unobtrusive, honest (does not fool users), long lasting, thorough through, environmentally friendly - and most important of all there is as few design in there as possible.

As one example Mary mentioned the flight booking site Hipmunk.com that gives users the chance to sort flights by least agony. It optimises for the best flying experience - not the best buying experience.

In the end it's all about building what the user needs - not selling what your build. It's about watching your users and learning about their needs.

Implementing design

One grave warning: Do not divorce design from development: Ford was a driver and racer himself, making him a great car designer. Wright flew the aircrafts he designed. Tom Edison thought through the whole process. Also in the tech industry the companies that are nowadays most successful were founded by people who are or at least were themselves very actively using the stuff they are building - think Bill Gates, Steve Wozniak and Steve Jobs, Sergey Brin and Larry Page. Takeaway is to do the job you are trying to automate, to have incremental development and delivery.

In the end it all boils down to how fast a learner you are: Not the most intelligent nor the strongest will survive. It's the one that is fastest to adapt to change that will be successful. One commonly used analogy for that is the so-called OODA cycle coming from John Boyd: The tighter your cycle of observing, orienting, deciding and acting is, the likelier you are to be successful among your competitors.

Speaking of release cycles Mary gave a very nice graph of various release cycle lengths.



In her experience teams that have deployment cycles of six months and above are likely to spend one third of their time hardening their software. If you are down to a quarter it's likely that the bottleneck is your business model. Maybe it is weird contracts or business partners expecting support for the exact version you shipped years ago. When you are down to a month per release you likely are running a software as a service model. As clarified later after a question from the audience for Mary this does include setups where customers basically buy a software including all future updates for a set period of time - including an automated roll-out. If you are down to daily or even just weekly releases you are likely running the product yourself. Mose likely then everyone is part of the team.

After her talk Mary answered various questions from the audience: According to her experience Scrum defines the team way too narrow by focusing on development only - with Scrum we are possibly getting perfect at doing the wrong thing very well - forgetting all about design and business.

One very general advise that leads to an interesting conclusion is to not waste your life, to not spend years of your life working on projects that are not needed by anyone. In the end this leads to realizing that failed projects are also caused by the developers not raising concerns and issues.

Mary is a huge proponent of trusting your team: Do not hand them detailed lists of user stories. You have hired great, senior people (or haven't you??) - give them an idea what you want and let them figure out the details by talking to the right people.

When asked for tools to support the lean startup process in a startup Mary emphasized the need for communication. Tools are likely to only slow you down: Get people to communicate, to sit at one table. Have a weekly or bi-weekly meeting to get everyone in sync. Otherwise let developers figure out their way to collaborate.

The talk was kindly hosted by Immobilien Scout, organised by agiliero with some tickets handed over to the Berlin Scrumtisch. Thanks guys.

The February Scrumtisch Berlin featured a talk by Lyssa Adkins, famously known for her publications on Coaching Agile teams. A mixture of fifty developers, scrum masters, coaches and product owner as well as one project manager followed Marion Eikmann's invitation. Thanks for organising the event, as well as thank you to Hypoport for providing the venue.

In her one-hour presentation she mainly focussed on two core topics: On the roles agile coaches have to fullfill as well as on the skill set needed by agile coaches. Being an agile coach (as well as being a scrum master, really), entails four very basic roles:

• Being a bulldozer - that is getting impediments out of the way. That can be as easy (or hard) as getting appropriate equipment for your developers or teaming up with the product owner to communicate with anyone trying to break the sprint.
  
• Being a servant leader - for a coach that means to work for the team, to ask the right questions and enable the team, to listen during dailies - instead of asking for additional reports: Most tools are already within the Scrum toolbox. The hard task is to identify them and use them correctly and efficiently.
  
• Being a shepherd - that may be as easy as getting people to attend the dailies, or as complex as communicating common values, a common goal.
  
• Being the guard of quality and performance - as a coach that means making degrading quality visible - and letting the Scrum team take the decision on how to deal with it.
  

However in reality each team is different, each sprint is different. So coaching really is similar to bing the river guide: Each trip is different. It is your task to make the team succeed and adapt to differing situations. To get to team to high performance - over and over again.

When talking about coaching what people need is a very specific skill set:

• To become a successful agile coach it helps to have coaching skills to be able to understand the client, to see their impediments and help the team become better by listening and asking powerful questions.
  
• Be a facilitator who organises sessions, meetings, conversations and may mediate in meetings.
  
• Have business domain expertise - that is to know process designs, figure out a companies strategy options.
  
• Be a great teacher to help people understand the basic concepts. That entails knowledge on course design, most likely design of exercises.
  
• Have mentoring skills to guide a team in the process.
  
• Know about lean and agile processes and stay up to date.
  
• Have the technical skills on what makes development successful, know the extreme programming techniques to help team excel.
  
• Have transformational skills - that is be familiar with change management, transformation leadership, knowing how to change organisations.
  

However being a coach remember that people are influenced not only by what you teach - but mostly by what you do. Most of what being a good coach means is invisible to the uninitiated outsider. It's about living the process you teach, staying true to the principles you want others to implement.

To get there it helps to get inspired, to talk with others in local meetups (just as with any coding practice: Try to find a common forum to exchange ideas and get fresh input). It may help to keep a value journal - not only to track your accomplishments and be able to prove them to higher management, but also to track where to improve yourself.

The talk provided several interesting starting points for exploring further. However with an added exercise sixty minutes covered only the very basic ideas. Especially for the skill sets needed to successfully coach teams it would be very interesting to learn more on what books to read or what courses to attend to learn more on each topic. Ultimately not only agile coaches benefit from being great teachers, mentors or facilitators.