Continuous Discussions – Metrics That Matter

On January 26th I participated in an online panel on the subject of Measuring DevOps, as part of Continuous

Discussions (#c9d9), a series of community panels about Agile, Continuous Delivery and DevOps. Watch

a recording of the panel:

Continuous Discussions is a community initiative by Electric Cloud, which powers Continuous Delivery at businesses like SpaceX, Cisco, GE and E*TRADE by automating their build, test and deployment processes.

I felt that the atmosphere in the webinar was nice and friendly. The other panelists were knowledgable and nice to talk with, that goes for the hosts too, who facilitated the discussion in a very professional way.

RapidShop Principles

It started with an experiment:

“If we can establish autonomous cross-functional, co-located teams that are aligned with direction, then we will be able to rapidly build products and services with high quality that customers want.”

Our goals were to improve flexibility for the business, reduce lead times and improve quality of the software delivered. to reduce lead time from an average of 2 months to an average of 1 week. We thought breaking down the silos between business and IT could help us in this pursuit and together we defined why we were doing this and what (in terms of measurable goals) we wanted to achieve.  4 months later we’d managed to

• reduce lead times from a best case of 2 months to an average of around 7 days
• release software continously, sometimes as much as 4-5 times a day (with zero downtime)
• break down silos between IT and business, creating a collaborative environment in a team consisting of both developers and business.

It has been a truly great experience, working closely together with some really awesome people. We started from scratch, both from a software and a process perspective, and gradually evolved our process. Our key principles  have been:

• Simplicity
• Emergence
• Experimentation
• Rapid feedback

Lets look at the principle of Simplicity first

Simplicity Will Improve Your Team Effectiveness

Maintaining focus can be hard when developing software. There are a bunch of things that has the ability to steal our attention during a day. Mail, meetings, other projects etc. Over the last months we have found that keeping things simple, and not waste time on stuff that we believe don’t give us any value. So we have the following principle:

“Don’t introduce things you don’t need”

From a technical perspective we always try to keep things as simple as possible. We therefore have:

• No branches
• No un-necessary libraries

 

We also try to delay architectural decisions, keeping options open as long as possible. But more on that in a later post. Lets get back to avoiding branches and libraries.

No branches

If you are doing continous delivery, having no branches is a key element. But sometimes this is easier said than done. At one point we were about to introduce a more complex branching strategy. The reason ? We had a couple of incidents where defects were introduced in production, we released code that weren’t ready for production. So we gathered around our desks and started to discuss a branching strategy. As the discussion progressed we started to get a bad feeling. Things were starting to get complex. The concern was raised, and since we all really prefer to keep things as simple as possible we decided to look deeper. Was the no branching strategy really the problem ? Looking at how the work flows it became evident that the problem was that we didn’t really follow our own process (our Kanban board had a separate testing phase, where all our features need to pass an acceptance test). So instead of complex branching strategy, we agreed that we needed to improve our own testing:.

No unnecessary libraries

From the beginning the team has really done it’s best to keep our architecture and code as simple as possible. Whenever someone wants to bring a library, one of us put the “sceptic hat” on and asks: “Do we really need this?”. As a consequence, our code is still quite simple with reasonable few external dependencies. This makes our software easier to maintain and we can move faster when writing new features. The same goes for architecture (more on that later).

But it’s not only from a technical perspective we have valued simplicity. From the very start we have tried to create as little process as necessary, letting the process grow as we learned. The focus for the team should be to write working software that solves a business needs. Everything else should help the team accomplishing that. If it doesn’t provide any value, we simply don’t do it. Thus, we have:

• No (un-necessary) meetings
• No emails
• No estimates
• No testers

More details about keeping your processes simple in later posts.

Rapid Shop’s Road to Continuous Delivery

In 4 months we have improved lead times from 2 months (at best) to an average of around 7 days. We have gone from a heavy delivery proces.s with deployments once every month and multiple handovers to a a lightweight process with no handovers. We now release almost every day, sometimes as much as 4-5 times a day. Big batches makes sense if you have high transaction costs. With the possibility to do zero downtime deployments and a transaction cost of a few minutes, batching stuff makes little sense. When a new feature is ready to be release, we ship it. We got into that position by taking small steps, solving one problem at at time. But we did so with some sense of urgency. We went as fast as we could without blowing stuff up. What we started with a wish to establish a better way of working, proposed as a hypothesis. We knew that we wanted to move in the direction of continuous delivery, other than that we started from scratch.

Hire Passionate People

“The secret to a successful team is this: look for the people who want to change the world”

– Mark Benioff, CEO of Salesforce

After several months of seeking alignment, we finally got funding for a 4 month experiment to test out a new way of working. We knew that chances of success would increase with skilled people in the team. But for this kind of mission we needed something more. We needed people who were not only skilled but also passionate about what they were doing. But most importantly, we needed people with a growth mindset (see Carol Dweck’s work to lean more about a Fixed VS Growth Mindset).

Towards A Learning Enterprise – Run Experiments

(This is the second post on our journey toward building a Leaner IT-organization, “Towards a Leaner Enterprise – The Beginning” is the first)

For the last 4 months I’ve been leading an experiment at my current client, a big Telecom company in Norway. The experiment is related to finding better ways of working. In a previous post, “Towards a Leaner Enterprise – The Beginning“, I have described what trigged the need for a change:  All the teams followed the same release process (one big common), releasing software simultaneously in big batches approximately every sixth week. The process had a touch of waterfall in it:

• some people were responsible for breaking business needs into user stories
• The stories were allocated to teams responsible for coding.
• The acceptance testing was then handed over to someone else
• And finally the software was deployed by Ops

Needless to say this created some problems for the organisation (long lead times, lot of errors in production systems etc.) We thought we could do better. The challenge is to make other people also see the same problems. We decided to propose an experiment to test out a new way of working

Defining a Hypothesis
The advantage with defining something as an experiment is that is more easily accepted by the organization, the word seems to make it less dangerous for people who would otherwise feel threatened by change. Thus, we avoided some organisational friction and reduced the risk of being “shot down”.  We proposed the following hypothesis:

“If we get a skilled, crossfunctional autonomous team working together with Digital channels (business), then we think Digital channels can deliver new features to the web shop faster and with better quality. The result will be added gross sales.”

But it’s no point delivering faster if we deliver the wrong things. So we used a substantial amount of time defining a direction that was aligned with the company’s direction. To be able to verify or dispute our hypothesis, we defined some targets that we wanted to reach:

• Reduce lead time from an average of 2 months to < 1 week
• Increase conversion rate by 10 % – leading to a gross add of 16,7 Mill NOK per year
• Increase gross add of 2700 customers per year

The length of the experiment was 4 months (February 1st – 31st of May) and the team were given some boundaries to operate within. The following picture shows how we defined what, why, measures and boundaries for the team (inspired by Stephen Bungay’s Art of Action)

How did we do

The cumulative flow diagram shows how we did in terms of delivering features to the users.

We reduced lead time from a best case of 2 months to an avaerage of around 7 days. Our historical data shows that 95 % of our stories can be delivered between 7.3 and 9.4 days. The conversion rate is up by almost 30 % as we speak. Top management has seen that colocated skilled developers that sit together with business people, deliver quality software with faster time to market than developers that follow a common and more command & control based delivery process. As a result of this, the organisation is establishing a new way of working, where autonomous cross-functional teams with a strong product focus are responsible for both building, shipping and run their own products and services. As Werner Vogels, Amazon CTO, once said:

You build it. You run it!

How we did it

That will be the topic of my next posts. As a teaser I can say that our method was based on emergence, simplicity and a goal to do continuous delivery. But more important than anything: Hire passionate and skilled people!

Towards a Leaner Enterprise

The beginning

How do you go about changing a system where the focus is command and control and the doing takes place a long way from where the thinking is done? This was the question I asked myself this autumn. Work was becoming increasingly boring each day.

Running a Pre-Mortem

As part of starting up a new product team we thought that it could be a good idea to start off by running a Pre-Mortem on the first day. I picked up the exercise from Dave Gray’s Gamestorming book. According to the description, running a Pre-Mortem is deceptively simple. That turned out to be true.

From IT to Product Mindset

Two weeks ago we started an experiment to test out how we can improve lead times of new features for one of our customer facing teams.

Background

The team is responsible for three customer facing channels and publish new content on a regular basis to the different channels. There are no developers on the team, so for more complex changes they are dependant on developers working off-site. The process for developing new features have many hand-overs. As a consequence, the following problems occur:

• Long lead times
• Poor quality

In 2015 this represents a problem when you need to react to changing customer needs. Currently, there is a thick wall between the business side and the IT-department, which I believe is the root cause of the many handovers.

My first post in a veery long time

It’s been quite a while since my last blog post. 2014 was a busy year, with both ups and downs, with a lot of learning atteched to it. Sometimes things don’t go quite as planned, that was certainly the case for a few things last year. Well, it’s a new year with lots of new possibilites. One of the things I will like to improve in my work is to become better at saying no. Being an optimistic guy, I realize that this can be hard.