Russ Lewis Compares Two Agile Adoption Projects at TfL
Gradual adoption or ‘all-in’? The decision on how to implement Agile in an organisation is a tough one. There are pros and cons to both and supporters for both approaches.
Jeff Sutherland and Ken Schwaber, the ‘fathers’ of Scrum (now of Scrum Inc and scrum.org respectively) are zealous in their ‘follow the recipe and it will work’ approach. Others take a more pragmatic approach, allowing the organisation to respond naturally to the stimulus of ‘agilisation’.
Case study 1: ‘Stick to the Recipe’ Agile Adoption
When I first “agilised” TfL there were just two analysts on the project. I was brought-in as the software development lead. Prior to my arrival, all software activities including design, analysis, documentation, delivery and operation was outsourced. It was a ‘clean slate’.
The analysts had deep knowledge of the rail network, passenger journeys and fares. I soon discovered that both were intellectual giants with unexpected talents.
Ross McKenna was our Product Owner. He had helped negotiate the agreement between TfL and the Train Operating Companies (TOCs) and understood the various stakeholders’ really well. His thoroughness and relentless attention to detail meant that when he reviewed our design diagrams and models he was able to find problems and more than once came up with brilliant solutions to the most challenging of issues.
Peter Svensson knew his way round the raw journey data and had written the database procedures that led to funding for the project. A shelf full of reference books gave little indication of the speed at which Peter could acquire advanced techniques. Within just a few weeks we were a fully Agile development team using a state-of-the-art development stack building a multi-tier enterprise application. Learning object-oriented programming, test-driven and domain-driven development was just the start; within 6 months Peter had mastered the mapping and network theory which enabled him to develop the algorithms for journey construction.
We had a Project Manager too. Jayne Davidson was familiar with TfL’s waterfall and outsourced approach to development and agreed to support our Agile approach in the Scrum Master role. She supported us sensitively and pragmatically, ensuring the project management office (PMO) was kept updated with our progress on their own terms and ensuring we were able to maximise working time in each sprint. Of the many projects Jayne managed, ours took the least ‘management’.
We had full support for what we did and the project’s success was obvious to everyone. Our product was launched just nine months later on Jan 2nd 2010 – the first time this department had ever designed and developed software for itself – with 99.99% accuracy. By 2014 it was allocating £75m revenue each week between TfL and the TOCs.
Case study 2: ‘Stealth Approach’ to Agile Adoption
Large organisations are political and without the success of the first Agile project at TfL, it is unlikely the subsequent project could have got very far. The next project, the central fares engine for TfL’s Contactless payment involved five teams and four years of work. We used Agile approaches where we could, but having more and diverse stakeholders meant that agilisation happened more slowly, even organically.
I was the link with the first Agile project, having done a large chunk of business modeling and analysis in the interim. The resulting analysis document proved to be highly valuable: leading to the creating of an IP store; regaining ownership of process documentation and providing the basis for the requirements of the upcoming fares engine.
Resistance to Agile
The Contactless payment project, then called the future ticketing project, had already been running for about a year and a team of Business Analysts had produced an abundance of textual documentation. I was asked to show them the Agile approach and introduce UML – a modelling notation used by analysts to diagram systems and interactions. Agile proved a step too far, but a diagramming tool was installed and I showed some of the analysts how to use it. Only one, Sunil Gokani was determined to stay with it, and he embedded diagrams into his documents, making them easier for developers to read.
Later-on, Sunil became the chief liaison between the business and the development team, contributing significantly to the project’s success. Although he reported to the manager of the (traditional waterfall) BA team, he was also a member of the Agile development team. Serving both traditional and Agile teams was undoubtedly uncomfortable at times for Sunil and it is to his credit that his work has helped define the role of Business Analyst in an Agile environment.
Agile / Waterfall Hybrid
In response to the rejection of Agile by the BAs, I assembled an hybrid Agile/UP/Waterfall methodology which my manager, had the wisdom to keep from ever seeing the light of day. Many similar hybrids have been attempted because Agile was considered too much of a stretch for the organisation. Although it seemed a good compromise to me at the time, I now see it as a destination that is inconsistent with Agile goals. Like following a diversion that leads you to a destination that’s not on any mainstream map.
Luckily, the managers at TfL recognised it was better to follow a regular, proven path towards agility.
Competing Visions of the Future
In 2009 I was hired to architect a solution for a central fares engine that could replace the journey calculation rules currently carried-out by the Oyster card readers. The design I produced comprised a 10 page document, used Service Oriented Architecture (SOA), and was capable of real-time fares calculation and payment.
At much the same time the main technical lead for the Oyster card project came out of retirement to help design the future ticketing project (FTP). Oyster had been a popular success for TfL and it was easy to see why people liked it so much. Meanwhile there was no compelling vision for FTP, it had many opponents and now there were three competing schemes, the third having been developed by MIT, who had been working previously on the problem with TfL.
The future ticketing project looked rather uncertain.
The Need for Leadership
What the detractors did not realise, was that senior management needed a higher level of confidence than they would had it been an outsourced project. I recently interviewed Shashi Verma, Director of Customer Experience and sponsor of the project. He told me that he would attend the various planning meetings to gauge the research and analysis that had been done. He wanted to be sure the team had revealed the true scope and complexities of the project. When I asked him if he used any specific measure of progress, he replied:
No amount of reporting will ever get you to the point where you can trust that the eventual outcome is going to be achieved.
If you can trust your team you can step back and let them get on with it.
The Unifying Vision
Once he was comfortable that he had teams and architectures that could deliver the Contactless payments vision, Shashi empowered his managers to begin construction of a proof of concept for the fares engine, based on the architecture I had designed. Under Sebastien Losq, we assembled a small team, set-up a whiteboard, drew-up a sprint plan and the future of the project immediately looked better.
Together with Sean Smith, my colleague on the project, we completed the prototype fares engine in just six sprints. We were joined by two Business Analysts, a Project Manager and an Architect, all from traditional waterfall backgrounds. Interestingly, apart from the PM, they found aspects of the Agile approach rather challenging. Although having been through the experience, three of them continued successfully to the next Agile project.
Meanwhile, Shashi was building on the success of the Oyster expansion that had just launched, and Agile was gaining confidence in the business. Communication beyond the team played a big part, especially at executive levels.
With a successful proof of concept, London Mayor Boris Johnson agreed to fund the £75m project to upgrade the entire TfL network. Sean stayed-on to lead the project, increasing the development resource to five Agile teams and earning the job title Agile Delivery Manager.
Contactless went live on bus and tram in 2012 processing 70,000 transactions a day and live on rail 2014. Savings are estimated at £20m a year.
Two Agile projects, two approaches to Agile adoption and two very different experiences.
The first was a team in a green-field environment, almost exactly living-up to the ideals described by the principles of the Agile Manifesto. The result was an obviously successful and happy project team that delivered within budget.
The second, in a situation perhaps more common in larger organisations, individual experience and a very real concern for failure combined with the uncertainty of a new approach. The result was equally successful, and a very powerful team emerged. However, it wasn’t always fun and a few egos got bruised along the way.
My conclusion is that Agile is something to aim for. If you can go there in one jump, that’s ideal, go for it. If not, I think it’s perfectly OK to be on the journey to becoming more agile. That journey has no time limit, and it’s a longer journey for some organisations than others. So you may as well sit-up, strap-in, and enjoy the ride.