Poor Foundations
What can be done early on to set your teams up for success? What kinds of things are worth investing time in and how much do you try to solve for not knowing exactly what is needed in the long run? Furthermore, how do you front load a project with all the foundational-type work (think continuous integration, frameworks, etc.) that will make the rest of the project go smoothly? Once you think you know what those things are, how do you convince management that this upfront cost will pay off in the long run? One thing is fairly certain: if you don’t invest in a solid foundation your Agile project will pay for it in the long run.
Real-Life Stories
Story 1: Building a Good Foundation
I was fortunate to be a technical lead of a team that was one of the first to start work on a multiyear, multi-million-dollar project. This team had an initial set of foundational items to complete that would pave the way for the rest of the development teams once they joined. Some of the things we started work on were caching frameworks, analytics frameworks, setting up Continuous Integration (CI) plans, and picking code-quality tools like Clover and PMD. There were some other teams working on similar items. Unfortunately, a decision was made to kick off the development for this project without completing many of the foundational items. Keep in mind that on a large project like this, once development starts it is very hard to stop development and work on foundational-type items. Furthermore, once teams started using frameworks that were incomplete, it was very hard to go back and switch to newer versions of those frameworks. As you can imagine, this created a lot of technical debt. In my experience, “this will get fixed later” usually means it will never get fixed.
Thoughts
In the beginning of a project there is a lot to think about. I think the items discussed next are worth at least thinking about. Of course, you’re going to get some things wrong. But doing some of these things can minimize the pain.
Put some serious thought into CI, if not Continuous Delivery (CD). There are many excellent resources on this topic, including Continuous Delivery by Jez Humble and David Farley (Pearson Education, 2011). This book explains the reasons for CI, some different approaches to automated testing, the various stages in a typical CI pipeline, and so on. Not all groups will need a full CI pipeline, but having even a basic CI pipeline setup for a codebase that has several developers (and you are trying to create software that is potentially shippable at the end of the Sprint) is vital.
As part of your CI strategy, it is important to set up the code quality gates for the project. This does a few things: makes sure all developers are following the same standards, helps prevent bugs, reduces technical debt, and ensures code is being adequately tested.
Put some thought into the processes your team uses. I do not believe this is in direct opposition to the Agile principle of “value people over process .” I am not talking about building in tons of process or anything that would build barriers to communication. Of course, you’re not going to get everything right; that is the purpose of Retrospectives and the chance to change things during every Sprint. But some things to think about are code review processes, architecture guidelines, and workflows (like those in Jira®) for moving a User Story through the normal User Story states. Another thing to think about is documenting the “Steps to Doneness,” which can help to make sure everyone is on the same page in terms of what “Done Done” means. Finally, it is good to have some process or agreement around writing automated functional testing.
Leaning on the knowledge of people who have experience using Agile can be very helpful—maybe an Agile Coach or someone on the team who takes the time to do the research and bring that knowledge back to the team. Learning from the mistakes of other teams/experts will go a long way to putting the team on the right track.
These processes are critical to have in place and not putting thought into them upfront will typically cost the project in the long run. Some people would say this concept is anti-Agile, but I don’t think that is the case. There is nothing in the Agile principles that say we don’t need to do design or set up foundational items. In fact, I think not doing so is one of the biggest mistakes that new Agile teams make.
So what are some of the ways we can tackle these items?
Set aside a “Sprint 0” to tackle foundational-type items or to set up your processes. This Sprint is usually a “free” Sprint where the team’s velocity will not be measured and the User Stories don’t necessarily have direct business value.
Scrum embraces a concept of Spikes, and Spikes can be used to tackle research-type items or some of these types of foundational items. Spikes are similar to the “risk mitigation ” in the Elaboration phase we used to use in RUP. Spikes should be used sparingly and are the exception, not the rule.
Dedicate teams to building out foundational items at the beginning of the project. On smaller projects, having a Sprint 0 might be sufficient. On larger projects, it might take a few Sprints.
In some companies you might have an architecture or advanced technologies group that can be engaged to help with some of these types of items. Use Sprint 0 or Spikes to talk with these teams and find out what they have that can be leveraged based on your platform needs.
Sometimes even the best of intentions can lead to very bad results. I saw an example of this on an Agile project where the architecture team put in place the quality gates for the client, which was on a LAMP stack. One of the mandates was that all PHP code had to have code coverage. That sounds great on the surface. But there were a few issues: (1) the web developers were not used to writing unit tests and (2) the code coverage number was a blanket number and did not take into account complexity, maintainability, or if the tests actually achieved anything. This led to tests that were hard to maintain and added very little value. Even worse, it led to perfectly good code being refactored just to allow for unit testing. Even with these side effects and the grumblings of the web developers, management did not want to change the code coverage floor.
It was not until toward the end of the project that team finally moved to using the CRAP (Change Risk Analysis and Prediction) metric in PHPUnit, which focused on testing complex code. By that point many brittle unit tests had been written and eventually deleted.
There had been so much wasted time and effort just because one group mandated 100% code coverage and then never followed up to see how this process was working in practice.
Thoughts
Putting quality standards in place is important, and in this case the intentions were good. Following are a few things that could have saved this project a lot of time and money:
One of the lessons learned from this project was to make a good case to management early on when you see red flags. Build a coalition of other developers who see the same issues. Form a proposal and show why things should be changed. If you can show that money will be saved and developers will be more productive, management should be willing to listen.
One way to address the lack of confidence in the application would have been to have adequate automated functional tests. Unfortunately, in this case, the automated functional tests were also lacking. I think that if an Agile team is doing behavior-driven development (BDD), there is an argument that you don’t need as many unit tests. If the automated functional tests are good, then you can have confidence that the application is behaving as expected. That is a better measure than pure code coverage (90% code coverage of an application that does not behave as expected does not mean much to the user).