Fake It Until You Make It
Because Agile has become the new “hot” trend over the last few years, it is tempting for an organization to just jump to saying “yes, we are doing Agile.” I’ve seen in several organizations where they rename a meeting a “morning Scrum,” and suddenly the organization is “doing Agile.” Sometimes it is just a misunderstanding of what Agile is all about. Other times it is more blatant. There are many things that are important for an organization to do as it adopts the Agile development methodology. Things like education and experience are important to the successful transition to Agile software development. Using buzzwords is not enough.
Real-Life Stories
I’ve been on several Scrum teams that, in reality, were using Water-Scrum-fall. What I mean is that on these teams, we were having the Scrum meetings, we were doing estimations, we were using BDD, and so on. However, when the demo meeting came around, we would show the Product Owner what was built and collect the points for those User Stories, but the reality is that what we had built was nowhere near shippable. We had “Steps to Doneness” that talked about quality standards, testing, and signoff by the Product Owner. But all these things overlooked a huge aspect: vertical integration. It was the elephant in the room on these teams and projects. Everyone knew that what was being demo-ed was not shippable, or even potentially shippable. This is because integration across multiple systems/services was not completed yet. Once we started integrating all the various layers in the application and across system boundaries, we easily had months of work left before the application could be shipped. This is why I would call it Water-Scrum-fall and not Scrum. I would argue that this should not be called Agile either, but management loved to call these successful Agile projects.
Thoughts
At the end of the Sprint, I think a Scrum team should have a shippable product. If not, then I don’t think your business is getting the most out of Agile. Delivering business value in an incremental way is one of the things I like most about Agile. The idea that development teams can deliver business value in small, incremental ways is one of the major benefits of using Agile.
Just completing a Sprint and meeting your point commitment is not really the point (no pun intended). Teams need to be honest about what is being demo-ed. If integration is not complete and some things are mocked, then team members clearly need to explain that to the organization. They must explain that even though the story was completed, it is not shippable and it will be shippable when story X is completed. As long as the business partners and the development teams are on the same page, there is no misrepresentation of what is being delivered.
There are plenty of articles on the value of CI and the role it plays in Agile, so I won’t go into detail here. But in a nutshell, the idea is that when you have a green build, what you are saying is that the product you just built is “potentially” shippable. The idea is to do integration early and often because doing the integration later is more painful. In most environments I’ve been in, the artifact built in the CI pipeline says we have something that “could” be deployed to Production. However, it usually does not go directly to Production without some sort of manual intervention. I’m only talking about CI here, and not CD. But in Story 1, a green CI build really meant nothing. Between inadequate functional testing and the fact that new features were not always fully integrated, the artifact built in the CI pipeline gave no real confidence that we could deploy it to Production.
So, the bottom line is to call a spade a spade. If you are practicing Water- Scrum-fall, that is fine, but don’t claim that you are practicing Scrum. Don’t claim to be delivering business value every two weeks when you are not building shippable software.
While on one e-commerce project, there was a mandate that everyone check into the mainline branch. This is because checking in code early and often into a mainline (or what some call trunk-based development ) is generally a good practice. The thinking is that by using one branch for all development it forces developers to integrate their code. This will raise issues/defects early on instead of postponing the integration until later. On this particular project the way we prevented new features from showing in Production was to use Feature Toggles. There is plenty of information on using Feature Toggles, but the gist of it is that you “hide” new features behind a toggle (it can be anything from an “if” statement or using some declarative mechanism like an XML file). You have the Feature Toggles “off” until you are ready to expose the new functionality and then toggle the feature “on.”
One of the main reasons this approach was chosen was that “this is how the big tech companies do it.” So if we do what they are doing, we must be Agile, right? Well, no.
Having used this approach on several projects where the mainline was a live application and we had several parallel projects going on, I can say it does not work. We had hundreds of toggles in the user interface and these toggles were never cleaned up. The other issue was that changes for projects were constantly breaking the application in Production. This was an e-commerce application, so these defects sometimes impacted revenue. We also had RESTful services, each in its own mainline branch. The problem was there was no toggle mechanism in these services. So code related to projects that might not go live for months was actually “live” code in Production.
Thoughts
In Story 2, the mainline development and use of toggles was somewhat of a knee-jerk reaction by the architecture team to the use of Feature Branches. I tend to agree with Martin Fowler’s comment, “Release toggles are a useful technique and a lot of teams use them. However they should be your last choice when you’re dealing with putting features into production.” On these specific Agile project teams we were using Feature Branches for every User Story. Our SCM (source configuration management ) was Perforce, so each Feature Branch was what is called a “hard” branch. As you can imagine, for a set of features taking months to complete, the amount of Feature Branches in Perforce grew significantly and the merging of these branches back into the main branch was painful.
Instead of using Feature Toggles or Feature Branches when you have parallel projects and sustainment all checking into the same branch, I recommend using Project Branches. You can think of these as Feature Branches, but at higher level. So instead of having a branch per long-running feature, you have a branch for long-running projects. Then, within each Project Branch you treat it as a mainline (check in early and often, run through CI on each commit, etc.).
But projects can run many months, and the whole time the main branch is having check-ins by the sustainment team. And other Project Branches are also moving along with changes. So how do you avoid merging at the end of the project when the pain of merging can be complete hell due to the amount of changes in each branch? One possible answer is to do merges down from the main branch into your Project Branch. Do this as part of the development iteration (i.e., the Sprint) so that the project branch never falls too far behind the main branch and merges should be less painful. This is a kind of hybrid approach of mainline development and Feature Branches, but it removes the need for toggles. That is not to say that toggles aren’t great and can be used for other purposes, but with the foregoing approach we don’t need hundreds of them littering the code base.
There is no single branching and merging strategy that will work for all teams. There are so many variables: SCM tool being used, team structure, how releases are scheduled, and so on. But the point is that checking code into a main branch and pushing it to Production when it is not meant to go live for months is a bad idea (with several caveats, of course) and the use of Feature Toggles is not always a practical solution to this problem. So get creative and look for ways to not push code to a Production environment until the code is actually meant to be live. This can be accomplished and from my experience it keeps your Production environment more stable.
While at one company I saw something I thought was humorous. Maybe it should not have been, but that was my initial reaction. Basically, I was on a Scrum team in an organization that had run several Agile projects with various degrees of success. The latest project by many accounts was a terrible failure in terms of practicing the values and principles of Agile. It was really more of a Water-Scrum-fall project. Several of the Scrum teams had abandoned many of the prescribed practices of Scrum and the idea of continuous improvement (in terms of learning how to build the right software, the right way) had ceased. It seemed like teams were all over the place in terms of what they were doing (some had Scrum meetings, others didn’t, some did automated tests, others didn’t, etc.).
So it was interesting when one day we received an e-mail offering Scrum training to “show how we do it at XYZ.” The e-mail said the class would be “led by a senior member from XYZ.” The irony was that when I asked most developers who had been in this organization for a while, they would tell me how things had really degraded in terms of Agile values and principles. I thought to myself, “I wonder who this senior expert is?” and I also thought, “This seems a little like the organization is lying to itself.”
Thoughts
Don’t get me wrong, I think offering training is great. Increasing employees’ knowledge so they can be better at their jobs is always a good thing. But management needs to be honest about what is actually being practiced day to day. As an organization, you want to think you are using Scrum, or Agile more generally, but maybe that is not the reality. This is actually one of my reasons for writing this book: in many organizations, this disconnect exists.
What if the training is a misrepresentation of Agile values and principles? What if those who attend have not been exposed to other Agile training or exposed to Agile in other companies? They might not know to question how things are being done. I am not saying there is a “right” or “wrong,” and that is actually one of the great things about Agile; the question “Are we doing Agile right?” is best answered by “Are you following the values?” and “Do what works for your team.” That would be fine, but in the organization in Story 3, things were not working.
One good thing about Scrum training, ideally by qualified professionals, is that when developers come back to their teams, organizations, and companies they should have a lot more questions about how things are being done and fresh ideas on what can be improved.
Story 4: Using the “Agile” Excuse
The issue with people in an organization misunderstanding what Agile software development means is that over time, everyone starts buying into the “we’re Agile” mentality when the reality could not be farther from the truth. For example, I saw in one organization how such a misunderstanding stops a group from improving. This organization proclaimed to be an “Agile” organization. Within this organization there was a Release Management team that was in charge of making sure all releases went to the Production environment smoothly. While in a meeting with the leader of this group I heard something that really showed me how much people can buy into the idea of being “Agile” when they are far from it. In this particular meeting we were discussing building out a new application and how it would be tested and ultimately released into Production. When we starting talking about how long code changes would take to get into Production, the leader of the Release Management team said “people need to understand that we are Agile and we release code every two weeks to Production.”
Thoughts
While it was true that we released code every two weeks, which happened to be the length of our Sprints, it had nothing to do with being Agile. The way a deployable artifact was created in this specific group could not have been further from how you would create an artifact using a CD pipeline. In this group the code was checked into multiple locations, one person controlled who could check in code, and ultimately an artifact was built and deployed to the Production environment. This artifact did not necessarily contain the latest code or reflect what was tested by developers in lower environments. For the most part, it was a manual process and error prone.
The fact that this organization released code every two weeks is not at all specific to Agile software development. Any company can have a two-week software release cycle. But because this group had a mind-set of “we’re Agile” and thought their process was fine, I never heard people question things like how the artifacts were being built, why it was a manual process, and why was there a lack of automated testing.
Because the group could simply say “we’re Agile” and move on, there was no serious effort to make things better. The danger of buying into the “we’re Agile” mentality occurs when Agile software development has been poorly adopted. If people already think they are “Agile” and doing things right, they won’t change. If you are in an organization where openness and transparency are not valued, then it makes it harder to face this issue head on. In Story 4, it was very clear that even management had bought into the “we’re Agile” mentality, which was unfortunate in the sense that things never improved.