Prediction Is Unpredictable

The predictive planning model is dangerous when we’re doing a standard consulting gig , like installing packaged software or building a data center. The practice of analogous estimating, in which we look at similar projects of like duration and set a price with some built-in buffer, neglects the fact that no projects are alike. As all experienced consultants learn, it’s not the technology that gets you, in most cases; it’s the humanity. Change resistance, difficult personalities, the unsaid and unknown make each engagement unique, no matter how “analogous” the specification may be. Engaging for a consulting gig is risky because the customer wants certainty and predictability in an uncertain and unpredictable domain. The client’s two big questions, how long and how much, are the exact questions we can’t answer in a predictive, big, upfront scope regime.

If estimating and planning a packaged software implementation is tricky, negotiating a consulting engagement that applies agile techniques is downright treacherous. We don’t know, before we engage, the state of the client’s culture, business model, or value chain, let alone the personalities we’ll be dealing with. The depth of their dysfunction is purposely disguised, in order to entice, rather than repel, the consultant. We don’t know, until we explore, the enterprise’s familiarity with agility. As difficult as it is for the client to articulate expectations for a packaged software implementation, guiding the client through an iterative engagement in which requirements will be emergent and evolving is much more demanding. As the hype cycle around agility has accelerated, many consulting firms claim the agile mantle, and many executives have formed their ideas about how agile works. Applying agility to the consulting relationship requires us to dispel myths and misconceptions, and employ all of our persuasive and educational skills to help our prospects understand how an agile consulting engagement would work, and why it would be to their benefit.

Consider a typical Big 5 consulting project from just a few years ago. The prospect sends out a number of requests for proposals (RFPs ) to their vendor list of consulting firms, or calls in a sales rep. Through a mix of business and technical language, the client tries to articulate the current state, the desired state, and the solution it expects the consulting firm to deliver. RFPs are rarely phrased in a way that enables the consulting firm to apply creativity or innovation; they’re more likely to state things like “the system shall have . . .” or “the end user will see . . .,” and the consulting firm becomes simply a fulfillment agent of a preconceived project, often with an preset due date. It’s not uncommon for the solution proposed to be unworkable, infeasible, or outright ridiculous. Even if prospects have actually devised an elegant solution, they rarely can communicate it precisely, thus leaving the consultant to make many assumptions, most of which will be wrong. The prospect, if we are lucky, will offer a “bidder’s conference ,” in which we get to ask a few questions along with all the other potential competitors. This often becomes, rather than an opportunity to uncover the devilish details, a jousting match between bidders, who strategically try to avoid giving away to their competitors either their solution ideas or their confusion.

After attending the bidder’s conference, most firms will huddle in a conference room and map out their reply. In responding to prospects, it’s common for consulting organizations to amplify or even distort their experience and expertise. They usually don’t feel they’re being dishonest; instead, those around the table tell each other, “If we get the work, we’ll figure it out.” So the guy who goes out for coffee becomes a “Java expert,” and the intern who managed the school carnival becomes a “Senior Project Manager.” Even without these distortions, the RFP response process is risky; many experienced consultants won’t even bother with them, heeding the old adage that if you have to respond to an RFP you’re already too late. We’re typically developing a solution to an imperfectly stated and poorly understood problem, or the wrong problem altogether. Once the solution is devised, the consultant’s proposal is delivered to the prospect with a nice glossy cover, and then the firm sits back and waits.

If the firm loses the work, it has invested a significant number of potentially billable hours on a failed bid. This may seem an inseparable part of the business, but I’ve seen firms that jump at every opportunity, literally proposing themselves out of business as they devote all their hours to long-shot bids that never come through. Like a bad-luck gambler, they go into every roll of the dice thinking “this could be the one!” Failing to qualify opportunities leads to bad behavior such as inflating credentials, pulling in “talent” off the street without proper vetting, and grossly mispricing deals due to lack of experience, or desperation.

If the firm wins the work, it typically finds, once it engages, that the consultant and the client have very different ideas of what was signed up for. In the big, upfront plan model, the consultant and the client begin the first phase, Requirements Definition, and, as they dig into the details of the broadly sketched scope outlined in the RFP, they find that each has visualized a completely different process and outcome than proposed, and that the boundaries of the project are far more elastic in the client’s mind than they are in the consultant’s. The consultant discovers first-hand all the ills of predictive planning that we’ve been discussing; the client has a vague and undefined vision of the outcome, and can’t articulate it in ways that we can decipher. Many stakeholders believe that they can add or subtract features based on their role, rank, and perspective, regardless of the scope defined in the proposal. All the vagaries of human and cultural personality manifest themselves and resist or block progress. Many deals that were celebrated as save-the-company coups turn out to be profitless death marches.

In the traditional consulting engagement , both the client and the consultant are focused on the so-called Iron Triangle , the constraints of scope, budget, and schedule. The client, in a bid to limit risk, typically focuses on constraining the schedule and cost by fixing the budget and timeline. Consultants, to protect their interests, concentrate on fixing the scope, to avoid the dreaded “scope creep” that plagues fixed-bid projects, thus constraining the customer to the features and functions they specified at the beginning of the engagement. Unfortunately, as we’ve learned from experience, neither of these strategies work. If consultants allow the client to fix the price and schedule, they often end up “eating” the overage, or disappointing on delivery date, when they discover the intricacies of delivering the client vision. If clients allow the consultant to fix the scope, anything that changes their requirements, such as shifting market conditions or new ideas about the project, enters an onerous change-control process that is designed to protect the upfront specification, rightly or wrongly. We set up an adversarial relationship from the start, as the client maneuvers to get the most for the least and the consultant jockeys to give the least for the most. The predictive, big, upfront plan consulting model is death on the relationship, hitching the client to a rigid specification that has no flexibility to bend with the circumstances, and dooming the consultant or firm to eternal hostilities, with the customer serially disappointed and the consultant marching to an arbitrary and impossible cadence.

Moving to an Agile Consulting Model

It’s easy to delineate the failings of the traditional consulting model . Failed projects, consulting industry consolidation, plus the spectacular market flameouts of the many Internet consulting shops that exploded during the market euphoria of the late 1990s and fell to Earth in the dot-com bust of 2000, illustrate that success in the consulting industry is hard to achieve and difficult to maintain. Even with its failings, however, those left standing, from the prestige firms like Bain and McKinsey to the hundreds of small IT consulting shops scattered around the country, have the opportunity to earn substantial fees and deliver high-impact projects. Like those industrial firms that have embraced lean and agile ideas to remain relevant, both the renowned global giants and the successful local players have revised their engagement models to leaner, more flexible, and more client-focused methods .

Search for “agile” on McKinsey’s¹ web site and you’ll find hundreds of thoughtful and valuable articles, case studies, and presentations that illustrate the firm’s deep experience , and deep thinking, on the agile transformation. Visit McKinsey’s Digital Labs ² site and you’ll see that the firm itself now engages using an agile model, touting its “app in a day” and “client capability” workshops that mention agility as a key component of their offerings. Open the May 2016 issue of Harvard Business Review (HBR) ,³ and you’ll see an article on “Embracing Agile.” Look further in HBR’s archives and you’ll find dozens of articles on agility, including discussions of agile strategy, agile marketing, agile adoption, and agile workforce management. The management theory cycle, from the major consulting firms to the academic management journals, is in full swing around agility. Still, all of this talk is about using agility within the enterprise. In terms of migrating the consulting firm itself to an agile model, available advice is scant.

For consultants and firms that see the value of engaging in a new way, the migration to agility is not that mysterious. I’ve seen many consulting firms that make an incremental evolution toward agility, even if starting from a waterfall model. The traditional model, delivered incrementally rather than as a “big bang,” with separate scope, schedule, and budget for each phase, is the first step toward agility for many consultants. In this evolution, consultants, rather than proposing a once-through, all-encompassing omnibus project, propose a detailed discovery process first, with its own scope, time box, and “cost box” (or budget), and with a clearly defined deliverable consisting of a findings document and presentation that reviews the unique needs, constraints, and circumstances found through our exploration. I sell this discovery phase as a valuable outcome of its own, helping the firm understand the implications in terms of possible risks, inherent constraints, and stakeholder expectations. While still structured as a traditional 4D engagement , we now have the opportunity to assess for ourselves the circumstances we’re walking into, and we enable the client to cap its investment, mitigate the risks of a new project, and determine whether the project is worth the investment , disruption, and risk. While we’re not in the business of discouraging clients from hiring us, we should be in the business of avoiding work that adds no value or drags us into dangerous waters.

The same theory applies to the other phases of a 4D engagement model . After completing the separately scoped discovery phase, we can propose a more informed design phase, present our solution ideas, and introduce our client to the concept of collaborative participation , and to its ability to make changes as its circumstances change. We can iterate through this phase until the client accepts a design, then scope and price the development and deployment phases in turn, each with an agreed timeline and budget. Each phase is entered with a more complete and realistic understanding of the situation, and each decision, for both the client and the consultant, is more informed and deliberate. We never get too far ahead of the planning horizon, and the client has the opportunity at every step to stop a low-value or problematic effort, thus avoiding the “project that wouldn’t die” scenario.

While not yet a true agile, iterative model, this phase-by-phase approach has many advantages for clients accustomed to the fixed-price, fixed-date, fixed-scope approach. They can still budget for these individual phases as projects, and so not dramatically upset their procurement and financial models . They begin to understand that they must participate iteratively to ensure they get what’s needed, and we start to break them of the habit of “over the wall’ specification documents. They see incremental value and have the chance to redirect the project if it strays from their expectations or requirements. We grant them visibility into our consulting process, so we can mutually determine whether we’re a good fit. We can progressively advise them toward a minimum viable product, and so start to guide them to lean project thinking, and wean them from the kitchen-sink style of specification. As in an agile environment, we’re evolving from a one-pass, big, upfront plan approach to a collaborative, iterative, flexible, and transparent relationship, without blowing up either their existing mental models or their enterprise procurement processes. I’m often presented with this challenge to agile consulting; “How can I sell the engagement without a project estimate?” With this approach, we can range-estimate each phase, stay within our planning horizon, and incorporate our learnings into each subsequent proposal. We’re constantly discovering and adapting as we go, thus giving us a better chance to estimate each phase from knowledge, and to propose a more suitable approach. This simple evolution from big-bang to distinctly proposed phases integrates agile thinking and benefits into a waterfall-style methodology, without radically changing the consulting engagement model.

Figure 13-1 is an example of a range estimate , by phase, for an actual merger project in which I engaged. Note that this is not a project about agility; rather, it’s a project in which I applied some basic agile ideas to executing a nontechnical project for a client with no agile experience. A quick glance reveals that this is an interim step between a big, upfront plan engagement and an iterative, incremental project in which the client commits to a piece at a time and has the right to change, adjust, or stop that project at any phase. Notice also that within each phase is a list of high-level features, each of which will have multiple tasks. This makes this sort of engagement amenable to an agile delivery approach, as we decompose features into epics and stories, and apply our agile practices to iterate toward an end result. While we are proposing to the client in the planning language the client understands, we’re also preparing ourselves to apply agile practices to the project, building a backlog and a series of sprints to incrementally deliver each phase.

Figure 13-1. An Agile consulting engagement proposal

Contracting for the Unknown

Let’s start this conversation by acknowledging that most consulting contracts, even if they explicitly agree to a fixed scope, budget, and schedule, are leaps into the unknown. We include the fixed elements for the convenience of the client, which wants to devote a specified budget to its project, have a guarantee of a complete deliverable, and know when its outcome will be ready. These are all reasonable goals, but their reasonability does not make them possible. One of our initial challenges as we migrate from these predictive, fixed contracting models to an agile approach is convincing the client that the fixed approach doesn’t work, and, furthermore, is not in the client’s interest. The Standish Group statistics on failed waterfall projects versus successful agile projects may be persuasive but they are not usually convincing. The corporate client has great pressure to stay with the waterfall, fixed consulting model . The strategic planning approach , the procurement standards, financial processes, and the risk-management impulses of most corporate clients tether them to a fixed model that is perceived as enabling a predictive budget and schedule, thus protecting the enterprise from unforeseen costs and delays, and a fixed scope, preventing the consultant from deviating from the enterprise’s stated requirements. Our job becomes the exposure of the risks of this model, and the education of the benefits to a consulting client of a collaborative, incremental, and change-friendly regime.

While citing the Standish statistics is an important part of the conversation, the real goal is to help clients grasp the benefit of embracing change as a competitive weapon in our turbulent environment, and to educate them on the increased level of choice, input, and guidance they can exert on the project as the result emerges. Clients will often insist on the fixed-contract model from one side of their mouth and then decry the inflexibility and change-resistance of their consulting partners from the other side. It’s the consultant’s responsibility to make the connection for them between the constraints of the fixed model they insist on and the ills they condemn. To migrate them to an agile engagement model, we need to connect in their minds their desire for a fixed scope with the change-avoidance regime that results. We need to remind them that the fixed budget and schedule that they insist upon rarely protects them from overages once they have sunk major investments into project development. Critically, we need to demonstrate to them that agility works, through an incremental exposure to the benefits of engaging collaboratively, with an exploratory mind-set and an eye to the real developments out in the world and the market.

It’s a fantasy to believe that an agile-naive client is going to contract for a “let’s see what happens” engagement with open budget and schedule. The consultant’s next challenge in migrating clients to an agile model is figuring out how to introduce them to the agile engagement without freaking them out. I often offer to give away, or refund when engaged, the initial discovery element of the engagement, especially with clients new to agile consulting, to illustrate that we can discover in short iterations, demonstrate our findings as we uncover them (with the caveat that further learnings may change our determinations), and add value immediately even in the discovery phase. Clients are often pleased to learn that, rather than waiting months for the outcome of our exploration, they can incrementally learn what we’re finding and then end up with a complete, contextual review based on our final determinations at the end of the effort.

Rather than contracting and delivering the discovery, and the subsequent phases, as fixed bids or ranges, we can now take the additional step toward agile by proposing to bid actual time expended for the discovery phase, with frequent presentations, similar to iterative demos, that not only show what we’ve discovered but also keep the client informed on the expenditures to date and the expected schedule, which evolves as we proceed and learn. The same approach is, of course, also applied to the design, planning, development, and deployment of the project outcome. Our evolution from a predictive to an agile consulting model is itself iterative and incremental. We begin by transitioning from a complete, fixed bid for the entire engagement, to a set of smaller ranged bids for each phase of the 4D model, and then incrementally move to decompose each phase into a series of iterative deliverables, with client collaboration and change-readiness built in. Eventually, of course, we hope that the client will completely adopt an agile model internally to correspond with the way we engage, and we dream that our example of agile engagement will awaken them to the benefits of agility. The rate of migration to this model, however, will be highly variable and subject, naturally, to the client’s culture, aspiration, and adaptability.

Summary

Helping our clients evolve from a predictive model to truly agile consulting engagements is an exercise in trust and relationship building. The separately phased approach, as noted, is not a true agile engagement. We can migrate even closer to an ideal agile relationship when the client is ready. This is a revolutionary transition for most corporate entities, as procurement, relationship, and financial norms are overturned. When the client requires a new software application, for example, we can apply an agile scrum process very similar to the one we utilize when we develop internal applications in an agile shop. We can collaborate with the customer to create a vision, a roadmap, and a set of features or epics to work from, and then decompose those epics into user stories that we can prioritize in a backlog. We can iterate through delivery as we’re accustomed to, demonstrating an incrementally advancing model of the client’s vision, modifying and reprioritizing as we go, and billing the client on a strictly hourly basis. This, of course, is the ultimate level of collaboration and trust, and requires, in my experience, a history of migration from predictive planning to staged phases, and then to an increasingly agile engagement based on proven success and escalating trust. The path from predictive planning to a true agile engagement model can take years, as we illustrate through results that the enterprise will get what it needs without being gouged on price or compromised on schedule.