Preface
I risked a lot professionally when, six years ago, I published a small book suggesting IT departments consider agile methods for building data warehouses. Like many radical ideas, this notion generated a strong storm of negative responses—at first. Within a couple of years the general tone of responses decidedly changed toward the positive. Looking back now, I am glad I persisted in popularizing this particular concept because agile data warehousing is an idea whose time has come. It has been extremely rewarding to have been part of its origins, so much so that with great expectations I am pleased to offer another, more extensive book detailing the practice.
As senior solutions architect and project manager in the field of data warehousing and business intelligence (DWBI), I speak or teach internationally at a half-dozen data warehousing conventions each year. I always ask who in the audience is building a data warehouse with an agile method. When I first started offering classes on agile DWBI at these events, perhaps one person out of a room of one or two hundred attendees would raise his hand. Now, six years later, sometimes half the room will answer yes. Moreover, folks approach me during class breaks to share their experiences with agile data warehousing. By and large, they are achieving wonderful results at some of the largest, best-known Fortune 500 companies in the world, representing industries as diverse as retail, aerospace, health care, insurance, manufacturing, and software.
One of my recent conversations with the director of business intelligence at a major transportation firm for which my company recently started consulting summarized nicely the results these organizations are experiencing: “I can’t tell you what a difference agile warehousing has made at our company in just six short months. Delivery speed is up, programmer productivity is up, business engagement is up, quality is up, staff accountability is up.” His DWBI department was benefitting deeply from the revolutionary aspects of the agile methods, including time-boxed iterations, embedded business partners, incremental deliveries, regular product demonstrations of truly shippable modules, integrated quality assurance, and frequent attention to process improvement.
The steady stream of positive testimonials I encounter has convinced me that the practice of agile data warehousing is off to a good start. But as strong as these case histories may be, agile methods are still not the pervasive approach for delivering business analytics. A project management style that yields such positive results deserves to be shared well beyond just the core of global companies. This book is powered by the desire to reach the next tier of companies, to introduce agile data warehousing with whoever might benefit among the Fortune 5000 and even the Fortune 50,000.
Answering the skeptics
Embedded within that goal is the desire to provide a counterpoint to those comments claiming that “agile warehousing cannot be done.” Over the past several years, I encountered this assessment in print, online, and in person, even at companies that would be wildly successful with the approach only a few months later. These skeptical individuals always had valid concerns that usually fell into one of two camps. One group thoroughly doubted the wisdom of working closely with the customer. “Customers never think carefully enough about what they want. They ask for too much at once. They change their mind at the drop of a hat.” With the customer embedded into the development process, programmers might deliver fast but the business partner will only have them chasing their tails, so why bother?
The other group doubted that any team working incrementally could deliver an enterprise-quality data warehouse. “DWBI requires solid data architecture. All data ambiguities must be resolved before you start coding, else you’ll have conflicting definitions and transformation rules across the corporation.” The business departments might get what they want if the developers rushed incremental enhancements into production, but the corporation will never have a single version of the truth, so why bother?
These objections may sound impossible to answer until one realizes that an agile data warehousing community exists today. We are delivering enterprise data warehouses, and incremental delivery plus close collaboration with business customers is integral to our success. With this book, I hope to illustrate the nature of these solutions so that those listening to the debate will hear both sides expressed well.
Intended audience
I anticipate that there will be three types of individuals listening to that debate, each hoping to hear something different. First, there will be DWBI professionals who are currently leading a development team, following traditional methods. If the statistics cited in the first chapter of this book have any validity, well over half of these team leads already feel that their developers are moving far too slow for their projects to succeed. These readers will be looking for a way to accelerate their programming in the next couple of days. This book provides a quick way to get started with agile methods and then a step-by-step path to enhance the method as each team matures with the practice.
The second group within the audience will probably be DWBI directors. They will be curious as to whether enterprise-level analytics can be implemented using the iterative and incremental approach that many of the agile methods champion. These readers might be willing to make some compromises upon the current methods of their departments, but they will not want to discard all the disciplined practices they have acquired and used successfully over the past decades of their careers. This book plans on providing a balanced approach that offers delivery speed while maintaining just enough process discipline to support corporate solutions.
Last, but not least, the audience for this book may well include project sponsors from the business side of a company. These individuals are undoubtedly afraid that the DWBI projects they have commissioned will be like many of the other projects they have pursued with the help of their IT departments or outside vendors. Development teams demand exhaustive requirements, disappear for months on end, and then return frequently with an application that is missing a large number of the features needed. These business readers are hoping to find a different way to work with IT, one that does not involve so much risk. This book offers a new collaboration model for business and IT in which projects start delivering early and provide frequent checkpoints throughout the development effort.
Parts and chapters of the book
The organization of the material presented in this book tries to answer first the most pressing questions listed earlier, questions regarding how to make developers doubly productive right away. The presentation then steadily pulls back the lens and folds in enterprise-level software engineering disciplines until it addresses the questions of all three of the parties listed previously. Part 1 is to make the reader quickly fluent in a practical, incremental delivery method. Its first chapter describes the problems with traditional methods that an agile approach is designed to address. Chapter 2 sketches a generic agile method called Scrum in order to provide teams with a baseline from which they can begin crafting their own agile solutions. Chapter 3 provides the mechanics of managing the development activity within an agile project room.
Part 2 of the book focuses upon an aspect of agile methods that new warehousing teams invariably find a tremendous challenge: organizing the project for both maximum customer satisfaction and fast incremental delivery. Chapter 4 presents how to author requirements in a uniquely agile way by employing “user stories.” Chapter 5 provides an agile approach to defining a DWBI project top down so that it has both the right boundaries and the right connections to enterprise architecture. It then offers an extensive example of writing user stories for a data warehousing project. For DWBI projects with significant data integration requirements, user stories typically prove too large for agile teams to deliver within a short development iteration. Chapter 6 solves this conflict using a slight variation on the user story format called the “developer story.” I have always felt that developer stories were the missing ingredient when others claimed agile methods do not work for DWBI. Chapter 7 provides another crucial element for agile data warehousing, namely estimating the level of effort for project work. The estimation techniques presented take place at three distinct levels: the work for the next iteration, the whole project before it is funded, and the remaining project once development is underway. This chapter also outlines techniques that I’ve drawn upon over the years to achieve early deliveries of the features business customers want most while simultaneously sequencing those deliverables so as to avoid senseless rework.
While Part 2 discusses how to define a DWBI project so that it is ready for agile development, Part 3 turns the perspective around to focus upon how to redefine an agile method for the demands of data warehousing. Chapter 8 outlines several adaptations one has to make to project room activities to successfully navigate the rigors of data integration requirements. Perhaps the most important of these adaptations is to “pipeline” the work to a modest degree so that teammates in each software engineering specialty receive the time needed to achieve high-quality work and enterprise objectives. Finally, Chapter 9 provides a step-by-step means of introducing the modified agile method to a new team and then discusses how to scale up successful implementations for managing a program of multiple projects. It concludes with a survey of further modifications data warehousing departments can make to Scrum, as offered by the lean software community. These modifications may well permit even faster deliveries with a lower method overhead for those projects that can meet the prerequisites.
As this book builds chapter by chapter to a complete solution for the agile DWBI project room, I will call out a few places where innovations outside the walls of the project room could be useful. In particular, there are aspects of requirements management, quality assurance, and data engineering that require resources from and planning by a company’s data warehousing department. These topics are addressed by a subsequent volume to this book that focuses upon getting agile data warehousing practices “right” for the enterprise.
Invitation to join the agile warehousing community
It is hoped that project team leads, DWBI directors, and project sponsors will all find substantial elements within this book that address the challenges they face with data warehousing projects in their workplace. I invite all who try the strategies and tactics presented here to contribute back a distillation of their experience by taking the agile data warehousing adoption survey, locatable though the companion site for this book. The results of this survey are presented regularly at data warehousing and information management conventions throughout the year. The collective insights the survey offers will be enriched by the input of all practitioners, no matter what the outcome of their projects. If a company performs well with the method offered in this book, as have so many other organizations, then the agile warehousing community will get a glimpse of the factors that reliably make projects succeed. If, however, a company’s first few agile warehousing efforts deliver mixed results, then the community still benefits as the survey identifies areas where further refinements to the method can be made. Either way, a more standard, baseline approach to fast delivery of business intelligence applications will emerge, increasing the probability of success for all who follow later.
Ralph Hughes
June 2012