Format of the developer story

As described in Chapter 2, the format for a user story is

Agile data integration teams should follow a similar template for developer stories in order to keep them consistent. A starting template for developer stories would be as shown in the following table. Teams with differing degrees and types of data integration work may well evolve the actual template they use.

The following list provides an example of a developer story from every level of data transform shown in Figure 6.1. These developer stories depict data flowing between the waypoints along the path laid out for product dimension data. Components from the template are set apart using braces. The user story for this sample developer story is abbreviated somewhat to save repetition.

“The {Stage Product Data} module will receive {the ability to detect and load only changed records} so our product owner can validate {that the warehouse is capturing the product records just added or changed in the operational system}, which further completes the {‘…decompose monthly revenue figures by customer and product…’ user story.}”

“The {Integrate Product Data} module will receive {the ability to include the western region’s product catalog for strategic products} so our product owner can validate the {company-consolidated view of all strategic products}, which further completes the {‘…decompose monthly revenue figures…’ user story.}”

“The {Dimensionalize Revenue Fact} module will receive {the ability link in products records} so our product owner can validate {revenue data by product in addition to customer}, which further completes the {‘…monthly revenue…’ user story.}”

“Our end-user application’s {Product Dimension Dashboard} components will receive {a new strategic/legacy product flag in its query definition screens} so our product owner can validate the {sales orders for the newly added strategic product catalogs}, which should complete our ‘…monthly revenue…’ user story.}”

Two aspects of this template make developer stories easier for the product owner to understand. First, each developer story links to a parent user story that the product owner himself authored for the developers. Second, he directly perceives some value for the business by reviewing the impact on data in a target data object. That value may only be one small step in a chain, but in each review data are meaningful as a component of a larger collection that the end users have requested.

Developers workshop within software engineering cycles

Because developer stories begin linking data structures to the data transformation processes that populate them, senior IT professionals will want to understand where developer stories fit within the enterprise architecture planning processes software engineers like to employ. Table 6.1 presents such a process, one borrowed heavily from the Zachman framework but with elements from the domain of data warehousing clearly added as subrows for each architectural viewpoint. Standard enterprise architectural planning requires that developers analyze and plan in a particular sequence: business modeling, next data architectures, and then functional application architectures. It also stipulates that conceptual thinking precedes logical modeling, which in turn should come before physical modeling. [Spewak 1993] This ordering safeguards that the data designs of applications will support all relevant business entities within the corporation accurately and consistently. This ordering also ensures the many applications that will comprise the organization’s processing capabilities across the decades will all support a single concept of company’s data, avoiding any conflicts between the representation of information that could make a single, enterprise-wide interpretation of data impossible.

Table 6.1 Developer Story Process Derivation Set in a Zachman Framework^a

^aAdapted for data warehousing from [Zachman 2011], versions 1992 and 1993.

^bLDM, logical data model; PDM, physical data model.

The predevelopment collaboration described among the product owner, project architect, data architect, and systems analyst follows the recommended ordering of development steps. During the initial project backlog interviews, the project architect accumulated a list of user stories and a business dimensional model. These artifacts both belong to the business architectural point of view and represent the functional and data portions of this viewpoint, as highlighted by the shading in Table 6.1. The fact that the business dimensional model and user stories were derived iteratively during the same set of interviews is immaterial to the framework and is, in fact, optimal from a pragmatic point of view in that it saves time and allows each artifact to double-check the findings recorded in the other.

With the initial project backlog interviews accomplished, the project architect involves the data architect to translate the user stories into developer stories. Reflecting upon the user stories and the application’s likely source data, the data architect drafts the project’s macro data architecture. He may also at the same time begin sketching a logical data model of the application. This work falls clearly in the logical modeling viewpoint of the enterprise planning framework and occurs after conceptual considerations, as preferred. The lead-off team also includes the systems analyst, who is asked to reflect upon whether business rules processing require that a developer stories be decomposed further. That analysis still takes place within the logical viewpoint, but represents a functional portion of the system model so that processing architecture follows data architecture as the framework requires. The fact that, during the developer story workshop, the data architect and the systems analyst may iterate between themselves—or even with the project architect and product owner—as they interpret developer stories is again immaterial to the framework and advantageous as far as the quality of the results matter.

Demonstrable

“Demonstrable” indicates that the deliverables for each developer story should result in something that can be demonstrated to the product owner during a sprint’s user demo. Therefore, each developer story can be accepted or rejected at the conclusion of the iteration.

The greatest challenge to delivering data integration services incrementally is that it can take two or more iterations to move all the necessary data elements from the operational source systems to the dashboard where they can be reviewed by the product owner. Using the DWBI reference data architecture to segment a user story is a great technique for making data transformation steps manageable for the developers. The danger with that approach, however, is that teams become largely occupied developing pieces for the invisible layers behind each end-user dashboard and neglect to review them with the product owner.

Waterfall methods incur tremendous risk when their teams stop communicating with the end-user community for months at a time while programmers construct the application. Agile warehousing teams want to avoid this practice and instead keep the users involved tightly with the project as the application takes shape. As they standardize, deduplicate, and merge operational data, adding history and derived columns, they need stakeholder feedback every few weeks to tell them where they have misunderstood the requirements or failed to apply a transformation rule properly. Without this regular feedback, they cannot hope to “fail fast and fix quickly.”

Luckily, the remedy requires only a small amount of effort. Every developer story must be defined so that it is demonstrable. Simply pushing data from one layer to the next is not enough. The team must also conclude every developer story with a display of results that the product owner can review. As shown in Figure 6.4, developers need to provide a simple dashboard for each layer of the reference architecture so that the product owner can examine the results of the developer story for himself. At the conclusion of each iteration, the developers should update these “product-owner portals” so that their business partner can review whatever new transforms have been added, no matter how small, and provide feedback.

Whereas on first thought it would seem that business users would have little interest in these intermediate steps data take, in practice the normally hidden data layers can prove very useful to the product owner. For example, by looking simply only at staged data, he can detect problems with source data quality the team had overlooked. He might run a simple reconciliation back to source to determine if the acquired data are complete and correct from a business perspective. He can validate the output of data cleansing transforms in the data quality layer of the DWBI reference architecture and help identify those areas of source data that have been too mangled by the data entry operators for a programmed routine to repair. A dashboard connected to staging and cleansing layers will allow him to try out different ways of joining the company’s data on business keys and discover business rules and other insights buried within corporate information that no one ever suspected was there. These product-owner portals to the intermediate data layers do not need to be elaborate, production-ready dashboards. Only the product owner and his chosen few from the business departments will evaluate the iteration results using these front ends. Often displaying only one or two table’s of data at a time will support the necessary evaluations. Building these simple front ends so the product owner can fully participate in validation during each sprint is a small price to pay for the user-based feedback they generate.

Independent

For developer stories, independent means teams need to define and sequence developer stories so that each can be coded when it arrives at the top of the project backlog. A story that fails this test cannot be worked on when scheduled because it depends on one or more other stories that have not been completed yet.

The independence of developer stories can be enhanced greatly by the way the data architect and systems analysts define them. In the sample data flow diagram presented in Figure 6.4, the team may find itself delayed in starting on the revenue trends universe because it depends on the booked revenue fact table, which involves some complex business rules. To make sure that the developer story is independent, the team leads could split it into two developer stories—a universe each for billed and booked revenue—allowing a good portion of the project to be developed before the allocation business rules were finished.

Layered

The “layered” criterion indicates that a single developer story shows development in only one layer of the DWBI reference data architecture. Focusing on one layer will also help considerably to make each story independent, especially if all its inputs and outputs connect to data objects located outside the layer in question. If developer stories scoped to either the integration or the presentation layer still do not feel “right sized,” the team can extend the layered concept and try restricting the story to a single rank of a tiered data model, as discussed in the next chapter.

Business valued

“Business valued” reminds the developers that they must define and communicate every developer story so that the product owner appreciates it as a significant step toward delivering the overall application. This criterion will probably be the one that requires the most effort and creativity from the technical members of the team. Many DWBI professionals believe this objective cannot be achieved. In their vision, business partners want to perform their analysis using fully dimensionalized data and could care less about what each dimension looks like individually. This interpretation is true only in a minority of projects and can be reduced even further if the team manages its rapport with the product owner properly.

Far more common are product owners who have struggled with one or more IT projects in the past, maybe even a data warehousing effort. They hated the fact that developers disappeared for months on end, only to return with an application that tragically failed to meet key business requirements. These project veterans comprehend all too well the tremendous risk a large data integration project represents, both in terms of expense and opportunity costs. Instead of being dismissive of the incremental steps involved in delivering data, the product owners will be thrilled at the chance to see and manipulate warehouse data at each step in the reference architecture, for such demonstrations assure them progress is being made and the risk they run by associating their name with the project is being reduced steadily.

True, the team may have to educate its product owner some regarding the reference architecture that defines the waypoint along which data added must reach as it moves through the warehouse. Most business partners these days have worked with data applications so they will understand readily the fact that data must be staged before it can be scrubbed and integrated before it can be arranged nicely for analysis. If the data warehousing terminology interferes with understanding, dimensions and facts can be reexpressed in business terms such as metrics and qualifiers to make them business meaningful.

Once the product owner is oriented properly, the team should keep the schematic data flow diagram for each user story on the project board. With such a graphic, the team can check off each element as demonstrated to the product owner so that progress toward the final objective—the dashboard—can be visualized and tracked easily.

Furthermore, the project architect can take on the role of finding business-relevant queries that will communicate to the product owner the progress represented by each intermediate delivery. As the product owner test drives a dashboard connected to one of the data layers of the warehouse, the project architect could provide the following commentary in order to underscore the business value of the developer stories being reviewed:

With a little creativity, the technical teammates will be able to fine-tune each developer story so that an interesting demonstration can be presented to the product owner. The small minority of stories that cannot be given a business-intelligible interpretation can be declared part of the 20% architectural reserve described in earlier chapters on generic agile project management.

Demonstration ideas are especially easy to come by if developers keep in mind that the product owner is being held accountable by company executives for the project’s enormous burn rate against budgeted funds. Like anyone in that situation, he will have a strong interest in seeing business-intelligible evidence that the team is making progress. It will not matter if that proof of progress comes through a hastily created BI portal connected to data on a system test server. As long as he can trace it to a waypoint on the high-level architecture diagram and perform some business-meaningful operation with it, he can make a compelling report to his management that the project is moving forward. We will explore how teams can even give their product owners an approximate percentage completion to report when we discuss project estimation techniques in the next chapter.

Estimable

This component of the test is borrowed directly from INVEST, where we stated that the story must describe work specifically enough for the team to be able to estimate the level of effort required to complete it. The next chapter discusses estimation at length.

Refinable

“Refinable” means that the wording of a developer story should be meaningful and well scoped, but the details are left to be filled in later. This component of the acronym is primarily a simple rewording of INVEST’s Not-Too-Specific criterion, but may actually be the more useful of the two notions. True, a team may waste valuable time by gathering too many specifics up front for a story, but, more importantly, the accuracy of one’s vision improves as the project progresses. Stating a story loosely at first and refining it close to the moment that its build-out begins positions teammates to discuss requirements and design at a time when their knowledge of both will be at their peak. Overall, product owner and developers will make the fewest mistakes if they organize the project with short statements of intent, planning to refine those statements into a fuller expression of requirements when they start transforming work upon a module.

One way to keep developer stories from descending into too much detail is to instruct teams to focus on only the “what” of a story, that is, what has to be accomplished, deferring the how for discussion within the development portion of the iteration in which the module will be built. For example, it will be enough to write a particular developer story stating that addresses will be cleansed, leaving out the details of how street and city names should be actually parsed into tokens and what lookup tables should provide the replacement strings need to standardize the components identified. The fact that the team includes a data architect and systems analyst who will be providing 80/20 specifications for the programmers when they begin to code a developer story ensures that the project will have a coherent data and processing architecture. Within that context, if a developer story provides just enough information that the team can determine if it fits comfortably within the time box of a sprint, all other details can be deferred until later.

Testable

This element is borrowed directly from the INVEST test, which stated a story should describe a unit of functionality that is small and distinct enough to be validated in a straightforward manner.

Luckily, developer stories are fairly close to the “units” that developers have in mind when they consider “unit testing” their code. This proximity makes validation of developer stories fairly easy for them to envision. We discuss testing for data integration projects in much greater detail in the enterprise volume of this two-book set.

Small

This criterion is also borrowed directly from the INVEST test, which suggests that every story should have such a limited scope that the work implied will consume only a fraction of the team resources available during a sprint. Fortunately, if the team applies the earlier criteria of layered, independent, and testable properly, “small” usually results. When this combination does not achieve that effect, there are secondary techniques for further decomposition, as discussed next.

Decomposition by rows

Breaking out candidate developer stories by identifying separately loadable sets of rows is possible when each source added to the warehouse results in an additional, independent set of records in the target tables. As long as each new source does not require updates to the rows already loaded, the team will find further substories packed within the candidate developer story that can be pursued in different iterations.

This technique is particularly useful when rows resulting from each new source derive from very different sources or transformation rules. By decomposing the candidate story into these distinctly transformed set of rows, the team allows itself to focus on just one complex transformation at a time, with probable gains in module testability and data quality as well.

Figure 6.6 depicts this decomposition technique using the example from our initial backlog discussion of the sales channel in the last chapter. During that conversation, stakeholders warned the team that each region of the company used very different business rules to determine which part of their organization was responsible for maintaining the relationship with a customer. The “load sales channel dimension” developer story suddenly appeared to the team as a module that would take many iterations to load.

Here the team drew upon the skills of the data architect. As part of the design process, data architects determine how dimensional data will be stored and how data will be joined to the records of the fact table. In this example, the data architect was able to assure the team that each region’s business rules will result in a new independent set of records in the sales channel dimension table. Furthermore, because there will be no snowflaking of the dimension involved, these records will link only to fact records sourced from the transaction systems of that region.

With those assurances, the system analysts could determine that each transaction source would simply layer another set of records into the sales channel dimension table without requiring modifications to previous load rows. That said, the “load sales channel” candidate story could then be subdivided into separate developer stories for loading the sales channel records from the unique source for each of the company’s sales regions. With the channels assigned according to the particular business rules of the region a sale belongs to, each of these developer stories dealt with only a single transformation pattern. Each developer story now appeared small enough that each of them would fit within a single iteration.

While performing this secondary decomposition, the team quickly revisited the other key points of DILBERT’S test besides small for each new developer story:

Secondary decomposition by row set works equally well for fact tables as it does for dimensions, as long as the load for each additional source results in an insert of new data without necessary updates to existing records. In our example, the data architect intends to design the fact table to hold revenue transactions. These are atomic, one-time events that pertain only to a particular region so that the record sets can be loaded independently. Therefore, should the “load revenue fact” developer story prove to be too large for a single iteration, it too can be decomposed into substories where each pertains to a specific data source. However, a story phrased as “load corporate sales channel order status fact table” would probably not be segmentable by regional row set because the fact records are now linking to dimension records that have been generalized to a level above region, and the status-oriented nature of the fact table implies that a previous fact record will have to be updated should a later transaction push it into another sales channel category.

Decomposition by column sets

The second of these techniques by which the data architect can further downsize developer stories is by splitting target data objects by the sets of column set to be loaded. This approach can be a bit trickier to think through, however. Load modules can be devastated by a change in the data structures. Because a large amount of code has to be updated, retested, and then promoted to production even if a single column name or its width changes, teams in general are very wary about changing the columns making up their target tables.

In some circumstances, however, the team can safely alter a table structure, allowing it to defer design decisions and constructively evolve a target schema as they work through a series of developer stories. In particular, as long as each new story requires that columns only be added, not changed, dropped, or renamed, then a series of developer stories based on distinct column sets can be pursued independently. Such stories will not entail reworking existing code, for any new column was not present when existing modules were developed, and therefore could not have been referenced by them.

A series of column-based developer stories can be envisioned using the example depicted in Figure 6.7. This example pertains to the product dimension table of the sample outlined in the previous chapter. Each new source system would provide product identifiers employing a unique set of native keys—keys that end users would want to reference in the dashboards. For this requirement, the data architect decided that each set of native keys would load into their own set of columns in the target table, exactly the design revealed in Figure 6.7. Over time the column sets will fill out moving left to right rather than top to bottom as occurred with the row sets of our previous technique.

There will be, of course, some shared columns pertaining to surrogate keys, a standardized identifier, and both warehouse and source meta data that will be populated no matter which data source is being processed. As long as the loading of shared columns remains consistent, no rework of existing code will be required. In fact, such consistency can be achieved easily by asking the developers to employ a reusable subprocess responsible for loading these shared columns.

As shown in Figure 6.7, the transforms populating the regional native keys can be decomposed into separate modules. Each of the modules will provide values for both its own specific target columns and the shared columns along the left edge of the table. The specific columns will hold the source system’s native keys for products, which will be convenient for the developers, as the source systems have wildly different formats for their product identifiers. The shared columns will receive the corporate product code that will allow rational consolidation of the separate product catalogs.

The decomposition by column set scheme allows developers to build separate load modules one at a time for each new source system rather than having to digest all the product identifier transformation rules in one iteration. The data architect can even defer specifying the attributes for each successive column set until the team is ready to build the transforms needed for a given source system, making even data design work incremental.

While decomposing the “load product dimension” story into developer stories for specific column sets, the team repeatedly applied DILBERT’S test to the new developer stories. Their analysis paralleled that of the row set approach given earlier, with exception of demonstrable. When working with a progressive set of rows, the team needed to provide only a changeable filter to allow the product owner to view the accomplishments of each story in turn. With column sets, however, the team will have to add new columns to the semantic layer of the product owner’s portal after every story so that the product owner can see data loaded into those new fields. Luckily, semantic layers are updated easily, especially when they provide single-table access as required here.

Decomposition by column type

The third means by which a data architect can further downsize developer stories is by splitting them across the different types of columns to be loaded. Large developer stories can also be decomposed along column sets that pertain to types of columns rather than individual sources. If the project architect can successfully educate the product owner about the differences between the column types included in most warehousing tables, then the team may well be able to define the developer stories to sequentially deliver data to different sets of column types as the iterations progress.

As portrayed by Figure 6.8, there is a natural division within warehouse tables between columns that receive simply replicated columns from source tables versus columns receiving derived values created by complex transformation rules. There are also columns for primary keys, foreign keys, and aggregates, as well as meta data from source or meta data generated by the warehouse itself.

Although a product owner instinctively wants all data made available immediately, project architects can share with them that there is a different business value provided by each type of column. Keys alone will enable counts to be calculated for the entities in the dimensional model. Counts are often the first business value stakeholders need to have when they begin working with newly loaded data. Textual replicated columns allow those counts to be filtered and combined selectively by values found in the source system. Numerical replicated columns can be summed and averaged. With such counts, sums, and simple filtering, the product owner can generate a first set of business insights to review. True, the capability is limited, but most product owners will agree that working with some information now is preferable to remaining blind to business operations until everything needed is in place.

By providing the column types needed for such simple analytics, developers will have bought themselves some time to work on delivering the derived columns that will need complex business rules coded. They can even consider dividing groups of business rules into distinct subwaves in order to make the remaining developer stories small enough to fit into sprints.

In the initial user story interview session described in the last chapter, finance had mentioned that postal addresses needed to be scrubbed before they would be reliable. In order to decompose the “deliver addresses” developer story, the team can consider segmenting the work between directly loadable columns versus columns needing scrubbing. In particular, postal address lines one and two (which typically hold street number, street name, and living unit identifier) will need standardization, but the city, state, and postal code attributes are well controlled by the source systems and can be loaded as is. Thus the work can be distributed across two iterations, the first delivering immediately replicable columns such as CISTAZIP, which will allow transactions to be analyzed down to the postal code level. In the second sprint, the team would finish up the scrubbed columns, allowing metric disaggregation down to the street and census block level.

In reaching small via this technique, the elements of DILBERT’S test largely track the pattern established in the previous two examples. For business valued in particular, however, the team needed to work a bit harder to explain for the product owner the new set of analyses made possible as each column type is populated in the target tables. Demonstrable required repeated updates to the product owner portal as new column sets became available. Separating transformation rules from other column types led to substories that make the developer stories independent. The tighter scopes of these new stories made the delivery increments more estimable and testable.

Decomposition by tables

The fourth technique by which the data architect can further downsize developer stories is by splitting the target tables they focus upon into special purpose objects. When faced with time-oriented entities such as a customer with so much history that queries began to run long, DWBI data architects realized long ago that they could separate these large tables into two versions: one table holding all history and a smaller one with only the last image for each native key. For many BI applications, because the vast majority of analyses focus on recent events or current status, these queries can be answered from current data only. Furthermore, queries reporting on trends are often based on only a few time points, such as the end of each month over the prior 3 years. Only rarely do the users truly mine the full life cycle of a topic and require information as of any arbitrary point in time. Given this natural division in usage, data architects can provide fast performance for the bulk of user queries by designating a “current history” table that holds data for only the most recent, frequently used time points. Such tables contain considerably less data to be processed by each query, allowing the system to return results far more quickly.

When a project involves a set of user stories requiring current data and others based on full history, the developer story for some of the larger dimensions can be split between “deliver current data” and “deliver full history.” This design pattern actually reflects what is commonly called a “type 4” dimension. [Kimball Group 2011] Developer stories for deliver current data typically require less complex logic than full history routines because old records for a native key can be simply replaced rather than marked inactive. Often the data volumes for current data sets are low enough that the developer story can require only a full refresh (“kill and fill”), making the load logic very straightforward.

Figure 6.9 depicts the decision to break a developer story for a large dimension into two. The data architect first adds a “current-only” version of the dimension entity to his data models. The team then defines a developer story for “deliver current customer data” and another for “deliver full history customer data.” Whereas the original developer story was considered so large it would take two sprints to complete, the new developer stories can each be delivered and demonstrated in a single iteration.

As far as DILBERT’S test is concerned, delivering the current table is clearly demonstrable as long as the project architect actively scopes the product owner’s review to questions involving latest customer segments. The business value will be certainly apparent, as the first developer story will provide data that address nearly all of the questions users have in mind. Because the fact records have separate foreign key columns linking them to the two versions of the segment dimension, those foreign keys can be loaded in separate processing runs, making the stories independent. As with the other segmentation techniques, the tighter scope of the new developer stories makes them both more estimable and testable.

Theoretical advantages of “small”

With the techniques considered earlier, we now have a deep toolkit for decomposing user stories as much as necessary to prepare them for iterative delivery. Our overall process is progressive by nature:

This process steadily reduces the scope or “size” of the request confronting the developers, yielding two important benefits. First, by reducing scope we steadily drive uncertainty out of each product owner request until the unknowns that remain can be managed successfully via the eye-to-eye collaboration that agile methods rely upon. By addressing detailed requirements and design through direct conversation, development teams can do without the large, up-front specifications that consume so much time and effort under waterfall methods.

The second benefit of steadily reducing the scope of a given developer story is that the size of the work units becomes increasingly homogeneous; this convergence allows the development process to proceed far more smoothly. Figure 6.10 depicts two projects, both being pursued iteratively, but taking very different approaches to the definition of their work packages. In both, the work bundles are trying to thread through a small aperture in a wall. This constraint represents each team’s finite capacity to turn requirements into working code during a single iteration.

In the first project, the developers have spent no time at all trying to break the work bundles down to a consistent size, especially one appropriate for the size of their aperture. As can be seen, such an approach is a prescription for delays and frustration when a particularly large work package plugs the opening for more than one iteration. These developers can expect to frequently appear at their user demo with nothing appreciable to show for the prior sprint. With sprints for a team of six coders costing somewhere between $75 K and $150 K per iteration (2010 dollars), such disappointments will increase the anxiety of the product owner and his sponsors quickly.

The second project, however, has taken a different approach. Its developers have taken the time and acquired the skills to decompose their work steadily until it sits in their backlog as a collection of more or less homogeneously sized work bundles. We will see in the next chapter how they can quantify module and work package size, but for now we can say that the developers acquire an intuitive sense of the complexities of their modules so that they can continue decomposing them until they all feel consistent and appropriate compared to their team capacity. As shown in the previous chapter, the team moves toward consistently sized user stories using the INVEST test and then to right-sized developer stories using DILBERT’S test.

The payoff is apparent in the bottom portion of Figure 6.10. By measuring their velocity each iteration, these developers have discovered their capacity to convert requirements into working code. They can think of their capacity as the aperture in the wall that is just right for a, say, 10-pound sack to fit through. Using the decomposition methods discussed in this chapter they have broken the work down into units of 10 pounds or less so that they can fill each sack with close to that exact amount of work. Once packaged in this way, they can move the sacks steadily through the aperture. By taking each request through the chain of epics, themes, and two kinds of stories, they know that there is no oversized request in any of the sacks that is going to clog up the gap in the wall. They will be able to steadily convert user needs into working code free of any complications due to work package size. Moreover, the team can optimize their work habits for the 10-pound sacks and acquire far more velocity than if they had to keep their work habits able to manage a wide range of work package sizes.

Readers should be careful not to interpret the story preparation described in the chapters of this part of the book as somehow returning to waterfall’s notion of “define everything down to the last detail” before a team begins system construction. Instead, the stack of project definition objects displayed in Figure 6.2 remains a single-sentence requirement identifier. The initial project backlog interviews and developer story workshops attempt to identify a whole project’s worth of work, but only for scoping, estimating, and scheduling purposes. The product owner and team expect the stories in the bottom half of the list to have considerably more precision than those at the top. Moreover, the data architect and systems analysts provide specifications only for developer stories that will be programmed in the next iterations, and those artifacts will be 80/20 specifications at that.

Of course, occasionally teams will underestimate a package. Agile accepts some surprises as the price to pay for faster project starts and higher delivery speeds. Agile’s iterative structure and sprint retrospectives give a team the chance to discuss the pain caused by such misestimation so that they can steadily improve their work-definition techniques. Yet, such misestimation will be the exceptional occurrence for agile warehousing teams that employ developer stories. By driving the work definition past the level of user stories, down to the developer’s notions of work units, these teams will uncover a good deal of the technical work hidden in a project’s user requirements. They will see this implicit technical work in enough detail that the risk of such nasty surprises becomes sufficiently small.

With developer stories and their decomposition techniques now illustrated, all that is left of our three-step project definition process is to estimate the size of the work packages so that a team can recombine them into the “10-pound sacks” described earlier. Techniques for such estimation and packaging are precisely the object of the next chapter.