CHAPTER 4
Why We Need This Now
We’ve outlined a better future. We’ve described why the status quo is arrayed against you. These dynamics have been true for a while. Why is now the time to act?
There are two main contributors to the timing:
- The status quo is getting exponentially worse.
- The alternative is finally gaining momentum.
The status quo is getting exponentially worse
As we saw in Software Wasteland, the current status quo regarding enterprise software implementation is a disaster. Typical project timelines and costs are routinely 10 times what modest best practices would suggest13. In the previous volume, we suggested some measures that would curb the worst of these excesses. This volume picks up where the other left off. Curbing the worst excesses is a step in the right direction, but what we want to do is take what we have learned and take it to the next level.
This book is about what next-generation enterprise software is going to look like. We explore it from first principles as well as some of the pioneers who have already established a beachhead in this brave new world.
First, let’s do a recap of what is really causing the problem so that we can juxtapose it with a new way of building systems.
Code creates maintenance
One thing we know with a great deal of certainty is the more code you have, the larger your maintenance burden. Some of this is the “latent defect rate.” We know that all software has latent bugs. These are flaws that haven’t yet manifested themselves because the right combination of circumstances has not happened. But they are there, lurking. Very sophisticated quality software development and testing can reduce the extent of latent defects, but nothing eliminates them.
In Code Complete, 14 Steve McConnell reports that the industry average defect rate is between 15 and 50 per KLOC (thousands of lines of source code). These “latent defects” manifest themselves when you least desire them. You have no choice but to staff up to handle this. Your 10 million lines of code system, therefore, has at least 150,000 latent defects.
Some “cleanroom” development environments have managed to get the latent defect rate of code as low as 3 per KLOC (a great reduction from 15-50), but this takes Herculean effort. It is far easier to reduce the defect rate by reducing the number of lines of code you are dealing with. When you drop the amount of code by half, the latent defect rate drops by half. It’s pretty easy to drop the amount of code in half and to do it over and over again. Indeed, we have done some projections that suggest that it is not unreasonable to suggest that the number of lines of code a large enterprise needs to support could drop by 100 to 1000-fold.
Further, even without the out-and-out failures that the latent defects usually germinate into, is the impact of change in the environment. Every change to regulation or business strategy, even changing computer operating systems, puts a strain on the code base. The larger the code base, the more likely there are hidden dependencies and the larger the strain.
A related issue is: whose problem is all this excessive code bloat? Certainly, the code that a firm has written or had written for it is their problem. Even any acquired software that has been modified.
What isn’t as obvious is that code you don’t even have direct access to has complexity roughly proportional to the size of the code base, which you eventually end up shouldering. If you use a DBMS (Database Management System) or ERP (Enterprise Resource Planning) that consists of hundreds of millions of lines of code, there is a very good chance this complexity is keeping you locked in, and that any slight change or desire to migrate will be hindered by the complexity of the code base.
Given that many large firms are dealing with billions of lines of code, they are teeming with diseconomy. Each new project makes the situation worse.
Perversely, many implementers brag about the number of lines of code they are bringing to bear or are creating for the solution, as if this were a good thing. The much-maligned Healthcare.gov implementation project at one point boasted of the 500 million lines of code that were involved in its implementation. It is as if a manufacturing firm were bragging about the amount of their toxic waste, as if this were a measure of productivity.
The main takeaway: application code is the problem. Strive to reduce it as much as you can.
Complexity creates high priests
Complexity, as we will use the term here, refers to the number and difference of interacting parts, and the variety of their interactions. In software systems complexity mostly manifests as lines of code, and most complex application systems became that way through complex database schemas.
My observation is that most people have a comfort zone around complexity. As they begin solving a problem, they bring in more data, more considerations, more factors, more structure, more tables, and more data in general. The solution gets a bit more involved and more nuanced.
New information is grafted onto initial structures. Often this creates additional complexity. Seasoned developers recognize they are doing this and set aside periodic development tasks to “refactor” and restructure their solution, based on new information that would have changed how they organize things had it come in a different sequence.
Whether the problem solver periodically refactors their solution or not, the solution gradually becomes more complex. Because this happens gradually, the solution architect has time to internalize and consolidate the complexity in their mind.
Everyone else has not had the same amount of time to absorb this complexity. This creates a monopoly of understanding in the mind of the creator, or often someone who has devoted a great deal of time to the solution. These experts, while helpful in the short run if you’re trying to interface with the solution, are actually part of the problem.
By running all requests through these “high priests,” we further entrench silos and complexity.
Application-centricity creates silos
Executives rail against “silos,” yet not a great deal of time is spent on understanding where silos come from, or how to prevent them. As a result, it is easy to rail against them even as you are re-implementing them.
We have silos because of our application-centric mindset. Every time we say, “Let’s not reinvent the wheel,” or, “Buy before build,” we commit ourselves to instantiating another application database. It will inevitably have copies of some other datasets (if it is an ordering system, it will no doubt have customer and product data), but this borrowed data has been converted, restructured, and renamed to conform to the application being implemented.
When we are finished, and when we have finally converted some of our old data to the new format and added new data, we will have successfully created yet another silo of data.
Very few projects recognize the full scope of the integration debt that implementing a new application system involves.
Silos create the need for integration
Between 40-70% of most IT budgets are spent on integration of some fashion. Logically speaking none of this is necessary. “Systems Integration” seems like a value-added endeavor, but it is only because of what made it necessary.
Imagine a world where cars were built for $20,000, but it took $40,000 worth of rework to make the car drivable. If non-drivable $20,000 cars are all there is, then the “car rework” industry is a high value-adding industry. Once someone figures out that the cars could have been made correctly in the first place, it becomes obvious that the rework was only value-added because of the faulty build process. Rework, as the quality movement in manufacturing has so succinctly pointed out, is waste.
This is where we stand with enterprise systems. Systems Integration is rework. Information silos create the need for integration.
Whether built in-house or acquired, the end result of decades of software implementation is a landscape of silos, expensive to attempt to integrate after the fact.
Legacy creates entrenchment
Everyone laments their “legacy” systems. For example, what is it about a payroll system implemented 20 years ago for $2 million that would make it cost $20 or $40 million to replace now (and at great risk)?
Keep in mind your legacy system was built with what we could now consider the equivalent of stone tools. We worked with a major Forest Products company that had a payroll system that had been written in Assembler Language on punched cards. Many of our State Agency clients have legacy systems written in languages like COBOL and Natural, which are not known for their developer productivity.
So why was it so much cheaper to build the original system than its replacement? Some people say that it is because the requirements are so much more complex now. I have built and implemented several payroll systems in my career, and I continue to be apprised of developments in the area by virtue of being, among other things, the chief payroll officer for our consulting firm. In my opinion, payroll has become about 10% more complex than it was 40 years ago. We have always had to calculate gross pay based on a set of rules that included which hours had been worked and which of them had to be considered overtime (and therefore be eligible for time and a half or double time). Payroll systems have always had to be aware of holidays and had to accrue and charge vacation hours. Payroll systems have always had to deal with a wide variety of pre-tax and post-tax deductions. We have always had to garnish wages for various reasons. Deductions have always been based on percentages and fixed amounts and often had annual caps. We have always had tax codes that varied based on marital status and deductions, and every state has always had their own tax tables, forms, and reporting periods.
As near as I can tell, a payroll system these days has a few more types of deductions and the states and federal government now allow online automated payment, which is a big plus. We often now add on some functionality for benefits management or skills management, but the core functionality of a payroll system is scarcely greater than that which was implemented in punched cards at a Forest Products company many decades ago.
One of the reasons that a payroll system implemented decades ago was so much easier and cheaper, despite the lack of good tools, was when the original system was built, there was almost nothing for it to be connected to, and nothing that depended on it.
Once the system is in place, other systems begin to rely on it, in subtle and overt ways. A downstream system builds a feed, and someone else takes extracts for their system; others use the identifiers from the first system and become dependent on them.
In a mature environment, replacing a legacy system is akin to performing a heart transplant: getting the old heart out and the new heart in is relatively straightforward, hooking up all the arteries, veins, getting nerves reconnected and keeping the patient alive throughout the process is where the work is.
Inflexibility creates shadow IT
“Shadow IT,” “Satellite systems,” “Rogue systems,” “Dark IT,” and even many “End User Computing” solutions refer to the smaller departmental solutions that grow up around your enterprise systems.
These systems further exacerbate the silo and legacy problems described above. Typically, each has a feed from the larger system (often an undocumented feed), which is one of the key contributors to keeping legacy systems in place—nobody knows how many of these systems are being fed from a corporate system and what would be impacted if it were replaced (and if the API changed).
The typical shadow IT system is built in Microsoft Access or Excel. It is remarkable how many mission-critical systems have one or more Excel spreadsheets in their data provisioning pipelines. There are several instances of firms using detailed Sarbanes Oxley Analyses to prove the provenance of the numbers in their financial statements. Many of these firms, to their surprise, found that their data had been processed through a series of spreadsheets.
What should cause alarm is the fact that 85% to 90% of spreadsheets have serious flaws. The error rate is 1% to 5% per cell.15 16 17
Why do people build and maintain shadow IT systems? The primary reason is the inflexibility of the corporate system. Gartner estimates that 35% of information system spending goes to shadow IT.18 In our observation, four of the most common reasons people build shadow IT systems are:
- To categorize or classify information in some way that is not supported by the corporate system.
- To add additional data attributes to a data record in a corporate system.
- To combine data from more than one source.
- To perform analytics – often relatively simple calculations appropriate for a spreadsheet.
All four of these are symptoms of a deeper problem: the corporate systems are too hard to change and are monolithic. If the corporate system were easy to change, these categories and additional data and analytics could have just been applied in place. Consider the waste in copying a set of corporate data to a local data store, adding a category, and performing some simple analysis. As a one-off, this isn’t a punishing bit of overhead, but once this becomes a regular job, the API must be supported, the download job must be scheduled, the satellite system must be maintained, and the results typically aren’t available to the source system because they have no place to store them.
Another perverse side effect of this state of affairs is the number and variety of these satellite system ossifies the legacy system. The stewards of the legacy system don’t know how many feeds are dependent on their data structures, and even the smallest change could have dire, unpredictable downstream consequences. Legacy maintenance grinds to a standstill.
The related problem is that corporate IT systems are just that: corporate. It is the same system for everyone. There is typically no mechanism for departments to share a system and yet have their own customization. In this book, we will describe how a Data-Centric view promotes this and allows us to have variety in the context of standardization.
Mega projects create mega failures
Large information systems continue to have embarrassingly high failure rates. The 2014 Standish Chaos report reported a rate of 41% failure and 55% “challenged.”19 To save a bit of math, that leaves only 4% “successful.”
While the overall success rate has been at this level for some time, there is more bad news in the Standish data: the bigger the project, the higher the failure rate. As our world and its data landscape grow increasingly complex, big projects are becoming the norm.
There is a silver lining to the observation that most large IT projects fail: Failed projects don’t make your overall information architecture worse than it was. However, most “successful” projects do. In the world of application-centric, every new successful project introduces another mostly redundant data model and dataset, and thereby increases the cost of systems integration.
As we will see in the next section, the key driver to enterprise information system cost is complexity.
Where application complexity comes from
Application software is complicated. It is among the most complex artifact ever produced by humans. A great deal of the complexity comes from the fact that every single line of code is different, and any line of code, when executing in a running application, may have adverse outcomes on other software. Even a simple application is complex, and most enterprise applications are not simple. Many have millions of lines of code. Changes to this code are fraught with risk.
The deeper question is, “where did the need for millions of lines of application code come from?”
Function Point Analysis is a tool for estimating the size of an application software project. The Function Point Users Group20 have been refining this approach for over three decades. The approach relies on counting the number of components (inputs, outputs, and the like) and the complexity of each. The chart on the facing page from Software Metrics21 summarizes the key part of the estimating methodology.
These factors are used to estimate how complex a software project will be. If you drill down, each of the factors eventually bottoms out into schema complexity. The number of “External Inputs” is the number of classes or tables or transactions that are being inputted, which is one type of schema. The evaluation of low, average, or high complexity is based on the number of elements per each input (and is therefore also driven by the complexity of the schema). Each of the other five components is equally 100% driven by the schema complexity. Adjustment factors, such as the complexity of the organization or the technical environment, are applied after the total number of points has been established.
It follows if you double the complexity of your schema, the number of function points will double, and the complexity and effort of your entire application will double. (The adjustment factor is a constant, so whether you are adding 25% or 125%, the end result is the project doubles in size).
What isn’t obvious from this methodology is how easy it is to double the size of the schema on an application project, and therefore double the size and complexity of the resulting system. Indeed, the mega projects are just several doublings of complexity, most of which are unnecessary. To get an idea of how easy it is to let complexity run rampant, what follows are a few case studies in complexity.
A case example in complexity
We worked with a firm that had hundreds of “products” (their product was data and so they referred to datasets or Application Programming Interfaces (APIs) as “products”). Most of these products had thousands of attributes. Occasionally they would try to migrate a customer from one of their products to another. This invariably led to trying to find someone who was an expert in the “from” product system and the “to” product system. It was very rare that they could find someone who was an expert in two systems. And anyone who was, was in such high demand it was hard to get his or her time.
Most product migrations were very difficult negotiations between two experts, who spent a great deal of time working out the similarities and differences of their data representations.
Ultimately, this firm produced an inventory of all the attributes they had in all their systems. There were 150,000 items in their inventory. This sheds a bit of light on how unnecessarily complex things can become.
Our core model for this firm had about 500 concepts in total. The stretch was that many of the distinctions in the 150,000 were not different attributes (60,000 of the 150,000 were address-related attributes) but were contextual. In a rich model, the context distinctions can be modeled in an economy of concepts. But mapping 150,000 attributes directly to their primitive representation would be time-consuming. What we did instead was create a faceted model that provided a way to group and distinguish the distinctions they had made in the 150,000 attributes and in the attributes in all their various products.
Implementing a set of systems with a collective schema of 150,000 attributes, when 500 would suffice, is not making the problem twice as complex as it needs to be, as I was suggesting above. It is making it 300 times as complex. To put it another way, they doubled the complexity 8 times. This is recounted to get a sense for the extent of the complexity that exists in many corporate environments and the opportunity for improvement, and to remind us that in the absence of determined effort to turn back complexity, it continues to creep.
Separation and isolation
Part of the problem stems from the separation that has been encouraged by the way we build systems. As systems become more complex, there is a tendency to splinter them off into smaller, more manageable pieces. Once separated, they become isolated. This makes it easier to work on because you don’t have to consider the impact on the rest of the firm, and therefore, each piece tends to develop on its own.
Designers and developers believe that the cost of coordinating and conforming are too high and prefer instead to isolate themselves and solve their own problems.
This natural tendency is one of the driving factors that lead to silo creation. This starts a vicious cycle. As the systems become more separated, the first thing they do is implement their own instance of a database. Very often, especially if the new application is a package, it will be not only a separate instance of a database, it will be a separate DBMS (Oracle versus Microsoft SQL versus IBM’s DB2). Once the separation is in place, the models begin evolving on their own. There is nothing drawing them back together and everything allowing them to drift further.
Agile development has a concept they call “technical debt.” It is the recognition that many short-term fixes erode the overall structure of the code built up to that point. They recognize if the quick fixes aren’t addressed, the system will gradually take on the characteristics of a legacy system (expensive to make simple changes). They set aside periodic sprints to “refactor” their code, which essentially means restructure the code as if you knew at the beginning what you know now. There are many patterns that emerge after the fact that create easier-to-modify systems.
The analogy for systems of systems is “integration debt.” Each point-to-point interface that you build adds to integration debt. When developers (or their managers) succumb to the attraction of isolation, they unknowingly introduce “integration debt.” Like all debt, it must eventually be repaid.
Humans in the loop
The way we build and integrate systems requires a great deal of “human-in-the-loop” processing. Even within a single application, the act of “joining” data between two tables requires a human, with knowledge of the metadata, to write the queries that will “join” the tables.
The problem becomes far greater when there is more than one systems involved. Humans are in the loop to extract data from one system and post it to another, or to access the APIs of one system and then those of another.
The “human in the loop” factor doesn’t drop to zero immediately when you begin your Data-Centric journey. As long as we use legacy systems, we will need humans to map them and more. However, we will gradually wean ourselves off a great deal of the non-value-adding portions of that work.
The negative network effect
There is a vicious cycle in the middle of all of this. As firms invest more and more in more complex systems, the cost to introduce a new system raises the cost even more, and the integration costs spiral ever upward.
The rising integration costs cause developers to try to isolate themselves, which leads to further development of silos. The need to get something done fast leads to short cuts, which often delay taking on the kind of change that will improve things.
Complexity math and the way out of the quagmire
There are three factors that generate the complexity in a typical enterprise. They are multiplicative; any one of them doubles the complexity of the whole. Doubling two of them quadruples the complexity of the whole and doubling all three amplifies it by eight times.
The three factors are:
- How many data models you have (which is essentially how many applications you have, as with very few exceptions, each of your applications has its own data model).
- How many concepts there are in each data model (this is the number of tables + columns or classes + attributes) that application programmers must program to (there are often concepts that are managed by the architecture that the application programmers needn’t be affected by).
- The average number of lines of code per concept (how many lines of code, on average, are added to a program when the schema is increased).
What follows is a very conservative example for a typical billion $ company:
1 Billion is a US billion (9 zeros and not a British Billion with 12 zeros). Each of these numbers is very conservative. Most of our clients have 3,000-15,000 applications. Few applications have under 1,000 concepts, and many purchased applications have 10,000 plus. QuickBooks, to use a well-documented example of a presumably well-written application, has 1,300 lines of code per concept.
I think a reasonable goal should be to drop each of these by a factor of ten. Strive to have 100 applications, each of which has, on average, 100 concepts they code to and use model-driven development to drop the LOC average to way under 100 per concept (for many use cases there will be zero lines of code, but the occasional bespoke app will bring the average up). We will discuss model-driven development later in the book, but for now take it to be an approach to delivering application functionality that does not rely on writing new code for every slight variation in application functionality.
This will result in 1 Million lines of code to manage, which is a fantastic 1000-fold reduction in complexity. That is a stretch goal. But if in pursuing this you “merely” dropped each in half (very doable), your overall complexity drops by a factor of 8 (or 88.5%).
Chapter Summary
The complexity of our information systems is directly proportional to their cost, both to implement as well as to operate and change. Complexity is what drives the inflexibility of the typical application information system. The cost of our information system is directly proportional to its complexity.
We are so, so far from optimal that opportunity abounds. We can make great strides by reducing the number of applications we manage (rationalization) by reducing the complexity of the models in our systems (through elegant design) and by reducing the amount of code needed per concept (through model-driven development and other techniques). Reducing all three has a synergistic effect. This is the aim of this movement.
The superiority of the Data-Centric approach has been around for a long time.22 Why is now the time to tackle this?