The preceding vision is clearly not the status quo in most firms. The rest of this book and its companion volumes will go into more detail, but let’s outline what is needed to get from here to there.
What it requires
Moving a firm from its application-centric habits to a new Data-Centric paradigm will require changing a lot of things simultaneously. Everything is arrayed against the change agent.
After living with their dysfunctional behavior for so many years (a sunk cost if ever there was one), people become invested in defending their dysfunctions rather than changing them.9
Marshall Goldsmith, Mojo
The resistance to changing to this new way will be both subtle and diffuse and, at the same time, pointed and specific.
This will require someone (or preferably a group of someone’s) who can understand and finesse the kind of resistance that an undertaking of this type will require. There will be overt and covert resistance, as well as pure inertia.
Inertial resistance
The inertial resistance is perhaps the trickiest. It isn’t intentional. It isn’t aimed at you. It isn’t even aware. In a large organization, executives have internalized a way of getting things done. It involves the particular seasonality of budgets. It concerns whose agendas need to be reinforced. There are norms around how projects gain momentum. In some organizations, the most successful gambits involve positioning one’s project to be the answer to regulatory issues. In others, the case needs to be made that a given system is a “burning platform” and must be abandoned as soon as possible. Others have the luxury of long-term plans to tie one’s initiatives to. Whatever the prime movers, most executives have worked out how to get projects on the docket and get them approved, perhaps without even being consciously aware of the strategies they are employing.
In addition to considerations around getting projects considered and approved, there are norms around project sizes and how much pre-work must be done to justify a project of the “normal” size. We worked with a State Agency that was funded on a biennial (every two years) basis. There were never funds assured for any project for more than two years in duration. That organization had learned, over decades, to scope projects to be a bit less than 24 months in duration, and to have interim deliverables such that they didn’t get to the end of the biennium and have nothing to show. A two-year project (typically with dozens of developers) demands a bit of planning: typically, at least a three-month feasibility study and a six-month requirements study. If everything goes well, the project is found to be feasible, and the requirements suggest a project scope that fits with the trial balloon budget numbers floated before the project scope had been nailed down.
There is nothing wrong with any of this. Indeed, this is a learned response to a set of goals and constraints unique to each organization. I bring this up because what you will be attempting to do will not fit this pattern. Therefore, it will be hard to find allies, because no one has any history with doing things this way. They may well agree with you and not know how to move forward.
Overt and covert resistance
I just covered the resistance of people who may actually want to do what you are intending to do. Let’s talk about those who don’t want this type of endeavor to succeed.
Enterprise IT is almost a $4 Trillion a year industry, growing at a healthy rate. There are tens of thousands of firms who are invested in continuing the growth trajectory they have been on for the last several decades. I met a Systems Integrator senior business development VP who was in the midst of closing in on an $800 Million CRM project. He had been working on this account for years. Do you think he was interested in how they could take 99% of the cost out of this project? (Spoiler alert: he wasn’t.) Had I been motivated, I’m sure I could have figured out how to take 10% out of the cost and risk of the project, and it’s something someone with an $800 million project would be interested in. But I’m not interested in perpetuating this madness.
On the other side, the buyer’s side, there are equally interesting incentives and habits to overcome. An executive will have been working on a major project for years. He or she will have compared notes with executives in other similar sized organizations and functions. After you have spent $5 million on a requirements project, which reinforced your preconceived belief that your new project was going to cost $120 Million (+/- a laughably precise $15 Million delivered by very earnest consultants whose confidence springs from the reams of spreadsheets backing up their conclusion), you would be hard pressed to launch an implementation project for $2 Million.
Furthermore, you have managers who have hundreds to thousands of direct reports, and very large budgets who are not motivated to think about drastically reducing the size and importance of their empires. You will find a sponsor of a rival project (as we did) who just committed $20 Million to a packaged application who are not interested in being embarrassed by a pilot project that shows that a better result could have been (was) delivered at a fraction the price. We are aware of another firm who built an incredible customer support system, using what we would describe here as a Data-Centric approach (although they didn’t call it that at the time). It was implemented at a major customer, and delivered unprecedented metrics, such as issue resolution on first call, call prevention, reduced number of agent handoffs, and reduced churn. And yet the project was cancelled. There were other executives who were betting on an alternative approach who, while they had inferior technology and methodology, had superior political skills, and carried the day.
I don’t say this to generate despair. I say this to reinforce my earlier point: this endeavor will require an executive who is senior enough to deal with these issues and, at the same time, junior enough to have the time and the inclination to take on a long-term change project of this type.
That is a lot of bad news. However, interestingly, there are a lot of things this change doesn’t require.
What it doesn’t require
First and foremost, this doesn’t require a large budget. Every case where we have seen this succeed, it has done so with a modest budget (6-12 full-time equivalents, typically at the core of the change process, and then a similar sized team for each conversion or initiative).
It does not involve exotic or proprietary technology. It can be, and has been, done with clever adaption of traditional technology, as we will see in the first set of case studies. If you choose to leverage Semantic Technology (which we make a case for later in the book), you will be pleased to find that once you get over the initial unfamiliarity hurdle, there is a wide array of tools and platforms built on a set of open standards in a way that enables true interoperability. You will find many open source alternatives for many parts of your architecture, and for those where you prefer the comfort of commercial support, their adherence to open standards prevents vendor lock in.
It does not involve extremely rare expertise. Every year there seems to be a new bidding war going on for the must-have expertise da jour. Currently, it is machine learning and artificial intelligence. The last few years it was data scientists. Next year it will be something else. When these fads hit, the cost for talent can rise considerably. You will need some talent, just not those that everyone else is looking for. One of the two talents you will need will be the ability to model well at the conceptual level. These skills are findable, and the companion book, Data-centric Pattern Language, will help someone who is well-versed in conceptual modeling come up to speed on the additional skills they will need to model in this environment. Likewise, for the foreseeable future, there will not be suitable fully formed platforms available to purchase, and you will likely have to staff a team to build out your architecture. The other talent you will need will be solid technical architects and developers who are fluent in modern languages, such as JavaScript and python or java. The other companion book, Data-centric Architecture, will flesh out the additional considerations they will need to incorporate.
This is a program, not a project
A project has an individual and specific goal. Achieving that goal is the completion of the project. Building a dam is a project. Assuring flood control, non-fossil fuel energy, or adequate sources of agricultural water is a program. A program has goals, but these goals evolve over time, and the goals require many interrelated projects to contribute to its success. A program may decide to change the sequence or type of projects it employs to achieve its ends. But it is still possible to tell whether a program is succeeding or not.
Decades of IT implementation have caused us to focus on projects. Each project is individually justified and executed. It is rare that they are seen as contributing (or detracting) from a larger scale program. That’s what is different about the Data-Centric movement. The Data-centric movement involves many projects, but it is a program. The success of the program rests on the degree to which the firm’s application functionality is based on a single, simple, extensible data model. As that overall program goal is achieved, many other metric-based goals will improve as well (e.g., cost to implement new functionality, percent of budget spent on integration, and IT spending as a portion of revenue).
The transition to a Data-Centric approach requires discipline and consistency
Organizing around programs rather than projects allows for shifting priorities. Successful completion of individual projects is not the key predictor of success. Yes, you will be running several projects, and most of them need to be at least partially successful, but this isn’t the main thing. You may be encouraging others to start projects, and their contribution to the program is more important than their own specific success.
You will launch projects that are only tangentially related to the program’s long-term goals. You may launch a project that is primarily about getting more skills in the organization. You may launch a project that makes the program more visible to the enterprise. Many of the early projects will have an immediate application payoff coupled with a contribution to the architecture.
A few early wins will provide the cover needed to deliver a few projects that are necessary but may not be as publicly defensible.
The IT fashion industry
What you will notice if you look through the long-term lens is that IT is a fashion industry just as much as the garment industry. We read trade magazines and go to conferences in order to find out what approaches and technologies are “trending.” Developers have learned that padding one’s resume with the latest languages and frameworks is the golden ticket to higher wages. Knowing the acronyms is key. Even pronouncing them correctly is important.
While executives are not expected to be technically up to date, they need to understand which technologies apply to their business and are expected to be making strategic bets in those areas. As I am writing this book, virtually every company in North America has a blockchain project in progress. 90% of them are expected to be abandoned10.
Gartner has made a mini-industry tracking where each emerging technology is on its inevitable trajectory. Gartner calls this trajectory the “hype cycle.”
Most people who have been in IT for a while recognize the pattern. In phase one a new technique or product is invented. The early adopters understand the technology and especially understand how the technology can contribute to real benefits. At this point, it is no different from diffusion of innovation as seen in other technologies.
IT technology differs just after we get through the early adopters. As a new technology begins to show promise and benefit, the hyping starts to kick in. Hype in the IT world is a many-headed beast. Developers jump on bandwagons, sometimes just out of curiosity or novelty.
Many developers are intelligent and curious and are drawn to new developments. They also are often handicapping technologies and trying to guess which ones will most enhance their future earning power. When a hyped technology goes mainstream, companies end up in a bidding war for the talent to implement it. Because these technologies are new, it is often hard to evaluate competency directly, and employers tend to rely on proxy measures, such as experience reported on a resume.
Adding fuel to the hype cycle fire are IT vendors who either promote the new IT technology directly or attach it to their current offering. Right now, for instance, all products are cloud enabled.
As news of the early successes begins to surface, executives are increasingly drawn in. It is a combination of fear of missing out and perceived competitive necessity, worrying, “if our competitors get a head start on us here, we might not be able to catch them.”
Once a technology begins to get hot, these three forces (developers padding their resumes, vendors pushing new technology and fear of missing out) combine to overheat it very rapidly. The “hype” (vendor marketing which is often over promising the potential outcome) combined with demand (developers and business people, each with their own motives) combine to create a rush to the technology. This corresponds to the peak of Gartner’s hype cycle.
Unfortunately, most of the people rushing in to fill the demand are not up to the task. Additionally, many of the projects meant to leverage the new technology were not well thought out as executives and developers rushed to apply the technology. This creates what Gartner calls the “trough of disillusion.” At this point, most of the people who adopted the new technology are disappointed because “this stuff doesn’t work.” Thus, the technology falls out of favor.
However, in most cases, the technology had legitimate value. After all the failures wash out, the few projects that had the right talent and target show the way forward. After some notable post-trough successes, executives feel safe in re-approaching the technology. Luckily by this time, the hype has come and gone. At this point, it is easier to find truly experienced developers, and the race for talent isn’t as severe.
Is the Data-Centric approach a fad?
By this point, you may be wondering whether this whole new Data-Centric approach might just be another passing fad. So far , it isn’t exhibiting those characteristics. There are no products hyping their “Data-Centric-ness,” nor are there developers leading with their Data-Centric street cred.
Could it become a fad? It is possible, and we want to stay on the alert for signs that it is happening, but I doubt it will. In my opinion, it will not be technology as much as mindset and discipline that prevent Data-Centric methods from becoming trendy.
As far as I know, “agile” never went through a hype cycle, and yet it has arrived. A great deal of development now is agile. Again, I think this is more because agile methods weren’t associated with a specific technology, but were more about a change in the way of doing things.
Can Data-Centric methods benefit from other fads?
Can we or should we “fad surf” for the benefit of our Data-Centric initiative? It may seem cynical or opportunistic, but I think we should. Initiatives like the transition to Data-Centric methods are hard to fund. The progress is quite gradual. Fads, on the other hand, are very easy to fund. If you time it right, your firm will be demanding whatever new technology that has suddenly become “it” in your industry.
Attaching your Data-Centric initiative to an emerging fad can be a win/win. You will gain funding and senior executives’ attention by becoming part of the flavor of the month that the enterprise is getting behind. The other side of the win (which we will pick up in the chapter “Data-Centric and Other Emerging Technology”) is that almost all the past, current, and upcoming IT fads that we are aware of actually benefit from a dose of data-centrism.
Fad surfing brings the danger of losing focus. This is tricky. The main thing becoming Data-Centric requires is consistency. A spacecraft on a mission to a distant planet can often get there faster by diverting and flying to another planet in a way that is designed to increase its speed through the “slingshot effect.”
You want to get a slingshot effect from your close encounters with technology fads without falling into their orbit. If you find yourself caught up in the hype of the fad, you will end up off course on your primary objective.
From Fad Surfing to New Discipline
Enterprises have a great deal of inertia around their information system implementation practices. While the fads of the technology industry can provide a nudge toward change, and can often free up budget for change, as has been demonstrated time and again, new technologies don’t in and of themselves create lasting change.
Lasting change comes from changing the way things are done. Our observation is the needed changes are in two main areas:
- New Modeling Discipline
- New Delivery Architecture
New modeling discipline
We have trained a generation of data modelers how to build silos. Those thousands of data models you have in your enterprise were each independently designed. In some cases, by competent designers and modelers. But there has been no discipline and no impetus for these designs to converge into a single data model as we suggest here. This will require discipline and a change in the modeling approach.
For an idea of the types of changes that will be needed in modeling, we can examine the undisciplined state of the art in data modeling as it is currently practiced.
If you work for a large enterprise, you have repeatedly built or bought applications that cover many of the same concepts. If you look at the data models behind these systems, you will be amazed at the ingenuity. You will be gobsmacked when you find out how many different ways there are to model the same thing. This overlap in concepts, and lack of overlap in implementation is exactly why systems integration is hard. If every application structured their common data the same and named all the same attributes the same, systems integration would be just a matter of copy and paste. But it isn’t. It isn’t even close.
Let’s examine where this variety comes from.
There is a great deal of methodology and mathematics around structuring a data model once you decide what you are modeling. This includes the practice referred to as normalization. Normalization is a step-by-step process to move from a poorly organized first cut of a domain design to one that optimizes its arrangement. If two modelers using the normalization process started from a similar, even flawed, starting point, they should end up with the same result.
The issue is that two modelers covering the same domain do not start from the same starting point.
It is worth examining briefly now, and in more detail in the companion volume, why two modelers will create wildly different starting models.
The first thing is that almost all data modelers start with a “clean sheet of paper.” That is, they go into a design session ready to model whatever they encounter. They interview subject matter experts who describe the data that they work within their own terms. Data modelers jot down these terms and begin experimenting with ways to organize it.
While they start with a clean sheet of paper, whatever they capture first has a profound impact on how the design occurs. From that starting point there are many more design choices. Almost any aspect of a model can be done at various levels of abstraction. Many aspects can be parameterized or intentionally made more flexible. At this point, all the terms are made up and gain their own sort of permanence.
The endpoint of this process is what gives us such rich variation just where we don’t want rich variation. And keep in mind, the model for a packaged application is based on a core set of interviews that were conducted, decades ago, usually, and often in industries and settings far removed from yours.
This is a discipline, and it will not come naturally or easily, but if we are going to have a Data-Centric enterprise, we are going to need to change the way we build our data models.
New delivery architecture
And we are going to need a new delivery architecture.
One reason this sounds daunting is that for most projects, we just accept the delivery architecture of our application packages. Really, it’s the average delivery architecture of the application packages we’re implementing now.
In the early 90s, most enterprises were writing “fat clients” (Visual Basic, Power Builder or their equivalents) with most of the application logic in programs installed on PC clients. They talked pretty directly to relational databases. In the late 90s, the stack mostly moved to server-based systems, mostly based on Java, still with relational databases. Most firms experimented with message-based architectures and/or API-driven architectures. These days, RESTful endpoints and JavaScript frontends are mostly the norm.
When I say that we will need a new delivery architecture, that sounds like a big ask. And yet, we have been continually changing our delivery architecture for as long as we have been implementing systems. The only thing that may be a bit scary is that we have mostly accepted architectures from packaged applications as the starting point or prototype architecture, and then extended from there. In the Data-Centric world, we are not going to have applications that are setting the norms for architecture because we are not going to have applications as we currently think of them.
As we suggested earlier and will elaborate more fully in the third part of this trilogy, this architecture will have very little application code. There will be small “applets” comparable to applications in the app store or Google Play, but there will not be monolithic applications as we know them.
Along the way, the new architecture must co-exist with existing systems. This is essential. New functionality in the Data-Centric architecture is going to be adding on to what is there more than replacing what exists. As your architecture matures, you may find point solutions where you can replace legacy functionality with the new more flexible type. But there will be such a long period of co-existence that one of the key aspects of the architecture will be how it will promote this easy integration.
Luckily, this is one of the strengths of this architecture. For right now, we are just going to state that standards and tools exist to make it easy for traditional systems and Data-Centric systems to interoperate. We will go into some of the specifics in the chapter on formal semantics.
As you bring functionality to the shared enterprise model, there will be code that performs some standard functionality in a more general fashion. The chapter on “model-driven everything” will explain how this works and what the scope will be. Some functionality that was traditional in the realm of the monolithic application, such as security, identity management, constraint management, and notification, will move from application logic to the architecture.
Chapter Summary
This is not going to be easy. There is no “quick win.”
The good news is this entire program will cost less than an average application package implementation. It will deliver incremental improvements along the way and, once the program gets into gear, will pay for itself as it goes.
But the changes are quite profound. It will change the way people think about design. It will change the architecture they use to implement systems. It will change the way systems are funded. Initially, you will have very few co-conspirators. You will be forced to adapt to whatever IT fad is being funded in order to fund continuous improvement. It will take a long time. By the time it becomes the new normal, there will be many who are willing to take credit for it.
But this is the nature of profound change. It’s not for everyone.