CHAPTER 6
The quagmire by sector

This book was gestated in large organizations. However, it will affect many other sectors of the economy.

How the quagmire looks for government

The government sector is as deep into the quagmire as it gets. We have done more work with state agencies than either local or federal, but our observations of all three suggest that the nonprofit sector is firmly in the quagmire.

Most of the popularized IT runaway projects have come from the public sector. The private sector is better about keeping these things out of the news.

Factors making it worse

What is it about the public sector that leads to their over-investment in legacy systems? The following factors conspire to make the situation worse for the government:

• Difficulty in hiring top talent
• Overly restrictive purchasing procedures
• Contractors that specialize in gaming the system
• Willingness to accept mediocracy
• Budget cycles
• Lack of incentive

Difficulty in hiring top talent

Government agencies often lament the fact that their restrictive pay grade systems prevent them from hiring the best and the brightest. There is some truth to this. Add to this the fact that there are no stock options, lavish bonuses, and the like.

For these reasons, hotshot programmers and architects who understand the latest and greatest tools and techniques are less likely to come to the ranks of civil service.

Overly restrictive purchasing procedures

Government agencies pride themselves in their objectivity and freedom from influence by slick marketing types. How this manifests is through very restrictive purchasing procedures. Most government agencies must go through a competitive bid process for all but the most minor procurement. The process itself adds considerable time to any project. There is the process of defining one’s requirements in enough detail to write an RFP, which takes months. Then the RFP process itself takes months, because the RFP has to be produced and published, time must be allowed for bidders to respond, followed by an evaluation period, interviews, and time to allow for protests before a project can begin. This can add six months to a project cycle.

If six months are allocated to procurement, there is a strong incentive toward larger projects. Why go out to bid for a one-month project? You will only get two months’ worth of work accomplished per year. Better to go with projects longer than a year.

However, this leads right into the larger project/larger failure rate problems.

Contractors that specialize in gaming the system

An entire sub-industry exists due to the size of the government software industry. Pejoratively called “beltway bandits” from their Headquarters near the I-495 beltway around Washington DC, these firms specialize in servicing the government markets.

There is nothing wrong with this per se, but what has happened is these firms have specialized in winning, expanding, and extending government contracts far more than they have specialized in performing them. This is because there is very little incentive in performance, and a great deal of incentive in capture (as they call winning a bid).

Because of the “objective” criteria in the RFPs, and how hard it is to objectively score reputation, the government ends up reinforcing this bad behavior. The beltway bandits know how to answer questions in a way that the contractors are more or less forced to be awarded maximum points. They specialize in protesting when they don’t win. Many government employees are fearful of protests, not only because it delays their projects even further, but because it holds them up to scrutiny. We have often worked for government agencies who lament the fact that they “had to” select a contractor that they knew could not perform, because their objective process lead them to a conclusion they knew wouldn’t work.

Another popular practice is getting involved with an agency in advance of an RFP and making sure there are requirements that only they can satisfy. This generally favors firms with large marketing firms who can dedicate resources to the long-term cultivation of contacts inside the agency.

Finally, there is the revolving door between agencies and contractors. There are some prohibitions against hiring contractors who employ ex-agency employees, but not nearly enough.

Willingness to accept mediocracy

There is a sort of learned helplessness in many of the agencies we’ve worked with. This is partially a product of their difficulty in hiring, and partially a result of the contractors learning to prey on them.

This has led to extreme conservatism. If you don’t believe you can hire and retain talent, you are unlikely to embark on custom development projects. If you believe that your contractors will game the procurement, you will define projects that have little risk and require little skill.

Budget cycles

Most agencies work in 1-2 year budget cycles. It is very hard to get long-term projects funded.

While this is a disadvantage for long-term improvement projects, in many ways I think it has been more blessing than curse. For many agencies, the 1-2 year cycle has forced it to concentrate on smaller (therefore less risky) projects, and especially to focus on projects that would become operational in far less than a year.

Lack of incentive

The private sector has personal and/or collective incentives that are missing from government and nonprofit sectors. In the private sector, the profit motive trickles down. If you can figure out how a software project can improve productivity or increase revenue, the incentive pay of the affected employees will help make that happen. The head of sales will be happy to get behind a project that will increase revenue, as it will improve commissions for all his salespeople, and an increase in revenue will likely lead to bonuses all around.

Government projects also go through much of the same motions when it comes to justifying projects, such as ROI calculations, featuring improved productivity and improved goal attainment. However, there isn’t the same kind of urgency because there aren’t personal incentives attached to the initiatives.

Factors making it better

The situation for government agencies need not be as dire as it is. I have painted with broad strokes a general set of problems amongst government agencies, but it needn’t always be so. At times government agencies have proven themselves to be motivated, adroit, and taking the right kinds of risks. Consider NASA in the 1960’s getting astronauts to the moon and back, or the Department of Transportation and the interstate highway system. We have worked with government agencies who have managed to overcome some of the negative inertia of the above factors and deliver above-average results.

While government agencies have a considerable set of factors to overcome, they have some things going for them that they need to learn to capitalize on. In particular, governments need to learn how to exploit some of the positive factors they have going for them:

• Stability
• Ability to hire pretty good talent
• Longer planning horizons
• Less reliance on packaged software
• Budget cycles

Stability

The stability of government agencies is a strength that isn’t leveraged much. Governments rarely go out of business. Businesses routinely do.

Government should be able to capitalize on this more. Many people feel the tug of stability and the potential to work on longer term projects. It is hard to work on long-term projects in firms obsessed with quarterly earnings.

Ability to hire pretty good talent

While government agencies cannot compete for the top talent, they can attract “pretty good talent.” We have been privileged to work with people who may not have been “top” but were quite competent. In the long run, having a stable group of competent people may outperform a revolving door of superstars.

Longer planning horizons

Governments are not going away. Their permanence is their strength. They can propose and execute projects that could take a decade or more. When I was with a consulting firm, we would help clients build “Long Range Information Systems Plans.” It was rare that we could get clients to pay attention to a three-year plan. I had the fortune of working with two mining companies, which, because of the nature of their business, could think in five year horizons.

Governments routinely think in longer terms–partly because of their permanence, but partly because of their pace. They know things take longer and they plan for it. This is great for making long-term gradual improvements. A small team working over a longer time frame often drastically outperforms a much larger team that attempts to do a project in a compressed timeframe.

Less reliance on packaged software

Even though the government market is large in total, each agency has its own requirements. As such, there isn’t a very large reachable market, and therefore there are not as many packaged options available. This may have been a gift in disguise. Many government agencies are less afraid than their private firm counterparts in taking on custom software development projects. When you are comfortable with custom development, you are more likely to change an existing system rather than buy and attempt to integrate another package.

Budget cycles

We mentioned above the negative on budget cycles, but there is a positive. Because of their budgeting cycles, most state and local agencies tend to one or two year projects. There is often no guarantee that a project will continue in the following fiscal year.

The government agencies’ response has been to package their projects into less than one-year phases that can be implemented and used even if the next phase isn’t funded. Further, because there is a hard stop to the phase, they are generally more conservative in what they put into a phase.

Essentially, they have invented their own form of agile. Longer sprints and epics to be sure, but they still shoot to have an installable product before the end of the budget year. It may not be the “Minimum Viable Product,” but it is often a subset of what they would have designed had they had the luxury of doing all their requirements up front.

This is an over generalization. There are agencies that do launch multi-year projects, and there are some that are practicing full agile development. However, I have seen this middle ground so much more often in state and local government that it deserves a mention, and deserves to be encouraged.

Government summary

Most government agencies are deeply in the application-centric quagmire. Some, especially at the Federal level are deep in the quagmire. As mentioned in the first chapter, some of the attempts to rise up were misinformed, and cost billions of dollars. Our guess is these agencies will have a long memory; it will be a long time until they muster the courage to take another run at it, but it is inevitable.

I’m reminded of a State Teachers’ Pension Agency we worked with. They have ancient legacy systems. The Y2K crisis gave them the impetus to take on a massive rewrite of their systems. After $20 million and nothing to show, they cancelled the project. It was almost ten years before the memory of that fiasco died down. We were brought in and designed their new data model and architecture. It was a rather straightforward design; the most complex part was how to engineer an orderly transition from their legacy systems to a properly decoupled Service-Oriented Architecture. We were not in the implementation business at the time. We expect this would be several million dollars and two to three years. We checked in again after three years and they were $34 million into the project with no end in sight. A less formal check in 2017 revealed that the team was still struggling.

One of the main problems with our industry is that there is far more money to be made by being incompetent than there is for being competent.

There are still far too many contractors who make far more money not implementing systems than there are contractors that can implement productively. Sadly, it will probably be another ten years before this agency gathers the collective will to take this on again. Ten more years of living with woefully inadequate systems.

The death and rebirth of the application software industry

There are some interesting signs of life in the application software industry.

The largest packaged application vendors still bulldoze ahead, mostly ushered along by their implementation ecosystem. As Vinnie Mirchandani points out in SAP Nation, SAP has had several attempts at remaking its core software but their multi-hundred billion dollar ecosystem brings them back to continued extensions of their flagship product, which is essentially their R/3 product from the 1980’s.

However, two trends should worry the large application software vendors. The first is their overreliance on maintenance revenue. Maintenance revenue is the 15-20% of the original purchase price that firms pay for the ongoing privilege of using the software, along with technical support and software feature addition and bug fixes. This is a cash cow for the application software companies. As we have mentioned throughout this book, it is very hard for companies to get off these legacy application systems, even though many are highly motivated to do so.

Somewhere, sometime soon, we will witness “The Last ERP Implementation.” Once something is inevitable, it is just a matter of time. We are convinced that the ERP economics are now at least 10 times more expensive than their alternatives and may be 100 times. Inertia will only keep this trend going for so long.

The second area of concern for the traditional application software vendors is their vastly worse economics to their newer nimbler, cloud-based competitors, as well as the existence of open source alternatives. Most of the established application vendors now offer cloud hosting but they haven’t changed their licensing or integration economics, so it only affects a small part of the equation. So far, the open source alternatives are not complete enough to be competitive. However, this is the perfect set up for a disruptive innovation. The application software vendors are seeing encroachment at the low end of their markets. They respond to the cloud economics by offering their own cloud versions, without changing their economics.

However, there is hope in the application software market.

The first glimmers are in the App Store and Play, the two marketplaces that Apple and Google have created for Apps on Smart Phones. When Apple launched the App Store, its entire concept was far from mainstream. The software market of 2007, the year of the debut of the iPhone looked like this:

Against this backdrop, Apple chose the price point to be 99 cents, or $1.99 for a premium offering (in addition to a large number of “free” offerings that would make money through ad revenue). One would be excused for considering that this would become a trivial footnote to the evolution of the software industry.

The App Store and Play now represent an $80 billion market.39 That is 20% of the traditional application software industry. This is a classic disruptor pattern playing out before our eyes. This was made possible through nearly 300 billion downloads.40 Reflect on this for a second. This is an $80 billion industry, barely 10 years old, whose average sales per item is 25 cents (of the 300 billion downloads many were free).

This industry has created an existence proof. We are beginning to see experiments around how to take this idea to the corporation.

At the moment, the early experiments aren’t able to scale to enterprise needs. One example is Zoho, which allows individuals to integrate their many apps together. Another is IFTTT (IF This Then That) which allows people to automate simple tasks. The problem with these apps, is that they rely on individuals owning their systems integration experience, which is something only a small percentage of the population are inclined to do. The other problem is that while it integrates information for an individual, it does not provide any integration for the firm.

In the companion book, we will take up what the prerequisites are to create this lightweight app experience on top of an integrated enterprise.

Four other related trends are hinting at a way out of the quagmire:

• APIs
• RESTful APIs
• Microservices
• Low code / no code

APIs

Legend has it that Jeff Bezos in 2002 dictated that henceforth all communication between applications would be through well-defined APIs (Application Programming Interfaces).41 APIs are a predefined way for other applications to access each other’s functionality.

APIs have been seen as the solution to the legacy problem. This is certainly a step in the right direction. We don’t know all of the details on how it was implemented at Amazon. But we do know it has helped Amazon a great deal. There are other firms that have implemented something similar without as good a result. The difference isn’t in the use or not of APIs; it is much more in how you do it. Just telling everyone to expose the functionality of their application in an API doesn’t solve the problem, because without a lot of discipline, each application will expose their arbitrarily different internal structures and little will have been gained. What is gained is the number of ways to integrate systems has been rationalized, such as by file-to-file transfer and direct database access. ETL has all been eliminated in favor of the API, which is progress.

RESTful APIs

Roy Fielding’s PhD dissertation42 reverse-engineered the Web to find the principles that made it successful. One thing he found was something he called “Representational State Transfer” (REST), which included several ideas. One idea is that an API provides a representation of a resource, such as an order might be represented by an HTML page, a pdf document, or a csv file. The second idea which has allowed the web to scale is that state is managed in the client. One very subtle implication of RESTful interfaces is that there are very few verbs with GET, PUT, POST and DELETE being the main ones, yet potentially thousands of nouns for the resources, such as Customers, Orders, and Product.

A traditional API architecture has lots of compound verbs such as getCustomer, getCustomerOrder, getAddress, updateAddress, and newAddress.

As we will see, the small number of verbs allows architectures to evolve that could better manage network cache and computer memory. If someone had recently issued a GET request for Order27, it would likely still be in computer memory. And the idea that PUTs and POSTs be “idempotent,” meaning that executing the same PUT or POST more than once wouldn’t change the state of the server, ushered in a set of architectures that no longer had to invest a great deal of infrastructure on guaranteeing that a message got processed only once.

Finally, relying on resources as the nouns greatly reduces the variety that was introduced with traditional APIs.

Microservices

Microservices is a style of developing enterprise applications that might be characterized as “fine grained SOA” (Service-Oriented Architecture). This is a very promising idea. Microservices are similar to libraries of code, the difference being, with a set of libraries you eventually include and compile the libraries into a monolith. The microservices approach strives to create similar functionality as you might find in a library, but to implement it as a standalone package that can be invoked through a service call. This turns a monolith into something that can be composed, potentially at run time.

Our belief is that a well-designed architecture might be able to recreate all of the behaviors of a legacy application with a few dozen microservices.

Low code / no code

The “low code / no code” movement is picking up steam. What has been true for a decade is now being more widely adopted. There are ways to build complex applications with little or no application code.

Salesforce was one of the early entries into the no code movement. Their contention, which is mostly correct, is that they could build complex applications without application code. This is essentially the “model driven approach.” In model driven, the idea is the application is partially or complete derived from a model of the application. In the 1980’s and early 1990’s, this was code generation. A model of an application could be input to a code generator that would create application code that was derived from the model. This was what the CASE (Computer Aided Software Engineering) tools were. In the early days, the CASE tools created a first version, or a scaffold of the application, upon which a programmer could add.

In the 1990’s the technique improved to where complex applications could be generated that did not require programmers to add code, and therefore the application could continue to be maintained by modifying the model.

One breakthrough that is marked by our patent43 is the ability to build a system where the model is the application; there is no application code, generated or otherwise.

This general idea has resurfaced now as the “low code / no code” movement. We are seeing much more movement in the area of building applications with little or no code (this is in stark contrast to the tens of millions of lines of code that comprise a typical ERP system).

Two industries under siege

There are two IT sub-industries that are doing very well now, but will be looking more and more like video rental stores in the near future. One is the systems integration business and the other is the legacy modernization business.

They both promote the idea that applications are inherently complex, and therefore it will take a great deal of effort to either integrate or replace them.

Systems integration

The systems integration industry is a bit of a euphemism. At one point, these were consultants called in expressly to “integrate” systems that had been allowed to grow up in a non-integrated fashion. This integration was often opportunistic, even building point-to-point interfaces if that was the most expedient way to solve the problem at hand.

Many of these projects were large. As these firms grew, they needed more and more large projects to feed the backlog.

Most “systems integration” projects now are implementation projects. The firm is implementing new functionality, which co-incidentally also involves a lot of integration.

As described in the first two chapters, many of these projects are 10-100 times more expensive than they need to be, and their success rate hovers in the 50% range, despite decades of methodological attempts to improve it.

As the systems integration business turns into a buyer’s market, expect to see the ranks of systems integrators thinned considerably.

Legacy modernization

There are two types of legacy modernization vendors: those that focus on obsolete infrastructure and those that focus on replacing obsolete applications.

If your existing systems are overly dependent on hardware, operating systems, databases, or programming languages that have become obsolete, you could be in the market for the first type of legacy modernization company.

If you have a major application that was coded in languages such as PowerBuilder, Natural, or DataBasic, it may behoove you to do a straight port to something that you can find programmers to work on.

Most of these vendors focus on a particular legacy platform. The key to success in these projects is to avoid the temptation to make functional changes to the system while the conversion is in progress. You will want as pure a “lift and convert” as is possible. You can make the functional changes in the new environment after the conversion settles down.

You should consider conversions like this to be tactical. Direct translation of your legacy code from one language to another will not magically make it more malleable. Nor will lifting a schema from one database and dropping it on another make it more than a little bit more flexible. However, it may buy you some time.

The other legacy modernization vendors concentrate on replacing your existing applications with more modern ones.

As we will discuss later in this chapter, what makes the lift and replace strategy so much harder than it seems, is that the existing systems do not have the same boundaries as the system being installed. We have this convenient fiction that an ERP system is an ERP system, but the truth is that every install has a wildly different footprint and has different sets of dependencies each of which has to be addressed.

The real problem is that most “new applications” have most of the same economic issues as the systems they are replacing. Indeed, many have the problem in an even more severe fashion. Most are far more complex and are harder to change than the systems they are replacing.

Outsourcing

The pendulum tends to swing both ways on these industries. Ever since H. Ross Perot invented outsourcing at EDS, firms have been alternately drawn to it, and often wished they hadn’t been.

Some of the early outsourcing was financially motivated. A firm, strapped for cash, could get an infusion from an outsourcing firm, which bought their IT assets in exchange for a long-term contract. The outsourcers aim, often achieved, was to make more over the years of the contract than the upfront investment. They could often do this even while reducing the cost per transaction, at least in part by employing ever-cheaper hardware and networks. Generally speaking, outsourcers were not motivated to reduce the cost of change, and were certainly not incentivized to make major architectural improvements. They merely sought to reduce the cost of change, as change orders were large and lucrative line items.

Conversely, the outsourcing arrangements that seem to have worked out best for clients were those in areas where there was little need for change and customization. Furthermore, there was a low switching cost, which allowed the client to continually shop the outsource arrangement. The traditional 100% outsourcing of an IT shop typically does not allow any part of the system to be switched out. This is especially problematic if the outsource contract locks in long-term rates that become unfavorable.

Email is pretty much a commodity now. Firms can outsource their email, without having major side effects on other systems. It is relatively easy to shop email, and switch when it becomes advantageous. Email is a stable stand-alone application. Over time email will become a feature in many systems, and firms will be looking for better ways to integrate their many communication channels. At that point, email may no longer be a good outsource candidate, but for now it is an exemplar.

Outsourcing parts of the stack will continue to create favorable economics. Cloud computing is outsourcing your hardware and operating system infrastructure. The internet is outsourcing your computer network.

On the other extreme, non-commodity, highly customized systems, such as ERP systems, are not very good candidates for outsourcing. The outsourcing arrangement tends to lock in the inefficient cost structure.

Our prediction is that outsourcing will continue to grow for the commoditized portions of the solution, and will gradually begin to shrink for the custom and changeable parts of the application infrastructure.

Offshoring

The offshoring industry is essentially moving work to places where labor is cheaper. In this regard, it is much like the offshoring of manufacturing or garment production. At one level, it is easier, as we can sling code around the world at the speed of light, where fabric and plastic still move at the speed of container vessels.

The other relevant aspect of offshoring software is that the technology is constantly changing, as are the requirements.

Each wave of offshoring boosts local demand for the skill being offshored.

Garment manufacturers originally moved production to China. As wages in the garment industry rose, many moved their operations to India. Even with their vast reserves of labor, wages began going up in China and India. Garment makers moved to Vietnam and Bangladesh44 as well as Indonesia, Malaysia, Sri Lanka, the Philippines, and Korea.45

So far, software outsourcers haven’t followed this pattern. This is at least in part due to language barriers, and the relatively low level of technical college training in the lower-wage countries. Software outsourcers have tended to go where talent is high, and the costs (while not at the Bangladesh garment level) are still competitive. These are countries like Bulgaria, Hungary, Lithuania, and Portugal.

Much of the offshoring business, indeed most programming, will become partially or completely automated in the coming decades, and dry up the demand for offshore technologists.

For the medium term, there is still an advantage in offshoring legacy maintenance, and some systems integration, but more and more firms will understand this to be a rear-guard action.

The other area that will prosper in the medium term will be advanced technology. Many pockets of expertise will spring up around technologies like machine learning, natural language processing and big data.

The new platform vendors

What are already emerging are new categories of platform vendors. There is a great attraction in being a platform vendor.

A platform vendor is someone who creates the infrastructure in which a whole ecosystem is based. The most valuable platforms are the “two-sided” platforms.

For instance, Amazon is a two-sided platform. It is a sales platform to buyers of consumer goods, but in the early days when it was only selling its own merchandise, it was a one-sided platform. Once it opened its platform up to vendors and allowed them to sell their own merchandise, it became a two-sided platform.

These two-sided platforms have incredible network effects. Once you have the majority of the online buyers, it is easier to attract additional sellers. Adding additional sellers just increases the demand on the buyer side.

While Amazon is a platform for physical goods, there are many purely digital platforms. Apple’s App Store (and its predecessor iTunes) and Google’s Play are application platforms. And Amazon’s AWS business is a platform for cloud offerings.

Salesforce started life as a SaaS (Software as a Service) provider, but once it opened up its development capability it too became a two-sided platform, and currently most of its revenue comes from applications they did not develop.

There are tools now that will allow for DIY (Do It Yourself) integration such as Zoho and IFTTT, but these may not scale well. There will evolve a few platforms, and eventually one dominant platform, that will flip the current application-centric paradigm on its head, and these will scale well.

How application centricity robs productivity from small business

Small business might be the segment of the economy least affected by the quagmire.

The very small business (1-10 employees) typically has an accounting system (QuickBooks more often than not) and a series of other task-specific applications, depending on what they do. They will often have SaaS (Software as a Service) applications for things like CRM (Customer Relationship Management) or Inventory Management. They will often have their email list managed by someone like Constant Contact or Mail Chimp.

If you look closely, you will notice that they have data integration issues. For example, the customers in the CRM system have to be re-entered into their accounting system in order to be invoiced. Some of their contacts are redundant between their CRM system and their email management system.

But the problem is more of a nuisance than a strategic barrier. It can mostly be handled with some redundant data entry, and if they miss a few, it’s not that big a deal.

Eventually, when the new platform vendors, described in the previous section, hit their stride, we expect the small business point solutions, whether on premise or cloud-based, will be under threat, but until then, this is the sector least affected.

As small firms get bigger, their applications systems tend to become more numerous and more complex. My peers in the CEO group called Vistage used to refer to our business as “cleaning up hairballs.” They said that growing small businesses need a “hairball prevention service.” They bought into the idea that there were things they were unwittingly doing that would eventually lead to issues that would require a great deal of effort to unwind, and that paying attention and preventing this could be worth a fair bit.

A growing body of research suggests that they were on to something. In the study of small business growth, several authors have identified “crises” that must be navigated for a small company to emerge and grow to become a large company46 especially Larry Greiner’s work on how companies grow through five predictable crises.47

The last crisis in Greiner’s original work was described as the crisis of red tape followed by the collaboration growth phase. This last phase has been reinterpreted as the integration phase, as most small companies got to this point building out silos as fast as they could.

Summary

The quagmire is playing out quite differently in different sectors. The public sector arguably has more legacy software per capita than any other sector. It has a few things going for it, such as an embrace of agile and lean, but far more working against it, including difficulty hiring the talent needed to get them out of the quagmire, and an industry (the “beltway bandits”) that is dedicated to keeping them in the quagmire.

The application software and the system integration industries may be the most threatened. In the end game, they are slated to go from about $400 billion per year for each of the two industries, to asymptotically close to $0 billion. This won’t happen overnight but it will be even more extreme than the change for the public sector, as the public sector will at least still exist, even though it will take them quite a while to crawl out of the wreckage of their legacy systems.

Small business and the digital native companies (those founded after 2000) will fare the best, as they have invested the least in the quagmire.