Chapter 19
Centers of Excellence
Abstract
Centers of excellence (COE) are people-oriented solutions that systemically address enterprise-wide problems of disconnected application and data silos. They operate cross-functionally with different business groups along with their supporting processes, applications, and technologies. COEs enable enterprises to share skills, expertise, and data across applications used in different groups so, as a whole, the enterprise is making better use of the applications, and resources are more easily shared, while investments are maximized. A COE can be an actual organizational unit or a virtual unit. The book covers two COEs: BI and data integration. The Business Intelligence COE (BI COE) coordinates and oversees BI activities, including resources and expertise. The Data Integration COE (DI COE) team focuses specifically on data integration: establishing its scope, defining its architecture and vision, and helping to implement that vision.
Keywords
BI COE; Business and IT; Center of excellence; COE; Data integration; DI COE; Enterprise-wide• BI and data integration centers of excellence (COE)
• Need for BI and DI COEs
• Business case for COEs
• Skills needed
• Organizational models
• Deliverables
• Best practices
The Purpose of Centers of Excellence
Teamwork is an essential ingredient in successful BI, as we discussed in Chapter 17. Centers of excellence (COEs) are organizational teams that help companies coordinate similar activities that are taking place in disparate groups all over the company.
COEs are people-oriented solutions that systematically address the problem of disconnected application and data silos across an enterprise. They operate cross-functionally with different business groups along with their supporting processes, applications, and technologies.
COEs enable enterprises to share skills, expertise, and data across applications used in different groups so, as a whole, the enterprise is making better use of the applications, and resources are more easily shared, while investments are maximized. A COE can be an actual organizational unit or a virtual unit.
This chapter will discuss COEs for BI and data integration.
• The Business Intelligence COE (BI COE) coordinates and oversees BI activities, including resources and expertise.
• The Data Integration COE (DI COE) team focuses specifically on data integration: establishing its scope, defining its architecture and vision, and helping to implement that vision.
Both of these COEs have occasionally been labeled as competency centers: BI competency center (BICC) and integration competency center (ICC). This book uses the term COE versus CC for psychological and marketing reasons. Clients have told me they do not wish to be merely competent, but rather strive for excellence. Creating a CC implies that you are currently incompetent, while creating a COE means you are trying to improve to being excellent (and it does not disparage your current efforts or skills). This distinction may sound humorous, but using the right terminology can help you convince management that the COE is needed and gets business and IT people onboard.
BI COE
The BI COE is a cross-functional team composed of business and IT people representing the BI stakeholders in an enterprise. Its goals are to:
• Expand the use of analytics to better manage and grow the enterprise’s business;
• Improve business and IT productivity in the design, development, deployment, and use of analytics;
• Increase business value and ROI from the investments of time, resources, and capital in analytics;
Based on organizational preferences, the BI COE span of control would be either:
• Advisory—team coordinates and oversees BI projects and activities
• Development—team includes hands-on people from both business and IT groups who design, develop, implement, and support BI applications throughout the enterprise.
The BI COE pools the skills and expertise of people doing analytical work and allows them to collaborate. The pooling may be in a centralized organization or a virtual team; there are several alternatives for collaboration, which we discuss later in this chapter. As with many of the people-oriented processes developed for BI, it is best to start small, learn from successes and mistakes, and expand to an enterprise-wide solution iteratively.
Why Businesses Need BI COEs
The reasons why businesses need BI COEs are deeply ingrained in the way they operate and are organized. These factors are never going to change, so it’s almost inevitable that a BI COE is going to be needed.
Organizational Structures Drives Reporting Silos
The need for a BI COE arises from the basic way that companies are organized—as silos. This isn’t a bad thing—it’s just an observation. Companies are typically organized along business functions such as sales, finance, marketing, and human resources. Companies may further be divided by geographic boundaries or the products or services they offer. Although larger enterprises are likely to have more complex and detached groups, small to midsize enterprises are often organized in this fashion, too.
Each business group builds or acquires applications to support their unique needs; these applications are different and separate from those in other groups, which have their own unique needs and customized solutions. Although many firms purchase enterprise resource planning (ERP) systems that support business processes, the reality is that even these applications are split into modules geared to specific group needs. Typically, these ERP systems have reporting capabilities that business people use. As their needs get more sophisticated, people may expand the ERP systems’ reporting solution, build new custom reporting applications, or use spreadsheets pulling in data from the ERP systems for reporting. Each of these approaches creates or expands the distinct islands, or silos, of applications and data.
Each group may be doing what is right for them, but as cross-functional, enterprise-wide reporting becomes a higher priority, this approach will become counter-productive. BI should keep pace with and support the cross-functional nature of an enterprise, not hold it back. However, each reporting silo adds to the escalating costs of integrating the data, making it consistent and reconciling differences in reports. BI cannot become truly cross-functional if it is hampered by silos; each silo makes it more difficult to achieve that goal.
Business Preferences Exacerbate the Problem
Many enterprises, both large and small, are using multiple BI tools. It is not uncommon, for example, for a Fortune 1000 company to be using more than a half dozen different BI tools. Some of these BI tools may be the result of different ERP applications bundling different BI tools in their reporting solutions. The business people using a particular ERP system typically have no choice but to use the bundled BI tool, whether they like the tool or not, because that is the only method to gain access to that system’s prebuilt reporting.
In addition to the BI tools bundled with ERP systems, it is very common when a customized reporting application is built for business groups to select a different BI tool. Typically, the BI tool selection process is done based on the needs of that business group and is independent of enterprise-wide considerations.
Enterprise IT groups that are driven by efficiency and cost considerations want to establish enterprise-wide standards for the technologies used. Although they have been very effective with technologies related to infrastructure, i.e., things that the typical business person is not all that concerned with (unless it does not work), they have been less successful on standardization when the business person visibly interacts with the technology. BI is one of those areas.
A business group might believe that the BI standard conflicts with what they feel is the best solution to their unique needs. They are focused on solving business problems, and are not really concerned if a BI tool meets IT’s criteria for a standard. Their dissatisfaction with the tool may stem from the fact that IT selected it without really understanding what the business needed, or because they have prior experience with another BI tool, or even because they have listened to a vendor claiming that another tool is superior. So for any of these reasons, the tools don’t get used, and the business people either use a BI tool they selected or continue using spreadsheets.
Although the business people understand what data they need better than anyone else, they may not be well versed in choosing a BI tool that best helps analyze that data. They’re not technology experts, after all. So, a vendor may convince them that their feature-rich product is the answer. However, product features aren’t the only things one should evaluate when choosing an application. What about how it meshes with the company’s existing architecture and technology strategies? Is it going to just create another silo that only meets a narrow, departmental need but conflicts with the rest of the company’s IT solutions?
The trends toward self-service BI also contributes to this problem as business groups assume, and are reassured by product vendors, that their BI needs will be solved by the ease of use and powerful analytical capabilities of the latest BI tool. First, there should always be a disclaimer with these purchases that the self-service BI tool is only as good as the data underneath it; and, second, no matter how self-sufficient a business group feels it is with these tools, they will eventually need to work with other business groups and IT as their analytical needs expand. The initial self-service BI applications look good, but the next silo has just been created and with it all the downsides that we have discussed.
As we covered in Chapter 16, silos are expensive and messy. It’s so easy (and tempting) to build them, and so expensive and complicated to solve the problems they create.
Building a Business Case for a BI COE
A BI COE helps organizations maximize their investments in BI. The benefits of a BI COE are:
• Increased BI business value and ROI
• BI initiatives are better aligned with the business and become more successful
• Investments in BI are well-spent
• Lower total cost of ownership (TCO) over time
• Enterprise-wide BI strategy and road map is agreed upon
• Processes enable cross-functional input, feedback, and decision-making
• Strategy and road map are communicated and promoted
• Improved business and IT productivity
• Enable self-service BI
• Data silos such as data shadow systems are reduced
• Leveraged BI skills, knowledge, and experiences across enterprise and over time
• Business groups benefit from one another’s BI experiences
• Maximize use of BI skills throughout the enterprise
For someone who has been involved in BI projects, it may be obvious that a BI COE can benefit your organization, but others may need some help understanding its value, especially if they’re being asked to help fund it and commit resources to it. Breaking down organizational silos and working collaboratively in a BI COE often encounters political resistance. This resistance typically is passive-aggressive in nature, meaning that everyone says they are for it, but behind the scenes they are acting like the silos will always exist. To avoid this resistance, it is critical to sell the BI COE, to openly discuss the problems with the status quo, and to get business groups’ resource commitments to the COE initiative. It may be helpful to frame it as follows:
1. Explain the value of BI
2. Point out the cost savings
3. Reinforce business and IT objectives
4. Demonstrate the improved alignment of resources
These points are all explained further, below.
Value of BI
If they don’t already realize it, executives need to understand the value of BI in the organization before they can even consider a BI COE. With Big Data and analytics so often appearing in the mainstream business press now, BI has gotten a big boost. However, closer to home, executives need to be shown examples of how critical the data from BI projects is to enabling the decision-making necessary to effectively manage and grow the business. Exposing the obvious reliance on BI for business success helps pave the way for funding and support of a COE.
Cost Savings
The silo approach to BI projects wastes money. Each group is on its own, figuring out what approach to use, what tools to buy, and hiring and training resources. No group has a chance to learn from one another’s mistakes. The overlapping and redundant nature of the BI projects greatly increases costs, lengthens the time to implement analytical processes, and creates lost business opportunities due to the backlog of analytical needs going unfulfilled.
Inconsistent or missing data across data silos creates a very expensive hidden cost and productivity drain (see Chapter 16). This inconsistency causes many business people to gather and reconcile data across these silos, taking significant time away from business analysis and decision-making.
The BI COE offers immediate cost savings even with just the economies of scale of shared resources, knowledge, and data. Then, it ultimately saves the enterprise money by avoiding unsuccessful, expensive BI projects and eliminating the need to reconcile data across silos.
Business and IT Objectives
With a BI COE, business groups have a formalized process to voice their needs: defining and prioritizing their data and analytical requirements across business groups. The COE team can build a BI strategy and road map that meet the needs of all the groups, and reduce their temptation to build silo-type solutions. The BI road-map is critical to elevate the perspective and timeline beyond individual BI projects.
Likewise, this helps the business groups gain a better understanding of how important it is to have a cohesive, enterprise-wide IT strategy. It reminds them that they don’t operate in a vacuum, and they can also learn from the experiences of other departments. The sum of the deliverables from the BI road-map should be greater that what the individual business group could achieve by itself and at a lower aggregate cost.
Monitor BI Performance with BI on BI
How do you know if a BI program is successful? Are the business people actually using the application, or are they returning to their data shadow systems? How easy is it for them to get the answers they need? The BI COE is where you can perform “BI on BI,” monitoring the performance and success of the BI program.
A Forrester Research report [1] sheds light on the BI on BI concept by pointing out that “Enterprises still rely largely on intuition and qualitative hearsay assessments of business users’ level of satisfaction with BI applications and tools.”
Using BI on BI, as the report points out, the BI COE can make recommendations on the following:
• Purchasing or weeding out software—taking into account consolidations, migrations, and upgrades
• Assessing the maturity of the BI environment
• Justifying new BI endeavors
• Confirming adherence to service level agreements (SLAs), clarifying the basis for charge-backs
• Justifying incentive compensation
• Complement governance, risk, and compliance activities
• Gauge BI performance against benchmarks
These efforts can help the BI COE increase the usefulness of a BI program by predicting the requests business people are going to make, and finding hidden data so it can be used for decision-making.
Improved Alignment of Resources
The BI COE can help the enterprise identify and better align its most skilled and experienced people with BI. Enterprises typically have a shortage of BI-savvy business and IT people, and the push for Big Data only makes that gap more acute. Where relying on ad hoc teams for each BI project would find each group starting from scratch, working with a BI COE would make it easier for groups to find resources—people, standards, processes, and templates—to accelerate the launch of their projects.
It also encourages more coordinated training. Although a single department might not have the budget for in-house, customized training, a BI COE can arrange for training for business people in various groups, ensuring they learn the basics of BI, the specific tools the company has chosen, and how to analyze the new streams of data they’ll be receiving. An enterprise-wide approach also enables the COE to establish “train the trainer” or mentoring programs staffed by internal staff, which is something that would not be practical if BI projects remained as silos.
The COE has insight into various business units, so it makes it easier to share data sets and results among them. The business units benefit from the cross-pollination of reports, dashboards, and analysis that this can enable.
BI COE Deliverables
The BI COE’s mission is to expand the use and business value of analytics in a cost- and labor-effective manner. The BI COE needs to interact with business groups, the BI development group, and the IT group that supports business application (Figure 19.1). Working with BI subject matter experts (SMEs) in the business groups, the COE determines what the BI consumers need. Working with systems-of-record (SOR) SMEs in the business application group, the COE learns where the data originates.
The process of establishing and operating a COE will include these deliverables:
• Establish processes for business and IT interaction (see Chapter 18)
• Requirement-gathering and prioritization across business groups
• Change and scope management
• Communication and marketing plans on the state of analytics in the enterprise
• Create a BI Portfolio (see Chapter 7 and Chapter 14)
• Assess analytics needs associated with:
- Business functions and processes
- Business people’s analytical styles and skills
• Design BI Portfolio capability and usage road-map
• Assist in implementing BI Portfolio incrementally
• Formulate policies and adherence standards (see Chapter 4 and Chapter 17)
• Access, privacy, and security
• Data governance (in conjunction with data governance team, if one exists)
• Reporting and analytics governance
• Create common and reusable processes (if there is a hands-on team) (see Chapter 14)
• Expand knowledge and skills (see Chapter 17)
• Determine skills, experience and knowledge of business BI consumers
• Obtain or deliver targeted business training
Skills Needed
The BI COE needs people with a variety of skills: technical, business, and analytic. Ideally, you’ll have people with combinations of these skills. It is important to have business and IT people working in the COE. The types of skills needed are:
Technical skills—These should include a mastery of the tools in the BI portfolio to access and analyze business data; ability to mentor others in the use of these tools, particularly self-service BI; being able to navigate data models, data relationships, and business metric definitions; understanding how the DW and BI systems work; and the ability to know how to match technology to the analytical needs of the business.
Although it is assumed all the technical skills will be provided by IT personnel, business “power users” are well suited for fulfilling that role in the BI COE. They are far more technical than the average business person, but they also understand the needs of a business group far more than the typical IT person. They are usually better at linking BI functionality to how it can be used in business processes and then explaining that to their business peers.
The BI development team will provide technical expertise to the BI COE, but how it is organized will determine the extent of that involvement. BI development members’ participation can range from key members, such as a BI architect or senior BI developer being COE advisors, to the other end of the spectrum, where the COE includes the entire BI development team.
Business skills—Some members of the team must understand what data the business groups, such as HR or finance, need and how they will use it in their jobs. They should be able to work with business BI consumers to determine their requirements and priorities. This may require a level of comfort in working with executives to understand their strategic goals, requirements, and priorities. They also need to be able to translate business requirements from business to technical terms for the BI development team.
Either business systems analysts from IT or business (finance, operational, or other) analysts from various business groups are the best candidates to fulfill this role. Both types of analysts spend their time understanding the business and translating their needs into an analytical context. Typically, these people are already assigned to support specific business groups or processes that enable the COE to select individuals to represent the enterprise via cross-functional collaboration.
Analytic skills—People skilled with analytics are becoming more essential in every enterprise, and especially in the BI COE. These people need to understand business issues and the data required to address those issues; work with IT to determine specific data needed; identify what data is available and its quality; be able to discern patterns, relationships, trends, and anomalies in data; assist in creating analytical processes; and assist business people in using those analytical processes. These people need to be proficient in at least some set of tools in the BI portfolio. The analytical role bridges both the technical and business roles already described, but with the analytical person being able to leverage their skills to create and apply analytical applications to business processes.
The demands of Big Data have created the increased need for a much more sophisticated analytical role: the data scientist. In order to create Big Data analytical applications and predictive models, they need to have skills in statistics, data mining, data aggregation, and potentially some programming skills. To develop the model, they also need to understand their business and industry. It also helps if they understand how to apply customer behavior and economic models.
Although Big Data means “a lot of data,” the reality is, in many cases, that data is incomplete and inconsistent. Developing analytical models means dealing with dirty data and gaps. Many “numbers” people have trouble dealing with these conditions.
Data scientists need to have expertise in statistics, economics, and business, in addition to the basic programming skills. Although the ranks of the software engineers and IT is an excellent place to start looking for people with these skills, it helps to broaden the search to reach to people with mathematics, actuarial, statistical, economics, engineering, or business backgrounds.
Organization and funding
Someone has to “own” the BI COE, but its success depends on its ability to guide BI in the entire enterprise. In many cases, it makes sense for a BI COE to be organized under the CIO. Ideally, IT already has a strategic role in the company, paving the way for the COE to be accepted and therefore effective in its role.
Alternatively, it could be located in a particular business group if that business drives the direction of the enterprise. For example, a retail company might be very marketing-focused, and that’s where their most crucial data resides. In many other enterprises, finance may be the business driver to cross-functional collaboration, so the CFO may sponsor and “host” the COE. If a BI COE is located in a business group, it’s essential that internal politics allow the center to represent the whole enterprise.
Because the BI COE can be a virtual group, it can overlap with other centers of excellence, such as the DI COE, also discussed in this chapter. As virtual groups, they can share resources and shift priorities more easily.
Someone also has to fund the COE, which can be complicated because it is designed to benefit many different cost centers. If it’s necessary to have its costs covered, its owner, which we’ll say is the IT group for this example, could cross-charge each group for their use of the center. This provides an opportunity to demonstrate the economic value of the COE, but on the down side, it can discourage groups on a tight budget from using it.
Another approach is to cross-charge other groups on a subscription basis. This way, groups that have made their annual payment aren’t discouraged from making good use of the COE. The more use it gets, the more effective it can be, and the more everyone’s investment grows.
Many enterprises use a shared funding approach rather than cross-charging. Using this method, the enterprise executives working with various business groups will agree on the funding level for the BI COE, similar to the way they fund the IT organization.
A BI COE that is funded solely by the CIO who carves out a portion of the IT budget to support the COE raises a red flag in regards to sustainability. Without an explicit commitment by business groups to fund the BI COE, their buy-in is in question and internal politics often hinder BI COE success.
Data Integration Center of Excellence
The data integration COE (DI COE) is an enterprise team driving the design and implementation of data integration to enable enterprise-wide analytics. Its goals:
• Develop data integration standards, best practices, and reusable processes spanning enterprise-wide integration uses within a data integration framework (DIF) (see Chapter 5)
• Improve productivity in the design, development, deployment, and use of data integration
• Increase business value and ROI from the investments of time, resources, and capital in data integration
In contrast to the BI COE, this team is composed primarily of IT personal who specialize in data integration. There’s a misconception that the DI COE only involves loading the data warehouse; in actuality, its scope encompasses all integration processing in business applications, business processes, MDM, BI applications, DW, and inter-application processes. In addition, its scope includes data gathering, exchange, and integration with:
• External stakeholders—partners, supplier, customers, partners
• Big Data—social media, web analytics, machine data
• Unstructured data
The DI COE pools the skills and expertise of people doing data integration work in either a single organization or a virtual team. This chapter covers alternative roles and responsibilities later.
Although this organizational structure is often associated with larger enterprises, it is just as critical for small and midsize enterprises. Smaller organizations may not use the term COE because it seems too pretentious or they don’t have many integration people, but the DI COE goals are just as applicable to them as they are to a larger firm.
As with other people-oriented processes developed for BI, it is best to start small, learn from successes and mistakes, and expand to an enterprise-wide solution iteratively as the integration maturity level rises within an enterprise.
Why Businesses Need DI COEs
Several factors contribute to the need for a DI COE. The reasons generally have to do with the project-oriented approach toward integration that creates silos, and peoples’ tendencies to stick with what they know, assuming it has worked in the past, not the limitations of technology.
Data Sources and Silos Exploding
Enterprises are experiencing an accelerating deluge of data volumes, varieties, and velocities, creating an expanding demand for data integration. Many people point out that the world’s data is doubling every year, but whether that is a real statistic or an “urban legend” isn’t clear. What is clear is that the problem is not just the explosion of data. It’s also the fact that businesses create and expand application, data, and reporting silos—further exacerbating the integration challenges.
Myriad Integration Approaches Make the Problem Worse
Each business group uses its own applications and processes to support its unique needs. These applications, which may be purchased or custom-built, include such systems as ERP; customer relationship management (CRM); supply chain management (SCM); budgeting, planning, and forecasting; sales-force automation; and many more that may be industry-specific or more narrowly supporting a particular business process.
The implementation, customization, and expansion of these applications is done on a project-by-project basis. Inter-application data exchange and integration are also done in projects that are tactical and stand-alone. Each of these integration projects starts with a clean slate:
• Gathering requirements
• Determine if and what type of integration software is needed
• If yes, select integration software
• Acquiring skilled resources
• Establishing processes, procedures, and standards
• Developing the integration application
• Deploying the integration application
Without a data integration framework, these projects are destined to “reinvent the wheel” each time in terms of defining an architecture, selecting integration software, and developing integration processes. There are myriad different types of integration tools and techniques that provide overlapping data integration processes:
• Business application developers will typically select tools and techniques in the enterprise application integration (EAI) software category as web services in an enterprise service bus (ESB), middleware, or service-oriented architecture (SOA).
• DW developers generally use extract, transform and load (ETL); or extract, load and transform (ELT) tools; and data quality tools, if needed.
• BI developers will use connectivity capabilities built into their BI tools; manually create custom code; or, use enterprise information integration (EII) or data virtualization software
• MDM developers will use MDM applications or use a combination of ETL and data quality tools
The tendency for each of these IT specialists to create their own architectures and select different integration software to essentially perform the same data integration processes results in integration silos. From an enterprise perspective, these silos increase costs, require more resources (with skills in short supply), and reduce productivity. As each silo is established, it creates yet another application that needs to be integrated and makes the problem worse. These silos will create an integration backlog, making it take longer for IT to enable the availability of integrated data needed for analytics.
Silos are Expensive
Each group may be doing what seems right for them, but their parochial approach can be wrong for the enterprise as a whole. Data silos create another version of data instead of actually integrating it. Silos drain company finances in several ways:
• Expense of creating and maintaining various silos
• Waste of overlapping and redundant projects
• Time spent reviewing and reconciling conflicting numbers from the various data silos
• Loss of business confidence in the numbers and their transparency
Even more wasteful and expensive data silos are created when companies undertake projects such as data migrations to support application consolidation. These can be due to corporate mergers, ERP instance consolidations, and master data consolidations (product and customer reference data). Although these are data integration projects, they are treated as stand-alone, one-off projects—further fragmenting already constrained resources and budgets.
When Data Integration Problems become Hard to Ignore
Many companies are blind to their data integration problems. All they see are their investments in ERP, CRM, SCM, and DW systems, and the terabytes of data stored in them. Smug with the knowledge that they have all the data that the business needs, they’re not even aware of the data silos surrounding them.
At some point, they start noticing inconsistent data, which is a symptom of a data integration problem. However, because they don’t understand the cause, they focus on relieving the symptoms with quick-hit solutions. For example, they may try to consolidate BI tools. While this may be a worthy goal unto itself, using a single BI tool will do nothing about the fact that the underlying data is coming from many disparate systems (ERP, CRM, SCM, data warehouses, data marts, etc.). The business is not getting different numbers because it is using different tools, but rather because each tool is associated with a different database where the data had been transformed differently than the other databases. The BI tool used is the tip of the iceberg; the data integration issue is what is below the waterline.
Building a Business Case for a DI COE
You now understand how issues with people, politics, and process have established integration silos throughout an enterprise that are costly, inefficient, and inhibit the use of data as an enterprise asset. How do you stop the madness? The key deliverables to breaking the silos and establishing an enterprise-wide integration initiative are:
1. Create the business case
2. Enlist sponsorship
3. Establish an integration investment portfolio
4. Create DI COE
An enterprise needs to create a DI COE to enable enterprise-wide integration. Too often, people assume that a COE has to be a centralized group within a company controlling and staffing all integration projects. Although that is one model that can be used, there are other alternatives that can be implemented based on the circumstances within an enterprise.
Build the Business Case
You have to establish the business and technical case for an enterprise-wide integration approach. It should clearly explain that the project-by-project, start-from-scratch approach is too costly, too inefficient from time and resource perspectives, and too ineffective, because each project is reinventing the wheel.
However, the justification isn’t just that a DI COE will save money and time in the long-run; more importantly, it’s that a DI COE is the key ingredient to truly implementing the consistent, comprehensive, and current data needed by the business. The business value of the integrated data is the primary selling point, with cost savings and efficiency secondary.
The business case needs to answer these questions:
• What business problems or opportunities are being addressed by the DI COE?
• Who will use it?
• What are the anticipated business benefits?
Enlist Sponsorship
The goal of the justification is to enlist sponsorship, both from your business and IT leadership. Although many initiatives start from the bottom in order to establish success and credibility, at some point, to truly achieve an enterprise-wide approach, your CFO and CIO have to become sponsors. The CFO is necessary both from a budget and a data governance (ownership) perspective, and your CIO has to commit the IT organization to approaching integration from an enterprise-wide perspective, not just on a tactical project basis.
The business is investing in projects and applications that require data integration, but historically, these are disjointed projects that have created data silos scattered across the enterprise. Someone needs to recognize that this is a systemic problem that needs to be addressed in order to better use IT investments and, more importantly, get the information needed by the business to manage the enterprise. The data integration problem needs an out-of-the-box approach that looks at the problem in a holistic manner.
The task of enlisting sponsorship involves three types of people or groups:
• The Evangelist—The first step in this journey is getting help from someone who sees the forest, not just the trees—someone who recognizes the need for change. This is usually an IT person involved in existing data integration efforts who sees the redundancy in data integration projects and understands the business benefit of eliminating it. This person becomes the data integration evangelist who preaches that there is a problem and that something must be done about it. Not having significant budgetary authority and being located deep in the IT organization means that the evangelist generally cannot single-handedly change the momentum of the company. Often, this IT person is someone with architectural responsibility.
• The Champion—The next person in the chain to keep the fire going is a champion—someone higher in the organization visible to and respected by either the CIO or CFO. The evangelist enlists the champion to continue selling the vision further up the organizational hierarchy. The champion also may lack significant budgetary authority, but is often the person who creates the business and IT case for budget submissions to the CIO or CFO. The champion needs to justify the solid business case of establishing data integration as an infrastructure program, just as e-mail and networks are treated in most enterprises today.
• Sponsors—Finally, the crucial link to success is getting the sponsors, with both the CIO and CFO signed on, to treat data integration as an investment portfolio. There is a wide range of organizational approaches to this—from actually having a single data integration budget to a more realistic approach of budgetary reviews of all projects with data integration components. The budgetary reviews would eliminate redundant or conflicting efforts and, possibly, combine the data integration of multiple projects to enable more expansive and complete coverage.
Establish a Data Integration Investment Portfolio
With multiple projects vying for the same budget, you need to evaluate and manage the data integration components by creating a data integration strategy and architecture, i.e., data integration framework (DIF). (The DIF is covered in detail in Chapter 5.) Although the DIF is a prerequisite to guiding the data integration program, it is crucial to make sure that you have funding.
The Portfolio Approach
The best way to ensure funding for the right projects is a data integration investment portfolio. With this approach, you examine each project to determine if a data integration component exists. If so, you position it within your DIF to classify what data integration functionality is being performed. Then you examine that component in the context of all the other data integration components of other projects. This review will determine if there are any overlapping or complementary aspects between projects, thereby eliminating redundant expenditures and leveraging work across projects.
Looking at data integration efforts from an enterprise perspective often enhances these efforts because you are no longer sub-optimizing the data integration efforts of each project to fit within that individual project’s budget and resource constraints. You can undertake a more robust effort when the enterprise is being used as a focus. In addition, you can modify some aspects that may have hindered the effort put into other projects in order to reach a better solution.
Money talks. You absolutely must have financial incentives such as budgeting to get buy-in for data integration projects. Enterprise-wide data integration, while appealing to many in the IT industry, is just an esoteric concept that will be overshadowed by short-term and parochial projects unless backed up by budgetary allocation and review.
Establishing the Portfolio
Creating the integration portfolio involves assessing where you are, determining where you want to be, and how you are going to close the gap.
First, you need to take an inventory of what data integration is already occurring in an enterprise. Existing IT projects may be under these labels:
• Performance management
• Customer data integration (CDI)
• Big Data
• Predictive analytics
• MDM
• Data warehousing
• BI
In addition to these IT labeled projects, it is a safe assumption that most, if not all, key business initiatives have integration components to gather the data and generate the business metrics needed.
The technologies that may be used include:
• ETL or ELT
• Data virtualization (formerly enterprise information integration (EII)
• Web or data services—evolved form of EAI
• Data quality
• MDM
• Data shadow systems or spreadmarts that are used extensively across enterprises using SQL scripts, ODBC/JDBC, Microsoft Access, and Microsoft Excel to “integrate” data
When assessing this inventory, examine what has the most significant impact on the enterprise, either in terms of business benefit or IT cost. You are not going to be able to solve all the problems at first, so concentrate on the areas that provide the best rate of return for your efforts and make the biggest impact.
After taking inventory, you need to determine what you want your integration portfolio to be in the future. You cannot succeed unless you have an agreed upon goal, i.e., architecture, that supports the information needs of the business. The integration portfolio needs to be an integral part of your information architecture and systems.
Finally, you need to create an integration road-map to move from your existing data silos to your desired information architecture. This will involve integration initiatives providing new capabilities along with projects that renovate or replace existing silos.
Organizational Models for Building the DI COE
There are several approaches to organizing the role and responsibilities of a DI COE. The approach you take depends on your enterprise’s people, politics, and processes. Too often, people assume that a DI COE has to be a centralized group within a company controlling and staffing all integration projects. Although that is one model that can be used, there are other alternatives that may be more advantageous based on the circumstances within an enterprise. Another incorrect assumption people make is that only large corporations need DI COEs.
The four organizational models most often used to implement DI COEs are:
• Best practices
• Technology standards
• Shared services
• Centralized services
Although the DI COE organizational choices above are listed in an increasing level of control, scope, and size, it is neither necessary nor recommended for an enterprise to step up the DI COE organization ladder. The organizational approach that best fits an enterprise depends on its unique situation, rather than some esoteric, one-size-fits-all model. These four models are explained in more detail in the following sections.
Sharing Best Practices Organizational Model
The simplest form of DI COE organization is a virtual team that documents and shares best practices among data integration projects. This allows each project to leverage past work without reinventing the wheel. Like all organizational models, it has pros and cons.
The pros of this approach are that it saves time, reduces costs, and enables each project team to concentrate on expanding the overall enterprise’s integration expertise, thereby contributing new or improved practices based on what they’ve learned. In this manner, enterprise integration expertise continues to grow project-by-project rather than being lost as each project is completed and resources inevitably scatter.
In addition, there is a considerable amount of industry knowledge gained from successful and unsuccessful integration projects, which has formed a set of data integration best practices. Tapping into that pool of best practices should improve an enterprise’s integration efforts and help avoid many of the learning mistakes everyone new encounters. Just to set expectations: there is no single recipe that applies to all situations, but rather a data integration cookbook of best practices that an enterprise needs to pick which recipes apply to them.
Very few corporate cultures lend themselves to this approach, however. The people on the virtual team are often doing this work on a voluntary basis in addition to their primary jobs. In difficult economic times, most people are already working many more hours than they did in the past and may not have the opportunity to invest the time that is needed. Virtual team members are often drawn back into their primary jobs as priorities change or crises emerge.
Enterprises must recognize the level of efforts that each contributor needs to make for the common good. If the investment in people is recognized and encouraged—by incorporating as objectives in employee reviews and substituting this work for lower priority work—then this approach can be very effective. If, however, this activity is just thrown on top of everyone’s already too-large workload, then this model will not be effective.
Another issue that can arise with this model is that each project team generally feels empowered to do their project their own way and figures they will do it better than others. New project teams may not have the bandwidth to study and use other projects’ best practices. In fact, they might not even know how they apply. Although this is a very appealing approach from a practical standpoint, it often has little effect on enterprise-wide data integration efforts.
Common Technology Organizational Model
The second DI COE organizational approach is to have a virtual team that not only publishes best practices, but also determines and recommends a common technology platform. The benefit of this approach is that it goes beyond sharing ideas by creating a technology platform that each independent integration project can tap into. This approach avoids the costly and time-consuming tool evaluation and selection process; avoids the cost of redundant or overlapping software and supporting infrastructure; and, if the integration standard is followed, provides the potential for a common pool of expertise in the enterprise that might be enlisted to work on various projects.
Standardizing on integration technology enables the integration projects to concentrate on creating business value rather than continually tying up resources in the evaluation, selection, purchase, training, development, and deployment of multiple integration technologies.
There are some concerns with this approach, however:
• First, how will the DI COE team make its selection without using various projects for requirements input?
• Second, the team often does not have the budget or resources to adequately pick a technology platform. This approach means that an additional budget item must be allocated to a pure IT technology project—often a difficult sell when budgets are tight.
• Third, while the team is examining the technology options, other projects are moving forward without the guidance that it can provide. In other words, more data silos may be built just as the DI COE looks at ways to prevent them.
• Finally, what are the real incentives for each project to adopt the common technology platform? Many corporate cultures will allow new projects to pick another platform anyway, because these projects must still deliver their objectives in addition to picking up the requirements of the technology platform. Many projects will justify moving forward on their own because of their limited budget, resources, and expertise.
Shared Services Organizational Model
The third organizational approach is to create a DI COE where a group of dedicated people develop the data integration components of individual projects. In this services model, the DI COE operates as a subcontractor within the individual projects. It determines the strategy, designs the architecture, and selects the technology platform as prerequisites to its development efforts. The shared services model allows companies to leverage work and resources across projects to develop an enterprise-wide data integration program. This organization enables the enterprise to develop deep data integration skills because the DI COE specializes in that area. This approach works extremely well, especially when reinforced by strong business and IT sponsorship along with supporting budgetary investments.
The individual integration project teams are still responsible for the development activities and deliverables; however, the DI COE assigns resources from their common pool of talent to work on those projects.
This is a terrific approach to developing and nurturing integration talent and expertise (from an employee perspective) and a very productive way to leverage individual skills, thus reducing costs (from an enterprise perspective).
This approach works extremely well, especially when reinforced by strong business and IT sponsorship, along with supporting budgetary investments. The main caution is that decentralized companies may be reluctant to use the shared services approach. However, with strong financial incentives and recognition that many IT services already operate in this manner, initial resistance can be overcome.
Centralized Services Organizational Model
The final organizational model, which requires the highest levels of skills and experience, is a completely centralized DI COE operation. Data integration is elevated as a primary program; projects are independent of the other IT and business application projects. The data integration projects are executed to build a data infrastructure supporting other business projects. All data integration components are pulled out of projects and placed into the data integration program. Just as with the shared services model, the DI COE determines the strategy, designs the architecture, and selects the technology platform as prerequisites to its development efforts. However, unlike the previous model, the DI COE operates the entire data integration project.
This is an extremely effective model if the corporate culture supports centralized development and control. However, in a decentralized culture, the shared services model will have a better chance of success.
DI COE Deliverables
The DI COE facilitates the design, implementation, and support of a data integration framework (DIF) that enables the integration of comprehensive, clean, consistent, and current data for the business in a cost- and resource-effective manner. The best practices to follow during this process include:
• Create DIF (see Chapter 5)
• Classify data integration needs by category and functionality
• Document DI use cases
• Design data integration architecture
• Create common and reusable processes (see Chapter 11)
• Design common standards and templates
• Design common data quality processes and standards
• Design common communications, “logging,” and alerts
• Design DI monitoring processes
• Design BI performance tuning processes
• Leverage integration software (see Chapter 13)
• Use a common set of data integration software
• Use DI software rather than manually coding
• Expand integration knowledge and skills (see Chapter 17)
• Determine skills, experience, and knowledge
• Obtain fundamentals training
• Obtain tools training
Skills Needed
There is an ongoing shortage of experienced data integration professionals. The skills gap is getting larger with the data deluge and growing integration backlog. Most people gain their data integration experience while manually building custom-coded processes; however, it’s limited. These manually coded projects have not provided an opportunity to learn data integration architectures, best practices, standards, and how to use sophisticated data integration software, never mind learn data integration fundamentals.
A skilled data integration specialist needs expertise in the following:
• Data modeling
• Database design
• Data profiling
• ETL, ELT, or data integration suites
• SQL
• Web and data services
• MDM
• DW design
• BI and analytics design
• Data and technical architectures
• Software development methodology
A good data integration specialist will have extensive data and software skills across a variety of integration uses. But the IT industry is in a quandary because as long as manually coded integration processes dominate projects then data integration specialists will not learn how to implement more sophisticated integration software that will break through the integration backlog. (Chapter 12)
How do we close the skills gap?
• Invest in training and ongoing education on integration fundamentals and tools
• Accept lower productivity in initial integration projects as an investment in gaining skills for the long term
• Create an on-the-job mentoring program with senior integration specialists as the mentors. (This means the senior staff time shifts from development to mentoring—another investment in the future with the associated short-term drop in productivity.)
• Create a DIF with its architecture, best practices, and standards as a way for integration specialists skills to grow over time
Enabling a Data-Driven Enterprise
Organizationally, the best approach to implement a sustained enterprise data management strategy is to establish a data governance program along with the BI and data integration COEs. The three-pillar approach to data management is a best practice for enterprises of all sizes, with the differences being that larger organizations will actually have three organizations, while smaller enterprises will have three processes.
The benefits of the data management trilogy:
• Leverages analytics to create business value
• Improves the productivity of both business and IT people
• In the long-term, it is the cost-effective approach to becoming a data-driven enterprise
The three organizations in the data management trilogy, as depicted in Figure 19.2, support each other’s goals and objectives. The data governance program will work with the DI COE to gather the data definitions from the SORs and with the BI COE to define the data and metrics used in BI applications. The data governance program will then work with the COEs to implement and enforce (or strongly encourage) the use of these definitions in the enterprise’s data integration and BI applications. There is an overlap in skills across the trilogy and, except possibly in the largest enterprises, there is also a sharing of people and resource commitments.
An organizational view of the trilogy is illustrated in Figure 19.3. The ideal or long-term goal is to have executive sponsorship support an enterprise-wide data-governance board that then drives both the BI COE and DI COE. The reality is that the most successful data management programs have started small and grown iteratively based on each tactical success story. Data management is not a goal unto itself, but rather an enabler to better analytics and better decision-making. It is merely an esoteric concept until it can be proved to provide business value.
The data governance board should leverage the COEs as the intermediaries and specialists who work directly with the SMEs for SORs and business intelligence applications. Data management involves very detailed and time-consuming tasks that are best delegated to the COEs. The data governance board needs to sift through that detail, determine what needs to be governed and what is not worth the investment, work with the executives and representatives of the business groups to prioritize these needs, work with the business groups on how to enforce and encourage data governance use in analytics, establish ongoing communication across the enterprise, market the efforts, and keep refreshing the program as the business changes and evolves. The most important contribution the trilogy makes is managing the politics that data management always evokes.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.