Chapter 18
Project Management
Abstract
Good project management helps ensure that projects are completed as planned, within budget, with good quality and, most important, that the completed project meets expectations. In larger organizations, establishing a business intelligence (BI) program helps ensure that all projects follow the same enterprise-wide BI vision and reduces overlap and redundancy. Individual projects should begin with an assessment, which puts everyone on the same page by answering questions about scope, schedule, resources, budget, products, etc. The project management approach should begin with defining the work breakdown structure (WBS) prior to developing any project schedules. Using the WBS is a project management technique that incrementally decomposes a project into smaller components based on its deliverables. It is visually represented as a hierarchical or tree structure. Project managers often debate over using waterfall or agile project methodologies for BI projects; we recommend a hybrid approach using the incremental development methodology and some variation of iterative development such as prototyping, rapid application development, or agile. The project phases are scope and plan; analysis and definition; architect and design; build, test, and refine; implement; and deploy and roll out. To create the project schedule, decompose the BI project WBS into greater levels of detail until it is sufficient to schedule tasks, assign resources, and track progress.
Keywords
Agile; BI assessment; BI program; BI project management; Project manager; Project methodology; Project schedule; Waterfall; Work breakdown structures (WBS)• Establishing a BI program
• BI assessments
• Work breakdown structure (WBS)
• Project methodologies
• Project phases
• Project schedule
The Role of Project Management
Without good project management, IT projects are prone to be late, over budget, low quality and, most important, they fail to meet expectations. The failures occur primarily because the project emphasis is on technology rather than the business need, and because there is poor communication between IT and the business.
Project management is critical for a business intelligence (BI) project’s success, yet teams often fail to give it proper attention; they tend to behave as if project management is not their problem. But actually, it’s everyone’s problem.
The role of the project manager is not trivial. This function carries major responsibility, but too often the person who becomes the project manager does not have the appropriate background or experience. In some cases, the person selected is a project manager without any BI experience; in others, the person is technically proficient but has no project management experience. The problem is especially acute if it is the first time the enterprise has developed a BI solution—a risky scenario because then there are no in-house experts to help.
BI project managers must not only manage the project team but also serve as primary advocates for the business users, i.e., BI customers, and keep active communications with all stakeholders. BI project managers must handle all the people, process, and political issues that arise throughout the project. They need to introduce and ensure that solid project management processes and best practices are followed by all members of the project team.
In this chapter, we will discuss the following project management subjects:
• BI program management
• Work breakdown structures (WBS)
• Project methodologies
• Project phases
Each of these topics is meant to enable a BI project manager to better plan and manage a BI project. There are cross references to the other sections of this book that discuss the details of the BI project tasks.
Establishing a BI Program
Enterprises are managed strategically by their senior management and operationally by business groups or departments such as marketing, sales, finance, and human resources. This division of labor enables specialization and fosters improved productivity. Although the individual business groups may have a considerable amount of autonomy in making tactical decisions, the enterprise must have an overall strategic vision and a set of guidelines under which these groups must operate.
Senior management will lay out a strategic vision and work with the various business groups to develop a plan—typically done at least once a year—to implement that decision. Those plans involve multiple business initiatives that are needed by the enterprise to achieve its strategic vision (see Figure 18.1). Typically, these business initiatives require cross-functional collaboration. Business initiative plans include business objectives and deliverables along with the schedules, budgets, and resources needed. An enterprise’s strategic vision and business initiatives are the glue that binds its business groups together.
Business groups within the enterprise need data and analytics to run day-to-day operations, so these groups tend to tactically drive BI projects. Although it is certainly understandable that business groups feel the urgency to initiate BI projects to get the data and analytics they need, tactical BI projects tend to create data, application, and technology silos, lost business and IT productivity, redundant or overlapping work and resources, and, quite simply, the inability to achieve success with BI on an enterprise-wide basis.
Just as an enterprise needs a strategic vision with an overall program of business initiatives to achieve it, BI needs a strategic vision within its overall BI program. The enterprise’s strategic vision will evolve over time and along with its products or services; so too will its data analytics needs. BI is not a “one and done” project, but an ongoing program that supports business needs.
Business Strategy Drives the BI Program
The chief information officer (CIO) needs to ensure that an overall BI program is established for the enterprise. When you consider that data is an enterprise asset, and analytics the key to leveraging that data, BI efforts can no longer be scattered and uncoordinated throughout an enterprise no matter what its size.
The BI program must support the enterprise’s business strategy (see Figure 18.2). In this context, it is clear that the BI program needs to focus on business enablement rather than on technology, which is just the opposite of the orientation of many BI projects.
Historically, when BI projects are driven tactically by individual business groups, each project determines its deliverables and priorities solely in the context of that group. This “bottom-up” approach ignores other groups’ priorities and inhibits investments in scope that may be needed in other groups. These missed or shortchanged investments often include important infrastructure or enterprise-wide data management work, such as master data management (MDM).
A BI program needs to use a “top-down” approach that leverages the existing business strategy and priorities. There is no reason for a BI program to independently determine the strategic vision of the enterprise. Keep in mind that using a “top-down” approach to set overall direction for the program does not imply that the program will be managed centrally. The choice of central or decentralized management of the BI program will be based on the enterprise’s organization and political culture.
Business Sponsors for the BI Program
Although the initial establishment of a BI program may be driven by the CIO, a key element of success will be finding a strategic business sponsor for that program. Since the primary objective of BI is enabling business rather than implementing technology, the business group needs to sign up to drive this program.
One of the common BI program business sponsors is the CFO. CFOs are the natural candidates because they are responsible for the accuracy and transparency of enterprise information, at least in regard to financial information, for enterprise stakeholders. They are legally responsible for the data that is published outside the enterprise and are subject to regulatory compliance and potential audits regarding the data. This responsibility means that they have a cross-organizational perspective that many of the sponsors may not have. In addition, they are responsible for driving enterprise-wide cost management, so they have keen interest in ensuring that the BI investments are managed appropriately. The CFO’s responsibility for driving the overall enterprise’s budgetary process enables it to be integrated with the creation and prioritization of the BI program.
Although the CFO is a natural candidate to be the BI program sponsor, there are other senior managers who may take the lead due to the industry of the enterprise or simply because of the political structures in that enterprise. In customer-oriented industries such as financial services, retail, and consumer goods, the senior vice president (SVP) of marketing or sales may be the BI program sponsor. In manufacturing, energy, or the utilities industries, the SVP of operations may take the lead. In all cases, a senior-level business executive with significant budgetary authority is typically the primary business sponsor. Although there may actually be more than one business sponsor, it is imperative from a political perspective that a primary BI business sponsor be designated to ensure that the program moves forward.
Determining the Portfolio of BI Projects
After identifying the business strategy and business initiatives that the BI program will support, IT can then work with the business sponsors to determine what portfolio of BI projects will be required to support those initiatives.
The CIO typically drives the selection of projects for the BI program. The CIO needs to be the IT advocate for the BI program and work with peers in senior management to ensure that this is an enterprise-wide effort. The CIO, of all corporate executives, needs to be concerned with both the “what” and “how” of the program—what are its goals and how is it going to accomplish them? The CIO is in the unique position to understand why long-term success—measured in business value and cost effectiveness—comes from developing an overall architecture to guide the short-term BI solutions needed by the business.
The BI portfolio needs to focus on business needs first, not IT needs. Getting the business the information it needs is priority; choosing software tools and other IT-centric issues are important only in terms of how they help achieve the business goals. IT groups too often create their list of projects and priorities based on a logical and mainly technical viewpoint. Their agenda is driven by upgrading hardware and software to take advantage of new capabilities that vendors have built into their products, along with support and maintenance. None of this necessarily matters to the business. Getting information matters to the business. Chances are that every business initiative in the company needs more data, so every project should be directly tied to a priority initiative. (See Figure 18.3.)
If the business needs are not priority, you really cannot expect a high level of business group participation. The business groups are focused on their priority business initiatives, so they cannot commit to a BI program that does not address these needs. And, as we discussed in Chapter 16, if the business group’s information needs are not addressed, they tend to create data shadow systems.
Sponsorship, Governance, and Business Participation
The BI program does not work in a vacuum. It needs to receive direction and influence from certain areas outside the BI team. Part of the project management responsibility is ensuring that three areas, which are particularly important in the BI effort, are in place. These areas are sponsorship, governance, and business group participation. (See Figure 18.4.)
Sponsorship
A BI program needs business sponsorship to justify the need for it, set its business priorities, and commit the resources necessary to design, develop, deploy, and use the resulting BI deliverables. The business sponsors need to establish an ongoing prioritization process that represents the enterprise stakeholders, because a BI program and its capabilities evolve over time. Enterprise-wide buy-in of the BI program’s priorities is essential; otherwise, business groups feel disenfranchised and will create their own data and analytical silos or data shadow systems.
Depending on the enterprise’s organization and budgetary processes, the funding may come directly from the business sponsor’s budget, carved out from the IT budget, or may be a pooling of budgets from various business groups. Regardless of where the funding originates, the business sponsors will be responsible for managing the BI program funding. Managing the BI program funding from an enterprise perspective enables a business sponsor and the CIO to do a more effective and efficient job of allocating funding for BI projects than if the projects were done on a piecemeal basis by business groups. This method of financing encourages foundational work and infrastructure investments, because it is shared across the enterprise. In addition, it reduces the likelihood of overlapping or redundant investments because of the enterprise visibility.
Business sponsorship is nearly always discussed in the context of funding. Although this is critical, there are two other areas that are just as essential from a business perspective: governance and business participation.
Program Governance
The BI program needs an overall governance model to ensure that it meets its goals and objectives for all its stakeholders. The governance model defines the accountabilities, responsibilities, roles, and levels of involvement by the stakeholders in the BI program. A common mistake is to concentrate exclusively on the roles and responsibilities of the BI project team rather than the BI program. If the governance model is project-focused, then the program typically does not get the level of business involvement or commitment that is necessary for success. The governance model also involves creating the processes necessary to gather and adjust requirements and priorities based on stakeholder involvement and feedback. These processes need to be transparent and communicated to all stakeholders.
Business Participation
The business needs to provide sponsorship (funding) and governance but it also needs to be actively involved with the BI program and individual BI projects. This participation may involve the following tasks, although they may not apply in every case:
• Providing input on requirements, refinement, validation, and prioritization
• Software evaluation
• Validation and refinement of specifications using techniques such as storyboarding, wireframes, and mock-ups
• Providing iterative feedback on BI design such as dashboards and visualizations
• Working with prototypes or proofs-of-concept (POCs)
• Agile, prototyping, rapid application development (RAD), or other team-developing BI deliverables
• User training
• User acceptance testing (UAT), systems testing, and parallel testing, if applicable
If the business has the personnel resources that can participate as active members of the team, then the probability of success and quality of the deliverables is significantly enhanced. The BI project manager needs to develop schedules and resource plans that include both IT and business. Business participants need to understand what is expected of them, the level of effort required, and when their involvement is planned to happen.
Typically, the most valuable business participants are already overloaded with work and need their manager’s commitment to free up their time to work on the BI project. It is the task of the IT group to determine realistically how much the business group really can and will contribute. Sometimes the business group says, with all good intentions, that it will be involved, but then does not have the resources to offer. The BI manager needs to be ever vigilant in monitoring the expected versus actual participation. If there is a variance, take corrective action. Avoid the common pitfall of tracking only IT resources while ignoring business participation. Too many BI projects have been derailed because lack of business participation led to schedule slip and adversely affected deliverables, completely surprising the stakeholders.
Build the BI Program Like a Software Firm
BI is not a one-time effort; you are in it for the long haul. So, the BI program needs to be built iteratively and allowed to grow incrementally. This approach accelerates business value, enables business agility, and ultimately lowers the total cost of ownership. Successful BI programs incrementally add data and analytical capabilities, expand the number of business processes and business BI consumers, and increase the supporting infrastructure and software.
There is often pressure to lay out a very high-level BI program plan quickly and almost immediately jump into BI development. This happens because there is an attitude that getting into hands-on development as soon as possible is desirable, and that people will figure out the details of the BI program as they go along. This approach turns every project into a standalone project needing to define its own infrastructure, data, and analytics capabilities. These standalone projects typically under-invest in infrastructure, short-change architecture, and fail to leverage potentially interconnected or shared work across projects. The BI program needs to plan not just the deliverables, but the “how” of those deliverables. Examining the program holistically enables you to plan the projects, building the long-term BI framework (architecture, data, integration, analytics and skills) incrementally and iteratively in a deliberate manner.
Think of the way a software company builds and updates its product. Just as with a BI program, a software product is going to evolve and expand over time. The software firm will develop and release an initial product V1.0, and follow that with incremental updates that add features and fix bugs. They take customer feedback into account as they decide what new functionality needs to be added. As they move forward, they will announce their product roadmap and software release schedule, laying out the growth in capability and features. Enterprises, therefore, should plan and manage their BI programs as if they were software products: design, schedule, and communicate a program roadmap and release schedule.
Figure 18.5 illustrates a sample template for a BI program release schedule with the following key attributes filled in:
• Timeframe: typically the month or quarter that major project or releases are completed
• Business sponsors: the business groups and executive funding the project
• Business constituency: the business groups that will use the resulting deliverables
• Business process affected by the release, and whether BI deliverables will be embedded in those processes
• Business applications involved in data integration or analytics
• Data subjects needed for analytics, extracted or integrated
• Source systems from which data is gathered, and extraction frequencies
• Service level agreements (SLAs) may be established for such factors as data currency, data load times (occur when there are data loading time periods that are agreed upon with source applications), query performance (typically associated with specific datasets or dashboards), length of time data is available, etc.
• Applications retired, if applicable, along with retirement criteria and dependencies
• Technology required: includes hardware, software, and cloud services, along with corresponding licenses and versions
• Budget allocation, if applicable, with capital, labor, and other expenses
• Project team (IT and business)
• Participating resources outside of the full-time BI project team
Balancing a BI program
As with any project, there are always trade-offs between the resources and time available in relation to the results desired. Since the luxury of unlimited resources and time are seldom a reality, this balancing act is common. As shown in Figure 18.6, the BI project manager needs to balance along the three dimensions of results, resources, and time. But even within these dimensions, further trade-offs can occur:
• Results: BI deliverables are a combination of software and data, and as such, much of the deliverables are hidden from the business people. Because of this, a BI project manager can adjust the deliverables of that project from a functional perspective, and can even change the completeness or quality of the deliverables.
• Resources: project resources can be adjusted based on labor such as IT staff, consultants, or business people; the quality of that labor, i.e., skills and experience; the budget; and the investment in infrastructure, i.e., hardware, storage, memory, etc.
• Time: ironically, this is the one variable that is generally determined even before any other variables. It is very common for the business sponsors to assign a completion date before any planning occurs. In one common scenario, there are particular business events that require the availability of BI deliverables. In this particular case, the BI program manager uses a time-boxed approach where time is fixed in the scope of the deliverables (results) and resources are variable. The second most common scenario, and one that is very reasonable when planning out an entire BI program, is to set up specific time intervals for the BI project release, again another time-boxed approach. BI project results and resources vary; all time is fixed.
A BI project is a very complex application with many factors influencing the scope: resources, results, and schedule. Some of the other key factors affecting the scope of a BI project include:
• Complexity and variety of analytical functionality
• Complexity and number of dashboards, reports, and visualizations
• Amount of data detail
• Amount of historical data
• Data currency or latency
• Complexity and extent of data integration
• Complexity and extent of data quality
• Number, variety, complexity, velocity, and volume of data sources (internal and external)
• Number of supported business users (internal and external) and variety of analytical styles
• Privacy, security, and regulatory requirements
Each of the above factors can be modified to affect the scope. Often, changes to scope are not readily apparent until the business people actually start using the BI deliverables hands-on, and even then it may take an extended period of time. Each of these attributes, therefore, needs to be defined and documented to set business expectations and to effectively manage the BI project.
Unfortunately, BI teams may compromise or short-change other areas to meet resource and time constraints that might not be immediately evident:
• Testing: unit, UAT, system, regression, and parallel
• Documentation: requirements, architecture, data models, data integration specifications and data mapping, BI specifications, production schedules, application and system workflows
Just like the scoping factors previously discussed, these areas also need to be defined and the level of effort agreed on to effectively manage the project and avoid unexpected and potentially harmful errors or omissions in the BI deliverables.
Internal Marketing of the BI Program
An important task for the BI project manager is to create and execute a communication strategy during the life of the project. This includes scheduled status reporting, providing ongoing project information, and enabling stakeholder feedback. Besides scheduling meetings, there are many types of collaboration tools that enable all levels of communication.
It is a mistake to assume that your key players know about current developments taking place in the BI program. There is often a need to increase internal awareness and market the BI program within the company. Business results are key to getting a project staffed and implemented, so an internal awareness and marketing campaign can be essential. Even after the project is approved and budgeted, it is important to engage business sponsors, IT staff, and future program users to promote enthusiasm and involvement in the new endeavor. Otherwise, although your project might be viewed as successful, funding may be cut anyway because of the business perception that it is already “good enough.”
The benefits of the program are not always obvious. Business group members may not be “connecting the dots” between what information they ultimately need to do their jobs and the process it will take to get them that information. Someone may have to make crystal clear the business value of the investments necessary for a BI project—and this is not a one-time effort; you may need to have an ongoing plan to assess the success of the BI program by gauging the perceptions and expectations of the business people and sponsors.
Also, just because a business group’s “power user” is on board and up-to-speed on the BI program, other critical members of the business group might not be so well informed. There might be a communication gap within the business group.
Before diving into the marketing work, meet with business users and sponsors to determine the perception of the program and future expectations. Then conduct a gap analysis to see where the needs are, and formulate an internal awareness and marketing campaign to increase the likelihood of successfully moving your program forward.
The marketing campaign might consist of a series of presentations, workshops, or training sessions delivered to business sponsors, business group members who will use it (existing or desired), and the IT team. The content should cover the business benefits of the BI program and should include workshops to provide information on what is involved in BI from a business, technology, and project perspective. These workshops should help with knowledge transfer and setting the right expectations for your program.
Effective marketing methods include:
• Enterprise portal, if applicable, where business group or program information is typically posted
• Collaboration tools used to share information, solicit feedback, and communicate information among stakeholders
• Social media, such as blogs
• Case studies or testimonials (just like product vendors)
• Lunch (or breakfast) and learn sessions (in person or web conferencing)
A BI program manager’s choices will depend on organizational culture and enterprise standards. The BI program manager must monitor and manage all marketing campaigns. Privacy, security, and governmental regulations all need to be considered.
BI Assessment
Before kicking off any BI project, there are many questions to be answered. What is the scope? What is the best way to proceed? How long will it take? How much will it cost and how much do we have to spend? Who needs to be involved? What products do we need? And so on. An assessment helps answer these questions and keep a project on track. It can be done either internally or by an outside consultant.
The end result of the BI assessment is a BI roadmap or program to achieve your business and technical objectives in a manner that is as cost and resource efficient as possible. Ideally, the roadmap should include a few potential scenarios providing different capability, timeline, and cost possibilities. To develop the roadmap, you’ll need to examine the four subject areas shown in Figure 18.7 in sufficient detail to quantify and qualify the roadmap’s plans, budgets, and timeline.
Done before a project begins, the BI assessment will help flesh out the scope and design. With detailed information on feedback and priorities, you can further refine your project schedule, resource plans, and deliverables, while mitigating risk. You will use this information to assess the status of the company in its BI program, where it should be, and what its current capabilities are. These observations will help refine objectives in terms of project, people, processes, and technology.
The high-level depiction of the BI assessment is provided in Figure 18.8. As this shows, the assessment covers these three dimensions:
• Architecture: information, data, technology and product
• Requirements and priorities: business and IT
• People: organization, governance, projects, and skills
Figure 18.9 shows how to conduct your assessment in three phases: discovery, analysis, and recommendations.
Discovery Phase—Assess Current Environment
The discovery phase, shown in Figure 18.10, explores the following areas:
• Business and IT requirements
• Organization, partnerships, processes, and skills
• Data, information, technology, and product architectures
Much of the information gathering for this phase is done through interviews with IT and business people, such as business sponsors, key members of business staff who will likely use the BI environment, and potential business power users. The following list highlights the subjects to discuss with business group members:
• Desired reporting and analysis: what capabilities do they need and what data do they need to get?
• Current reporting and data shortcomings and workarounds: this information will help you with the gap analysis
• Priorities: what is most important for them in terms of the BI project and the information they need for decision-making?
• Analytical skills: what skills do they already have?
IT staff to be interviewed include potential project team members, DBAs, project management, and IT management. The IT discussions should include an overview of the existing application and reporting environment and then reviewing more details regarding the architectures, requirements, and future plans. Have the following discussions with IT:
• IT directions/plans, project methodology, product standards, application direction
• Skill level in database, reporting/BI, data modeling, applications
• Review source systems/applications
• Major business applications: overview, reporting, database/OS
• Other business applications, spreadsheets, or access databases that are relevant to cross-group reporting
• Demos of reporting applications
Background materials to review during the discovery phase may include data source systems, data models, business and technical requirements, samples of existing reports or analysis conducted that will be replaced by the BI solution, existing technical infrastructure, current IT application plans, and business initiatives driving the need for BI.
Analysis Phase—Identify the Gaps
During the analysis phase, you will review the conversations, discussions, and interviews along with the BI requirements and materials gathered in the discovery phase. The analysis involves examining the architecture, organization, and requirements in terms of where you are now and where you would like to move your BI environment. Determine the gaps between the two, validate your ability to achieve your future end state, and determine what you would need along all three of these dimensions to achieve success.
Figure 18.11 shows an overview of the analysis phase. The tasks for the analysis phase include:
• Developing a list of requirements for areas like enterprise-wide reporting, enterprise-wide shared data, and data cleansing needs
• Creating a list of current reporting capabilities
• Performing a gap analysis of what is desired, what is reasonable, and what gets added to the to-do list for BI, data integration, data cleansing, and MDM
• Developing a product vendor short list
• Developing product demos for the recommendation phase. This may involve obtaining sample data and reports from the business and asking vendors to create the demos
• Developing a high-level program plan with timeline, costs, and resources
• Developing program and project plans, including high-level timeline, cost, and resources estimates, data sources, and reporting capabilities
Recommendations Phase—Scope, Priorities, and Budget
During the recommendation phase, you will present the results of the analysis phase along with a roadmap that has a vision for the BI program. Ideally, it will also include an iterative series of projects to achieve your overall BI program objectives.
The recommendations phase will involve feedback and recommendations on the current state of your readiness in relation to your requirements and capabilities. These recommendations will involve your architecture (information, data, technology, and product) and your organization (program/project plans, resources/skills/training, and rough cost estimates.) See Figure 18.12 for an overview of the recommendations phase.
The tasks in this phase include:
• Reviewing the program roadmap with the business and IT to go over timeline/budget and the iterative implementation of the BI program
• Marketing the BI program to the business
• Reviewing the skills assessment for business and IT, discussing how it affects the timeline and product selection, and what training is then required
• Reviewing the product and technology roadmap with IT, including product shortlists and demos
Work Breakdown Structure
Typically, the project manager is either a technology-oriented person without much project management background or a project manager without much BI experience. When enterprises are new to BI, it is likely that the project manager will not have any BI or project management background.
There is a tendency in BI projects to require the project manager to develop a detailed project schedule before any work gets underway. If the BI project manager has limited or no experience with BI and/or project management, then this is likely an impossible task. This is a difficult task, even if the person has experience in project management, because of the complexity of a BI project. There are often many unknowns or high-level-only requirements at the beginning of the project, making it difficult to create a detailed project schedule. The best project management approach at the beginning of both the BI program and any BI project is to define a WBS prior to developing any project schedule.
WBS Defined
The WBS concept was developed by the U.S. Department of Defense in the early 1960s and later documented by the Project Management Institute for use outside the defense industry. WBS is a project management technique that incrementally decomposes a project into smaller components based on its deliverables. It is visually represented as a hierarchical or tree structure.
With WBS you start with the project’s ultimate objective, then successively subdivide it into smaller, manageable components that scope out areas such as deliverables, time, resources, cost, and span of responsibility. Each component or node in the tree structure needs to have a defined scope, timeframe, budget, resources, and person responsible for its delivery.
WBS provides a framework to plan, schedule, and manage a BI program and individual BI projects. It is important to note that the primary purpose of the WBS is to decompose the project by deliverables rather than defining a detailed, task-oriented project schedule. Developing a detailed project schedule is a natural follow-on creating a WBS.
When dealing with a systems integrator (SI) or outside contractor, use a WBS to create the statement of work (SOW) for the SI, who then develops a proposal with detailed plans and costs. Even if an SI is not engaged in the BI project, the BI stakeholders should consider the WBS to be its SOW.
WBS Design Principles
There are several principles that help with the design and development of the WBS. They use parent-child terminology to describe levels. They are as follows:
• 100-percent rule: a WBS contains 100% of the work required to deliver all project deliverables. Each level in the hierarchy needs to be equal to 100% of the project, and the children of each parent need to represent the totality of their parent.
• Mutual exclusivity: every node in the WBS needs to be exclusive to the parent and not overlap with any other branches in the tree structure. Besides enforcing the 100-percent rule, the purpose of this is to eliminate ambiguity and duplicate work.
• Plan deliverables: a WBS is a breakdown of the project’s deliverables or scope rather than activities. Concentrating on what will be delivered rather than how it will be delivered keeps a WBS from getting bogged down into too many details and becoming too time-consuming to develop.
• 8/80 rule: the WBS components at the lowest level of detail should not require fewer than 8 hours of effort, or the WBS will be too detailed. In addition, the most detailed components should not require more than 80 hours of work, otherwise it is assumed that the components are too high-level to manage. The 8/80 rule is not an absolute rule, but rather a “rule of thumb” that experienced, successful project managers follow.
A project may have multiple WBSs representing different views of the project from various stakeholders’ perspectives. The two most common WBS structures for an application development project are:
• Functional-oriented breakdown: representing the viewpoint of one or more categories of the stakeholders. These categories may be the business sponsors and the business people who will use the resulting BI application.
When multiple WBSs are created, each needs to follow the rules specified above and, in addition, needs to be connected to the underlying project plan to ensure completeness and control. Using multiple WBSs enables more effective communication and better program and project management with a diverse project stakeholder community.
Multiple WBSs for BI
Using WBSs is a very effective project management tool for planning and developing BI solutions. Too often, the planning and scheduling of a BI project becomes a project unto itself, consuming a large part of the time that was thought to be used for designing, developing and deploying the solution. Because the WBS does not involve creating the project task-oriented schedule, the BI project manager can determine project scope and estimate the time, cost, and resources needed by decomposing deliverables, while not getting bogged down in the minutia of the tasks. In addition, since the WBS is independent of any specific project methodology, the BI project manager is not forced into a one-size-fits-all project approach. The BI project manager can tailor the methodology choice based on the project scope and BI maturity of the enterprise. In fact, for more complex or larger projects, it is recommended to use a hybrid approach for the project methodology. This is described further in this chapter.
There are several WBSs to support the design, development, deployment, and expansion of an enterprise BI solution:
• BI program WBS
• BI project WBS
• BI architecture WBS
BI Program WBS
The WBS is an effective planning tool for an overall BI program. Just as a software firm would plan out its product development over series of releases that incrementally and iteratively expand its functionality, so too should a BI program be planned. The BI program’s WBS decomposes the program into a series of releases with their own scope, i.e., deliverables, timeframe, and costs, as depicted in Figure 18.13. Each of the BI releases is then decomposed into the underlying functionality and data that will be delivered.
BI Project WBS
Create a BI program WBS prior to creating the individual BI project WBS. As depicted in Figure 18.14, the BI project WBS is broken down at the highest level by project phases (more on this in the project task section of this chapter). Each of the phases is then broken down by its key deliverables. Each one of the deliverables is further broken down into more detailed underlying deliverables, but as mentioned previously the decomposition should not go down to task level nor should it get too detailed.
This WBS is an effective planning tool to develop a list of the key deliverables showing their interdependencies, level of effort, estimated cost, planned resources, and milestone dates. This WBS describes what will be built and what resources are needed, but does not yet discuss how it will be built, which is deferred to other project management activities.
This WBS is an effective communication tool between stakeholders because it deals with the “what’s” rather than the “how’s.” The BI project WBS is used to solicit budget and resource commitments along with agreement about schedule milestones.
BI Architecture WBS
The third WBS is tied to the BI architecture, and has proven to be very useful at planning architectural deliverables and components for the BI program and individual BI projects. As depicted in Figure 18.15, the BI program or project is broken into the four BI architectures: information, data, technical, and project. (See the discussion in Part III of this book.)
Each of the architectural nodes is further broken down into the deliverables and components of that specific architecture:
• Information architecture contains the BI applications that will be developed and then will be broken down further until the lowest level of detail is specific BI dashboards, visualizations, or reports.
• Technology architecture is broken down into the underlying technologies that are used to create the BI deliverables. Technologies may include BI, analytics, data integration, databases, data profiling, data modeling, data cleansing, MDM, source control, job scheduling, storage, and appliances.
• Project architecture is broken down by the delivery and availability of specific products. At the highest level, this may include product categories, while at the lowest level it may list specific products.
BI Architectural Plan
The BI architecture WBS defines the key deliverables and components for the overall BI program, and is the companion to the BI Program WBS that defines the key business BI deliverables for the program. This enables the BI program manager to examine the program deliverables holistically and then determine the best sequence for incrementally developing the architecture. Once this is done, the manager can assign the architectural component to individual projects to design, develop, and deploy. As depicted in Figure 18.16, the four architectures span each phase in the individual BI projects.
The BI program manager should define four architectural plans:
• Information
• Data
• Technical
• Product
The information architecture plan defines what BI functionality needs to be built and what data will be accessed in the context of the business requirements for the BI solution. This plan should detail the following:
• Business applications accessed
• Business processes impacted
• Business and data subjects used
• Business groups and business people defining and using BI solution
• BI and analytic capabilities required
The data architecture plan defines what data is used to support the information architecture. This plan should detail the following:
• Data sources
• Source applications, databases, and files
• Subset of data accessed: tables, columns, fields
• Historical data requirements
• Data integration and transformation
• Data quality and cleansing
• Processing requirements
• Metrics and monitoring
• Define requirements for changes in historic data and approaches for:
• AS-IS, AS-WAS and AS-PLANNED data difference
• Slowly changing dimensions
• Data currency or latency
• Handling of errors, omissions, and exceptions
The technology architecture plan lists what technology is needed to support the information and data architectures:
• BI and analytic tools
• Data integration or extract, transform, and load (ETL)
• Application integration
• Database
• Data modeling
• Data profiling
• Metadata management
• Application development: source code management, job scheduling, backups, recovery
The product architecture plan lists what products are needed to support the technology architecture above. The products selected should comply with the appropriate enterprise standards, although often the BI project introduces many new technologies and products into the enterprise. If particular technologies and products are not being used by the enterprise, then it is likely that the BI project team will need to conduct a product evaluation and selection process. See Chapter 7 for more details.
BI Projects Are Different
From a high-level perspective, project plans for a BI application look very similar to a typical IT application development project. In fact, the project phases can be identical. However, there are fundamental differences between BI and typical IT applications that significantly affect the selection of the appropriate project methodology and the detailed breakdown of its project plan.
These differences are summarized in Table 18.1.
Whereas most companies standardize their business transactional processing and select enterprise resource planning (ERP) to implement that processing, BI has remained a highly customized application. From a high level, there are BI functions and data subject areas that are common within industry groups or business functions such as finance, sales, or marketing. However, the details and specific implementations of those functions and data structures vary widely from enterprise to enterprise. Business performance metrics and detailed data used in the calculation will vary wildly not only across enterprises, but sometimes across business groups within the same enterprise. Although an enterprise can bring in an expert in a particular ERP application to implement their application with much certainty for business and data requirements, that certainty does not exist when implementing an enterprise-specific BI implementation that is highly customized.
Table 18.1
Key Differences between Enterprise Applications and BI Applications
Not only is each BI implementation highly customized at the detail level, but, typically, the details will not be readily apparent at the beginning of the project, regardless of how thorough the discovery processes for business data requirements. Not until the business people actually start using BI applications along with the associated data can the BI team complete the detailed business and data requirements. As we discussed in Chapter 16, much of detail processing that the business is using to gather and analyze data is hidden in undocumented spreadsheets and data silos.
The primary difference between selecting a project methodology and defining a detailed project plan is the uncertainty in the business data requirements at the beginning of the project. This means that the typical waterfall methodology—in which business people communicate their requirements and then IT implements a project to satisfy them—will not work. This one of the key reasons that most BI projects fail to meet business expectations.
Project Methodologies
A systems development life cycle (SDLC), sometimes called an application development methodology, is the term used by software engineers and IT professionals to describe the process used to plan, design, develop, test, and deploy software applications. Software engineers working within a technology firm will always use an SDLC when developing and maintaining software products that are sold to customers. IT professionals, on the other hand, might not use a formal project methodology when developing their applications. This is especially true in small to medium-sized businesses, but is also evident even in larger enterprises.
The primary reason for the difference between software engineers and IT professionals is that software engineers are developing and maintaining the products that their company sells. These software products are therefore the assets of the technology companies and need to be treated as such. Using a common project methodology is the best way for technology firms to design, maintain, support, and expand their products. Using a methodology enables a technology firm to better manage its resources, makes it easier to plan, and establishes documentation as part of the deliverables.
In contrast, the applications that the IT group develops for its company typically are not viewed as an enterprise asset, and the IT group is often viewed as overhead or a company cost center. In addition, business people want their application ASAP and are typically unconcerned about, or unaware of, how that application has been built. With these viewpoints, it is not surprising that a software development project methodology is typically not high on the priority list of many IT groups.
Another difference between IT and software engineers is the “accidental” application phenomena. It is a very common scenario for a business person to ask IT to develop something for them “quick and dirty” with the intention of the code being a one-off. However, the business person starts using it and then goes back and asks IT to add more to it. This continues, resulting in an application that was never designed because it was never planned in the first place. This is the same phenomenon that occurs when a one-off spreadsheet becomes a full-fledged data shadow system. Under this scenario, IT would never use a formal project methodology to build the application.
Without a software development project methodology, it is common for large projects to be late, over budget, or fail to deliver business value, while smaller projects may produce short-term praise but typically create redundant or fragmented data silos. Other shortcomings from failing to use a methodology are:
• BI applications are typically not documented
• Testing gets short-changed
• The application gets increasingly difficult to support
• The application becomes more costly to expand
To succeed both in the short run managing individual BI projects and in the long run in a sustained and expanding BI program, IT groups need to use software development project methodologies. The key for this to happen is that an enterprise must believe that the information used in analytics is a corporate asset, as are the BI applications that transform data into that business information. When both BI applications and information are considered corporate assets, the need for software development project methodologies becomes clear.
Battling Project Methodologies
There has been a battle in recent years regarding which project methodology an enterprise should use for business intelligence: waterfall versus agile. The waterfall methodology has been the traditional approach used in the industry, while agile is a newer approach that is especially popular in Web development.
Waterfall
The waterfall methodology uses a sequential or linear approach to software development. The project is broken down into a sequence of tasks, with the highest level grouping referred to as phases. A true waterfall approach requires phases that are completed in sequence and have formal exit criteria, typically a sign-off by the project stakeholders. A typical list of waterfall tasks would include:
• Scope and plan project
• Gather and document requirements
• Design application
• Develop application and perform unit tests
• Conduct system testing
• Perform UAT
• Fix application as appropriate
• Deploy application
The waterfall methodology is a formal process, with each phase comprising a list of detail tasks with accompanying documentation and exit criteria. Larger enterprises often require the use of SDLC methodology products, particularly in larger IT application projects. This is also the approach that SIs use when building IT applications for their customers, since budget, resources, deliverables, and scope have to be managed on a very disciplined basis.
The advantages of the waterfall methodology are that:
• Requirements are completed early in the project, enabling the team to define the entire project scope, create a complete schedule, and design the overall application.
• It improves resource utilization because tasks can be split to be worked in parallel or grouped to leverage resource skills.
• It is a better application design because there is a more complete understanding of all the requirements and deliverables.
• The project status is more easily measured based on a complete schedule and resource plan.
The disadvantages of the waterfall methodology are that:
• It is often difficult, particularly in BI, to get complete business requirements up front in a project, because business people have not really thought through in detail what they need, and business requirements can change during the project.
• It requires a very detailed breakdown of the tasks and deliverables for the overall BI application, which may be beyond the project team’s capability or experience at the start of the project.
• Although waterfall projects do not inherently have to span lengthy periods of time, it is very common for these projects to span months or quarters because of the emphasis on trying to get everything done at one time, i.e., the “big bang” approach. The likelihood of projects being late, over budget, and failing to meet expectations rises as the timeframe for an IT project significantly increases.
Agile
The primary alternative to waterfall is the agile project methodology. Agile is an iterative team-based approach where requirements and deliverables evolve through cross-functional collaboration. Rather than creating schedules with detail tasks, a list of deliverables is defined, with the team gathering requirements and creating the deliverables in time-boxed phases called sprints. Time boxing is a project planning technique where a fixed time period is assigned to deliverables (scope) and resources rather than the traditional, i.e., waterfall, approach that fixes scope and then determines the time and resources necessary to implement. With the typical agile workflow:
• Deliverables are prioritized by the business consumer and assigned to a sprint
• Requirements gathering, design, development, and testing are completed during the sprint
• Business consumers are active in each sprint, reviewing and evaluating work
• Any work not completed is reprioritized for future deliverables
• The work iterates into the next sprint and loops through same sequence
The emphasis of the agile methodology is rapid and iterative development with considerable interaction between the developers and the business people who will be the application customers. This methodology is popular in Web development and garners a lot of enthusiasm from many industry pundits.
The advantages of the agile methodology are:
• By isolating individual deliverables, work is highly focused and time to completion is accelerated
• Customers are highly involved, enabling more interaction in development and greater ownership of results
The disadvantages of the agile methodology are:
• Individual deliverables are time boxed and rapid, but to achieve this, scope might be reduced and quality of deliverables compromised
• If the overall project requires many interdependent deliverables, then the lack of an overall architecture and framework may result in, at best, a suboptimal design and, at worst, a system whose parts do not work together
• With larger projects, there may be wide variance in expectations about project costs, overall functionality, and quality
The False Choice between Project Methodologies
The battle of waterfall versus agile project methodologies is not as clear cut as advocates of each methodology proclaim. There are assumptions about selecting a single methodology that are false or misleading:
• A specific methodology will work on any type of IT application project
• A specific methodology will work on any size project in regard to scope, resources, deliverables, and time
• There are only two choices of methodologies: waterfall or agile
• Only one methodology can be used by an enterprise
Misleading Examples
When debating the use of waterfall versus agile, advocates base their choice on experience with specific types of IT applications. Waterfall advocates base their experience on developing enterprise applications that are transactional oriented, while agile advocates typically use the creation of standalone BI dashboards, reports, or visualizations. The comparison of the project characteristics between these two types of applications is outlined in Table 18.2.
The key take-away from the comparison is that neither application being used for comparison truly reflects a full life-cycle BI project that includes data sourcing, data integration, database design, and likely multiple BI dashboards, reports, or visualizations. The enterprise application does have the same scope as a BI project; however, the BI project does not have well-defined functionality or data requirements. The standalone BI dashboard does share the lack of well-defined functional requirements with the BI project; however, it assumes that the data is in place, no data integration is needed, and the scope is typically limited. It appears, based on these characteristics, that neither waterfall nor agile methodologies are appropriate fits for a BI project.
Table 18.2
Differences between Enterprise and BI Applications
| Enterprise Applications (Transaction Oriented) | BI Dashboards, Reports, and Visualizations | |
| Methodology | Waterfall | Agile |
| Scope | Business process or enterprise | Typically limited to specific group of business people and processes |
| Data | Well defined | Assumed in place |
| Data integration | Not applicable | Not applicable |
| Functionality | Well defined | Needs to be defined |
One Size Does Fit All
BI project teams range from dozens of IT staff with dedicated roles—such as ETL developer, business analyst, or BI developer in a large enterprise—to a two-to-three person team where every person is involved in all types of development roles. Waterfall works well with large teams but not so much with the small team, where people are frequently switching between roles. Agile, on the other hand, works extremely well with small, dedicated, standalone teams, but its strength works against it when many small teams have to work together in an interdisciplinary fashion.
Based on these characteristics, the selection of waterfall versus agile is more influenced by team size and whether many people working in small groups need to interoperate.
More Than Two Choices
Waterfall and agile are not the only methodologies that are available for a project manager to choose from. The various options include:
• Incremental development: mix of waterfall and iterative development methodologies
• Spiral development: mix of waterfall and prototyping development methodologies
• Rapid application development (RAD): iterative development methodology using prototypes
• Prototyping: not a formal methodology for large projects but an approach to iteratively developing applications
• Object oriented: object-oriented analysis and design methodology
• Unified process: an iterative development methodology framework based on unified modeling language (UML)
The agile methodology also has variations such as extreme programming and scrum.
Mixing Project Methodologies
Both incremental and spiral development methodologies are really plans of waterfall and iterative methodologies. With these methodologies, waterfall is used for the overall project framework, with the project divided into subprojects that are developed iteratively. The subprojects may use miniwaterfall, prototyping, RAD, or agile methodologies to complete the deliverables. The overall project is tracked in the more traditional waterfall fashion.
Hybrid BI Project Methodology
Our recommended BI project methodology is a hybrid approach using the incremental development methodology and some variation of iterative development such as prototyping, RAD, or agile. We will refer to this hybrid as the BI iterative and incremental development methodology. The comparison of the hybrid methodology with waterfall and agile is depicted in Table 18.3.
Table 18.3
BI Iterative and Increment Development Methodology
| Enterprise Applications (Transaction Oriented) | BI Dashboards, Reports and Visualizations | BI and DW (with Data Integration) Systems | |
| Methodology | Waterfall | Agile | Hybrid: Iterative and incremental with agile, RAD, prototyping |
| Scope | Business process or enterprise | Typically limited to specific group of business people and processes | Enterprise, across business groups and processes; may expand to customers, prospects, partners, suppliers, and other stakeholders |
| Data | Well defined | Assumed in place | Data sources, historical quality and metrics need to be identified and documented; needs will expand |
| Data integration | Not applicable | Not application | Key functionality includes consistency, cleansing, quality, and MDM |
| Functionality | Well defined | Needs to be defined | Business requirements need to be defined but will change and expand; analytical functionality will expand |
The BI methodology uses:
• Incremental methodology to define the BI program with its releases, i.e., BI projects
• Waterfall methodology to define an overall framework for an individual BI project and establish interdependencies between subprojects
• Iterative approach to design and develop applications within a subproject
The choice of the iterative development approach will be based on the enterprise’s project management culture, project teams’ experience, and the scope of the subproject. The choice of the specific iterative method is not as critical as commitment by the team to use that approach for that subproject.
Regardless of how much time is spent gathering business requirements, it is inherent in the nature of BI development that the business people will either poorly grasp what their analytical and data needs are, or they will change those needs once they start hands-on with the BI application. Many teams following the waterfall methodology are frustrated with business people changing their minds and adding new functionality or modifying requirements, but those changes are inevitable. BI project teams need to understand that business people are BI consumers and these changes increase the business value of the BI applications. Therefore, they should adopt better methodologies to enable application improvements instead of pushing back.
BI Project Phases
At a high level, a BI project shares the same phases as a typical IT application development project. The project phases, as depicted in Figure 18.17, are:
1. Scope and plan
2. Analyze and define
3. Architect and design
4. Build, test and refine
5. Implement
6. Deploy and roll-out
The recommended BI project phases resemble an IT application development project that follows a waterfall methodology; however, it deviates from that methodology in several key aspects.
First, waterfall-based methodology requires that a phase must be completed before the next phase starts. Our recommended BI project structure has overlapping phases. And as a project is further decomposed into functional groupings that may be performed in parallel, as you will see later in this chapter, there may be several overlapping phases related to specific BI functionality. The functional groups are visually represented by “swim lanes,” which is a processing modeling technique used in UML activity diagram modeling and business process modeling notation.
Second, as opposed to waterfall-based methodology that works sequentially from the start, our recommended BI structure incorporates three feedback or iteration loops. These feedback loops are:
1. Scoping: the initial iteration examining business, technical, and budgetary requirements is always performed at a high level. More detail is always necessary to refine those requirements and adjust deliverables, costs, resources, and timetable. Locking in the scope early and without enough details is a sure recipe for slipping schedule, exceeding budget, and ultimately failing to meet expectations. On the other hand, it is essential to establish some bounds on the requirements, costs, and timeframe at the beginning of the project. Building in the scoping feedback loop enables discovery, flexibility, and better buy-in to the project.
2. Prototyping or development: it is very common in BI and data warehousing projects to need to actually start working with the data, its integration, and the analytical capability before you get a chance to work on the functional design or truly understand its complexity. Writing complete detailed specs before the developers get hands-on with the data is a fool’s exercise. As we discussed previously, the development feedback loop uses an iterative approach that may include the blended use of prototyping, RAD, or an agile methodology. The specific techniques to use are highly dependent on the complexity of the project, as well as the size and experience of the development team. Also, as we discussed in the WBS section, the development activities are split into various swim lanes with feedback loops identifying dependencies across the lanes.
3. Testing: this phase includes informal business group feedback and interaction; UAT; user testing; functional testing; system testing; and, potentially, regression or parallel testing. When dealing with the variety and time span of data, it is essential to build the feedback loop to deal with errors, omissions, and surprises. Many of the issues encountered during testing require changing the programs and then retesting. In addition, business people will often identify items that they forgot to ask for during the requirements gathering process. This feedback loop enables some, but not all, of these changes to be incorporated into the deliverables.
Scope and Plan Phase
This phase involves defining project scope in terms of high-level requirements, timeframe, resources, and costs. It includes creating the project budget and obtaining sponsorship if that has not already been done. A key decision will be about who will actually undertake the project:
• Internal team
• Internal team augmented with external resources
• External team
If an external team is going to be used, then typically a request for proposal (RFP) process will be completed during this phase to select the team. This selection process will typically involve a written RFP sent out to candidates, who will then submit proposals with an SOW that includes deliverables, project schedule, resource plan, and cost estimates. The exit criteria will be a signed agreement that incorporates the SOW as the basis for the project.
Although the rigor of an RFP process is not necessary if an internal team is performing the BI project, it is still best practice to provide an SOW with the same content that an external vendor will provide (of course, without the sales and marketing materials).
There can be significant risks in this phase if the business and data requirement details are not clear. Analyzing the details may reveal unexpected issues regarding data quality and completeness, or may lead to changes in analytical requirements that will affect the scope and timeline of the project. One of the best practices for BI projects with vague requirements is to undertake a BI assessment. The BI assessment (discussed earlier in this chapter) is a combination of the first two phases of the BI project with an accompanying feedback loop. A BI assessment enables an iterative and stepwise refinement of the requirements, significantly reducing uncertainty and, more formally, revising scope and timelines in the initial phases of the project. This significantly reduces the likelihood that the project will be late and over budget, and better sets expectations for deliverables.
Analysis and Definition Phase
The key deliverables in this phase are detailed business, data, and quality requirements derived from the high-level requirements documented in the scope and plan phase. The analysis tasks examine what is being asked for and compare that to the current status of data in reporting. The in-depth comparison results in a gap analysis that reveals what can be leveraged, what needs to be replaced or retired, and what needs to be built.
From the data perspective, source system analysis is the key task to determine:
• what data is being asked for
• the current sources of that data
• most important, the state of the data: consistency, completeness, and quality
It is critical, when profiling data, to examine in detail the state of the data as far back in history as is needed by the business. As we discussed previously, data does not age well. Many BI projects have included unpleasant, last-minute surprises when historical data differed from current data because the project failed to look at that historical data in this phase.
Although IT and business people should be interacting throughout the project, this phase will be the most people-intensive portion of the project. Although there are certainly software tools to assist in profiling data and collaborating with stakeholders, analyzing and defining the detailed requirements includes intensive interaction between business BI consumers in the BI development team.
For more in-depth discussion of this phase, see Chapter 11, and Chapter 14.
Architect and Design Phase
In this phase, the BI development team begins to feel that it is laying the foundations for the BI solutions. The key deliverables are the BI solution blueprints: information, data, technology, and product architectures. These include the design for data integration processing, business intelligence applications, data models, and their technology underpinnings.
This is a reminder that our project methodology is not a strict waterfall where previous phases have to be completed before subsequent phases can start. In our methodology, phases will overlap and will provide feedback for refinement of other phases. In this manner, the architectural blueprints can be designed, if resources are available, while the analysis and definition phase is underway. The architectural frameworks do not need all the details from the analysis phase to be designed. The frameworks are similar to an architect’s blueprint of a house that lays out all the critical specifications, but leaves details such as bathroom fixtures and furniture to a later design.
The assumption in this phase is that technology and products have been acquired by the organization and the architectures use these in their design. If new products need to be selected and purchased, then the RFP process (in the scope and plan phase) needs to be expanded to incorporate this selection process.
For more in-depth discussion of this phase, see these other parts and chapters:
• Part III
• Part IV
Build, Test, and Refine Phase
As the architecture and design phase is underway, the construction phase, i.e., build, test, and refine can start. There have been many advances over the years in the technologies and products used to build, test, and implement BI solutions, as we discussed in Chapter 4 and Chapter 7. These advances have not only expanded BI functionality, but have improved productivity, quality, and time to market.
In addition, current technology has enabled much better interaction between the developers and the BI consumers in the design, testing, and refinement of the BI solutions. There are several alternative project methodologies to encourage this interaction such as agile, prototyping, proof of concept (POC), and RAD. Regardless of which methodology is used, the key to success is getting the business person involved in examining the solutions both from a data and BI functional viewpoint. A crucial mistake made by many projects is to have user involvement in the requirements and then re-engage the BI consumer for UAT. No matter how much time is spent on gathering requirements, business people will add to or modify those requirements once they start interacting with the data in the BI application. It is a fairly common occurrence for the business person to have forgotten detailed steps performed or data needed in the business process supported by the BI application being built. The best way to really get a complete set of process and data requirements is to have the business BI consumer interact with the BI application as it is being built and the data integrated.
For more in-depth discussion of this phase, see the following parts of this book:
• Part V
• Part VI
Implementation Phase
The implementation phase is the first phase that shifts to primarily IT-oriented deliverables, which are:
• Migrating software and data to testing environments
• UAT
• System testing
• Parallel or regression testing, if applicable
• Migrating software and data into the production environments
In the past, the BI project team would work with the enterprise’s in-house production support group to complete these tasks. Nowadays, it is much more common to have one or more components of the BI solution and hosted environments in the cloud rather than just residing in on-premise data centers. When the BI solution or portions thereof reside off-premise, the BI project team will likely coordinate with both their in-house production support and the off-premise production support.
It is critical that the processes are defined and the success criteria documented for each of these tasks. Nothing should be ambiguous regarding the success or failure of any of the testing processes. Before the start of each of these tests, business and IT have to be involved in committing to whatever criteria have been agreed to. The success criteria should be in the form of an SLA between the BI project team and the BI consumers, who are likely represented by the business sponsor.
Both cloud and virtualization technologies have enabled organizations to perform more complete and thorough testing much more cost effectively than they could ever do in the past. Successful testing is absolutely essential prior to deploying BI solutions to obtain BI business value and avoid the business costs of faulty decision making based on faulty BI applications.
For more in-depth discussion of the testing deliverables in this phase, see these parts of the book:
• Part V
• Part VI
Migrating software, sometimes referred to as promoting, from development to test environment and subsequently to a production environment, needs to follow a documented process with a checklist to ensure successful completion. It is recommended that both the application software and data migrate together from one level to another, along with any supporting documentation and procedures. The application software includes all the data integration, BI and any other software involved in moving, transforming, or analyzing data. The software also includes any procedures used to create or modify the data structures. In addition, the data model and applicable metadata should be included.
Deploy and Roll-Out Phase
The moment of truth has arrived as the BI system goes into production and business people start using the BI applications in their daily jobs. This phase should be handled just the way a software firm would release a product to its customers, by offering:
• Product communication
• Product training
• Product support
• Product marketing
BI project communication channels―BI portal, collaboration tools, e-mails, social media, and meetings―should transition naturally in the deployment phase to an overall BI program perspective. The communication with the stakeholders should include status, timetables, support mechanisms, and training availability.
Business person BI training should include education on both how to use the BI application and what data is available with it. The BI educational plan developed during the build phase will be executed during this phase. Encourage business people to take the BI education before they use the application, as this will improve their productivity and significantly reduce misuse and misunderstanding of the system data.
The BI program should leverage any existing standard support processes, such as a help desk, that are deployed for IT applications. If not, the BI program needs to establish an effective support structure for its consumers.
It is best practice to provide at least two tiers of support. The first tier handles basic support issues such as gaining access to the software, connectivity issues, and an explanation of basic capabilities. This first tier handles the bulk of the support inquiries regarding BI. If basic instructions are segmented into easy-to-find explanations on a portal or Web site, sometimes even presented in a short video, then business people may be able to more quickly use the BI solution. Of course, a help desk process manned by IT staff will often be necessary because a business person may not even know what questions to ask. Business interaction with help desk personnel may be by phone, e-mail, chat, or a more formal help desk software package.
The second or higher level tiers supply more in-depth support. This high-level tier support is typically provided by the BI project team, the subject matter experts (SMEs) for the system. Typically, a business systems analyst (BSA) from the BI team is the first point of contact by the business person requesting support. In many cases, BSAs can answer the support inquiries, but when that is not possible they will confer with the BI project member whose expertise is needed.
The BI solution needs to be marketed during the deployment stage to the existing business people as well as any business groups and people that are being targeted for the roll out. Business people are busy, and no matter what the value of the BI solution, it will take a commitment of their time to get them trained and started in using it. You need to proactively market the business value to them.
BI Project Schedule
The BI project is composed of the six phases as depicted in Figure 18.17. These phases represent the full life cycle of a project from planning through deployment. An effective approach to creating a BI project schedule is to decompose the BI Project WBS (Figure 18.14) into greater levels of detail until it is sufficient to schedule tasks, assign resources and track progress. The level of detail will vary based on:
• Complexity of the project: more details
• How extensive an interactive methodology is used: fewer details, more milestones
• Enterprise project management standards: likely more details
• Project manager skills and BI knowledge
Divide BI project tasks and milestones into functional groupings. The groupings, which are visually represented by swim lanes in Figure 18.18, are:
• Business intelligence
• Data and database
• Data integration
• Infrastructure
• Program and project management
The functional groups are based on the three major technology categories (BI, data, and data integration) and the two enabling categories of infrastructure and project management. This is very useful in planning and managing related tasks and skills. These groupings serve the purpose of organizing by team skillsets and related tasks. As BI project teams or groups get larger, they will likely be organized along these same skillsets.
The swim lane visually illustrates the sequence of tasks as they iteratively and incrementally progress through a project. From this level of detail, the project manager should construct a detailed BI project plan with tasks, dependencies, and resource allocations. The initial level of detail or key tasks performed within the functional groupings are illustrated in Figure 18.19.
BI Swim Lane
The BI functional group contains the primary set of activities (Figure 18.20) in which the BI project team interacts with its customers, i.e., business people. The BI team roles responsible for these tasks are BSA and BI developer. The project manager takes on the supporting role of account manager to the business people who are this project’s stakeholders, serving as their advocate to ensure that their views and feedback are properly represented.
This grouping is responsible for the two primary deliverables:
• Business and data requirements
• BI deliverables such as dashboards, visualizations, reports, or advanced analytics
The BSA works with the business BI stakeholders to gather, prioritize, and document business requirements, including analytical capabilities and data needed. The BSA delves into the details such as the business rules, transformations, algorithms, key performance indicators (KPIs), and data quality metric requirements. These requirements are likely to be further refined and modified during the BI development activities.
The BI developer designs, develops, and tests the BI solutions, i.e., the deliverables requested by the BI consumers. If using an iterative methodology such as agile, prototyping, or RAD, the developers work closely with a selected set of business BI consumers throughout this process and use a feedback loop to iteratively develop the BI solution. The four key milestones during this process are:
• Design preliminary BI solutions
• Gather user feedback and perform unit tests
• Refine and publish BI deliverables
• Conduct UAT with selected business BI consumers
If this is a large project supporting multiple business departments, the deliverables are grouped together to reflect their respective business constituencies. With sufficient resources, the subgroups can work independently of each other to improve productivity.
These activities are dependent on two tasks:
• Scope-priorities-budget task, which provides the high-level description of deliverables to be used as the basis for the requirements–discovery activities
• Data-and-database-task, which supplies the data schema for BI design and actual data to use in development and testing
Data and Database Swim Lane
The data and database functional group contains the data-focused activities (Figure 18.21) for designing and populating (loading) all data structures needed to support BI. The BI team roles responsible for these tasks are the data architect, data modeler, and database administrator (DBA). The data architect oversees all data-focused activities and coordinates with BI personnel responsible for interdependent activities.
The two primary deliverables from this grouping are:
• Data design: logical and physical
• Data loaded into all relevant BI data structures
The data architect and data modeler examine the business and data requirements to start designing the data architecture and data models. If there is already a BI solution in place, they will examine what data is needed, what data already exists in the DW, and what data is missing in that DW. They will design new BI data structures and modify existing ones to fill in the data gaps they identify. If the enterprise is new to BI, a data architecture will be designed followed by supporting data models.
The key milestones for data design are:
• Data profiling: examine current and historical data sources
• Data models: logically design
• Data stores: physically design and create
This functional group is responsible for supplying the data models and data architecture to the BI, data integration, and infrastructure groups. The group depends on the BI group to supply the business and data requirements, the data integration group to load the data, and the infrastructure group to set up and manage the technology to support the data structures.
The data and database functional group validates that the following data loads have completed properly:
• Sample data to be used for prototyping
• Test data to be used for more extensive prototyping and validation
• Current and historical data to be used for development, systems testing, and implementation
Data Integration Swim Lane
The data integration (DI) functional group contains the integration-focused activities (Figure 18.22) that design, develop, test, and deploy all integration-related processing. The BI team roles responsible for these tasks are DI architect and DI developers. The DI architect will oversee all integration-focused activities and coordinate with BI personnel responsible for interdependent activities.
The two primary deliverables from this grouping are:
• Source systems analysis
• Data integration processes
The DI architect and DI developer examine the business and data requirements to determine what source systems need to be integrated. If there is already a BI solution in place, then they determine what data is already sourced compared to the data needed, to identify any data source gaps. If the enterprise is new to BI, they need to examine all the data sources.
The next step is to conduct a source systems analysis. The DI team works with the source systems’ SMEs, if available, to gain a better understanding of what is available and its data quality. After those discussions, the DI team should conduct hands-on data profiling to drill into the details of both current and historical data. If there are no SMEs, the team will have to rely totally on data profiling.
After source system analysis, the DI developers use the results to obtain the schemas for the BI data structures from the data-focused group to design the source-to-target mappings and accompanying data workflows. The mappings and workflows are the basis for the DI specifications. In larger projects, either the DI architect or senior-level DI developer creates the DI specifications and provides them to the hands-on DI developers.
The source systems milestones are:
• Source systems requirements
• Source-to-target mappings
The DI developers use the specifications to design, develop, and test the integration processes. If this is a large project, the DI development is grouped by business subject and then by data source to improve consistency and productivity. The iterative and incremental approach is best for performing this data integration work, because there are many interdependencies across the code and the developers should heavily rely on common standards, templates, and applets (this is covered in Chapter 12).
The key milestones for data integration are:
• Data integration processes designed and developed
• Data integration processes tested
• Data integration processes validated
The DI team loads the sample, test, and production BI data structures, and works with both the data and BI teams to validate the data. The DI team works with the infrastructure team, obtaining support for the data integration, databases, and technology they need to perform their work.
Infrastructure Swim Lane
The infrastructure functional group is responsible for all the activities involving the care and feeding of the technologies used for BI, as shown in Figure 18.23. Typically, the team is composed of IT staff personnel from an enterprise’s systems group. This group manages the enterprise’s infrastructure such as networks, hardware (servers, PCs, tablets), storage, and databases. Nowadays, some or all of these may actually be hosted in off-premise cloud environments, in which case this group is the liaison with the outside vendor that provides the hosting solution.
The two primary deliverables from this grouping are:
• Architectures: design and implementation of the technology and product architectures
This group’s primary interaction with the BI team is with the BI architect. It is that BI architect who will design the initial technology and product architectures. In addition, the BI architect likely handles any product evaluation efforts. The infrastructure group works with the BI architect to develop the detailed technology and product architectures that will be implemented.
After the architectures have been designed and the product selected, the infrastructure group’s responsibilities include setting up the technology environments, including such deliverables as:
• Acquiring appropriate products with licenses
• Installation and configuration of the products
• Enabling product access and their usability by the BI project team and appropriate business users
• Ensuring appropriate privacy, security, and regulatory compliance
The infrastructure group is responsible for ongoing operations, including monitoring and performance tuning. This group includes the BI environment in its backups, auditing, and disaster recovery processes.
Program and Project Management Swim Lane
This functional group is responsible for all the BI program and project planning and management activities. The primary responsible individual is the BI project manager. If the BI project is small, this person may also be the BI group manager; in a larger organization, there may be a dedicated project manager. In organizations that have project management offices (PMOs)―typically Fortune 500 companies―the BI project manager would be assigned from the PMO.
It is a best practice for the business sponsor to designate a person responsible for coordinating any activities between business people and the BI project team. In addition, this person manages any deliverables that the business is responsible for. This person is typically considered the peer of the BI project manager and, in some organizations, may be a project co-manager. This person is the project team’s business advisor.
Potentially, there is a third individual who forms a project management triad: BI technical advisor. This role is needed when the BI project manager and business advisor are not very knowledgeable about BI itself or if the BI team has many technologists to be managed. The typical candidates for this role are either the BI architect or a very experienced outside consultant. Besides providing BI subject matter expertise to the BI project manager and business advisor, this person manages the day-to-day activities of the technical staff.
The project management related activities include:
• Business justification
• Schedule and resource planning
• Schedule and resource management
• Change management
• Status reporting
• Issues resolution
• Communication with all BI stakeholders
• Coordination between the BI project team and others involved in the project
• Risk management and contingency planning
For more in-depth discussions of project management roles, see Chapter 17.
The project manager is also responsible for any coordination necessary between the BI project team and the following organizations or processes, if they exist:
• Data governance
• Analytics and report governance
• BI COE
• DI COE
For more in-depth discussions of these concepts, see these other chapters of the book:
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.