Introduction

Only four types of organizations need to worry about data quality:

Those that care about their customers,

Those that care about profit and loss,

Those that care about their employees, and

Those that care about their futures.

–Thomas C. Redman (2006)

The Reason for This Book

Information is currency. In today’s world of instant global communication and trends that turn on a dime, up-to-date and reliable information is essential to effective competition. Whether it is market research, patent applications, manufacturing improvement metrics, taking sales orders, or receiving payments, information is as essential to the functioning of a business as it is to providing that business with a competitive edge.

Information quality contributes to that edge by delivering the right information, at the right time, in the right place, to the right people. Human beings cannot make effective business decisions with flawed, incomplete, or misleading data. Humans need information they can trust to be correct and current if they are to do the work that furthers business goals and objectives.

The right information can help inventory managers keep the supply chain lean, help CEOs make long-term plans for growth based on accurate and dependable performance measures, help social services identify high-risk youth who need their help, and instill trust in voters that election results are accurate. Understanding the quality of the information can allow decision makers to account for the impact of poor-quality data.

The wrong information can make the difference between a satisfied customer and a frustrated customer, or several thousand frustrated customers. In some cases, incorrect information can make the difference between life and death. Even if the stakes are not as high as that, poor-quality data impact the health of any organization. Lost business, magnified by the effects of wrong information propagated throughout information systems, can lead to millions of dollars in lost revenue. Businesses must use sound practices for improving information quality and reducing error to prevent loss of business and to produce satisfied customers. Every organization—for-profit, nonprofit, government, educational institutions—depends on information to support its customers and whatever the organization provides.

This book provides a systematic approach for improving and creating data and information quality within any business. It explains a methodology that combines a conceptual framework for understanding information quality with the techniques, tools, and instructions for improving and creating information quality. The Framework for Information Quality is the conceptual framework and The Ten Steps are the processes for implementing its concepts.

The methodology can be applied in the following ways by choosing the appropriate steps, activities, processes, and techniques:

For information quality-focused projects, such as a data quality assessment of a particular database or assessing the business impact of an issue that is the result of poor-quality data with the goal of identifying root causes and implementing improvements.

In the course of daily work whenever you are responsible for managing data quality or the work you do impacts data quality.

To integrate specific data quality activities into other projects and methodologies. For example, Enterprise Resource Planning (ERP) migration, building a data warehouse, or other application development and implementation.

The methodology can also form a foundation to create your own improvement methodology or to integrate data quality activities into a company’s standard project life cycle or software/solution development life cycle.

Every company is different, yet the underlying approach to data quality described in the pages that follow applies to all companies and all types of data, whether finance, research, development, procurement, manufacturing, sales and marketing, order management, human resources, and so on. It applies to numerous types of organizations—businesses and corporations of all sizes, educational institutions, government agencies, and nonprofit and charitable organizations—because all depend on information to succeed.

Intended Audiences

Are you:

• An individual contributor or practitioner?
• A member of a project team?
• A project manager?

Do you find yourself in any of the following types of situations?

• Poor-quality data are causing issues for the organization, but no one is sure of their real impact and how much should be invested in dealing with those issues.
• Your company is implementing (or has implemented) an ERP application. Poor-quality data are having an impact on project timelines and are hampering the migration and test results. Once in production, confidence in the information is low and data previously used by one business function are being used in end-to-end processes, with poor results.
• Your new application development project goes live on time, maybe even under budget; the user-acceptance tests are complete and the solution is being used. A few months later, however, complaints start to surface. Additional staff is needed to handle data quality and reconciliation needs. It is discovered that the final solution did not have all the information those involved need to make decisions. Or worse, incorrect information is presented to decision makers leading to costly mistakes.
• Because of mergers and acquisitions, your company is starting a project to integrate data from the acquired companies. The project team has a tight schedule, yet you already know there are quality issues with the data to be integrated.
• The organization has invested heavily in purchasing data from external sources but cannot depend on its quality to meet business needs.
• The data warehouse has been in production for more than a year. Users from the business intelligence group don’t trust the reports, complain about the quality, and are reverting to their own spreadsheets for verification.
• The company invested in a major master data clean-up project, such as customer, vendor, employee, or product. A few years later another clean-up project is started because the data quality declined and is causing issues for the business.
• Your organization has purchased a data quality tool, but doesn’t know how to best implement it and use it effectively.
• You are involved in a Six Sigma project at the company and need more help regarding the information and data aspects of the project.
• Managing data quality is an important part of your daily responsibilities.

If these situations sound familiar, this book is for you! Whether you are just starting a project or are already in production, it is not unusual to find that information quality issues prevent the company from realizing the full benefit of its investment in the projects. The business therefore does not receive the expected improvements to operations, decision-making and business intelligence processes, and customer satisfaction.

There is hope! Although problems with data and information quality can seem urgent and overwhelming, an iterative information quality process of accountability, root cause analysis, prevention, ongoing monitoring, continuous improvement, and communication makes the issues solvable. Here you will learn processes, techniques, and ideas to address the data quality challenges. The same techniques and tools presented in this book can also be used during system development and implementation projects to prevent or minimize data quality challenges from the beginning.

This book has what you need to address information improvement at every stage—from an initial assessment project to fine-tuning current information processes to developing and implementing IT solutions that produce the most current, most relevant information to use.

Structure of This Book

This book describes a methodology, Ten Steps to Quality Data and Trusted Information™, which consists of a framework, concepts, and processes for improving and creating information and data quality.

The Ten Steps are concrete instructions for executing information and data quality improvement projects. The steps combine Data Quality Dimensions (aspects or features of information used for defining, measuring, and managing data) with Business Impact Techniques (qualitative and quantitative techniques for analyzing the impact of data quality issues). Performing the first steps in the process help present a picture of the current state of data and information quality in the business or organization and produce a convincing rationale for pursuing data and information quality. Subsequent steps address the business needs revealed by the picture of the current state of information quality. The descriptions of The Ten Steps process include templates, examples, and advice for executing them, with the goal of reaching a future state of excellence in information quality and information processes.

To implement The Ten Steps effectively, it is necessary to understand some key concepts about information and data quality; for this reason, several of the concepts are presented first. The Framework for Information Quality (FIQ) provides the conceptual foundation for understanding the components necessary for information quality. The framework includes the very important Information Life Cycle, which brings insight into the life of information as it is created, applied, stored, maintained, and ultimately disposed. The groundwork is also laid by defining other concepts such as the data quality dimensions, business impact techniques, data categories (groupings of data with common characteristics or features such as reference, master, transactional, and metadata), data specifications (standards, models, business rules, metadata, and reference data), data governance and stewardship, and best practices and guidelines for implementation.

Main Sections

This book contains the following main sections:

Chapter 1, Overview—a summary of the information and data quality approach.

Chapter 2, Key Concepts—the philosophy and fundamental concepts that are integral components of the methodology and on which The Ten Steps process is built.

Chapter 3, The Ten Steps Process—the process flow, instructions, advice, examples, and templates for completing information and data quality improvement projects.

Chapter 4, Structuring Your Project—advice about approaches to data quality projects, timing, and assembling a team. Use this chapter to help structure your information and data quality improvement project.

Chapter 5, Other Techniques and Tools—techniques that can be applied in various ways throughout the methodology.

Chapter 6, A Few Final Words—a summary of the other chapters and some words of encouragement.

Appendix, Quick References—this section pulls together materials that were presented throughout the book into an easy-to-read reference format.

Glossary—an alphabetical list of terms discussed in the book together with their meanings.

Bibliography—the list of books, articles, and websites used during the writing of the book.

Conventions

Italicized text is used to indicate references to the steps and important words or concepts in this book (e.g., Step 1—Define Business Need and Approach). The list that follows contains a description of the callout icons you will see in boxes at various places within the text.

Definition	Defines key word or phrase
Key Concept	Describes significant, essential, and/or important ideas
Best Practice	Tips for implementation and recommendations for excellence and success based on experience
Warning	Things to watch out for
Communicate	Describes communication activities, which are critical success factors that help support what is happening throughout the project
Checkpoint	Guidelines, in the form of questions, to help you determine completion of each step and readiness to move on to the next step

How to Use This Book

The goal of this book is to teach the basic concepts of information quality and to provide enough specific step-by-step instructions so that you can improve and create quality information.

In today’s fast-moving world, readers need to be able to get information and find what they need quickly. This book’s format is conducive to reading front to back, but also to quickly finding items of particular interest at any point in time. It is expected that the book will be used as a reference guide, with the reader returning to it when new data quality situations arise. For example, here are two common laments:

• “We know data quality is important, but we don’t know how to get started.”
• “ Some of us know data quality is important, but we don’t know how to show the value of information quality.”

What’s here will help anyone who is responsible for the quality of data and information: program managers, project managers, practitioners, individual contributors, and internal and external consultants. Practitioner job titles include data analysts, business analysts, developers, database administrators, subject-matter experts, information consumers or knowledge workers, data modelers, and data stewards. This book also provides techniques for assessing business impact that can be used to show the importance of information quality.

This book operates under the “just enough” principle. Readers are given just enough background on the underlying concepts to understand the components necessary for information quality and to provide the foundation for the instructions. The step-by-step instructions provide just enough structure for readers to understand what needs to be done and why. The methodology is designed for the users in a way that allows them to pick and choose the steps applicable to their specific projects. The beauty of the approach is that it provides just enough structure to know how to proceed, but enough flexibility so that those using it can also incorporate their own data quality improvement knowledge, tools, and techniques.

When the words business or company or organization are used, it means any type of organization that uses data and information to support its goals. For-profit corporations are obvious, but it also means other organizations such as government, nonprofits, and educational institutions. Many of the examples center around customer information because that is familiar to most people and can be used to easily illustrate the concepts and techniques. Customer means whoever uses what an organization provides. However, what is presented here applies to all types of information (e.g., customer, employee, product, sales and marketing, manufacturing, research) in any industry (e.g., high tech, financial services, insurance, biotech, retail, manufacturing). All organizations depend on quality information to support whatever they provide.

This approach is not specific to, nor does it require, any particular data quality software (e.g., data profiling or cleansing tools), but the methodology can be used to help implement these tools more effectively if they are available to you.

The companion website for this book—www.books.elsevier.com/companions/9780123743695—contains more information, including links to additional resources.

Project Managers

For this group, the following will provide background so that you can help set up a data quality project and make good judgments about which data quality activities are needed to achieve the project’s goals. Read the Introduction and Chapters 1 and 2. After doing so, you will understand the Data Quality Dimensions and Business Impact Techniques so that you and your team can make good decisions about which to use to achieve goals.

In addition, read the first few pages at the beginning of each of the steps in Chapter 3, focusing on the Step Summary tables and the sections on Business Benefit and Context. Next, read Chapter 4 for information on resources and other valuable information about setting up a project.

Project managers don’t need to know the same level of detail as team members. But you do need to understand enough about the data quality approach so that you can support data quality, whether in the form of a focused data quality project or data quality activities integrated into another project plan or methodology. Project managers influence the priority of team members’ time. Your support is required to confirm that data quality is appropriately incorporated and implemented into a project—which will greatly increase your chance of success.

Individual Contributors and Practitioners

The detail in this book is for you! You are the ones who will implement what is needed for success. Become familiar with its contents. The information is presented in a logical format, from concepts to more specific detail, as you move through the book. You may not need to read it cover to cover, but know what is included and then use it as a reference as various data quality situations arise.

All the information in Chapter 2, Key Concepts, is provided as a basis of education. These underlying concepts will help you understand the necessary information quality components and provide the foundation for the step-by-step instructions in The Ten Steps process. The steps are the how-to for information quality and are designed so that you can pick and choose those steps that are applicable to your projects and/or issues.

Even with the how-to, it is expected that you will combine the concepts and the instructions to make good judgments as to how to best apply them to your particular situation. Key to success is the ability to select the relevant steps and determine the right level of detail needed to address a specific situation. The methodology presented in this book provides enough structure so that you know how to proceed, but enough flexibility so that you can incorporate your own knowledge, tools, and techniques into the process.

Of course, the support of management, along with appropriate investments in time, money, and people, are essential for success. Use the ideas here to help explain information quality and gain management support. You will need it. Ultimately, however, this book is for you—the individual contributors and practitioners who are chartered to make data quality a reality!

There are numerous situations where this methodology can be applied. Use what you learn here as a starting point to get going, bring value to your organization, and enjoy the journey!