This book is aimed at IT professionals or curious adults who wish to learn about the foundational concepts that can increase the likelihood for IT project success. These concepts were commonly used and were highly successful at IBM in the 1960s and 1970s. These concepts have morphed over the decades as thousands of organizations and people outside of IBM began to use them. My personal and subjective observation is that these foundational concepts are not utilized sufficiently today. This book is intended to broadly communicate the concepts to the best of my recollection. To the extent IT projects deviate from these concepts, they under-deliver, overcharge, are late, or flat out fail.
The three foundational concepts in data engineering are data models, value chain models, and team dynamics. Data models define the meaning or semantics of the “things of interest” to the enterprise. Value chain models define the flow of “things of interest” from external actors into the processes of the enterprise, define how the processes add value to the “things of interest”, and define how the completed “things of interest” are passed to external actors as finished goods or services. Team dynamics means that people need to be conscious of their behavior and how it impacts the ability of the team to deliver value successfully. People need to be aware of the uniqueness of each individual, as well as how we are all members of the same enterprise.
Your IT project is very likely to succeed if you are clear on data and process, as well as maintaining constructive interpersonal relationships. Also, data and process exist with different viewpoints defined by different audiences and purposes. I learned these data and process concepts from John Zachman and Ted Codd at IBM in the late 1970s. The interpersonal relationship ideas have evolved from my own circuitous journey of trial and error over many decades. I present this trio of concepts by using stories from my life, in hopes of ease of consumption, breadth of audience, and clarity through example.
Data Modeling Ideals versus Reality
Data modeling cannot be effective as an independent design artifact. Consistent success with data modeling requires a related value chain and constructive team dynamics.
This book will describe three important data modeling artifacts called the Conceptual Data Model (CDM), the Logical Data Model (LDM), and the Physical Data Model (PDM). These three types of models will be described as idealized concepts in their pure form. This is similar to Platonic idealism, the archetypes that describe the perfect form of a thing while recognizing that the real world is built of things that always vary from the ideal. The value of these three ideal concepts is that we can take a data model and categorize which portions of it align with each concept.
Of course, when I sit down to do a CDM for my shipping department I hope to align perfectly with the CDM ideal for the entire modeling effort. The reality is that I have rarely achieved this ideal due to time, lack of access to experts, complexity of the problem, or my personal defects. When time or budget expire on the shipping department CDM effort, my team sits down and discusses which portions of the CDM are true and which portions are leaning towards LDM or PDM. We probably won’t be able to fix the errors, but we can all use the shipping department CDM model with full consensus and awareness of the imperfections. We can compensate for CDM weaknesses as we do the LDM and PDM.
Agreeing on where the imperfections exist and how they vary from the ideal provides a surprising degree of value as the project moves beyond data modeling and into design and coding.
Each data model will have a corresponding value chain model. All of the inputs and outputs in the value chain will be represented in the data model. A value chain is a set of processes that take low value things and transform them into high value things. Value is defined in the context of your customers. If my customers are hungry carnivores, then hamburgers have value to my customer. If, on the other hand, my customers are hungry vegetarians, then salads would be of more value than hamburgers. Value chains exist within a single business but can also exist across multiple businesses. Each step in a value chain represents a specialization in the division of labor. Specializations can be outsourced or in-sourced to optimize profitability.
We will touch upon some ancient wisdom on human behavior, as well as some contemporary brain research, to present ideas on how we can behave to optimize collaboration, efficiency, and the probability of IT project success.