4.1 Introduction

The word metadata has a number of meanings, not all obviously-related to one another. Several formal definitions exist, most of them defining the word to mean “data about data.” The word seems to have been coined in 1969 by Jack E. Myers, who intended to choose a term with no particular meaning in the data management community. In the ensuing years, the word has been enthusiastically adopted by the data community, but with varying meanings.

Multiple online dictionaries define the word. Wikipedia¹ says this: “Metadata has come to be used to refer to data about data.” Dictionary.com² includes this definition: “Data about data. In data processing, metadata is definitional data that provides information about or documentation of other data managed within an application or environment.” Both definitions, and others, are consistent in saying that metadata is data about data. Another helpful definition that we found on the World Wide Web³ is this: “Metadata is information about a thing, apart from the thing itself.” While wholly consistent with the more terse definitions, this one calls out a particularly important characteristic about metadata – it is something different than what it is describing. A paper we encountered⁴ contains an interesting discussion of metadata and related concepts as they are used to find information. Although the paper concentrates on two particular kinds of metadata (semantic and catalog – see the following paragraph), its contents may be helpful to readers of this book who want a better grasp of the concepts involved.

We have found four different usages of the word metadata (sometimes spelled “meta-data” and sometimes given as two words: meta data) in the data management community, briefly described next. For each meaning, we provide an adjective that clarifies the intent of that meaning. (We have chosen to use the term data field or sometimes just field to mean some component of data that is an identifiable “piece” of some data structure. The use of “field” is not intended to evoke the image of some particular structure, such as a row.)

In this chapter, we introduce each of the meanings of the word, with examples. At least one example in each section is provided in an XML context.

Since metadata is “data about data,” it’s easy to get confused by the terminology. To avoid confusion, we use the bare word data to mean the data that an application needs to do its job. Similarly, we use the word metadata, with or without an adjective, to mean that data that somehow describes that other, application-required, data.

Although we do not discuss this further, it is worth mentioning that many environments provide still higher-level metadata that describes other metadata. This concept can be extended to as many levels as needed, as illustrated in the ISO Reference Model for Data Management.⁵

4.2 Structural Metadata

Structural metadata is metadata that describes the structure, type, and relationships of data. For example, in a SQL database, the data is described by metadata stored in the Information Schema and the Definition Schema. In the international standard for SQL, part 11⁶ (called SQL/Schemata) specifies SQL views and base tables that describe a SQL database. The SQL standard provides quite a number of components that applications can define within a database: tables, columns in those tables, views, user-defined types, attributes and methods of those user-defined types, other types of user-defined routines, parameters of those methods and other routines, constraints of various sorts, triggers, and so forth. Each of those objects is described by rows that appear in one (or more) of the various views of the Information Schema and one (or more) of the tables of the Definition Schema.

The descriptions of those objects specify such information as the name of the object, the characteristics of the object, and the relationships that the object has with other objects of the same sort and/or of different sorts.

For example, the SQL standard’s most fundamental object is the table, which roughly corresponds to the relation in the relational model of data. Tables are composed of one or more columns, each of which has a specified data type.⁷ SQL/Schemata provides a TABLES view and a COLUMNS view that, together, describe each of the tables in a database as well as each column of each of those tables. In addition, each constraint that is defined for a table or for one of its columns is described in one or more of several views: TABLE_CONSTRAINTS, COLUMN_CONSTRAINTS, CHECK_CONSTRAINTS, and REFERENTIAL_CONSTRAINTS. Those views are specified in such a way that they retrieve their information from underlying “base tables” with the same name.

Consider the SQL table definition in Example 4-1.

This definition creates a table whose fully qualified name is BOOK_CATALOG.QUERYING_XML.MOVIES (SQL automatically converts ordinary identifiers to uppercase). This table contains three columns, named MOVIEJD, MOVIE_TITLE, and MOVIE_DESCRIPTION. Each of those columns is given a particular data type, which means that each value stored in that column must be a value of that type. One of the columns has a NOT NULL constraint defined on it, meaning that no value stored in that column can be the null value.

The structural metadata that describes SQL tables looks something like the table definition in Example 4-2 (for brevity, we have omitted many of the columns of these metadata tables).

Each row of the TABLES table specifies the fully qualified name of some table. Since TABLES is itself a table, it includes a row containing metadata about itself – that is, SQL’s metadata is self-describing. The name of each table is provided in the form of three columns: TABLE_CATALOG, TABLE_SCHEMA, and TABLE_NAME. The metadata also specifies which of four sorts of table that table is: base table, view, or one of two types of temporary table. Several constraints are specified to govern values stored in the TABLES table. One, named TABLES_PRIMARY_KEY, specifies that the values of the combination of the three _NAME columns must be unique within the table. Another, called TABLES_FOREIGN_KEY_SCHEMATA, mandates that values of the combination of the two columns TABLE_CATALOG and TABLE_SCHEMA be found in columns of the same names in the SCHEMATA table.

Our MOVIES table would be described by a row in the TABLES table with the values in Table 4-1.

The columns of our MOVIES table are defined in a different Definition Schema table, COLUMNS. The columns of each row of the COLUMNS table include the fully qualified name of some column of some table, the ordinal position of that column within its parent table, the data type of the column, and whether or not the column is nullable. The COLUMNS table also specifies several constraints, such as a primary key that requires the combination of values of the four_NAME columns to be unique within the COLUMNS table, that no two columns occupy the same ordinal position in their parent table, and so on. For example, the columns of our MOVIES table would be described by rows in the COLUMNS table with the values given in Table 4-2.

The column in Table 4-2 named DTD_IDENTIFIER deserves a brief explanation. The values of that column are foreign keys into the DATA_TYPE_DESCRIPTOR table. The corresponding rows in that latter table contain information about every variation of data type in the database. For example, if some column C1 in some table T1 has the data type INTEGER and some other column C2 in some other table T2 has the data type INTEGER, there will be two rows in the DATA_TYPE_DESCRIPTOR table, each row identifying the name of the column and the data type (INTEGER), as well as some implementation-specific value that uniquely identifies each row in the table. That unique value is the value stored in the DTD_IDENTIFIER column of a row in the COLUMNS table for the given column.

The Definition Schema base tables fully describe every table, column, constraint, etc. in the database. The structure of each table is fully defined by specifying each of its columns and the order in which those columns appear in the table. The data type of each column (meaning, of course, the data type of the value of that column for every row in the table) is specified. And, of course, the relationships between various columns and between the columns and their own values are also specified.

You might wonder why so much space is spent describing SQL metadata in a book whose subject matter is XML. Simply put, the broad familiarity that SQL has in the software community makes it an obvious choice for introducing the subject of structural metadata.

The world of XML is not always quite as convenient as the SQL world, largely because of the difference in data model. SQL’s data is completely regular, because every row in a table contains a value of every column of the table; in some rows, the value for a given column might be the null value (if the table’s and a column’s constraints permit null values), but there are no “cells” of a table that are simply “not there.”

By contrast, as you read in Chapter 1, “XML” is inherently semistructured, which means that some data elements might be missing entirely from a given XML document. As a result, the nature of XML structural metadata is significantly different from that of SQL.

Metadata for XML documents must provide necessary information about the names and the “types” of each component of those documents. In the context of XML, the word type has a somewhat less obvious definition than it does in SQL. In particular, the type of an XML element may include information such as the attributes and children of the element. The type of an XML element might be some simple type, such as ordinary text, but it might also be a complex type that permits the element to have child elements and even mixed content (combination of child elements and text). When XML Schema⁸ (or other, analogous facility) is used, an element’s content or an attribute can be given a more traditional “data type,” such as integer, character string, or datetime.

For example, consider an XML document that we might have chosen to represent information about actors, as seen in Example 4-3. (Not having gender-based biases, we have chosen to characterize both males and females as “actors.”)

We readily see that the element “actors” is made up of one or more “actor” elements. Each “actor” element contains a “name” element, a “gender” element, and one or more “film” elements, “film” elements each contain a “title” element, one or more “role” elements, and optionally an attribute named “runtime.” The data type of the nonelement content of each element is obviously character string, but the data type of the “runtime” attribute seems to be integer.

The information in the immediately preceding paragraph is the structural metadata of the element “actors.” It can also be characterized as the type of that element. Of course, in order for this structural metadata to be useful to computer systems, it has to be presented in some machine-readable form, perhaps in an XML form (such as XML Schema) or a non-XML form (such as Document Type Definitions or even SQL-like tabular representations).

It is important to observe that none of this structural metadata – for either SQL or XML – offers any hope of understanding the meaning of the data. Sure, we human readers can reasonably presume that an element named “title” might contain the title of a film or that an attribute named “runtime” probably contains the running time of a film (but in what units?). But, absent any other information about the meaning of all of those names and the content of the elements and attributes, no application program is likely to “understand” the data in any significant sense.

In summary, structural metadata serves to describe the data components, the types of those components, and their relationships to one another. However, it has nothing to do with the meaning of the data being described. Structural metadata is discussed in greater detail in Chapter 5, “Structural Metadata.”

4.3 Semantic Metadata

Semantic metadata is metadata that describes the “meaning” of data. Of course, the very term “meaning of data” requires an explanation, if for no other reason than we’ve found that different individuals interpret it differently. We believe that there are two distinct applications of the term semantic metadata that are relevant here. The first applies to the meaning of data values, the other to the meaning of the names of things that can take on such values.

In any context in which data exist, there are values that represent certain concepts. Associating specific values with specific concepts is one way of assigning meaning to the data made up of those values. For example, when we needed to capture the gender of an actor in Example 4-3, we chose to use the values “Male” and “Female” to represent men and women, respectively. Other creators of such data might have chosen “male” and “female,” “M” and “F,” “0” and “1,” or even “” and “” to represent the same concepts.

There is an international standard⁹ that defines how to represent this particular information in a standardized way. This standard specifies that “0” represents “Not known” (which might apply to an essentially androgynous character, such as the one played by Julia Sweeney in the film It’s Pat), “1” represents “Male,” “2” represents “Female,” and “9” represents “Not applicable” (which could be applied to inanimate “actors,” such as robots, that lack a human gender). It also seems to recommend that the term used to describe this concept is “SEX.” Thus, ISO 5218 contains semantic metadata about gender (sex). Unfortunately, that semantic metadata is not readily accessible to computer programs since it is in an ordinary text document.

Which brings us to the second application of the term semantic metadata: the meaning of names of things (fields, columns, elements, even variables) that can be assigned values associated with some meaning. In Example 4-3, we put the name “gender” on the element specifying the (human) sexes of our actors. Had we followed the recommendation of ISO/IEC 5218, we might have named that element “SEX” (or perhaps, to be easier on the eyes, “sex”). Had we chosen to apply ISO 5218 by naming the element according to that standard’s conventions, then the standard would provide the semantic metadata to tell us what the element name “SEX” means.

But how can the names of such “things” and the values assigned to them be given a “meaning” that can be handled by computer programs? Part of the answer lies in the notion of metadata registries. A metadata registry, managed by some registration authority, provides a mechanism by which the names of “things” and the values assigned to them can be managed, making them easier to find and interpret in various data sources.

Another ISO standard, ISO/IEC 11179, is a multipart standard in which each part standardizes a separate aspect of constructing and managing metadata registries. Various parts of this standard describe “a conceptual model of a [metadata registry] and the processes of classification, naming, identification, forming definitions, and registration in order to make data understandable and shareable.”¹⁰ We emphasize that semantic metadata exists to make data more understandable (to programs and to humans) but also more sharable (by programs and by humans).

The framework of that standard states the following in its introduction:

According to Clause 1, “Scope,” of this framework, “[ISO/IEC 11179] applies to the formulation of data representations, concepts, meanings, and relationships between them to be shared among people and machines, independent of the organization that produces the data.” Other parts of ISO/IEC 11179 address the use of metadata registries to record the meanings of the tags that make up various XML vocabularies. Other standards or additional parts of ISO/IEC 11179 could be created that address the use of XML for semantic metadata representation. (We are unaware of efforts to create such standards or parts of ISO/IEC 11179 at this time, but hope springs eternal!)

Design of a metadata registry for some particular purpose might well be driven by an ontology, which is defined by the Wikipedia as an “attempt to formulate an exhaustive and rigorous conceptual schema within a given domain, a typically hierarchical data structure containing all the relevant entities and their relationships and rules (theorems, regulations) within that domain.” The Wikipedia also mentions that T. R. Gruber described an ontology as “an explicit specification of conceptualization.” Computer programs may be written to take advantage of (machine-readable) ontologies to apply a sort of machine understanding of data described by those ontologies. The terms used by such ontologies may well be managed in some metadata registry, such as those specified according to ISO/EEC 11179.

There are several important XML-related standards under development that will allow the development and application of ontologies – and thus semantics metadata – to pages of the World Wide Web. Arguably the most important of these is OWL,¹¹ which provides XML-based mechanisms for defining ontologies that can be used to provide semantic metadata about XML documents on the web.

In summary, semantic metadata is the description of the meanings of values and of the names of data components, at least for human reference and possibly for programmatic use as well.

4.4 Catalog Metadata

Catalog metadata specifies information about identifying and locating data that (usually) cannot be found in the data itself. Catalog metadata includes the kind of information you might use to locate data in a repository of some sort. For example, the title, author, publisher, ISBN, and publication date of a book are pieces of catalog metadata that makes it easier to locate a particular book than searching all known books for some passages dimly remembered from the text of the book.

There are countless mechanisms in common use for cataloging resources of many different sorts, including books, periodicals, conferences, buildings, web pages, automobiles, words, people, theories, galaxies, molecules, cancers, tools, etc.

For example, until rather recently (historically speaking, of course), public and other libraries cataloged the books they held by using index cards stored in large cabinets with many file drawers just the size of the index cards – a card catalog. The cards themselves contained additional information about the book, such as the full title, the author, the publisher, and the publication date. Because this medium was so labor-intensive, books were almost never cataloged in multiple ways. Therefore, locating a book required having an idea of the topic of the book and using that knowledge to identify the most likely Dewey Decimal Number of the subject matter of the book.

Once the Dewey Decimal Number for the subject matter is known, the section of the card catalog in which books of that topic were found can be located. After that, a linear search of the cards associated with books of that subject is performed. Either the card identifying the desired book is found or the absence of such a card indicates that the book was not cataloged under that subject. All in all, this process was workable – demonstrated by the fact that it was in use for quite a few years – but tedious and time-consuming, both for catalogers and for seekers. The advent of computers and their application to libraries provided the opportunity to search for books based on a variety of knowledge about the book: complete or partial titles, complete or incomplete author information, ISBN, and so forth.

Of course, other resources used other mechanisms to catalog individual instances of the resource, such as postal addresses to identify buildings, vehicle identification numbers (VINs) to identify motor vehicles, and taxonomical names to identify species of organism. Each type of resource requires a different set of identifying data to catalog instances of the resource type. Catalog information about books is considerably different than catalog information about chemical compounds or catalog information about stars and galaxies.

Examination of many types of resources for which cataloging is necessary led to a generalization of the requirements for cataloging, which in turn led to standardized vocabularies for catalog metadata. One important standard in this field is the Dublin Core,¹² a set of metadata elements that can be used to describe any resource, that is, anything that has identity. The metadata elements specified by the Dublin Core are: Title, Creator, Subject, Description, Publisher, Contributor, Date, Type, Format, Identifier, Source, Language, Relation, Coverage, and Rights.

One possible Dublin Core representation of a movie appears in Table 4-3.

It may not be completely obvious that there could be a number of different ways to organize the Dublin Core representation of “The Scariest Sci-Fi Thriller in Years!,” as TNT Roughcut called this film. But it’s very easy to see how knowing the information shown in Table 4-3 would help locate the film in a catalog, web store, etc.

In the XML world, a well-known example of a catalog metadata standard is the Resource Description Framework (RDF). One of the documents in this standard, the Primer,¹³ clarifies that RDF is intended “for representing information about resources on the World

Wide Web. It is particularly intended for representing metadata about Web resources, such as the title, author, and modification date of a web page, copyright and licensing information about a web document, or the availability schedule for some shared resource.” In the RDF model, assertions about resources take the form of a subject, an object, and a predicate that specifies the relationship between the subject and the object. For example, the website http://sqlx.org (the subject) is maintained (the predicate) by the authors of this book (the object). Of course, most subjects, including this one, have many characteristics that can be represented in this way. That website has a title, it has a date of most recent update, it has several web pages, and so forth. Each of those characteristics can be represented in the RDF model. In fact, many (or most) of the objects themselves are subjects for other predicates. Thus, a complete RDF description of an entity often forms a tree structure – for which XML is extremely well suited. RDF is discussed more fully in Chapter 18, “Finding Stuff.”

Dublin Core and RDF are not competing ways of creating and representing catalog metadata about documents. Dublin Core defines a number of metadata elements used to describe documents and represents a consensus among information retrieval specialists about the minimal information necessary to identify and locate such documents. By contrast, RDF is an architecture for representing and organizing metadata (or, indeed, data) that does not predetermine what that metadata must be.

It is not uncommon to find environments in which RDF is used to record Dublin Core metadata. For example, the following RDF-like assertion identifies the title of the movie described in Table 4-3:

(In this case, “our” is a prefix representing information about stuff we own – including movies – and “dc” is a prefix representing concepts belonging to the Dublin Core specifications. As you will see in Chapter 18, those prefixes are only illustrative.)

In summary, catalog metadata is information that makes it easier to locate desired resources among a collection of like resources. Catalog metadata is not homogenous, though. We identify three subcategories of catalog metadata:¹⁴ descriptive (or bibliographic, supporting discovery and interpretation of data), administrative (addressing rights management, physical media descriptions, encoding conventions, and so on), and preservational (to track the lineage or provenance of data, the archival requirements, etc.). There are undoubtedly many other ways of subdividing the notion of catalog metadata.

4.5 Integration Metadata

Integration metadata is metadata that makes it possible to pull together data designed or created by different organizations, where the data from all sources is intended to have the same purpose.

For example, one organization might represent information about actors by using the data elements we illustrated in Example 4-3, while a different organization might choose the approach shown in Example 4-4.

Even a casual examination of the XML in Example 4-3 and Example 4-4 reveals that the two have a great deal in common but that there are important differences as well. For example, Example 4-3 contains an element named “actors,” while Example 4-4 contains one named “Actors-and-Actresses.” Most of the data in each of the two examples can be found in the other, even though some of the data appears as an attribute in one and as a child element in the other, the names of the elements differ, the sequence of elements differ, and some data in one example (the element “released” in Example 4-4) has no analog in the other example.

Integration metadata could provide a mapping between the two designs, as illustrated in Table 4-4. (We hasten to observe that the problem of mapping between two XML vocabularies, however similarly they are structured, is significantly more complex in practice than we have space to explore here.)

Of course, several questions are left unanswered by the information captured in Table 4-4, such as the meanings of the values for the element “gender” in Example 4-3 and for the attribute “code” of the element “sex” in Example 4-4. Clearly, the table could be enhanced to add such mapping information as well. For example, it happens that the values of the element “sex” in Example 4-4 follow the practices of ISO 5218. (In fact, it is exactly this sort of problem that is addressed by the use of semantic metadata – understanding the meanings of data names and values.)

Given the kind of mapping illustrated in Table 4-4, application programmers – or, indeed, application generators – can readily integrate the data represented in the two XML documents given in Example 4-3 and in Example 4-4.

We are unaware of any existing standards meant specifically for integration metadata, but it’s worth pointing out that the creation such facilities is enhanced through the use of semantic metadata implementations, such as an ISO 11179 metadata registry.

In summary, integration metadata is information that assists in correlating data designed by different organizations or individuals.

4.6 Chapter Summary

In this chapter we have introduced the term metadata and the four forms of metadata that affect life in the information technology industry: structural metadata, semantic metadata, catalog metadata, and integration metadata.

Of these four forms, we believe that it is most urgent to learn more about structural metadata and semantic metadata. Only then does catalog metadata become an important concept. Because we are not aware of standardized facilities for creating and using integration metadata, we do not further discuss that form in this book.