Embedded Tagging

When people refer to “embedded tagging,” they are talking about the powerful ability to add structure to data elements—grouping them in various ways to provide support for organizational objectives. Knowledge workers are familiar with relational and tabular data storage using the Relational Database Management System, or RDBMS. While XML is fully capable of representing these types of data, most XML documents are stored hierarchically. An XML document is simply a text document that contains at least one XML element (pair of tags). Embedded tagging is the process of including sub-elements to the primary data element in a document. This means that any data element in an XML document can have subordinate data elements underneath it, which inherit the higher-level “container” element properties. As an example, consider a communications framework expressed in XML. The communications system uses the framework to keep track of who communicated with whom, and when the communication took place. All of this is metadata and it should be maintained using modern data management technologies. The actual content of the communication (in this case, a quick note) is stored in a separate location and is often the only portion of the system to use any sort of database at all. Figure 2.1 shows how the data and the metadata can be integrated and structured into a result.

As the need arises to have a more complete picture of a particular communication that has taken place, it would be helpful to combine the two pieces of data into a unified document. Figure 2.1 shows how the XML message is combined with the framework to produce a resulting document that contains all of the data of both. While this should be considered a reasonably advanced use of XML, it really isn’t that complex. The XML message is “embedded” inside the <content> element of the framework document. In this way, XML documents are quite like Russian nesting dolls; inside of one XML element is another, which itself contains another, and so on. With nesting dolls, there is a limit based on how small or large a doll you can actually pick up and hold, but with XML, there is no predefined limit.

Tag embedding in general simply allows XML documents to be mixed and matched, combined, taken apart, and reassembled. Just as we inserted one document into another in this example, we might also have extracted one document from another. The business value that arises from this is that no direct one-to-one mapping from an XML document to a data source or system is necessary. One document may represent several sources, or several XML documents may be combined to represent one source. The impact to data management is that data can be structured according to how it will be understood and used, rather than according to artificial platform or representational constraints.

Meta-Lanuae

A meta-language is a language that describes another language. XML is a meta-language that has several important capabilities:

Data managers understand that XML provides two specific opportunities. Organizations can develop languages for very specific tasks, tailoring their language to the requirements at hand, and they can also develop “bridge languages” to permit communication in ways that are less cumbersome than strict application of standards.

Vocabularies

The vocabulary of an XML document often refers to the concepts, elements, and attributes that are present in the document. Since XML elements and attributes are just text names, it is possible to label each item of data in a document with a name that best represents the data’s meaning or applicability. The vocabulary of a document is then the collection of all of the elements that are found in a document, along with their meanings. XML as a whole has a way to formalize and express these vocabu-laries—this is usually done in the form of a DTD (Document Type Definition) or an XML Schema, although these two are only the most recognized among many options.

Vocabularies are important because they are intricately tied to the meaning of the XML document. When converting from one form of XML document to another, it is crucial to understand the vocabulary of both documents in order to make sure that the document comparison and conversion is apples-to-apples, and not apples-to-oranges. This problem arises when similar terms mean different things in different contexts, as described in Chapter l’s example of “secure the building.” Data managers are often justifiably interested in these vocabulary reconciliation issues, since they are crucial to ironing out issues of semantics in data.

XML-Wrapped

The term “XML-wrapped” is often used as a buzzword, but at its core, it just means that the data in question has XML elements surrounding key data items, and that the document has been given some sort of XML structure. It is important to keep in mind that just because data is “XML-wrapped” does not mean that it is useful or easy to work with, just that the data is represented as XML.

Many data managers have seen quite a bit of data that is coming “XML-wrapped,” and will see even more of it as time goes on. Data managers can understand “XML-wrapped” to mean that someone along the line is attempting to pair meaning with his or her data.

 There is no effective way to identify the content of a page. HTML tags describe the visual layout of a page. Web browsers

use the tags for presentation purposes, but the actual text content has no meaning associated with it. To a browser, text is only a series of words to be presented visually.

 HTML creates problems locating content with search engines. Because there is no semantic data, there is no automatic way that search engines can determine meaning, except by indexing relevant words. Search engines have improved tremendously over the past few years, but clever search algorithms still occasionally fail, and are no substitute for representing data properly.

 Many capabilities require special extensions to HTML. In order to perform more complex functions such as dynamic content, special extensions are needed that don’t necessarily have much in common with HTML. For example, ask your local web developer, and you may find that he or she is familiar with a number of different computer languages, all of which have different syntaxes that must be cobbled together to produce the desired effect. Examples of this are DHTML, JavaScript, CSS, and others.

 HTML has encouraged “bad habits.” In the early days of HTML, browsers and other software that interpreted HTML came with a variety of different exceptions and special cases to correctly interpret badly written HTML that didn’t conform to the standard. As users got accustomed to this, taking shortcuts in HTML became the norm, to the point that the standard has warped slightly. Allowing bad habits in HTML has had the strange side effect of making the task of parsing what was originally a very simple markup language into an incredibly complex task.

Tags Must Come in Pairs

In the XML vernacular, a tag or an element is the marker that is put around a piece of data. It effectively acts like parentheses in writing, showing the reader where some comment or piece of information starts, and where it ends. Figure 2.2 shows how XML doesn’t have a tabular or field structure like other data representation formats. In other formats, where the data begins and ends is largely a function of where it falls in a grid-like structure. Because there is no strictly enforced grid-like structure in XML, data must be delimited clearly. There are no exceptions to this rule, although in some XML documents you may see what are called “singleton” tags—these appear as XML tags that contain a slash (“/”) character after their name. This is XML short hand for an opening and a closing tag; it just means that the implied content within the tag is empty. Singleton tags are not exceptions to the rule that all tags must come in pairs; they are simply a shortcut in the case where the tag contains no content. In the example provided here, the singleton tag for “name” is used—note the slash before the end of the tag. This is the same as expressing the “name” tag as <name first=“David” last=“Allen”></name>.

Tag Pairs Can Be Nested Inside One Another at Multiple Levels

The first rule, that tags always come in pairs, essentially acts to identify the XML concept of a piece of data. The second rule, that tags can be nested within one another, is what provides XML documents with their real structure. If one tag is nested within another, we can say that there is a “parent” tag (the outer tag) and a “child” tag (the inner tag). When this principle is applied to complex data structures, the result is an XML tree. The very first element in an XML document can be seen as the root of the tree, with various branches corresponding to the child tags of the first element. Those sub-tags may in turn have their own child tags, each of which represents further branching of the tree.

There are several reasons why this hierarchical structure is used. When documents are structured in this way, it allows software to treat the entire document (or subsections of it) as a tree internally. Trees are data structures that are extremely well understood in computer science terms—there are already a multitude of algorithms in existence that can effectively process trees in a variety of ways. This allows a large body of previous research on trees to be applied to XML documents. But that is a low-level technical reason. At the user level, it permits easy representation of

common parent-child relationships in data. It also allows related data items to be “grouped” together under a common parent tag for semantic reasons. In the relational model, this logical grouping of related pieces of data is often done at the record level, with one record holding a number of different data points that logically hang together. Figure 2.3 illustrates a conversion between tabular data and XML-wrapped data. In XML documents, that might translate into an XML structure where each record of data is captured as a “chunk” of XML with an enclosing tag, such as the “flight” tag seen in this example.

One interesting point about the hierarchical nature of XML that rarely gets mentioned is this: While there are established methods of converting tabular and relational data structures into hierarchical structures, the opposite is not true. Converting hierarchical structures into relational or tabular data can be extremely difficult, not because the hierarchical structure is somehow more powerful than the relational form of data storage, but simply because of differences between the two models. Converting from one model to the other is a challenge often faced when organizations want to store XML data in relational databases. In later chapters, we will discuss XML databases and how their storage model may be better in certain cases.

XML Is Unicode Based

Unicode is a relatively new form of encoding characters in computers. The need for Unicode resulted from the recognition that the primary standard for representing characters (known as ASCII) was extremely limited and basically only allowed representation of characters from Latin alphabets. As computers crossed more international boundaries and came to be used in almost every human language on the planet, it became desirable to have a single way of representing characters that could cover languages that do not necessarily use a Latin alphabet. In short, Unicode provides a unique number for every character, no matter what the platform, no matter what the program, no matter what the language. ^**

XML uses UTF as the default character encoding for all documents. UTF in turn is a way of representing the numbers and characters from Unicode as bytes that can be stored. This basic feature of XML makes a lot of sense when examined together with XML’s other features. It certainly wouldn’t do to create a generalized data representation markup language that was unable to express characters from different languages. XML’s use of Unicode makes it easy to exchange documents over national and cultural boundaries, and does not require the romanization of foreign words. XML element names, attribute values, and document data can be specified in a way that is easiest for the author and is still interoperable, making XML truly an international format.

Designing Your Own Language

XML allows element names to be whatever the author prefers. Data publishers are free to refer to pieces of data in their own terms. But with this capability comes some responsibility. In order for a defined language to make sense, it too must conform to several rules that help give the language meaning for other computers and software packages. It wouldn’t do to have everyone choose their own unique set of 20 different elements, mixing them freely and in any order. The confusion that would result from this would be analogous to people randomly ordering the words in their sentences, perhaps occasionally mixing in words from other languages. The results might sometimes be humorous, but would not often be of much use. So the definition of any language within XML should include the following three pieces of information that form the rules of the language.

These rules are not strictly required. It is possible to use XML documents without following any of these rules, but it is rarely a good idea. If XML documents could be thought of as cities, with individual data items as citizens, then these rules are like a sheriff and a court system. While it’s technically possible to do without them, nobody will like the resulting anarchy.

Figure 2.4 illustrates how creating a list of allowed tags in the document is similar to writing a dictionary for the language. It lays out which terms can be used. Words that are in this dictionary make sense in the language, just like the words in an English dictionary are understandable to speakers of English. The nesting structure of the defined tags must also be defined, to give some meaning to the collection of “words” in the language. This is similar to grammar rules in the English language. Speakers of English cannot simply order words in any way they want, since result it would sentences in garbled!

In this example, we see two documents that contain identical data, with different nesting conventions. On the left of the figure is a standard representation with a “route” tag clearly specifying that the flight goes from Washington to New York City. The right side contains the same data, but it is rearranged, so it has a different meaning. Without solid and enforced nesting rules, this second document might be interpreted as a flight with no departure time that goes from one place named “559” to another place named “9:30”!

As an illustration of how XML itself is a “meta-language,” consider how most of the time when new XML languages are created, the tags that are allowed in the document and the nesting rules that must be in place are themselves expressed in the XML document. The two major XML architectural components used to accomplish this are referred to as DTDs and XML Schemas. Later, we will go into more depth about these particular components and how they fit into the overall XML component architectural model.

The last part of defining any created XML language is specifying how the tags should be processed. At times, it may be important to deal with certain data items before others, or even to skip processing some items depending on the values of others. The need for formalized rules about how the tags are processed varies with the complexity of the language that is created. The more optional tags and special cases that exist in the language to represent particular data structures, the wiser it is to lay out a set of rules that describe how the language is processed, and what that processing means in terms of the phenomenon the data represent.

XML is to data what Java is to programming languages. The excitement over Java was that you could “Write it once. Play it back anywhere!” With XML capabilities you can “Wrap your data once in XML and utilize it anywhere!” XML represents what Mike Hammer terms a disruptive technology—that is, a technology that has the potential to impact-related processes (Hammer & Champy, 1993). XML has the potential to simplify our data management operations in much the same way that EDI simplified data communications exchange.

HTML versus XML

With regard to the way the rules of the two markup languages work, there are a number of differences between HTML and XML. In HTML, there is a fixed set of tags that are allowed to be included in HTML documents, while all other tags are ignored. In XML documents, on the other hand, tag formatting can be specified for any application—there is no fixed set of tags that are acceptable in base XML. While tag limitations can be put in place on XML documents for particular applications, as discussed earlier, there are no inherent limitations on XML itself as there are in HTML. Another difference between the two concerns how browsers interpret the data. HTML documents are interpreted in terms of a visual layout, whereas XML documents can be interpreted in a number of different ways according to what data is actually in the document. XML technologies exist that allow browsers to display data documents in a visually pleasing way just as HTML would have done, while at the same time preserving the metadata and semantic context of the data.

In an excellent article, two of XML’s creators describe two of the motivations that led them to develop XML (Bosak & Bray, 1999). The first was Internet congestion. Using HTML required updating a quality field by accessing a server and reloading the page. With XML, more work can be accomplished on the client side. The second motivation was to address information location difficulties. With HTML, you cannot search for any-thing marked as “price” on the web beyond searching for the string “price.” XML-based data is “self describing,” which means that devices will be more capable of local analysis of information. As we shall see, the goals of XML’s creators directly complement several data management objectives.

Now that we have taken a look at some of the important terms to understand with XML and what rules of the road need to be followed, it’s time to take a look at examples of how XML is actually used in practice.

 One way is to increase the number of applications that can understand the XML-wrapped finance data. The parser itself does not understand the data, but allows the application to understand it by using the metadata.

 The other way of expanding XML capabilities is to increase the number of tags that each application is able to “understand.” Adding new sets of tags might be as simple as replacing a table. Technically, all that is happening is that you are expanding the rules that each application understands and trying to maintain them so each application might implement the same rules using different technologies. Adding “understanding” of XMLwrapped data from new applications is done using a single source of entry and maintenance.

 Reward rates (within and across partners)

 Distinguishing different status program participants

 Integrating rotating promotional offers

 Integrating with airline booking systems

 Integrating with affiliated airline web sites

 Printing accurate and timely statement packages

1. Split the system into smaller chunks. Enable better understanding of it using the “divide and conquer” approach.

2. Allow communication between areas of the system using an XML message bus. By using this bus, components could talk to one another in the way seen in Figure 2.8.

3. Reengineer the components simultaneously using modern methods and technologies. By breaking down the system into parts, different groups could work on each part individually, speeding progress.

The Challenge

MedCorp approached Data Blueprint about converting about 15,000 legacy HTML documents that contained internal documentation into an XML format. MedCorp actually stored their documents in a number of different formats, but most often referred to the HTML edition. One problem they were experiencing was that the format used to create the print edition of their documentation was not the HTML edition, so that when corrections were made to the HTML edition, they often didn’t find their way back to the original. With several different formats floating around the organization, change management was rapidly becoming a nightmare with the growing need for resources to keep all editions in sync enough for them to be useable.

Display of the documents was also an issue. Ideally, MedCorp wanted to be able to deploy these documents to other platforms as well, such as wireless phones or PDAs for salesmen in the field, but they weren’t about to try to manage yet another edition of the documentation, given all the problems they were already having. What was really needed was a way to store the documents centrally in one place, where all modifications would be made. The actual presentation of the documents could then be regenerated automatically from the source documents by pressing a button as the need arose. Enter XML. One of its most basic strengths is write it once and read it everywhere (see Figure 2.12).

The Process

The project consisted of different parts, which would in the end yield a master set of documentation for MedCorp that could be easily managed, edited, and repurposed. The steps of the conversion process included

Important Outcomes and Insights

There were several important project outcomes:

The last one is hardly typical but bears repeating. Earlier recognition of unanticipated business requirements would have made it possible to increase efficiency. Instead, many business requirements surfaced after conversion freezes—other business requirements were assumed; for example, the need to create new materials, new linking and navigation structures within XML, and code retention from the original HTML through to the XML.

Given this last bullet point, the real lesson was that it is important to identify all architectural decisions before the process begins, to prevent the substantial amount of work that it takes to change those decisions and assumptions midstream. When organizations underestimate the value of the baseline architecture, they are more likely to neglect proper planning in the initial stages, and later in the project they may want to change some of their earlier faulty assumptions. One prominent example that required a work-around was the determination of the actual physical XML file sizes. Selection of a physical size below a certain “object size” required the development of a programmatic work-around. The difference forced by the rework is illustrated in Figure 2.14.

To wrap up this example, we offer a few important points on the process of converting data from its original format to XML:

Metadata Identification and Extraction

The gigantic document itself is fairly well structured, laying out the hierarchy of functions and processes and how they relate to each other. Still, there is no way to effectively take advantage of this structure since searching through the documents for particular strings is slow, and often returns “false positives” of a particular term used in a context other than what was meant. (For example, searching for a particular function might first find it in the context of being a subordinate of some other function, rather than finding its actual definition.) Ideally, the document would be extensivelycross-referenced, allowing the reader to jump back and forth from any point to any other point according to what was relevant at the time. Since the functions and processes are arranged hierarchically, it would be useful to be able to move up the tree and down it, but also across it, enabling the analyst to take a look at the sibling nodes in the tree for any particular item.

What is needed here is effective management and application of the metadata, since that is really all the document is. XML in and of itself does not automatically do this, but what it does provide is a solid framework in which the concepts of interest can be articulated and exploited. The first step is to take the original document and use its inherent structure to build a corresponding XML document that marks each element of data in the document appropriately. The purpose of this stage is to attach names and meanings to individual pieces of data, rather than having them float around in amorphous blocks of text. The second stage then is to exploit this newly created structure to make the document more readable and searchable, so that the analysts can better understand the legacy system they are working with and spend more of their time solving problems, and less of it working around the annoyances of the original manual’s inadequate data representation.

Figure 2.16 shows a conversion example. It is it clear that the text of the document implies a certain structure to the data without actually specifying that structure. Converting the information into XML allows particular data items to be explicitly referred to according to their metadata. For example, when searching the original Word document for the term “Function_5,” many dozens of matches might have been found—one where Function_5 was defined, one where it was listed as a child of another function, yet another where it was listed as a reference from some other process. By wrapping the data in XML, the analyst can essentially “ask” the document more specific questions using the metadata—“Show me the item that refers to Function_5 as a child,” instead of “Show me everything about Function_5.”

Presentation

Once the data has been extracted from its original semi-structured format, some way of presenting it to users must be devised. The original Word document itself was in something of a presentation format in that the document was page oriented, and had the concept of particular fonts and sizes attached to portions of text. Even though the original document was meant for presentation, now that the data has been moved to an XML format, a number of additional presentation options are available. Normally in Word or HTML format, the presentation would have been hard-wired and would not be modifiable without a serious reworking of the document. Putting the data into an XML format helps us focus on the data of the document, and to see the presentation as arising as a consequence of the relationships inside of the data, instead of seeing presentation as a unique way to simply throw the data up onto the screen.

Access to the metadata that is clearly marked in the XML document opens up new presentation possibilities. The original Word document contained a seemingly endless procession of blocks of text, each describing a particular function or process. With the addition of contextual data in the XML document comes the ability to have the presentation layer take advantage of it. In this case, the presentation of the document was completely reengineered to take advantage of the data about the relationships between the data items. Instead of presenting “chunks” of information about each process, the data was rearranged into a family tree, and navigation became possible up, down, and across the tree. The processes and functions in the document were no longer thought of as blocks of text, but as valuable links between other processes and functions.

Further, since the complete list of what constitutes a function or a process was available, it became possible to have an “index” into the document at any time. From within a browser, the user could view all of the data about a particular function or process, while also having links to related information, and an index on the left-hand side of the screen showing all other data items that were in the same family. The goal of presentation should be to provide a number of different ways of looking at the data wherever possible, since the more options the information consumers have in terms of drilling into the data, the more likely they are to find what they need quickly. The only way these different options can be provided is by understanding the relationships inherent in the data, and how everything fits together. Understanding the relationships in the data and for-malizing that understanding is the process of identifying and documenting the metadata.

Converting Documents to XML

For structured documents like the one in this example, the process of converting the data from its original form to XML is relatively straightfor-ward. Particular items in predetermined places in the source data correspond to well-known data items. There are a number of up-and-coming technologies that allow unstructured data in a number of different formats to be converted into XML documents. Generally, there are few problems with converting tabular and relational data into XML; most of the work in that case deals with finding the appropriate tags and nesting rules that form an overall coherent data model. However, in the case of unstructured information, the source data can be quite a bit more unruly. Take for example most current HTML pages, which describe how the page looks, but nothing at all about what is actually in the page. The analyst can either choose to use one of several widely available software packages to parse the data out of the unstructured format and wrap it in XML, or develop a small custom program to extract particular pieces of information out of documents. Typically, the amount of effort that is required to convert documents is directly related to how structured the document already was. For highly structured documents, less effort will be needed, while for very unstructured documents, the process is far more challenging.

In the context of extracting data, we sometimes refer to the idea of “metadata markers,” or particular “flags” seen in unstructured data that would normally signify to humans that a particular type of data is coming. In the above example, the word “Function” followed by a colon might act as a metadata marker for the function data. Converting unstructured information to XML is largely an exercise of locating the appropriate metadata markers, extracting the data that it is marking, and wrapping the result in a more appropriate XML tag. The XML tags in a document are themselves basically metadata markers. The advantage to using them over unstructured formats is that they were designed expressly for that purpose, while unstructured formats generally have ad hoc metadata markers that are frequently misleading or incomplete. Figures 2.17 and 2.18 illustrate the conversion outcome.

Overall, this example touches on several different issues. For data managers, it illustrates how XML can be useful for presenting, storing, and manipulating metadata about systems. It also provides another example of how document conversion can be accomplished, and how this conversion is not always done from source data, but sometimes from semi-structured information that represents system documentation. XML is a valuable tool for data managers in many different phases of system migration projects. In this example, we refer to the metadata of the system being migrated; in other cases, XML may be involved with the actual data being migrated, it might be used as a method of communication between the new system and other systems, or in any number of other areas.