Chapter 13. XML

Because computers are computers and people are people, they generally have different requirements when it comes to getting their data into a usable format. XML is an attempt to arrange data in a structure that is usable for both people and software.

XML has really come into vogue in recent years, but its roots are quite old. It’s derived from SGML (Standard Generalized Markup Language), as is HTML (cousins!). SGML in turn came from GML (Generalized Markup Language), a “metalanguage” (a language that describes another language) designed by IBM back in the 1960s. So, blame IBM if you want to, but either way, you will come in regular contact with XML as you develop .NET applications.

I might as well tell you right from the start: either you will love XML, or you will hate it, but probably both. It’s a strange beast, this XML is, as you would expect from any acronym that takes letters from the middle of the words it represents (eXtensible Markup Language). XML represents an alphabet of data manipulation technologies, an alphabet that strangely has seven Xs. But enough of the teasing; let’s extend our understanding of this basic .NET technology.

What Is XML?

XML is nothing more than a data format that is both human readable and machine readable. Have you ever tried to open a Microsoft Word document with Notepad? Good luck (see Figure 13-1). Although you can usually sift out the main text of the document, most of what you see is gobbledygook. That’s because it is in a proprietary binary format. It’s proprietary because, frankly, you shouldn’t be poking your fingers in there. That’s what Microsoft Word is for. And it’s binary because you can store a lot of information conveniently in a little bit of disk space. With such a file, I can store my data anyway I choose. In fact, I can write my data out willy-nilly, and not have to get permission from anyone, because it’s mine, mine, all mine.

Figure 13-1. This chapter in Notepad

Binary files are great for storing any kind of data: numbers, strings, base-64 encrypted images, streams of networking data chatter, anything. The problem is that unless you know the exact structure that you used to write it out, there is little chance of ever getting the data back. This is good if your goal is secrecy, but if you ever need to share that data with another person or program, or worse yet, debug the output from your errant program, you’re in for a tough time. If one little byte gets messed up, the whole file might be useless.

There are, of course, other ways to store your data. For files that store records of data, tab-delimited and CSV (comma-separated values) files provide a convenient transfer medium, in a more human-friendly format. For instance, consider this data from Microsoft’s sample “Northwind Traders” database, stored as comma-separated values.

Now that’s better. This data is pretty easy to understand. Each piece of data is grouped by commas, and the first row indicates what each column contains. And the best part is, many programs already know how to read files in this format. If you save this data in a text file with a “.csv” extension, and open it in Microsoft Excel, the data automatically appears in columns as expected.

But it could be better. For instance, what do those “652” and “9874” values refer to anyway? And is it correct that the unit price of Aniseed Syrup is “On Sale?” Sure, I can load this data into my program, but can I do anything with it? At least it’s an easy read for both people and computer programs, and isn’t that what I said XML was all about?

Well, yes. Although XML includes rules and features that make it more flexible than your average text data file, it’s not that different. For all the hype, XML is just a way of storing data. Any of the fancy-schmancy XML traits discussed in this chapter could be performed easily with data stored in more simple text or binary proprietary formats. In fact, it is often quicker and more convenient to develop using a proprietary format, because your data will contain exactly and only what you need, without any fluff.

That being said, XML does include many aspects that make it a strong contender when considering a data format.

• It’s straightforward to read. Each data element includes a type of title. Good titles make for good reading.
• It’s easy to process. All data includes starting and ending tags, so a program can process the data without much effort. And one bad element won’t necessarily ruin the whole file.
• It’s flexible. You can store any type of data in XML. It is just a text file, after all. If you have a certain XML file format used in version 1 of your program, and you add features to it in version 2, you can do it in a way that still allows version 1 programs to use version 2 files without breaking.
• It’s self-describing. XML includes several methods that let you describe the content of a given XML file. Two of the most popular are: DTD (Document Type Definition) and XSD (XML Schema Definition). You use these tools to indicate exactly what you expect your data file to contain. Additionally, XML allows you to embed comments in the content without impacting the actual data.
• It’s self-verifying. There are tools available, including tools in .NET, which can confirm the integrity and format of an XML file by comparing the content to the associated DTD or XSD. This lets you verify a file before you even process it.
• It’s an open standard. XML has gained widespread acceptance, even across divergent computer platforms.
• It’s built into .NET. This is going to be the biggest reason for using it. In fact, you won’t be able to get away from XML in .NET, even if you try. It’s everywhere.

But there’s bad news, too.

• It’s bulky. XML content contains a lot of repetitive structural information, and generally lots of whitespace. You could abbreviate many of the structure elements, and remove all the whitespace (XML doesn’t require it), but that would remove the human-readable aspects of the data. Some platforms, such as cell phone browsers, like to keep data small. XML is anything but small.
• It’s text. Wait a minute, this is a good thing—most of the time. Sometimes you just need to store binary data, like pictures. You can’t really store true binary data in an XML file without breaking one of the basic rules about XML: text only! Often, binary data is encoded in a text-like format, such as base-64 (which uses readable characters to store binary data).
• It’s inefficient. This comes from having data in a verbose semi-human-readable format, rather than in terse, compact binary form. It simply takes longer for a computer to scan text looking for matching angle brackets than it does to move a few bytes directly from a lump of binary data into a location in memory.
• It’s human readable. There are not many secrets in an XML file. And while you could encrypt the data elements in the file, or the entire file for that matter, that would kind of defeat the purpose of using XML.
• It’s machine readable. If you are expecting the average Joe to pick up an XML printout and read it in his easy chair, think again. XML is not appropriate for every type of data file.
• It’s not immune to errors. As I keep repeating, XML is just a text file. If you open it in Notepad and let your five-year-old pound on the keyboard, the content will have problems. XML is not a panacea; it’s just a useful file format.

The XML Rule

Before we look at some actual XML, you need to know The Rule. You must obey The Rule with every piece of XML text you write.

That’s it. Don’t forget it. Obey it. Live it. I’ll explain what it means later.

XML Content

There’s no better way to learn about XML then to start looking at it. If you’ve never used XML, but you’ve written some HTML, then this should look somewhat familiar.

Some Basic XML

Here’s a simple chunk of XML for you to enjoy.

Hey, I didn’t say it was going to be interesting. As I mentioned before, it’s just data, but it is useful data, and here’s why.

• It’s obviously XML. This is clear from the first line, which always starts with “<?xml....” This line also indicates the XML version number, which tells XML processing routines (parsers) to adjust behavior if needed. That’s foresight.
• It’s structured. XML is a hierarchical data structure. That is, you can have data elements embedded inside of other data elements to any depth you want. Every element is bounded by a set of tags. In this sample, the tags are hello, there, world, totalCount, and goodbye. Tags always appear inside of <angle brackets>, and always appear in pairs, as in <hello>...</hello>. (This is where The Rule, “If you open it, close it,” comes in.) Don’t forget the “/” just before the tag name in the closing bracket. This syntax lets you organize your data into specifically arranged named units. For tag pairs that have nothing in between them, you can use the shortened syntax “<tagname  />,” as I did with the goodbye tag. By the way, XML tags are case sensitive, so type carefully.
• It’s readable. It’s human readable, thanks to all the whitespace, although you could remove it all and still have XML. It’s also computer readable because of the consistent use of tags.
• It’s a single unit of data. All XML files have a single root element in which all other elements must appear. In the sample, <hello> is the root element. Once that element is closed (through its ending tag), you can’t add any additional elements. Nope. Nada.
• It’s got comments. See that “<!--...-->” line? That’s a comment. You can stick comments here and there just like they were free-floating tags.
• It’s got attributes. XML supports two varieties of data: real data and attributes. Real data values come between the in-most tag pairs, as with think XML and 694.34 in the sample. Attributes provide extended information about the tags themselves. I included an attribute named target in the world element. The content of all attributes must be in quotes. I could have made this attribute a sub-element instead, and a lot of people do. There is disagreement among programmers as to when data should be an element or an attribute. Let your conscience be your guide.

So there you have it—some clean, clear XML data.

Some Basic—and Meaningful—XML

Let’s see what that comma-delimited data from Northwind Traders I listed previously could look like in XML.

Moving the data to XML has greatly increased the size of the content. But with an increase in size comes an increase in processing value. I was immediately able to get some benefit from the hierarchical structure of XML. In the original data, supplier was just another column. But in the XML version, all the data is now grouped into supplier sections, which makes sense (at least, if that is how I was planning to use the data).

You can also see that I followed The Rule. Every opening tag has a matching closing tag. Whatever you do, don’t forget The Rule.

Now, you’re saying to yourself, “Tim, I could have grouped the data by supplier once I loaded the comma-delimited data into my program.” And to that I say, “You’re right.” I told you that XML was just another data format. By itself, the XML content is not all that sexy. It’s really the tools that you use with your XML data that make it zoom. Because XML uses a consistent yet generic structure to manage data, it was a snap to develop tools that could process consistent yet generic data in ways that look interesting and specific.

What About the Human-Readable Part?

One of the tools used with XML is the double acronym XSLT, which stands for XSL Transformations (XSL stands for eXtensible Stylesheet Language). XSLT is a hard-to-use scripting language that lets you transform some XML data into whatever other data or output format you want. It’s just one of a handful of XSL-related languages created to manipulate XML data in complex ways. Ready for some hands-on XSL fun? Take the useful chunk of XML listed previously (the <productList> sample), and replace the first “?xml” line with the following two lines:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="hello.xsl"?>

Save all of that beautiful XML text to a file on your desktop as hello.xml. Next, put the following XSLT script into another file on your desktop named hello.xsl. (Notice that I break one line with the marker so that the content could fit in this book. Please don’t really break the comma-separated list on that line in the file.)

I told you it was hard to use, and even harder to look at. OK, now for the show. I have Internet Explorer 6 installed on my system, but this should work with most current browsers. Open the hello.xml file in your browser, and voilà, the following beautifully formatted text should appear.

Now that’s more like it. XML and XSLT together have made this advance in data technology possible. (I did cheat a little in this example. You will notice the <BR/> entries in the XSLT script that don’t appear in the final output. I added these just to make it look right in your browser.) But seriously, while I was able to generate a comma-separated data set with XSLT, more common tasks for XSLT include generating nicely-formatted HTML based on XML data, or generating a new XML document with a specific alternative view of the original data. How does it work? Basically, the <xsl:template> elements tell the parser to look for tags in the XML document that match some pattern (like “supplier”). When it finds a match, it applies everything inside the <xsl:template> tags to that matching XML tag and its contents. The pattern specified in the “match” attributes uses an XML technology called XPath, a system to generically search for matching tags within your XML document.

Sounds confusing? Well, it is, and don’t get me started on how long it took to write that short little XSLT script. XSLT scripting is, blissfully, beyond the scope of this book. Of course, there are tools available to make the job easier. But XSLT is useful only if the XML data it manipulates is correct. You could write an XSL Transformation to report on data inconsistencies found in an XML document, but it won’t work if some of the tags in your document are misspelled or arranged in an inconsistent manner. For that, you need another advancement in XML technology: XSD.

XML Schemas

XSD (XML Structure Definitions) lets you define the schema—the “language” or “vocabulary”—of your particular XML document. Remember, XML is a wide-open generic standard; you can define the tags any way you want and nobody will care, at least until you have to process the tags with your software. If they aren’t correct, then your processing will likely fail. XSD lets you define the rules that your XML document must follow if it is to be considered a valid document for your purposes. (DTD, or Document Type Definition, is a similar, though older, technology. It’s widely support by XML tools, but it is not as flexible as XSD. There are also other schema definition languages similar to XSD, but because XSD is built right in to .NET, we’ll focus on that.)

XSD schemas are every bit as endearing as XSLT scripts. Let’s create an XSD for our original sample <productList> XML listed previously. First, we need to change the top of the XML to let it know that an XSD schema file is available. Change this:

<?xml version="1.0"?>
<productList>

to this:

These directives tell the XML parser to look in hello.xsd for the schema. They also define a namespace; more on that later. The hello.xsd file contains the following schema.

It looks nasty, doesn’t it? Actually, it’s more straightforward than XSLT. Basically, the schema says that for each element (or “tag” or “node”) in my XML document, here are the sub-elements and attributes they contain, and the data type of each of them. You can even create your own pseudo-data types (actually, limiting factors on existing data types), as I did with the “YesOrNoType” data type, which limits the related value to the strings “Yes” and “No.”

You can look at the XML file with the attached XSD schema in your browser, but it wouldn’t be all that interesting. It just shows you the XML. But schemas will be useful when you need to assess the quality of XML data coming into your software applications from external sources.

XML Namespaces

The product list in the XML shown earlier is nice, but someone else could come up with a product list document that is just as nice, but with different naming and formatting rules. For instance, they might create a document that looks like this:

The data is all the same, but the tags are different. Such a document would be incompatible with software written to work with our original document. Running the document through our XSD would quickly tell us that we have a bogus data set, but it would be nicer if something told us that from the start. Enter namespaces. Namespaces provide a convenient method to say, “This particular tag in the XML document uses this XSD-defined language.” Notice the start of the XSD schema shown previously.

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

This line sets up a namespace named xs by using the xmlns attribute. (The “:xs” part tells XML what you want to call your namespace.) The value of the attribute is a URI (Uniform Resource Identifier), just a unique value that you are sure no one else is going to use. Typically, you use a web site address for your own company; the web site doesn’t have to exist. You could even put your phone number there, just as long as it is unique.

The most common way to use a namespace is to prefix the relevant tags in your XML document with the new namespace name, as in xs:schema instead of just schema. This tells the parser, “If you are checking my syntax against an XSD schema, then use the one that I defined for the xs namespace.” You can also use a “default” namespace for a given element and all its descendants by including the xmlns attribute in the outermost element. Then all elements within that outermost element will use the specified namespace. I used this method in one of the preceding examples.

<productList xmlns="SimpleProductList"...

For basic XML files that will be used only by your program, you may not need to bother with namespaces. They really come in handy when you are creating XML data that uses some publicly published standard. There are also instances where a single XML file might contain data related to two or more distinct uses of XML. In this case, different parts of your XML file could refer to different namespaces.

As with other parts of the XML world, XSD and namespaces are not all that easy to use, but they are flexible and powerful. As usual, there are tools, including tools in Visual Studio, which let you build all of this without having to think about the details.

As I keep saying, XML is just data, and if your program and data don’t understand each other, you might as well go back to chisel and stone. XML and its related technologies provide a method to help ensure your data is ready to use in your application.

Using XML in .NET with System.Xml

XML is used internally in the .NET Framework to pass all types of data around. From ADO.NET to Web Services, XML is the data format of choice for shuffling information to and from Framework services. Fortunately, we aren’t reprogramming the entire Framework, so most of that isn’t really important to our understanding of .NET development. But some Framework classes expose XML services right out there for the programmer to use.

Because XML is no fun to manage as a big chunk of text, .NET includes several classes that manage XML data. All of these tools appear in the System.Xml namespace and its subordinate namespaces.

• System.Xml. The main collection of XML-related classes.
• System.Xml.Schema. Classes that create and use XSD schemas.
• System.Xml.Serialization. Classes that read and write XML documents via a standard .NET stream.
• System.Xml.XPath. Classes that implement the XPath technology used to search XML documents.
• System.Xml.Xsl. Classes that enable XSL Transformations.

The features included in each class tie pretty closely to the structure of the XML, XSD, and XSLT documents themselves. They include a whole lot of features that weren’t covered previously, because there are gobs of ways to manipulate XML data.

The Basic XML Classes, Basically

The System.Xml namespace includes the most basic classes you will use to manage XML data. An XmlDocument object is the in-memory view of your actual XML document:

Dim myData As New System.Xml.XmlDocument

Your document is made up of declarations (that <?xml...?> thing at the top), data elements (all the specific tags in your document), attributes (inside of each starting element tag), and comments. These are represented by the XmlDeclaration, XmlElement, XmlAttribute, and XmlComment classes, respectively. Together, these four main units of your document are called nodes, represented generically by the XmlNode class. (The four specific classes all inherit from the more basic XmlNode class.) Usually, when you build an XML document by hand in memory, you use the individual classes like XmlElement. Later on, when you need to scan through an existing document, it is easier to use the generic XmlNode class.

Let’s build a subset of our sample XML product data.

Declare all the variables you will use, and then use them.

It really works, too. To prove it, put this code in the click event of a button, and end it with the following line:

products.Save("c:products.xml")

Run the program and view the c:products.xml file to see the XML product data. There are many different ways to use the XML classes to create an XML document in memory. For instance, although I used the SetAttribute method to add attributes to the supplier and product nodes, I could have created separate attribute objects, and appended them on to these nodes, just like I did for the main elements.

So, this is nice and all, but what if you already have some XML in a file, and you just want to load it into an XmlDocument object? Simply use the XmlDocument object’s Load method.

For those instances where you just want to read or write some XML from or to a file, and you don’t care much about manipulating it in memory, the XmlTextReader and XmlTextWriter classes let you quickly read and write XML data via a text stream. But if you are going to do things with the XML data in your program, the Load and Save methods of the XmlDocument object are a better choice.

Finding Needles and Haystacks

In our sample data, all of the products appear in supplier groups. If we just wanted a list of products, regardless of supplier, we ask the XmlDocument to supply that data via an XmlNodeList object.

For a more complex selection of nodes within the document, the System.Xml.XPath namespace implements the XPath searching language, which gives you increased flexibility in locating items. The Visual Studio documentation describes the methods and searching syntax used with these classes.

Schema Verification

An XmlDocument object can hold any type of random yet valid XML content, but you can also verify the document against an XSD schema. If your XML document refers to an XSD schema, includes a document type definition (DTD), or uses XDR (XML Data Reduced Schemas, similar to XSD), an XmlReader, when configured with the appropriate XmlReaderSettings, will properly compare your XML data against the defined rules, and throw an exception if there’s a problem.

XML Transformations

Before we move on to the project code, let’s look at XSL Transformations in the .NET classes. It’s no more difficult than any of the other manipulations of XML. Just as there are many ways to get XML source data (from a file, building it by hand with XmlDocument, and so on), there are many ways to transform the data. If you just want to go from input file to output file, the following code provides a quick and efficient method. XSL Transformation is generally a performance-poor activity. But you can speed up performance by putting your source XML document into an XPathDocument object instead of a plain XmlDocument object.

Summary

There are a lot of useful features in the various System.Xml namespaces, and you can manage complex data in very effective ways. It’s not always the most efficient way to manage data, but if you have structured hierarchical data, it may be the most direct and clear method.

Although XML lurks everywhere in the .NET Framework, and in all applications written using .NET, you could actually write large and interesting applications without looking at a single line of XML content. Even if your application needs to interact with XML content, the classes in the System.Xml namespace effectively shield you from the text manipulation nightmare that is XML.

XML is a very useful and flexible data format that is here to stay. Although it will always lack the speed of more compact data standards, its benefits are numerous. There is a move to introduce a “binary XML” format as a standard. Currently it’s meeting with much resistance from the standards and development communities. As a .NET programmer, you won’t really have to worry about that. If binary XML does become a standard, you will likely continue to use the same classes and methods introduced in this chapter, with the possible addition of an OutputFormat (“Text” or “Binary”) property.

Project

The administrator of the Library system will want to see statistics and information at a glance, or run various reports that provide meaningful summary or detail views of system data. As a programmer, I could try to add every conceivable type of report that the user may need, but I have learned from experience that this is not possible. Users always want the moon, usually in the form of some weird esoteric report that I know they will use once and never look at again (although they will call once a year asking for the same report to be written again). I don’t like recompiling and re-releasing the entire application every time a user needs a new report. Instead, I keep the reports outside the application, stored as separate programs. Then, from one form in the main application, I make all of those external reports available in a nice convenient list.

To implement this generic feature, I use a report configuration file, a simple XML file that contains information on the available reports, and how to run them. I want my selection list to have indented items, so that I can visibly group reports for convenience. To do this, I will make my XML file into an unlimited depth hierarchy, with each level representing a further level of displayed indent. For instance, let’s say I wanted the following outline of reports (with report group titles in bold).

Detail Reports
     Daily Report
     Monthly Reports
          Monthly Value
          Monthly Inventory
Summary Reports
     Inventory Summary

The XML configuration would follow this structure.

Of course, this is greatly simplified (not to mention non-compliant) XML. In addition to the hierarchy, I also want to include support for a variety of reporting methods. To keep things simple, the Library project will include three methods of initiating reports.

1. Built-in Reports. The application includes a limited number of reports that are permanently built in to the main application (assembly). The reports are numbered, starting from 1, and at this time I have five reports in mind. The designer of the XML configuration file can choose to include these in the display of reports or not by simply including or not including them in the file. In the absence of a configuration file, these reports will appear in the list by default. In addition to the report number (1 to 5), each entry has a display text and a long description.
2. Application Reports. These reports are separate and distinct EXE files, and are started via standard application initiation methods. Each entry includes a display text, the full path to the application, optional arguments, a flag to pass the identity of the user initiating the report, and a long description.
3. URL Reports. These reports are simply calls to web pages, or any other valid URL. For instance, you could include a report entry that does a “mailto:” to the local organization’s help desk. Each entry includes the display text, the URL itself, and a long description.

The project activities in this chapter involve both coding and documentation of the new external resource (the XML file format).

Update Technical Documentation

First, let’s add clear documentation on the structure of the XML configuration file. There is no easy way to communicate the structure of an XML file to an ordinary user. Although such documentation is a requirement, hopefully the application will also include a tool to let an administrator build the configuration file. Such a program, sadly, is not included in this book’s project. It is left as an exercise for the reader. (I always wanted to say that.)

This technical description appears in the Technical Resource Kit document, originally developed in Chapter 4, “Designing the Database.”

Create Report Entry Class

With .NET’s ability to store whole objects as ListBox items, we can create a custom class that contains all the information needed to select and run a report from the list of reports. This class is fairly simple, with nothing but basic public fields, plus an overridden ToString function, used by the ListBox control to properly display each list item.

In the Library Project, add a new class file named ReportItem.vb through the Project Add Class menu command. Add the following enumeration to the file, but add it outside the Class . . . End Class boundaries. This enumeration indicates what type of entry each list item represents.

To this same file, add the members of the ReportItem class. This class contains all the information we need to run reports loaded from the configuration file.

Design the Report Form

Librarians and administrators use the Select Report form (see Figure 13-2) to view reports. The form includes a ListBox control that displays all reports and report groups, a Run button that starts a report, and a Close button that returns the user to the main form. A label displays the full description of a report, when available, just below the ListBox.

Figure 13-2. The Select Report form

Add a new form file named ReportSelect.vb through the Project Add Windows Form menu command. Add the controls and settings as listed here.

Adjust the tab order of the new controls by selecting the form, and then using the View Tab Order menu command.

Although the administrator has probably given useful names to each report, the terseness of each report name may still confuse the user. Each report includes an optional full description. As the user selects reports from the list, an event handler updates the FullDescription label just below the main list. Add this event handler member to the class.

Populate Reports from Configuration File

The RefreshReportList method loads the data from the report configuration file, and processes the results. Eventually, the location of this file will be recorded in the application’s configuration file, but we won’t be adding that until a later chapter. For now, let’s put in a hard-coded test file location, and mark it for later update.

Because the report configuration file allows nested report groups to any level, we need to use a recursive routine to repeatedly descend to each successive level. The LoadReportGroup routine, called by RefreshReportList, adds all report items and report groups within a starting report group. It’s initially called from the reference point of the root <reportList> element. Each time that it finds a child <reportGroup> element, it calls itself again, but this time starting from the reference point of the child <reportGroup> element.

Add the form’s Load event, which loads in the content from the configuration file.

Running the Reports

Now that all of the groups and items appear in the list, we have to run the actual reports. The ActRun button’s Click event handles this duty. For now, we will just add the framework to support the calling of each report. The built-in reports will be added in Chapter 20, “Reporting.”

For external reports, the event handler calls the Process.Start method. This amazing method accepts either a standard command-line expression, or any valid URL or web page address.

Connecting the Select Report Form

To make the reports available to the user, we must enable a link to the report form from the main form. We included a distinct panel on that form just for printing reports. The ActDoReports button on that panel triggers a call to the new report selection form. Create a new event handler for the ActDoReports button and add the following code.

' ----- Show the reports form.
ReportSelect.ShowDialog()

Now that we have a firm grasp on the world of XML, we’ll let Visual Basic do all the hard work of manipulating it for application configuration purposes.