XQuery and XPath

XPath started out as a language for selecting elements and attributes from an XML document while traversing its hierarchy and filtering out unwanted content. XPath 1.0 is a fairly simple yet useful recommendation that specifies path expressions and a limited set of functions. Later versions of XPath are much more than that, encompassing a wide variety of expressions and functions, not just path expressions.

XQuery and XPath overlap to a very large degree. They have the same data model and the same set of built-in functions and operators. XPath is essentially a subset of XQuery. XQuery has a number of features that are not included in XPath, such as FLWORs, user-defined functions, and XML constructors. This is because these features are not relevant to selecting, but instead have to do with structuring, sorting query results or more complex programming.

The two languages are consistent in that any expression that is valid in both languages evaluates to the same value using both languages.

XQuery Versus XSLT

XSLT is a W3C language for transforming XML documents into other XML documents or, indeed, documents of any kind. There is a lot of overlap in the capabilities of XQuery and XSLT. XSLT makes use of XPath, so it has the same data model and supports all the same built-in functions and operators as XQuery, as well as many of the same expressions.

Some of the differences between XQuery and XSLT are:

• XSLT implementations are generally optimized for transforming entire documents. Whether they load the entire input document into memory or use the streaming features of XSLT 3.0, the use case is typically handling a single document at a time. XQuery is optimized for selecting fragments of data, for example, from a database. XQuery is designed to be scalable and to take advantage of database features such as indexes for optimization.

• XQuery has a more compact non-XML syntax, which is sometimes easier to read and write (and embed in program code) than the XML syntax of XSLT.

• XQuery is designed to select from a collection of documents as opposed to a single document. FLWORs make it easy to join information across (and within) documents. XSLT stylesheets can also operate on multiple documents, but XSLT processors are not particularly optimized for this less common use case.

Generally, when transforming an entire XML document from one XML vocabulary to another, it makes more sense to use XSLT. When your main focus is selecting a subset of data from an XML document or database, you should use XQuery. The relationship between XQuery and XSLT is explored further in Chapter 27.

XQuery Versus SQL

XQuery borrows ideas from SQL, and many of the designers of XQuery were also designers of SQL. The line between XQuery and SQL may seem clear; XQuery is for XML, and SQL is for relational data. However, increasingly this line is blurred, because relational database vendors are putting XML frontends on their products and allowing XML to be stored in traditionally relational databases.

XQuery does not replace SQL for the highly structured data that is traditionally stored in relational databases. The two can coexist, with XQuery being used to query less-structured data, or data that is destined for an XML-based application, and SQL continuing to be used for highly structured relational data.

Chapter 26 compares XQuery and SQL, and describes how they can be used together.

XQuery and XML Schema

XML Schema is a W3C standard for defining schemas, which can be used to validate XML documents and to assign types to XML elements and attributes. XQuery uses the type system of XML Schema, which includes built-in types that represent common datatypes such as decimal, date, and string. XML Schema also specifies a language for defining your own types based on the built-in types.

If an input document to a query has a schema, the types can be used when evaluating expressions on the input data. For example, if your item element has a quantity attribute, and you know from the schema that the value of the quantity attribute is an integer, you can perform sorts or other operations on that attribute’s value without converting it to an integer in the query. This also has the advantages of allowing the processor to better optimize the query and to catch errors earlier.

XQuery users are not required to use schemas. It is entirely possible (and common) to write a complete query with no mention of schemas. However, a rich set of functions and operators are provided that generally operate on typed data, so it is useful to understand the type system and use the built-in types, even if no schema is present. Chapter 14 covers schemas in more detail.

Input Documents

Throughout this book, the term input document is used to refer to the data that is being queried, which is most often XML. The XML that is being queried can, in fact, take a number of different forms, for example:

• Text files that are XML documents

• Fragments of XML documents that are retrieved from the Web using a URI

• A collection of XML documents that are associated with a particular URI

• Data stored in native XML databases

• Data stored in relational databases that have an XML frontend

• In-memory XML documents

Some queries use a hardcoded link to the location of the input document(s), using the doc or collection function in the query. Other queries operate on a set of input data that is set by the processor at the time the query is evaluated.

Whether it is physically stored as an XML document or not, an XML input document must conform to other constraints on XML documents. For example, an element cannot have two attributes with the same name, and element and attribute names cannot contain special characters other than hyphens, underscores, and periods.

In addition to XML input, it is also possible to query JSON documents and simple text files. This can be done via the json-doc and unparsed-text functions, or through in-memory data structures.

The Query

An XQuery query could be contained in a text file, embedded in program code or in a query library, generated dynamically by program code, or input by the user on a command line or in a dialog box. Queries can also be composed from multiple files, known as modules.

A query is made up of three parts: a version declaration, a prolog, and a body, in that order.

• The optional version declaration says what version of XQuery you are using, for example 3.1. If the version declaration does not appear, the processor makes an assumption about the version based on which version it supports.

• The optional query prolog contains various declarations that are used in evaluating the query. This includes namespace declarations, variable declarations, user-defined functions, and other settings. These declarations are discussed in relevant sections throughout the book and summarized in Chapter 12.

• The query body contains one or more expressions, separated by commas, that indicate what the query should return.

So far, the examples in this book have had only a query body. Example 2-1 shows a query with all three parts. As you can see, a semicolon separates the version declaration and each of the two declarations in the prolog. The query body contains two expressions, a constructed h1 element, and a FLWOR. The comma after the h1 element is used to separate the two expressions in the query body.

xquery version "3.1";
            
declare namespace html = "http://www.w3.org/1999/xhtml";
declare variable $orderTitle := "Order Report";

<h1>{$orderTitle}</h1>,
for $item in doc("order.xml")//item
order by $item/@num
return <p>{data($item/@num)}</p>

<h1>Order Report</h1>
<p>443</p>
<p>557</p>
<p>557</p>
<p>563</p>
<p>784</p>
<p>784</p>

The Context

A query is not evaluated in a vacuum. The query context consists of a collection of information that affects the evaluation of the query. Some of these values can be set by the processor outside the scope of the query, while others are set in the query prolog. The context may include such values as:

• The context item, which determines the context for path expressions in the query, i.e., what input documents are being queried

• Current date and time, and the implicit time zone

• Names and values of variables that are bound outside the query or in the prolog

• External function libraries built into your processor

The Query Processor

The query processor is the software that parses, analyzes, and evaluates the query. The analysis and evaluation phases are roughly equivalent to compiling and executing program code. The analysis phase finds syntax errors and other static errors that do not depend on the input data. The evaluation phase actually evaluates the results of the query based on input documents, possibly raising dynamic errors for situations like missing input documents or division by zero. Either phase may raise type errors, which result when a value is encountered that has a different type than expected. Errors in XQuery all have eight-character names, such as XPST0001, and they are described in detail in Appendix C.

There are a number of implementations of XQuery. Some are open source, while others are available commercially from major vendors. Many are listed at the official XQuery website at http://www.w3.org/XML/Query. This book does not delve into all the details of individual XQuery implementations but points out features that are implementation-defined or implementation-dependent, meaning that they may vary by implementation.

The Results of the Query

The query processor returns a sequence of values as the results. The results are often XML elements (or entire documents), but a query could also return a result that is not XML, for example a string or an array of integers. Depending on the implementation, these results can then be written to a physical file, sent to a user interface, or passed to another application for further processing.

Writing the results to a physical XML document is known as serialization. In your query you can specify that you want the output serialized as XML, HTML, XHTML, text, or JSON. “Serializing Output” covers serialization options in more detail.

Nodes

Nodes are used to represent XML constructs such as elements and attributes. Nodes are returned by many expressions, including path expressions and constructors. For example, the following path expression returns four product element nodes:

doc("catalog.xml")/catalog/product

The node hierarchy

An XML document (or document fragment) is made up of a hierarchy of nodes. For example, suppose you have the document shown in Example 2-2.

Example 2-2. Small XML example

<catalog xmlns="http://datypic.com/cat">
  <product dept="MEN" xmlns="http://datypic.com/prod">
    <number>784</number>
    <name language="en">Cotton Dress Shirt</name>
    <colorChoices>white gray</colorChoices>
    <desc>Our <i>favorite</i> shirt!</desc>
  </product>
</catalog>

When translated to the XQuery data model, it looks like the diagram in Figure 2-3. (Depending on the processor, there may also be text nodes, not shown in the diagram, for the line breaks and spaces used to indent the XML document.)

Figure 2-3. A node hierarchy

string(doc("catalog.xml")/catalog/product[4]/number)

string(<desc>Our <i>favorite</i> shirt!</desc>)

data(doc("catalog.xml")/catalog/product[4]/number)

Atomic Values

An atomic value is a simple data value such as 784 or ACC, with no markup, and no association with any particular element or attribute. An atomic value can have a specific type, such as xs:integer or xs:string, or it can be untyped, meaning that it is assigned the generic type xs:untypedAtomic.

Atomic values can be extracted from element or attribute nodes using the string and data functions described in the previous section. They can also be created from literals in queries. For example, in the expression @dept = 'ACC', the string ACC is an atomic value. The result of the entire expression is also an atomic value; it is a Boolean true/false value.

The line between a node and an atomic value that it contains is often blurred. That is because all functions and operators that expect to have atomic values as their operands also accept nodes. For example, you can call the substring function as follows:

doc("catalog.xml")//product[4]/substring(name, 1, 15)

The function expects a string atomic value as the first argument, but you can pass it an element node (name). In this case, the atomic value is automatically extracted from the node in a process known as atomization.

Unlike nodes, atomic values don’t have identities. It’s not meaningful (or possible) to ask whether "abc" and "abc" are the same string or different strings; you can only ask whether they are equal.

Sequences

Sequences are ordered collections of items. A sequence can contain zero, one, or many items of any kind. For example, you could have a sequence of atomic values, or a sequence of nodes, or a sequence that contains both atomic values and nodes.

The most common way that sequences are created is that they are returned from expressions or functions that return sequences. For example, the expression:

doc("catalog.xml")/catalog/product

returns a sequence of four items, which happen to be product element nodes.

A sequence can also be created explicitly using a sequence constructor. The syntax of a sequence constructor is a series of values, delimited by commas, surrounded by parentheses. For example, the expression (1, 2, 3) creates a sequence consisting of those three atomic values.

You can also use expressions in sequence constructors. For example, the expression:

(doc("catalog.xml")/catalog/product, 1, 2, 3)

results in a seven-item sequence containing the four product element nodes, plus the three atomic values 1, 2, and 3, in that order.

Sequences do not have names, although they may be bound to a named variable. For example, the let clause:

let $prodList := doc("catalog.xml")/catalog/product

binds the sequence of four product elements to the variable $prodList.

A sequence with only one item is known as a singleton sequence. There is no difference between a singleton sequence and the item it contains. Therefore, any of the functions or operators that can operate on sequences can also operate on individual items, which are treated as singleton sequences.

A sequence with zero items is known as the empty sequence. In XQuery, the empty sequence is different from a zero-length string (i.e., "") or a zero value. Many of the built-in functions and operations accept the empty sequence as an argument, and have defined behavior for handling it. Some expressions will return the empty sequence, such as doc("catalog.xml")//foo, if there are no foo elements in the document.

Sequences cannot be nested within other sequences; there is only one level of items. If a sequence constructor is inserted into another sequence constructor, the items created by the inserted sequence constructor become full-fledged items of the result sequence. For example:

(10, (20, 30), 40)

is equivalent to:

(10, 20, 30, 40)

Quite a few functions and operators in XQuery operate on sequences. Some of the most used functions on sequences are the aggregate functions (count, min, max, avg, and sum). In addition, union, except, and intersect operators allow sequences to be combined. There are also a number of functions that operate generically on any sequence, such as index-of and insert-before.

Like atomic values, sequences have no identity. You can’t ask whether (1, 2, 3) and (1, 2, 3) are the same sequence; you can only compare their contents.