11.2.1 Concepts

Every language has concepts that are necessary to an understanding of the language and how to use it. XQuery is no exception. Here are a few terms we consider especially important to know.

• The Data Model instances can be created by parsing, and perhaps validating, XML documents, thereby creating Infosets or PSVIs. The Data Model spec describes how to derive a Data Model instance from an Infoset or a PSVI. They can also be created by other means, such as by programs that directly generate Data Model instances for XQuery engines to evaluate.

• Similarly, the In-Scope Schema Definitions can be created by parsing XML Schema documents, thereby generating XML Schemas. Alternatively, they can be created by other means, analogous to direct Data Model instance generation.

• The static context is initialized from the environment (e.g., by the XQuery implementation), as is the dynamic context. Both are affected by other parts of the Processing Model, such as the inclusion of in-scope schema definitions.

• The execution engine (perhaps “evaluation engine” would be more appropriate) acts on the Data Model instances provided to the XQuery expression being evaluated and (normally) generates other Data Model instances. Those instances may be serialized when query evaluation has completed, but the Processing Model does not require that. Data Model instances so generated can be passed directly to other processes, perhaps another XQuery expression evaluation.

• The execution engine depends on the dynamic context and, by implication, on the static context, while the process of parsing an XQuery expression, and converting it into whatever internal execution constructs that the execution engine uses, depends only on the static context.

11.3.1 The Static Context

Whenever an XQuery expression is processed, the set of initial conditions governing its behavior is called the static context. The static context is a set of components, with values that are set globally by the XQuery implementation before any expression is evaluated. The values of a few of those components can be modified by the query prolog (see Section 11.7), and the values of a very few can be modified by the actions of the query itself. Figure 11-1 illustrates just where in the XQuery processing model the static context is used.

Table 11-1, adapted from the XQuery 1.0 specification, summarizes the components of the XQuery static context and how their values can be changed. For an explanation of the meaning of each component identified in the first column of the table, please refer to the XQuery 1.0 specification.

In the headers of the rightmost three columns, “Implementation” means “the XQuery implementation,” “Prolog” means “the XQuery prolog,” and “Expression” means “the expression itself.” Within the rows of those columns, an “s” means that the corresponding object can set the value of the component, an “a” means that the object can augment³ the value of the component, and a dash (” – “) means that the object cannot change the value of the component.

When a component’s value can be changed by the implementation or by the prolog, the initial value can be overwritten and/or it can be augmented. Some components can be overwritten, while others cannot. Other components can be augmented, while others cannot. For such fine detail, we recommend that you consult the XQuery 1.0 specification.

11.3.2 The Dynamic Context

The dynamic context represents aspects of the environment that may change during the evaluation of an XQuery or that might be changed by environmental factors other than the XQuery implementation itself. Some people, us included, view the static context as part of the dynamic context; others don’t.

Table 11-2, also adapted from the XQuery 1.0 specification, summarizes the components of the XQuery dynamic context and how their values are set. For an explanation of the meaning of each component identified in the first column of the table, please refer to the XQuery 1.0 specification. In this table, “y” means that the corresponding object can change the value of the component and “ – “ means that it cannot.

For additional details, we recommend that you consult the XQuery 1.0 specification.

11.5.1 Literal Expressions

The most primitive kind of expression in XQuery is a literal. A literal is a character string that lies in the lexical space⁵ of one or more data types (the lexical space of a data type is the collection of character strings that can be used to express any possible value of that data type). For example, the character string 1.1E1 is a literal lying in the lexical space of the XML Schema data types xs:float and xs:double, while ‘Mars Attacks!’ is a literal lying in the lexical space of xs:string, and 3.14159 is a literal in the lexical space of xs:decimal, xs:float, and xs:double.

Broadly speaking, a literal is an expression whose value is itself. Therefore, the value of 3.14159 is, well, 3.14159 and the value of 1.1E1 is 11. As you see in Grammar 11-3, numeric literals come in three “flavors”: integer literals, decimal literals, and double literals. String literals can be enclosed in double quotes (” “) or in apostrophes (’’, sometimes called “single quotes”). The characters permitted in a string literal include all Unicode characters other than ampersands (&). To include a double quote in a literal enclosed in double quotes, you simply, well, double it. Similarly, to include an apostrophe in a literal enclosed in apostrophes, you simply use two consecutive apostrophes. You can also use character references of the form &#xnnnn; (where “nnnn” is one to six hex digits specifying the Unicode code point for the desired character). This comes in handy when you need to use characters that might not appear on your keyboard, such as &#x222D; for the character triple integral: ∭. Finally, you can use one of the five predefined entity references defined by XML itself (<, >, &, ", and ’).

11.5.2 Constructor Functions

An expression that is almost as primitive as a literal is a constructor function invocation. As you saw earlier, the value 3.14159 is a valid literal in the lexical space of three data types: xs:decimal, xs:float, and xs:double. Because XQuery is a strongly typed language, it’s sometimes necessary to specify more precisely the data type you want a literal to be.

XQuery provides constructor functions for this purpose. Constructor functions are defined in the Functions and Operators specification (about which you learned in Chapter 10), but they are sufficiently central to XQuery itself that we briefly mention them here. For the purposes of the XQuery grammar, constructor functions are invoked exactly like ordinary functions.

To ensure that your literal 3.14159 is a value of xs:double, you can write the constructor function invocation xs:double (3.14159) or the equivalent constructor function invocation xs:double(“3.14159”). If your query requires a decimal value instead, you could simply write xs:decimal (3.14159). However, XQuery is clever enough to infer that the literal 3.14159 by itself is intended to be of type xs:decimal and not of type xs:double or of type xs:float.

By contrast, values of some data types can be constructed only with an explicit constructor function invocation or an explicit cast from string to the desired data type. For example, to express the date commonly known as American Independence Day, it is not sufficient to write 1776–07–04. XQuery would interpret that to mean “the integer value 1765” (the result of subtracting 7 from 1776 and then subtracting 4 from that result). Instead, one must write xs:date (“1776–07–04”) or “1776–07–04” cast as xs:date.

XQuery automatically provides a constructor function for every built-in data type defined in XML Schema Part 2, as well as for every data type derived from them in any schemas that you might import (see Section 11.7) in your query.

11.5.3 Sequence Constructors

Another simple kind of XQuery expression is the sequence constructor. There are many ways in XQuery of generating a sequence – after all, the fundamental building block in the Data Model is the sequence (and every XQuery value is a sequence of zero, one, or more items). As a result, it is technically accurate to say that every expression evaluates to a sequence.

In fact, the XQuery grammar fragment in Grammar 11-4 defines XQuery expressions in general to be a sequence of ExprSingles. Note that this is not a type of primary expression.

There are only two types of sequence constructor in XQuery. One of these uses commas (,) as the operator that constructs a sequence from two items, as illustrated in Example 11-1. In general, this first form of sequence constructor can be used in any context where a general Expr is appropriate; however, in a context where a single value (ExprSingle in XQuery terms) is required, a sequence constructed using the comma operator can be used only when enclosed in parentheses. (This is because a sequence of values separated by commas without surrounding parentheses is not recognized in the XQuery grammar as a single value. It requires the parentheses to group the values into a single value that is a sequence.) By convention in this book, we enclose all such sequences in parentheses.

Don’t be fooled: the result of the sequence constructor in Example 11-1 is entirely different from the character string ‘This reviewer gives 3 stars to this film’. Example 11-1 results in a sequence of five items:⁶ an xs:string value, an xs:integer value, and three more xs:string values. By contrast, the character string is a sequence of only one item: a single xs:string value.

Sequences constructed with comma operators are not limited to containing items of atomic types. They can contain any sort of item, including elements, XML comments, and so forth (but, as you read earlier in this chapter, not other sequences).

The other kind of sequence constructor is called the range expression, written RangeExpr in Grammar 11-4. (The BNF nonterminal symbol AdditiveExpr is addressed in the next subsection, Arithmetic Expressions. For our purposes here, it’s just an expression that evaluates to a value of type xs:integer.)

A range expression constructs a monotonically increasing sequence of consecutive integers beginning with the value of the first (or only) AdditiveExpr in the RangeExpr and ending with the value of the last (or only) AdditiveExpr in the RangeExpr, as illustrated in Example 11-2, which constructs the sequence (5, 6, 7, 8, 9, 10).

If the second AdditiveExpr is specified and its value is less than the value of the first AdditiveExpr, then the RangeExpr evaluates to an empty sequence, represented in XQuery as an empty pair of parentheses: ( ). (If you need to generate a sequence of consecutive integers in descending order, you can apply the F&O function fn:reverse( ) to a range expression.)

11.5.4 Variable References

Variable references are another kind of primitive expression in XQuery. As you can see in Appendix C: XQuery 1.0 Grammar, XQuery allows the declaration of variables in query prologs. In addition, variables are declared as part of certain other expressions, particularly the for and let clauses of FLWOR expressions. Of course, those variables are of little use unless they can be referenced in XQuery expressions. The name of a variable is a QName and is always preceded by a dollar sign ($) when the variable is being defined and when it is being referenced. Grammar 11-5 provides the syntax of variable references. Recall that a QName is made up of two parts: an optional namespace URI, lexically represented by a namespace prefix, and a local name. Two variable references reference the same variable if their local names are the same and the namespace URIs bound to their namespace prefixes are the same.

A variable reference is syntactically invalid if there is not a variable of the same QName in the in-scope variables (see Table 11-1). If there exists a variable named “studio,” then that variable is referenced using “$studio.”

11.5.5 Parenthesized Expressions

A parenthesized expression is exactly what its name implies – an expression surrounded by parentheses, as expressed in Grammar 11-6. Note that the contained expression is, in fact, optional, allowing a bare pair of parentheses – ( ) – to be used as the representation of an empty sequence.

In an XQuery expression, parentheses can be used to force a desired precedence of operators that is different from the default precedence. For example, the expression “2*3+4” has a different result – 10 – than the expression “2*(3+4)” – 14. Parentheses can also be used where they don’t change the semantics of an expression, perhaps to make the precedence in an expression explicit, or for aesthetic purposes.

11.5.6 Context Item Expression

In Chapter 9, “XPath 1.0 and XPath 2.0,” as well as in Section 11.2, you learned that XPath and XQuery nearly always have a context item (as well as a context position and context size) that is used as the context in which many expressions are evaluated.

In XQuery, the context item is referenced using the syntax shown in Grammar 11-7. This is commonly referred to as “dot.”

A context item expression evaluates to the context item (which may be either a node or an atomic value). Evaluation of a context item expression when the context item is undefined results in an error.

11.5.7 Function Calls

A function call, like almost everything else in XQuery, is an expression. In Appendix C: “XQuery 1.0 Grammar,” we see that functions are declared using syntax that includes the name of the function (a QName) and a pair of parentheses that optionally includes a comma-separated list of parameter declarations. Once declared, a function can be invoked as part of an XQuery expression, returning a value of the type specified when the function was declared. The Functions and Operators specification defines a number of functions that are always available to use in XQuery expressions. Other functions can be made available for use in an XQuery expression in three ways: They can be declared in the XQuery prolog, they can be imported from a library module, and they can be provided by the external environment as part of the static context.

A function call (or function invocation), shown in Grammar 11-8, bears some resemblance to the function declaration syntax mentioned in the previous paragraph. The important difference, of course, is that a function call specifies arguments that provide the values for the function’s parameters.

When a function call is evaluated, the name of the function has to be equal to the name of a function in the static context, and the number of arguments in the function must be equal to the number of parameters in the function’s declaration.

Function calls are evaluated in several steps.

An example of a function declaration and a corresponding function call is given in Example 11-3.

11.5.8 Filter Expressions

A filter expression is merely any primary expression followed by zero or more predicates, as specified in Grammar 11-9. The result of a filter expression comprises each item returned by the primary expression for which all of the predicates are true. If there are no predicates, then the value of the filter expression is exactly the same as the value of the primary expression.

The order of items in the result of the filter expression is the same as the order in which those items appeared in the primary expression.

You were exposed to predicates in Chapter 9, “XPath 1.0 and XPath 2.0,” so they are not addressed in detail in this chapter. Recall that a predicate is an expression whose value is a Boolean value, such as a comparison expression. Below, Section 11.5.11 discusses Boolean-valued expressions that are used in predicates.

11.5.9 Node Sequence-Combining Expressions

Now that we’ve covered the simpler kinds of expressions, let’s look at expressions that combine node sequences – union, intersect, and except. The syntax used for these sequence-combining expressions is seen in Grammar 11-10. The operands of these three operators are node sequences, not values (as they are in SQL, for example). Consequently, it is not possible to evaluate an expression such as (1, 2) union (2, 1) – the contents of the two sequences are values and not nodes.

One of these, using the union operator (equivalently, the vertical bar operator, |), returns a sequence containing all nodes that appear in either of its node sequence operands. Another, using the intersect operator, returns a sequence containing only those nodes that appear in both of its node sequence operands. The third, using the except operator, returns a sequence containing all nodes that appear in its first node sequence operand but not in the second.

All three of these expressions eliminate duplicate nodes (based, of course, on node identity) and, unless the ordering mode is unordered, return the result node sequence in document order. Example 11-4 illustrates some of these expressions. For the purposes of these examples, assume that “A” represents the movie node corresponding to the movie Absolute Power, “B” represents the movie node for the film Below, and “C” represents the movie node of the film Corruption, and also assume that the document containing these three nodes happens to contain them in that sequence: A followed by ? followed by C. The XQuery comments preceding each example indicates the value computed by the expression.

11.5.10 Arithmetic Expressions

Let’s get back to basics. XQuery supports the basic kinds of arithmetic operations that most programming languages provide: addition, subtraction, multiplication, and division; it also provides a modulus operator. In addition (pun noted, but not intended), XQuery provides unary plus and minus operators.

Because XQuery is not primarily intended as a mathematical computation language, it does not provide built-in operators for operations such as exponentiation, extraction of roots, or logarithmic computations. (However, we do anticipate that each community will develop libraries of user-defined functions to support the operations on which their work depends.)

The syntax of XQuery’s arithmetic operators is shown in Grammar 11-11, and a few examples are seen in Example 11-5.

Because the hyphen (–) is used as the subtraction operator, as the negation operator (“unary minus”), and as a valid character in XML names, XQuery requires that the subtraction operator be preceded by white space if it could possibly be mistaken as part of the preceding token. For example, “MyStars-1” is a valid XML name; if your intent is to subtract 1 from the rating of a film (number of stars) given by a reviewer, then XQuery requires that to be expressed something like this: “MyStars -1” or “MyStars – 1.”

In an AdditiveExpr, the plus sign (+) indicates addition of the values of the two operands, and the hyphen, also called a minus sign (-), specifies the subtraction of the value of the second operand from the value of the first.

In a MultiplicativeExpr, the asterisk (*) means multiplication of the values of the two operands. In many programming languages, a slash (/) is used to indicate division. However, XQuery uses the slash as a path expression operator, so the keyword “div” was chosen to indicate division and a second keyword, “idiv,” indicates integer division – specifically, division of the value of the first operand by the value of the second. The keyword “mod” indicates the modulus operation (which, simplified, means to return the remainder of a division operation instead of returning the quotient).

In Example 11-5, we use XQuery comments to state the result of the example.

In XQuery, numbers are handled using rules that are needed when mixing integers, decimal numbers, and floating-point numbers. These rules say that any arithmetic operation that involves numbers of two different data types requires one number to be “upcast,” or “promoted,” to the type of the other. XQuery deals with only four of the many XML Schema numeric types – xs:integer, xs:decimal, xs:float, and xs:double. Even though, in XML Schema, there is no type derivation relationship between xs:decimal, xs:float, and xs:double (but xs:integer is derived from xs:decimal), XQuery treats them as though there were such a relationship. Example 11-6 illustrates the numeric type hierarchy and provides a couple of examples.

If an operation requires promotion of one value to the type of the other value, then xs:integer values can be promoted to any of the other three types, xs:decimal values can be promoted to xs:float or xs:double, and xs:float values can be promoted to xs:double. If the variable $ i is of type xs:integer and the variable $ j is of type xs:float, then the expression $i – $j requires that the value of $ i be promoted to xs:float before the operation is performed; it also requires the result of the operation to be of type xs:float.

Each operand of an arithmetic operator is evaluated in four steps:

If, after these steps have been applied to both operands, one or both operands are not of a type suitable for the operation – such as an effort to subtract a value of type xs:decimal from a value of type xs:IDREF – an error is raised. Even when the operands are both of suitable types, errors can be raised by the operation itself, such as an attempt to divide by zero.

As we have seen, XQuery has a rich set of arithmetic operators, but that set will be complemented by function libraries that provide even more functionality.

11.5.11 Boolean Expressions: Comparisons and Logical Operators

There are two kinds of expressions in XQuery that produce Boolean results. One kind, comparison expressions, provides the ability to compare two values. The other, logical expressions, allow the combination of Boolean values, such as those produced by comparisons.

Comparison expressions can be divided into three categories: value comparison, general comparison, and node comparison. Value comparisons are used to compare two single values, general comparisons are (for all practical purposes) quantified comparisons – also called existential comparisons – that can be used to compare sequences of any length, and node comparisons are used to compare two nodes.

The grammar of comparison expressions is presented in Grammar 11-12 (slightly modified for clarity from the grammar as published in the XQuery specification). Note that the three types of comparison use different sets of operators. It’s tempting to conclude that the ordinary comparison operators (=, >, etc.) could have been used for all three types; however, if XQuery had done so, it would be impossible in many instances to determine whether any given comparison was intended to be a value comparison, a general comparison, or a node comparison.

Value comparisons require that the value of each operand be determined. The steps are very similar to those involved in determining the values of the operands of arithmetic expressions, except for the fourth step:

If the values of the two operands have types that are compatible for the purposes of comparison, then they are compared. If the value of the first operand is equal to, not equal to, less than, less than or equal to, greater than, or greater than or equal to the value of the second operand, then the comparison using the “eq,” “ne,” “lt,” “le,” “gt,” or “ge” operator, respectively, is true; otherwise, the comparison is false. Some value comparisons are illustrated in Example 11-7.

General comparisons, as we said earlier, act as existential comparisons. By “existential comparison” we mean this: If there exists at least one value in the sequence that is the value of the first operand that has the proper comparison relationship (using the value comparison rules!) with at least one value in the sequence that is the value of the second operand, then the general comparison is true; otherwise, it is false.

In principle, every value belonging to each of the two sequences is compared to every value in the other sequence. In practice, implementations are very often able to determine the result without actually doing so many comparisons, so the rules of XQuery allow implementations to return the result without compulsively comparing every combination of values. One consequence of this permissive rule is that there may be errors that would result from comparing some particular value in the first sequence with some other specific value in the second sequence, but the implementation might return a true/false result and not raise the error. The second example in Example 11-8 illustrates such a situation.

Of course there are a few rules to cover the relationships between the types of the operands:

Example 11-8 provides a few sample general comparison expressions.

The final example is a valid general comparison, even though the operands are single values – remember that a single value is a singleton sequence containing that value.

General comparisons do not behave like comparisons that use the same operators in most other languages (it’s value comparisons that behave like comparisons in those other languages, albeit with different operators). The differences are caused by the existential semantics of general comparison. Therefore, even though “(1, 2) = (2, 3)” is true and “(2, 3) = (3, 4)” is true, “(1, 2) = (3, 4)” is false! That is, general comparisons are not transitive. Similarly, both “(1, 2) = (2, 3)” and “(l, 2) != (2, 3)” are true – inverted operators do not imply inverted results.

Node comparisons are different from both value comparisons and general comparisons, in that they do not compare values at all, but compare nodes based on their identities. In a node comparison, if either operator evaluates to a sequence of more than one node, then an error is raised. If either operand evaluates to an empty sequence, then the result of the comparison is also the empty sequence (node comparisons can have three values: true, false, and the empty sequence).

If the two operands have the same identity – that is, they are the same node – then the “is” comparison is true; otherwise, that comparison is false. If the first operand is a node that appears earlier in document order than the node identified by the second operand, then the ”<<” comparison is true and the “>>“ is false. There is an exception to this rule: If the ordering mode is unordered (see Section 11.4.7, “Function Calls”), then the results are nondeterministic, because document order is not maintained. Example 11-9 demonstrates these principles.

As you have seen in this discussion of comparison expressions, the result of such an expression is either a Boolean value (true or false) or the empty sequence. A frequent requirement in applications is to combine the results of multiple comparisons: “I want to find the MPAA rating of all movies whose titles contain the word ‘Outlaw’ that were released after 1965.” In this example, two comparisons are combined: “titles contain the word ‘Outlaw’” and “released after 1965” – both must be true for the movies I want to find. The ability to combine multiple comparison expressions is provided in XQuery with logical expressions.

A logical expression is an expression that permits the combination of Boolean values to achieve an aggregated Boolean result. Logical expressions in XQuery operate on the effective Boolean value of their operands and, naturally, follow the rules of Boolean algebra. XQuery defines only two Boolean operators: and and or. (Many languages include a third Boolean operator: not. In XQuery, the functionality of that operator is provided by a function, fn:not.)

The behaviors of these two operators are defined in Table 11-3 and Table 11-4. The cells that contain “true or error” or “false or error” imply that an implementation may choose to determine the value of the expression from the value of the one operand that does not generate an error, or it may choose to raise an error instead. It’s worth pointing out that XQuery, like SQL and many other programming languages, but unlike a few popular languages such as C, does not require evaluation of the operands of logical expressions in any particular order. Instead, implementations are free to reorder the evaluation of the operands for such reasons as query optimization.

Comparison expressions and logical expressions can be combined in powerful ways, making up arbitrarily complex predicates that are used in the FLWOR expression’s where clause and in the predicates of path expressions. But users new to XQuery must be careful to use the correct operator when comparing two items. Remember that most languages use the symbol “=” to mean “this value is equal to that value,” while XQuery uses it in an existential sense to mean “any item in this sequence is equal to any item in that sequence.” In order to get the semantics that “=” provides in other languages (including gaining protection from situations where it is possible for one or both operands to be a sequence of length greater than 1), XQuery expressions use the “eq” operator instead.

11.5.12 Constructors – Direct and Computed

One of the strengths of XQuery (over, say, XPath) is its ability to construct XML nodes and thus to build up brand new XML fragments or complete documents in the result of a query. In addition to the constructor functions and sequence constructors that we discussed earlier in this section, XQuery provides two different classes of constructors for nodes. Document nodes can be constructed using only one class of constructors, while five of the other six node types can be constructed by both classes. XQuery does not represent namespace bindings as nodes, so there is no way in XQuery to construct namespace nodes (the seventh node type).

The two classes of node constructor are direct constructors and computed constructors. Direct constructors use an XML-like syntax, while computed constructors use a syntax based on enclosed expressions. (An enclosed expression is an expression enclosed within curly braces: {…}.)

11.5.13 Ordered and Unordered Expressions

XML, used as a markup language, creates documents that are inherently ordered. Think about a book that is marked up in XML – the second chapter must always follow the first chapter, and the paragraphs in each chapter must always be in the sequence in which the author wrote them. As a result, XQuery treats the XML that it queries as ordered (and, in particular, it handles that XML in document order) unless instructed to do otherwise. One way in which an XQuery can be instructed to “do otherwise” is through the order by clause (see Section 11.6.3), through which the author of an XQuery forces the results of an expression to be ordered according to specified criteria.

However, because XQuery is sometimes applied to information that doesn’t represent books or other traditional “documents” – such as relational data, as you’ll see in Chapter 15, “SQL/XML” – the notion of “document order” is not always a meaningful one. Instead, any ordering to be applied to query evaluation is imposed as part of the query (such as the order by clause just mentioned) or is an artifact of the optimizations that the query processing engine applies to the evaluation of that particular query, often based on factors such as indexes or other physical storage facets.

In order to provide applications with the ability to write queries that selectively bypass considerations of inherent ordering, XQuery provides, as primary expressions, both ordered expressions and unordered expressions. The syntax of these two expressions appears in Grammar 11-16.

In Section 11.2.3, our description of XQuery’s static context included a component named “ordering mode.” When either an ordered expression or an unordered expression appears as a part of an XQuery expression, the ordering mode in the static context is set to ordered or unordered, respectively, for the lexical scope of the Expr that appears between the curly braces ({}). Of course, that Expr can be any XQuery expression and thus can have arbitrarily deep nesting of other expressions, including other ordered and unordered expressions.

The ordering mode affects the behavior of most step expressions (as discussed in Chapter 9, “XPath 1.0 and XPath 2.0”), the set operators (union, intersect, and except), and FLWOR expressions that don’t have an order by clause. If the ordering mode of those expressions is “ordered,” then the node sequences that they return are in document order; if the ordering mode is “unordered,” then the node sequences are in an implementation-dependent order. (Note, however, that the ordering mode has no effect on elimination of duplicate nodes from those node sequences.) Because the order of nodes in those node sequences is implementation-dependent, the behavior of certain functions, such as fn:position( ), as well as numeric predicates in path expressions, is nondeterministic.

In addition to ordered and unordered expressions, XQuery provides the fn:unordered( ) function that takes any sequence (not necessarily of nodes) and returns it in a nondeterministic order. This function is not a “randomizing” function – that is, it might well return the sequence in its original order. It merely gives permission to the XQuery evaluation engine to reorder the sequence if necessary for reasons such as performance optimization.

11.5.14 Conditional Expression

Generally speaking, a conditional expression is one that returns one of two values based on the evaluation of a predicate. (This is not the “if statement” used by imperative languages that causes execution to take one of two branches.) In most languages offering conditional expressions, as in XQuery, those expressions are defined using the keyword if. In fact, XQuery’s grammar uses the BNF nonterminal symbol “IfExpr” to define such expressions, as seen in Grammar 11-17.

When an IfExpr is evaluated, the IfTestExpr is first evaluated to find its effective Boolean value, as described in Section 11.2. If the effective Boolean value is true, then the result of the IfExpr is the result of evaluating the IfTrueExpr. Otherwise, the result of the IfExpr is the result of evaluating the IfFalseExpr. Example 11-12 illustrates how we might decide which of two movies to watch tonight based on which of two other movies was released first.

11.5.15 Quantified Expressions

In XQuery, quantified expressions provide the ability to do existential quantification (“Does at least one of these values meet this criterion?”) and universal quantification (“Do all of these values meet this criterion?”). Let’s examine the syntax of quantified expressions in Grammar 11-18.

The Quantifier keyword some causes existential quantification to be evaluated, while the keyword every causes universal quantification. Each QuantifiedInClause declares a variable, whose type may optionally be specified, and binds it to the sequence of items resulting from the evaluation of the QuantifiedBindingSequence expression.

A variable declared in one QuantifiedInClause can be used in the QuantifiedTestExpression, and even in the QuantifiedBindingSequence of the QuantifiedInClauses that follow its own QuantifiedBindingSequence. (Wow! What a mouthful.)

The result of a QuantifiedExpr that specifies some is true if at least one evaluation of the QuantifiedTestExpression results in a value of true, while the result of a QuantifiedExpr that specifies every is true only if every evaluation of the QuantifiedTestExpression results in true. When some is specified and the result of the QuantifiedBindingSequence is the empty sequence, the result is false. Why? Because there are no values for which the QuantifiedTestExpression can evaluate to true. By contrast, when every is specified and the result of the QuantifiedBindingSequence is the empty sequence, the result is true, because there are no values for which the QuantifiedTestExpression can evaluate to false.

Example 11-13 illustrates the use of quantified expressions.

11.5.16 Expressions on XQuery Types

When you read Appendix C: XQuery 1.0 Grammar, you will see that a sequence type is the (data) type of something that can appear in a Data Model sequence – which is pretty much anything recognized by the Data Model. Sequence types can be specified (using the sequence type syntax) in variable declarations, as well as in function parameter declarations and results.

There are several other places in XQuery where sequence types are specified. In a couple of these, the sequence type itself is tested. The syntax of the five additional expressions in which sequence types are used – instance of, typeswitch, cast, castable, and treat – is shown in Grammar 11-19.

Let’s examine them one at a time.

An InstanceOfExpr is used to determine whether a given expression has a particular sequence type or not. Example 11-14 provides examples of using this expression and one example illustrating how it might be put to use in the context of a larger expression.

A query uses a TypeSwitchExpr to choose one of several expressions based on the dynamic (run-time) type of a test expression. In Example 11-15, you see an example of a type switch expression and an example of using it in context.

It is often necessary to convert values of one data type to another data type, depending on the specific needs of a query. For example, your query might retrieve a string from some element, knowing that the string is a sequence of digits, convert the string to an integer, and then use that integer value in a computation. The cast expression provides that capability for XQuery, as illustrated in Example 11-16.

There are several reasons why a cast might fail. The value being cast is first atomized. If atomization results in a sequence longer than 1, a run-time error is raised. If atomization results in an empty sequence and the sequence type was specified without the “?” (indicating that an empty sequence is permitted), a run-time error is raised. If the static type of the value being cast is not one that can be converted to the target type as indicated in the Functions and Operators specification, a run-time error is raised. Finally, if the actual value being cast cannot be converted to the target type, a run-time error is raised.

Which brings us to the next expression, the castable expression. Sometimes, in the context of a query, a cast is required under conditions where the query cannot guarantee that the values being cast are always appropriate for the target type. If such a cast is attempted, a run-time error is raised. But run-time errors are generally Not A Good Thing, especially when queries may be very complex and long-running – nobody wants her query to simply report “Error” after running for 15 minutes. (Unfortunately, XQuery 1.0 doesn’t have a way for a query to detect and handle errors – such as the try/catch blocks used in some languages.)

The castable expression allows you to write your queries in a self-protective manner, so casts that would fail at run time can be avoided.

Frequently, your query knows that a value being used is always of a specific known type or of a type derived from that known type. For example, your query might have to deal with data off the web that has not been carefully constructed with attention paid to certain details. One element, let’s call it RegionCode, in the data might be instances of either xs:integer or my:DVDRegionCode, which is derived from xs:integer. But the query author wants to ensure that only values representing region codes (that is, whose type is my:DVDRegionCode but not xs:integer) are actually processed and is willing to endure a run-time error if any other sort of data is encountered. The first treat expression, illustrated in Example 11-18, is used to provide this capability. A more relaxed query author might decide that values of either xs:integer or my : DVDRegionCode are acceptable, but not values of xs:double or xs:float. The second treat expression in the example illustrates this usage.

The purpose of the treat expression is to allow a query author to provide a guarantee that instance data being queried have appropriate data types. It has particular value when the static typing features is implemented and in use, because it provides information that the static type evaluation algorithms can use to determine and enforce the type correctness of expressions that use treat.

11.5.17 Validation Expression

Every XQuery expression that does not raise an error evaluates to some result, which is a sequence of items. As discussed both earlier in this chapter and in Chapter 10, that sequence might contain no items (the empty sequence), one item (singleton), or more than one item. The items in the sequence might be atomic values or complex values (such as XML documents or elements). When the result of an XQuery expression – whether it’s the “top-level” expression (that is, the QueryBody of an XQuery Module) or some expression nested deep within a QueryBody – is an XML document or an element, it’s very useful to know whether the result is valid or not (and even just how valid it is!) according to the associated XML Schemas.

In order to validate the result of an XQuery expression, the XML Schema or Schemas against which that result is to be validated either must be implicitly included in the environment in which the XQuery expression is evaluated, or it must have been imported via the use of the import schema clause in the XQuery prolog.

The result of successfully validating some node is a copy of that node (with a different identity!) in which it and all of its descendent nodes have been annotated with a validity assessment and a Data Model type. If validation fails, an error is raised.

XQuery supports validation of the results of expressions through the validate expression, whose syntax is given in Grammar 11-20.

The syntax of the validate expression is deceptively simple. Why? Because it depends on the rules of XML Schema Part 1⁹ to provide the detailed semantics of validation. The actual process of validation is discussed in Chapter 10 of this book.

The validated node either corresponds directly to the node being validated or, for a validated document node, to the only element child of the document node.

Example 11-19 provides a few examples of successful validation and validation efforts that will raise errors.

11.6.1 The for Clause and the let Clause

The XQuery spec introduces the for and let expressions with the unfortunate sentence “The purpose of the for and let clauses in a FLWOR expression is to produce a tuple stream in which each tuple consists of one or more bound variables.” Those of you familiar with relational theory will certainly recognize the word tuple, as will many others.

In this context, a tuple is a binding of a variable to a sequence of (zero or more) values or, by extension, pairs (or triples, etc.) of such bindings, depending on the number of variables used in the combination of for clauses and let clauses. A tuple stream is a sequence of such tuples that can be considered in turn.

Consider the for clause illustrated in Example 11-21, which calls on the movies document in Example 11-20 (seen in earlier chapters as well). Of course, as we can tell from the syntax of FLWOR expressions, a for clause cannot appear alone – at a minimum, it must be followed by a return clause.

Note that the result contains three instances of the variable $m, each instance being bound to a separate movie element from the original document. That result is a tuple stream comprising three tuples, each of which is an instance of the variable and the value (called its binding sequence) to which that instance is bound.

The for clause iterates over the items in the binding sequence, binding the variable to each of those items in turn. When the ordering mode applied to the FLWOR expression is ordered, the tuple stream is also ordered (in the same order as the binding sequence); when the ordering mode is unordered, the tuple stream’s order is implementation-dependent.

A for clause can have multiple variable bindings, as shown in Example 11-22, which depends on the two documents seen in Example 11-20. In this case, the two variables, $m and $ s, are each associated with a binding clause, but they are not independent of one another. Instead, each binding of $m is associated with every binding of $s. The XQuery 1.0 spec says, “The resulting tuple stream contains one tuple for each combination of values in the respective binding sequences.” Those of you familiar with the relational model – or with vector operations from your math classes – will recognize this as a cross product.

This concept can be extended arbitrarily to cover as many variables as the for clause supplies.

If the ordering mode in effect for a FLWOR expression whose for clause defines multiple variables is ordered, then the first variable provides the primary sort order, the second provides the secondary sort order, and so forth.

The let clause also binds variables with the values returned by expressions, but without iteration. Instead, the let clause binds its variables with the entire value of their respective expressions – the entire sequence, not one item of the sequence at a time. A let clause that binds two or more variables generates a single tuple containing all of the variable bindings, as illustrated in Example 11-23.

The scope of the variables bound in a for clause or a let clause is every subexpression in the same FLWOR expression that follows the individual for clause or a let clause in which the variable is bound (but not, of course, the expression to which the variable is bound). Consequently, an expression such as that in Example 11-24 is possible (if not necessarily useful in this case).

One often misunderstood implication of the rule that the “scope of the variables bound … is every subexpression … that follows“ is that a variable declared in one clause (a for clause, for example) can be apparently redeclared in a subsequent clause (a let clause, perhaps) in the same FLWOR expression, as illustrated in Example 11-25. However, that apparent redeclaration does no such thing. Instead, the second declaration is actually a declaration of a new variable of the same name, whose declaration obscures the previously declared variable of that name. As a result, expressions in clauses following the let clause (in this example) can never access the value of the variable $i declared in the for clause; all such efforts will see only the other variable $i declared in the let clause.

In both for clauses and let clauses, each variable being bound may be specified to have an explicit type. If the value bound to a variable with an explicitly declared type does not match that type, using the rules of sequence type matching, then a type error is raised.

In a for clause, a bound variable can be accompanied by a positional variable, whose value is an integer that represents the position of each value in the bound variable’s binding sequence in turn. Repeating Example 11-21 with a positional variable, we get the same results, as shown in Example 11-26.

11.6.2 The where Clause

As the grammar in Appendix C: XQuery 1.0 Grammar shows, FLWOR expressions can optionally include a where clause, the purpose of which is to filter the tuples generated by the preceding for and/or let clauses. The ExprSingle contained in the where clause, called the where expression, is evaluated once for each of those tuples, and only those tuples for which the effective Boolean value of the where expression is true are retained.

In Example 11-27, we have coded a FLWOR expression fragment in which a where clause is applied to a positional variable generated in a for clause that is otherwise identical to Example 11-26.

Note that the result in Example 11-27 is identical to the result in Example 11-26, except that one tuple – the tuple in which the value of $i and the value of the myStars attribute are both 3 – is absent.

And, yes, the where clause really is that simple. We leave as an exercise for the reader to determine the result of the FLWOR fragment in Example 11-28.

11.6.3 The order by Clause

The order by clause is used to reorder the tuples in the tuple stream generated by the for and/or let clauses, possibly filtered by a where clause. If a FLWOR expression does not contain an order by clause, then the order of tuples is determined by the for and/or let clauses and by the ordering mode (ordered or unordered, as discussed earlier in Section 11.4.13). If an order by clause is present, then it determines the order of those tuples based on values present in the tuples themselves. (Note that the ordering done by the order by clause is done by values and not by nodes or by node identity.)

An order by clause has one or more ordering specifications (OrderSpec), each of which contains an ExprSingle and an optional ordering modifier (OrderModifier). The ExprSingle is evaluated using the variable bindings in each tuple. The relative ordering of two tuples is determined by evaluating each OrderSpec, in left-to-right sequence, until an OrderSpec is encountered for which the two tuples do not compare equal. When evaluating an OrderSpec:

The optional OrderModifier can specify that the ordering is to be ascending or descending (that is, whether the tuples are delivered with the lowest values appearing first or last). It can also specify whether empty sequences and, for values of type xs:float and xs:double, the special value NaN (Not a Number), are sorted as greater than all other values (empty greatest) or less than all other values (empty least). The OrderModifier can also specify a collation that governs how xs:string (and, because of the cast cited earlier, xsd:untypedAtomic) values are compared.

If the order by clause specifies stable, then for any two tuples that compare equal for every OrderSpec, the relative order of those tuples is the same as in the original tuple stream. If stable is not specified, then the relative order of two such tuples is implementation-dependent.

Example 11-29 illustrates a FLWOR expression fragment that uses an order by clause.

11.6.4 The return Clause

Every FLWOR expression contains a return clause (which is why previous examples in this section are characterized as FLWOR fragments). The ExprSingle contained in a return clause is evaluated once for each tuple that is produced by the for clauses and/or let clauses and/or where clause and/or order by clause. The results of these evaluations are concatenated (as if they were assembled using the comma operator) into a sequence; the resulting sequence is the value of the FLWOR clause.

Example 11-29 can be completed by adding a return clause, as shown in Example 11-30.

The result in Example 11-30 is a sequence of two values, each the value of the attribute myStars of the element movie contained in a tuple in the tuple stream produced by the for clause, filtered by the where clause, and sorted by the order by clause.

In Appendix A: The Example, you will see other examples of FLWOR expressions.

11.8.1 Prologs

A MainModule is a Prolog followed by a QueryBody. We have seen examples of QueryBodys, and we have referred to the contents of the Prolog. Now it’s time to see exactly what the Prolog is.

The Prolog (frequently called the query prolog to distinguish it from the XML document prolog) provides syntax that allows authors of XQuery modules to declare several things that affect the behaviors of XQuery expressions. Some of the items that can be specified in a query prolog – such as boundary space policy, ordering mode, and default collation – override the implementation defaults for those items. Others – such as variable and namespace declarations – may augment implementation defaults, but do not override them. Information about each of these items is available in Table 11-1, and you can find the details of which can be overridden and that can be augmented in the XQuery 1.0 specification in its section entitled “The Static Context.”

It is a syntax error if any of these declarations are specified more than once in a query prolog. Some other declarations are permitted to appear more than once:

11.8.2 Main Modules

A main module is one that contains, in addition to a (possibly empty) query prolog, a query body – an expression that is evaluated when the module is invoked. A query has exactly one main module. Evaluating the expression that is the query body of a main module is the same as executing, or running, the query.

How a main module is invoked is very much left to the XQuery implementation. Some implementations may provide a command-line interface or a graphical user interface (GUI) that allows a query to be typed directly by a user. Other implementations may provide ways to embed XQueries into some other programming language, such as Java, C, or Python. Still others might allow applications to invoke methods in some application programming interface (API) and pass the text of XQueries and main modules for evaluation (see Chapter 14, “XQuery APIs,” for more information). Still others might provide for a GUI facility that builds queries without having to enter the character strings conforming to XQuery syntax. We expect that all implementations will provide at least one of these methods, and that some will provide more than one.

However, the sequence of events once a main module is invoked is well defined. In fact, it is generally described in Section 11.2.2, “The XQuery Processing Model.” That description does not cover every detail, so here is a more precise list of the steps involved in the invocation of a main module. Of course, before these steps can be performed, the invoking environment has to provide input data in the form of a Query Data Model instance, possibly by parsing a serialized XML document into an Infoset, perhaps performing Schema validation on that Infoset to produce a PSVI, and then transforming the result into a Data Model instance. (Note that several of these steps depend on the implementation’s providing optional features: schema import, modules, and static typing are all optional.)

Complete examples of main modules can be found in Appendix A: The Example.

11.8.3 Library Modules

Library modules support the notion of modularizing applications, which is done for reasons of design, maintenance, and code reuse. A library module comprises only a ModuleDecl and a Prolog. The ModuleDecl defines the target namespace of the library module, while the Prolog contains declarations for functions and variables that are exported (made available) for importation (inclusion) by other library modules and by main modules. Example 11-32 contains a scenario of importing library modules.

Some of the components of Example 11-32 deserve a few additional words of discussion.