The Structure of a Regular Expression

The regular expression syntax of XQuery is based on that of XML Schema, with some additions. Regular expressions, also known as regexes, can be composed of a number of different parts: atoms, quantifiers, and branches.

Representing Individual Characters

A single character can be used to represent itself in a regular expression. In this case, it is known as a normal character. For example, the regular expression d matches the letter d, and def matches the string def, as you might expect. Each of the three single characters (d, e, and f) is its own atom, and can have a quantifier associated with it. For example, the regular expression d+ef matches the strings def, ddef, dddef, etc.

Certain characters, in order to be taken literally, must be escaped because they have another meaning in a regular expression. For example, the asterisk (*) will be treated like a quantifier unless it is escaped. These characters, called metacharacters, must be escaped (except when they are within square brackets): ., , ?, *, +, |, ^, $, {, }, (, ), [, and ].

These characters are escaped by preceding them with a backslash. This is referred to as a single-character escape because there is only one matching character. For convenience, there are three additional single-character escapes for the whitespace characters tab, line feed, and carriage return. Table 19-4 lists the single-character escapes.

You can also use the standard XML syntax for character references and predefined entity references in regular expressions, as long as they are in quoted strings. For example, a space can be represented as  , and a less-than symbol (<) can be represented as <. This can be useful for special characters. It is described further in “XML Entity and Character References”.

Table 19-5 shows some examples of representing individual characters in regular expressions.

Representing Any Character

The period (.) has special significance in regular expressions: it matches any character except a line feed (#xA) or carriage return (#xD). The period character represents only one matching character, but a quantifier (such as *) can be applied to it to represent multiple characters.

Table 19-6 shows some examples of the wildcard escape character in use. For the third example in the table, assume a line feed character between f and o in the third column. This string does not match unless you are in dot-all mode.

It is important to note that the period loses its wildcard power when placed in a character class expression (within square brackets).

XQuery functions that use regular expressions allow you to indicate that the processor should operate in dot-all mode. This is specified using the letter s in the $flags argument. In dot-all mode, the period matches any character whatsoever, including the line feed (#xA) and carriage return (#xD). See “Using Flags”.

Representing Groups of Characters

Sometimes characters fall into convenient groups, such as decimal digits or punctuation characters. Three different kinds of escapes can be used to represent a group of characters: multi-character escapes, category escapes, and block escapes. Like single-character escapes, they all start with a backslash.

Character Class Expressions

Character class expressions, which are enclosed in square brackets, indicate a choice among several characters. These characters can be listed singly, expressed as a range of characters, or expressed as a combination of the two.

Reluctant Quantifiers

XQuery supports reluctant quantifiers, which allow part of a regular expression to match the shortest possible string. Reluctant quantifiers are indicated by adding a question mark (?) to the end of any of the kinds of quantifiers identified in Table 19-1.

For example, given the string reluctant and the regular expression r.*t, the regular expression could match reluct or reluctant. Since a standard quantifier (*) is used, the match is on the longest possible string, reluctant. If the regular expression were r.*?t instead, which uses a reluctant quantifier, it would match reluct, the shorter of the two strings.

Reluctant quantifiers come into play when replacing matching values in a string. Table 19-12 shows some examples of calls to the replace function that use reluctant and non-reluctant quantifiers.

Reluctant quantifiers have no effect on simply determining whether a string matches a regular expression, which explains why they are not supported in XML Schema. It may seem that the regular expression r.*?tly would not match the string reluctantly because r.*?t would match the shorter string reluct, leaving an extra antly which does not match the pattern ly. However, this is not the way it works. Reluctant quantifiers do not indicate that only the shorter string matches, just that the processor uses the shorter of the two matches if called on to perform a replacement or some other operation. Any of the quantifiers in the examples in Table 19-2 could be replaced by reluctant quantifiers, and the list of matching and non-matching strings would be the same.

Anchors

XQuery adds the concept of anchors to XML Schema regular expressions. In XML Schema validation, the regular expression is expected to match the entire string, not a part of it. For example, the regular expression str matches only the string str and not other strings that contain str, like 5str5. The XQuery matches function, on the other hand, will return true if any part of the string matches the pattern. For example, matches("5str5", "str") returns true because a portion of the string matches the regular expression str.

Because of this looser interpretation, it is sometimes useful to explicitly say that the expression should match the beginning or end of the string (or both). Anchors can be used for this purpose. The ^ character is used to match the beginning of the string, and the $ character is used to match the end of the string. For example, the regular expression ^str specifies that a matching string must start with str. Table 19-13 shows some examples that use anchors with the matches and replace functions.

XQuery functions that use regular expressions allow you to indicate that the processor should operate in multi-line mode. In multi-line mode, anchors match not just the beginning and end of the entire string, but also the beginning and end of any line within the string, as indicated by a line-feed character (#xA). This is specified using the letter m in the $flags argument, as described in “Using Flags”.

Back-References

XQuery supports the use of back-references. Back-references allow you to ensure that certain characters in a string match each other. For example, suppose you want to ensure that a string is a product number delimited by either single or double quotes. The product number must be three digits, followed by a hyphen, followed by two uppercase letters. You could write the expression:

('|")d{3}-[A-Z]{2}('|")

However, this would allow a string that starts with a single quote and ends with a double quote. You want to be sure the quotes match. You could write the expression:

'd{3}-[A-Z]{2}'|"d{3}-[A-Z]{2}"

but this requires repeating the entire pattern for the product number. Instead, you can parenthesize the expression representing the quotes and refer back to it by using an escaped digit. For example, the expression:

('|")d{3}-[A-Z]{2}1

is equivalent to the prior example, but it is shorter and simpler. The atom 1 indicates that you want to repeat the first parenthesized expression, namely ('|"). The characters that match the first parenthesized expression must be the same characters that match the back-reference. This means that the regular expression does not match a string that starts with a single quote and ends with a double quote.

The parenthesized sub-expressions are numbered in order from left to right based on the position of the opening parenthesis, starting with 1 (not 0). You can reference any of them by number. You can use as many digits as you want, provided that the number does not exceed the number of sub-expressions preceding it.

Using Flags

Four XQuery functions use regular expressions: matches, replace, tokenize, and analyze-string. Each of these functions accepts a $flags argument that allows for additional options in the interpretation of the regular expression, such as multi-line processing and case insensitivity. Options are indicated by single letters; the $flags argument is a string that can contain any of the valid letters in any order, and duplicates are allowed.

The $flags argument allows five options:

s

The letter s indicates dot-all mode, which affects the period wildcard (.). (This is known as single-line mode in Perl.) This means that the period wildcard matches any character whatsoever, including line-feed and carriage return characters. If the letter s is not specified, the period wildcard matches any character except the line-feed (#xA) or carriage return (#xD) character.

m

The letter m indicates multi-line mode, which affects anchors. In multi-line mode, the ^ and $ characters match the beginning and end of a line, as well as the beginning and end of the whole string. For the purposes of the m flag, the line-feed (#xA) character delimits lines.

i

The letter i indicates case-insensitive mode. This means that matching does not distinguish between normal characters that are case variants of each other, as defined by Unicode. For example, in case-insensitive mode, [a-z] matches the lowercase letters a through z, uppercase letters A through Z, and a few other characters such as a Kelvin sign. The meaning of category escapes such as p{Lu} is not affected.

x

The letter x indicates that whitespace characters within regular expressions should be ignored. This is useful for making long regexes readable by splitting over many lines. If x is not specified, whitespace characters are considered to be significant and must match those in the string. If you want to represent significant whitespace when using the x flag, you can use the multi-character escape s.

q

The letter q means that all metacharacters (such as [ or *, that would normally need to be escaped to be interpreted literally) are taken literally even without being escaped. For example, a.b would normally mean “a followed by any character followed by b,” but if used with the q flag, it will only match “a followed by a period followed by b.” This can be useful if you want to make simple text replacements without worrying about regular expression characters. The q flag can be used with the i flag, but if it is used with any of the other flags, it renders them ineffective.

If no flag options are desired, you should either pass a zero-length string, or omit the $flags argument entirely. Table 19-14 shows some examples of how the $flags argument, which is the third argument of the matches function, affects interpretation of regular expressions. They assume the following variable declaration (where the line break is significant):

declare variable $address := "123 Main Street
Traverse City, MI 49684";

Using Sub-Expressions with Replacement Variables

The replace function allows parenthesized sub-expressions (also known as groups) to be referenced by number in the replacement string. In the $replacement string, you can use the variables $1, $2, $3, etc., to represent (in order) the parenthesized expressions in $pattern. This is very useful when replacing strings on the condition that they come directly before or after another string—for example, if you want to change instances of the word Chap to the word Sec, but only those that are followed by a space and a digit. This technique can also be used to reformat data for presentation. Table 19-15 shows some examples.

The variables are bound in order from left to right based on the position of the opening parenthesis. The variable $0 can be used to represent the string matched by the entire regular expression. If the variable number exceeds the number of parenthesized sub-expressions in the regular expression, it is replaced with a zero-length string.

If you wish to include the character $ in your replacement string, you must escape it with a backslash (i.e., $), as shown in the fifth example. Backslashes must also be escaped in the $replacement string, as in \.

Starting in version 3.0, if you have a set of parentheses that you don’t want to count for the purpose of sub-expressions, you can make it a non-capturing group by putting a ?: directly after the left parenthesis. For example, without non-capturing groups, the following query only refers to the second, third, and fourth groups.

replace("2015-10-18", "(d{2})(d{2})-(d{2})-(d{2})", "$3/$4/$2")

If the first group is changed to be a non-capturing group, the variable numbers need to change, as in:

replace("2015-10-18", "(?:d{2})(d{2})-(d{2})-(d{2})", "$2/$3/$1")

With this change, the first group is no longer counted as the first one.