Description

A token identifies a unit of information. Usually, “tokens” are the result of some processing pass that has performed lexical analysis and divided a data set into the smallest units of information used for subsequent processing.

Exactly what constitutes a token varies by context.

See Also

classname, property, symbol, type

Examples

<article xmlns='http://docbook.org/ns/docbook'>
<title>Example token</title>

<para>In parsing, line ends are turned into the <token>CRLF</token>, all other
whitespace becomes <token>WHITESP</token>.
</para>

</article>