As the name implies, the SimpleTokenizer class performs simple tokenization of text. The INSTANCE field is used to instantiate the class as shown in the following code sequence. The tokenize method is executed against the paragraph variable and the tokens are then displayed:

SimpleTokenizer simpleTokenizer = SimpleTokenizer.INSTANCE;
String tokens[] = simpleTokenizer.tokenize(paragraph);
for(String token : tokens) {
    System.out.println(token);
}

When executed, we get the following output:

Let
'
s
pause
,
and
then
reflect
.

Using this tokenizer, punctuation is returned as separate tokens.

As its name implies, this class uses whitespaces as delimiters. In the following code sequence, an instance of the tokenizer is created and the tokenize method is executed against it using paragraph as input. The for statement then displays the tokens:

String tokens[] = WhitespaceTokenizer.INSTANCE.tokenize(paragraph);
for (String token : tokens) {
    System.out.println(token);
}

The output is as follows:

Let's
pause,
and
then
reflect.

Although this does not separate contractions and similar units of text, it can be useful for some applications. The class also possesses a tokizePos method that returns boundaries of the tokens.

The TokenizerME class uses models created using Maximum Entropy (maxent) and a statistical model to perform tokenization. The maxent model is used to determine the relationship between data, in our case, text. Some text sources, such as various social media, are not well formatted and use a lot of slang and special symbols such as emoticons. A statistical tokenizer, such as the maxent model, improves the quality of the tokenization process.

A TokenizerModel class hides the model and is used to instantiate the tokenizer. The model must have been previously trained. In the next example, the tokenizer is instantiated using the model found in the en-token.bin file. This model has been trained to work with common English text.

The location of the model file is returned by the method getModelDir, which you will need to implement. The returned value is dependent on where the models are stored on your system. Many of these models can be found at http://opennlp.sourceforge.net/models-1.5/.

After the instance of a FileInputStream class is created, the input stream is used as the argument of the TokenizerModel constructor. The tokenize method will generate an array of strings. This is followed by code to display the tokens:

try {
    InputStream modelInputStream = new FileInputStream(
        new File(getModelDir(), "en-token.bin"));
    TokenizerModel model = new TokenizerModel(modelInputStream);
    Tokenizer tokenizer = new TokenizerME(model);
    String tokens[] = tokenizer.tokenize(paragraph);
    for (String token : tokens) {
        System.out.println(token);
    }
} catch (IOException ex) {
    // Handle the exception
}

The output is as follows:

Let
's
pause
,
and
then
reflect
.

This tokenizer mimics the Penn Treebank 3 (PTB) tokenizer (http://www.cis.upenn.edu/~treebank/). It differs from PTB in terms of its options and its support for Unicode. The PTBTokenizer class supports several older constructors; however, it is suggested that the three-argument constructor be used. This constructor uses a Reader object, a LexedTokenFactory<T>argument, and a string to specify which of the several options to use.

The LexedTokenFactory interface is implemented by the CoreLabelTokenFactory and WordTokenFactory classes. The former class supports the retention of the beginning and ending character positions of a token whereas the latter class simply returns a token as a string without any positional information. The WordTokenFactory class is used by default. We will demonstrate the use of both classes.

The CoreLabelTokenFactory class is used in the following example. A StringReader instance is created using paragraph. The last argument is used for the options, which is null for this example. The Iterator interface is implemented by the PTBTokenizer class allowing us to use the hasNext and next method to display the tokens.

PTBTokenizer ptb = new PTBTokenizer(
    new StringReader(paragraph), new CoreLabelTokenFactory(),null);
while (ptb.hasNext()) {
    System.out.println(ptb.next());
}

The output is as follows:

Let
's
pause
,
and
then
reflect
.

The same output can be obtained using the WordTokenFactory class, as shown here:

PTBTokenizerptb = new PTBTokenizer(
    new StringReader(paragraph), new WordTokenFactory(), null);

The power of the CoreLabelTokenFactory class is realized with the options parameter of the PTBTokenizer constructor. These options provide a means to control the behavior of the tokenizer. Options include such controls as how to handle quotes, how to map ellipses, and whether it should treat British English spellings or American English spellings. A list of options can be found at http://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/process/PTBTokenizer.html.

In the following code sequence, the PTBTokenizer object is created using the CoreLabelTokenFactory variable ctf along with an option of "invertible=true". This option allows us to obtain and use a CoreLabel object which will give us the beginning and ending position of each token:

CoreLabelTokenFactory ctf = new CoreLabelTokenFactory();
PTBTokenizer ptb = new PTBTokenizer(
    new StringReader(paragraph),ctf,"invertible=true");
while (ptb.hasNext()) {
    CoreLabel cl = (CoreLabel)ptb.next();
    System.out.println(cl.originalText() + " (" + 
        cl.beginPosition() + "-" + cl.endPosition() + ")");
}

The output of this sequence is as follows. The numbers within the parentheses indicate the tokens' beginning and ending positions:

Let (0-3)
's (3-5)
pause (6-11)
, (11-12)
and (14-17)
then (18-22)
reflect (23-30)
. (30-31)

The DocumentPreprocessor class tokenizes input from an input stream. In addition, it implements the Iterable interface making it easy to traverse the tokenized sequence. The tokenizer supports the tokenization of simple text and XML data.

To illustrate this process, we will use an instance of StringReader class that uses the paragraph string, as defined here:

Reader reader = new StringReader(paragraph);

An instance of the DocumentPreprocessor class is then instantiated:

DocumentPreprocessor documentPreprocessor =
      new DocumentPreprocessor(reader);

The DocumentPreprocessor class implements the Iterable<java.util.List<HasWord>> interface. The HasWord interface contains two methods that deal with words: a setWord and a word method. The latter method returns a word as a string. In the next code sequence, the DocumentPreprocessor class splits the input text into sentences which are stored as a List<HasWord>. An Iterator object is used to extract a sentence and then a for-each statement will display the tokens:

Iterator<List<HasWord>> it = documentPreprocessor.iterator();
while (it.hasNext()) {
    List<HasWord> sentence = it.next();
    for (HasWord token : sentence) {
        System.out.println(token);
    }
}

When executed, we get the following output:

Let
's
pause
,
and
then
reflect
.

Here, we will use the StanfordCoreNLP class as demonstrated in Chapter 1, Introduction to NLP. However, we use a simpler annotator string to tokenize the paragraph. As shown next, a Properties object is created and assigned the annotators tokenize and ssplit.

The tokenize annotator specifies that tokenization will occur and the ssplit annotation results in sentences being split:

Properties properties = new Properties();
properties.put("annotators", "tokenize, ssplit");

The StanfordCoreNLP class and the Annotation classes are created next:

StanfordCoreNLP pipeline = new StanfordCoreNLP(properties);
Annotation annotation = new Annotation(paragraph);

The annotate method is executed to tokenize the text and then the prettyPrint method will display the tokens:

pipeline.annotate(annotation);
pipeline.prettyPrint(annotation, System.out);

Various statistics are displayed followed by the tokens marked up with position information in the output, which is as follows:

Sentence #1 (8 tokens):
Let's pause, 
and then reflect.
[Text=Let CharacterOffsetBegin=0 CharacterOffsetEnd=3] [Text='s CharacterOffsetBegin=3 CharacterOffsetEnd=5] [Text=pause CharacterOffsetBegin=6 CharacterOffsetEnd=11] [Text=, CharacterOffsetBegin=11 CharacterOffsetEnd=12] [Text=and CharacterOffsetBegin=14 CharacterOffsetEnd=17] [Text=then CharacterOffsetBegin=18 CharacterOffsetEnd=22] [Text=reflect CharacterOffsetBegin=23 CharacterOffsetEnd=30] [Text=. CharacterOffsetBegin=30 CharacterOffsetEnd=31]

LingPipe supports a number of tokenizers. In this section, we will illustrate the use of the IndoEuropeanTokenizerFactory class. In later sections, we will demonstrate other ways that LingPipe supports tokenization. Its INSTANCE field provides an instance of an Indo-European tokenizer. The tokenizer method returns an instance of a Tokenizer class based on the text to be processed, as shown here:

char text[] = paragraph.toCharArray();
TokenizerFactory tokenizerFactory = IndoEuropeanTokenizerFactory.INSTANCE;
Tokenizer tokenizer = tokenizerFactory.tokenizer(text, 0, text.length);
for (String token : tokenizer) {
    System.out.println(token);
}

The output is as follows:

Let
'
s
pause
,
and
then
reflect
.

These tokenizers support the tokenization of "normal" text. In the next section, we will demonstrate how a tokenizer can be trained to deal with unique text.