Another option to load the documents is through cc.mallet.pipe.iterator.CsvIterator.CsvIterator(Reader, Pattern, int, int, int), which assumes all of the documents are in a single file and returns one instance per line extracted by a regular expression. The class is initialized by the following components:

• Reader: This is the object that specifies how to read from a file
• Pattern: This is a regular expression, extracting three groups: data, target label, and document name
• int, int, int: These are the indexes of data, target, and name groups as they appear in a regular expression

Consider a text document in the following format, specifying the document name, category, and content:

AP881218 local-news A 16-year-old student at a private 
   Baptist...  
AP880224 business The Bechtel Group Inc. offered in 1985 to...  
AP881017 local-news A gunman took a 74-year-old woman hostage...  
AP900117 entertainment Cupid has a new message for lovers 
   this...  
AP880405 politics The Reagan administration is weighing w...  

To parse a line into three groups, we can use the following regular expression:

^(\S*)[\s,]*(\S*)[\s,]*(.*)$

There are three groups that appear in parenthesies (), where the third group contains the data, the second group contains the target class, and the first group contains the document ID. iterator is initialized as follows:

CsvIterator iterator = new CsvIterator ( 
fileReader, 
Pattern.compile("^(\S*)[\s,]*(\S*)[\s,]*(.*)$"), 
  3, 2, 1)); 

Here, the regular expression extracts the three groups separated by an empty space and their order is 3, 2, 1.

Now let's move to the data-preprocessing pipeline.