StringTokenizer was the first and most basic tokenizer and has been available since Java 1. It is not recommended for use in new development as the String class's split method is considered more efficient. While it does provide a speed advantage for files with narrowly defined and set delimiters, it is less flexible than other tokenizer options. The following is a simple implementation of the StringTokenizer class that splits a string on spaces:

StringTokenizer tokenizer = new StringTokenizer(dirtyText," "); 
while(tokenizer.hasMoreTokens()){ 
  out.print(tokenizer.nextToken() + " "); 
} 

When we set the  dirtyText variable to hold our text from Moby Dick, shown previously, we get the following truncated output:

Call me Ishmael. Some years ago- never mind how long precisely...

StreamTokenizer is another core Java tokenizer. StreamTokenizer grants more information about the tokens retrieved, and allows the user to specify data types to parse, but is considered more difficult to use than StreamTokenizer or the split method. The String class split method is the simplest way to split strings up based on a delimiter, but it does not provide a way to parse the split strings and you can only specify one delimiter for the entire string. For these reasons, it is not a true tokenizer, but it can be useful for data cleaning.

The Scanner class is designed to allow you to parse strings into different data types. We used it previously in the Handling CSV data section and we will address it again in the Removing stop words section.

Apache Commons consists of sets of open source Java classes and methods. These provide reusable code that complements the standard Java APIs. One popular class included in the Commons is StrTokenizer. This class provides more advanced support than the standard StringTokenizer class, specifically more control and flexibility. The following is a simple implementation of the StrTokenizer:

StrTokenizer tokenizer = new StrTokenizer(text); 
while (tokenizer.hasNext()) { 
  out.print(tokenizer.next() + " "); 
} 

This operates in a similar fashion to StringTokenizer and by default parses tokens on spaces. The constructor can specify the delimiter as well as how to handle double quotes contained in data.

When we use the string from Moby Dick, shown previously, the first tokenizer implementation produces the following truncated output:

Call me Ishmael. Some years ago- never mind how long precisely - having little or no money in my purse...

We can modify our constructor as follows:

StrTokenizer tokenizer = new StrTokenizer(text,","); 

The output for this implementation is:

Call me Ishmael. Some years ago- never mind how long precisely - having little or no money in my purse
and nothing particular to interest me on shore
I thought I would sail about a little and see the watery part of the world.

Notice how each line is split where commas existed in the original text. This delimiter can be a simple char, as we have shown, or a more complex StrMatcher object.

Google Guava is an open source set of utility Java classes and methods. The primary goal of Guava, as with many APIs, is to relieve the burden of writing basic Java utilities so developers can focus on business processes. We are going to talk about two main tools in Guava in this chapter: the Joiner class and the Splitter class. Tokenization is accomplished in Guava using its Splitter class's split method. The following is a simple example:

Splitter simpleSplit = Splitter.on(',').omitEmptyStrings().trimResults(); 
Iterable<String> words = simpleSplit.split(dirtyText);  
for(String token: words){ 
  out.print(token); 
} 

This splits each token on commas and produces output like our last example. We can modify the parameter of the on method to split on the character of our choosing. Notice the method chaining which allows us to omit empty strings and trim leading and trailing spaces. For these reasons, and other advanced capabilities, Google Guava is considered by some to be the best tokenizer available for Java.

LingPipe is a linguistical toolkit available for language processing in Java. It provides more specialized support for text splitting with its TokenizerFactory interface. We implement a LingPipe IndoEuropeanTokenizerFactory tokenizer in the Simple text cleaning section.

We will use the string shown before from Moby Dick to demonstrate some of the basic String class methods. Notice the use of the toLowerCase and trim methods. Datasets often have non-standard casing and extra leading or trailing spaces. These methods ensure uniformity of our dataset. We also use the replaceAll method twice. In the first instance, we use a regular expression to replace all numbers and anything that is not a word or whitespace character with a single space. The second instance replaces all back-to-back whitespace characters with a single space:

out.println(dirtyText); 
dirtyText =    dirtyText.toLowerCase().replaceAll("[\d[^\w\s]]+", " "); 
dirtyText = dirtyText.trim(); 
while(dirtyText.contains("  ")){ 
  dirtyText = dirtyText.replaceAll("  ", " "); 
} 
out.println(dirtyText);  

When executed, the code produces the following output, truncated:

Call me Ishmael. Some years ago- never mind how long precisely -
call me ishmael some years ago never mind how long precisely

Our next example produces the same result but approaches the problem with regular expressions. In this case, we replace all of the numbers and other special characters first. Then we use method chaining to standardize our casing, remove leading and trailing spaces, and split our words into a String array. The split method allows you to break apart text on a given delimiter. In this case, we chose to use the regular expression \W, which represents anything that is not a word character:

out.println(dirtyText); 
dirtyText = dirtyText.replaceAll("[\d[^\w\s]]+", ""); 
String[] cleanText = dirtyText.toLowerCase().trim().split("[\W]+"); 
for(String clean : cleanText){ 
  out.print(clean + " ");
} 

This code produces the same output as shown previously.

Although arrays are useful for many applications, it is often important to recombine text after cleaning. In the next example, we employ the join method to combine our words once we have cleaned them. We use the same chained methods as shown previously to clean and split our text. The join method joins every word in the array words and inserts a space between each word:

out.println(dirtyText); 
String[] words =    dirtyText.toLowerCase().trim().split("[\W\d]+"); 
String cleanText = String.join(" ", words); 
out.println(cleanText); 

Again, this code produces the same output as shown previously. An alternate version of the join method is available using Google Guava. Here is a simple implementation of the same process we used before, but using the Guava Joiner class:

out.println(dirtyText);  
String[] words =    dirtyText.toLowerCase().trim().split("[\W\d]+"); 
String cleanText = Joiner.on(" ").skipNulls().join(words); 
out.println(cleanText); 

This version provides additional options, including skipping nulls, as shown before. The output remains the same.

Text analysis sometimes requires the omission of common, non-specific words such as the, and, or but. These words are known as stop words are there are several tools available for removing them from text. There are various ways to store a list of stop words, but for the following examples, we will assume they are contained in a file. To begin, we create a new Scanner object to read in our stop words. Then we take the text we wish to transform and store it in an ArrayList using the Arrays class's asList method. We will assume here the text has already been cleaned and normalized. It is essential to consider casing when using String class methods—and is not the same as AND or And, although all three may be stop words you wish to eliminate:

Scanner readStop = new Scanner(new File("C://stopwords.txt")); 
ArrayList<String> words = new    ArrayList<String>(Arrays.asList((dirtyText)); 
out.println("Original clean text: " + words.toString()); 

We also create a new ArrayList to hold a list of stop words actually found in our text. This will allow us to use the ArrayList class removeAll method shortly. Next, we use our Scanner to read through our file of stop words. Notice how we also call the toLowerCase and trim methods against each stop word. This is to ensure that our stop words match the formatting in our text. In this example, we employ the contains method to determine whether our text contains the given stop word. If so, we add it to our foundWords ArrayList. Once we have processed all the stop words, we call removeAll to remove them from our text:

ArrayList<String> foundWords = new ArrayList(); 
while(readStop.hasNextLine()){ 
  String stopWord = readStop.nextLine().toLowerCase(); 
  if(words.contains(stopWord)){ 
    foundWords.add(stopWord); 
  } 
} 
words.removeAll(foundWords); 
out.println("Text without stop words: " + words.toString()); 

The output will depend upon the words designated as stop words. If your stop words file contains different words than used in this example, your output will differ slightly. Our output follows:

Original clean text: [call, me, ishmael, some, years, ago, never, mind, how, long, precisely, having, little, or, no, money, in, my, purse, and, nothing, particular, to, interest, me, on, shore, i, thought, i, would, sail, about, a, little, and, see, the, watery, part, of, the, world]
Text without stop words: [call, ishmael, years, ago, never, mind, how, long, precisely

There is also support outside of the standard Java libraries for removing stop words. We are going to look at one example, using LingPipe. In this example, we start by ensuring that our text is normalized in lowercase and trimmed. Then we create a new instance of the TokenizerFactory class. We set our factory to use default English stop words and then tokenize the text. Notice that the tokenizer method uses a char array, so we call toCharArray against our text. The second parameter specifies where to begin searching within the text, and the last parameter specifies where to end:

text = text.toLowerCase().trim(); 
TokenizerFactory fact = IndoEuropeanTokenizerFactory.INSTANCE; 
fact = new EnglishStopTokenizerFactory(fact); 
Tokenizer tok = fact.tokenizer(text.toCharArray(), 0, text.length()); 
for(String word : tok){ 
  out.print(word + " "); 
} 

The output follows:

Call me Ishmael. Some years ago- never mind how long precisely - having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world.
call me ishmael . years ago - never mind how long precisely - having little money my purse , nothing particular interest me shore , i thought i sail little see watery part world .

Notice the differences between our previous examples. First of all, we did not clean the text as thoroughly and allowed special characters, such as the hyphen, to remain in the text. Secondly, the LingPipe list of stop words differs from the file we used in the previous example. Some words are removed, but LingPipe was less restrictive and allowed more words to remain in the text. The type and number of stop words you use will depend upon your particular application.

We often not only want to find text but also replace it with something else. We begin our next example much like we did the previous examples, by specifying our text, our text to locate, and invoking the contains method. If we find the text, we call the replaceAll method to modify our string:

text = text.toLowerCase().trim(); 
toFind = toFind.toLowerCase().trim(); 
out.println(text); 
 
if(text.contains(toFind)){ 
      text = text.replaceAll(toFind, replaceWith); 
      out.println(text); 
} 

To test this code, we set toFind to the word I and replaceWith to Ishmael. Our output follows:

call me ishmael. some years ago- never mind how long precisely - having little or no money in my purse, and nothing particular to interest me on shore, i thought i would sail about a little and see the watery part of the world.
call me ishmael. some years ago- never mind how long precisely - having little or no money in my purse, and nothing particular to interest me on shore, Ishmael thought Ishmael would sail about a little and see the watery part of the world.

Apache Commons also provides a replace method with several variations in the StringUtils class. This class provides much of the same functionality as the String class, but with more flexibility and options. In the following example, we use our string from Moby Dick and replace all instances of the word me with X to demonstrate the replace method:

out.println(text); 
out.println(StringUtils.replace(text, "me", "X")); 

The truncated output follows:

Call me Ishmael. Some years ago- never mind how long precisely -
Call X Ishmael. SoX years ago- never mind how long precisely -

Notice how every instance of me has been replaced, even those instances contained within other words, such as some. This can be avoided by adding spaces around me , although this will ignore any instances where me is at the end of the sentence, like me. We will examine a better alternative using Google Guava in a moment.

The StringUtils class also provides a replacePattern method that allows you to search for and replace text based upon a regular expression. In the following example, we replace all non-word characters, such as hyphens and commas, with a single space:

out.println(text); 
text = StringUtils.replacePattern(text, "\W\s", " "); 
out.println(text); 

This will produce the following truncated output:

Call me Ishmael. Some years ago- never mind how long precisely - 
Call me Ishmael Some years ago never mind how long precisely

Google Guava provides additional support for matching and modify text data using the CharMatcher class. CharMatcher not only allows you to find data matching a particular char pattern, but also provides options as to how to handle the data. This includes allowing you to retain the data, replace the data, and trim whitespaces from within a particular string.

In this example, we are going to use the replace method to simply replace all instances of the word me with a single space. This will produce series of empty spaces within our text. We will then collapse the extra whitespace using the trimAndCollapseFrom method and print our string again:

text = text.replace("me", " "); 
out.println("With double spaces: " + text); 
String spaced = CharMatcher.WHITESPACE.trimAndCollapseFrom(text, ' '); 
out.println("With double spaces removed: " + spaced); 

Our output is truncated as follows:

With double spaces: Call Ishmael. So years ago- ...
With double spaces removed: Call Ishmael. So years ago- ...

Sometimes we simply need to validate whether a piece of data is of a specific type, such as integer or floating point data. We will demonstrate in the next example how to validate integer data using the validateInt method. This technique is easily modified for the other major data types supported in the standard Java library, including Float and Double.

We need to use a try-catch block here to catch a NumberFormatException. If an exception is thrown, we know our data is not a valid integer. We first pass our text to be tested to the parseInt method of the Integer class. If the text can be parsed as an integer, we simply print out the integer. If an exception is thrown, we display information to that effect:

public static void validateInt(String toValidate){ 
try{ 
      int validInt = Integer.parseInt(toValidate); 
      out.println(validInt + " is a valid integer"); 
}catch(NumberFormatException e){ 
      out.println(toValidate + " is not a valid integer"); 
 
} 

We will use the following method calls to test our method:

validateInt("1234"); 
validateInt("Ishmael"); 

The output follows:

1234 is a valid integer
Ishmael is not a valid integer

The Apache Commons contain an IntegerValidator class with additional useful functionalities. In this first example, we simply duplicate the process from before, but use IntegerValidator methods to accomplish our goal:

public static String validateInt(String text){ 
      IntegerValidator intValidator = IntegerValidator.getInstance(); 
      if(intValidator.isValid(text)){ 
            return text + " is a valid integer"; 
      }else{ 
            return text + " is not a valid integer"; 
      }      
} 

We again use the following method calls to test our method:

validateInt("1234"); 
validateInt("Ishmael"); 

The output follows:

1234 is a valid integer
Ishmael is not a valid integer

The IntegerValidator class also provides methods to determine whether an integer is greater than or less than a specific value, compare the number to a ranger of numbers, and convert Number objects to Integer objects. Apache Commons has a number of other validator classes. We will examine a few more in the rest of this section.

Many times our data validation is more complex than simply determining whether a piece of data is the correct type. When we want to verify that the data is a date for example, it is insufficient to simply verify that it is made up of integers. We may need to include hyphens and slashes, or ensure that the year is in two-digit or four-digit format.

To do this, we have created another simple method called validateDate. The method takes two String parameters, one to hold the date to validate and the other to hold the acceptable date format. We create an instance of the SimpleDateFormat class using the format specified in the parameter. Then we call the parse method to convert our String date to a Date object. Just as in our previous integer example, if the data cannot be parsed as a date, an exception is thrown and the method returns. If, however, the String can be parsed to a date, we simply compare the format of the test date with our acceptable format to determine whether the date is valid:

 
public static String validateDate(String theDate, String dateFormat){ 
      try { 
            SimpleDateFormat format = new SimpleDateFormat(dateFormat); 
            Date test = format.parse(theDate); 
            if(format.format(test).equals(theDate)){ 
                  return theDate.toString() + " is a valid date"; 
            }else{ 
                  return theDate.toString() + " is not a valid date"; 
            } 
      } catch (ParseException e) { 
            return theDate.toString() + " is not a valid date"; 
      } 
} 

We make the following method calls to test our method:

String dateFormat = "MM/dd/yyyy"; 
out.println(validateDate("12/12/1982",dateFormat)); 
out.println(validateDate("12/12/82",dateFormat)); 
out.println(validateDate("Ishmael",dateFormat)); 

The output follows:

12/12/1982 is a valid date
12/12/82 is not a valid date
Ishmael is not a valid date

This example highlights why it is important to consider the restrictions you place on data. Our second method call did contain a legitimate date, but it was not in the format we specified. This is good if we are looking for very specifically formatted data. But we also run the risk of leaving out useful data if we are too restrictive in our validation.

It is also common to need to validate e-mail addresses. While most e-mail addresses have the @ symbol and require at least one period after the symbol, there are many variations. Consider that each of the following examples can be valid e-mail addresses:

One option is to use regular expressions to attempt to capture all allowable e-mail addresses. Notice that the regular expression used in the method that follows is very long and complex. This can make it easy to make mistakes, miss valid e-mail addresses, or accept invalid addresses as valid. But a carefully crafted regular expression can be a very powerful tool.

We use the Pattern and Matcher classes to compile and execute our regular expression. If the pattern of the e-mail we pass in matches the regular expression we defined, we will consider that text to be a valid e-mail address:

public static String validateEmail(String email) { 
      String emailRegex = "^[a-zA-Z0-9.!$'*+/=?^_`{|}~-" +
          "]+@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\." + 
          "[0-9]{1,3}\])|(([a-zAZ\-0-9]+\.)+[a-zA-Z]{2,}))$"; 
      Pattern.compile(emailRegex); 
      Matcher matcher = pattern.matcher(email); 
      if(matcher.matches()){ 
            return email + " is a valid email address"; 
      }else{ 
            return email + " is not a valid email address"; 
      } 
} 

We make the following method calls to test our data:

out.println(validateEmail("[email protected]")); 
out.println(validateEmail("[email protected]")); 
out.println(validateEmail("myEmail")); 

The output follows:

[email protected] is a valid email address
[email protected] is a valid email address
myEmail is not a valid email address

There is a standard Java library for validating e-mail addresses as well. In this example, we use the InternetAddress class to validate whether a given string is a valid e-mail address or not:

    public static String validateEmailStandard(String email){ 
        try{ 
            InternetAddress testEmail = new InternetAddress(email); 
            testEmail.validate(); 
            return email + " is a valid email address"; 
        }catch(AddressException e){ 
            return email + " is not a valid email address"; 
        } 
    } 

When tested against the same data as in the previous example, our output is identical. However, consider the following method call:

    out.println(validateEmailStandard("myEmail@mail")); 
 

Despite not being in standard e-mail format, the output is as follows:

myEmail@mail is a valid email address

Additionally, the validate method by default accepts local e-mail addresses as valid. This is not always desirable, depending upon the purpose of the data.

One last option we will look at is the Apache Commons EmailValidator class. This class's isValid method examines an e-mail address and determines whether it is valid or not. Our validateEmail method shown previously is modified as follows to use EmailValidator:

public static String validateEmailApache(String email){ 
      email = email.trim(); 
      EmailValidator eValidator = EmailValidator.getInstance(); 
      if(eValidator.isValid(email)){ 
            return email + " is a valid email address."; 
      }else{ 
            return email + " is not a valid email address."; 
      } 
} 
 

Postal codes are generally formatted specific to their country or local requirements. For this reason, regular expressions are useful because they can accommodate any postal code required. The example that follows demonstrates how to validate a standard United States postal code, with or without the hyphen and last four digits:

public static void validateZip(String zip){ 
      String zipRegex = "^[0-9]{5}(?:-[0-9]{4})?$"; 
      Pattern pattern = Pattern.compile(zipRegex); 
      Matcher matcher = pattern.matcher(zip); 
      if(matcher.matches()){ 
            out.println(zip + " is a valid zip code"); 
      }else{ 
            out.println(zip + " is not a valid zip code"); 
      } 
} 

We make the following method calls to test our data:

out.println(validateZip("12345")); 
out.println(validateZip("12345-6789")); 
out.println(validateZip("123")); 

The output follows:

12345 is a valid zip code
12345-6789 is a valid zip code
123 is not a valid zip code

Names can be especially tricky to validate because there are so many variations. There are no industry standards or technical limitations, other than what characters are available on the keyboard. For this example, we have chosen to use Unicode in our regular expression because it allows us to match any character from any language. The Unicode property \p{L} provides this flexibility. We also use  \s-', to allow spaces, apostrophes, commas, and hyphens in our name fields. It is possible to perform string cleaning, as discussed earlier in this chapter, before attempting to match names. This will simplify the regular expression required:

public static void validateName(String name){ 
      String nameRegex = "^[\p{L}\s-',]+$"; 
      Pattern pattern = Pattern.compile(nameRegex); 
      Matcher matcher = pattern.matcher(name); 
      if(matcher.matches()){ 
            out.println(name + " is a valid name"); 
      }else{ 
            out.println(name + " is not a valid name"); 
      } 
} 

We make the following method calls to test our data:

validateName("Bobby Smith, Jr."); 
validateName("Bobby Smith the 4th"); 
validateName("Albrecht Müller"); 
validateName("François Moreau"); 

The output follows:

Bobby Smith, Jr. is a valid name
Bobby Smith the 4th is not a valid name
Albrecht Müller is a valid name
François Moreau is a valid name

Notice that the comma and period in Bobby Smith, Jr. are acceptable, but the 4 in 4th is not. Additionally, the special characters in François and Müller are considered valid.