Partitioning text into sentences is also called Sentence Boundary Disambiguation (SBD). This process is useful for many downstream NLP tasks that require analysis within sentences; for example, POS and phrase analysis typically work within a sentence.
In this chapter, we will explain in further detail why SBD is difficult. Then we will examine some core Java approaches that may work in some situations, and move on to the use of models by various NLP APIs. We will also examine training and validating approaches for sentence detection models. We can add additional rules to refine the process further, but this will work only up to a certain point. After that, models must be trained to handle both common and specialized situations. The later part of this chapter focuses on these models and their use.
The SBD process is language dependent and is often not straightforward. Common approaches to detect sentences include using a set of rules or training a model to detect them. A set of simple rules for detecting a sentence follows. The end of a sentence is detected if:
- The text is terminated by a period, question mark, or exclamation mark
- The period is not preceded by an abbreviation or followed by a digit
Although this works well for most sentences, it will not work for all of them. For example, it is not always easy to determine what an abbreviation is, and sequences such as ellipses may be confused with periods.
Most search engines are not concerned with SBD. They are only interested in a query's tokens and their positions. POS taggers and other NLP tasks that perform extraction of data will frequently process individual sentences. The detection of sentence boundaries will help separate phrases that might appear to span sentences. For example, consider the following sentence:
"The construction process was over. The hill where the house was built was short."
If we were searching for the phrase "over the hill", we would inadvertently pick up it here.
Many of the examples in this chapter will use the following text to demonstrate SBD. This text consists of three simple sentences followed by a more complicated sentence:
private static String paragraph = "When determining the end of sentences "
+ "we need to consider several factors. Sentences may end with "
+ "exclamation marks! Or possibly questions marks? Within "
+ "sentences we may find numbers like 3.14159, abbreviations "
+ "such as found in Mr. Smith, and possibly ellipses either "
+ "within a sentence …, or at the end of a sentence…";