There are other rules that can be used to perform SBD. LingPipe's HeuristicSentenceModel class uses a series of token rules to perform SBD. We will present them here, as they provide insight into what rules can be useful.
This class uses three sets of tokens and two flags to assist in the process:
- Possible Stops: This is a set of tokens that can be the last token of a sentence
- Impossible Penultimates: These tokens cannot be the second to last token in a sentence
- Impossible Starts: This is a set of tokens that cannot be used to start a sentence
- Balance Parentheses: This flag indicates that a sentence should not be terminated until all matching parentheses are matched in that sentence
- Force Final Boundary: This specifies that the final token in an input stream should be treated as a sentence terminator even if it is not a possible stop
Balance parentheses include () and []. However, this rule will fail if the text is malformed. The default token sets are listed in the following table:
|
Possible Stops |
Impossible Penultimates |
Impossible Starts |
|---|---|---|
|
. |
any single letter |
closed parentheses |
|
.. |
personal and professional titles, ranks, and so on |
, |
|
! |
commas, colons, and quotes |
; |
|
? |
common abbreviations |
: |
|
" |
directions |
- |
|
'' |
corporate designators |
-- |
|
). |
time, months, and so on |
--- |
|
U.S. political parties |
% | |
|
U.S. states (not ME or IN) |
" | |
|
shipping terms | ||
|
address abbreviations |
Although LingPipe's HeuristicSentenceModel class uses these rules, there is no reason they cannot be used in other implementations of SBD tools.
Heuristic approaches for SBD might not always be as accurate as other techniques. However, they may work in a particular domain and often have the advantages of being faster and using less memory.