Stemming is the process of reducing inflected or sometimes derived words to their stem, base, or root form, for example, a stemming algorithm might reduce Riding and Rides, to just Ride. Stemming is done to improve search result recall, but at the expense of some precision. If you are processing general text, you will improve your search results with stemming. However, if you have text that is mostly proper nouns, such as an artist's name in MusicBrainz, then anything more than light stemming will hurt the results. If you want to improve the precision of search results but retain the recall benefits, you should consider indexing the data in two fields, one stemmed and the other not stemmed. The DisMax query parser, described in Chapter 5, Searching, and Chapter 6, Search Relevancy, can then be configured to search the stemmed field and boost by the unstemmed one via its bq or pf options.

Many stemmers will generate stemmed tokens that are not correctly spelled words, such as Bunnies becoming Bunni instead of Bunny or stemming Quote to Quot; you'll see this in Solr's Analysis screen. This is harmless since stemming is applied at both index and search times; however, it does mean that a field that is stemmed like this cannot also be used for query spellcheck, wildcard searches, or search term autocomplete—features described in later chapters. These features directly use the indexed terms.

Here are stemmers suitable for the English language:

<filter class="solr.KeywordMarkerFilterFactory"
  protected="protwords.txt" />
<filter class="solr.StemmerOverrideFilterFactory"
  dictionary="stemdict.txt" />
<filter class="solr.PorterStemFilterFactory" />

The purpose of synonym processing is straightforward. Someone searches using a word that wasn't in the original document but is synonymous with a word that is indexed, so you want that document to match the query. Of course, the synonym need not be strictly those identified by a Thesaurus, and they can be whatever you want, including terminology specific to your application's domain.

Here is a sample analyzer configuration line for synonym processing:

<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>

The synonym reference is to a file in the conf directory. Set ignoreCase to true for the case-insensitive lookup of synonyms.

Before describing the expand option, let's consider an example. The synonyms file is processed line-by-line. Here is a sample line with an explicit mapping that uses the arrow =>:

i-pod, i pod =>ipod

This means that if either i-pod (one token) or i then pod (two tokens) are found in the incoming token stream to this filter, then they are replaced with ipod. There could have been multiple replacement synonyms, each of which might contain multiple tokens. Also notice that commas are what separate each synonym, which is then split by whitespace for multiple tokens. To customize the tokenization to be something more sophisticated than whitespace, there is a tokenizerFactory attribute, but it's rarely used.

Alternatively, you may have lines that look like this:

ipod, i-pod, i pod

These lines don't have => and are interpreted differently according to the expand parameter. If expand is true, the line will be translated to the following explicit mapping:

ipod, i-pod, i pod =>ipod, i-pod, i pod

If expand is false, the aforementioned line will become this explicit mapping, in which the first source synonym is the replacement synonym:

ipod, i-pod, i pod =>ipod

It's okay to have multiple lines that reference the same synonyms. If a source synonym in a new rule is already found to have replacement synonyms from another rule, then those replacements are merged.

There is a simple filter called StopFilterFactory that filters out certain so-called stop words specified in a file in the conf directory, optionally ignoring case. The example usage is as follows:

<filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>

When used, it is present in both index and query analyzer chains.

For indexes with lots of text, common uninteresting words such as "the", "a", and so on, make the index large and slow down phrase queries that use them. A simple solution to this problem is to filter them out of the fields in which they often show up. Fields likely to contain more than a sentence are ideal candidates. Our MusicBrainz schema does not have content like this. The trade-off when omitting stop words from the index is that those words are no longer queryable. This is usually fine, but in some circumstances like searching for To be or not to be, it is obviously a problem.

Solr comes with a decent set of stop words for the English language. You may want to supplement it or use a different list altogether, if you're indexing non-English text. In order to determine which words appear commonly in your index, access the Schema Browser menu option in Solr's admin interface. All of your fields will appear in a drop-down list on the form. In case the list does not appear at once, be patient. For large indexes, there is a considerable delay before the field list appears because Solr is analyzing the data in your index. Now, choose a field that you know contains a lot of text. In the main viewing area, you'll see a variety of statistics about the field, including the top 10 terms appearing most frequently. If you can't see the term info by default, click on the Load Term Info button and select the Autoload checkbox.

Another useful text analysis option to enable searches that sound like a queried word is phonetic translation. A filter is used at both index and query time that phonetically encodes each word into a phoneme-based word. There are many phonetic encoding algorithms to choose from: BeiderMorse, Caverphone, Cologne, DoubleMetaphone, Metaphone, RefinedSoundex, and Soundex. We suggest using DoubleMetaphone for most text, and definitely BeiderMorse for names. However, you might want to experiment in order to make your own choice.

The following code shows how to configure text analysis for phonetic matching using the DoubleMetaphone encoding in the schema.xml file:

<!-- for phonetic (sounds-like) indexing -->
<fieldType name="phonetic" class="solr.TextField"
     positionIncrementGap="100">
   <analyzer>
       <tokenizer class="solr.StandardTokenizerFactory"/>
       <filter class="solr.DoubleMetaphoneFilterFactory"
          inject="false" maxCodeLength="8"/>
   </analyzer>
</fieldType>

The previous example uses the DoubleMetaphoneFilterFactory analysis filter, which has the following two options:

Note that the phonetic encoders internally handle both uppercase and lowercase, so there's no need to add a lowercase filter.

In the MusicBrainz schema that is supplied with the book, a field named a_phonetic is declared to use BeiderMorse because that encoding is best for names. The field has the artist name copied into it through a copyField directive. In Chapter 5, Searching, you will read about the DisMax query parser that can conveniently search across multiple fields with different scoring boosts. It can be configured to search not only the artist name (a_name) field, but also a_phonetic with a low boost so that regular exact matches will come above those that match phonetically.

Here is how BeiderMorse is configured:

<fieldType name="phonetic" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <!-- ... potentially others ... -->
    <filter class="solr.BeiderMorseFilterFactory" ruleType="APPROX"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.BeiderMorseFilterFactory" ruleType="EXACT"/>
  </analyzer>
</fieldType>

Notice the difference in ruleType between the query and index analyzers. In order to use most of the phonetic encoding algorithms, you must use the following filter:

<filter class="solr.PhoneticFilterFactory" encoder="RefinedSoundex" inject="false"/>

The encoder attribute must be one of those algorithms listed in the first paragraph of this section, with the exception of DoubleMetaphone and BeiderMorse, which have dedicated filter factories.

Usually, the text indexing technology is employed to search entire words. Occasionally, however, there arises a need for a query to match an arbitrary substring of an indexed word or across them. Solr supports wildcards on queries (for example, mus*ainz), but there is some consideration needed in the way data is indexed.

It's useful to first get a sense of how Lucene handles a wildcard query at the index level. Lucene internally scans the sorted terms list on disk starting with the nonwildcard prefix (mus in the previous example). One thing to note about this is that the query takes exponentially longer for each fewer prefix character. In fact, Solr configures Lucene to not accept a leading wildcard to ameliorate the problem. Another thing to note is that stemming, phonetic, and other trivial text analysis will interfere with these kinds of searches, for example, if running is stemmed to run, then runni* would not match.

<filter class="solr.ReversedWildcardFilterFactory" />

<fieldType name="nGram" class="solr.TextField"
       positionIncrementGap="100">
   <analyzer type="index">
       <tokenizer class="solr.StandardTokenizerFactory"/>
       <!-- potentially word delimiter, synonym filter, stop words, NOT stemming -->
       <filter class="solr.LowerCaseFilterFactory"/>
       <filter class="solr.NGramFilterFactory" minGramSize="2"             maxGramSize="15"/>
   </analyzer>
   <analyzer type="query">
       <tokenizer class="solr.StandardTokenizerFactory"/>
       <!-- potentially word delimiter, synonym filter, stop words, NOT stemming -->
       <filter class="solr.LowerCaseFilterFactory"/>
   </analyzer>
</fieldType>

Usually, search results are sorted by relevancy via the score pseudo-field, but it is common to need to support conventional sorting by field values too. And, in addition to sorting search results, there are ramifications to this discussion in doing a range query and when showing facet results in a sorted order.

It just so happens that MusicBrainz already supplies alternative artist and label names for sorting. When different from the original name, these sortable versions move words like "The" from the beginning to the end after a comma. We've marked the sort names as indexed but not stored since we're going to sort on it but not display it—deviating from what MusicBrainz does. Remember that indexed and stored are true by default. Because of the special text analysis restrictions of fields used for sorting, text fields in your schema that need to be sortable will usually be copied to another field and analyzed differently. The copyField directive in the schema facilitates this task. The string type is a type that has no text analysis and so it's perfect for our MusicBrainz case. As we're getting a sort-specific value from MusicBrainz, we don't need to derive something ourselves.

However, note that in the MusicBrainz schema there are no sort-specific release names, so let's add sorting support. One option is to use the string type again. That's fine, but you may want to lowercase the text, remove punctuation, and collapse multiple spaces into one (if the data isn't clean). You can even use PatternReplaceFilterFactory to move words like "The" to the end. It's up to you. For the sake of variety in our example, we'll be taking the latter route; we're using a type title_sort that does these kinds of things.

By the way, Lucene sorts text by the internal Unicode code point. You probably won't notice any problem with the sort order. If you want sorting that is more accurate to the finer rules of various languages (English included), you should try CollationKeyFilterFactory. Since it isn't commonly used and it's already well documented, we'll refer you to the wiki page https://cwiki.apache.org/confluence/display/solr/Language+Analysis#LanguageAnalysis-UnicodeCollation.

Solr includes many more token filters:

There are some other miscellaneous Solr filters we didn't mention for various reasons. For common-grams or shingling, see Chapter 10, Scaling Solr. See the all known implementing classes section at the top of http://lucene.apache.org/core/4_10_4/analyzers-common/org/apache/lucene/analysis/util/TokenFilterFactory.html for a complete list of token filter factories, including documentation.