The scoring factors that have just been described relate to Lucene's default scoring model. It's known as the Vector Space Model, also referred to as simply TF-IDF due to its most prominent components. This venerable model is well known in the information retrieval community, and it has been the only model Lucene supported since the beginning. Lucene/Solr 4 supports four more models, including a well-known one called BM25. BM25 has been the subject of many research papers including those from recognized search experts at Google and Microsoft. It's often pitted against the Vector Space Model portrayed as an improvement, provided its parameters are tuned appropriately.

In Lucene, the relevance model is implemented by a Similarity subclass, and Solr provides SimilarityFactory for each one. Naturally, they have their own unique tuning parameters. In order to use BM25, simply add the following to your schema.xml:

   <similarity class="solr.BM25SimilarityFactory">
     <float name="k1">1.2</float>
     <float name="b">0.75</float>
   </similarity>

Don't forget to re-index. The preceding excerpt will have a global effect on relevancy for the schema. It's also possible to choose a different similarity per field. Beware that doing so is problematic if you are using TF-IDF at all; you'll see different scores for TF-IDF depending on whether the similarity is configured at the field level or as the global default, with regard to the query norm and coordination factor.

For more information on configuring scoring models (similarities) in Solr, see the wiki at http://wiki.apache.org/solr/SchemaXml#Similarity and Solr's Javadoc API for the factories. But for real guidance on the specific models, you'll have to start Googling.

At index-time boosting, you have the option to boost a particular document specified at the document level or at a specific field. Chapter 4, Indexing Data, shows the syntax; it's very simple to use for the XML and JSON formats. The document-level boost is the same as boosting each field by that value. This is internally stored as part of the norms number. Norms must not be omitted in the relevant fields in the schema. It's uncommon to perform index-time boosting because it is not as flexible as query time. That said, index-time boosting tends to have a more predictable and controllable influence on the final score, and it's faster.

At query-time boosting, we described in the previous chapter how to explicitly boost a particular clause of a query higher or lower, if needed, using the trailing ^ syntax. We also showed how the DisMax query parser's qf parameter not only lists the fields to search but allows a boost for them as well. There are a few more ways DisMax can boost queries that you'll read about shortly.

An invaluable tool in diagnosing scoring behavior (or why a document isn't in the result or is but shouldn't be) is enabling query debugging with the debugQuery query parameter. There is no better way to describe it than with an example. Consider the fuzzy query on the artists' index:

a_name:Smashing~

We would intuitively expect that documents with fields containing Smashing would get the top scores, but that didn't happen. Execute the preceding query mentioned with debugQuery=on.

In the following code, the fourth document has Smashing as part of its name but the top three don't:

  <doc>
    <float name="score">3.999755</float>
    <str name="a_name">Smashin'</str>
  </doc>
  <doc>
    <float name="score">3.333129</float>
    <str name="a_name">Mashina</str>
  </doc>
  <doc>
    <float name="score">2.551927</float>
    <str name="a_name">Slashing Funkids</str>
  </doc>
  <doc>
    <float name="score">2.5257545</float>
    <str name="a_name">Smashing Atoms</str>
  </doc>

The first and third documents have words that differ from smashing by only one character, the second by two. What's going on here? Let's look at the following debug output, showing just the second and fourth docs for illustrative purposes:

<lst name="explain">
  <str name="Artist:227132">
    3.333129 = (MATCH) sum of:
      3.333129 = (MATCH) weight(a_name:mashina^0.71428573 in 166352) [DefaultSimilarity], result of:
        3.333129 = score(doc=166352,freq=1.0 = termFreq=1.0 ), product of:
          0.2524328 = queryWeight, product of:
            0.71428573 = boost
            13.204025 = idf(docFreq=1, numDocs=399182)
            0.026765013 = queryNorm
          13.204025 = (MATCH) fieldWeight(a_name:mashina in 286945),
          product of:
            1.0 = tf(termFreq(a_name:mashina)=1)
            13.204025 = idf(docFreq=1, numDocs=399182)
            1.0 = fieldNorm(field=a_name, doc=286945)
  </str>
<!--... skip ...-->
  <str name="Artist:93855">
    2.5257545 = (MATCH) sum of:
      2.5257545 = (MATCH) weight(a_name:smashing in 9796) [DefaultSimilarity], result of:
        2.5257545 = score(doc=9796,freq=1.0 = termFreq=1.0 ), product of:
          0.32888138 = queryWeight, product of:
            12.287735 = idf(docFreq=4, numDocs=399182)
            0.026765013 = queryNorm
          7.6798344 = fieldWeight in 9796, product of:
            1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
            12.287735 = idf(docFreq=4, maxDocs=399182)
            0.625 = fieldNorm(doc=9796)
  </str>

What we see here is the mathematical breakdown of the various components of the score. We see that mashina (the term actually in the index) was given a query-time boost of 0.75 (under-boosted), whereas smashing wasn't. We expected this because fuzzy matching gives higher weights to stronger matches, and it did. However, other factors pulled the final score in the other direction. Notice that the fieldNorm for Mashina is 1.0, whereas Smashing Atoms has a fieldNorm of 0.625. This is because the document we wanted to score higher has a field with two indexed terms versus just the one that Mashina has. Another factor is that the IDF for mashina, 13.2, is higher than for smashing, 12.3. Upper/lower case plays no role. So, arguably, Mashina is a closer match than Smashing Atoms to the fuzzy query Smashing~.

How might we fix this? Well, it's not broken, and the number four spot in the search results isn't bad. So this result is arguably in no need of fixing. This is also a fuzzy query that is fairly unusual and arguably isn't a circumstance to optimize for. For the fuzzy query case seen here, you could use DisMax's bq parameter (to be described very soon) and give it a non-fuzzy version of the user's query. That will have the effect of boosting an exact match stronger. Another idea is to enable omitNorms on a_name in the schema; however, that might reduce scoring effectiveness for other queries.