Let's build an index with sample documents to understand the examples in a better way.

First, create an index with the name profiles:

curl -XPUT 'localhost:9200/profiles'

Then, put the mapping with the document type as candidate:

curl -XPUT 'localhost:9200/profiles/candidate'

{
 "properties": {
   "geo_code": {
     "type": "geo_point",
     "lat_lon": true
   }
 }
}

Please note that in the preceding mapping, we are putting mapping only for the geo data type. The rest of the fields will be indexed dynamically.

Now, you can create a data.json file with the following content in it:

{ "index" : { "_index" : "profiles", "_type" : "candidate", "_id" : 1 }}
{ "name" : "Sam", "geo_code" : "12.9545163,77.3500487", "total_experience":5, "skills":["java","python"] }
{ "index" : { "_index" : "profiles", "_type" : "candidate", "_id" : 2 }}
{ "name" : "Robert", "geo_code" : "28.6619678,77.225706", "total_experience":2, "skills":["java"] }
{ "index" : { "_index" : "profiles", "_type" : "candidate", "_id" : 3 }}
{ "name" : "Lavleen", "geo_code" : "28.6619678,77.225706", "total_experience":4, "skills":["java","Elasticsearch"] }
{ "index" : { "_index" : "profiles", "_type" : "candidate", "_id" : 4 }}
{ "name" : "Bharvi", "geo_code" : "28.6619678,77.225706", "total_experience":3, "skills":["java","lucene"] }
{ "index" : { "_index" : "profiles", "_type" : "candidate", "_id" : 5 }}
{ "name" : "Nips", "geo_code" : "12.9545163,77.3500487", "total_experience":7, "skills":["grails","python"] }
{ "index" : { "_index" : "profiles", "_type" : "candidate", "_id" : 6 }}
{ "name" : "Shikha", "geo_code" : "28.4250666,76.8493508", "total_experience":10, "skills":["c","java"] }

Once the file is created, execute the following command to put the data inside the index we have just created:

curl -XPOST 'localhost:9200' --data-binary @data.json

If you look carefully at the example, the documents contain the data of the candidates who might be looking for jobs. For hiring candidates, a recruiter can have the following criteria:

You can construct a simple bool query in combination with a term query on the skills field along with geo_distance and range filters on the geo_code and total_experience fields respectively. However, does this give a relevant set of results? The answer would be NO.

The problem is that if you are restricting the range of experience and distance, you might even get zero results or no suitable candidates. For example, you can put a range of 0 to 100 kilometers of distance but your perfect candidate might be at a distance of 101 kilometers. At the same time, if you define a wide range, you might get a huge number of non-relevant results.

The other problem is that if you search for candidates who know Java, there is a chance that a person who knows only Java and not any other programming language will be at the top, while a person who knows other languages apart from Java will be at the bottom. This happens because during the ranking of documents with TF/IDF, the lengths of the fields are taken into account. If the length of a field is small, the document is more relevant.

Elasticsearch is not intelligent enough to understand the semantic meaning of your queries, but for these scenarios, it offers you the full power to redefine how scoring and document ranking should be done.

A weight function allows you to apply a simple boost to each document without the boost being normalized: a weight of 2 results in 2 * _score. For example:

GET profiles/candidate/_search
{
  "query": {
    "function_score": {
      "query": {
        "term": {
          "skills": {
            "value": "java"
          }
        }
      },
      "functions": [
        {
          "filter": {
            "term": {
              "skills": "python"
            }
          },
          "weight": 2
        }
      ],
      "boost_mode": "replace"
    }
  }
}

The preceding query will match all the candidates who know Java, but will give a higher score to the candidates who also know Python. Please note that boost_mode is set to replace, which will cause _score to be calculated by a query that is to be overridden by the weight function for our particular filter clause. The query output will contain the candidates on top with a _score of 2 who know both Java and Python.

Java example:

The previous query can be implemented in Java in the following way:

This uses the value of a field in the document to alter the _score:

GET profiles/candidate/_search
{
  "query": {
    "function_score": {
      "query": {
        "term": {
          "skills": {
            "value": "java"
          }
        }
      },
      "functions": [
        {
          "field_value_factor": {
            "field": "total_experience"
          }
        }
      ],
      "boost_mode": "multiply"
    }
  }
}

The preceding query finds all the candidates with Java in their skills, but influences the total score depending on the total experience of the candidate. So, the more experience the candidate has, the higher the ranking they will get. Please note that boost_mode is set to multiply, which will yield the following formula for the final scoring:

_score = _score * doc['total_experience'].value

However, there are two issues with the preceding approach: first is the documents that have the total experience value as 0 and will reset the final score to 0. Second, Lucene _score usually falls between 0 and 10, so a candidate with an experience of more than 10 years will completely swamp the effect of the full text search score.

To get rid of this problem, apart from using the field parameter, the field_value_factor function provides you with the following extra parameters to be used:

Java example:

The preceding query can be implemented in Java in the following way:

script_score is the most powerful function available in Elasticsearch. It uses a custom script to take complete control of the scoring logic. You can write a custom script to implement the logic you need. Scripting allows you to write from a simple to very complex logic. Scripts are cached, too, to allow faster executions of repetitive queries. Let's see an example:

{
  "script_score": {
    "script": "doc['total_experience'].value"
  }
}

Look at the special syntax to access the field values inside the script parameter. This is how the value of the fields is accessed using Groovy scripting language.

To see some of the power of this function, look at the following example:

GET profiles/candidate/_search
{
  "query": {
    "function_score": {
      "query": {
        "term": {
          "skills": {
            "value": "java"
          }
        }
      },
      "functions": [
        {
          "script_score": {
            "params": {
              "skill_array_provided": [
                "java",
                "python"
              ]
            },
            "script": "final_score=0; skill_array = doc['skills'].toArray(); counter=0; while(counter<skill_array.size()){for(skill in skill_array_provided){if(skill_array[counter]==skill){final_score = final_score+doc['total_experience'].value};};counter=counter+1;};return final_score"
          }
        }
      ],
      "boost_mode": "replace"
    }
  }
}

Let's understand the preceding query:

Do not try to work with the analyzed fields while writing the scripts. You might get weird results. This is because, had our skills field contained a value such as "core java", you could not have got the exact matching for it inside the script section. So, the fields with space-separated values need to be set as not_analyzed or the keyword has to be analyzed in advance.

Java example:

The previous query can be implemented in Java in the following way:

As you can see, the script logic is a simple string that is used to instantiate the Script class constructor inside ScriptScoreFunctionBuilder.

We have seen the problems of restricting the range of experience and distance that could result in getting zero results or no suitable candidates. Maybe a recruiter would like to hire a candidate from a different province because of a good candidate profile. So, instead of completely restricting with the range filters, we can incorporate sliding-scale values such as geo_location or dates into _score to prefer documents near a latitude/longitude point or recently published documents.

function_score provide to work with this sliding scale with the help of three decay functions: linear, exp (that is, exponential), and gauss (that is, Gaussian). All three functions take the same parameter, as shown in the following code and are required to control the shape of the curve created for the decay function: origin, scale, decay, and offset.

The point of origin is used to calculate distance. For date fields, the default is the current timestamp. The scale parameter defines the distance from the origin at which the computed score will be equal to the decay parameter.

The origin and scale parameters can be thought of as your min and max that define a bounding box within which the curve will be defined. If we wanted to give more boosts to the documents that have been published in the past 10 days, it would be best to define the origin as the current timestamp and the scale as 10d.

The offset specifies that the decay function will only compute the decay function of documents with a distance greater that the defined offset. The default is 0.

Finally, the decay option alters how severely the document is demoted based on its position. The default decay value is 0.5.

GET profiles/candidate/_search
{
  "query": {
    "function_score": {
      "query": {
        "match_all": {}
      },
      "functions": [
        {
          "exp": {
            "geo_code": {
              "origin": {
                "lat": 28.66,
                "lon": 77.22
              },
              "scale": "100km"
            }
          }
        }
      ],"boost_mode": "multiply"
    }
  }
}

In the preceding query, we have used the exponential decay function that tells Elasticsearch to start decaying the score calculation after a distance of 100 km from the given origin. So, the candidates who are at a distance of greater than 100 km from the given origin will be ranked low, but not discarded. These candidates can still get a higher rank if we combine other function score queries such as weight or field_value_factor with the decay function and combine the result of all the functions together.

Java example:

The preceding query can be implemented in Java in the following way:

In the preceding example, we have used the exp decay function but, the commented lines show examples of how other decay functions can be used.

Last, as always, remember that Elasticsearch lets you use multiple functions in a single function_score query to calculate a score that combines the results of each function.