The only field "statically" defined in this schema is the uri field, which is used to represent the original uri of the resource as a uniqueKey, and the _version_ fields, that we will analyze in a while. Note that in our particular case, every resource will have a specific unique uri, so we can avoid using a numeric id identifier without loosing consistency. For the moment, uri will be a textual value (string), and _version_ should be a numeric one (long) useful for tracking changes (also needed for the real-time get feature).

We have decided to define all the fields as indexed and stored for simplicity; we have explicitly decided that our dynamic fields should be multiValued and fullText, because they will receive all the values from every other field.

For the fullText field, we have defined a new specific type named text_general. Every user will perform searches using a combination of words, and our queries on the fullText field should be able to capture results using a single word or combination of words, and ignoring case for more flexibility. In short, the terms written by a user in the query will generally not be an exact match of the content in our fields, and we must start taking care of this in our fullText field.

If we want to define a customized fieldType, we should define a couple of analyzers in it: one for the indexing phase and the other for the querying phase. However, if we want them to act in the same way, we can simply define a single analyzer, as in our example.

An analyzer can be defined for a type using a specific custom component (such as <analyzer class="my.package.CustomAnalyzer"/>) or by assembling some more fine-grained components into an analyze chain, which is composed using three types of components, generally in the following order:

There are many components that we could use; a nonexhaustive list of components can be consulted when configuring a new field at http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters.

We have defined a _version_ field, which can also take only a single value. This field is used to track changes in our data, adding version information. A version number for a modified document will be unique and greater than an old version. This field will be actually written by Solr itself when <updateLog/> is activated in the solrconfig.xml file. Because we need to obtain the last version values, we also need to have our version field to be stored.

I suggest you start using stored fields for your tests; this way, it's easy to examine them with alternative codecs, such as SimpletextCodec. Even if this will cost you some more space on disk, you can eventually evaluate whether a field needs to be stored during later stages of the incremental development.

Once we have defined the first version of our configuration files, we can index the example Mona Lisa entity we used earlier to think about the fields in our schema. First of all, we will play with the first version of the schema then later we will introduce other field definitions substituting the dynamic ones.

In the following examples, we will use the json format for simplicity. There are some minor changes in the structure between json and XML posts. Note that every cURL command should be written in one line even though I have formatted the json part as multiline for more readability.

Both XML and json are widely used on Internet applications and mashups, so I suggest you become familiar with both of them. XML is, in most cases, used for rigorous syntactic checks and validation over a schema in case of data exchange. Json is not a real metalanguage as XML, but it is used more, and more often for exposing data from simple web services (think about typical geo search or autocomplete widgets that interact with remote services) as well as for lightweight, fast-linked approach on the data.

Using JSON with cURL in these example can give you a good idea on how to interact with these services from your platform/languages. As an exercise, I suggest you play the same query using XML as the result format, because there are minor differences in the structure that you can easily study yourself:

When Solr performs an update, generally the following steps are followed:

As you may expect, this process could produce a lot of fragmentation on the index structure ("holes" in the index structure) derived from the delete, and the delete and add operations over a very big index could take some time. It's then important to optimize indexes when possible, because an optimized index reduces the redundancy of segments on disk so that the queries on it should perform better.

Generally speaking, the time spent for adding a new document to an index is more than the time taken for its retrieval by a query, and this is especially true for very big indexes.

In particular context, when we want to perform a commit only after a certain number of documents have been added in the index, in order to reduce time for the rewriting or optimization process, we could look at the soft autocommit and near-realtime features. But for the moment, we don't need them.

The atomic update is a particular kind of update introduced since Solr 4. The basic idea is to be able to perform the modification of a document without necessarily having to rewrite it. For this to be accomplished, we need the addition of a _version_ field, as we have seen before.

When using atomic updates, we need to define our fields to be stored, because the values are explicitly needed for the updates, but with the default update process we don't need this because the full document is deleted and readded with the new values. The copy-field destination, on the other hand, should not necessarily be defined as stored.

If you need some more information and examples, you can look in the official wiki documentation at http://wiki.apache.org/solr/Atomic_Updates.

The following simple schema, however, can help us visualize the process:

When performing an atomic update, we can also use the special attribute update on a per field basis. For example, in the XML format:

<add overwrite="true">
  <doc>
    <field name="uri">http://en.wikipedia.org/wiki/Mona_Lisa</field>
    <field name="title" update="set">Mona Lisa (modified)</field>
    <field name="revision" update="inc">1</field>
    <field name="museum" update="set">Another Museum</field>
    <field name="_version_" >1</field>
  </doc>
</add>

The corresponding json format has a slightly different structure:

[
  {
    "uri" : "http://en.wikipedia.org/wiki/Mona_Lisa",
    "title" : {"set":"Mona Lisa (modified)"},
    "revision"  : {"inc":1},
    "museum" : {"set":"Another Museum"},
    "_version_" : 1
  }
]

The values for this attribute define the different actions to be performed over the document:

The _version_ value is handled by Solr itself and when passed, it generates a different behavior for managing updates, as we will see in the next section.

Another interesting approach to use when updating a document is optimistic concurrency, which is basically an atomic update in which we provide the _version_ value for a field. For this to work, we first need to retrieve the _version_ value for the document we want to update, then if we pass the retrieved version value to the update process, the document will be updated as it exists. For more info please refer:

http://wiki.apache.org/solr/Atomic_Updates

If we provide a nonexistent value, it will be used with the following semantics:

If we guess the current _version_ value for performing updates on a certain specific document, we only have to make a query for the document (for example, by uri):

>> curl -X GET 'http://localhost:8983/solr/paintings_start/select?q=uri:http://en.wikipedia.org/wiki/Mona_Lisa&fl=_version_&wt=json&indent=true'

Retrieve the _version_, and then construct the data to be posted. In this example, we have also anticipated the fl (fields list) parameter, which we will see in the next chapter.