A <dataSource/> tag specifies, as you might guess, the source of data referenced by an entity. This is the simplest piece of the configuration. The type attribute specifies the type, which defaults to JdbcDataSource. Depending on the type, there are further configuration attributes (not listed here). There could be multiple data sources but not usually. Furthermore, with the exception of JdbcDataSource, each type handles either binary or text but not both. The following is a listing of available data source types. They all have a name ending with DataSource.

If you were looking for a MailDataSource, then there isn't one. The MailEntityProcessor was coded to fetch the e-mail itself instead of decoupling that function to a data source.

Following the data sources is a <document/> element, which contains one or more <entity/> elements referencing an Entity Processor via the processor attribute; the default is SqlEntityProcessor. An entity processor produces documents when it is executed. The data to produce the documents typically comes from a referenced data source. An entity that is an immediate child of <document> is by default a root entity, which means its documents are indexed by Solr. If the rootEntity attribute is explicitly set to false, then the DIH recursively traverses down until it finds one that doesn't have this marking. There can be sub-entities, which execute once for each parent document and which usually reference the parent document to narrow a query. Documents from a sub-entity are merged into its root entity's document, producing multivalued fields when more than one document with the same field is produced by the sub-entity.

The entity processors have some common configuration attributes and some that are unique to each one.

Entity processors all have a name ending with EntityProcessor. The following are a list of them:

Within an <entity/> tag are <field/> elements that declare how the columns in the query map to Solr. A field element must have a column attribute that matches the corresponding named column in the SQL query. Its name attribute is the Solr schema field name that the column is going into. If it is not specified, then it defaults to the column name. When a column in the result can be placed directly into Solr without further processing, there is no need to specify the field declaration, because it is implied.

An attribute of the entity declaration that we didn't mention yet is the transformer. This declares a comma-separated list of transformers that create, modify, and delete fields and even entire documents. The transformers are evaluated in order, which can be significant. Usually, the transformers use attributes specific to them on a given field to trigger that it should take some action, whether it be splitting the field into multiple values or formatting it. The following is a list of transformers:

By the way, DIH transformers are similar to Solr UpdateRequestProcessors described at the end of this chapter. The former operates strictly within the DIH framework, whereas the latter is applicable to any importing mechanism.

The following is the mb-dih-artists-jdbc.xml file with a rather long SQL query:

<dataConfig>
  <dataSource name="jdbc" driver="org.postgresql.Driver" url="jdbc:postgresql://localhost/musicbrainz_db" user="musicbrainz" readOnly="true" autoCommit="false" />
  <document>
    <entity name="artist" dataSource="jdbc" pk="id" query="
      select
        a.id as id,
        a.name as a_name, a.sortname as a_name_sort,
        a.begindate as a_begin_date, a.enddate as a_end_date,
        a.type as a_type,
        array_to_string(array(select aa.name from artistalias aa where aa.ref = a.id ),
          '|') as a_alias,
        array_to_string(
          array(select am.name from v_artist_members am where am.band = a.id order by am.id), '|') as a_member_name,
        array_to_string(array(select am.id from v_artist_members am where am.band = a.id order by am.id),
          '|') as a_member_id,
        (select re.releasedate from release re inner join album r on re.album = r.id where r.artist = a.id order by releasedate desc limit 1)
          as a_release_date_latest
      from artist a
          "transformer="RegexTransformer,DateFormatTransformer, TemplateTransformer">
      <field column = "id" template="Artist:${artist.id}" />
      <field column = "type" template="Artist" />
      <field column = "a_begin_date"
        dateTimeFormat="yyyy-MM-dd" />
      <field column = "a_end_date"
        dateTimeFormat="yyyy-MM-dd" />
      <field column = "a_alias" splitBy="|" />
      <field column = "a_member_name" splitBy="|"/>
      <field column = "a_member_id" splitBy="|" />
    </entity>
  </document>
</dataConfig>

If the type attribute on dataSource is not specified (it isn't here), then it defaults to JdbcDataSource. Those familiar with JDBC should find the attributes in this example familiar, and there are also others available. For a reference to all of them, see the Solr Reference Guide.

The main piece of an <entity/> used with a database is the query attribute, which is the SQL query to be evaluated. You'll notice that this query involves some subqueries, which are made into arrays and then transformed into strings joined by spaces. The particular functions used to do these sorts of things are generally database specific. This is done to shoehorn multivalued data into a single row in the results. It may create a more complicated query, but it does mean that the database does all of the heavy lifting so that all of the data Solr needs for an artist is in one row.

In this example, we're going to import an XML file from the disk and use XSLT to do most of the work instead of DIH transformers.

curl 'http://localhost:8983/solr/mbartists/update/xslt?tr=artists.xsl&commit=true' -H 'Content-type:text/xml' --data-binary @downloads/artists_veryshort.xml

<dataConfig>
 <dataSource name="artists" type="FileDataSource" encoding="UTF-8"/>
 <document name="artists">
   <entity name="artist" dataSource="artists" url="downloads/artists_veryshort.xml" processor="XPathEntityProcessor" xsl="cores/mbtype/conf/xslt/artists.xsl"
     useSolrAddSchema="true">
   </entity>
 </document>
</dataConfig>

This dataSource of type FileDataSource is for text files. The entity URL is relative to the baseUrl on the data source; since it's not specified then, it defaults to the current working directory of the server. To see the referenced XSLT file, download the code supplement for the book.

An interesting thing about this example is not just the use of XSLT, but the use of useSolrAddSchema, which signifies that the resulting XML structure follows Solr's XML <add><doc><field name=… structure. Our input file is an HTML table and the XSLT file transforms it. These two options are best used together.

In this example, we have a configuration that crawls all PDF files in a directory and then extracts text and metadata from them:

<dataConfig>
  <dataSource type="BinFileDataSource" />
  <document>
    <entity name="f" dataSource="null" rootEntity="false" processor="FileListEntityProcessor" baseDir="/my/file/path" fileName=".*pdf"
      recursive="true">
      <entity name="tika-test" processor="TikaEntityProcessor" url="${f.fileAbsolutePath}" format="text">
        <field column="Author" name="author" meta="true"/>
        <field column="title" name="title" meta="true"/>
        <field column="text" name="text"/>
      </entity>
    </entity>
  </document>
</dataConfig>

The FileListEntityProcessor is the piece that does the file crawling. It doesn't actually use a data source but it's required. Because this entity is not a root entity, thanks to rootEntity="false", it's the sub-entity within it that is a root entity, which corresponds to a Solr document. The entity is named f and the sub-entity tika-test refers to the path provided by f via f.fileAbsolutePath in its url. This example uses the variable substitution syntax ${…}.

The TikaEntityProcessor part is relatively straightforward. Tika makes a variety of metadata available about documents; this example just used two.

The DIH supports what it calls a delta import, which is a mode of operation in which only data that has changed since the last import is retrieved. A delta import is only supported by the SqlEntityProcessor and it assumes that your data is time-stamped. The official DIH approach to this is prominently documented on the wiki. It uses a deltaImportQuery and deltaQuery pair of attributes on the entity, and a delta-import command. That approach is verbose, hard to maintain, and slow compared to a novel alternative documented at http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport.

Essentially, what you can do is introduce a timestamp check in your SQL's WHERE clause using variable substitution, along with another check if the clean parameter was given to the DIH in order to control whether or not a delta or full import should happen. Here is a concise <entity/> definition on a fictitious schema and dataset showing the relevant WHERE clause:

<entity name="item" pk="ID"
  query="SELECT * FROM item
    WHERE '${dataimporter.request.clean}' != 'false'
      OR last_modified > '${dataimporter.last_index_time}'">

Notice the ${…} variable substitution syntax. To issue a full import, use the full-import command with clean enabled: /dataimport?command=full-import&clean=true. And for a delta import, we still use the full-import command, but we set clean to false: /dataimport?command=full-import&clean=false&optimize=false. We also disabled the index optimization since it's not likely that this is desired for a delta import.