As with virtually every distributed storage approach, in order to support the kind of size envisioned for the Knowledge Base, it is necessary to horizontally partition or "shard" the data. In the case of Stardog, this form of sharding is achieved utilizing industry-standard-defined SPARQL federation via the SERVICE extension as described in the W3C SPARQL 1.1 Federated Query specification (https://www.w3.org/TR/sparql11-federated-query/).

The recommended approach in Stardog is to "shard" the data into separate databases that form logical containers of similar types of data, such as Campaigns, Malware, Identity/Organizations, and Provenance.

An example of SPARQL federation is shown here:

PREFIX foaf:   <http://xmlns.com/foaf/0.1/>
SELECT ?s
{
  ?s a foaf:Person .
  SERVICE <http://example.org/{db-name}/query> {?s foaf:knows ?o }
}

Since a failure to communicate with the federation endpoint would normally result in a termination of the query, it's possible to indicate that the federated query is optional. Additionally, it is possible for a query to involve multiple federation points. The following example illustrates a query that utilizes two federation points and makes getting results from one optional:

PREFIX foaf:   <http://xmlns.com/foaf/0.1/>
SELECT ?person ?interest ?known
WHERE
{
  SERVICE <http://example.org/people/query> {
    ?person foaf:name ?name . 
    OPTIONAL {
      ?person foaf:interest ?interest 
      SERVICE <http://example.org/people2/query> {
        ?person foaf:knows ?known . } }
  }   
}

Stardog optimizes communications between databases hosted on the same nodes/cluster as well as between Stardog clusters. Federations with a remote RDF SPARQL-compliant data source are performed utilizing the SPARQL 1.1 Protocol.

Stardog’s builtin full-text search system indexes data stored in Stardog for information-retrieval queries. The indexing strategy creates a "search document" per RDF literal. Each document consists of two fields: literal ID and literal value. Customizing Stardog's each programmatically can be done through the use of a user-defined Lucene Analyzer.