Managing bulk actions

Executing automatic operations on items via a single call is often the cause of a bottleneck if you need to index or delete thousands/millions of records: the best practice in this case is to execute a bulk action.

We have discussed bulk actions via the REST API in the Speeding up atomic operations (bulk) recipe in Chapter 4, Basic Operations.

Getting ready

You need an up-and-running Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe in Chapter 2, Downloading and Setup.

A Maven tool, or an IDE that natively supports it for Java programming such as Eclipse or IntelliJ IDEA, must be installed.

The code of this recipe is in the chapter_14/nativeclient directory and the referred class is the BulkOperations.

How to do it...

For managing a bulk action, we will perform these steps:

  1. We'll need to import the required classes to execute bulks via the native client:
            import org.elasticsearch.action.bulk.BulkRequestBuilder; 
        import org.elasticsearch.client.Client; 
        import org.elasticsearch.common.xcontent.XContentFactory; 
        import org.elasticsearch.script.Script; 
 
        import java.io.IOException; 
        import java.net.UnknownHostException;
  2. We'll create the client, remove the old index if it exists, and create a new one:
            public class BulkOperations { 
        public static void main(String[] args) throws 
        UnknownHostException { 
        String index = "mytest"; 
        String type = "mytype"; 
        Client client = NativeClient.createTransportClient(); 
        IndicesOperations io = new IndicesOperations(client); 
        if (io.checkIndexExists(index)) 
            io.deleteIndex(index); 
 
        try { 
            client.admin().indices().prepareCreate(index) 
                    .addMapping(type, XContentFactory.jsonBuilder() 
                            .startObject() 
                            .startObject(type) 
                            .startObject("properties") 
                            .startObject("position").field("type", 
                            "integer").field("store", "yes").endObject() 
                            .endObject() 
                            .endObject() 
                            .endObject()) 
                    .execute().actionGet(); 
        } catch (IOException e) { 
            System.out.println("Unable to create mapping"); 
        }
  3. Now we can bulk-index 1000 documents, adding the bulk index actions to the bulker:
            BulkRequestBuilder bulker = client.prepareBulk(); 
        for (Integer i = 1; i <= 1000; i++) { 
            bulker.add(client.prepareIndex(index, type, 
            i.toString()).setSource("position", i.toString())); 
        } 
        System.out.println("Number of actions for index: " +  
        bulker.numberOfActions()); 
        bulker.execute().actionGet();
  4. We can bulk-update the previously created 1000 documents via a script, adding the bulk update action to the bulker:
            bulker = client.prepareBulk(); 
        for (Integer i = 1; i <= 1000; i++) { 
            bulker.add(client.prepareUpdate(index, type, 
            i.toString()).setScript(new Script("ctx._source.position += 
            2"))); 
        } 
        System.out.println("Number of actions for update: " + 
        bulker.numberOfActions()); 
        bulker.execute().actionGet();
  5. We can bulk-delete 1000 documents, adding the bulk delete actions to the bulker:
            bulker = client.prepareBulk(); 
        for (Integer i = 1; i <= 1000; i++) { 
            bulker.add(client.prepareDelete(index, type, 
            i.toString())); 
        } 
        System.out.println("Number of actions for delete: " + 
        bulker.numberOfActions()); 
        bulker.execute().actionGet();
  6. We can now free up the resources used:
            io.deleteIndex(index);  
        //we need to close the client to free resources 
        client.close(); 
          } 
       }
  7. The result will be as follows:
            Number of actions for index: 1000 
        Number of actions for update: 1000 
        Number of actions for delete: 1000

How it works...

Before executing these bulk actions, a client must be available, an index must be created, and document mapping can optionally be created.

We can consider  bulkBuilder as a collector of different actions:

IndexRequest or IndexRequestBuilder 
UpdateRequest or UpdateRequestBuilder 
DeleteRequest or DeleteRequestBuilder 
a bulk formatted array of bytes.

Generally, when used in code, we can consider it as a List in which we add actions of the supported types:

  1. To initialize  bulkBuilder we use the following code:
            BulkRequestBuilder bulker=client.prepareBulk();
  2. In the previous example we added 1,000 index actions (IndexBuilder is similar to the previous recipe one):
            for (Integer i=1; i<=1000; i++){ 
            bulker.add(client.prepareIndex(index, type,  
        i.toString()).setSource("position", i.toString())); 
        }
  3. After adding all the actions, we can print (for example) the number of actions and then execute them:
            System.out.println("Number of action: " + 
        bulker.numberOfActions()); 
        bulker.execute().actionGet();
  4. After executing bulkBuilder, the bulker is empty.
  5. We have populated the bulk with 1,000 update actions:
            bulker = client.prepareBulk(); 
        for (Integer i = 1; i <= 1000; i++) { 
            bulker.add(client.prepareUpdate(index, type, 
        i.toString()).setScript(new Script("ctx._source.position += 
        2"))); 
          }
  6. After adding all the update actions, we can execute them in bulk:
            bulker.execute().actionGet();
  7. Next, the same step is done with the delete action:
            for (Integer i=1; i<=1000; i++){ 
                    bulker.add(client.prepareDelete(index, type,   
                    i.toString())); 
        }
  8. To commit the delete, we need to execute the bulk.

Note

In this example, to simplify it, I have created bulk actions with the same type of actions, but, as described previously, you can put any supported type of action into the same bulk operation.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
13.58.108.102