Creating a plugin

Native plugins allow several aspects of the Elasticsearch server to be extended, but they require a good knowledge of Java.

In this recipe we will see how to set up a working environment to develop native plugins.

Getting ready

You need an up-and-running Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe in Chapter 2, Downloading and Setup.

A Maven tool, or an IDE that supports Java programming, such as Eclipse or IntelliJ IDEA, is required.

The code to this recipe is available in the chapter17/simple_plugin directory.

How to do it...

Generally, Elasticsearch plugins are developed in Java using the Maven build tool and deployed as a ZIP file.

To create a simple JAR plugin, we will perform the following steps:

  1. To correctly build and serve a plugin, some files must be defined:
    • pom.xml is used to define the build configuration for Maven.
    • es-plugin.properties defines the namespace of the plugin class that must be loaded.
    • <name>plugin.java is the main plugin class, which is loaded at startup and also initializes the action's plugin.
    • plugin.xml assemblies, which defines how to execute the assembly steps with Maven. It is used to build the ZIP file to deliver the plugin.
  2. A standard pom.xml file is used for creating a plugin containing the following code:

    The Maven pom.xml header:

            <?xml version="1.0" encoding="UTF-8"?> 
        <project xmlns="http://maven.apache.org/POM/4.0.0" 
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
                 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0   
        http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
           <name>elasticsearch-simple-plugin</name> 
           <modelVersion>4.0.0</modelVersion> 
           <groupId>com.packtpub</groupId> 
           <artifactId>simple-plugin</artifactId> 
           <version>${elasticsearch.version}</version> 
           <packaging>jar</packaging> 
           <description>A simple plugin for Elasticsearch</description> 
           <inceptionYear>2013</inceptionYear> 
           <licenses>...   </licenses>
  3. The parent pom.xml file is used to derive common properties or settings:
            <parent> 
            <groupId>org.sonatype.oss</groupId> 
            <artifactId>oss-parent</artifactId> 
            <version>7</version> 
        </parent>
  4. Some properties are mainly used to simplify the dependencies:
            <properties> 
            <elasticsearch.version>5.1.1</elasticsearch.version> 
            <maven.compiler.target>1.8</maven.compiler.target> 
            <elasticsearch.assembly.descriptor>${project.basedir}
            /src/main/assemblies/plugin.xml
            </elasticsearch.assembly.descriptor> 
            <elasticsearch.plugin.name>simple-plugin
            </elasticsearch.plugin.name> 
            <elasticsearch.plugin.classname>
            org.elasticsearch.plugin.simple.SimplePlugin
            </elasticsearch.plugin.classname> 
            <elasticsearch.plugin.jvm>true</elasticsearch.plugin.jvm> 
            <tests.rest.load_packaged>false</tests.rest.load_packaged> 
            <skip.unit.tests>true</skip.unit.tests> 
        </properties>
  5. A list of JAR dependencies:
            <dependencies> 
             <dependency> 
                 <groupId>org.elasticsearch</groupId> 
                 <artifactId>elasticsearch</artifactId> 
                 <version>${elasticsearch.version}</version> 
                 <scope>compile</scope> 
             </dependency> 
             <dependency> 
                 <groupId>org.apache.httpcomponents</groupId> 
                 <artifactId>httpclient</artifactId> 
                 <version>4.5.2</version> 
             </dependency> 
             <dependency> 
                 <groupId>org.apache.logging.log4j</groupId> 
                 <artifactId>log4j-api</artifactId> 
                 <version>2.3</version> 
             </dependency> 
             <!- Testing dependencies --> 
             <dependency> 
                 <groupId>org.hamcrest</groupId> 
                 <artifactId>hamcrest-core</artifactId> 
                 <version>1.3</version> 
                 <scope>test</scope> 
             </dependency> 
             <dependency> 
                 <groupId>org.hamcrest</groupId> 
                 <artifactId>hamcrest-library</artifactId> 
                 <version>1.3</version> 
                 <scope>test</scope> 
             </dependency> 
      
             <dependency> 
                 <groupId>junit</groupId> 
                 <artifactId>junit</artifactId> 
                 <version>4.12</version> 
                 <scope>test</scope> 
             </dependency> 
         </dependencies>
  6. A list of Maven plugins required is to build and deploy the artifact:
            <build> 
            <plugins> 
                <plugin><!- for compiling --> 
                    <groupId>org.apache.maven.plugins</groupId> 
                    <artifactId>maven-compiler-plugin</artifactId> 
                    <version>3.1</version> 
                    <configuration> 
                        <source>1.7</source> 
                        <target>1.7</target> 
                    </configuration> 
                </plugin> 
                <plugin><!- optional for executing tests --> 
                    <groupId>org.apache.maven.plugins</groupId> 
                    <artifactId>maven-surefire-plugin</artifactId> 
                    <version>2.12.3</version> 
                    <configuration> 
                        <includes> 
                            <include>**/*Tests.java</include> 
                        </includes> 
                    </configuration> 
                </plugin> 
                <plugin><!- optional for publishing the source --> 
                    <groupId>org.apache.maven.plugins</groupId> 
                    <artifactId>maven-source-plugin</artifactId> 
                    <version>2.3</version> 
                    <executions> 
                        <execution> 
                            <id>attach-sources</id> 
                            <goals> 
                                <goal>jar</goal> 
                            </goals> 
                        </execution> 
                    </executions> 
                </plugin> 
                <plugin>><!- for packaging the plugin --> 
                    <artifactId>maven-assembly-plugin</artifactId> 
                    <version>2.3</version> 
                    <configuration> 
                        <appendAssemblyId>false</appendAssemblyId>                      
                        <outputDirectory>${project.build.directory}
                        /releases/</outputDirectory> 
                        <descriptors>                                                    
                        <descriptor>${basedir}/src/main/assemblies
                         /plugin.xml</descriptor> 
                        </descriptors> 
                    </configuration> 
                    <executions> 
                        <execution> 
                            <phase>package</phase> 
                            <goals><goal>single</goal></goals> 
                        </execution> 
                    </executions> 
                 </plugin> 
              </plugins> 
          </build> 
        </project>
  7. In the JAR, there must be a src/main/resources/plugin-descriptor.properties file, which defines the entry point class that must be loaded during plugin initialization. This file must be embedded in the final JAR; it is usually put in the src/main/resources directory of the Maven project. It's generally rendered with the data taken from the Maven properties. For example:
            classname=${elasticsearch.plugin.classname} 
        version=${project.version}
  8. The src/main/java/org/elasticsearch/plugin/simple/SimplePlugin.java class is an example of the basic (the minimum required) code that needs to be compiled for executing a plugin:
            package org.elasticsearch.plugin.simple; 
 
        import org.elasticsearch.plugins.Plugin; 
 
        public class SimplePlugin extends Plugin { 
 
        }
  9. To complete compiling and deploying the workflow, we need to define a src/main/assemblies/plugin.xml file used in the Maven assembly step. This file defines the resources that must be packaged into the final ZIP archive. Let's take a look:
            <?xml version="1.0"?> 
        <assembly> 
            <id>simple-plugin</id> 
            <formats> 
                <format>zip</format> 
            </formats> 
            <includeBaseDirectory>false</includeBaseDirectory> 
           <dependencySets> 
               <dependencySet> 
                  <outputDirectory>/</outputDirectory> 
                  <useProjectArtifact>true</useProjectArtifact> 
                  <useTransitiveFiltering>true</useTransitiveFiltering> 
                    <excludes> 
                    <exclude>org.elasticsearch:elasticsearch</exclude> 
                </excludes> 
              </dependencySet> 
           </dependencySets> 
         </assembly>

How it works...

Several parts make up the development life cycle of a plugin, such as designing, coding, building, and deploying. To speed up the build and deployment steps, which are common to all plugins, we need to create a Maven pom.xml file.

The preceding pom.xml file is a standard for developing Elasticsearch plugins. This file is composed of:

  • Several section entries used to set up the current Maven project. In detail, we have:
  • The name of the plugin (that is, elasticsearch-simple-plugin):
        <name>elasticsearch-simple-plugin</name>
  • The groupId and artifactId used to define the plugin artifact name:
        <groupId>com.packtpub</groupId> 
        <artifactId>simple-plugin</artifactId>
  • The plugin version:
        <version>${elasticsearch.version}</version>
  • The type of packaging:
        <packaging>jar</packaging>
  • A project description with the starting year:
        <description>A simple plugin for Elasticsearch</description> 
        <inceptionYear>2016</inceptionYear>
  • An optional license section, in which we can define the license for the plugin. For the standard Apache one, the code should look like this:
        <licenses> 
           <license> 
              <name>The Apache Software License, Version 2.0</name> 
              <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url> 
              <distribution>repo</distribution> 
           </license> 
        </licenses>
  • A parent pom, used to inherit common properties. Generally for plugins, it is useful to inherit from the sonatype base pom.
        <parent> 
            <groupId>org.sonatype.oss</groupId> 
            <artifactId>oss-parent</artifactId> 
            <version>7</version> 
        </parent>
  • Global variables set are typically in the properties section, the Elasticsearch version and other library versions are set. The properties are used to modularize the Maven strings (replace the properties variable value in different part of pom.xml). Here we can also define variables that will be resolved during Maven execution.

    Tip

    It is very important that the Elasticsearch JAR version matches the Elasticsearch cluster version to prevent issues arising from changes between releases.

  • A list of dependencies for compiling a plugin, the Elasticsearch JAR and the log4j library are required for the compile phase. If you need to test it, remember to add the test JAR as dependencies.
  • The Maven plugin section contains a list of Maven plugins that execute several build steps. We have:

    The compiler section, which requires a source compilation. The Java version is fixed at 1.8:

                <plugin> 
                <groupId>org.apache.maven.plugins</groupId> 
                <artifactId>maven-compiler-plugin</artifactId> 
                <version>3.5.1</version> 
                <configuration> 
                    <source>${maven.compiler.target}</source> 
                    <target>${maven.compiler.target}</target> 
                </configuration> 
            </plugin>

    The source section, which enables the creation of source packages that are to be released with the binary (useful for debugging):

                <plugin> 
                <groupId>org.apache.maven.plugins</groupId> 
                <artifactId>maven-source-plugin</artifactId> 
                <version>2.3</version> 
                     <executions> 
                     <execution> 
                        <id>attach-sources</id> 
                        <goals> 
                            <goal>jar</goal> 
                        </goals> 
                   </execution> 
                 </executions> 
            </plugin>

    The assembly section, which builds a ZIP taking a configuration plugin.xml file and puts the output in the releases directory:

                <plugin> 
                <artifactId>maven-assembly-plugin</artifactId> 
                <version>2.3</version> 
                <configuration> 
                    <appendAssemblyId>false</appendAssemblyId> 
                    <outputDirectory>${project.build.directory}
                    /releases/</outputDirectory> 
                    <descriptors> 
            <descriptor>${basedir}/src/main/assemblies/
            plugin.xml</descriptor> 
                    </descriptors> 
                </configuration> 
                <executions> 
                    <execution> 
                        <phase>package</phase> 
                        <goals><goal>single</goal></goals> 
                    </execution> 
                </executions> 
            </plugin>

Related to pom.xml, we have the plugin.xml file, which describes how to assemble the final ZIP file. This file is usually contained in the /src/main/assemblies/ directory of the project.

The most important sections of this file are as follows:

  • formats: Here the destination format is defined:
            <formats><format>zip</format></formats>
  • excludes sets in dependencySet: Contains the artifacts to exclude from the package. Generally, we exclude the Elasticsearch JAR as it's already provided in the server install:
            <dependencySet> 
            <outputDirectory>/</outputDirectory> 
            <useProjectArtifact>true</useProjectArtifact> 
            <useTransitiveFiltering>true</useTransitiveFiltering> 
            <excludes> 
                <exclude>org.elasticsearch:elasticsearch</exclude> 
            </excludes> 
        </dependencySet>
  • includes sets in dependencySet: Contains the artifacts to include into the package. They are mainly the required JAR files used to run the plugin:
            <dependencySet> 
            <outputDirectory>/</outputDirectory> 
            <useProjectArtifact>true</useProjectArtifact> 
            <useTransitiveFiltering>true</useTransitiveFiltering> 
            <includes>... truncated ...</includes> 
        </dependencySet>

During plugin packaging, the include and exclude rules are verified and only files that are allowed to be distributed are put in the ZIP.

After having configured Maven, we can start to write the main plugin class.

Every plugin class must be derived from Plugin one and it must be public otherwise it cannot be loaded dynamically from the JAR:

import org.elasticsearch.plugins.Plugin; 
public class SimplePlugin extends Plugin {

After having defined all files required to generate a ZIP release of our plugin it is enough to invoke the maven package command. This command will compile the code and create a zip package in the target/releases directory of your project: the final ZIP file can be deployed as a plugin on your Elasticsearch cluster.

In this recipe, we configured a working environment to build, deploy and test plugins. In the next recipes, we will reuse this environment to develop several plugin types.

There's more...

Compiling and packaging a plugin are not enough to define a good lifecycle for your plugin: a test phase for testing your plugin functionalities needs to be provided.

Testing the plugin functionalities with test cases reduces the number of bugs that can affect the plugin when its released.

It is possible to add a test phase in the Maven build pom.xml.

Firstly, we need to add the package dependencies required for testing Elasticsearch and Lucene. These dependencies must be added for the test:

<dependencies> 
  <dependency> 
    <groupId>org.apache.lucene</groupId> 
    <artifactId>lucene-test-framework</artifactId> 
    <version>${lucene.version}</version> 
    <scope>test</scope> 
  </dependency> 
  <dependency> 
    <groupId>org.elasticsearch.test</groupId> 
    <artifactId>framework</artifactId> 
    <version>${elasticsearch.version}</version> 
    <scope>test</scope> 
  </dependency> 
</dependencies>

The order is very important, so make sure to put the lucene-test-framework at the top of your dependencies; otherwise problems with loading and executing tests may occur.

For the unit and integration test, the Elasticsearch community mainly uses the hamcrest library (https://code.google.com/p/hamcrest/). To use this, you need to add its dependencies in the dependencies section of the pom.xml:

<dependency> 
    <groupId>org.hamcrest</groupId> 
    <artifactId>hamcrest-core</artifactId> 
    <version>1.3</version> 
    <scope>test</scope> 
</dependency> 
 
<dependency> 
    <groupId>org.hamcrest</groupId> 
    <artifactId>hamcrest-library</artifactId> 
    <version>1.3 </version> 
    <scope>test</scope> 
</dependency>

Note

Note that the compiling scope is test, which means that these are dependencies applied only during the test phase.

To complete the test part, we need to add a Maven plugin, which executes the tests:

<plugin> 
    <groupId>org.apache.maven.plugins</groupId> 
    <artifactId>maven-surefire-plugin</artifactId> 
    <version>2.12.3</version> 
    <configuration> 
        <includes><include>**/*Tests.java</include></includes> 
    </configuration> 
</plugin>

The includes section lists all the possible classes that contain tests via the glob expression.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.116.87.250