Creating a distributed configuration service

As soon as your application and especially your application load gets bigger, you might want to share the load between several hosts. So you would setup several Play application nodes, all having the same configuration file, which is easily possible with Play. However, you would have to restart all application nodes, when you change one parameter. This is useful when you change something fundamental, like your database configuration. However there are cases where it is not useful to have a downtime of your application. This is where the principle of a centralised configuration service becomes useful. This recipe shows a simple example of how to implement such a service.

You can find the source code of this example in the examples/chapter7/configuration-service directory.

Getting ready

You should set up memcached and the example application written here should run on two different ports. So, after installing memcached, it might have been started by your distribution already, if you are using Linux. Otherwise you can start it by typing: the following

memcached

As you need more than one instance of a running Play application, you could install it on two nodes. And you need to enable memcached as the caching server. Both can be done in the application.conf:

memcached=enabled
memcached.host=127.0.0.1:11211

%app01.http.port=9001
%app02.http.port=9002

The first two lines enable memcached, and the last two lines make use of the so-called application IDs. You can now start the application with a special ID via this command:

play run --%app01

This instance will bind to port 9001. Now open another terminal, copy the directory of the application to another directory, change to the copied directory, and enter the following:

play run --%app02

Now you will have one memcached and two Play instances of your application running. Do not forget to keep the directories in sync, when making changes.

The example itself will have two features. First, it allows you to set parameters on any host, which are then cached and can be retrieved from any other host. Second, it implements a good heartbeat function, which allows the admin to check quickly which hosts are alive—this would be interesting for monitoring and alerting purposes.

How to do it...

Three routes are needed for this example:

GET     /nodes           Application.nodes
GET     /config/{key}    Application.getConfig
POST    /config/{key}    Application.setConfig

One job, which registers itself on application startup in the cache, is needed:

@OnApplicationStart
public class CacheRegisterNodeJob extends Job {

   public static final key = "CS:nodes";

	public void doJob() {
		Set<String> nodes = Cache.get(key, Set.class);
		if (nodes == null) {
			Logger.info("%s: Creating new node list in cache", this.getClass().getSimpleName());
			nodes = new HashSet();
		}
		nodes.add(Play.id);
		Cache.set(key, nodes);
	}
}

Another job, which features a heartbeat operation, is needed:

@Every("10s")
public class CacheAliveJob extends Job {

	private static final String key = "CS:alive:" + Play.id;
	
	public void doJob() {
		Date now = new Date();
		Logger.info("CacheAliveJob: %s", now);
		Cache.set(key, now.getTime(), "15s");
	}
}

The last part is the controller code:

public class Application extends Controller {

	public static void getConfig(String key) {
		String value = (String) Cache.get(key);
		String message = String.format("{ %s: "%s" }", key, value);
		response.setContentTypeIfNotSet("application/json; charset=utf-8");
		renderText(message);
	}
	
	public static void setConfig(String key) {
		try {
			Cache.set(key, IOUtils.toString(request.body));
		} catch (IOException e) {
			Logger.error(e, "Fishy request");
		}
		throw new Status(204);
	}

    public static void nodes() {
    	Set<String> nodes = Cache.get("CS:nodes", Set.class);
    	Map<String, List> nodeMap = new HashMap();
    	nodeMap.put("alive", new ArrayList<String>());
    	nodeMap.put("dead", new ArrayList<String>());
    	
    	for (String node : nodes) {
    		if (Cache.get("CS:alive:" + node) == null) {
    			nodeMap.get("dead").add(node);
    		} else {
    			nodeMap.get("alive").add(node);
    		}
    	}
    	
    	renderJSON(nodeMap);
    }
}

How it works...

The simple part is—as you can see—using the cache as a node wide configuration service. In order to test this functionality, you should start two instances of this application and set the configuration parameter on the first node, while receiving it on the second:

curl -v -X POST --data "This is my cool configuration content" localhost:9001/config/myParameter
* About to connect() to localhost port 9001 (#0)
*   Trying ::1... connected
* Connected to localhost (::1) port 9001 (#0)
> POST /config/myParameter HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
> Host: localhost:9001
> Accept: */*
> Content-Length: 37
> Content-Type: application/x-www-form-urlencoded
> 
< HTTP/1.1 204 No Content
< Server: Play! Framework;1.1;dev
< Content-Type: text/plain; charset=utf-8
< Set-Cookie: PLAY_FLASH=;Path=/
< Set-Cookie: PLAY_ERRORS=;Path=/
< Set-Cookie: PLAY_SESSION=83af29131d389f83adc89cea9ba65ab49347e58a-%00___ID%3A138cc7e0-5e11-4a52-a725-4109f7495c8f%00;Path=/
< Cache-Control: no-cache
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection #0

As you can see, there is no data sent back to the client in the response body, and therefore a HTTP 204 can be sent back. This means that the operation was successful, but nothing was returned to the client. Now, checking the data can be easily done on the other host:

curl -v localhost:9002/config/myParameter
* About to connect() to localhost port 9002 (#0)
*   Trying ::1... connected
* Connected to localhost (::1) port 9002 (#0)
> GET /config/myParameter HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
> Host: localhost:9002
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: Play! Framework;1.1;dev
< Content-Type: application/json; charset=utf-8
< Set-Cookie: PLAY_FLASH=;Path=/
< Set-Cookie: PLAY_ERRORS=;Path=/
< Set-Cookie: PLAY_SESSION=ad8a75f8a74cea83283eaa6c695aadb1810f15c3-%00___ID%3A256a3510-a000-45cb-a0a0-e4b98a2abf53%00;Path=/
< Cache-Control: no-cache
< Content-Length: 56
< 
* Connection #0 to host localhost left intact
* Closing connection #0
{ myParameter: "This is my cool configuration content" }

By connecting to port 9002 with cURL the second Play instance is queried for the content of the cache variable. A (hand-crafted) JSON reply containing the configuration content is returned.

The second part of this recipe is the heartbeat function. It is pretty cool to play with. When checking the CacheAliveJob, you will see that it runs every 10 seconds, and puts some data in the cache with an expiry time of 15 seconds. This means, if after 15 seconds no new entry is inserted, it will vanish from the cache.

However, in order to be able to check whether a node is active, one must know which nodes were online before. This is what the CacheRegisterNodeJob does. It gets a set with nodes from the cache, adds its own name to this set, and stores it back in the cache.

So there is now one list with nodes which have been added upon start up to this list. Using this list with node names, and then checking whether there are any cache entries which show that these nodes are up, enables us to create some sort of heartbeat service.

If you did not stop any of your two nodes, just check whether they are up with cURL:

curl localhost:9002/nodes

The above should return this:

{"dead":[],"alive":["app02","app01"]}

Now it is time to stop one of the instances. Stop the one running on port 9002, wait a little more than 15 seconds and call cURL again on it:

curl localhost:9001/nodes

This time it should return this:

{"dead":["app02"],"alive":["app01"]}

To show you that the cache works as expected, you can even stop the other instance as well; start the second one again and you should be able to call cURL again (of course, always be aware of the 15 second timeout):

curl localhost:9002/nodes

The above will get you this:

{"dead":["app01"],"alive":["app02"]}

Be aware that if you are in DEV mode, the first reply of the nodes query in this case will return both hosts as dead. This is because the whole application only starts when you issue the request above, which in turn means that the first run of the CacheAliveJob has not been run yet, so you have to wait another ten seconds. This is a problem you will not meet in production.

Now put up the first instance again and you should get the response that both nodes are up. Again pay attention here, if you are in DEV mode and wondering why the node you are querying is still marked as dead.

There's more...

If you have taken a closer look at the jobs you might have spotted the flaw in the design. If you bring up two nodes in parallel, they might get the set of nodes of the cache, add their own node, and store it back. This of course might imply losing one of the two nodes. Unfortunately, the Play cache implementation does not help here in any way.

More on memcached

Memcached is pretty powerful. Play supports multi memcached hosts as cache per default, by setting more hosts in the configuration:

memcached.1.host=127.0.0.1:11211
memcached.2.host=127.0.0.1:11212

In order to understand this configuration a little background know-how on memcached is needed. Every memcached client (in this case the library included with Play) can have its own hashing function on how to distribute data to a cluster of memcache hosts. This means, both servers are completely independent from each other and no data is stored twice. This has two implications. If you use such a multi-node memcached setup in play, you have to make sure that every application node includes a configuration for all memcached hosts. Furthermore, you cannot be sure that any other library you are using for accessing memcached use the same hashing algorithm.

See also

If you are iclined to want to know more about cache, check out the recipe Writing your own cache implementation in Chapter 6, where you can write your own cache implementation.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.118.0.145