The following librados application is written in C and shows how to use the watch or notify functionality in RADOS. Ceph enables a client to create a watcher on an object and receive notifications from a completely separate client connected to the same cluster.

The watcher functionality is implemented via callback functions. When you call the librados function to create the watcher, two of the arguments are for callback functions, one is for what to do when a notification is received and another is for what to do if the watcher loses contact or encounters an error with the object. These callback functions then contain the code you want to run when a notification or error occurs.

This simple form of messaging is commonly used to instruct a client that has a RBD in use that a snapshot is wished to be taken. The client who wishes to take a snapshot sends a notification to all clients that may be watching the RBD object so that it can flush its cache and possibly make sure the filesystem is in a consistent state.

The following example creates a watcher on an object named my_object and then waits. When it receives a notification, it will display the payload and then send a received message back to the notifier.

1. Create a new file with a .c extension and place the following into it:

       #include <stdio.h> 
       #include <stdlib.h> 
       #include <string.h> 
       #include <syslog.h> 

       #include <rados/librados.h> 
       #include <rados/rados_types.h> 

       uint64_t cookie; 
       rados_ioctx_t io; 
       rados_t cluster; 
       char cluster_name[] = "ceph"; 
       char user_name[] = "client.admin"; 
       char object[] = "my_object"; 
       char pool[] = "rbd"; 

       /* Watcher callback function - called when watcher receives a
       notification */ 
       void watch_notify2_cb(void *arg, uint64_t notify_id, uint64_t
       cookie, uint64_t notifier_gid, void *data, size_t data_len) 
       { 
         const char *notify_oid = 0; 
         char *temp = (char*)data+4; 
         int ret; 
         printf("Message from Notifier: %s
",temp); 
         rados_notify_ack(io, object, notify_id, cookie, "Received", 8); 
       } 

       /* Watcher error callback function - called if watcher encounters
       an error */ 
       void watch_notify2_errcb(void *arg, uint64_t cookie, int err) 
       { 
         printf("Removing Watcher on object %s
",object); 
         err = rados_unwatch2(io,cookie); 
         printf("Creating Watcher on object %s
",object); 
         err = rados_watch2(io,object,&cookie,watch_notify2_cb,
         watch_notify2_errcb,NULL); 
         if (err < 0) { 
           fprintf(stderr, "Cannot create watcher on %s/%s: %s
", object,
           pool, strerror(-err)); 
           rados_ioctx_destroy(io); 
           rados_shutdown(cluster); 
           exit(1); 
         } 
       } 

       int main (int argc, char **argv) 
       { 
         int err; 
         uint64_t flags; 

         /* Create Rados object */ 
         err = rados_create2(&cluster, cluster_name, user_name, flags); 
         if (err < 0) { 
           fprintf(stderr, "Couldn't create the cluster object!: %s
",
           strerror(-err)); 
           exit(EXIT_FAILURE); 
         } else { 
           printf("Created the rados object.
"); 
         } 


         /* Read a Ceph configuration file to configure the cluster
         handle. */ 
         err = rados_conf_read_file(cluster, "/etc/ceph/ceph.conf"); 
         if (err < 0) { 
           fprintf(stderr, "Cannot read config file: %s
",
           strerror(-err)); 
           exit(EXIT_FAILURE); 
         } else { 
           printf("Read the config file.
"); 
         } 
         /* Connect to the cluster */ 
         err = rados_connect(cluster); 
         if (err < 0) { 
           fprintf(stderr, "Cannot connect to cluster: %s
",
           strerror(-err)); 
           exit(EXIT_FAILURE); 
         } else { 
           printf("
 Connected to the cluster.
"); 
         } 

         /* Create connection to the Rados pool */ 
         err = rados_ioctx_create(cluster, pool, &io); 
         if (err < 0) { 
           fprintf(stderr, "Cannot open rados pool %s: %s
", pool,
           strerror(-err)); 
           rados_shutdown(cluster); 
           exit(1); 
         } 

         /* Create the Rados Watcher */ 
         printf("Creating Watcher on object %s/%s
",pool,object); 
         err = rados_watch2(io,object,&cookie,watch_notify2_cb, 
         watch_notify2_errcb,NULL); 
         if (err < 0) { 
           fprintf(stderr, "Cannot create watcher on object %s/%s: %s
",
           pool, object, strerror(-err)); 
           rados_ioctx_destroy(io); 
           rados_shutdown(cluster); 
           exit(1); 
         } 

         /* Loop whilst waiting for notifier */ 
         while(1){ 
           sleep(1); 
         } 
         /* Clean up */ 
         rados_ioctx_destroy(io); 
         rados_shutdown(cluster); 
       } 

2. Compile the watcher example code:

       $ gcc watcher.c -o watcher -lrados

3. Run the watcher example application:

4. The watcher is now waiting for a notification. In another Terminal window, using rados, send a notification to the my_object object which is being watched:

5. You can see that the notification was sent and an acknowledgement notification has been received back. If we look at the first Terminal window again, we can see the message from the notifier: