One of the wonders of working with OpenCV and computer vision is that the physical interface can become optional. There is no requirement that the user touch anything. This dramatically frees you as the designer, allowing you to literally create an interface out of air. With the recent emphasis on lighter, more transparent, and more minimal interfaces using computer vision to capture user input seems thrilling. Before you get too excited by the concept, though, you need to consider the ramifications of that lack of interface. Interfaces are what they are because users require information about what information they can send, what might be done with the information that they send, and how that system to which they are sending the information will use the information.

Your messages to the user about what they are doing and how the system receives information about what they are doing becomes absolutely vital when making computer vision interfaces. You’ll want to avoid the vague “you do something, it does something back” mode of interaction. While this can be fun to experiment with and is an excellent way to learn about computer vision and test certain ideas, it doesn’t make for a particularly rich experience. One of the great realizations that started the flurry of touchscreen interfaces was the precise reactivity of the prototypes of Jeff Han’s Media Wall. These were touchscreens that reacted to user movements in real time and tracked fingertips and gestures with great precision, making the interaction both precise and natural. Once the touchscreen started to become an interactive surface that was not only dynamic but was precise as well, it became possible to develop new and different kinds of applications with them. Touchscreens had been around for a long time before 2004, but they were ATM screens or kiosks where the user touched a button much the same way as if they clicked a mouse. When working with purely gestural interfaces, it becomes even more important to ensure that the modes of input and the feedback to the user are precise, timely, and intelligible. How you go about doing that is entirely up to you.

Creating a gesture library, a set of specific gestures that your application will look for and react to, is important if you’re going to make an application that will use gestures, whether those gestures will be input into a touchscreen, will be in the air, or will be the movement of a body. Defining ahead of time what gestures your application will be looking for will make your decisions about how to structure your application and how to test it a great deal easier for you. With all the recent developments in touchscreens, there is an informal but growing library of recognized gestures: two fingers slide, pinch, tap to select, and so on. Dan Saffer wrote an excellent book called Designing Gestural Interfaces (O’Reilly) that details many of these gestures and what meaningful uses for them might be.

You should think about how to signal what a device is interpreting a gesture to be by providing either some textural feedback or some visual feedback. Many touchscreen devices draw a small white dot around the user’s finger so that the user can see where the application thinks their finger is located. A few other suggestions to think about include making an easy help gesture that users can call and making sure that users are given prompts. If you’re using gestures or specific actions, make sure you signal them clearly.

There are a wide range of projects that use motion tracking and computer vision techniques. The projects range from LaserTag, which tracks a laser pointer on a surface several meters away; to medical imaging, which helps detect cancerous tumors; to military applications; to projects like the NOR_/D Touchscreen that will be explored later in this chapter. This notion of the computer-mediated environment has percolated throughout the design, architectural, and artistic communities. Public artists like Greyworld make frequent use of computer vision. Pieces like The Monument to the Unknown Artist use the gestures of passersby to pose a robotic sculpture, relying on computer vision to determine the pose of the onlooker. Performance pieces like Messa di Voce by Golan Levin and Zachary Lieberman use computer vision to track the movements of a performer on a stage and particularly the motion of their heads. Computer vision is widely used in dance, in musical performance, and in video pieces in dance clubs to widen the physical space in a sense or to make a performer or a participant metaphorically “exist” in more than one place. Advertising or product demonstrations use computer vision to grab a viewer’s attention with graphics that mirror their movements. These computer vision techniques are becoming popular for video artists in dance clubs, as attention getters in shopping malls, in building lobbies, and on the street to attract attention for an advertisement, as well as being tools for artists.

Some of the most remarkable achievements in computer vision have been much more oriented toward engineering. In 2005 a race called the Grand Challenge was held where entrants equipped a car with a computer vision system to navigate desert terrain for several hours. Gary Bradski was a member of the winning team and used OpenCV in the vision system. Almost all robots and robotics applications have some element of computer vision to help them navigate terrain or analyze objects. Several companies are working with computer vision and gesture recognition to create gaming systems that operate without touch and without the need for a controller.

ofxOpenCV is an add-on, but it’s included with the “fat distribution,” the full distribution containing all of the add-ons, so you won’t need to download it separately. Take a look in the add-ons directory of your oF folder, and see whether the ofxOpenCV folder is there. If so, you don’t need to do anything. If you don’t find it, you might need to download it from http://addons.openframeworks.cc or grab a different version of the oF download. As with other add-ons for oF, you’ll need to include the main file for the add-on in an include statement:

include "ofxOpenCv.h"

Once you’ve done that, you should be ready to go. You may want to run the sample application that comes with the ofxOpenCv add-on.

One of the most important aspects of this add-on is its contour finding capability. This is handled through the ofxContourFinder class that performs the blob finding and tracking task. In the .h file of your application, define an instance of the ofxCvContourFinder class:

ofxCvContourFinder contourFinder;

You’ll see more about how to use this method in the following example, but for the moment, since this is one of the two key methods of ofxCvContourFinder, you should understand how the findContours() method works. Here is the signature followed by a description of the parameters:

virtual int  findContours( ofxCvGrayscaleImage& input, int minArea,
    int maxArea, int nConsidered, bool bFindHoles, bool bUseApproximation = true);

Once you’ve detected the blobs in an image, there are two useful properties of ofxContourFinder that you’ll use to determine what has been found in the contour finding:

Commonly after contour detection, you’ll want to use the nBlobs variable to loop through the blobs contained in the blobs vector:

for (int i = 0; i < contourFinder.nBlobs; i++){
    contourFinder.blobs[i].draw(360,540);
}

This merits a look at ofxCvBlobs. As mentioned earlier, a blob defines an area identified by the OpenCV plug-in as being a contiguous object in the scene. Here are some of the properties of this class:

When contourFinder detects a contiguous contour, it stores the information about that region as a blob. Later in this chapter you’ll use those blobs to track objects over time. First, though, look at the code for reading the contours of an image, detecting blobs, and drawing them to the screen:

#ifndef _CTRS_APP
#define _CTRS_APP

#include "ofMain.h"
#include "ofxOpenCv.h"

class contoursApp : public ofBaseApp{

public:

    void setup();
    void update();
    void draw();

    void keyPressed  (int key);

    ofVideoGrabber vidGrabber;
    ofxCvColorImage colorImg;

The three ofxCvGrayscaleImage images are used together to determine what has changed between the current frame and the background. The grayImage is used to convert the color image from the video into grayscale, and the grayBg image stores an image of the background. These two are then compared using the absDiff() method of the ofxCvGrayscaleImage object, which creates a new image based on the differences between the current image and the image of the background and stores it in the grayDiff image:

    ofxCvGrayscaleImage grayImage;
    ofxCvGrayscaleImage grayBg;
    ofxCvGrayscaleImage grayDiff;

ofxCvContourFinder is the object that will detect the contours in the image, finding borders of objects or shifts in color and light:

    ofxCvContourFinder contourFinder;
    int threshold;
    bool bLearnBackground;
};
#endif

Here’s the .cpp file:

#include "contoursApp.h"

void contoursApp::setup(){

        bLearnBackground = false;

    vidGrabber.setVerbose(true);
    vidGrabber.initGrabber(320,240);

Now, allocate memory for the array of pixels that the image will use to save the incoming pixel data:

    colorImg.allocate(320,240);
    grayImage.allocate(320,240);
    grayBg.allocate(320,240);
    grayDiff.allocate(320,240);
}

void contoursApp::update(){
    vidGrabber.grabFrame();
    //do we have a new frame?
    if (vidGrabber.isFrameNew()){

Now ofxCvColorImage has all of its pixels set from the newest frame that the video camera has captured. If you don’t have a camera attached to your computer when you run this code, you can always use the ofVideoPlayer class to play back a video to use in contourDetection:

        colorImg.setFromPixels(vidGrabber.getPixels(), 320,240);
        grayImage = colorImg; // convert our color image to a grayscale image
        if (bLearnBackground == true) {
            grayBg = grayImage; // update the background image
            bLearnBackground = false;
        }

Now, find the difference between the background and incoming image to look for changes in the two images:

        grayDiff.absDiff(grayBg, grayImage);

To ensure that the grayDiff image doesn’t contain too much noise, increase the contrast on it. You might want to make this higher or lower depending on the lighting situation in the video image:

        grayDiff.threshold(20);

Here’s the call to the findContours() method. In this case, the smallest blobs that contourFinder is looking for are 10 pixels, and the largest are one tenth the size of the screen. contourFinder is looking for two blobs and for holes in those blobs:

        contourFinder.findContours(grayDiff, 10, (340*240)/4, 2, true);
    }
}

void contoursApp::draw(){
    colorImg.draw(0, 0, 32, 240);
    ofSetColor(0xffffff);
    ofRect(0, 0, 320, 240);
    ofSetColor(0x000000);

Now, loop through all the blobs that contourFinder has detected, and draw them:

    for (int i = 0; i < contourFinder.nBlobs; i++){
        contourFinder.blobs[i].draw(0, 0);
    }
}

If the user hits a key, then set the learnBackground parameter to true, meaning that the background should be captured again so the new images will be compared with the correct background. This can be good to do if the light changes in a room or if the camera moves. Otherwise, you’ll find too many false matches in your image:

void contoursApp::keyPressed  (int key){
    bLearnBackground= true;
}

Now that you’ve seen the basics of detecting blobs, the next step is to track them over a period of time.

Now that you have an understanding of both how to work with ofxOpenCV in oF and what blob tracking is, you can begin to look at tracking blobs over time. Tracking a blob or a shape across a surface is, at its core, a matter of comparing all the blobs found in a scene with the blobs found in the previous frame of the scene. Just as in the earlier ofxOpenCV examples, this can be done using the ofxContourFinder class within the application to detect the contours of the objects. The key difference is that once a blob has been detected, then its data is stored in a vector. On the next frame, when the update method of the application is called, all the blobs are captured again and compared with the previous blobs to see which blobs from the previous frame best correspond to the blobs of the current frame. Figure 14-2 shows a single blob being tracked over several frames.

Figure 14-2. Tracking a hand over several seconds

The 0 in the center of the blob indicates the position of the blob within the vector of discovered blobs. That 0 is drawn at the centroid, or center point of the blob. When looking through tracking code, you’ll often see references to the centroids of the blobs. This example makes the blob tracking easy because the object is in high contrast to its background and there is really only one object of interest in the frame. Generally, the more objects there are to track, the less accurate the blob tracking becomes, and the lower the contrast between objects. The more precise the control that you need over the tracking data, the more careful you’ll need to be when setting up your environment to ensure that the images have the proper contrast.

Here is the general algorithm for blob tracking:

The first three steps are handled by an instance of contourFinder, and the fourth step is handled by a new file, the ofxCVBlobTracker. This file is available in the code downloads for this chapter, so to conserve space, only the .h header file of the blob tracker is shown here:

#ifndef oF_CV_BLOBTRACKER_H
#define oF_CV_BLOBTRACKER_H

#include <map>
#include <vector>
#include "ofMain.h"

#include "ofxCvTrackedBlob.h"
#include "ofxCvConstants_Track.h"

class ofxCvBlobTracker {
    public:
        vector<ofxCvTrackedBlob>  blobs;
        ofxCvBlobTracker();
        void trackBlobs( const vector<ofxCvBlob>& blobs );

Determine the order by which the present blobs came into existence:

        int findOrder( int id );

The getById() method returns a reference (note the & symbol) to the blob in the blobs vector that has the ID passed to the method:

        ofxCvTrackedBlob& getById( int id );

The draw() method draws all the tracked blobs, drawing the contour of the detected blob and drawing the ID number at its center:

        void draw( float x, float y );

The blobs’ history, that is, all the tracked and detected blobs, is stored in the history vector, which is a vector of vectors. Each vector represents a capture of all the blobs from the screen, which are then compared to the newest blobs coming in to determine which ones most likely represent the same objects:

    protected:
        vector<vector<ofxCvTrackedBlob> > history;

You can find the .cpp file of the ofxCvBlobTracker class in the downloads for this chapter.

Now, take a look at ofBaseApp, which uses ofxCvBlobTracker to find and track blobs in a live camera feed and draw them to the screen:

#include "ofMain.h"

#include "ofxCvBlobTracker.h"
#include "ofxCvTrackedBlob.h"
#include "ofxCvConstants_Track.h"

class trackerApp: public ofBaseApp {

    public:
        int cwidth;
        int cheight;

Here is the ofVideoGrabber instance to capture video:

        ofVideoGrabber  vidGrabber;

As in the previous application, there are three images. The first stores bitmap data from the camera, the second is a grayscale image that will be thresholded to sharpen the contrast of the changed pixels, and the third stores the background for comparison against the incoming image:

        ofxCvColorImage  colorImg;
        ofxCvGrayscaleImage  grayImg;
        ofxCvGrayscaleImage  bgImg;

ofxCvBlobTracker doesn’t actually capture the blobs; it simply stores blobs detected by the ofxCvContourFinder instance and compares them for similarities. In the .cpp file of the application, you’ll see how they are used together:

    ofxCvContourFinder  contourFinder;
    ofxCvBlobTracker  blobTracker;

    int threshold;
    bool bLearnBackground;
};

Here is the trackerApp.cpp file:

#include "trackerApp.h"

void trackerApp::setup() {
    ofSetFrameRate( 60 );
    threshold = 60;
    bLearnBackground = true;

Start the videoGrabber, and allocate memory for all the images:

    vidGrabber.initGrabber( 320, 240 );
    colorImg.allocate( 320, 240 );
    grayImg.allocate( 320, 240 );
    bgImg.allocate( 320, 240 );

Since the blobTracker is set up to notify a listening application when detected blobs have been moved, the tracker is set to notify this application. You do this by passing the blobTracker the this pointer so that it will call the blobMoved(), blobOn(), and blobOff() methods of the application:

    blobTracker.setListener( this );
}

In the application’s update() method, a new frame is grabbed from the videoGrabber and converted into a grayscale image:

void trackerApp::update() {
    ofBackground( 100, 100, 100 );
    vidGrabber.grabFrame();
    if( vidGrabber.isFrameNew() ) {
        colorImg = vidGrabber.getPixels();

The = here copies the color pixels of the colorImg variable into the ofxCvGrayscaleImage after converting them to grayscale. This uses something called operator overloading that allows different operators, =, +, or –, for instance, to have different meanings depending on the class. It’s too advanced a topic for this book, but you can learn more about it in one of the more advanced C++ texts listed in Chapter 18:

        grayImg = colorImg;
        if( bLearnBackground ) {
            bgImg = grayImg;
            bLearnBackground = false;
        }
        grayImg.absDiff( bgImg );
        grayImg.blur( 11 );
        grayImg.threshold( threshold );

contourFinder is passed the grayscale image, and then the discovered blobs are passed to ofxCvBlobTracker:

        contourFinder.findContours( grayImg, 50, 20000, 10, false, true);
        blobTracker.trackBlobs( contourFinder.blobs );
    }
}

void trackerApp::draw() {
    ofSetColor( 0xffffff );
    colorImg.draw( 20,200 );
    grayImg.draw( 360,200 );

This time, you draw from blobTracker so that all the blobs will be drawn to the screen rather than all the detected contours as you did in the previous example:

    blobTracker.draw( 20,200 );
  }

class ofxCvBlobListener {
    public:

        virtual void blobOn( int x, int y, int id, int order ) = 0;
        virtual void blobMoved( int x, int y, int id, int order ) = 0;
        virtual void blobOff( int x, int y, int id, int order ) = 0;

};

class Person {

public:
    virtual void speak() = 0; // here's the virtual method

};

class FrenchPerson : public Person {
public:
    void speak() { printf(" Je parle Francais. "); }
};

class ArgentinePerson : public Person {
public:
    void speak() { printf(" Hablo castellano "); }
};

class testApp : public ofBaseApp, public ofxCvBlobListener {

#ifndef _TEST_APP
#define _TEST_APP

#include "ofMain.h"
#include "Ball.h"
#include "ofxVectorMath.h"
#include "ofxCvBlobTracker.h"
#include "ofxCvMain.h"

#define APP_WIDTH 1000
#define APP_HEIGHT 800

#define vid_width 640
#define vid_height 480

class blobForceApp : public ofBaseApp, public ofxCvBlobListener {

public:

    void setup();
    void update();
    void draw();

    Ball pushedBall;
    Ball physicsBall;

    int cwidth;
    int cheight;
    ofVideoGrabber  vidGrabber;
    ofxCvColorImage  colorImg;
    ofxCvGrayscaleImage  grayImg;
    ofxCvGrayscaleImage  bgImg;

    ofxCvContourFinder  contourFinder;
    ofxCvBlobTracker  blobTracker;

    int threshold;
    bool bLearnBackground;

    ofxVec3f blobCenter;
    bool checkCollision();

    void keyPressed( int key );
    void blobOn( int x, int y, int id, int order );
    void blobMoved( int x, int y, int id, int order );
    void blobOff( int x, int y, int id, int order );

    ofxVec3f gravityForce;
    ofxVec3f handForce;

};

#endif

#include "blobForceApp.h"

void blobForceApp::setup() {
    ofSetFrameRate( 60 );
    threshold = 60;

    blobCenter.x = 400;
    blobCenter.y = 500;

    gravityForce = ofxVec3f(0.0, 1.0f, 0.0);
    handForce = ofxVec3f(0.0, 0.0f, 0.0);
    pushedBall.bounce = 0.8;
    pushedBall.maximum_velocity = 2;
    physicsBall.bounce = 0.8;
    physicsBall.maximum_velocity = 2;

    vidGrabber.initGrabber( vid_width, vid_height );
    colorImg.allocate( vid_width, vid_height );
    grayImg.allocate( vid_width, vid_height );
    bgImg.allocate( vid_width, vid_height );
    blobTracker.setListener( this );
}

void blobForceApp::update() {

    bool bNewFrame = false;
    #ifdef _USE_LIVE_VIDEO
        vidGrabber.grabFrame();
        bNewFrame = vidGrabber.isFrameNew();
    #else
        vidPlayer.idleMovie();
        bNewFrame = vidPlayer.isFrameNew();
    #endif
    if( bNewFrame ) {

        #ifdef _USE_LIVE_VIDEO
                colorImg.setFromPixels(vidGrabber.getPixels(), 320,240);
        #else
                colorImg.setFromPixels(vidPlayer.getPixels(), 320,240);
        #endif

        grayImg = colorImg;

        if( bLearnBakground ) {
            bgImg = grayImg;
            bLearnBakground = false;
        }
        grayImg.absDiff( bgImg );
        grayImg.blur( 11 );
        grayImg.threshold( threshold );

        contourFinder.findContours( grayImg, 200, 20000, 1, false, false );
        blobTracker.trackBlobs( contourFinder.blobs );
    }

    pushedBall.addForce(gravityForce);
    pushedBall.addForce(handForce);
    pushedBall.updateBall();
    physicsBall.addForce(gravityForce);
    physicsBall.updateBall();

    float tampedVal;
    float absVal;
    absVal = fabs(handForce.x);
    if(absVal > 0) {

        if(absVal > 2 ) {
            tampedVal = sqrt(absVal);
        } else {
            tampedVal = absVal - 0.1;
        }
        handForce.x = ( handForce.x > 0 ) ? tampedVal : −1 * tampedVal;
    }
    absVal = fabs(sqrt(handForce.y));
    if(absVal > 0) {
        if(absVal > 2 ) {
            tampedVal = sqrt(handForce.y);
        } else {
            tampedVal = absVal - 0.1;
        }
        handForce.y = ( handForce.y > 0 ) ? tampedVal : −1 * tampedVal;
    }
    if(    blobCenter.x == 0 && blobCenter.y == 0 ) { return; }

    if(checkCollision()) {
        physicsBall.collision(&blobTracker.blobs[0]);
    }
}

void blobForceApp::draw() {
    ofSetColor( 0xffffff );
    ofSetLineWidth(1);
    blobTracker.draw( 20,200 );
    pushedBall.drawFrame();
    physicsBall.drawFrame();
    ofSetColor( 0xffffff );
    ofSetLineWidth(3);
    ofLine(500, 400, 500+(handForce.x * 50), 400+(handForce.y * 50));
    ofCircle(500+(handForce.x * 50), 400+(handForce.y * 50), 10);
}

void testApp::keyPressed( int key ) {
        bLearnBackground = true;
        handForce.set(0.0, 0.0, 0.0);
        physicsBall.location.x = 400;
        physicsBall.location.y = 0;
}

void testApp::blobOn( int x, int y, int id, int order ) {}

void testApp::blobMoved( int x, int y, int id, int order) {
    ofxCvTrackedBlob b = blobTracker.getById(id);
    handForce.set(0.5 * b.deltaLoc.x, 0.5 * b.deltaLoc.y, 0.0);
    blobCenter.x = ( APP_WIDTH / vid_width ) * b.centroid.x;
    blobCenter.y = ( APP_HEIGHT / vid_height) * b.centroid.y;
}

void testApp::blobOff( int x, int y, int id, int order ) {}

bool testApp::checkCollision() {
    //difference vector of the 2 circles' positions
    ofxVec3f cDiff = physicsBall.location - blobCenter;
    float c = cDiff.dot( cDiff ) - 10000;
    if( c < 0 ) {
        return true;
    }
    return false;
}

There are many other topics that you can explore in OpenCV that there isn’t space in this chapter to explain and explore properly. These are all used frequently and you can find more information on using them in Gary Bradskis OpenCV, in the OpenCV documentation, or on the oF forums at openframeworks.cc/forums.

Figure 14-5. Creating a convex hull from points and from the image of a body

Usually apparent motion matches actual motion, which can be helpful for detecting motion where you don’t need to track blobs but simply want to detect movement. For instance, many reactive installations merely want to react to the general movement in front of the camera, not track blobs.

Figure 14-6. Detecting movement using apparent motion; on the left, apparent motion is detected correctly, while on the right, you can see how apparent motion might not match actual motion

One way to use gestures in a Processing application is to use ezGestures, a gesture recognition library. It analyzes mouse or Wiimote movements that you make while dragging the mouse or while holding the A button of the Wiimote down. Then it compares the motion against a gesture map that you’ve created for all the gestures in which your application should be interested. It’s an excellent way to experiment and prototype gestural applications. The library understands four directions: UP, DOWN, LEFT, and RIGHT. Using these simple directions, the application can recognize, for instance, the difference between a circle and an L shape or the difference between clockwise or counterclockwise gestures. It can’t, however, tell the difference between a square and a circle or recognize diagonal gestures, and it doesn’t pay attention to how large an area is being covered, among other things.

There are two types of GestureListeners used in the ezGestures library. In the first type, concurrent listeners try to match the gesture to the pattern while the gesture is still being performed. This means that a gesture can be recognized and acted upon before the button is released. The second type starts with Post; for instance, PostVShakeListener tries to match the gesture to the pattern after the gesture is complete, which means after the button is released. This is how the following example works.

One thing to understand about the gesture detection library is that it uses a regular expression to determine what gesture is being drawn. A regular expression is really just a way of making a pattern of letters. While they can be extremely complex and difficult to read, they can be powerful tools. You don’t need to understand too much about how they work to use the gesture detection library. You’ll see a few strange symbols at the beginning of the string that is passed to the PostGestureListener constructor method; for instance, a circle is described as LDRU, or left, down, right, up. To make sure that any and all circles are found in the messages that are sent to the PostGestureListener method, you’d use the caret (^) to indicate that anything can come before the circle gesture and the dollar sign ($), which is the last thing in the string passed to the PostGestureListener. Here is an example of creating a gesture listener that listens for a circle gesture:

PostGestureListener pgl = new PostGestureListener(this, brain, "^(LDRU)$");

For each gesture that you want to listen for, you’ll need to create a new PostGestureListener object that will have the string that represents the gesture passed to it in the constructor for that object:

import net.silentlycrashing.gestures.*;

GestureAnalyzer brain;
PostGestureListener boxListener;


void setup() {
    // initialize the gesture listeners
    size(500, 500);

    brain = new MouseGestureAnalyzer(this);
    brain.setVerbose(true);
    boxListener = new PostGestureListener(this, brain, "^(RDLU)$");
    boxListener.registerOnAction("drawBox", this);
}

void draw() {
    line(pmouseX, pmouseY, mouseX, mouseY);
}

void drawBox() {
    rect(mouseX, mouseY, 200, 200);
}

The previous example used PostGestureListener, but there are four other classes that use concurrent gesture recognition:

In the following very simple example, the listeners will listen concurrently; that is, they won’t wait for the gesture to be finished. Instead, they’ll simply call their action method whenever the gesture being listened for is heard. ConcurrentHShakeListener just listens for a horizontal sweep back and forth, or in ezGestures terms RLRL or LRLR. In this simple application, drawing the square gesture creates a new 3D box using the box() method, and the horizontal shake clears all the boxes:

import net.silentlycrashing.gestures.*;
import net.silentlycrashing.gestures.preset.*;

GestureAnalyzer brain;
ConcurrentGestureListener boxListener;
ConcurrentHShakeListener shakeListener;

int numBoxes = 0;
SimplePoint[] boxes;

void setup() {
    // initialize the gesture listeners
    size(500, 500, P3D);
    lights();

   boxes = new SimplePoint[100];

As before, the MouseGestureAnalyzer is created and passed a reference to the application itself:

    brain = new MouseGestureAnalyzer(this);
    brain.setVerbose(true);

ConcurrentGestureListener is initialized and given the box string as its gesture to listen for:

    boxListener = new ConcurrentGestureListener(this, brain, "(RDLU)$");
    boxListener.registerOnAction("createBox", this);

ConcurrentHShakeListener is created and passed a reference to the MouseGestureAnalyzer instance. Since the gesture that it’s listening for, the horizontal shake, is already programmed, there’s no need to pass it another string to use as its gesture:

    shakeListener = new ConcurrentHShakeListener(this, brain);
    shakeListener.registerOnAction("shakeHeard", this);
}

void draw() {
// line(pmouseX, pmouseY, mouseX, mouseY);
    background(122);
    int i;

Using the camera() method as in Chapter 13, the camera is positioned based on the user’s mouse position. This could, of course, be the relative position of a Wiimote or another kind of accelerometer connected to the computer via an Arduino:

    camera(0.0, mouseY, 0, // eyeX, eyeY, eyeZ
         width/2, height/2, 0.0, // centerX, centerY, centerZ
         0.0, 1.0, 0.0); // upX, upY, upZ

Here, the translate() method sets the location where the box will be drawn, and the s property of the SimplePoint is used to determine the size of the box:

    for( i = 0; i<numBoxes; i++) {
       pushMatrix();
           translate(width/2 - 100, height/2 - 100, 0);
           translate(boxes[i].x, boxes[i].y, boxes[i].z);
           box(boxes[i].s);
       popMatrix();
    }
}

void createBox() {
    SimplePoint p = new SimplePoint();
    p.x = random(100);
    p.y = random(100);
    p.z = random(50);
    p.s = random(40);
    boxes[numBoxes] = p;
    numBoxes++;
}

When the horizontal shake gesture is heard, all the boxes will be cleared from the screen:

void shakeHeard() {
    println(" shaken ");
    numBoxes = 0;
}

This is a simple class to store the data about where the box should be drawn:

class SimplePoint {
    float x;
    float y;
    float s;
    float z;
}

One of the great advantages of the ezGestures library is that it works well with the Wiimote remote control. To help you work with the Wiimote, the ezGestures library has several different listeners that listen for buttons and actions specific to the Wiimote.

ofxSimpleGesture is an add-on for openFrameworks that lets you do simple gesture recognition using either a mouse or a video camera. To start, download the add-on from http://addons.openframeworks.cc. It’s loosely based on an algorithm developed by Didier Brun and was originally written in ActionScript for Flash Player. You can add your own custom shapes to it or use a predefined alphabet. The ofxSimpleGesture plug-in has a createStandardAlphabet() method that creates the base alphabet using modified shapes that can easily be recognized by the controller.

Figure 14-7 shows the shapes of each letter. The M, for instance, is broken into four separate gestures. As the user makes the shapes, the ofxSimpleGesture plug-in stores a record of the gestures and then attempts to determine what shape the user might be trying to make. The drawing that the user makes may be slightly different than the model for that gesture. The ofxSimpleGesture plug-in will take a best guess as to what the intended gesture is.

As you can see from the letter shapes shown on the left, the strokes to create the letters are simplified in a few cases to make it easier for the application to read the user’s gestures. You certainly don’t need to use these preprogrammed alphabetic gestures; you can easily create your own.

Figure 14-7. The ofxSimpleGesture plug-in uses simplified representations of letters to read gestures.

The ofxSimpleGesture requires that the application extend the IGestureListener class, in addition to the ofBaseApp class. This means that the application has a new method, onGesture(), that can be called by the ofxSimpleGesture plug-in when a gesture has been detected.

The declaration for an application using the ofxSimpleGesture might look like this:

#ifndef _TEST_APP
#define _TEST_APP
#include "ofMain.h"
#include "ofxGestures.h"

class gestureApp : public ofBaseApp, public IGestureListener {

    public:

        void setup();
        void update();
        void draw();

        void mouseDragged(int x, int y, int button);
        void mousePressed(int x, int y, int button);
        void mouseReleased();

Here’s the onGesture() method that the ofxGestureRecognizer instance uses to tell the application that it has detected a gesture:

        void onGesture(const string& name);

Here’s the declaration of the ofxGestureRecognizer instance:

        ofxGestureRecognizer ofxg;
        string gesture;

    };

#endif

In the .cpp file of the gestureApp, the ofxGestureRecognizer instance calls the setListener() method to set the callback methods of the application:

#include "gestureApp.h"

void gestureApp::setup(){
    ofBackground(255,255,255);
    ofxg.setListener(this);

The createStandardAlphabet() method creates all the letters in the standard alphabet, as shown previously in Figure 14-7:

    ofxg.createStandardAlphabet();
    gesture = "";
}

void gestureApp::update(){
    // nothing here
}

void gestureApp::draw(){
    ofBackground(255,255,255);
    ofSetColor(0x000000);
    ofDrawBitmapString(gesture, 100, 100);
}

void gestureApp::mouseDragged(int x, int y, int button){
    ofxg.addMousePosition(x, y);
}

Start the capture by calling the startCapture() method of the ofxSimpleGesture instance. This begins logging all the user’s actions to determine the gesture:

void testApp::mousePressed(int x, int y, int button){
    ofxg.startCapture(x, y);
}

The gesture recording stops when the mouse is released:

void gestureApp::mouseReleased(){
    ofxg.endCapture();
}

Next is the callback method for the ofxGestureRecognizer that indicates to the application that the gesture has been found:

void gestureApp::onGesture(const string& name) {
    gesture = "Your gesture was ";
    gesture += name;
}

TouchKit, shown in Figure 14-9, is a modular multitouch development kit with the aim to make multitouch readily available in an open source fashion. TouchKit is comprised of software and a hardware component. The TouchKit team provides the source files and welcomes you to use, study, and appropriate the code and schematics for your work or projects. TouchKit is a plug-and-play solution for simple projects and an easily extendable base for experimental and cutting-edge endeavors.

Figure 14-9. The NOR_/D TouchKit

You can find more information on NOR_/D and its projects at touchkit.nortd.com/. TouchKit will also be explored in greater detail in Chapter 16.

Tuio is a protocol for tabletop tangible user interfaces that helps provide a common way to describe events. It’s used in a few different existing table interfaces like the reacTable, developed in Barcelona, and the tDesk, developed in Germany. It defines information about objects on a table surface, such as finger and hand gestures performed by the user or the movement of controller objects with fiducial markers like those shown in Figure 14-10.

Figure 14-10. Fiducial markers that can be tracked across a table and used as controls by a user

The Tuio protocol is implemented using the OpenSound Control (OSC) that was discussed in Chapter 12, and is usable on any platform supporting this protocol. There is an ofxTuio add-on available for oF that lets a table interact with an oF application. The Tuio library can be used with the Touchlib library for Processing or oF as well. You might want to look into some of the development efforts around the Tuio implementation if you’re interested in developing your own touchscreen devices.

Touchlib is another library that can be used to create multitouch interaction surfaces. It tracks blobs of infrared light and sends your information about these blobs as events. These events are named things like finger down, finger moved, and finger released. It includes a configuration application and a few demos to get you started. It will interface with most types of webcams and video capture devices. It currently works only under Windows, but it is being ported over to other platforms.

reacTIVision is another open source computer vision framework. It can be used for tracking fiducial markers and for multitouch finger tracking in table-based tangible user interfaces (TUIs) and multitouch interactive surfaces. It is being developed by Martin Kaltenbrunner and Ross Bencina at the Music Technology Group at the Universitat Pompeu Fabra in Barcelona, Spain, as part of the reacTable project. reacTable is a novel electronic music instrument based using a tabletop multitouch tangible user interface. Martin Kaltenbrunner has created a library for working with reacTIVision that can be used for creating touchscreen applications using Processing.