In this section, we'll look at some Android-specific tasks: one is rendering an overlay on top of the camera view and the second is reading media files on Android.
The overlay is helpful for general information and debugging, and looks nice too! Think of it like the heads up display on a consumer camera.
The reading media files section is something we don't use in this chapter (we read media files using Python). However, if you decide to write an Android app that processes videos on the device itself, this section should get you started.
Now that we have the camera preview working, we want to render some additional information on top of it. We'll be drawing three things; first, a red circle to indicate whether recording is active, second, the current gyroscope values (angular velocity and estimated theta) just for information, and third, a safety rectangle. When stabilizing, we'll probably be cropping the image a bit. The rectangle will guide your video recording to stay within a relatively safe zone.
Along with this, we'll also be setting it up so that you can create buttons on this overlay. Simple touch events can be used to execute specific functions.
You don't need this section for the application to work, but it's a good idea to know how to render on top of an OpenCV camera view while recording.
Before we start working on the overlay widget, let's define a supporting class, Point3. Create a new class called Point3 with three double attributes:
public class Point3 {
public double x;
public double y;
public double z;
}We start by defining a new class, CameraOverlayWidget.
public class CameraOverlayWidget extends SurfaceView implements GestureDetector.OnGestureListener, SurfaceHolder.Callback {
public static String TAG= "SFOCV::Overlay";
protected Paint paintSafeExtents;
protected Button btn;
protected GestureDetector mGestureDetector;We've subclassed this from SurfaceView to be able to render things on it. It also implements the gesture detector class so that we'll be able to monitor touch events on this widget.
private long sizeWidth = 0, sizeHeight = 0;
// Stuff required to paint the recording sign
protected boolean mRecording = false;
protected Paint paintRecordCircle;
protected Paint paintRecordText;
// Calibrate button
private Paint paintCalibrateText;
private Paint paintCalibrateTextOutline;
private Paint paintTransparentButton;
private RenderThread mPainterThread;
private boolean bStopPainting = false;
private Point3 omega;
private Point3 drift;
private Point3 theta;
public static final double SAFETY_HORIZONTAL = 0.15;
public static final double SAFETY_VERTICAL = 0.15;We define a bunch of variables to be used by the class. Some of them are Paint objects – which are used by the SurfaceView to render things. We've created different paints for the safety rectangle, the red recording circle, and the text.
Next, there are variables that describe the current state of the recorder. These variables answer questions like, is it currently recording? What's the size of the video? What is the latest gyro reading? We'll use these state variables to render the appropriate overlay.
We also define some safety fractions – the safety rectangle will have a margin of 0.15 on each edge.
protected GestureDetector.OnGestureListener mCustomTouchMethods = null;
protected OverlayEventListener mOverlayEventListener = null;And finally, we add a few event listeners – we'll use these to detect touches in specific areas of the overlay (we won't be using these though).
Let's look at the constructor for this class:
public CameraOverlayWidget(Context ctx, AttributeSet attrs) {
super(ctx, attrs);
// Position at the very top and I'm the event handler
setZOrderOnTop(true);
getHolder().addCallback(this);
// Load all the required objects
initializePaints();
// Setup the required handlers/threads
mPainterThread = new RenderThread();
mGestureDetector = new GestureDetector(ctx, this);
}Here, we set up some basics when the object is initialized. We create the paint objects in initializePaints, create a new thread for rendering the overlay and also create a gesture detector.
/**
* Initializes all paint objects.
*/
protected void initializePaints() {
paintSafeExtents = new Paint();
paintSafeExtents.setColor(Color.WHITE);
paintSafeExtents.setStyle(Paint.Style.STROKE);
paintSafeExtents.setStrokeWidth(3);
paintRecordCircle = new Paint();
paintRecordCircle.setColor(Color.RED);
paintRecordCircle.setStyle(Paint.Style.FILL);
paintRecordText = new Paint();
paintRecordText.setColor(Color.WHITE);
paintRecordText.setTextSize(20);
paintCalibrateText = new Paint();
paintCalibrateText.setColor(Color.WHITE);
paintCalibrateText.setTextSize(35);
paintCalibrateText.setStyle(Paint.Style.FILL);
paintCalibrateTextOutline = new Paint();
paintCalibrateTextOutline.setColor(Color.BLACK);
paintCalibrateTextOutline.setStrokeWidth(2);
paintCalibrateTextOutline.setTextSize(35);
paintCalibrateTextOutline.setStyle(Paint.Style.STROKE);
paintTransparentButton = new Paint();
paintTransparentButton.setColor(Color.BLACK);
paintTransparentButton.setAlpha(128);
paintTransparentButton.setStyle(Paint.Style.FILL);
}As you can see, paints describe the physical attributes of the things to draw. For example, paintRecordCircle is red and fills whatever shape we draw. Similarly, the record text shows up white with a text size of 20.
Now let's look at the RenderThread class—the thing that does the actual drawing of the overlay. We start by defining the class itself and defining the run method. The run method is executed when the thread is spawned. On returning from this method, the thread stops.
class RenderThread extends Thread {
private long start = 0;
@Override
public void run() {
super.run();
start = SystemClock.uptimeMillis();
while(!bStopPainting && !isInterrupted()) {
long tick = SystemClock.uptimeMillis();
renderOverlay(tick);
}
}Now let's add the renderOverlay method to RenderThread. We start by getting a lock on the canvas and drawing a transparent color background. This clears anything that already exists on the overlay.
/**
* A renderer for the overlay with no state of its own.
* @returns nothing
*/
public void renderOverlay(long tick) {
Canvas canvas = getHolder().lockCanvas();
long width = canvas.getWidth();
long height = canvas.getHeight();
// Clear the canvas
canvas.drawColor(Color.TRANSPARENT, PorterDuff.Mode.CLEAR);Now, we draw the safety bounds of the camera view. While stabilizing the video, we'll inevitably have to crop certain parts of the image. The safe lines mark this boundary. In our case, we take a certain percentage of the view as safe.
// Draw the bounds
long lSafeW = (long)(width * SAFETY_HORIZONTAL);
long lSafeH = (long)(height * SAFETY_VERTICAL);
canvas.drawRect(lSafeW, lSafeH, width-lSafeW, height-lSafeH, paintSafeExtents);If we're recording, we want to blink the red recording circle and the recording text. We do this by taking the current time and the start time.
if(mRecording) {
// Render this only on alternate 500ms intervals
if(((tick-start) / 500) % 2 == 1) {
canvas.drawCircle(100, 100, 20, paintRecordCircle);
final String s = "Recording";
canvas.drawText(s, 0, s.length(), 130, 110, paintRecordText);
}
}Now we draw a button that says "Record" on it.
canvas.drawRect((float)(1-SAFETY_HORIZONTAL)*sizeWidth, (float)(1-SAFETY_VERTICAL)*sizeHeight, sizeWidth , sizeHeight, paintTransparentButton);
final String strCalibrate = "Calibrate";
canvas.drawText(strCalibrate, 0, strCalibrate.length(), width-200, height-200, paintCalibrateText);
canvas.drawText(strCalibrate, 0, strCalibrate.length(), width-200, height-200, paintCalibrateTextOutline);While recording the video, we will also display some useful information—the current angular velocity and estimated angle. You can verify if the algorithm is working as expected or not.
if(omega!=null) {
final String strO = "O: ";
canvas.drawText(strO, 0, strO.length(), width - 200, 200, paintCalibrateText);
String strX = Math.toDegrees(omega.x) + "";
String strY = Math.toDegrees(omega.y) + "";
String strZ = Math.toDegrees(omega.z) + "";
canvas.drawText(strX, 0, strX.length(), width - 200, 250, paintCalibrateText);
canvas.drawText(strY, 0, strY.length(), width - 200, 300, paintCalibrateText);
canvas.drawText(strZ, 0, strZ.length(), width - 200, 350, paintCalibrateText);
}
if(theta!=null) {
final String strT = "T: ";
canvas.drawText(strT, 0, strT.length(), width - 200, 500, paintCalibrateText);
String strX = Math.toDegrees(theta.x) + "";
String strY = Math.toDegrees(theta.y) + "";
String strZ = Math.toDegrees(theta.z) + "";
canvas.drawText(strX, 0, strX.length(), width - 200, 550, paintCalibrateText);
canvas.drawText(strY, 0, strY.length(), width - 200, 600, paintCalibrateText);
canvas.drawText(strZ, 0, strZ.length(), width - 200, 650, paintCalibrateText);
}And, with this, the render overlay method is complete!
// Flush out the canvas
getHolder().unlockCanvasAndPost(canvas);
}
}This class can be used to spawn a new thread and this thread simply keeps the overlay updated. We've added a special logic for the recording circle so that it makes the red circle blink.
Next, let's look at some of the supporting functions in CameraOverlayWidget.
public void setRecording() {
mRecording = true;
}
public void unsetRecording() {
mRecording = false;
}Two simple set and unset methods enable or disable the red circle.
@Override
public void onSizeChanged(int w,int h,int oldw,int oldh) {
super.onSizeChanged(w, h, oldw, oldh);
sizeWidth = w;
sizeHeight = h;
}If the size of the widget changes (we'll be setting it fullscreen on the preview pane), we should know about it and capture the size in these variables. This will affect the positioning of the various elements and the safety rectangle.
public void setCustomTouchMethods(GestureDetector.SimpleOnGestureListener c){
mCustomTouchMethods = c;
}
public void setOverlayEventListener(OverlayEventListener listener) {
mOverlayEventListener = listener;
}We also have a few set methods that let you change the values to be displayed on the overlay.
public void setOmega(Point3 omega) {
this.omega = omega;
}
public void setDrift(Point3 drift) {
this.drift = drift;
}
public void setTheta(Point3 theta) {
this.theta = theta;
}There are other functions that can be used to modify the overlay being displayed. These functions set the gyroscope values.
Now, let's look at some Android-specific lifecycle events such as pause, resume, and so on.
/**
* This method is called during the activity's onResume. This ensures a wakeup
* re-instantiates the rendering thread.
*/
public void resume() {
bStopPainting = false;
mPainterThread = new RenderThread();
}
/**
* This method is called during the activity's onPause method. This ensures
* going to sleep pauses the rendering.
*/
public void pause() {
bStopPainting = true;
try {
mPainterThread.join();
}
catch(InterruptedException e) {
e.printStackTrace();
}
mPainterThread = null;
}These two methods ensure we're not using processor cycles when the app isn't in the foreground. We simply stop the rendering thread if the app goes to a paused state and resume painting when it's back.
@Override
public void surfaceCreated(SurfaceHolder surfaceHolder) {
getHolder().setFormat(PixelFormat.RGBA_8888);
// We created the thread earlier - but we should start it only when
// the surface is ready to be drawn on.
if(mPainterThread != null && !mPainterThread.isAlive()) {
mPainterThread.start();
}
}
@Override
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
// Required for implementation
}
@Override
public void surfaceDestroyed(SurfaceHolder holder) {
// Required for implementation
}When the surface is created, we set up the pixel format (we want it to be transparent, we make the surface of the type RGBA). Also, we should spawn a new thread to get the overlay rendering going.
With that, we're almost ready with our overlay display. One last thing remains—responding to touch events. Let's do that now:
@Override
public boolean onTouchEvent(MotionEvent motionEvent) {
boolean result = mGestureDetector.onTouchEvent(motionEvent);
return result;
}
@Override
public boolean onDown(MotionEvent motionEvent) {
MotionEvent.PointerCoords coords =new MotionEvent.PointerCoords();
motionEvent.getPointerCoords(0, coords);
// Handle these only if there is an event listener
if(mOverlayEventListener!=null) {
if(coords.x >= (1-SAFETY_HORIZONTAL)*sizeWidth &&coords.x<sizeWidth &&
coords.y >= (1-SAFETY_VERTICAL)*sizeHeight &&coords.y<sizeHeight) {
return mOverlayEventListener.onCalibrate(motionEvent);
}
}
// Didn't match? Try passing a raw event - just in case
if(mCustomTouchMethods!=null)
return mCustomTouchMethods.onDown(motionEvent);
// Nothing worked - let it bubble up
return false;
}
@Override
public void onShowPress(MotionEvent motionEvent) {
if(mCustomTouchMethods!=null)
mCustomTouchMethods.onShowPress(motionEvent);
}
@Override
public boolean onFling(MotionEvent motionEvent,MotionEvent motionEvent2,float v, float v2) {
Log.d(TAG, "onFling");
if(mCustomTouchMethods!=null)
return mCustomTouchMethods.onFling(motionEvent,motionEvent2,v, v2);
return false;
}
@Override
public void onLongPress(MotionEvent motionEvent) {
Log.d(TAG, "onLongPress");
if(mCustomTouchMethods!=null)
mCustomTouchMethods.onLongPress(motionEvent);
}
@Override
public boolean onScroll(MotionEvent motionEvent,MotionEvent motionEvent2,float v, float v2) {
Log.d(TAG, "onScroll");
if(mCustomTouchMethods!=null)
return mCustomTouchMethods.onScroll(motionEvent,motionEvent2,v, v2);
return false;
}
@Override
public boolean onSingleTapUp(MotionEvent motionEvent) {
Log.d(TAG, "onSingleTapUp");
if(mCustomTouchMethods!=null)
return mCustomTouchMethods.onSingleTapUp(motionEvent);
return false;
}These functions do nothing but pass on events to the event listener, if there is any. We're responding to the following events: onTouchEvent, onDown, onShowPress, onFlight, onLongPress, onScroll, onSingleTapUp.
One final piece of code remains for the overlay class. We've used something called OverlayEventListener at certain places in the class but have not yet defined it. Here's what it looks like:
public interface OverlayEventListener {
public boolean onCalibrate(MotionEvent e);
}
}With this defined, we will now be able to create event handlers for specific buttons being touched on the overlay (the calibrate and record buttons).
Once you've written the media file, you need a mechanism to read individual frames from the movie. We can use Android's Media Decoder to extract frames and convert them into OpenCV's native Mat data structure. We'll start by creating a new class called SequentialFrameExtractor.
Most of this section is based on Andy McFadden's tutorial on using the MediaCodec at bigflake.com.
public class SequentialFrameExtractor {
private String mFilename = null;
private CodecOutputSurface outputSurface = null;
private MediaCodec decoder = null;
private FrameAvailableListener frameListener = null;
private static final int TIMEOUT_USEC = 10000;
private long decodeCount = 0;
}mFilename is the name of the file that's being read, it should only be set when an object of SequentialFrameExtractor is created. CodecOutputSurface is a construct borrowed from http://bigflake.com that encapsulates logic to render a frame using OpenGL and fetches raw bytes for us to use. It is available on the website and also in the accompanying code. The next is MediaCodec—Android's way of letting you access the decoding pipeline.
FrameAvailableListener is an interface we'll create in just a moment. It allows us to respond whenever a frame becomes available.
What is TIMEOUT_USEC and decodeCount?
private long decodeCount = 0;
public SequentialFrameExtractor(String filename) {
mFilename = filename;
}
public void start() {
MediaExtractor mediaExtractor = new MediaExtractor();
try {
mediaExtractor.setDataSource(mFilename);
} catch(IOException e) {
e.printStackTrace();
}
}We've created a constructor and a new start method. The start method is when the decoding begins and it will start firing the onFrameAvailable method as new frames become available.
e.printStackTrace();
}
MediaFormat format = null;
int numTracks = mediaExtract.getTrackCount();
int track = -1;
for(int i=0;i<numTracks;i++) {
MediaFormat fmt = mediaExtractor.getTrackFormat(i);
String mime = fmt.getString(MediaFormat.KEY_MIME);
if(mime.startswith("video/")) {
mediaExtractor.selectTrack(i);
track = i;
format = fmt;
break;
}
}
if(track==-1) {
// Did the user select an audio file?
}Here, we loop over all the tracks available in the given file (audio, video, and so on) and identify a video track to work with. We're assuming this is a mono-video file, so we should be good to select the first video track that shows up.
With the track selected, we can now start the actual decoding process. Before that, we must set up a decoding surface and some buffers. The way MediaCodec works is that it keeps accumulating data into a buffer. Once it accumulates an entire frame, the data is passed onto a surface to be rendered.
int frameWidth = format.getInteger(MediaFormat.KEY_WIDTH);
int frameHeight = format.getInteger(MediaFormat.KEY_HEIGHT);
outputSurface = new CodecOutputSurface(frameWidth,frameHeight);
String mime = format.getString(MediaFormat.KEY_MIME);
decoder = MediaCodec.createDecoderByType(mime);
decoder.configure(format,outputSurface.getSurface(),null,0);
decoder.start();
ByteBuffer[] decoderInputBuffers =
decoder.getInputBuffers();
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
int inputChunk = 0;
boolean outputDone = false, inputDone = false;
long presentationTimeUs = 0;With the initial setup done, we now get into the decoding loop:
while(!outputDone) {
if(!inputDone) {
int inputBufIndex =decoder.dequeueInputBuffer(TIMEOUT_USEC);
if(inputBufIndex >= 0) {
ByteBuffer inputBuf =decoderInputBuffers[inputBufIndex];
int chunkSize = mediaExtractor.readSampleData(inputBuf, 0);
if(chunkSize < 0) {
decoder.queueInputBuffer(inputBufIndex,0, 0, 0L,
mediaCodec.BUFFER_FLAG_END_OF_STREAM);
inputDone = true;
} else {
if(mediaExtractor.getSampleTrackIndex()!= track) {
// We somehow got data that did not
// belong to the track we selected
}
presentationTimeUs =mediaExtractor.getSampleTime();
decoder.queueInputBuffer(inputBufIndex,0, chunkSize, presentationTimeUs, 0);
inputChunk++;
mediaExtractor.advance();
}
}
} else {
// We shouldn't reach here – inputDone, protect us
}
}This is the input half of the media extraction. This loop reads the file and queues chunks for the decoder. As things get decoded, we need to route it to the places we need:
} else {
// We shouldn't reach here – inputDone, protect us
}
if(!outputDone) {
int decoderStatus = decoder.dequeueOutputBuffer(
info, TIMEOUT_USEC);
if(decoderStatus ==
MediaCodec.INFO_TRY_AGAIN_LATER) {
// Can't do anything here
} else if(decoderStatus ==
MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
// Not important since we're using a surface
} else if(decoderStatus ==
MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
MediaFormat newFormat = decoder.getOutputFormat();
// Handled automatically for us
} else if(decoderStatus < 0) {
// Something bad has happened
} else {
if((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != ) {
outputDone = true;
}
}
boolean doRender = (info.size != 0);
decoder.releaseOutputBuffer(decoderStatus, doRender);
if(doRender) {
outputSurface.awaitNewImage();
outputSurface.drawImage(true);
try {
Mat img = outputSurface.readFrameAsMat();
if(frameListener != null) {
Frame frame = new Frame(img, presentationTimeUs,
new Point3(), new Point());
frameListener.onFrameAvailable(frame);
}
} catch(IOException e) {
e.printStackTrace();
}
decodeCount++;
if(frameListener!=null)
frameListener.onFrameComplete(decodeCount);
}
}
}
medaiExtractor.release();
mediaExtractor = null;
}This completes the output half of the decode loop. Whenever a frame is complete, it converts the raw data into a Mat structure and creates a new Frame object. This is then passed to the onFrameAvailable method.
Once the decoding is complete, the media extractor is released and we're done!
The only thing left is to define what FrameAvailableListener is. We shall do that now:
public void setFrameAvailableListener(FrameAvailableListener listener) {
frameListener = listener;
}
public interface FrameAvailableListener {
public void onFrameAvailable(Frame frame);
public void onFrameComplete(long frameDone);
}
}This is a common pattern in Java when defining such listeners. The listeners contain methods that are fired on specific events (in our case, when a frame is available, or when the processing of a frame is complete).