Abstract
This chapter features a family collage sketch. The critical programming concepts are playing a video clip and handling video together with images and rectangles. The rectangles represent the category of graphics. The distinct types of items are defined using classes and subclasses.
In addition to introducing the use of video, this chapter and its example can be viewed as another lesson on classes. You can go back to Chapter 4 and review the bouncing things example. The code for repositioning of the pieces by mouse actions has some similarities to creating a line and then moving an image on the line example in Chapter 4.
The source code material includes an extra example, a demonstration of using line drawing on canvas or an image or a video for directions for an origami model. Illustrations are given as a teaser.
Programming Concepts
This section provides general background on video and the notions of shallow vs. deep copying. I then move to the details of how video is handled in Processing and more on classes and subclasses.
Video
Digital video files come in a variety of formats, just like images. It should be easy to accept that a considerable amount of data is involved, potentially a full image for every frame of the video. Some formats perform compression across frames as well as within frames. The term codec is used for the software or hardware device used for encoding and decompressing video. There are trade-offs to make between quality of the video and size of the file. There also are differences in the speed of going from the stored, digital format to presentation on the screen. It also can be important to ask if the video is to be streamed or acquired all at once. Regarding quality of the image, you might need to think about whether this video is to be viewed on a small screen, like a phone; a typical computer monitor; a high-definition TV; or projected on a large screen for viewing by a big audience.
Copying a Video
A general concept in computing is shallow copying vs. deep copying. It relates to the issue of whether your code is dealing with a value or a reference to a value. These issues arise in all programming languages and are not confined to working with videos or images. If your code copies the reference, it will be to the same value. In my first version of the collage, my duplicate method of the class I named MovieItem copied the reference to the Movie object. Therefore, the method produced two items in the window playing the same movie, frame by frame. I decided that I wanted the duplicate operation to produce a new, distinct copy of the video. With this approach, Annika does a round with herself. We can think of the movies being at different points in the reel. I did this by implementing a deep copy. You can examine the code in the “Program” section.
Processing Programming Features
The critical Processing programming features are video and more on classes and subclasses.
Video
where snowman.mov is a file that has been placed in the data folder of the sketch. The this term refers to the PApplet defined for the sketch and is one of the few situations in which we need to think about the Java program being created. When you get to the “Implementing the Family Collage Sketch” section, you will see the Movie constructor in use.
where the video is displayed and the dimensions of the display are dependent on the variables (xpos, ypos, iwidth, and iheight) set and maintained by my code.
You can use what you learned in previous chapters to access a video on the Web or ask the user to identify a video on the local computer. Do keep in mind that videos can be quite large, so you do need to make sure the video is fully loaded before attempting to use it. Do also read about obtaining streaming video, including video from webcams and other cameras.
Now what I have written about the use of the Movie class has a lot of detail. I hope you can appreciate that most of the implementation of video is done for us within the code of the Library.
Classes and Subclasses
My initial objectives for my family collage are to be able to move each item by dragging with the mouse, duplicate an item, and delete an item. I then realized that sometimes items get under other items, so I wanted a way to move an item to the top. I also decided that I wanted to be able to pause and restart a video. As I have written, I also decided to treat a duplicate of a video as a distinct entity. My implementation has the Item class and the ImageItem and MovieItem subclasses. I use a tab to hold all the class definitions. I name the tab definitions, which means that the sketch folder has a PDE file named Definitions.pde. The Item class is simply a rectangle. All the items are stored in an array.
In addition to the constructor methods, the methods are isOver, removeIt, display, move, duplicate, restart, and pauseMovie. Because of how I invoke the methods, which you will see in the code, I need to define all methods in the parent class. The restart and pauseMovie methods in the Item class are empty.
The example demonstrates the power of classes and subclasses to share the coding that is alike across classes and supply distinct coding when required.
Under the Covers
Reinforcing what has been said, videos are large, complex entities, and Processing (Java) handles them as such, playing each video independently of everything else in its own thread. The image function used for static images is used to display the current frame of a video. The function movieEvent is invoked by the underlying Java program in the same way that mousePressed, mouseReleased, mouseDragged, and keyPressed are invoked. The parameter to movieEvent indicates which Movie object had the event, so the call in the body of movieEvent to m.read(); responds to the event by updating the Movie with the available frame. If the event was something other than the arrival of a new frame, the last frame would be reread. Note that the one function handles all Movie objects.
Because videos are large, it can be critical to make sure that videos no longer in use are treated in such a way that garbage collection can reclaim the space. For the family collage sketch, this meant that I wrote code to assign the value null to the variable referencing the video when the user chose to delete it.
Family Collage Operation Overview
5 images. Image a is a bright rectangular box. Image b shows a child sitting at a table. Image c shows 1 child and 3 adults. Image d shows a child climbing a wall. Image d is a bright rectangular box. A label on the top reads c for copy comma d for delete comma t for move to top comma p for pause video comma r for restart.
Opening window for family collage
The image with the little girl at the table is a frame of a video clip that is being played. This frame was captured when I used the Grab utility to get the screenshot. The collage starts off with the video clip, two static images, and two rectangles. Each of the items can be repositioned. It also is possible to copy any item, move the copy around, and delete any item. The video clip can be paused and restarted.
8 images. 1 has 2 bright rectangular boxes. 2, a child sitting at a table with both hands raised. 3, a child sitting at a table. 4 and 5, 1 child and 3 adults. 6, a child climbing a wall. 7 and 8, are bright rectangular boxes. A label on the top reads c for copy comma d for delete comma t for move to top comma p for pause video comma r for restart.
Collage after some manipulation
You also should take note of the two rectangles at the upper left corner, as one is slightly offset from the other. My duplicate method creates the second item slightly offset from the position of the original.
Implementing the Family Collage Sketch
Once you understand how videos work, I claim that the implementation is straightforward and based on what you already know about classes and subclasses and mouse events for dragging.
Planning
I decided to have the different types of items in my collage implemented as subclasses of an Item class and to store all Item objects in an array. Because I was providing a way to move items around, I needed to erase the window and redisplay everything: all the items and the instructions. Methods to display an item would be present in each class. Processing provides the ArrayList construct, which you read about in the snake example in Chapter 5, with its own remove method. However, I chose to use a standard array to hold all the Item objects, mainly to show that it was possible to write a function for removing an element, which I did in the removeFromItemsArray, using two for-loops.
Each Item has object variables indicating the horizontal and vertical coordinates. The move method performs an incremental move. The change amounts are calculated using mouseX-pmouseX and mouseY-pmouseY.
Function Table for collage6 Tab
Function | Invoked by | Invokes |
|---|---|---|
setup | Underlying Java program | Item, ImageItem, MovieItem |
draw | Underlying Java program | Appropriate display method |
overWhich | mousePressed | isOver |
keyPressed | Underlying Java program | overWhich, swapThem, appropriate duplicate, removeIt, restart, pauseMovie |
swapThem | keyPress | |
removeFromItemsArray | removeIt | |
mousePressed | Underlying Java program | overWhich |
mouseDragged | Underlying Java program | move |
mouseReleased | Underlying Java program | |
movieEvent | Underlying Java program | The read method for a Movie |
Programming the Family Collage Sketch
Code for Class definitions
class Item { | Header for Item |
float xpos; | Horizontal coordinate can change |
floatypos; | Vertical coordinate can change |
float iwidth; | Width |
float iheight; | Height |
int cred; | Redness |
int cgreen; | Greenness |
int cblue; | Blueness |
Item(float x,float y,float w,float h,int red,int green, int blue) { | Header for Item constructor |
xpos = x; | Sets initial horizontal coordinate |
ypos = y; | Sets initial vertical coordinate |
iwidth = w; | Sets width |
iheight = h; | Sets height |
cred = red; | Sets redness |
cgreen = green; | Sets greenness |
cblue =blue; | Sets blueness |
} | Closes constructor |
Boolean isOver(float x,float y) { | Header for isOver method |
return ((x>xpos)&&(y>ypos)&&(x<(xpos+iwidth))&&(y<(ypos+iheight))); | Returns the result of calculation against four sides |
} | Closes isOver |
void removeIt(int i) { | Header for removeIt |
removeFromItemsArray(i); | Removes from the Items array |
} | Closes removeIt |
void display() { | Header for display |
fill(cred,cgreen,cblue); | Sets the color |
rect(xpos,ypos,iwidth,iheight); | Draws rectangle |
} | Closes display |
void move(float dx,float dy) { | Header for move |
xpos +=dx; | Incrementally adjusts xpos |
ypos +=dy; | Incrementally adjusts ypos |
} | Closes move |
void duplicate() { | Header for duplicate |
Item copy; | For the copy |
copy = new Item(xpos+10, ypos+10,iwidth,iheight,cred,cgreen,cblue); | Creates new Item, offset position |
items = (Item[]) append(items,copy); | Adds to Items array |
} | Closes duplicate |
void restart() { | Header for restart |
} | Closes empty method |
void pauseMovie() { | Header for pauseMovie |
} | Closes empty method |
} | Closes Item class definition |
class MovieItem extendsItem{ | Header for MovieItem subclass of Item |
Movie imovie; | iMovie references the Movie object |
String movieFileName; | Holds movie file name, used by duplicate |
PApplet paref; | Holds reference to the PApplet, used by duplicate |
MovieItem (float x,float y,float w,float h,String mfn, PApplet par ) { | Constructor |
super(x,y,w,h,255,255,255);//sets up white rectangle | Calls parent constructor to set base variables |
imovie = new Movie(par,mfn); | Sets reference to the Movie |
movieFileName = mfn; | Sets the name of file |
paref =par; | Sets reference to PApplet |
imovie.loop(); | Starts the movie |
} | Closes constructor |
void removeIt(int i) { | Header for removeIt |
imovie.stop(); | Stops the movie |
imovie = null; | Extra precaution to remove the link to movie, for garbage collection |
super.removeIt(i); | Calls parent method |
} | Closes removeIt |
void duplicate() { | Header for duplicate |
Itemcopy; | Will hold the copy |
copy = new MovieItem(xpos+10, ypos+10,iwidth,iheight,movieFileName,paref); | Creates the MovieItem; this will start the new Movie. Note reference to the paref |
items = (Item[]) append(items,copy); | Adds to items |
} | Closes duplicate |
void display() { | Header for display |
image(imovie, xpos, ypos,iwidth,iheight); | Draws in window the current frame |
} | Closes display |
void restart() { | Header for restart |
imovie.jump(0); | Goes to first frame |
imovie.loop(); | Starts movie |
} | Closes restart |
void pauseMovie() { | Header for pauseMovie |
imovie.pause(); | Pause (uses the method of Movie object) |
} | Closes pauseMovie |
} | Closes MovieItem class definition |
class ImageItem extends Item { | Header for ImageItem |
PImage myImage; | Reference to PImage |
String filename; | File name, used by duplicate |
ImageItem (float x,float y,float w,float h, String imagefilename) { | Header constructor |
super(x,y,w,h,255,255,255);//sets up white rectangle | Calls parent constructor to set base variables |
filename =imagefilename; | Saves file name |
myImage = loadImage(imagefilename); | Sets PImage |
} | Closes constructor |
void duplicate() { | Header for duplicate |
Item copy; | For copy |
copy = new ImageItem(xpos+10, ypos+10,iwidth,iheight,filename); | Creates new ImageItem, offset |
items = (Item[]) append(items,copy); | Adds to Items array |
} | Closes duplicate |
void display() { | Header for display |
image(myImage, xpos, ypos,iwidth,iheight); | Draws image |
} | Closes display |
} | Closes class definition |
Code for collage6
import processing.video.*; | Imports video library |
MovieItem myMovieItem; | Used by setup |
Item[] items = {}; | Will hold all the items, starts out empty |
Item curItem = null; | Will hold the item being dragged |
void setup() { | Header for setup |
size(1000, 1000); | Sets window |
Item myItem1 = new Item(10,30,100,200,250,0,200); | Creates rectangle item |
items = (Item[]) append(items,myItem1); | Adds to items |
Item myItem2 = new Item(500,800,200,100,0,100,100); | Second rectangle |
items = (Item[]) append(items,myItem2); | Adds to items |
myMovieItem = new MovieItem(250,200,300,200,"snowman.mov",this); | Creates MovieItem, including creating Movie, starting Movie. This refers to the PApplet |
items= (Item[]) append(items,myMovieItem); | Adds to items |
ImageItem myImage = new ImageItem(10,500,205,154,"pigtails1.JPG"); | Creates first ImageItem |
items = (Item[]) append(items,myImage); | Adds to items |
myImage = new ImageItem(600,300,300,400,"climbing.jpg"); | Creates second ImageItem |
items = (Item[]) append(items,myImage); | Adds to items |
} | Closes setup |
void draw() { | Header for draw |
background(255); | Erases window |
text("c for copy, d for delete, t for move to top, p for pause video, r for restart.", 5,20); | Outputs the instructions |
for (int i=0; i<items.length;i++){ | Loop through items |
items[i].display();//use appropriate method | Displays each item |
} | Closes for-loop |
} | Closes draw |
int overWhich() { | Header for overWhich. The function determines the first item the mouse is over |
for (int i=0; i<items.length;i++) { | Loop through items |
if (items[i].isOver(mouseX,mouseY)){ | Is it over the item |
return i; | Returns the index (exit function) |
} | Closes if true clause |
} | Closes the for-loop |
return -1; | Did not exit function in the for-loop, so return -1 |
} | Closes overWhich |
void keyPressed() { | Header for keyPressed |
int i; | Will hold the return value of overWhich |
i = overWhich(); | Invokes overWhich |
if (i < 0) { | If negative, means not over any item |
return; | Return; no action |
} | Closes if true clause |
else { | Else |
switch(key) { | Switch depending on letter pressed |
case 'c': | |
items[i].duplicate(); | Copies item at ith position and adds to items array |
break; | Leaves switch |
case 'd': | |
items[i].removeIt(i); | Deletes ith element from items array |
break; | Leaves switch |
case 't': | |
swapThem(i,items.length-1); | Swaps the item with the last; that is, the topmost |
break; | Leaves switch |
case 'r': | |
items[i].restart(); | Restarts the item |
break; | Leaves switch |
case 'p': | |
items[i].pauseMovie(); | Pauses the item |
break; | Leaves switch |
default: | |
println("invalid key pressed"); | Shows up on console |
break; | Leaves switch (not strictly necessary because this is the last) |
} | Closes switch |
} | Closes else clause |
} | Closes keyPressed |
void swapThem(int j,int k) { | Header for swapThem; general function, although only used to swap with the topmost item |
Itemtemp; | Needs a placeholder |
temp = items[j]; | Sets item at j |
items[j] = items[k]; | Swaps in item at k |
items[k] = temp; | Sets item at k |
} | Closes swapThem |
void removeFromItemsArray(int i) { | Header for removeFromItemsArray |
Item[] tempitems = new Item[items.length-1]; | Creates new array, size one less than the size of current Items array |
for (int k = 0; k<i;k++) { | For loop up to item to remove |
tempitems[k] = items[k]; | Copy over kth item |
} | Closes loop |
for (int k=i+1;k<items.length;k++) { | For loop starting after item to remove |
tempitems[k-1] = items[k]; | Copy over kth item |
} | Closes loop |
items =tempitems; | Sets items to the array just created and populated |
} | Closes removeFromItemsArray |
void mousePressed() { | Header for mousePressed |
int i; | Will hold the index of item |
i = overWhich(); | Find out if on any item |
if (i>=0) { | If this is a valid item |
curItem = items[i]; | Sets curItem |
} | Closes if true clause |
} | Closes mousePressed |
void mouseDragged() { | Header for mouseDragged |
float dx; | Will hold incremental x amount |
floatdy; | Will hold incremental y amount |
if (curItem!=null) { | Only do this if curItem is set |
dx = mouseX- pmouseX; | Calculates horizontal change |
dy = mouseY- pmouseY; | Calculates vertical change |
curItem.move(dx,dy); | Sets curItem to move these amounts |
} | Closes if clause |
} | Closes mouseDragged |
void mouseReleased(){ | Header for mouseReleased |
curItem = null; | Sets curItem to null, which means no more dragging |
} | Closes mouseReleased |
void movieEvent(Movie m) { | Header for movieEvent |
m.read(); | Reads the next frame for the specified Movie |
} | Closes movieEvent |
Things to Look Up
You can and should read the documentation on the use of video. This would include a video in a file and videos captured from cameras. If you have a webcam on your computer, you can investigate how to include it in a sketch.
How to Make This Your Own
Add other types of items, including drawings, perhaps what you made for Chapter 1.
Inspired and instructed by the last two chapters, incorporate ways for collage makers to access image files and video files on their own computers or on websites.
The collage example given here, by allowing multiple copies of the one video, results in Annika singing a round with herself. If you decide to allow more than one movie, either by uploading the video file(s) to the data folder or providing the capability of the collage maker adding videos dynamically, you might want to address the issue of too much sound. One approach I took for a program using JavaScript was to specify a sound level for each video. However, there now are policies involving autoplay of video on the Web that made me opt for muting audio on my collage program.
You can incorporate video controls. Perhaps you want the restart operation to continue playing where the video was paused.
The image shows an inverted triangle. It has an arrow from right to left outside, and 3 lines inside the triangle. Below the triangle reads, fold in half to the left. 2 labels below read go back and next step.
Step in making kissy fish
The image shows an open diamond-like shape. The text below reads, stick your finger in its mouth and move the inner folded material to one side. 2 ovals below read, go back and next step.
Step before throat surgery
The image shows an open diamond-like shape. The text below reads, throat fixed. 2 ovals below read, go back and next step.
Model after throat surgery
The image shows a fish-like object held by hands. The text below reads talking fish. 2 ovals below read, go back and next step.
Video showing kissy fish in operation
Thus, the sketch demonstrates the use of graphics, image, and video. The reason I omitted extensive coverage of this example was not because I felt the programming was too difficult, but because of the complexity of the algebra and trigonometry required for the line drawings and origami is my interest and, possibly, not yours. You are welcome to try it out and examine the coding. The origami directions sketch might inspire you to think about a process you know that consists of steps where you can decide what the best way is to convey each individual step: a drawing made dynamically, an image, a video, or perhaps an audio file or something else.
Review previous examples and think about incorporating video clips, perhaps as an initial, splash type of opening screen, or as a response to some action. Perhaps you can make your chicken fly off if hit by the slingshot.
What You Learned
This chapter introduces incorporating video into your sketches. The Movie object, or, more precisely, a reference to the Movie object, was a variable in a class. The use of classes and subclasses demonstrated the object-oriented approach that provides a way to be consistent about what code can be shared and what must be different in dealing with a set of things.
What’s Next
The next chapter describes the implementation of the paper-and-pencil game, Hangman. I provide two approaches. In one, the player enters letters by typing keys on the keyboard. In the other, a button is provided for each letter. My intention is to encourage you to realize that there generally are multiple ways to implement something even as simple and familiar as this game. For my sketch, I make use of a very small word list provided as a CSV file in the data folder. I use some of my favorite words and discuss scaling up.