Motion capture in the entertainment field is the descendant of rotoscoping, a technique still used by some traditional animation studios to copy realistic motion from film footage onto cartoon characters.

The rotoscope device was invented and patented by cartoonist Max Fleischer in 1915, with the intent of automating the production of cartoon films. The device projected live-action film, a frame at a time, onto a light table, allowing cartoonists to trace the frame's image onto paper. The first cartoon character ever to be rotoscoped was Koko the Clown. Fleischer's brother, Dave, acted out Koko's movements in a clown suit. Fleischer wanted to use Koko to convince the big studios to use the new process for their cartoon projects. The sale was difficult because it had taken Fleischer about a year to produce the initial 1-min cartoon using the technique, so he couldn't market it as a mass production tool. Eventually, Fleischer realized that rotoscoping would be a viable technique only for certain shots that required realistic motion.

Walt Disney Studios used some rotoscoping in 1937 to create the motion of human characters in Snow White. Snow White herself and the Prince were partially rotoscoped. The decision to use rotoscoping wasn't a matter of cost, but of realistic human motion. In fact, Snow White went tremendously over budget due to the complexity of the animation.

Rotoscoping has been adopted over the years by many cartoon studios, but few actually admit using it because many people in the animation industry consider it cheating and a desecration of the art of animation.

A two-dimensional (2D) approach, rotoscoping was designed for traditional, hand-drawn cartoons. The advent of 3D animation brought about the birth of a new, 3D way of rotoscoping. Hence, motion capture.

Some of the current motion capture technologies have been around for decades, being used in different applications for medical and military purposes. Motion capture in computer graphics was first used in the late 1970s and early 1980s in the form of research projects at schools such as Simon Fraser University, Massachusetts Institute of Technology, and New York Institute of Technology, but it was used in actual production only in the mid-1980s.

In late 1984, Robert Abel appeared on a talk show and was asked if he would soon be able to counterfeit people digitally. “We are a long ways from that,” he replied. “We haven't even figured out human motion, which is the basis, and that's a year away.” A week and a half later, Abel received a visit on a Friday afternoon from a creative director from Ketchum, a prominent advertising agency. The visitor brought six drawings of a very sexy woman made out of chrome. She was to have Kathleen Turner's voice and would be the spokesperson for the National Canned Food Information Council, an association formed by Heinz, Del Monte, Campbell's, and a number of big players that sold canned food. They felt they had to make a powerful statement because the idea of buying food in cans was becoming obsolete, so they wanted to do something really different and outrageous, and they wanted it to air during the Super Bowl in January 1985. “Can you do it?” asked the client. “You're certainly here a lot earlier than I would have planned,” replied Abel, and asked the client to wait until the end of the weekend for an answer.

At that time most computer graphics consisted of moving logos, landscapes, and other hard objects, and Robert Abel and Associates had already become a player in that market, along with MAGI, Triple-I (Information International, Inc.), John Whitney's Digital Productions, and PDI, all of which had their own proprietary software, because at that time there was almost no off-the-shelf animation software and whatever was available was still in its infancy. Abel's software was initially based on bits and pieces from Bell Labs, Evans and Sutherland, JPL, and other places, and was augmented over time by his group.

The next step would be to animate a digital character. “For storytelling, which was really our goal, we had to have human characters,” recalled Abel, “because nobody better than a human character is able to convey emotion and story. We come from a long line of storytellers that go back maybe 35,000 years, and although the forms may change from cave paintings to digitally made motion pictures, it's still the same thing, it's the passing on of stories.” Creating the first animated digital character would open a Pandora's Box of many new challenges, such as creating realistic skin, hair, and expression. But first they had to deal with the motion.

Abel and his team decided to lock the doors and not leave the building until Monday morning. If by then they didn't have the solution figured out, they would have to pass on the project. Robert Abel and Associates' background in shooting miniatures with motion control cameras since the late 1960s and early 1970s was the key to the solution. They knew that the answer to their problem would have to do with motion and control, except this time it would be human motion. Keyframe character animation was not an option at the time, so they decided to find a way to track the motions of a woman acting the part of the character. It made sense to shoot the woman with several cameras from different points of view, and then use this footage to create a motion algorithm.

Seven people worked throughout the weekend. “Several of us got into our underwear,” recalled Abel. “We got black adhesive dots and we put them on our bodies and we would photograph each other with Polaroid cameras and then we would lay out these Polaroids so we could see how they changed from angle to angle.” They continued this slow deductive reasoning process until Sunday at 3 A.M., when they decided that it would take a few weeks to digitize all the motion. It would be close, but they felt that they could do the job in the 8-week schedule that the client had established.

Among the people involved in this project besides Bob Abel were Bill Kovacs and Roy Hall, who later became cofounders of Wavefront Technologies; Frank Vitz, now a Senior Art Director at Electronic Arts; Con Pederson, cofounder of Metrolight Studios a few years later; Randy Roberts, who directed the project and still a commercial director; Richard Hollander, former president of Blue Sky/VIFX and of the feature film division at Rhythm and Hues, now at Pixar; Neil Eskuri, currently visual effects director at Rainmaker in Vancouver; Charlie Gibson, Oscar-winning visual effects supervisor for Babe and Pirates of the Caribbean; and John Hughes, who later became cofounder and president of Rhythm and Hues.

They found a woman who was very pretty and graceful and had been a dancer and a model. They had decided that motion on 18 hinge points would be necessary to achieve the desired result, so with black magic markers they put black dots on each of her 18 joints. A stool with a 360° axis of rotation in the middle was assembled so that the model could sit and perform the moves without obstacles. The team photographed her from multiple points of view, and then managed to import the images to SGI Iris 1000 systems. These workstations appeared in early 1984 and were the first model produced by Silicon Graphics. They were then able to analyze the difference in measurement between pairs of joints (e.g., the elbow and the wrist) for each point of view and to combine them to come up with a series of algorithms that would ultimately be used to animate the digital character. This process was done on a frame-by-frame basis and took 4½ weeks.

At the same time, the wire-frame model was built in separate sections, all rigid parts. The motion algorithms were applied to all the combined moving parts, and the animation was output as a vector graphic. “We then had to deal with the basic issue of wrapping her body in chrome,” said Abel. “Of course, there is no way in the world we could do ray-tracing to real reflective chrome the way those guys do it at SIGGRAPH, with those multimillion dollar supercomputers. We had VAX 750s, which were early DEC computers.” This problem was solved by Charlie Gibson, who figured out a way of texture mapping the body so that when it moved, the map would animate following the topology of the body. Today we call this a reflection map.

The last challenge was to render the final spot, all 30s of it, in the 2 weeks that they had left. “The good and the bad news is this,” Abel announced. “We don't nearly have the horsepower, but the VAX 750 is a staple of almost every university, laboratory, and engineering place in the country.” They ended up using 60 additional VAX 750s around the country, from Florida to Alaska to Hawaii, and even a few places in Canada. The final spot, “Brilliance,” was rendered and pieced together about 2 days before the delivery date. It is now known by most people in the industry as “Sexy Robot.” Abel, who passed away in 2001, was a pioneer in visual effects, computer graphics, and motion capture.

Pacific Data Images (PDI) was the oldest operating computer animation studio until it was completely acquired by DreamWorks in 2000, and it played a big part in the history of performance animation. Founded in 1980 in Sunnyvale, California, by Carl Rosendahl, and later joined by Richard Chuang and Glenn Entis, it wasn't until 8 years later that the studio would produce its first project using some kind of human tracking technology.

Over the years, PDI used different types of tracking devices that fit particular project needs, ranging from custom-built electromechanical devices to electromagnetic and optical tracking systems, but it wasn't PDI's intention to specialize in motion capture. “We use technology where it is appropriate,” says Richard Chuang. “We are not a one-technique company; our goal is not to be a master of any one thing, just be good at a lot of them.”

Brad deGraf also experimented with performance animation while working at Digital Productions, using Jim Henson's Waldo device. After Digital Productions was purchased by Omnibus, Brad left and joined forces with Michael Wahrman, also from Digital Productions, and founded deGraf/Wahrman Production Company.

In 1988, Silicon Graphics contracted the services of deGraf/Wahrman to create a demonstration piece for their new 4D models. The piece was an interactive animation called “Mike the Talking Head,” which was showcased at SIGGRAPH'88. For the first time, an animated character was able to interact with the audience. The controls that animated the character in real time were operated by a puppeteer during the conference. deGraf/Wahrman's proprietary software was used to create an interface between the controls and the rendering engine and to produce interpolated instances of the character's geometry. The new Silicon Graphics 4D workstation had the horsepower to render the character in real time.

When deGraf and Wahrman dissolved their partnership, Brad joined Colossal Pictures and started rewriting the performance animation software. In 1993, he developed “Moxy,” a character for the Cartoon Network that was operated in real time using an electromagnetic tracker. In 1994, he founded Protozoa, a spin-off from Colossal's performance animation studio. His focus has been on real-time performance animation solutions, including software and character development for different media, including television and the Web. ALIVE, Protozoa's proprietary performance animation software, supports multiple input devices, including motion capture systems, joysticks, and MIDI controllers. It also outputs to different formats, including live television and the World Wide Web via VRML (Virtual Reality Modeling Language).

In 1986, Jeff Kleiser began experimenting with motion capture while he was at Omnibus Computer Graphics. “We used the optical system from Motion Analysis in Santa Rosa, California, to encode martial arts movement for use in a test for Marvel Comics,” recalls Kleiser. “Results were disappointing due to the alpha code we were working with.”

In 1987, after Omnibus closed, Kleiser joined forces with Diana Walczak, who had been sculpting human bodies, and founded Kleiser-Walczak Construction Company. Their specialty would be to build and animate computer-generated actors, or Synthespians.

“After creating our first Synthespian, Nestor Sextone in 1988, we got together with Frank Vitz and went back to Motion Analysis to capture motion for our second digital actor, Dozo, in creating the film Don't Touch Me, in 1989,” says Kleiser. “We were only able to get about 30s of usable motion capture, and we had to recycle it to fill the 3.5min of the song we had written.” Vitz had been working with Robert Abel and Associates and had some motion capture experience, as he had been part of the team that created “Brilliance.”

Over the years, Kleiser-Walczak has created digital actors for special venue, commercial, and feature film projects. They created dancing water people for the Doug Trumbull stereoscopic ride “In Search of the Obelisk,” located inside the Luxor Hotel in Las Vegas. Using a Flock of Birds electromagnetic system by Ascension Technology Corporation, they also created digital stunt doubles for Sylvester Stallone, Rob Schneider, and others for the film Judge Dredd. In their most recent use of motion capture, Kleiser-Walczak produced “Trophomotion,” a commercial spot for Stardox athletic braces in which two basketball trophies come to life. They used a combination of keyframe animation and motion capture, which they achieved with an optical system manufactured by Adaptive Optics in Cambridge, Massachusetts.

Their latest project is computer imagery for “The Amazing Adventures of Spiderman,” a ride for Universal Studios' Islands of Adventure, in Orlando, Florida. “We tested mocap [motion capture] for this project, but it quickly became clear that superhero characters need to have super-human motion, and that keyframe animation gave us the look and flexibility we wanted,” says Kleiser. “All the animation in the project was therefore done with keyframe animation.”

In the early 1990s, projects utilizing motion capture in computer graphics were starting to become part of actual production work, so companies whose main business was based on this technology started to surface. Medialab, Mr. Film, Windlight Studios, SimGraphics, and Brad deGraf at Colossal Pictures concentrated their efforts on real-time applications that included character development and puppeteering, while Biovision, TSi, and Acclaim embraced the non-real-time technology for the up-and-coming video game market. At the same time, commercial production using the now traditional motion capture techniques was initiated by Homer and Associates.

In the early 1990s, a few companies started offering motion capture services to other production companies. The early players were Biovision in San Francisco and TSi in Los Angeles and San Francisco. The systems used were optical and the software to solve the motion data was all proprietary, because the few companies that sold optical motion capture equipment didn't have biomechanical solvers that could deliver data useable for animation to any of the current 3D animation software programs. Entertainment was a brand new market for them as they have been servicing mostly the Life Sciences sector at that time, so they looked at the service bureaus to help them access the market.

Owning and operating an optical system at the time was not only very expensive, but extremely technical and not user friendly by any means. The data was very difficult to collect and there was always a risk that the final data would be unusable. Verifying the data during the shoot wasn't possible as it took a very long time to process. Initially, at TSi we had mostly clients in the video gaming industry. Video game consoles were becoming more powerful and could render more complex motions, so most game developers started using motion capture when service bureaus started doing business. We were servicing clients from all over the world for a few years until motion capture systems became more user friendly. In 1997, TSi sold their biomechanical solver and animation software plug-ins to Motion Analysis Corporation. Optical systems started including the necessary software to produce performance animation and other service bureaus started opening in many countries. Also, larger video game developers such as Electronic Arts started acquiring systems to produce in-house graphics. Today it is a given that any video game that has human characters is expected to have used motion capture to generate the performance data.

Optical motion capture is a very accurate method of capturing certain motions when using a state-of-the-art system. It is not a real-time process in most cases; immediate feedback is not possible on the target character unless motions are not too complex and there aren't too many performers. Data acquired optically can require extensive postprocessing to become usable, so operating costs can be high.

A typical optical motion capture system is based on a single computer that controls the input of several digital CCD (charge-coupled device) cameras. CCDs are light-sensitive devices that use an array of photoelectric cells (also called pixels) to capture light, and then measure the intensity of the light for each of the cells, creating a digital representation of the image. A CCD camera contains an array of pixels that can vary in resolution from as low as 128 × 128 to as high as millions of pixels. The state of the art today is 16 million pixels (megapixels), but that number will continue to increase every year.

Obviously, the higher the resolution, the better, but there are other trade-offs. The samples-per-second rate, or frame rate, has to be fast enough for capturing the nuances of very fast motions. In most cases, 60 samples per second are more than enough, but if the motions are very fast, such as a baseball pitch, a lot more samples per second are required. By today's standards, a CCD camera with a resolution of 16 megapixels would be able to produce up to 120 samples per second at that resolution. To capture faster motions, the resolution has to be dropped. Another important feature is shutter synchronization, by which the camera's shutter speed can be synchronized with external sources, such as the light-emitting diodes (LEDs) with which optical motion capture cameras are usually outfitted.

Some of the most modern cameras process the imagery locally before sending it to the computer, saving broadband and processing time of huge amounts of data. This can be very useful, especially when the system has a large number of ultra-high-resolution cameras, each capturing hundreds of images per second.

The number of cameras employed is usually no less than 8 and no more than 32, but there are cases where hundreds of cameras are used as I will explain later. They capture the position of reflective markers at speeds anywhere between 30 and 2000 samples per second. The cameras are normally fitted with their own light sources that create a directional reflection from the markers, which are generally spheres covered with a material such as Scotch-Brite tape. Red light sources are preferred because they create less visual distortion for the user. Infrared is also used but it is slightly less effective than visible red. The marker spheres can vary from a few millimeters in diameter, for facial and small-area captures, to a couple of inches. The Vicon system, shown in Figure 1.5, is an example of a state-of-the-art optical system that can accommodate up to hundreds of cameras.

Most optical systems were designed and manufactured for medical applications; as such, they lacked many features that are important to computer graphics applications. The Vicon 8 system was the first system designed with computer graphics in mind. It was the first able to support SMPTE time code, a time stamp used by most film and television applications. Even if you videotaped your capture session, there was no easy way to match the video to the actual motion data. Having time code in the motion data allows you to edit the motion files, as you would do with live-action video, and properly plan the association of the characters with background plates. Another very useful feature that the Vicon 8 introduced was the fact that reference video of the session could be synchronized, or genlocked, with the actual data collection. In addition to video genlock, the Vicon 8 software could shoot AVI movie files at the same time as it captured. These movie files are great reference for data postprocessing and application.

The optical system must be calibrated by having all the cameras track an object with known dimensions that the software can recognize, such as a cube or a wand with reflective markers. By combining the views from all cameras with the known dimensions of the object, the exact position of each camera in space can be calculated. If a camera is bumped even slightly, a new calibration must be performed. It is a good idea to recalibrate the system many times during a capture shoot, since any kind of motion or vibration can shift the position of a camera, especially if the studio is located on unstable ground.

At least two calibrated views are needed to track a single point's 3D position, and extra cameras are necessary to maintain a direct line of sight from at least two cameras to every marker. That doesn't mean that more cameras are better, because each additional camera increases postprocessing time. There are other methods for minimizing occlusions that are implemented in software and used during postprocessing. The most time- and cost-effective solution is different for every case, depending on the type, speed, and length of the motion, as well as on the volume of capture and the available light. Figure 1.6 shows a performance being filmed on an optical motion capture stage.

Once the camera views are digitized into the computer, it is time for the postprocessing to begin. The first step is for the software to try to produce a clean playback of only the markers. Different image processing methods are used to minimize the noise and isolate the markers, separating them from the rest of the environment. The most basic approach is to separate all the groups of pixels that exceed a predetermined luminosity threshold. If the software is intelligent enough, it will use adjacent frames to help solve any particular frame. The system operator has control over many variables that will help in this process, such as specifying the minimum and maximum lines expected per marker, so the software can ignore anything smaller or bigger than these values.

The second step is to determine the 2D coordinates of each marker for each camera view. This data will later be used in combination with the camera coordinates and the rest of the camera views to obtain the 3D coordinates of each marker. The third step is to actually identify each marker throughout a sequence. This stage requires the most operator assistance, since the initial assignment of each marker has to be recorded manually. After this assignment, the software tries to resolve the rest of the sequence until it loses track of a marker due to occlusion or crossover, at which point the operator must reassign the markers in question and continue the computation. This process continues until the whole sequence is resolved and a file containing positional data for all markers is saved.

The file produced by this process contains a sequence of marker global positions over time, which means that only each marker's Cartesian (x, y, and z) coordinates are listed per frame and no hierarchy or limb rotations are included. It is possible to use this file for computer animation, but a more extensive setup is required inside the animation software in order to resolve the final deformation skeleton to be used. Experienced technical directors can benefit by using this file's data, because it allows more control over what can be done in a character setup. For the average user, however, the data should be processed further, at least to the point of including a skeletal hierarchy with limb rotations. Most optical systems include data-editing systems that allow the user to produce the rotational hierarchical data before importing it to animation software. Figure 1.7 shows a sample screen of the Vicon Blade integrated motion capture platform that is shipped with the Vicon MX optical motion capture system.

For the past couple of decades, there have been no major changes in the way motion capture is done. Since the late 1980s, optical has been the preferred method in the entertainment and life sciences fields, as accuracy is more necessary than real-time measurement.

Technological advances in hardware since optical systems emerged have been mostly in camera count and camera resolution. Meanwhile, other technologies have remained stagnant in their advances. Software improvements have been mostly in finding clever ways to obtain somewhat real-time feedback and better ways of using and managing data.

Radio frequency (RF) is the new emerging technology in position tracking. Not to be confused with using RF to deliver data wirelessly, but rather using RF to calculate position measurement. Currently, there are several types of RF-based systems, but none of them has the accuracy needed for performance capture. However, it is clear that this will be the next state of the art in motion capture.

Electromagnetic motion capture systems are part of the six degrees of freedom electromagnetic measurement systems' family and consist of an array of receivers that measure their spatial relationship to a nearby transmitter. These receivers or sensors are placed on the body and are connected to an electronic control unit, in most cases by individual cables.

Also called magnetic trackers, these systems emerged from the technology used in military aircraft for helmet-mounted displays (HMDs). With HMDs, a pilot can acquire a target by locating it visually through a reticule located on the visor. A sensor on the helmet is used to track the pilot's head position and orientation.

A typical magnetic tracker consists of a transmitter, 11–18 sensors, an electronic control unit, and software. A state-of-the-art magnetic tracker can have up to 90 sensors and is capable of capturing up to 144 samples per second. The cost ranges from $5000 to $150,000, considerably less than optical systems.

The transmitter generates a low-frequency electromagnetic field that is detected by the receivers and input into an electronic control unit, where it is filtered and amplified. Then, it is sent to a central computer, where the software resolves each sensor's position in x, y, and z Cartesian coordinates and orientation (yaw, pitch, and roll).

Magnetic trackers such as the Flock of Birds by Ascension Technology Corporation use direct current (DC) electromagnetic fields, whereas others, such as the Polhemus Fastrak, use alternating current (AC) fields. Both these technologies have different problems associated with metallic conductivity. AC trackers are very sensitive to aluminum, copper, and carbon steel, but not as sensitive to stainless steel or iron, whereas DC trackers have problems with ferrous metals, such as iron, but not with aluminum and copper.

Many of these conductivity problems are caused by the induction of a current in the metal that creates a new electromagnetic field that interferes with the original field emitted by the tracker. These new fields are called eddy currents. Some magnetic trackers use special algorithms to compensate for these distortions by mapping the capture area, but these calibrations only work with static, predefined problem areas such as metallic structures in buildings. In most cases, it is better to avoid any high-conductivity metals near the capture area. This limitation makes the magnetic tracker difficult to transport to different stages or sets.

The latest use of magnetic systems is in consumer video gaming. Sixense Entertainment, a Los Gatos, CA–based company, is developing a wireless gaming controller (Figure 1.10) based on a low-power AC field that will be compatible with computer and console video games. Devices such as these will take the performance gaming trend started by the Nintendo Wii to the next level.

Magnetic trackers in the entertainment industry are used mostly for real-time applications such as live television, live performances, and location-based or Internet virtual reality implementations. Sometimes, they are used as part of puppeteering devices. A performer can author the body motions of a character with the magnetic tracker while someone else is performing the facial expressions and lip syncing using a face tracker or a data glove. At the same time, a puppeteer can be animating the character's eyes using a simple mouse. They are also used to obtain global positioning for other systems, such as inertial motion capture suits.

Electromechanical performance capture suits have been around for a while. They are inside-in systems based on a group of structures linked by potentiometers, gyroscopes, or similar angular measurement devices located at the major human joint locations. The newer versions use MEMS (Microelectromechanical Systems) inertial sensors placed over a Lycra or spandex suit. The idea is that the suit measures all human limb rotations.

Potentiometers are components that have been used for many years in the electronics industry, in applications such as volume controls on old radios. A slider moving along a resistor element in the potentiometer produces a variable voltage-potential reading, depending on what percentage of the total resistance is applied to the input voltage.

MEMS inertial sensors are also angular measurement devices, but they are much smaller and more accurate. They are basically integrated circuits combined with micromachined moving parts. Good examples of devices that use MEMS technology are the Nintendo Wii input device and the iPhone.

One big drawback of performance capture suits is their inability to measure global translations. In most cases, an electromagnetic or ultrasound sensor is added to the configuration to solve this problem, but that subjects the setup to the same disadvantages as the electromagnetic or ultrasonic systems, such as sensitivity to nearby metals or bad accuracy and drift. In addition, the design of most of these devices is based on the assumption that most human bones are connected by simple hinge joints, so they don't account for nonstandard rotations that are common to human joints, such as in the shoulder complex or the lower arm. Of course, this can actually be a benefit if the mechanical setup of a particular digital character calls for such types of constraints.

A good example of a performance capture suit is the Xsens MVN, shown in Figure 1.11. This suit uses inertial sensors based on MEMS technology.

Digital armatures can be classified into two types: (1) keyframing or stop-motion armatures and (2) real-time or puppeteering armatures. Like the mechanical suit, both types consist of a series of rigid modules connected by joints whose rotations are measured by potentiometers or angular sensors. The sensors are usually analog devices, but they are called digital because the resulting readings are converted to digital signals to be processed by the computer system. These armatures are typically modular in order to accommodate different character designs.

Keyframing armatures were initially used to help stop-motion animators animate digital characters; they are not really considered motion capture systems because they are not driven by a live performer. I mention them because most commercially available armatures can also be used as real-time armatures. Some proprietary armatures are dual purpose such as a device initially called the Dinosaur Input Device (DID), which was devised by Craig Hayes at Tippett Studio in Berkeley, California. The name was conceived because this unit was used to animate some of the digital dinosaurs in Jurassic Park. Later, the device was used to animate the bugs in Starship Troopers and it was called the Digital Input Device (Figure 1.12).

The basic concept behind keyframing armatures is that the animator poses the device manually to generate each keyframe in the animation. The character in the animation software is set up with a mechanical structure equivalent to the armature's. By pressing a key, the animator uses the computer to record the armature's pose into the digital character for a particular frame in time. This is done via a driver program that connects the device, typically plugged in to a serial port, to the animation software. Once all the key poses are recorded, the software treats them as regular keyframes that might have been created with the software by itself.

Puppeteering armatures are very similar to keyframing armatures, except the motion is captured in real time as performed by one or more puppeteers. An example of such a setup is the proprietary armature developed by Boss Film Studios to capture the motion of Sil, the alien character that H.R. Giger designed for the film Species.

Digital armatures are not very popular any more. The last commercially available example of a digital armature was the Monkey 2 by Digital Image Design in New York City (Figure 1.13). This unit could be used as both a keyframe and real-time armature. It was modular, so it could be assembled in different joint configurations. The first-generation Monkey had a fixed configuration, which made it unusable for any nonhumanoid applications. The typical cost for a 39-joint Monkey 2 setup was approximately $15,000, which included all the necessary parts as well as driver software for most well-known animation packages.

There are several mechanical and optical devices for capturing facial motion. The most popular are the real-time optical face trackers, consisting of a camera that is placed in a structure attached to the performer's head so that it moves with the performer. The device captures the motion of small markers placed in different areas of the face. Unfortunately, these are 2D devices that cannot capture certain motions such as puckering of the lips, so the data is all in one plane and not very realistic. Three-dimensional facial motion data can be captured with an optical system using two or more cameras, yielding a much better result, but not in real time.

There are also newer optical systems that don't require markers, but are based on optical flow analysis. Such a system was used for the facial performance capture for the film Avatar.

The state of the art in facial motion capture is based on a combination of optical capture and software algorithms. The three major types of modern software algorithms to process facial optical data are muscle/skin simulators, dynamic blend shape solvers, and photogrammetry builders. There can also be combinations of two or three of these techniques. Most such algorithms exist as proprietary developments of visual effects companies, such as the muscle simulation software Sony Pictures Imageworks developed for films like The Polar Express or the dynamic blend shape solver used for Avatar, developed by Weta in New Zealand.

In clinical circles, motion capture is called 3D biological measuring or 3D motion analysis. It is used to generate biomechanical data to be used for gait analysis and several orthopedic applications, such as joint mechanics, analysis of the spine, prosthetic design, and sports medicine. It has always been the largest segment user of the technology, but entertainment uses are growing faster.

There have been a great number of studies of gait performed with patients of all ages and conditions. The first ones were made by Etienne Jules Marey and Eadweard Muybridge in the late 1800s using photographic equipment.

Muybridge's studies started when he was hired by Leland Stanford, governor of California, to study the movement of his race horses in order to prove that all four hooves left the ground simultaneously at a given point during gallop. In 1876, Muybridge succeeded by photographing a galloping horse using 24 cameras (Figure 1.18). He then continued his studies of human and animal motion for many years. His paper on the subject, “Animal Locomotion,” was published in 1884 and is still one of the most complete studies in the area.

A professor of natural history, Etienne Marey used only one camera to study movement, as opposed to the multiple-camera configuration used by Muybridge. Even though they met in 1881, Marey and Muybridge followed separate paths in their research. Studies continued in this fashion for a century, but until the introduction of optical motion capture systems in the 1970s, the research yielded almost no benefits.

Of the current life sciences' applications, gait analysis is very useful because it accurately separates all the different mechanisms that are used during the multiple phases of a walk cycle in a way that makes it easy to detect certain abnormalities and changes. For example, gait analysis helps to measure any degree of change in conditions such as arthritis or strokes. It is also used along with other tests to determine treatment for certain pathological conditions that affect the way we walk, such as cerebral palsy. Rehabilitation by gait training is used for patients with pelvis, knee, or ankle problems.

There are many other potential applications for motion capture in the medical field. Once the current systems' limitations are surpassed, it will be possible to use motion capture to align patients and imagery in operating rooms, to compensate patient movements during MRI sessions, or even to make widely available the current applications that are possible only in special labs. The requirement will be a real-time motion capture system with great accuracy that can function in any environment.

Sports analysis is a major application of motion capture. 3D data is being used extensively to improve the performance of athletes in sports such as golf, tennis, gymnastics, and even swimming by studying the performance of professionals and dissecting and classifying the different components of the movement. As motion capture technology improves to the point at which undisturbed data collection is possible, the potential uses will become even greater in this field.

There are a few motion capture studios across the country dedicated exclusively to the analysis of sports, especially golf. For a few hundred dollars, any golfer can have his or her swing analyzed or compared with the swing of a professional golfer. Visualization software allows the studios to study the athlete's motions to find any problem areas.

The market for sports analysis is mostly amateur sports enthusiasts, but professionals look for help as well. Biovision, a company that used to operate three sports analysis studios across the country, claimed to have helped professional golfer John Daly to find a problem related to opposite movements of his hips and shoulders that happened for one-tenth of a second during his swing. Tiger Woods has done his share of motion-captured golfing for all his PGA video games. His swing has been analyzed and broken down to the smallest component in order to teach others what a perfect swing must look like.

The benefit of using motion capture rather than video for this kind of sports application is that motion capture yields a 3D representation of the motion that can be examined from any possible angle. This is true even if the video could be recorded at higher speed than the normal 30 frames per second. To analyze a fast sports action such as a golf swing or a baseball pitch, it is required to have at least a few hundred frames or samples per second.

The future of motion capture in sports is very bright. Newer systems will be able to carry out sports analysis outside of the lab. All track and field events will be analyzed at the track, its actual venue, and so will all other sports. In addition, the data captured will have many additional uses, such as real-time integration into video games via gaming networks such as X-Box live. It will be possible to watch an auto race from any desired camera in real time rendered on the gaming console and introduce a virtual car to participate in the race remotely.

Broadcasters are always looking for ways to enhance the sports watching experience. Having accurate position data of athletes in any sport is a very attractive prospect for generating eye-catching graphics and previously unavailable statistics. Once a game of football can be captured in real time, we will be able to watch rendered instant replays from any angle. The data will be used as an ongoing biomechanical profile for each player. It could even be used to diagnose injuries in real time.

Figure 1.19 shows how motion capture data could be used to enhance the broadcast of a soccer match. For the application in the image only one captured marker is required per player and that makes this application one of the closest ones to being possible. There are a few systems that analyze video and deliver motion data of such events, but technology is still not up to par for real-time applications. Instead, most current systems are mostly used to calculate estimated speed and total distance covered by each player. In a year or two, this is the way we'll be starting to look at broadcast sports.

The entertainment industry is the fastest growing market segment for motion capture; although not yet the largest, it certainly is the segment with the highest profile. Of the different entertainment applications, the use of motion capture in video games is currently the most widespread and the best accepted and understood. Television and feature film applications of motion capture are still migrating from an experimental market to a viable medium.

Motion capture is applied in law to produce reconstructive videos of events. These videos are used as evidence in trials to aid the jury in understanding a witness's opinion about a particular order of events. According to a study conducted by the American Bar Association, this kind of evidence is much more effective with jurors than any other demonstrative evidence.

An example of an effective use of motion capture in the legal context is the video produced by Failure Analysis to recreate the events of the murder of Nicole Brown Simpson and Ronald Goldman during the O.J. Simpson trial. The video wasn't actually used as evidence in the trial, but it serves as an example of the kind of animation that could be used in a court of law.

To be admissible, the animation must be very plain, without any complex lighting, texture mapping, or detailed models, and, like any other evidence, it must meet the local and federal rules of evidence. In most cases, it has to be accompanied by supporting testimony from the animation creators, during either trial or deposition. Many animation companies specialize in all aspects of evidence animation, from its preparation to the presentation in court.

Motion capture has been used in the military for many years, mainly in tracking helmets for pilots and other operators of combat vehicles. Tracking the line of sight of pilots has proven to be a great way to help aim weapons and to receive critical visual information.

Other uses in this field include troop training aid through simulation. Currently, it is mostly used for production, but eventually will be widely used to acquire exact troop position in the virtual or real training field. Positioning of troops and equipment in large area exercises will allow the instructors to evaluate everything that happens in the exercise by looking at a large real-time computer-generated visualization. Troops then will be debriefed using the 3D created simulation, so they can learn where to improve.

In secured installations or amusement parks, personnel and visitors can carry a tag at all times, allowing the tracking of all their movements in real time. Tagging will allow the system to provide entry privileges to certain areas and will also keep a history of personnel and visitor flow in the facility. Certain items and equipment can also be tagged in order to keep real-time account of its position at all times. There are already some GPS and RTLS versions of this application. Legoland in Denmark has had an RTLS child-tracking system for several years now. For child safety at the mall or anywhere else, parents can track their children by having them carry a mobile phone that they can track online.

Granted this is not performance capture, but it is capture of motion and it is relevant because it potentially is a much larger market than any other that exists today, even in entertainment. In fact, asset tracking is already a fast-growing market in hospitals and other facilities using RTLS.

Motion capture is used in the engineering process for design, safety, and ergonomic testing. Through simulated human interaction with engineering models, engineers determine the functionality of their design prior to building expensive prototypes. Also, through better enabling augmented reality technology, motion capture will open the door to more efficient work flow and training methods.

Ford Motor Company has developed a system called Human Occupant Package Simulator (HOPS). The system is basically a large database of captured movements of drivers and passengers inside vehicles. Ford designers use digital humans driven by these motion datasets inside virtual vehicle designs to analyze their interactions with the vehicles. This helps them improve the ergonomics of their designs as much as possible before having to build models and prototypes.