PDI had provided a CG Batman stuntman for Batman Forever, using optical motion capture as a means to collect the performance of a gymnast performing on a stage doing flips and other movements on rings. Batman and Robin required much more elaborate digital stunts, many of them happening in the air. PDI was again contracted to create these stunts using motion capture.

The motion capture services were provided by Acclaim Entertainment, the computer game company that had the video game rights for the Batman property and that had earlier provided the motion capture services for Batman Forever. They used their proprietary four-camera optical system to capture all the necessary motions for the stunt sequences.

Acclaim's team captured several of the shots at Acclaim's motion capture stage in New York, except for the shots that belonged to the sequence in which Batman and Robin skyboard away from an exploding rocket while in pursuit of Mr. Freeze (played by Arnold Schwarzenegger), trying to recover a diamond he had stolen. Because all the action in this sequence happened in the air, John Dykstra, visual effects supervisor, decided to bring Acclaim's optical motion capture equipment to a vertical wind tunnel used for military training. The tunnel was located at the Fort Bragg military base in North Carolina, and the performers were not actors, but members of the US Army's Golden Knights parachuting team. Dykstra directed the performances in the tunnel, and Acclaim processed the data and delivered it to PDI.

“We took the skeleton files from Acclaim and converted them to a setup,” explains Richard Chuang, who was the digital effects supervisor for PDI. “We also converted all the motion data into our own animation space.” Dick Walsh, character technical director at PDI, had set up a system that allowed them to adjust the differences in physical attributes between the digital model and the performer (Figure 2.1). “We actually had a certain flexibility with the use of motion capture data from different performers on the same character,” recalls Chuang. The data was then used to move a skeleton that would drive PDI's muscle system, also written by Walsh. In addition, animators had the ability to blend the captured data with keyframe animation, with or without inverse kinematics. The system didn't require animators to modify the motion data. They worked on clean curves and were able to blend the animation and the captured data using animatable ratios. The system was especially useful when blending between captured motion cycles.

“In both cases we ended up using probably around 20% motion capture and 80% animation,” notes Chuang, referring to both Batman films. “What we found is that because the motion is always captured in a stage, it really has nothing to do with the final performance you need for the film. For the director to have control of the final creative, we ended up taking motion capture as a starting point, and then animating on top of it. Then we'd be able to dial between motion capture and animation whenever we needed to.”

Part of the problem with motion capture in this case was that the motions were for the main action of the sequence and the director wasn't present to direct the performance. When you capture motion for background characters, it is okay for the director to delegate the task, but in a case such as this, the director would expect to have first-hand creative control over the final performance. That meant the performance would have to be modified by the PDI team to his specification. Larry Bafia, animation director at PDI's commercial and film effects division, worked on the air sequence. “It was the first time that I ever had to deal with motion capture data at all, so I had no idea what I was getting into at the time,” recalls Bafia. “This is a character that comes from a comic book and it really has to look like a superhero,” he adds. “Some of the poses had to be exaggerated and it wasn't something that a human could do in a wind tunnel.” The team kept most of the captured data for the motions of the character's torso, especially the actions of weight shifting on the skyboard. They used keyframe animation on many shots to enhance the pose of the arms and to animate the fingers, which were not included in the motion data. Inverse kinematics was used mostly on the legs.

The motion capture session was directed by the special effects supervisor, John Dykstra, who tried to capture as many different shots as possible for the director to choose from. This was done even before background plates were available, so no final camera angles and framing were available. “What you do is you capture all the possibilities, and then based on that you try to edit something that will fit your film,” says Chuang. “But by the time you get your background plates you can end up with something completely different.”

PDI technical directors prepared desktop movies of all the captured motions so that, after the plates became available, the team could decide which actions would apply for each particular shot. “One of the problems was the fact that a lot of the shots were long enough where the particular motion capture cycle didn't have quite enough activity,” recalls Bafia. Because of this, the team had to blend several captured motions into single shots. An example of this is the shot in which Robin catches the diamond after Mr. Freeze loses it. “I had to start the front end of the shot with one piece of motion capture where [Robin] is basically staying alive and balancing himself on the board and getting ready for spotting this diamond and catching it,” recalls Bafia. After catching the diamond, Robin had to do a spin-around, not unlike that of a surfer. “Because of the restrictions inside the wind tunnel, the guy could do the spin but he had to pull up early so that he wouldn't fall too far and drop down toward the fan that was actually suspending him,” says Bafia. PDI had to use inverse kinematics to maintain the character's feet on the board, and keyframe animation to blend between two motion files that had a reversed feet stance—all this while keeping the elements in the captured data that John Dykstra was interested in maintaining.

The fact that the performers weren't aware of where the final camera would be located was a problem that translated into the performance. “I would like to see a situation where you actually try motion capture after you shoot the film versus before, so you know where the background plate is going to be, and the director directs the action accordingly,” notes Chuang.

Regardless of the expected problems with captured performances, the resulting shots were extremely successful. PDI had the experience to use whatever they could out of the motion data and discard the rest. Therefore, the project was completed in a timely manner, the clients were happy, and PDI made a good profit. The difference between this and other projects using motion capture was that for Batman and Robin, motion capture was used to achieve a certain look, not to save money.

Before the original Shrek animated film started production, DreamWorks Feature Animation had been developing it to be mostly based on performance animation.

The decision to use performance animation on Shrek was based not only on cost (which indeed played a big part in the decision), but also on the desire to achieve a certain look. Before Shrek was even in preproduction, PropellerHead Design (Rob Letterman, Jeffrey Abrams, Andy Waisler, and Loren Soman) produced a test of a similarly shaped character called “Harry” for a potential television show for HBO. The motion capture for the test was done at TSi.

For the Harry test, the propellerheads (as they were known in the Shrek production) prepared an appliance that was worn by their performer. The appliance would add the volume that the performer needed to approach the proportions of Harry. This was a new idea that we had never seen done before, and it seemed like a good one. During our first of two sessions, we placed the markers on the appliance and captured the body motions with our optical system. The tracking of the data was a challenging feat because it was an uncommon setup, but we managed to get it done with our ancient six camera system.

In the second session, we were supposed to capture Harry's finger motion. We had never tried to capture the motion of fingers with an optical system, and we didn't guarantee that it would work. It was strictly a developmental session. We placed the small markers that we usually used on face capture on the finger joints. The performer sat down in a chair, and we immobilized his hands as much as possible because the wrist movement had already been captured in the previous session and we knew we couldn't capture the fingers if the wrists were moving. We captured several passes of the actor moving his fingers with the timing of the dialogue, but in the end it was impossible to track those motions, and we had to scratch the idea of optical finger capture. Facial capture wasn't even tested for this project.

The propellerheads finished the Harry test, filling in the additional animation, plus texture and lighting. The end result was stunning. The Harry test laid the foundation for the production of Shrek at DreamWorks. The propellerheads would coproduce the film, which would be animated using techniques similar to those in the Harry test: optical motion capture with body extensions or appliances.

The DreamWorks studio in Glendale was geared up with Silicon Graphics computers running different animation platforms and a brand new 10-camera Motion Analysis optical motion capture system. A production staff was assembled and production of a 30-s test began. Character animators started to work with the optical motion data. Several months later, the test was shown to Jeffrey Katzenberg, DreamWorks' cofounder and the partner responsible for animation films. Shortly thereafter, the production was halted and the project went back into development. The propellerheads were no longer involved. Exactly what happened is still a mystery, but motion capture was ruled out for this project. Among other things, the production made a mistake by using experienced character animators to work with the captured motion data, even from the stage of retargeting. It is not certain if that actually affected the test, but it would have eventually become a big problem.

Another character that was supposed to be animated by performance was Godzilla, in the 1998 film with the same name. Future Light had developed a real-time motion capture system based on optical hardware by Northern Digital. The system was connected to a Silicon Graphics Onyx with a Reality Engine, and it could render an animated creature in real time. Future Light was a division of Santa Monica Studios and a sister company of Vision Art, one of the CG studios involved in the visual effects production for Godzilla. Director Roland Emmerich liked the system because he could see the creature motion in real time, so the production proceeded with the idea of using the system as much as possible for the animation of the creature.

The character design was proportionally close to the human body, so it was possible to map human motion into the character's skeleton without much retargeting. According to Karen Goulekas, associate visual effects supervisor, the motion capture worked technically, but it didn't yield the results they were looking for esthetically, so they decided to abort that strategy and use keyframe animation instead. In the end, the animation for all the creatures, including Godzilla and its offspring, was done with keyframe animation.

“The reason we pulled the plug on using the motion capture was, very simply, because the motion we captured from the human actor could not give us the lizard-like motion we were seeking. The mocap could also not reflect the huge mass of Godzilla either,” adds Goulekas. “During our keyframe tests, we found that the Godzilla motion we wanted was one that maintained the sense of huge mass and weight, while still moving in a graceful and agile manner. No human actor could give us this result.” In addition, they had Godzilla running at speeds as high as 200 mph, with huge strides; this proved to be impossible to capture.

In general, using human motion to portray a creature like Godzilla, even if it is proportionally close to a human, is a mistake. In previous Godzilla films, it was always a guy in a suit performing the creature. In this version as initially conceived, it would also be a guy in a suit, except it would be a digital suit.

Sinbad: Beyond the Veil of Mists was the first full-length CG feature that used mostly performance animation. The film was produced by Improvision, a Los Angeles production company, and Pentafour Software and Exports, one of India's largest and most successful software companies.

The story concerns an evil magician named Baracca, who switches places with a king by giving him a potion. The king's daughter, who is the only one who knows that the switch took place, escapes, eventually running into Sinbad. The Veil of Mists refers to a mythical place where Sinbad and the princess have to go to find the answer to defeating Baracca and saving the king. The main character, Sinbad, was voiced by Brendan Fraser. Additional cast included Leonard Nimoy, John Rhys-Davies, Jennifer Hale, and Mark Hamill.

With an initial budget of $7 million, it is no secret that motion capture was adopted by the producers primarily as a cost-effective alternative to character animation. They had originally planned to finish the whole film in 6 months, thinking that using motion capture would be a huge timesaver. Evan Ricks, codirector of the picture, was approached by the producers with a rough script. “They already knew that they wanted to do motion capture,” recalls Ricks, who is a cofounder of Rezn8, a well-known CG production and design studio in Los Angeles. He told the producers that a film of this magnitude would probably take at least 2 years of production, but they insisted on trying to finish it on schedule.

Given the constraint that the production would have to utilize motion capture, Ricks initially explored the possibility of using digital puppeteering to animate the characters in real time, using that as a basis for animators to work on. After conducting a few tests, they found that there weren't enough off-the-shelf software tools available to produce this kind of project in real time, so they decided to use an optical motion capture setup.

Motion capture would be used for human characters' body motions only. They couldn't find a good solution at that time to capture facial gestures optically, so they tried to use digital puppeteering instead. Finally, they decided that all facial animation would be achieved by shape interpolation. Pentafour had acquired a Motion Analysis optical motion capture system that was sitting for some time unused in India, but the production decided to use the services of the California-based House of Moves and their Vicon 370E optical system. The reason was that in Hollywood there were more resources and expertise readily available for the shoot, such as stages, actors, set builders, and motion capture specialists.

The production rented a large sound stage at Raleigh Studios and started building all the props and sets that the actors would interact with during the capture session. They even built a ship that sat on a gimbal that could be rocked to simulate the motion of the ocean. “That introduced all kinds of problems in itself,” notes Ricks, “because all of a sudden you don't know exactly where the ground is and you have to subtract that all out.”

In addition to the people from House of Moves, the production had the services of Demian Gordon, who was brought in as motion capture manager. Gordon had worked extensively with motion capture before when he was at EA Sports. His job was to help break down the script, work closely with House of Moves, help design marker setups for all characters, and find solutions to problems that arose during production.

The House of Moves' Vicon system was brought into the sound stage and installed using a special rig that rose about 30 ft off the floor, which kept all seven cameras away from the action on the stage. In addition, special outfits were made for the performers to wear during the sessions.

Ricks wanted to shoot the motion capture sessions on tape from many angles as if it were a live television show. He was planning to edit a story reel after the motion capture was done and to use it while producing the CG images. A camera crew was hired initially, but was eventually dismissed by the production company because of its high cost. “We paid the price later,” says Ricks. “The camera can really help the director to visualize what's going to be on the screen. I guess you can stand by and use a finder, but with live action, the very next morning you can see dailies. You can see what you actually shot and if the angle works. With motion capture, you get a single reference and end up with mechanics more than cinema.”

Even if they had been able to tape all the captured shots, they had to deal with the motion capture system's inability to synchronize video and capture scan rates. They couldn't use SMPTE time code, because at that time no optical system was able to generate it, although it is a very important aspect of production work. According to Ricks, the lack of a good way to synchronize all the elements played a big part in the extension of the production schedule.

Because of the compressed schedule, the crew had to capture thousands of shots in about 2 months. According to Ricks, 4 months would have been ideal. “We weren't able to finish all the storyboarding before we started the capture, so sometimes we would be literally looking over the editor's shoulder, telling him what we wanted, or looking at new storyboards that had just been drawn, and then go over to the stage around the corner and capture it. Not the ideal way to work.”

Typical problems during the motion capture sessions included performers losing or relocating markers. On projects of this magnitude, it becomes very difficult to keep track of exact marker placements, which creates inconsistencies in the motion data. They also had to deal with obstruction problems, where cameras didn't have a direct line of sight for all markers.

Many of the shots had more than one performer interacting at the same time. They were able to capture up to four characters simultaneously, although it became somewhat problematic to clean up the data. “A lot of it took place aboard ship, and there're six people on the crew, so there's a lot of interactions,” says Ricks. Fighting sequences were especially difficult to capture, particularly when somebody would fall on the floor and lose markers.

The production was set up so that all design work, preproduction, initial modeling of characters, and motion capture work were done in the United States. The animation side, including the application of captured data, modeling of sets and secondary characters, shading, and rendering, would be handled by the Pentafour CG studio in India, which was a relatively new studio whose experience was mainly in the production of art for CD-ROM titles. Ricks wanted to use very experienced people to handle the most challenging parts of the project, creatively and technically, leaving the supporting tasks to the inexperienced crew.

Overall, the first 6 months of production had been extremely productive. Evan Ricks and Alan Jacobs had managed to rewrite the script practically from scratch, while Ricks and the production designer developed the look. They hired the staff and cast, recorded the voice track, storyboarded, built sets to be used for motion capture, tested motion capture alternatives, and designed and built all the digital characters and sets. They even stopped regular production for over a month to create a teaser for the Cannes Film Festival. All the motion capture was also finished within those first 6 months. Everything was looking great and the staff was energized. “I was continually told by visitors and investors how beautiful our designs were, and how new and different as opposed to a ‘Disney’ look,” says Ricks.

Unfortunately, after the completion of the motion capture, the producers had a mistaken perception of the work left and didn't understand what had been achieved in those 6 months. In their view, most of the technical hurdles in the production had been surpassed. Not only did they not see the need to fulfill repeated requests to hire additional experienced personnel, but they also started laying off their most experienced staff members. Finally, the remainder of the production was moved to the Pentafour studio in India.

The difficult part started after House of Moves finished postprocessing and delivered the data. “There was a serious underestimation in how long it takes to weight the characters properly,” notes Ricks, who describes some bizarre problems with the deformations of the characters. The deformation setup was much more time consuming than they had originally estimated. The problem was augmented by the fact that they relied exclusively on off-the-shelf software, and there's a lack of commercial tools for handling captured motion data. A project of this scale would normally have some sort of in-house R&D support, but the budget didn't provide for it.

The mechanical setups for the characters were created using rotational data. For a large project like this, with so many characters and interactions, it would have been beneficial to create customized setups using also the translational data, using markers as goals for inverse kinematics or as blended constraints. When using rotational data, the mechanical setup has already been done by the motion capture studio, and not necessarily in the best possible way for the project at hand. “[House of Moves] hadn't done tons of this stuff at that point either, and they had to staff up. It was a learning experience for everybody,” notes Ricks, “but they did a very good job under the circumstances.”

A typical problem with captured motion data is in the unmatched interactions between characters and props or other characters. The crew in India has been responsible for cleaning up all these problems, including the inability to lock the characters' feet on the floor. This particular problem can be fixed by using inverse kinematics, but it can also be hidden by framing the shots in such a way that the characters' contact with the floor is not visible. Ricks was trying not to let this problem rule the framing of shots.

As Ricks predicted, the length of the production of the film was close to 2 years. “If you deconstructed just about any recent film, you'd find out what the director originally wanted and what actually took place are very often two different things, and that was no different here,” says Ricks.

“The key is to use this experience to identify the strengths of mocap, to realize that its value is currently not as broad as many think. It is a new camera, not an end in itself,” concludes Ricks.

Robert Zemeckis had teamed up with Tom Hanks to bring the famous children's book The Polar Express to the screen. The original idea was to give the film the same painterly look as Chris Van Allsburg's book. The methodology to achieve the look would be to use live-action characters shot in front of green screen in order to place them in digitally generated backgrounds. Once the elements were integrated, some kind of filter would be applied to give it the look.

A day was scheduled to bring Tom Hanks to a soundstage at Sony Pictures to do a green screen test. Ken Ralston and Scott Stokdyk were the visual effects supervisors for the test. I was helping Scott with the preparations and we thought it would be interesting to try to capture Tom's face and see how accurate looking we could get it. One must remember that while there had been some interesting facial motion capture examples at the time, there still wasn't a believable side-by-side test of the face of a known person, so we weren't exactly sure what the result would look like.

We had Vicon bring a system to the soundstage for the day of the test. Once Tom was there, we spent most of the time shooting the green screen. Afterward, we were given a few minutes to do our thing. We scanned Tom's head with a laser scanner and slapped about 50 markers in his face on places where I thought would give us the movement we needed. We then had Tom say a few lines in front of the cameras. Nobody but Scott and I had any hope that this test would turn into something. Ken said to me that he was sure it wasn't going to succeed and Zemeckis didn't even pay attention to us when we were doing it. The day after the test almost everyone had forgotten we captured anything. I was the only person working on the motion capture test. Everyone else in the crew was working on the green screen test.

After several days, we started seeing the first green screen tests in dailies. They weren't looking good at all. The static elements in the shot were fine with the painterly look, but the live-action actors looked like they had a skin disease.

In the meantime, I was working away on the mocap test. I had already written a facial muscle-based solver that was used on Stuart Little II for talking cats and I was modifying it to work with captured data. Once I set up the system with Tom's facial model and started driving it with the data we captured, I was happily surprised. The test actually looked like Tom and it was moving the right way. It clearly required more data but you could see Tom in there.

After several weeks, the painterly look tests weren't looking any better. I hadn't been asked to show anything as most people had forgotten or assumed what I was doing didn't work. When I finally showed it in dailies, Ken Ralston looked at it and said “this may actually work. Let's show it to Bob.” Once Zemeckis saw it, he decided to pull the plug on the green screen test and use mocap for The Polar Express. I wasn't expecting that reaction. I still didn't know how extrapolate from that test into a whole 90-min film with lots of characters interacting with each other. This was a scary time, but it got worse when I was told that I had to have the ability to capture face and body together and we'd start shooting in about 5 months.

Producer Jody Echegaray and I had to start by choosing the right equipment and assembling the team. We started talking to the different equipment manufacturers and also interviewing all the possible candidates. We knew that we pretty much had to hire almost anyone who had any experience with motion capture to cover the scope of this project. Our first hire was Demian Gordon, who would be our motion capture supervisor. He had just finished working on the Matrix sequels and was able to bring with him a lot of good people to fill many of the slots in our team. Part of his expertise was also the design of camera layouts for capture stages, so I asked him to calculate how many cameras would be needed to meet the spec that we were asked to deliver. He estimated that we needed no less than 70 to 80 cameras to cover the face and body markers of up to four people at a time.

None of the existing manufacturers had the capabilities we needed, so we started to talk to them about working with us to daisy chain several of their systems together. As we told some of them what we were thinking, most decided to pass and at least one of them told us we didn't know what we were doing. The only manufacturer willing to work with us was Vicon, and between them and us we started designing the final layout, calibration methodology, and pipeline to achieve the final spec. Brian Nilles, Vicon's CEO, was instrumental in having his engineering team work on their software to allow us to hook up five Vicon systems together. All of us were on constant communication and as soon as they could ship some equipment we started to assemble a massive truss for placing cameras inside a soundstage at the Culver Studios. At the beginning, the assembled system was very delicate and just the wrong setting would make it malfunction. There was also the issue of calibration, as there was no known way to do this with a daisy-chained system. Initially, we had to calibrate five systems separately and then merge the calibrations, but eventually Vicon provided us with software to merge them on the fly. The problem was that the calibration had to be seen by all cameras and apparently in some particular order. We had to come up with a “dance” that D.J. Hauck, our head mocap operator, had to perform perfectly every time in order for the calibration to take. In the next couple of months, we came up with protocols and procedures for everything that could happen and started educating the production team on what to expect and what was possible and what wasn't.

We ended up with three different mocap stages: the main stage, where body and facial data would be captured, and two larger stages, where only body data would be captured. The main stage ended up with 84 cameras; 70 of them were Vicon cameras and would be used strictly for face capture. The rest of the cameras were operated by Giant Studio and would be used for body capture. Giant Studio was also responsible for the operations of the two larger sets, one of which was 36,000 ft³ (30 × 60 × 20 ft), the largest ever done at the time and used for large captures with many performers. The other large stage was 27,000 ft³ (30 × 30 × 30 ft) and was used mostly for stunt action because it had more height. The main stage was only 1300 ft³, which is 10 × 10 × 13 ft high.

We felt the stage was ready to shoot only about a week before the actual production would start. We held a massive meeting with all the different groups that would be on set and with the director and we went step by step over what was to happen on set. I also had to answer hundreds of questions from people who had never even heard about motion capture and were about to be dropped in the middle of the largest motion capture effort ever done. After that day, we started training the different groups about their new tasks. The makeup artists' job was to apply face markers and also to make sure that the marker sets were always complete during the shoot. Wardrobe people were in charge of mocap suits and body markers. Grips were in charge of bringing in and out of stage props that had to be built mostly with chicken wire. Also, the first assistant director and production assistants had to be educated about their new roles to get the stage ready for shooting.

I have to say that the crew for that shoot was ideal. Zemeckis understood immediately what he could and couldn't do. The makeup crew, led by Dan Striepeke, had to place 152 markers on up to 24 faces every day before shooting could begin, and they learned to do it very quickly and very accurately. Demian Gordon, our stage supervisor, had no problem stopping the shoot if any problem arose. He was never intimidated by Zemeckis, Hanks, or anyone else and because of that we never had any data surprises. We shot for 45 days without a major problem and everyone enjoyed the process. I heard Zemeckis say that he would never shoot a live-action movie again. He was so excited that he brought lots of people to see the stage. Among the people I saw, there were several studio heads and famous actors and directors like Steven Spielberg, James Cameron, and others.

An important aspect of the way Zemeckis decided to shoot his film is that he didn't want to frame shots on set or even look at real-time graphics of the performances. The small set was designed as an intimate setting where he could work with the actors. Not many people were allowed to stand inside the set.

When the shoot was done, we had to move into postproduction. We set up three departments to deal with the captured data. First was the Rough Integration Group, whose job was to create an assembled sequence from all the different pieces captured in different passes and stages. This sequence was played in real time and using a virtual camera setup the director of photography would design the shots. During The Polar Express we called that process Wheels, and it was specifically designed for Zemeckis' style of filmmaking.

The second group was the data-tracking group, whose responsibility was to track and deliver facial data from the shots created and used by editorial. The third group, called “Final Integration,” would bring each shot into Maya and integrate body and face data with the camera created by the DP. The rest of the pipeline was more typical of an animated feature. Shots went on to animation, layout, cloth, effects, color and lighting, and finally compositing.

“[The] Polar Express was in terms of motion capture probably the most groundbreaking, not necessarily on the imaging side, but because of the technology that had to be put together. Nobody had actually done body and face together at that point,” says Jerome Chen, visual effects supervisor.

Robert Zemeckisv second directorial effort in the performance capture arena was Beowulf. His initial goal was, as with The Polar Express, to create stylized environments and characters, but again the project evolved into the almost-photo-real realm. “Zemeckis wants to see realistic textures, realistic aspects to the environment and to the characters,” recalls Jerome Chen. “He doesn't gravitate toward the stylized designs; he likes it to be realistic. … Once again you are starting out trying to be stylized and ending up wanting to see more detail in the characters once you see realistic motion on them.”

Beowulf's pipeline was almost identical to the one used for The Polar Express. The main difference was in the data acquisition. Imageworks upgraded to Vicon's latest hardware included cameras with much higher resolution and the ability to accommodate many more of them, over 250 to be more specific. The volume was bigger so all the capture was done in a single stage. Also, the crew had a lot more experience from having done The Polar Express and Monster House. “A lot of the protocols established in the stage capture were now perfected,” says Chen. “We budgeted more time in Beowulf to do facial animation, particularly preserving volume in the face and on the lips.” The facial system in Beowulf was different from the one used on The Polar Express. It was mostly driven by FACS (Facial Action Coding System, described in Chapter 1), where actors would have a number of predetermined expressions and an algorithm would decide the optimal combination of expressions for each frame. They also paid a lot of extra attention to the eyes.

The Wheels process, now called Camera Layout, was still a major piece in the postproduction process as all the shots were designed there. For Zemeckis' style of filmmaking, that process worked very well because he is a big fan of complex camera work. Camera Layout allowed him to experiment and tweak the camera angles as many times as he wanted.

The main difference between the production pipelines of The Polar Express/Beowulf and Avatar is in the way the camera work is designed. Zemeckis likes to have a director of photography design the shots well after the actual capture, while Cameron likes to design his cameras while he's capturing the performances. For that purpose, he used a virtual camera he called “Simulcam” that was captured at the same time and in the same way as the performers. It was very important to have real-time feedback in order to achieve this and Giant Studio was able to provide it.

Avatar had been in the works for many years but it didn't start production until 2005. “Avatar wasn't done before because the technology didn't exist to do the movie before,” says Jon Landau, producer. “People get confused about what the technology was. They think it's the 3D. It has nothing to do with the 3D. It was the facial performance of the CG characters. We knew that for this movie to work the characters needed to be emotional and engaging, and that wasn't there.”

The second large difference between Avatar and its predecessors was the fact that the facial data acquisition was done using a helmet-mounted camera and no facial markers. Image-based analysis was used to obtain the facial sampling. “Over the years, we did some testing ourselves in another project called Brother Termite, where we did some image based facial capture. That proved to us that there was some validity there. Then in seeing things like Gollum in Lord of the Rings and what other people were doing we felt in 2005 that if we pushed the technology that we could be the impetus to get it to where we needed it to go,” says Jon.

I asked Jon about his take on motion capture and what was their intention when they decided to use it for Avatar. “We don't call it motion capture, we call it performance capture. We wanted the creative choices of performance to be made by actors and not made by animators, and that takes nothing away from great animators, because the animators are a very important step in the process, but the decisions that are made of when to stare and not blink and when to twitch—those are made by actors because that's what they do. The animators were responsible for showing that performance came through in the CG characters and that is a huge skill into itself. Performance capture is not a money saver to us, but a performance saver. If an actor gets that great performance, they have to only do it once. In fact it's a pure way of acting. Sigourney Weaver, on her first day on the set I said to her, so what do you think, and she said to me, It's easier than working on a theatre stage, because on a theatre stage you have to remember to play out to the audience, you can't always play off of your actors. Here I can play off of my actors and I can move and do what I want. It's liberating and I know that what I do, you will realize,” he adds.

The Polar Express, Monster House, and Beowulf all had performers wear facial markers. It's just a matter of taste, as we did give Zemeckis the choice of having a helmet with cameras. He thought actors wouldn't like it and it would hurt the facial performance. The drawback of not using helmets is in the capture volume size, as cameras surrounding it need to be able to see a'll the facial markers. Zemeckis was happy with this compromise because he wanted a small, intimate space where he could interact with the actors better. Cameron's style is different in the sense that he likes to edit his film as he's shooting it, in fact, he would stop shooting for days or weeks to edit what was shot and then he would continue shooting again. That is why cameras had to be already done, and he needed to have body and facial real-time feedback as well in order to build his cut so early in the process.

“We stand on the shoulders of those that came before us. We ride on top of what [The] Polar Express did and Polar was an incredible first step in this arena,” he concludes.