Figure 11.1 © Paolo Margari (www.flickr.com/paolomargari).
Chapter 9
Fixing Compositional Problems
“The more I get of you, the stranger it feels.”
—Seal
One of the aspects of digital postproduction that seem to get producers the most excited (particularly the ones with a significant background in working with film) is its ability to quickly and easily recompose shots. A medium shot can potentially become a close-up, widescreen footage can be formatted for narrower displays, and images can be blown up from SD video to 35 mm resolution. And all this for far less expense than reshooting.
The reality, of course, is not quite this straightforward. Although digital technology makes all of these things easy, there is still a price to be paid —almost all of these processes can diminish the inherent quality of the original in some way.
The Crop
Sometimes you need to adjust footage to fit a particular frame. For instance, you may need to add high-definition material to a narrower standard definition production. You have two options: you can squash the HD image to fit the SD frame, which will distort it (all your actors will look taller and thinner), or you can crop it, cutting some of the original composition.
What becomes important at this point is the region of interest, that is, the part of the image you want to keep. Most of the time, cropping footage will keep the image centered by default (performing what's known as a center cut-out). Most of the time that produces the desired result, but bear in mind that you can usually combine the crop with repositioning the image, for instance, in the case of cropping HD to fit an SD frame, keeping the top part of the frame and losing the bottom instead.
Figure 9.2 A frame of HD video. © Andrew Francis (www.xlargeworks.com).
Figure 9.3 The frame squashed to fit an SD frame.
Figure 9.4 The frame cropped to fit an SD frame.
Figure 9.5 The frame cropped and repositioned to fit an SD frame.
The Letterbox
Crops are useful for a number of situations, but sometimes it's more important to retain the complete, original image, rather than filling the frame. For example, if you need to make an SD reference tape that contains HD elements, you'd better be sure that the whole image is there in case anything important is happening at the sides. In this case, what you need is a letterbox, not a crop.
A letterbox retains the original shape of the footage, typically filling the rest of the frame with black. On the plus side, you get the complete composition, but on the minus side, you get a lot of wasted space in the frame (a letterbox usually implies a reduction in resolution as well, and so will be less detailed than the original).
Figure 9.6 A frame of HD video letterboxed to fit an SD frame. © Andrew Francis (www.xlargeworks.com).
Digital Scaling
There are many situations in postproduction that require the picture size of a shot to be changed. Unfortunately, the process of resizing digital images is far from perfect. Intuitively, it should be a very simple process, like using a zoom lens on a camera to get closer to the action, or even like using an enlarger in a photographic darkroom to make the picture fit a bigger page.
Figure 9.7 © Timothy Lloyd (www.timlloydphoto.com).
Both of these examples use an optical process that magnifies details that were previously too small to see clearly. With digital images, however, you're limited to what's already there. And because what's already there is just pixels, tiny colored square boxes, when you enlarge a digital image too far, you get … big colored square boxes.
Having said that, this side effect goes unnoticed most of the time. The main reason you don't see scaled-up pixels in digital images very often is that almost every image processing application automatically performs interpolation on the raw data to produce the resized result. This uses any of a variety of algorithms to create detail between the pixels in the original. Very often, this process works well, generating passable results. However, it can also lead to a variety of artifacts and should therefore be avoided whenever it's not needed.
TIP
Refer to Chapter 6 for techniques for removing many of the artifacts that can be caused by digital scaling, such as aliasing.
Figure 9.8 Close-up of pixels after resizing using nearest neighbor interpolation.
Figure 9.9 Close-up of pixels after resizing using bilinear interpolation.
Figure 9.10 Close-up of pixels after resizing using Lanczos interpolation.
How to Use High-Quality Scaling
Digital photo editing applications often provide more options for scaling footage than video editors.
Most video processing applications are built for speed. There's a trade off between an application's responsiveness and interactivity and the complexity of the processing it can do. It's acceptable to use scaling algorithms on digital video footage, provided that footage will still play back in real time.
Many of these applications, as a result, don't provide much in the way of choice of algorithms for scaling footage. This is a shame because there are a large number of digital image processing applications that provide a great number of different options, each yielding different results. This technique will allow you to leverage the flexibility of image processing applications for digital video footage.
1. Load the footage.
2. Render it as a series of still images at its native resolution (refer to Appendix 1 for more information on this process). Make sure the images are numbered sequentially.
3. Open one of the images in your image processing application.
4. Resize the image to the desired size, trying different methods until you find one that provides the best result.
5. Batch process the remaining images using these settings, again ensuring that the resized files are numbered sequentially.
6. Load the new image sequence into a timeline.
7. Render out the sequence to the desired format.
Figure 9.11
TIP
The problems associated with scaling (as well as the potential solutions) apply equally whether you are enlarging or reducing footage. Using different algorithms can produce varying results when making footage smaller, with some producing smoother results and others producing sharper results. Be aware that an algorithm that works very well when scaling footage up will not necessarily be suitable for scaling the same footage down.
Aspect Ratio
The aspect ratio of an image is the ratio of its width to its height. High-definition video has an aspect ratio of 16:9 (which may also be written as a decimal: 1.78), meaning that for every 16 units of width (be it measured in pixels, centimeters, whatever), the height will be 9 units. Standard definition video, on the other hand, usually has an aspect ratio of 4:3 (1.33), although there are wide-screen variants of SD that can be 16:9, just like HD.
With film, it gets even more complicated, as the aspect ratio will depend on the gate used during the shoot. This means that the aspect ratio of a strip of film footage might be 1.66, 1.85, or even 2.35. On a given production, it's not uncommon to find footage that has been shot with different aspect ratios for inclusion in the final cut.
With very few exceptions, the aspect ratio of any output format must be consistent. Regardless of how the material was shot, it must be formatted for 16:9 for HD video and usually for either 1.85 or 2.35 for theatrical release. Most of the time, any discrepancies can be resolved simply by cropping the footage to fit. That's fine when working with formats that have a lot of detail, such as 35mm film, but cropping SD video to make it 16:9 can quickly reduce it to something close to VHS quality.
Figure 9.13 Different aspect ratios.
Safe Areas
Safe areas are a legacy of old television sets. It was difficult to manufacture televisions that would ensure that the displayed picture would fit exactly within the frame. So many manufacturers overcompensated, with the result that part of the picture was outside of the frame and therefore not visible (but at least their customers didn't complain about part of their screens being blank).
In turn, producers found that the edges of their pictures were being cropped on a large percentage of the audience's televisions. So anything happening at the edge of a frame may or may not have been visible, which was especially problematic for necessary items like text. The solution, therefore, was to define the safe regions, where elements would be guaranteed to be visible regardless of the type of display. Unfortunately, these safeguards are still enforced in the digital era, and therefore require careful checking before output.
For visible action, the safe area is defined as 3.5% of the width and height from the edge, and for titles, 5%. Refer to Appendix 2 for more specifics.
Figure 9.14 Safe areas within an HD frame. Earth © BBC Worldwide Ltd 2007.
How to Turn HD into SD
Changing HD into SD requires more than a simple resize.
The following technique assumes you want to fill the SD frame (otherwise, you should just letterbox the footage to retain the entire frame).
1. Load the footage.
a. Ensure that the aspect ratio is preserved. If your application doesn't do this automatically, calculate the new width by multiplying the new height by 16 and then dividing by 9.
b. Ideally, you should use a system that doesn't commit the resizing operation until you render it out, to allow for interactive experimentation.
2. Crop the footage left and right to match the output width. Apply the crop evenly to the left and right to get a center cut-out.
a. If you don't want to use a center cut-out, reposition the frame horizontally before cropping.
b. For a more advanced application of this process, set key frames to dynamically adjust the position and scaling of the frame according to the desired point of interest in the shot (this is known as a pan-and-scan).
c. Make sure you pay attention to action-safe and title-safe regions for the final output composition.
d. If the footage has text that cannot be separated from the picture and cannot be made to be title-safe in the final composition, the chances are it will have to be cropped out completely and then added back on again separately.
3. Render out the final result, using settings appropriate to the output format.
Figure 9.15
How to Turn SD into HD
Turning SD into HD is particularly challenging, as there are many factors to consider.
When trying to give SD video footage the properties of HD video, you suffer from several disadvantages:
• The picture area is much smaller (a single frame of HD can have as many as four times as many pixels as a frame of SD).
• The frame is much narrower. This means it has to be resized to be significantly larger than HD in order for it all to fit in the frame.
• Any noise in the image will be greatly exacerbated.
• If it will be cut alongside actual HD footage, there may be noticeable differences in the picture quality and sharpness between the HD material and the resized SD material.
What this means is that you have far less flexibility in the process of going from SD to HD than you do from HD to SD. This technique tries to retain as much vertical detail as possible, at the cost of distorting the image slightly.
1. Load the SD footage. Make a note of details such as the size (in pixels) of the picture area, the frame rate, and so on. You'll need to compare these with those of your output HD format.
2. You'll probably have to noise reduce (using the techniques in Chapter 6), and it's best to do so before any scaling.
3. Stretch the picture horizontally as much as possible without it being noticeably distorted (I usually find that stretching it to an aspect ratio of around 14:9 (by 16.7%) gives reasonable results).
4. Scale this result (maintaining the new aspect ratio) until it has the width of the HD output format.
a. Because there will be significant scaling at this point, it's worth referring to the technique on page 154 for using high-quality scaling.
5. Crop the top and bottom of the resized frame to the height of the output HD format.
a. If you don't want to use a center cut-out, reposition the frame vertically before cropping.
b. For a more advanced application of this process, set key frames to dynamically adjust the vertical position of the frame according to the desired point of interest in the shot.
6. Follow the techniques in Chapter 3 for improving the sharpness of the final shot.
a. Ironically, you can add some noise back in at this stage to make it perceptually sharper, as it will appear to give more definition to the (now smaller) pixels.
7. Render out the final result.
TIP
You may be able to squeeze a few more pixels out of SD footage by utilizing the technique on page 166 for extending framing.
Figure 9.16
Shoot and Protect
If you know ahead of time that you're going to be shooting on a format that has a different aspect ratio from the destination format, you can plan for the difference using a technique known as shoot and protect.
The basic idea is that you compose your shots so that the main action happens within the framing of the destination format (once it has been cropped), but you still shoot the entire frame. Getting this technique to work properly means that you also have to consider how the safe areas will change once the shot is cropped, and so it's advisable to have corrected guides on the viewfinder.
Appendix 2 has a chart showing how to calculate recommended shooting ratios for different formats.
Figure 9.17 16:9 protected for 4:3. Earth © BBC Worldwide Ltd 2007.
Figure 9.18 4:3 protected for 16:9. Earth © BBC Worldwide Ltd 2007
Pixel Aspect Ratio
Pixels are thought of as being square, and they certainly are for display devices (monitors, digital projectors, iPhones …), but pixels in digital footage are actually rectangular. To better understand the difference, imagine you are building a house. You find an architect to draw up detailed plans, even working out how many bricks you'll need. Like everything in postproduction, something inevitably goes wrong at the last minute, and so on the day you're due to start building, a shipment of square bricks turns up. They're only slightly narrower than they were supposed to be, but your trusty architect does some quick sketching. Using these square bricks, the overall house is going to be narrower by several feet. However, she also points out that you could just order a few more of these square bricks now to end up with the same overall shape. The house won't use regular bricks like you wanted, but at least it will retain her stunning design.
Figure 9.19 © Jack James.
Rectangular pixels exist because they enable a wider image to be packed into a smaller file. In the film world, cinematographers can use anamorphic lenses to squeeze a wider, more panoramic image onto the same area of film. If you look at the negative of film shot in this way, it looks like it's been squashed inward horizontally. The reason you don't see it like that at the cinema is because the projector uses another anamorphic lens to unsqueeze it onto the screen. Exactly the same principle is used in the digital world, with some images squeezed together to use less pixels overall, then automatically unsqueezed for display purposes. Probably the most common example of this is in standard definition widescreen formats. SD frames are almost square (4:3 aspect ratio), but widescreen (16:9) images can be squeezed to fit this frame, meaning the same equipment can be used for both, from the tapes to the editing systems. All that needs to happen to make the process foolproof is for the footage to be adjusted automatically for display so that it look correct when you view it.
In order to determine the shape of a pixel, digital footage has a property known as the pixel aspect ratio, which is simply the width of a pixel divided by its height. So a perfectly square pixel has a pixel aspect ratio of 1.0, and a wide pixel might be more like 1.5.
One of the problems with all of this (other than there being numbers involved) is that sometimes the pixel aspect ratio can mysteriously disappear under certain situations. A particularly common problem during exports or conversions is that footage that looked fine on one system suddenly looks squashed on another. This happens either because the pixel aspect ratio was not written into the file by the first system, or because the second system has ignored it.
Figure 9.20 A group of square pixels (with a pixel aspect ratio of 1.0).
Figure 9.21 A group of non-square pixels, such as those found with DVCProHD footage (with a pixel aspect ratio of 1.5).
Figure 9.22 Approximating nonsquare pixels using square ones.
Video and Graphics
Managing the pixel aspect ratio is particularly important when you are importing or exporting graphics. Most digital video footage uses non-square pixels, while most digital photos and graphics use square pixels. Because of this, mixing the two can often result in elements getting squashed and stretched.
In this situation, it may be a good idea to convert all the footage to have a common pixel aspect ratio (for example, of 1.0), stretching it outward rather than squeezing it inward (thus creating new pixels) in order to get the best quality results. You may also see a slight quality benefit by doing this if your footage will be put through lots of processes that may degrade it further.
Appendix 2 lists the pixel aspect ratios of common formats.
How to Remove Objects from Shots
A perfect shot can be ruined by a misplaced boom, but can sometimes be rescued in post.
1. Load the footage.
2. If you're lucky, the object to be removed will be in the foreground. If that's the case, it can be removed using cloning techniques, in which case you can simply render out the result.
3. Otherwise, you'll first need to create a clean plate, removing the obstructing foreground elements. There are a few ways to do this, depending upon the picture content.
a. Scrub through the footage to find a point where the foreground objects have moved, revealing the background. Clone as much of the background as possible onto a frame that has the object to be removed, as a new layer. You may have to reposition the cloned regions to properly line them up.
b. Repeat using other frames as a cloning source until the foreground elements are removed.
c. If necessary, use cloning techniques to refine the clean plate.
d. Repeat this process for every frame in which the object to be removed is visible. You'll probably be able to use the new clean plate frames as a cloning source, which should make the process a little faster.
4. Once the clean plate has been constructed, mask it so that it only obscures the object to be removed.
Figure 9.23
5. Use cloning techniques to finish the job.
6. Render out the sequence.
TIP
This is the point where we start to stray into the realm of visual effects, as some of these techniques require a skill set similar to those required for digital compositing. However, I'm only scratching the surface here, and it's well worth learning some compositing skills even if you never intend to use them, as they can be useful for fixing all sorts of problems in postproduction. For example, the skill of match-moving (correctly positioning layered elements with respect to factors such as perspective), though a very deep and complex subject, makes creating and applying clean plates yield much better results.
How to Censor Objects
The visibility of a commercial brand, or a person who didn't sign a release, needn't mean a reshoot.
This technique is favored in documentary productions more than in other types, as it can be distracting to the viewer, but it can be useful when there are no other options.
1. Load the footage.
2. Mask the area to be censored on one frame.
3. Track the mask to the area across the shot.
4. Dynamically adjust the position and size of the mask so that it constantly obscures the object in question.
TIP
You can get different aesthetic results depending upo the type of filter you use to obscure the area in question. For instance, using a heavy Gaussian blur filter produces a much less distracting result.
5. Apply a filter, such as mosaic, to the masked area across the shot.
6. Render the result.
Figure 9.24
How to Replace Skies
Turning gray skies blue or adding stormy clouds can dramatically alter the mood of a shot.
1. Load the footage. If you want to change the pictorial content of the sky, you'll also need footage of the new sky to be inserted.
2. Key the existing sky. Depending upon the sky itself (whether it's cloudy, for example), this may prove tricky.
3. Use garbage masks to further isolate the sky. Remember to match the final mask to the motion of the camera, using tracking if necessary.
4. Blur the resulting mask.
5. Use the new mask to form the new sky.
a. Color-correct the original footage, using the mask to limit the extent of the correction to the sky.
b. Alternately, layer the sky footage over the original footage, constraining it by the mask. You'll need to track the motion of the new sky to the underlying layer if you do this.
Figure 9.25
TIP
A very simple way to add interest to shots with skies is to create a mask from a gradient, starting from the top of the frame, down to the horizon. Then apply a slight color correction using this mask to get the desired result.
How to Replace Logos
Replacing logos can be a painstaking process, but a real money-saver nonetheless.
1. Load the footage.
2. Follow the directions on page 162 for removing objects to remove the original logo. Don't render anything just yet.
3. You should now have two layers: the original footage and the masked clean plate.
4. Add the new logo onto a new layer.
5. You'll now need to match-move the logo so that it conforms to the movement of the objects in the scene.
a. Use trackers, particularly multiple-point trackers if possible, as these will aid in the rotation and scaling of the logo as well as the position.
b. Use warping techniques to ensure the perspective looks correct.
c. Use cloning techniques where needed to improve the integration with the rest of the scene.
d. Color-correct the logo to match the lighting of the surrounding region. Pay particular attention to shadows that may fall across the logo, using masked color corrections to simulate shadows where necessary.
e. Check every frame that the logo is visible in, adjusting as required.
Figure 9.26
6. Mask the logo layer to reveal foreground objects that should appear to be in front of it.
7. Go over the entire sequence with a cloning tool to fix problem spots.
8. Render out the result.
TIP
Software such as Imagineer's Monet (www.imagineersystems.com) can take a great deal of the guesswork and monotony out of the process of replacing logos by using tracking algorithms specifically designed for that purpose.
How to Extend Framing
This will push your system (and your patience) to the limit, but can result in footage that is otherwise impossible to obtain.
The field of digital photography has a technique for creating panoramic images much bigger and more detailed than the camera in question is capable of shooting. The basic idea is that you take multiple photographs of a scene, changing the objective slightly each time (but keeping everything else the same), resulting in a set of images that effectively overlap each other to some degree. You can then stitch these images together and adjust the result so that it's seamless.
In motion pictures we don't usually have the luxury of being able to shoot exactly the same subject from very slightly different viewpoints over and over again, but we do have one main benefit, which is access to lots of images.
1. Load all the available footage for the shot. Tr y to get as much of what was shot for a given scene as possible, which can include outtakes and different setups.
2. Using the original shot as a point of reference, arrange the footage in layers so as to extend the frame as much as possible.
a. Reposition, scale, rotate, and warp the footage as necessary.
b. As a rule of thumb, the greater the difference in camera angle from the original shot, the farther from the center it should be.
c. Don't agonize about keeping everything in sync, unless there are a lot of foreground objects moving about.
d. Remember to use frames before and ahead of the current frame, particularly if there is any amount of camera movement in the shot.
e. You may also be able duplicate layers and then flip or flop them to increase the available area or fill in gaps.
3. Scrub through the footage and keyframe the position, scaling and so on so that the elements don't appear to drift around.
a. At this point you may discover that choices you made earlier regarding the selection of the additional footage just don't work, and you may have to start over.
Figure 9.27
4. Depending upon what is happening in the scene, you may have to separate the foreground and background (and possibly in between) elements. Mask the regions in question and copy them to separate layers.
5. Scrub through the sequence again to ensure everything is lining up properly, and make adjustments as required.
6. Crop the composition so far to the desired aspect ratio, making sure there are no gaps at any point.
7. Color-correct the various layers so that they blend together.
8. Use cloning to make all the gaps seamless.
9. Play through the result and render it out.
How to Add Pseudo-Reflections
Simulating realistic reflections is probably best left to an experienced compositor, but this will give the subtle impression of a reflection quite easily.
Depending upon how you want the reflection to work, this can be a relatively easy process.
Figure 9.28
1. Load the footage and determine where you want the reflection to occur.
2. Duplicate the footage and mask the region to appear in the reflection.
3. Depending upon where the axis of the reflection is, you will need to flip or flop the result. For example, if the reflection is to appear on the ground, flip the shot vertically; if it's on a window, flop it horizontally.
4. Track the result over the course of the shot so that any camera movement doesn't give it away.
5. Depending upon the perspective of the scene, you may need to warp the reflection to make it look natural.
6. Add an additional mask to the layer, a gradient mapping the distance from the camera across the reflection. For example, for a ground reflection, the mask should start at white on the horizon, reaching black in the foreground. This will need the same tracking data applied to it.
7. Use this mask to control the opacity of the reflection layer. You want the reflection to appear strongest where it is closest to the subject, so the mask may need to be tweaked at this point.
8. Blur the resulting layer to taste.
a. You can also experiment with using other filters, depending upon the effect you're after.
9. Render the result.