CHAPTER 4
The Digital Video Workflow

We’ve talked about the fundamentals of compression. Now we’re going to dive into how to use this stuff in the real world for real projects. While all the issues covered in this chapter will be gone over in loving detail over the course of the rest of this book, it’s important to establish a general understanding of the digital video workflow to see how the different processes fit together.

Planning

The first phase of any production is planning. Don’t underestimate its importance. The best way to achieve good results is to know what “good” is for your project before you start. Even if you’re in charge of just a small portion of the overall project, knowing how it fits into the overall structure and having a solid understanding of the goals of the project will help you deliver optimal results on time and on budget.

You should define three basic, interdependent, elements as early as possible: content, communication goals, and audience. If you have a good grip on these three, the process will go much more smoothly.

Content

What is the nature of your content? Music video? Movie trailer? Feature film? Training video? Annual report? Produced on film or video? Letterboxed or full frame? HD or SD? PAL or NTSC? How long is it?

In an ideal world, you’d be able to design content that’ll compress with high quality on your target platforms. But very often we have to start with existing content produced for other purposes. Either way, understanding the source as well as possible is critical, as you’ll need to finesse the process to play to your content’s strengths and mitigate its weaknesses. The right compression settings can vary quite a bit with different types of content, even when targeting the same format and data rate. And there’s often plenty of processing required before the content hits the codec that can have a big impact on quality.

In general, the lower the bitrate and simpler the device you’re targeting, the more impact different types of content have on compression.

Communication Goals

Most of us don’t compress video just for the fun of it. We’re making compressed media for someone to watch, and thus we have a communication goal that compression is meant to achieve. Having a clear understanding of that goal is essential. However, it’s all too common for folks to not define explicitly what they’re trying to achieve with a project.

But how we encode a particular clip can vary radically with different communication goals. Take a music video, for example. If you were a member of the band and are embedding the video into your web site, with the goal of selling albums and concert tickets, you’d make sure to use enough bits for the audio to sound great, and would use technologies that most users wouldn’t need an extra install in order to watch. Conversely, if you were the cinematographer of the video, and you were putting the video on your online demo reel, you’d emphasize video quality over audio. And since you’re aiming for a more technical audience, you might use a technology with higher system requirements to make sure the experience is very high quality. If you’re trying to sell your ability to make beautiful images, it’s better for someone to not see your reel at all than to see it presented badly; there’s never a second chance at a first impression.

Audience

We also need to know who’s going to try to watch the content, what experience they’re hoping for, and what their playback device is capable of.

The first order of business is to predict what formats the targeted device can play back. If it’s a mobile phone, there’s a limited number of formats it can play, so you need to use one of those. If it’s a PC (be it Windows, Mac, or Linux) format support can be a lot broader but with more variation in what a particular PC may support in formats, performance, and bandwidth. And, of course, there’s the possibility of requiring the user to update or install software for an improved experience.

If your target audience is a particular corporate intranet with the same identical system image and a known minimum hardware spec, count your blessings. Your choice of media player and format will be an easy one (almost always Windows Media), and you’ll know how much bandwidth you can have.

But if your audience is anyone who accesses a general interest web site via the Internet, things can get very complex. Making a single file that 90 percent of the audience can watch is pretty easy, but getting to 99 percent is a lot more complicated, as you have to start including Linux systems and a variety of mobile devices, plus ancient slow computers. You can get quickly into cases where you have a single or a few encodes (perhaps WMV and H.264 .MP4), but need to do a lot of web-side programming to dynamically embed the file in the available media player, whether it be Windows Media Player, Silverlight, Flash, Moonlight, VLC, Safari, or another player.

The other critical concern is bandwidth—how many bits a second can the user get? And knowing bandwidth isn’t just knowing what grade of DSL or cable they’re connected to. Even if they get the full rated bandwidth, a household could have multiple computers and game systems in use, with bandwidth split among them. And a single user can be running multiple applications that are all consuming bandwidth at once; that system update in the background is using bits that could otherwise be used for video. So you want to know the real-world range of bandwidth is that users are likely to have, and from that decide what you want to support. Fortunately, we now have good adaptive streaming technologies that can detect and select streams based on real-world bandwidth, so things are easier if you’re using one of those.

Be warned that a client may say, “It’s critical that everyone sees this video!” But embedding an http://foo.mpg link to a MPEG-1 file is as universal as you can get. Almost every computer shipped since the mid-1990s can play that out of the box. But no one uses that, because it launches an external player, and good quality requires a much higher bitrate than modern codecs; you’d be trading a lower bar of codec support for a higher bar of bandwidth required. For every project, you need to draw the line on how ubiquitous it really needs to be to meet the communication goals, and that’s a tradeoff between cost and time in authoring, complexity of management, and the minimum experience you’re not willing to deliver below. That gets down to devices (no phones?), operating systems and player combos (no Linux without Moonlight, Flash, or VLC installed?), and bandwidth (nothing below 200 Kbps?).

After all, there are still millions of people using dial-up, and the industry has abandoned delivering compressed media to them; no one with dial-up has any expectations of getting a decent video experience. And it’s not worth sweating how to stream full-length movies to phones; people just don’t consume content that way. Better to understand how they would want to experience that content, and figure out how to deliver it in a way that suits the user and device.

Balanced Mediocrity

So how do you balance content, communication goals, and audience? You’re trying to achieve what I call, tongue-only-partially-in-cheek, balanced mediocrity.

For video on the web, you’ll often not have as much bandwidth to deliver or as much CPU power to play back as much resolution, frame rate, or detail as your luscious source may contain. Compromises must be made.

Finding the right balance of elements can feel like making everything mediocre, so you’re not spending bits making one aspect perfect at the expense of everything else getting worse. Balanced means “in the right proportion.” So, given our music video example, you want to make the ratio between video and audio data rates be such that each seems about as good as the other, relative to your goals. The relative part there is critical—you’d rate “quality” differently between the version for the cinematographer and the version for the band. So, even though you’d go through the same decision-making process, bitrate and format choices would be quite different in the end.

A good test for balanced mediocrity is to determine whether changing any particular parameter diminishes the overall experience—for example, when raising the data rate hurts more by slowing download times than it helps by improving video quality, and reducing data rate hurts video quality more than it helps by reducing download times. Or increasing resolution hurts more by increasing artifacts than it helps by increasing detail, and reducing resolution would hurt detail more than it would help by reducing artifacts. The bitrate requirement, video quality, resolution, and artifacts may not all be quite good as you’d like them to be, but if you’ve got the right balance of mediocrity for your project, you’ll have done what you can.

Production

Production is the actual shooting of the film or video, creating the animation, or otherwise creating the raw assets of the project. This is often the most expensive and most difficult part of any project, so getting it right is critical. And remember that the phrase “We’ll fix it in post” should be black humor, not an actual plan.

Professional video and film production is much, much harder than video compression, and it requires an experienced crew. One of the biggest mistakes many companies and individuals make in creating web video is thinking that because the content is going to be delivered on the web, production standards don’t matter. This is completely untrue. Amateur video on the web looks at least as bad as amateur video on television.

There are many important, irrevocable decisions made during production. For video production, the content needs to be shot as progressive or interlaced, at 4:3 or 16:9 aspect ratios, in black-and-white or in color; at a certain frame rate; with a cinematographic style that can run the gamut from Oprah to Battlestar Galactica. Of course, there is no one correct answer to such questions. It all depends on the nature of each project. (Well, shooting interlaced is never the correct answer anymore.)

Production is covered in Chapter 5.

Postproduction

Postproduction is when the snippets of images and sounds created in production are turned into a final product. Postproduction includes editing, compositing, audio sweetening, and so on.

Much of the style of a production comes during post. Different editing styles can have dramatic effects on the final results, both in terms of the uncompressed project and in terms of the compressed version—MTV-style fast cutting yields a totally different viewing experience than long, slow cuts. Likewise, MTV-style cutting is enormously more difficult to compress than more conventional, sedate editing.

Postproduction issues are discussed in Chapter 5.

Acquisition

If the source is a live video signal or on a tape, it’ll need to be converted to a form that can be compressed. The good news is that more and more, an all-digital workflow is possible, where the output of postproduction is a file that can be used directly rather than being captured.

But, capture is required when all you have access to is a tape camera, flash card, and so on. Tape is the most challenging and expensive method, particularly with professional formats, as the decks can cost into six figures. More and more cameras shoot in a compressed format like DV, DVCPROHD, HDV, or AVCHD that can be losslessly captured as a bitstream using USB, FireWire, or a card reader. Things get harder with analog sources and formats that offer only uncompressed digital output. These require capture cards and sufficient storage speed and capacity. Digitization isn’t particularly difficult, but it does require the right equipment. Luckily, getting DV format video off a MiniDV or DVCAM tape is trivially easy with modern computers. Capture and Digitization is covered in Chapter 6.

Preprocessing

Preprocessing is everything done between the source video and audio and what is finally handed off to the codec. This includes cropping, scaling, deinterlacing, image adjustment, noise reduction, and audio normalization. While this may happen behind the scenes in consumer products, it’s always being applied. The output of preprocessing is the final series of images in the native sampling and quantization (typically 8-bit 4:2:0) that the codec will take.

The goal of preprocessing is to take content optimized for whatever it was created for and transform it into the best version to be compressed. In essence, it’s about making the content as much like itself as possible, given its playback environment. We want blacks to be black and the image shape to match the display. Preprocessing requires the most craft and subjectivity of any part of the compression workflow. On quality-critical projects, I find myself spending far more keyboard-and-mouse time doing preprocessing than on codec settings. In the end, the codec can never output anything better than the source it’s given—problems in preprocessing can ruin quality that the best compression can’t fix.

Preprocessing is covered in Chapter 7.

Compression

If you cared enough to buy this lovely book, you already know compression is the Big Kahuna. In the compression process, the preprocessed content is turned into the final video and audio bitstreams and the file that contains them.

Before compression, many decisions need to be made, including choosing formats, codecs, data rates, and frame rates. While picking the right compression settings seems daunting, it becomes much easier when you you’ve got good answers to our three questions:

The bulk of this book is about the compression phase, although we’ll continue to talk about the earlier stages throughout. Make sure to read the chapters specific to the formats you’re using.

Delivery

Delivery is how the files get to the user. This can be on a DVD, Blu-ray, or CD-ROM, downloaded from a service, streamed to a media player, embedded in a web page, sideloaded onto a phone, and so on. The files can be standalone or integrated into a larger project such as a web site or computer game.

More complex delivery schemes often have their own, more stringent issues. Delivery issues with particular technologies are covered within the chapters about those technologies.