Install distributed processing in Final Cut Studio
Create QuickClusters (render farms) and virtual clusters
Distributed processing harnesses the collective encoding power of two or more computers on the same local area network (LAN)—sometimes referred to as a render farm. By spreading the encoding load across multiple computers, you can reduce processing time compared to using a single computer that’s shouldering the entire job.
Before performing the tasks in this lesson, familiarize yourself with the Distributed Processing Setup Guide in Help > Compressor Help. You can also download the Apple Qmaster Distributed Processing Setup Guide.pdf from the Compressor Support section of Apple’s website. Distributed processing works best with robust hardware environments in which the media is centralized on SANs (storage area networks) and connected to the Final Cut Studio workstations over a high-speed fiber channel network.
During the Final Cut Studio installation, the following screen appears that allows you to create an Apple Qmaster service node.
This window allows you to quickly set up distributed processing on the target computer by automatically designating it as an unmanaged service node—a computer available to encode network jobs. By default, the Compressor Service Sharing option is set to No. So, if you want the target computer to process batches on the network, select Yes, and then click Continue.
Apple Qmaster, used in conjunction with Mac OS X and Compressor, allows you to create a simple, easily configured distributed processing network referred to as a QuickCluster. Before creating a network, consult the following table to verify that your hardware meets the minimum requirements:
Hardware/Software Requirements | |
|---|---|
Network | All computers must be physically connected to a local network. (Gigabit Ethernet, Fiber Channel, and so on are suitable. Wireless connections are not recommended.) All computers must be on the same subnet of the local network. |
Computer | All computers must be running the same version of Mac OS X, version 10.5.6 or higher. All computers must be running the same version of QuickTime, version 7.6 or higher. For optimal performance and compatibility, the same version of Final Cut Studio must be installed on each system to ensure that all Compressor settings are available to the QuickCluster. Non-matching configurations of Mac OS X, QuickTime, or Final Cut Studio are not supported. All computers must have read access to the source media and read/write access to the output destination. |
You need to designate one computer as the cluster controller. This computer will send the encoding instructions with all the frames to the rest of the computers—the node computers—during processing.
If you selected Yes on the Distributed Processing Screen during the Final Cut Studio installation (see “Installing Distributed Processing in Final Cut Studio”), your computer will already be configured as an available node.
If you did not, you can easily configure your computer as a node by matching the following settings in Qmaster preferences:
Note
You might want to limit the available processors to retain computing power on your local computer. For example, you could make one processor in a computer available for distributed Compressor processes and leave the other processor available for local Final Cut Pro editing.
Qmaster can display the status of active services in the menu bar alongside the clock, AirPort status, Spotlight, and so on.
You do not have to create a separate workflow for distributed processing. First, ensure that all targets in the batch are ready for distributed processing, and then click the Submit button in the Batch window.
After clicking Submit in the Batch window, Compressor lets you decide where to send the batch for encoding.
You can use the Batch Monitor to oversee encoding jobs on a QuickCluster, just as you monitor jobs processing on a local computer.
Distributed processing adds multiple variables to the encoding mix. Use the following table to troubleshoot some common issues:
Problem | Solution |
|---|---|
Sequences exported directly from Final Cut Pro are not encoding in the QuickCluster. | When exporting sequences directly from Final Cut Pro to Compressor, you must install Final Cut Pro on each node computer for the QuickCluster to work properly. All the nodes must have access to the media referenced in the Final Cut Pro project—that is, the volume with the assets must be mounted on each computer. Note that every running network instance of Final Cut Pro must have a unique product serial number. |
Node computers do not have access to some or all of the components of a QuickTime reference movie. | Use self-contained source media by having Final Cut Pro (or any other application creating the source media) write the entire contents of a movie to a single file, or consolidate the media on shared storage such as an Xsan. |
Unsatisfactory quality issues arise from multi-pass encoding jobs sent to the QuickCluster. | Allowing Compressor to segment the job produces faster encoding times because different nodes are working concurrently on separate sections of the job. A potential problem arises with multi-pass presets. With unsegmented encoding, a single computer will analyze the whole media on the first pass to inform its compression decisions. When a job is segmented, each node reviews only a portion of the media on the first pass, so overall compression decisions are based only on that source segment. This may not produce the best results for source media that changes visual style considerably during its length—for example, a documentary that has static interviews and dynamic B-roll. Clearing the Allow Job Segmenting checkbox will output more consistent quality on multi-pass jobs, but will also reduce the overall distributed processing speed. |
With any multiprocessor or multicore Mac, you can create a virtual cluster that treats each of your processors (or cores) as if it were a separate node computer. On a computer with robust resources, this can offer significant speed advantages during encoding, especially for codecs like H.264 that are not natively multi-threaded, because Qmaster will segment the job and send work to the separate processors as if they were different nodes on the network.
For example, if you want to encode a movie for iPhones (using, by default, the H.264 codec) and you submit the job to This Computer, the Compressor engine will only be able to harness one of your processors (cores) because H.264 is not multi-threaded. However, if you create a virtual cluster, submit the job to This Computer, and allow Compressor to harness any unmanaged services, all of the processors (cores) that you’ve made available will be tasked to the job.
To create a virtual cluster, follow the steps in “Setting Up the Node Computer” to configure your system as an unmanaged node with available service instances. Also ensure that the target has Job Segmenting enabled (see “Submitting a Batch to a QuickCluster”).
You can also create a QuickCluster within your local computer to have, in essence, a single system that acts as both controller and node. Follow the steps above for creating a QuickCluster by choosing “QuickCluster with services” from the Qmaster pane and select the Managed checkbox for the Compressor service. Submit batches to your local QuickCluster
Distributed processing in Compressor adds a layer of technical complexity to the process of encoding media. Use the following table to guide you when submitting encoding jobs to a virtual or remote cluster.
Codec | Distributed Processing Considerations |
|---|---|
AIC, Apple ProRes | These codecs are multi-threaded and will take advantage of all available processors (cores). There is no minimum clip duration for job segmenting. |
MPEG-2 | This codec is multi-threaded. The best performance for local clusters will be achieved when using half of the available services. For example, on an eight-core machine, use four services. |
H.264 | This codec is not multi-threaded. You must have job segmenting enabled for this codec to take advantage of distributed processing. The minimum clip duration for segmenting is two minutes. Note: Using H.264 encoding for Blu-ray does not take advantage of distributed encoding, and the “Allow Job Segmenting” option is dimmed in the Encoder pane of those targets. |
Any codec used in an Episode custom setting | The Episode workflow in Compressor does not take advantage of distributed processing. You will need to use the proprietary Episode Engine as an alternative to Apple Qmaster. Refer to the Episode documentation for more information. |
Note
For more information on distributed processing, visit the Compressor website (www.apple.com/finalcutstudio/compressor).
To evaluate your understanding of the concepts covered in this lesson and to prepare for the Apple Pro Certification Exam, download the online quiz at www.peachpit.com/apts.compressor.