CHAPTER 10
The Transition to IT: Issues and Case Studies
CONTENTS
10.0 Issues in the Transition to IT
10.1 Organizational and Financial
10.2 Technical Operations and Support
10.2.2 Live Hardware/Software Upgrades and Repair
10.2.3 Integration with Legacy Equipment and Systems
10.2.4 Monitoring and Systems Management
10.2.5 Standards and Interoperability and Vendor Lock-In
10.3 Operations, Users, and Workfl ow
10.3.1 Managing Disruptions During the Transition
10.4.1 Case Study: KQED, Channel 9, San Francisco
10.4.2 Case Study: PBS, NGIS Project
10.4.3 Case Study: Turner Entertainment Networks— Centralized Broadcast Operations
10.5 Generic AV/IT System Diagram
10.6 The IT-Based Video System—Frequently Asked Questions
10.7 It’s a Wrap—Some Final Words
10.0 ISSUES IN THE TRANSITION TO IT
The move to IT will have its share of pains and joys. Despite the rosy picture painted in Chapter 1 about the workflow benefits to hybrid A/V+IT infrastructures, there remain practical issues that can be daunting for some. Depending on your starting point and end goals, the road to IT may be fairly simple or dreadfully complex. In the latter case, the IT pill may kill the patient. How can we avoid this sorry end? As with most new adventures, proceed with caution and with eyes wide open. The section that follows outlines the key factors that should be considered before embarking on any infrastructure change.
Following the transition issues will be coverage of three case studies: KQED’s digital plant, PBS’s NGIS project, and Turner Entertainment’s digital facility. Finally, a short FAQ will cover some common questions.
10.0.1 Look Before You Leap
Moving to AV/IT likely will not be a knife switch cutover for most users. Consider the following possible transition scenarios:
• New facility from the ground up—no or little legacy equipment or formats. This is the green-field case.
• Upgrade of select pieces of traditional A/V with the IT hybrid.
• Wholesale upgrade of existing facility—legacy issues galore.
In all but a few cases, the move to IT should improve media workflows, but will it? This depends on what the end design goals are. Here are some choices to consider:
1. Replace traditional VTR/tape workflow with IT/servers but keep tape-like workflow.
2. Replace portions of old workflows with improved workflows.
3. Perform wholesale upgrade to new workflows.
One error some planners make is to duplicate the old workflow but use new equipment (#1). It is tempting not to change what works. It is tempting to keep all the people and institutional experience in place even though new workflows have major advantages. One case in which the old workflows were completely discarded is the PBS engineering project called NGIS. This is one of our case studies and shows what wonderful benefits can be achieved by implementing #3.
Changing from an old but culturally established workflow to a new one (despite the proven advantages) can be a gut-wrenching experience for the operations and maintenance staff. Just the thought of changing from the comfortable to the new is difficult for many of us. So what factors need to be accounted for? Figure 10.1 shows the three domains that are normally affected by any new infrastructure or major workflow change. The stakeholders from each domain need to be involved before any migration plan is in place. If not, there will be unhappiness and discord in many parts of your organization as they realize that the changes did not involve them even though they are affected in major ways. Let us see how each of these domain members are stakeholders in the process of change.
FIGURE 10.1 Transition planning: Stakeholders of interest.
10.1 ORGANIZATIONAL AND FINANCIAL
When the horizon of change becomes apparent, we often hear reasons why the move is a bad idea. Consider a few classic lines:
“It is too risky to change now.”
“It will not work in our market.”
“The union will never approve of this.”
“We cannot afford that.”
How can an organization reduce the staff’s anxiety level? It is understandable that some resistance will occur. Change management is an art. Cultural change does not occur overnight. Some of the steps needed to move forward with a project are as follows:
• Provide education for all those who will be affected—why, when, and how.
• Have a change leader in the organization to evangelize the new workflows and infrastructure.
• Sell the vision in small steps.
• Consider consolidating the media engineering and IT staffs or, at the very least, cross-pollinating these groups. This is a politically charged issue no matter which way it tips.
• Stay focused on short-term goals, but always keep the long-term vision as a guiding light.
• Secure visible support from executive management.
• Do not bite off more than you can chew.
• Make sure the transition strategy includes “what-if” scenario planning.
No project should move forward without a proper justification for the funding. For grand visions, the CEO and CFO will demand to see the return on investment (ROI) and total cost of ownership (TCO) analysis before proceeding. ROI can be a difficult metric to calculate, especially if the new workflows are unfamiliar. For more modest transitions, some of the questions may not be fitting, but they are still good reference points for consideration. Questions of interest follow:
Will it generate new revenue potential, how much, and when?
What are the most compelling reasons to make the change?
Will it lower the TCO compared to current operations? Fewer operational staff, lower maintenance costs, less ongoing capital spending, less floor space, less power usage?
What is the equipment lifetime?
Do we have the skill sets needed to be successful? Should we merge the video engineering and IT departments? Make changes in staff skill set? Redeploy some resources? Is training needed?
What is the initial capital cost for the equipment and installation?
What are the costs to upgrade the building for air conditioning, power, and security to support the new installation?
What is the timeline for the transition? Will it be done in phases? Will there be any disruption in our current operations and delivered product?
What is the overall ROI expected over the lifetime of the installation?
All these factors were important to the participants involved with the case studies considered in this chapter. For the Turner Entertainment and PBS cases, planning and execution were several-year events. The scale of the execution is enormous, as we shall see. It is beyond our scope to detail each of the aforementioned questions. However, if you are planning a transition, having solid answers to these questions is essential for your success.
10.2 TECHNICAL OPERATIONS AND SUPPORT
The second domain of stakeholders comprises engineering and ongoing technical support. In some ways, the change to AV/IT is the most challenging for this group. Why? Because traditional A/V experts are not always comfortable in the IT world. Also, the IT experts may lack a working A/V knowledge. IT Media Engineer is a new title, and few of the staff are fully comfortable taking on its mantle and working in both spheres. Here are some of the classic complaints:
• “IT networks cannot carry video in real time.”
• “No one is trained to operate or repair it.”
• “We have tried that before and it did not work.”
• “The viruses and worms will kill us.”
Of course, the eight technical advantages discussed in Chapter 1 are not all peaches and cream. Each has some counterpoint in reality. Every engineering choice is a verdict; “give me this for that” in effect. Very rarely is an engineering course set without weighing the pros and cons. So what are the main technical issues to weigh when contemplating the switch to the AV/IT world? Figure 10.2 outlines the five main topics for this discussion. The list is not exhaustive and assumes that equipment performance, reliability, and scalability are already factored into the system transition plan.
FIGURE 10.2 System technical and support issues.
10.2.1 Life Cycle Issues
Life cycle issues are a red flag for many engineering managers. Some IT components, such as PCs, servers, some software, Ethernet cards, disk drives, and more, have notoriously short life spans. In a way, Moore’s law is responsible for this. We might call it Moore’s law of obsolescence. This is not just a headache for end users, but for equipment manufacturers too. They must support obsolete gear and stock a lifetime supply (5 years normally) of some spare parts or continue the product in production. Also, manufacturers often have to quickly redesign products due to lack of parts. This is a royal pain. Competent vendors account for this and deflect any obsolescence anguish away from the end user as much as possible. In the end, any purchase decision should factor life cycle into the plans.
10.2.2 Live Hardware/Software Upgrades and Repair
In the normal IT environment, components can be upgraded during off hours. Likely, we have all seen the message: “The email server will be down for 30 minutes tonight between 8 and 8:30 for a software upgrade. You will have no email access during that time.” We learn to live with and work around the small inconvenience, but imagine a similar message, “The TV station’s video server will be down tonight between 8 and 9 to install a software patch.” This will not fly. The mantra for many media facilities is 24/7/365.
Knowledgeable vendors have designed much of their equipment to be upgradeable while in use. At first blush, this may seem nearly impossible. However, with a degree of equipment redundancy (to support fault tolerance), this is practical. For example, a video server may be configured as N + 1 (see Chapter 5), and the spare unit may substitute for the component being upgraded. There are many clever methods to upgrade while “live.” It is important that you query the vendor on these matters before a purchase decision. Incidentally, for some upgrades, both an A/V and an IT expert may need to be present. Depending on either alone may result in long delays when trouble is encountered, especially during off hours.
Another aspect is the initial transition from the old workflow to the new. How will this transition be managed? What sorts of interruptions will we encounter? What is the most intelligent way to manage this with minimal disruption to our operations? Should we run the new workflow in parallel with the old for a time to test for compliance and equivalence of functionality? Should the new workflow be enabled and tested in phases? For cases in which the system is composed of multiple versions of the same basic idea (e.g., station group implementation for 10 stations), should we implement one first and then follow with the others? These questions should be answered before making the leap to any new workflows.
10.2.3 Integration with Legacy Equipment and Systems
When a company or organization is building a new A/V facility, the issue of legacy integration may be a moot point. However, for most transitions, the move from old to new cannot be done without accounting for legacy systems and media/metadata formats. Consider the case of CNN’s digital feeds and edit system. In 2003 CNN initiated a phased installation to an AV/IT-based news production architecture from a tape-based one. As part of its New York City facilities, CNN installed 29 SD ingest encoders for recording news feeds from the field and elsewhere, 20 SD playout decoders, 15 professional NLEs, and 50 proxy editor/browsers. Media clients attach to ~950 hr of fault-tolerant, mirrored A/V storage. Most interconnectivity is based on Ethernet, mirrored IP switches, and some Fibre Channel. A similar system is also installed at CNN’s Atlanta headquarters. These are not green-field installations, and on-air news production needed to continue as migration to the new system was phased in. Of course, the CNN integration is not typical of most transition scenarios due to its size and scope.
10.2.4 Monitoring and Systems Management
Every system, whether traditional A/V or hybrid IT, needs some sort of fault, warning, and diagnostic reporting provision. The IT world offers a standardized, mature, and encompassing solution set as shown in . Compared to standard A/V gear reporting, IT systems management is very sophisticated and holistic. Most vendors provide their own user interface for access to the most common management operations.
10.2.5 Standards and Interoperability and Vendor Lock-In
Most AV/IT systems are composed of different vendors’ equipment in a tightly integrated configuration. Ideally, the end user has the choice to substitute one system element for another. However, regarding hybrid AV/IT systems, most providers require that no third-party “substitutes” [commercial off the shelf (COTS)] be used. For example, a data server element in a live-to-air environment may require very special operational specs to meet the needs of live failover or an element (Fibre Channel switch, IP router, etc.) is located in a critical data flow and requires guaranteed throughput. For most standard IT applications, a little slower or faster element may not impact workflow at all, whereas for AV/IT the element specs are crucial for operations. A two video frame (~65 ms) equivalent delivery delay in an email application will have zero impact on the end user. Compare that to the same delay for a critical A/V application where a lip-sync problem manifests itself. Even content-neutral elements such as IP routers and switches can have a major impact on system performance under corner case loading and failover scenarios. As a result, while the end user may be tempted to substitute COTS equipment in a configuration, the reality of guaranteed performance under all operating modes makes this a risky decision.
Product support is another aspect of using COTS in an end-user configuration. A vendor’s system design is normally tested and validated with known equipment. Each element may have specific error and status reporting methods that the system provider is depending on. The overall system status and health are strongly dependent on knowing the exact nature of each element. If a COTS element is substituted for a vendor-provided element, the vendor support contract may be breached. After all, configuration documentation is the heart of any big system, and changing components to suit the needs of the end user (normally to save money or repair time) will make support a nightmare.
All similarly specified IP routers are substitutable, right? Well, for some applications this may be true, but the performance and operational characteristics are never exactly equivalent between, say, a Cisco router and one from Foundry. Configuration methods are often completely different, internal components are rarely the same (power supply diagnostics will be different), and so on. Any substitute of a mission-critical system component will only cause longterm grief to the end user and the providing vendor.
With that preamble, it is not difficult to see that some vendor lock-in is inevitable. However, this comes with the upside of guarantees in performance and support. No end user wants to be completely locked into a vendor’s products. That is why standards and interoperability are still of value. After all, most systems need to import files from external sources, so the file formats should be “open.” Also, metadata need to be open in terms of access and formats. When evaluating a vendor’s solution, make certain that the environment is open for file import and export. Because many systems have control points (record input A/V, playback, and so on), associated APIs and protocols should be well documented and open for third-party access. Ask many questions of any provider so you know which system domains are open and closed. Only then can you make intelligent buying decisions.
10.3 OPERATIONS, USERS, AND WORKFLOW
The third category of interest is where the rubber meets the road: the user experience. The end user usually gets the ear of the engineering and management staff when selecting new gear. Many purchase decisions are a direct result of what the operations staff demand. Changing workflows or the operational experience is never easy, and end users are often reluctant to venture into uncharted waters. Some operators (especially A/V editors) are comfortable with their favorite interface and can be intransigent when it comes to trying something new. This behavior comes from several motivations:
• “If it is not broken, do not fix it.”
• “We do not know if the new way will work.”
• “I like vendor XYZ’s user interface, so do not ask me to learn a new one.”
• “Changing products now will slow us down—too much to learn. If you want it done today, then let me use what I already know.”
For sure, these are valid comments, but in the context of the greater good, they are inhibitors to more efficient workflows and all that IT promises to deliver. Let us look at some of the factors that may impede the adoption to AV/IT systems.
10.3.1 Managing Disruptions During the Transition
Managing disruptions is a crucial aspect of the move to any new workflow. Planning needs to be done to minimize any potential reduction in the quality and amount of output. Using a mirror training system is one way to reduce staff anxiety and opportunity for error. In the case of Turner Entertainment, they built a new facility and ran a new channel in parallel before decommissioning the old channel. This gave the staff some time to learn the new workflows and iron out the bugs. Of course, this was a challenge since the staff needed to operate both old and new simultaneously, but this proved a wise choice in the end.
Training is a key to smooth transitions. This may come from vendorprovided classes or in-house mentoring. As with most of us, once we are comfortable with a new process, then we are eager to spread our wings. Without proper training, we should expect the worst—problems at every turn. The poet Longfellow said, “A single conversation across the table with a wise man is better than ten years’ mere study of books.” How true. So get educated by attending industry events such as SMPTE Technical Conferences and Networld + Interop conferences. Meet the people driving AV/IT forward. Attending conference-related tutorials is a great way to learn. Subscribe to several A/V and IT industry trade magazines. Subscribe to free Web-based A/V and IT newsletters. Attend local A/V and IT industry events.
10.3.2 Know Thy Workflow
During the course of developing this chapter, I interviewed several facility engineers and end users. When contemplating the change to a new workflow, they acknowledged that there was the inevitable step of reviewing the documentation of the existing workflows. “Where is the documentation for our workflows?” was the refrain. Often, no one answered back. As it turns out, existing workflows were part of the culture of doing the job and not some well-thought-out and documented procedure. This was an eye opener to some of those about to embark on improving the old workflow, since they did not have a good handle on what they wanted to improve! This is a dangerous starting position for a new venture. Of course, every case is different, and each transition plan eventually comprehends the status quo to the extent needed. The lesson was clear to all: you cannot improve what you do not understand. If you are thinking of changing workflows, take the time to document what you have so the transition to new ones will not have as many surprises.
Chapter 7 reviews the fundamentals behind all A/V workflows. This is a high-level discussion but will give you some food for thought as you plan new workflows.
10.4 CASE STUDIES
The three short case studies to follow investigate the workflows, challenges, and successes of each new design. In all cases, the existing media enterprise planned and implemented (or using a systems integrator) the transition from traditional A/V to an AV/IT system. In one case the transition is a work in progress, whereas for others the job is done. For each case study, stakeholders were interviewed for their take on the success and issues with the undertaking. These three are representative of many similar facilities worldwide. One is a typical public TV station, one is a TV network facility, and one is a large multichannel broadcaster. The coverage is brief, and you may not find a direct parallel to a project you have in mind. Nonetheless, the general methods and lessons from these studies are applicable to just about any size AV/IT project. The following case studies are covered:
• KQED San Francisco—A flagship station of public broadcasting and a provider of PBS programming. It is one of 349 member stations serving commercial-free programming. In 2003, KQED replaced its tape-based master control chain with an IT-based one.
• PBS—The U.S.-based Public Broadcasting Service is executing a longterm plan to convert its member station A/V stream feeds over satellite to file transfers. Migrating from traditional video feeds to file transfer yields great economies. This transition is ongoing in 2009.
• Turner Entertainment Networks—In September 2003 TEN rebuilt its entire broadcast operations center in Atlanta. As one of the premier broadcasters worldwide, TEN has developed an IT-based, distributed operations center for 22+ channels.
10.4.1 Case Study: KQED, Channel 9, San Francisco
KQED1 is a major public broadcaster and PBS affiliate. In 2003, on-air operations were based mainly on tapes and VTRs for time-delaying PBS programming. Yes, the station has some live local programming, but the lion’s share of on-air content comes from stored materials. Most delays were anywhere from a few hours to a few years. The workflow is not identical to a commercial station, but there are many similarities.
KQED replaced most of its manual videotape chain with an automation system, a video server system, and a data tape archive. Figure 10.3 shows the major components in the system using IT-based control and stored program content. The legacy system had all the classic inefficiencies inherent in a tapebased system:
FIGURE 10.3 KQED digital on-air system.
• Linear, non-networked material
• Tape-based, VTR maintenance headaches
• Little automation, no HD chain
• No support for upgrading to multichannel
A comprehensive plan for a new infrastructure should do more than eliminate the pains of the old one. In addition to moving beyond videotape, some of the goals of the new infrastructure were as follows:
• Moving from 20 hr per day to broadcasting 100 hr (six channels) without increasing staff.
• Maximizing the use of automation—ingest, file movement, storage management, playout, and switching.
• Providing support for HD along the entire chain.
• Providing support for eight audio channels per video channel.
• Addressing file transfer needs. Support for ingesting files (e.g., from PBS), archiving files, and playing out files. PBS file ingest is a future need, so the new system also supports recording and playing of traditional A/V streams.
10.4.1.1 The Configuration
Figure 10.3 shows the chief components of the new system. Central to all operations are two video servers using Avid’s MediaStream. Each is attached to separate SAN storage. Two servers provide protection in the event of one failing. One is used primarily as an ingest server (it has 31 I/O), and the other as a playout air server (16 I/O). In the event of one malfunctioning, the other one can take over. Material comes in from satellite or tape and is recorded automatically into the ingest server. Once it is ingested, trimmed, and QA’d by personnel, it is pushed to archive and to air server if it airs within 24 hr.
The ADIC Scalar 1000 archive system is a major system component. It has a capacity of 1,000 tapes. AIT-3 tapes are used with a capacity of 100GB each, so each tape supports 18.5 hr of MPEG2 at 12 Mbps (SD) per tape for a total of 18,500 hr for the archive. A total of 12,000 hours of mixed SD/HD is a practical limit. HD is stored at 19.3 Mbps and some at 45 Mbps, so this also reduces total storage hours. When AIT-4 tapes are used, the capacity will double. EMC’s AVALONidm storage management software automatically and transparently manages files in the ADIC tape system, optimizing the placement of data to match service levels.
Most of the stored materials are received from PBS. KQED has the rights to rebroadcast a show four times over a 3-year period for most of the programs. The automation is provided by Harris Automation, ADC100. It manages all A/V movement. Six days in advance of air time, the traffic department gives automation a list of programs for all channels. The Harris ADC100 locates the programs in the archive and moves them to the ingest server. Twenty-four hours before air time, they get copied to the air server from the ingest server’s storage. Six days’ notice is required for the “pull list”—that gives plenty of time to resolve any material location problems. The day before air time, interstitials are pulled from the archive and loaded into the ingest/protect and air servers. At air time, both servers play out in parallel all channel programming with fault tolerance and peace of mind.
When PBS initiates program delivery via file transfer, KQED will be ready for this new form of ingest. PBS has yet to define the exact A/V file format. It will likely be a constrained MXF file with MPEG2 essence. The Advanced Media Workflow Association (AMWA) is developing a best practice document to describe the MXF file constraints for program distribution. This is ongoing work in 2009. Operations are running smoothly, and the engineering staff and operators are gaining confidence in the new workflow. National TeleConsultants was the systems integrator for the project.
Of course, not every aspect of the migration went smoothly. KQED had issues with device configuration and establishing stability of some software elements. Some vendors overpromised their deliverables. The learning curve from traditional manual to full automation operations took time to negotiate.
10.4.2 Case Study: PBS, NGIS Project
PBS2 is the main public broadcaster in the United States with 170 non-commercial, educational licensees operating 349 PBS member stations.
Headquartered in Alexandria, Virginia, PBS broadcasts several programming genres, each on a different channel, which in turn are consumed by the member stations. For many years, program distribution to member stations has been via satellite A/V streaming. Local stations recorded the feeds and scheduled playback based on their local needs, as discussed in the KQED case given earlier.
PBS is engaging the Next Generation Interconnect System (NGIS) project, which will completely revamp how it receives (from content providers) and distributes programming to member stations. This case study outlines the issues with the old workflow and the motivations and plans to upgrade to a file-based workflow nationwide. NGIS is a work in progress, but the ideas are instructive for review even though it is not complete. To put the project into perspective, Figure 10.4 illustrates media flow among providers, PBS, and member stations. NGIS targets the programming flow in and out of PBS.
FIGURE 10.4 The PBS supply chain.
10.4.2.1 Motivations to Move from the Status Quo
Let us start off by reviewing the current roadblocks to smooth program exchange between entities in Figure 10.4. The following list breaks the issues into three camps: programming inputs to PBS, PBS output streams, and member station operations.
1. Programming produced by outside sources:
• All incoming content (files) enters a QA process.
• Corrective measures are applied as needed.
• Program and segment timings are checked and metadata are updated as necessary.
2. Real-time streaming distribution to member stations:
• No metadata are included with programs. All program-related metadata are sent out-of-band—errors are common.
• Programs are fed multiple times to cover U.S. time zones, thus wasted bandwidth.
• Satellite rain fades impact end users.
• 179 stations do basically the same thing:
• They use direct-to-air transfer or a tape/server delay process.
• Many stations record earlier time feeds to avoid a possible future rain fade during the correct playout time.
• As stations move to video servers, the feed recording/playout workflow is not efficient.
Table 10.1 outlines advantages to moving to a file-based ingest/playout workflow and away from tape ingest and A/V stream feeds to member stations. NGIS is aimed at removing the current inefficiencies and adding new performance features along with future proofing the design.
Table 10.1 Comparative Workflow Advantages of NGIS
The last entry in Table 10.1 enables a big savings in satellite use and overall distribution bandwidth. By sending files instead of streams, NGIS increases the overall transmission efficiency by about a factor of six. Fortunately, because PBS member stations do not broadcast network programs live for the most part, transferring program files (especially HD) to stations using non-real-time delivery saves considerable satellite bandwidth and is a great way to cut costs.
10.4.2.2 The NGIS System Outline
The following list outlines the salient features and technical advantages of the NGIS:
• All SD and HD files are compressed MPEG (or other) wrapped in MXF.
• Most content is distributed as files using IP over satellite.
• Content is sent once, providing accurate content distribution.
• Intermittent transmission outages do not impact file delivery.
• Operations are automated as much as possible.
• Received content is temporarily cached on an edge server at the station side.
• The number of transponders and costs is reduced.
• Missing packets are requested and re-sent using Internet connection.
Sending files from point to multipoint over satellite is non-trivial. One missing bit can ruin nearly a second of MPEG video, so any loss is unacceptable. NGIS uses two strategies to guarantee 100 percent file delivery integrity. The first is a very robust forward error correction (FEC) means to correct up to 20 min of lost data! This requires sophisticated coding but is achievable. It is a necessity during a heavy rain fade. A 20 min loss at 80 Mbps (transponder rate) is equal to 12GB of recoverable data. Very impressive. See Chapter 2 for more information in FECs and error correction strategies.
A second strategy is to automatically ask for a retransmission of corrupt packets if the FEC cannot recover the errors. The backchannel request must be used sparingly; otherwise, retransmissions will kill the file transfer advantages. For example, a FEC that can recover only 0.1 s of data will force many retransmissions, and the satellite bandwidth usage will be worse than with legacy A/V streaming.
Figure 10.5 shows a NGIS test configuration for the PBS side of the equation. At a scheduled time, automation retrieves files from the ADIC archive, prepares them for transmission, and feeds them to member stations over satellite. Note the Internet connection. It is required for retransmission requests and for member station requests for nonscheduled programming materials. On the station side (Figure 10.6), IP packets are received by the “catch PC.” It applies the FEC to correct file data errors, asks for retransmission if needed, and optionally reformats the file for the member station legacy video server. Most stations already have video servers, supplied from the usual suspects, and it is not practical for PBS to send files formatted for each server type; that would waste bandwidth. As a result, the catch PC optionally reformats (lossless normally) the incoming PBS MXF file into a format usable by the installed legacy server. The prototype shows an Omneon Spectrum server. For the final rollout, various models will be used, depending on legacy installations on a perstation basis (see the KQED case given earlier).
FIGURE 10.5 Prototype NGIS PBS-side hardware.
FIGURE 10.6 Prototype NGIS stationside hardware.
NGIS also defines format requirements for vendors supplying programming to PBS. When WGBH produces a new Frontline episode for PBS, it would format the program within the NGIS guidelines for metadata, closed caption, MPEG format specs, and so on. Any errors inherent during PBS ingest may get propagated to all member stations. Hence, NGIS is keen to define a standard ingest file format and improve the quality of the total distribution process.
Upon completion, NGIS promises to be a very efficient and practical way for PBS to receive and distribute programming. NGIS sets a model for how syndicated programming, news, and commercials may be distributed to commercial stations. Of course, there is activity in this area. See, for example, the services and products from Bitcentral (www.bitcentral.com) and Pathfire (www.pathfire.com) that are in general use and similar to the overall themes of NGIS.
Making the transition to NGIS is filled with challenges. Some of the issues needing to be resolved are the following:
• Member stations are currently using video servers from a host of vendors. The master PBS file format will likely be MXF with MPEG2 essence. However, because this format may not be compatible with all installed station-side video servers, file translation will be implemented as needed.
• PBS will package metadata with each program file. Local stations may decide to use these data to improve their operational workflow.
• PBS needs to coordinate with the 170 member stations so they have the proper file receive infrastructure, as discussed earlier. These stations need to start thinking files, not streams, as the new workflow.
• Funding from the Corporation of Public Broadcasting (CPB) is needed to complete NGIS. Writing proposals and working with the budget process are time-consuming and filled with iterations.
10.4.3 Case Study: Turner Entertainment Networks—Centralized Broadcast Operations
In January 1970, Ted Turner purchased UHF channel 17 (WJRJ) in Atlanta and renamed it WTCG. From that humble beginning, TEN3 now operates 22 network feeds (plus time zone feeds), including three of the top five cable networks in the United States—downtime is not an option. Channels are fed to satellite and cable distributors in North and South America. In September 2003, TEN moved all broadcast operation to a new 198,000 square foot building (see Figure 10.7).
FIGURE 10.7 Broadcast operations building—part of the TEN campus in Atlanta.
Image courtesy of TEN.
Migration from a videotape-based to an IT and file-based architecture was likely the biggest broadcast-related project of its kind at the time.
The following list enumerates the motivations behind the project.
• Provide for anticipated network growth
• Improve control and management of Turner physical and intellectual property assets
• Reduce current operational compromises and meet the growing demands of the Turner Entertainment Group
• Provide a platform for implementation of new media-based business initiatives
• Consolidate technical resources
• Implement a total digital infrastructure with fiber optic connectivity for file exchange
• Facilitate the implementation of HD digital TV
• Facilitate the migration from a videotape-based storage and workflow to a digital asset-based environment
• Enable upgrade of the traffic system and other key broadcast-specific operating systems
Now, that is a tall order. Figure 10.8 illustrates a high-level view of TEN’s broadcast inventory management (BIM) system. At its heart is the media operations center (MOC). This portion is responsible for ingesting A/V materials from tape, utilizing six short form bays, four long form bays, and 20 cache engines. Tape ingest formats include IMX, D2, and SRW (Sony HD) tape formats. Ingest video servers are used to temporarily cache newly ingested commercial and promotional content. Next, the cached materials are transferred as files into EMC near-line storage arrays and the Asaca AM 1450 DVD (Blu-ray disc) backup libraries. Pro-Bel Automation’s Sextant software moves the programming files via the AVALONidm (storage management software) from the EMC CLARiiON FC4700 arrays to the playout pods as determined by channel scheduling needs. Program content is cached directly to the air playout servers.
FIGURE 10.8 Broadcast inventory management system.
Image courtesy of TEN.
Each pod is designed with parallel playout chain redundancy. Each channel chain (A and B) has separate automation control. Video servers (mirror of content) synchronously play out the channels under Pro-Bel control. GVG MC2100 master control switchers are used. There are 49 automation/air chains in all, with a total of 200+ associated computers.
The Cartoon Network has two dedicated StorageTek Archives (Powderhorn 9310) for storing 6,000 cartoons. Total capacity for the archive is 240TB. While all other channels have near-line storage (with backup, not archive), Cartoon relies on deep archive due to the frequency of playback and total storage needed. Figure 10.9 shows a composite view of the 500-rack equipment room.
FIGURE 10.9 Composite view of TEN’s 500 racks of equipment.
Image courtesy of TEN.
Figure 10.10 outlines the benefits and payoffs for the BIM project. The five payoffs on the right side are of universal value to any commercial media operation. It is obvious that migrating to IT has big advantages for TEN. Of course, there were challenges along the way. A few of the biggest obstacles were as follows:
FIGURE 10.10 Benefits and payoffs of the BIM project.
Image courtesy of TEN.
• Building one of the first SMPTE 292M (serial HD link)-compliant facilities required unique installation methods to preserve the long-term integrity of the cabling.
• Developing the BIM system, which is not an off-the-shelf solution, required the cooperation of five major vendors in order to complete the system successfully.
• Painstaking analysis of current workflows was required in order to best understand what BIM should provide.
• Work with many vendors over 12 to 24 months was required to develop new products in order to meet the vision of what the file-based, HD-compliant infrastructure needed to deliver.
10.5 GENERIC AV/IT SYSTEM DIAGRAM
In the final analysis, the three case studies and many others not outlined here are patterned after the generic converged AV/IT video system shown in Figure 10.11. Figure 10.11 illustrates the major architectural themes discussed in this book. By rearranging, factoring, scaling, and duplicating elements, a business or an organization can implement any AV/IT system, including the three case studies. For sure, this description is overly simplified, but it represents the new paradigm for IT-based video systems. Note the use of the four classes of media clients, as discussed in Chapter 2. Also note the use of three levels of storage including SAN/NAS, as outlined in Chapter 3. Prominent, too, is the use of IP networking, as discussed in Chapter 6. Ideally, all elements are tied into centralized systems management. Concepts such as media formats, protocols, security, software architectures, and reliability are implied but not distinctly shown. Of course, there is not a single central IP router; the one in Figure 10.11 represents an aggregate of network switches, including redundancy methods.
FIGURE 10.11 The generic converged AV/IT video system.
Figure 10.11 is useful as a touchstone when developing converged AV/IT systems. It is a reminder of the chief elements required for many designs.
Well, that concludes the three case studies. If space would permit, many more from Europe, Asia, and the United States could be reported on. Just about every new broadcast and professional installation has a large component of IT spread throughout. The FAQ to follow concludes the discussion about AV/IT systems. Chapter 11 is an overview of traditional A/V technology. The contents are a flash course in digital video basics.
10.6 THE IT-BASED VIDEO SYSTEM—FREQUENTLY ASKED QUESTIONS
Thinking of migrating to a converged AV/IT system? If so, the following FAQ will be of value to you. It covers select issues commonly encountered by facility planners. The FAQ is brief; likely, it will not answer all of your questions. However, it should give you a springboard to start your planning process.
Q1: What is a practical plan to convert to an IT-based A/V workflow?
A: Do not boil the ocean. Convert one workflow at a time. Start with ingest or playout or archive or editing or graphics. If you are starting from scratch, you may be able to do a complete IT design. For most designs, the move to IT will be incremental and must integrate with a legacy system. Remember that file-based technology and workflows are enabled using hybrid A/V+IT systems components.
Q2: Is there is downside to the migration to IT-based video?
A: Sure. Creating video systems using IT is not a panacea. There are hidden costs, operational issues, and implantation issues. Some of those issues are staff education, interop falling short of vendor promises, too many standards to track, proprietary aspects, the need to maintain the desired network QoS, the need to keep it secure, software upgrades, IT gear obsolescence, required software patches, unified element management, and consistent metadata and format use. Despite these issues, each can be overcome by applying the principles outlined in the book’s chapters. Proof, too, is demonstrated by many successful AV/IT installations worldwide.
Q3: How should a small TV station or other A/V facility approach the move to IT?
A: Work with trusted vendors and system integrators who have a proven track record in offering and implementing IT-based systems. Educate yourself and the staff. Visit facilities that have made the move and interview the stakeholders. Make decisions based on the expertise of the IT and A/V staff; do not go it alone.
Q4: Where should I expect the most gains from the switch to IT?
A: Networked efficiencies and reach, content tracking and usage, less pure video to test and calibrate, flexible workflows, speed of content access, virtual facilities not bounded by distance, no videotape handling, the ability to ride the IT wave to lower costs, and higher performance. Remember not to duplicate a tape workflow with IT equipment but to leverage file transfer delivery times to save money and relax the demands on equipment performance and reliability.
Q5: What aspects of IT-based media client video flow will dominate: file transfer, IP streaming, or RT direct-to-storage access?
A: Based on current trends in facility design, file transfer is most common, followed by direct-to-storage access, followed by streaming. Low bit rate proxy video will be streamed to client stations. Your mileage may vary based on workflow needs.
Q6: What standards should we give special attention to?
A: For A/V file interop, focus on MXF and MPEG/DV; for storage access, CIFS, NFS, and iSCSI; for metadata, MXF, XML, SMEF, SMPTE, and EBU recommendations; for networking, IETF and IEEE; for editing, AAF and, of course, SDI and countless SMPTE standards for traditional A/V interfacing.
Q7: How should we approach asset management?
A: There is no simple answer. It depends on size of the installation, workflow needs, amount of programming, material repurposing plans, and other factors. Spend time to get this right, as it is the key to many workflow efficiencies. Again, learn from others who have already made the move. Do not try to boil the ocean.
Q8: In what format should we archive our A/V materials?
A: Your solution depends on several factors, so there are several answers. If you ingest materials from digital cameras, save in the camera’s native compressed format. If you ingest from legacy videotape or live video feeds, select a facility-wide encoding compression format. Choices range from DV 25/50/100, SMPTE’s VC-3 HD format, and one of the many MPEG2 or MPEG4 formats at bit rates and line structures (HD, SD) from 10 to 450 Mbps or fully uncompressed video for very high-end work. Audio formats range from uncompressed 16/20/24 bits, Dolby-E, or a variety of other formats. Archive choice also depends on material repurposing plans. For SD/HD play-to-air only applications, lower video bit rates suffice. As MXF becomes mature, many facilities will choose to work and archive with it as a wrapper format.
Q9: What are the key AV/IT design parameters?
A: Give attention to the Top 10.
1. Functionality of the total workflow (includes MAM)
2. Interop with legacy systems and intersecting workflows
3. Essence and metadata formats
4. Element and system reliability (SPOF, NSPOF, MTTR)
5. System scalability (number of clients, storage, network size, bandwidth)
6. Network topology and QoS
7. Software architectures (Web services, middleware, Java EE, .NET, standalone)
8. Storage requirements (online, near-line, and archive, hours, QoS)
9. Security across all system elements
10. Element management (system management techniques)
Q10: We are thinking of developing new IT-based workflows for our A/V needs. How should we start?
A: Know thy current workflow first. Spend time to document how you do things now before you design replacement workflows. Time spent up front will enable you to design better workflows later. Make sure any providing vendors agree to your system requirements.
Q11: How should we qualify a new AV/IT installation?
A: Develop acceptance test criteria. Before signing the check for a completed systems installation, make sure it all works according to plan. Develop a list of test actions. A target list may contain items such as glitch-free A/V recording for 5 hours, glitch-free playout for 5 hours (or a lot more), file transfer speeds, storage access rates, component failure responses, device failover responses, security audit, media client responsiveness, power cycle responses, alarm reporting, MAM functionality, and many more. The bottom line is: make sure you are buying a useful system that is not filled with holes.
Q12: Any parting words?
A: AV/IT is still maturing. Some workflows are mature and ready for prime time (news story production, ingest, playout, proxy browsing, cataloging, graphics playout, and production). Others are taking baby steps as with live production. Understand your workflow and map to stable technology. Do not bet on the latest unproven concepts for mission-critical applications. Traditional A/V technology is not dead. The fat lady is not singing, but she is practicing her scales.
Q13: What do you see in the crystal ball for technology?
A: Oh boy. Well, looking near the surface, IT costs will continue to decline, and element/link performance will continue to increase. That is an easy one. Traditional A/V will find applications for years, but AV/IT will continue to move into broadcast and other professional applications. Storage will support 1,000 “viewing quality” HD movies on a single 7TB disc drive by 2014. Open systems will take hold, and A/V vendors will offer (begrudgingly, for all but the pioneers) media clients to work with third-party online COTS storage for mission-critical applications. Green thinking and data center virtualization will drive purchasing decisions. Expect more and more software SD/HD processing with less and less video-specific hardware. Videotape will find shelf space at the Smithsonian. Looking deeper—now this is amazing—wait, it is getting cloudy. We will just have to wait and see.
Q14: What do you see in the crystal ball for A/V systems design?
A: Well, open systems for sure. Expect more use of Web services possibly coupled with SOA strategies. Expect to see rich media clients of all types, workflow management tools, more MAM, lots of creative tools for graphics, video and audio editing, and authoring. File transfer will rule the day, with A/V streaming taking a back seat. Live event production will continue to use traditional A/V methods for many years. It takes a pioneer to change the status quo of event production, and there is not one on the horizon. A/V devices will be managed using IT methods—at last. Oh yeah, and of course, IP- and Web-based everything.
10.7 IT’S A WRAP—SOME FINAL WORDS
Well, that is it—insights from the real world on the convergence of A/V and IT. Yes, the AV/IT train is moving down the track at high speed propelled by Moore’s law and the eight forces outlined in Chapter 1. The future looks bright for A/V system design, and it will be a fun ride over the next few years as the remaining transition issues get settled. Keep your saw sharp, and you will not be left behind as our industry moves forward. The next chapter outlines traditional A/V technology without special attention to IT. If you are new to A/V, then this chapter will be of interest.
1 This case study is based on information provided by Larry Reid, Sr., Director/Chief Broadcast Technology Officer of KQED.
2 Much of this original material is based on a presentation (and private discussions) given at the NYC SMPTE Technical Conference on November 12, 2003, by Thomas Edwards, then Senior Manager, Interconnection Engineering, PBS. Thanks also to Jerry Butler for updates in 2008.
3 The materials for this case study were provided by Clyde Smith, Senior VP, Broadcasting Engineering R&D, QA and Metrics of the Turner Broadcasting System; Naveed Aslam, Senior Director, Broadcast Technology and Engineering; Jack Gary, Director Projects and Integration Engineering; and Rick Ackermans, Director of Engineering, Turner Broadcasting Systems Network Operations.