i
i
i
i
i
i
i
i
15
Using
Multimedia
in VR
No VR environment would be complete without the ability to play movies,
DVDs, feed live video, work with video cameras or carry out multichannel
videoconferencing and indulge in a bit of distributed or augmented reality.
As soon as one hears the terms live video, DVD movies etc., one shudders
at the thought of the complexity and size of the software requirements. Just to
write the decoding functions for a DVD player application is a daunting task.
New encoding and compressing algorithms
1
are cropping up all the time, and
it is quite impossible to write every application so that it can cope with every
codec that it is likely to encounter.
It doesn’t help either when you realize that there is a different aspect to
the meaning of real-time when it comes to constructing players for video and
audio material. We have been discussing real-time 3D graphics, and if we
can render our view of the 3D scene in less than 25 ms, for all intents and
purposes, it is real time. But what if we can’t meet that target? It plays a
little slower; 15 frames per second is still acceptable. After that, we have to
compromise and reduce the quality of the rendering or the complexity of
the scene. For audio and video work, slow-motion replay is unacceptable.
Imagine watching a whole movie in slow motion. Or, have you ever listened
to a slowed down tape recording? In these technologies, the best thing one
can do is drop frames or leave out bits of the audio. We need a different
approach to real-time programming, where essentially what one is doing is
keeping in sync with a real-time clock. If your DVD player program takes
a few milliseconds longer to render the frame, it will have to adapt by, say,
1
In the context of TV, video, DVDs, AVI files, streaming movies etc. the universally
adopted term for the computer code that implements the encoding and decoding process is
codec.
385
i
i
i
i
i
i
i
i
386 15. Using Multimedia in VR
omitting the occasional frame. If your player is running on a faster processor,
it must not play the movie as if in fast forward, either; it must pause and wait
to synchronize with the real-time clock.
The good news is that every PC made today has the processing power
to easily play back movies at the required real-time speed, so the only
complication we have is to ensure that they play smoothly and don’t slow
down or speed up due to some big calculation being done in the back ground.
Gathering these thoughts together brings us to the subject of this chapter,
DirectShow. We mentioned this briefly in Chapter 12 when discussing appli-
cation program tools and templates. DirectShow is a powerful and versatile
API for multimedia applications. It facilitates acquiring data from sources,
processing the data and delivering it to a destination. It also has a nomen-
clature, a jargon, in which one can discuss and describe what is going on and
what one would like to do. DirectShow jargon speaks of signals coming from
sources passing through filters en route from sources to sinks. The signal path
may split to pass through different filters or to allow multiple outputs. For
example, when playing a movie, the video goes to the screen, and the audio
goes to the loudspeakers. DirectShow uses a neat graphical metaphor, called
a FilterGraph, that depicts the filter components and their connections (such
as those shown in Figure 15.1). It takes this metaphor further, to the level
of the API, where it is the job of the application program to use the COM
interfaces of the DirectShow API to build a FilterGraph to do what the ap-
plication wants to do (play a movie, control a DV camcorder etc.) and send
the signals through it. DirectShow had a slow start. It began life under the
title ActiveMovie, but in the early days, it was not popular, because multime-
dia application developers could do everything they wanted using either the
media control interface (MCI) or video for Windows (VfW). It wasn’t until
the multimedia explosion with its huge video files and the easy attachment
of video devices to a PC, none of which really worked in the MCI or VfW
approach, that DirectShow came into its own. Now, and especially in the case
of VR where there are few off-the-shelf packages available, when we need to
incorporate multimedia into an application, DirectShow offers the best way
to do so.
15.1 A Brief Overview of DirectShow
Meeting DirectShow for the first time, it can seem like an alien programming
concept. The terms filter, signal and FilterGraph are not traditional program-
ming ideas, and there are other complicating issues.
i
i
i
i
i
i
i
i
15.1. A Brief Overview of DirectShow 387
In DirectShow, an application program does not execute its task explic-
itly. Instead, it builds a conceptual structure. For example, the FilterGr aph
contains components which are called the filters. The application then con-
nects the filters to form a chain from a source to a sink. The route from
source to sink is essentially a signal path, and in the context of a multime-
dia application, the signal consists of either audio or video samples. Once
the FilterGraph is built, the application program instructs it to move samples
through the filters from source to sink. The application can tell the graph
to do many things to the data stream, including pausing the progress of the
samples through the filter chain. But just what is appropriate depends on the
nature of the source, sink and filters in between.
To make the idea of a FilterGraph and the signal processing analogy a
little more tangible, we will quote a few examples of what might be achieved
in an application program using DirectShow FilterGraphs:
• The pictures from a video camera are shown on the computer’s
monitor.
• A DVD movie is played on the computer’s output monitor.
• The output from a video camera is saved to a multimedia file on disk.
• A movie or DVD is decoded and stored on the computer’s hard drive
using a high compression scheme.
• A series of bitmap pictures is combined to form a movie file.
• Frames from a movie are saved to bitmap image files on disk.
• An audio track is added to or removed from a movie file.
Figure 15.1 shows diagrammatically how some of these FilterGraphs might
look. Indeed, the DirectShow SDK includes a helpful little application pro-
gram called GraphEdit which allows you to interactively build and test Fil-
terGraphs for all kinds of multimedia applications. The user interface for
GraphEdit is illustrated in Figure 15.2. The four filters in F igure 15.1 per-
form four different tasks. In Figure 15.1(a), the output of a digital video
recorder is compressed and written to an AVI file. The signal is split so that
it can be previewed on the computer’s monitor. In Figure 15.1(b), a video
signal from a camera is mixed with a video from a file and written back to
disk. A preview signal is split off the data stream. The mixer unit can insert a
number of effects in real time. In Figure 15.1(c), a video stream from an AVI
i
i
i
i
i
i
i
i
388 15. Using Multimedia in VR
Figure 15.1. An example of four FilterGraphs.
file is combined with a live audio commentary. The signals are encoded to the
DVD standard, combined and written to DVD. In Figure 15.1(d), two video
sources from disk files are mixed and written back to file with simultaneous
preview.
Figure 15.2. The GraphEdit interactive tool for testing DirectShow FilterGraphs. It
allows the program’s author to use any of the registered filters available on the com-
puter, connect them together and experiment with their action.
i
i
i
i
i
i
i
i
15.1. A Brief Overview of DirectShow 389
We could continue to list many more applications of DirectShow. The
fact that all of them arise out of mixing and matching a small (but extensi-
ble) set of FilterG raph components illustrates the power of the concept. We
can also see a pattern emerging. Every FilterGr aph starts with a source that
generates a stream of samples: video frames from a camera, images in a set
of bitmap files etc. It ends with a sink: a window on the monitor, some
additional bytes appended to a recompressed movie file etc. In between the
source and sink, the samples pass through a filter where they can be altered
in some way. For example, the DVD MPEG2 encoded data stream might be
decoded or the color balance in a video signal compensated for poor lighting
conditions etc.
DirectShow identifies three types of filter:
1. Source filters. These could be a video camera, a movie file, a microphone
or an audio file.
2. Sink filters. A video renderer. A renderer filter is a filter that displays
video pictures or plays audio, or in fact, any filter that presents multi-
media content. A file writer filter stores the media samples to disk.
3. In-place filters. These change the data stream in some way. They may
even split the data stream so that for example the output of a video
camera can be previewed on the monitor at the same time as it written
to an AVI file.
When filters are linked together, media samples are said to travel from an
upstream filter (such as a source) to a downstream filter (such as a sink). Al-
though we have classified both video cameras and movie file readers as source
filters, there are some significant differences. At the other end of the filter
chain, the types of output filters have their differences, too. A file writer is
different from a window presenting live video grabbed from a webcam, for
example. Rather than thinking about a subclass of filters, D irectShow intro-
duces the idea of a graph as conforming to a push model or a pull model. In
a push mode of operation, the source or upstream filter is the driver; it sets
the timing. In a pull FilterGraph, the downstream filter sets the timing and
controls the flow of samples. A typical pull graph application would be a me-
dia file player (movie or audio), because the presentation to the viewer (done
by the filter at the end of the chain) must appear as smooth as possible and
not play at a speed dictated by the speed at which the file can be read. A push
model would apply, for example, when capturing live video from a camera to
an AVI file.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.