i
i
i
i
i
i
i
i
18.5. A Multi-Projector VR Theater System 517
send the part of the image that the slave is to display. The second
thread will handle control messages to be sent to the client.
The slave also starts two communications threads. One will retrieve the
image from the master; the other waits for commands from the master
and posts messages into the message loop to execute those commands.
• When the master is instructed to display an image, it reads and decodes
the pixel data into an RAM buffer (stereoscopic images are stored in
left-over-right format). This is labeled 1 in Figure 18.7.
• The master sends a control signal to the slave to tell it that a new image
is ready to be grabbed (Step 2).
• The slave tells the master to dispatch the part of the image it needs
to form the peripheral edges of the display (Step 3). The slaves local
buffer is filled (Step 4).
• The master and slave a pply the distortion and blendin g to their re-
spective parts of the image and send the output to their pair of display
projectors (Steps 5a and 5b).
18.5.2 Distortion and Blending
Distor tion and blending are configured manually using a test image and grid.
The parts of the image, as shown in Figure 18.6, are rendered as a texture
on a quadrilateral mesh. By interactively moving the corners of the mesh
and controlling the bending along the edges, the shape of the projection is
altered to facilitate some overlap between adjacent projections and correct for
a nonflat screen. The application program and these features are partially
shown in action in Figure 4.11. The system only has to be configured once.
Parameters to control blending and overlap between adjacent projections are
recorded in a file. This data is used to automatically configure the projections
each time the programs start.
Displaying images and movies as objects painted with an OpenGL texture
works very well. It may be used for a stereoscopic display and can easily cope
with the refresh rates required for video output. Rendering the output of a
3D graphics program requires an additional step. Since we must render an
OpenGL-textured mesh to form the output, we cannot render a 3D scene
into the same output buffers. Thus, we do not render the scene into either
the front or back frame buffers; instead, we render it into an auxiliary buffer.
i
i
i
i
i
i
i
i
518 18. Building on the Basics, Some Projects in VR
(Most graphics adapters that support OpenGL have at least two auxiliary
buffers.) The content of the auxiliary buffer is then used as the source of
a texture that will be painted onto the mesh object. This may involve an
extra step, but it does not usually slow down the application, because none of
these steps involve the host processor. Everything happens within the graphics
adapter.
18.5.3 Synchronization
Synchronization, command passing and data distribution between the mas-
ter and slave programs are accomplished using TCP/IP
5
over Ethernet. Our
application uses the Windows sockets
6
programming interface, and because
it is multi-threaded, we don’t have to worry too much about the blocking ac-
tion that used to beset Windows sockets programming. The master and slave
programs open listening sockets on high-numbered ports and wait for data or
commands to be sent from the other side.
We use our own simple protocol for sending commands by encoding the
action into an eight-byte chunk; for example, to initiate a data transfer. The
slave’s control thread waits in an infinite loop using the
recv(...) function,
which only returns once it gets eight bytes of data from the server which were
sent using the
send(..) function. The slave acts upon the command by
using PostMessage(...) to place a standard Windows WM COMMAND mes-
sage into the message loop. When the slave’s control thread has posted the
message, it goes back to sleep, again waiting on the
recv(...) function
for the next command. All the other master/slave conversations follow this
model. When requesting that an image be sent by the master to the slave, the
eight-byte data chunk tells the slave how much data to expect and the two
programs cooperate to send the whole image. The TCP protocol guarantees
that no data is lost in the transmission.
At first glance, the code for all these client-server conversations look quite
complex, but that is only because of the care that needs to be taken to check
for errors and unexpected events such as the connections failing during trans-
mission. In fact, we really only need to use four functions to send a message
from one program to another:
socket(..) creates the socket, connect()
5
Transmission Control Protocol/Internet P rotocol is a standard used for the communica-
tion between computers, in particular for transmitting data over networks for all Internet-
connected machines.
6
Windows sockets (Winsock) is a specification that defines how Windows network soft-
ware should access network services, particularly TCP/IP.
i
i
i
i
i
i
i
i
18.5. A Multi-Projector VR Theater System 519
connects through the socket with a listening program, send(...) sends the
data and
closesocket(..) terminates the process.
The receiving code is a little more complex because it usually runs in a
separate thread, but again it basically consists of creating a socket, binding the
listener to that socket with
bind(..), accepting connections on that socket
with accept(..) and receiving the data with recv(..).
Along with the main programs, you will find an example of a small pair
of message sending and receiving programs. This illustrates in a minimal way
how we u se the Windows sockets for communication over TCP/IP.
So far, we have concentrated on displaying image sequences using two
PCs and four video outputs. To extend this idea to similar applications that
present panoramic movies and 3D content poses some difficulties:
• Movies must present their images at rates of 30 fps. Even the smallest
delay between the presentation of the central and peripheral parts of
the frame would be quite intolerable. Wide-screen images require a
high pixel count, too; for example, a stereoscopic movie is likely to
require a frame size of 3200 × 1200 pixels, at the minimum. The
movie player project we include with the book restricts the display to a
two-output stereoscopic wide-screen resolution of 1600 × 1200, so it
requires only a single host. To go all the way to a four output movie
player, it would be necessary to render two movies, one for the center
and one for the periphery. Each would need to be stored on its own PC,
and the master and slave control signals designed to keep the players in
synchronism.
• In a four-output real-time 3D design, the communications channel
would need to pass the numerical descriptions of the 3D scene and
image maps from master to slave. Under most conditions, this data
distribution would only need to be done in an initialization stage. This
transfer might involve a considerable quantity of data, but during nor-
mal usage (with interaction or scripted action, for example), a much
smaller amount of data would need to be transferred—carr y ing such
details as viewpoint location, orientation, lighting conditions and ob-
ject locations. Our example project code restricts itself to using a single
host and projecting onto a curved screen with two projectors.
In this section, we have highlighted some key features of a suite of core
programs to deliver a multi-projector cave-type immersive virtual environ-
i
i
i
i
i
i
i
i
520 18. Building on the Basics, Some Projects in VR
ment. Larger commercial systems that can cost in excess of $100k will typi-
cally implement these functions in custom hardware. However, the common
PC’s processing and GPU capability now allow anyone with the right soft-
ware to experiment with VR caves at a fraction of that cost. More details of
how this project works are included in the commentary on the CD.
18.6 Using Image-Processing and
Computer-Vision Libraries
In Section 8.4, the OpenCV and Vision SDK software libraries were intro-
duced. These two libraries can be easily integrated into programs that need
special purpose functionality such as overlays and removing perspective dis-
tortion. Since the subjects of image processing and computer vision could
each fill a book, we are only going to pro vide a framework for using OpenCV,
typically for processing moving images.
Our examples come in the form of a special inline DirectShow filter and
two short applications programs that use the filter. The programs comprise:
• Afilter. As part of the OpenCV distribution [6], a DirectShow fil-
ter called ProxyTrans provides a method to apply any of the OpenCV
image processing functions. Unfortunately, it didn’t work for us, and
so we provide our own simpler code based on a minor modification
of the
EZRGB24 sample that ships as part of DirectX. By adding one
additional interface method to the EZRGB24 project, our filter allows
any program which inserts it into a FilterGraph datastream to define
a callback function. The callback function is passed as a pointer to
an OpenCV image structure representing the image samples. Once we
have access to every image sample passing through the FilterGraph, it is
possible to apply any OpenCV process to it. Please consult the readme
file for details of ho w the component is registered with the operating
system. A registered DirectShow filter may be used by any program
(and in the
graphedit utility).
• AVI processor. This example shows how to use our filter in a scenario
where we wish to process a movie stored in an AVI file. It previews the
action of the image process and then writes it into another AVI file.
i
i
i
i
i
i
i
i
18.7. Augmented Reality (AR) 521
With the exception of illustrating how to instantiate our filter com-
ponent and use the callback function, all the elements of the program
have been discussed in earlier chapters. The same applies to our next
program.
• Camera-processor. This example illustrates how to apply OpenCV im-
age processing functions to a real-time video source.
18.7 Augmented Reality (AR)
In Chapter 8, we discussed virtual-reality videoconferencing, where the idea
is to map video images onto virtual screens. This is one form of an exciting
series of recent developments going broadly under the heading of AR. We can
put together a small project by using the ARToolKit [1] to illustrate another
form of AR: that is, placing virtual objects into real-world scenes.
The ARToolKit is a group of functions that allows us to easily give
the illusion that virtual elements actually exist in the real environment
around us. This can be done stereoscopically, but with a simple webcam
and a cheap set of video glasses (su ch as the Olympus Eye-Trek that are nor-
mally used for watching TV or DVDs), we can achieve the same effect (see
Figure 18.8).
In this project, we will adapt one of the ARToolKit’s simple examples so
that a 3D mesh model may be placed virtually into the real world we see
before us, as for example in Figure 18.9.
Figure 18.8. The combination of a basic webcam stuck to a simple pair of video
classes forms a low cost form of head mounted display suitable for the augmented
reality project.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.