135
9
AutomaticDynamicStereoscopic3D
Jason Hughes
Steel Penny Games
Stereoscopic rendering is not new. In film, it has been used for years to make
experiences more immersive and exciting, and indeed, the first theatrical showing
of a 3D film was in 1922 [Zone 2007]. A cinematographer can design scenes for
film that take advantage of the 3D projection without unduly taxing the audience,
intentionally avoiding the various inherent issues that exist with a limited field
projection of stereoscopic content. Further, an editor can trim unwanted frames
or even modify the content to mask objectionable content within a frame.
Interactive stereoscopic 3D (S3D), however, is fairly new territory. Those
same problems that filmmakers consciously avoid must be handled by clever
programmers or designers. Sometimes, this means changing some camera angles,
or adjusting a few parameters in certain areas of an environment. Sometimes this
is not an option, or it may be infeasible to change the content of a title for S3D
purposes. It’s likely that some visual issues that arise from player control are bet-
ter dealt with rather than avoided, such as window violations. While some re-
search is available to guide an implementation, practical examples that handle
dynamic environments well are hard to come by. This is an exciting time for
game programmers: some experimentation and creativity is actually required on
our parts until best practices become standard. Here follows the results of our
work with Bluepoint Games in porting the excellent Shadow of the Colossus and
ICO to PlayStation 3 in full HD and S3D.
9.1GeneralProblemsinS3D
All stereoscopic content, whether film or games, must be especially mindful of
the audience’s experience. Some problems that must be avoided in any S3D ex-
perience are described in this section.
136
Figure 9
.
does not.
Windo
w
A wind
o
the scre
e
front of
lax (clo
s
the brai
n
has no
demons
t
Conve
r
Unlike
w
to refoc
u
for peop
periods
o
the vie
w
It is
jects gr
o
mind, it
constant
3D effe
c
.
1. A window
w
Violatio
n
o
w violation
o
e
n while not
and behind t
h
s
er to the vi
e
n
failing to
m
visual occlu
s
t
rates a wind
o
r
genceFati
g
w
hen viewin
g
u
s on objects
le to experie
n
o
f time. This
w
er’s eyes are
worth notin
g
o
w more dist
a
is generally
p
l
y poking ou
t
c
t.
violation occ
u
n
s
o
ccurs when
a
exactly at t
h
h
e focal plan
e
e
wer). This v
e
m
erge two im
a
s
ion to acc
o
o
w violation.
g
ue
g
2D images,
as they cha
n
n
ce radical c
h
muscular fat
i
stressed, the
y
g
that the ey
e
a
n
t
and incre
p
referable to
k
t
of the scree
n
rs when one e
y
a
n object cli
p
h
e focal plan
e
e
, it is more
d
e
ry irritating
,
a
ges of an o
b
o
unt for the
with S3D, t
h
ge in depth.
I
h
anges in foc
a
i
gue may no
t
y
will be har
d
e
’s angular
a
a
ses as they
k
eep objects
o
n
. This reduc
e
9.Automat
i
y
e sees part o
f
p
s against the
e
distance.
W
d
isturbing w
h
,
unnatural f
e
b
ject that it
p
missing inf
o
h
e audience
m
I
t is uncom
m
a
l depth, cert
a
t
be noticed i
m
d
-pressed to e
a
djustment e
f
approach th
e
o
f interest at
e
s eye strain,
i
cDynamicS
t
f
an object but
e
left or right
W
hile this ha
p
h
en in negati
v
e
eling is the
p
erceives par
t
o
rmation. Fi
g
m
ust use eye
m
on in natura
l
a
inly not for
e
m
mediately,
b
njoy content
f
fort decreas
e
e
viewer. Wi
t
a distance ra
t
b
ut still yiel
d
t
ereoscopic3
the other
edges of
p
pens in
v
e paral-
r
esult of
t
ially but
g
ure 9.1
muscles
l
settings
e
xtended
b
ut once
further.
e
s as ob-
t
h this in
t
her than
d
s a nice
D
9.1GeneralProblemsinS3D 137
The film industry has already learned that the audience becomes fatigued if
rapid shot changes force a refocusing effort to view the scene. It is unclear
whether this fatigue is due to the eye muscles being unable to cope with contin-
ued rapid convergence changes or whether the brain is more efficient at tracking
objects spatially over time but perhaps must work harder to identify objects ini-
tially and then tires quickly when constantly reparsing the scene from different
viewpoints. In any case, giving those shot changes relatively similar convergence
points from shot to shot helps reduce the stress placed on the audience.
Accommodation/ConvergenceDeviation
Another problem with S3D is when the brain interprets the focal distance to be
significantly closer or farther than the location of the TV screen. This happens
when tension placed on the muscles inside the eye that controls the focal point of
the eyes differs significantly from the expected tension placed on muscles outside
the eye that controls angular adjustment of the eyes. Figure 9.2 demonstrates a
situation where accommodation and convergence are different. When the brain
notices these values being in conflict with daily experience, the illusion of depth
begins to break down. Related factors seem to involve the content being dis-
played, the size of the screen, the distance that the viewer sits from the screen,
and the spacing between the viewer’s eyes. It’s complicated, but in short, don’t
push the effect too much, or it becomes less convincing. For a deeper examina-
tion of visual fatigue and how people perceive different content, see the report by
Mikšícek [2006].
Figure 9.2. Accommodation is the physical distance between the viewer and the display
surface. Convergence is the apparent distance between the observed object and the
viewer.
Covergence
Accommodation
138 9.AutomaticDynamicStereoscopic3D
9.2ProblemsinS3DUniquetoGames
In addition to the issues above, S3D rendering engines need to take various other
issues into account in order to present a convincing image to each eye at interac-
tive rates. The film industry has an easier time with these problems than games
do because the artist designing the scene can ensure optimal viewing conditions
for each frame of the movie. They can also afford to devote many minutes worth
of CPU cycles to each frame, whereas games get only a fraction of a second to
achieve acceptable results. Worse, games typically give the control to the player
and often do not have the luxury of predictably good viewing conditions in S3D
for every frame. A dynamic solution that is reactive to the current rendered scene
would be ideal.
KeystoneDistortions
It should be noted that simply rendering with two monoscopic projection matri-
ces that tilt inward around a forward axis creates keystone distortions. Figure 9.3
shows an exaggerated example of how incorrectly set up viewing matrices ap-
pear. This kind of distortion is similar to holding two sheets of paper slightly ro-
tated inward, then overlapped in 3D—the intersection is a line rather than a
plane, and it warps the image incorrectly for stereoscopic viewing. For the brain
Figure 9.3. Keystone distortion, also known as the tombstone effect, occurs when a rec-
tangle is warped into a trapezoidal shape, stretching the image. This can be seen when
pointing a projector at a wall at any nonperpendicular angle. This image shows how two
monoscopic projections tilted inward produce a final image that the brain cannot merge
as a single checkerboard square.
+=
9.2ProblemsinS3DUniquetoGames 139
to merge or fuse two images correctly requires that the focal plane be aligned in a
coplanar fashion between the two images. Instead, the correct projection matrices
are asymmetrical, off-axis projection matrices. These contain shearing so that the
corners of the focal plane exactly match between the left and right stereo images
but have a different point of origin. We don’t reproduce the math here because it
can be found elsewhere with better descriptions of the derivation [Bourke 1999,
Schertenleib 2010, Jones et al. 2001].
2DImage‐BasedEffects
A staple of modern monoscopic games are the image-based effects, commonly
called “post effects.” These cheap adjustments to rendered images make up for
the lack of supersampling hardware, low-resolution rendering buffers, inability
for game hardware to render high-quality motion blurs, poor-quality lighting and
shadowing with image-space occlusion mapping, blooms and glows, tone map-
ping, color gamut alterations, etc. It is a major component of a good-quality ren-
dering engine but, unfortunately, is no longer as valuable in S3D. Many of these
tricks do not really work well when the left/right images deviate significantly
because the alterations are from each projection’s fixed perspective. Blurring
around the edges of objects, for example, in an antialiasing pass causes those
edges to register more poorly with the viewer. Depending on the effect, it can be
quite jarring or strange to see pixels that appear to be floating or smeared in
space because they do not seem to fit. The result can be somewhat akin to win-
dow violations in the middle of the screen and, in large enough numbers, can be
very distracting. When considering that post effects take twice the processing
power to modify both images, a suggestion is to dial back these effects during
S3D rendering or remove them entirely since the hardware is already being
pushed twice as hard just to render the scene for both eyes separately.
HUDandSubtitles
The most difficult problem to solve for S3D and games is the heads-up display
(HUD). User interfaces (UIs) generally have some issues with current-generation
LCD monitors that do not change colors rapidly enough to prevent ghosting be-
tween eyes (but this technological problem will probably improve in the next few
years to become less of an issue), which makes interesting 3D UIs difficult to
enjoy with high-contrast colors.
A systemic problem, though, is where to draw subtitles and UI elements in
the game. If they are drawn at the focal plane, then there is no ghosting on LCD
monitors since both left and right images display the UI pixels in the same place.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.