33-D Shape Information Recovery

Stereo vision introduced in the previous chapter requires determining which point in one image corresponds to a given point in the other image. This is often a difficult problem in real situation. To avoid this problem, various techniques using different 3-D cues for information recovery based on a single camera (position is fixed but several images may be taken) are proposed, Pizlo (1992). Among the intrinsic properties of scene, the shape of a 3-D object is the most important. Using a single camera to recover the 3-D shape information is often called “shape from X,” in which X can represent the illumination change, shading, contour, texture, motion, and so on. Some early important work in this kind was conducted in the 1970s, Marr (1982).

The sections of this chapter are arranged as follows:

Section 3.1 introduces a photometric stereology method for determining the orientation of the surface in a scene using a series of images with the same viewing angle but different illumination.

Section 3.1 discusses the principle and technique of acquiring the surface orientation of a moving object by detecting and calculating the optical flow field.

Section 3.2 discusses how to reconstruct the surface shape of the object according to the different image tones produced by the spatial variation of the brightness on the surface.

Section 3.3 presents the principles of three techniques for restoring the surface orientation based on the change (distortion) of the surface texture elements after the imaging projection.

Section 3.4 describes the relationship of object depth with the focal distance changes caused by focusing on the objects of different distances. This indicates that the object distance can be determined based on the focal length making the object clear.

Section 3.5 introduces a method for computing the geometry and pose of a 3-D object using the coordinates of three points in an image under the condition that the 3-D scene model and the focal length of the camera are known.

3.1Photometric Stereo

Photometric stereo is an important method for recovering surface orientation. This method requires a set of images, which are taken from the same viewing angles but with different lighting conditions. This method is easy to implement, but requires the control of lighting in the application environment.

3.1.1Scene Radiance and Image Irradiance

Scene radiance and image irradiance are related but different concepts. The former is the power emitted from a unit solid angle per unit area on the surface of the light source, with the unit Wm−2sr−1. The latter is the power per unit area on the surface of the object, with the unit Wm−2.

3.1.1.1The Relationship Between Scene Radiance and Image Irradiance

Now consider the relationship between the radiance at a point on an object in the scene and the irradiance at the corresponding point in the image, Horn (1986). As shown in Figure 3.1, a lens of diameter d is located at a distance λ from the image plane. Let a patch on the surface of the object have an area δO, while the corresponding image patch has an area δI. It is supposed that the ray from the object patch to the center of the lens makes an angle α with the optical axis and that there is an angle 0 between this ray and a surface normal. The object patch is at a distance z from the lens (−z means that the direction from lens to object points to −Z), measured along the optical axis.

The solid angle of the cone of the rays leading to the patch on the object is equivalent to the solid angle of the cone of the rays leading to the corresponding patch in the image. The apparent area of the image patch as seen from the center of the lens is δI × cos α. The distance of this patch from the center of the lens is λ/cos α. The solid angle subtended by this patch is δI × cos α/(λ/cos α)2. The solid angle of the patch on the object as seen from the center of the lens is δO × cos θ/(z/cos α)2. The equivalence of the two solid angles gives

$\frac{\text{δ}O}{\text{δ}I}=\frac{\cos \alpha }{\cos \theta }{\left(\frac{z}{\lambda }\right)}^2(3.1)$

Since the lens area is π(d/2)2, the solid angle seen from the object patch is

$\Omega =\frac{\pi d^2}{4}\cos \alpha \frac{1}{{(z/\cos \alpha )}^2}=\frac{\pi }{4}{\left(\frac{d}{z}\right)}^2{\cos }^3\alpha (3.2)$

The power of the light originating from the patch and passing through the lens is

$\text{δ}P=L\times \text{δ}O\times \Omega \times \cos \theta =L\times \text{δ}O\times \frac{\pi }{4}{\left(\frac{d}{z}\right)}^2{\cos }^3\alpha \cos \theta (3.3)$

where L is the radiance of the surface in the direction toward the lens. Since no light from other areas reaches this image patch, the irradiance of the image at the patch is

$E=\frac{\text{δ}P}{\text{δ}I}=L\times \frac{\text{δ}O}{\text{δ}I}\times \frac{\pi }{4}{\left(\frac{d}{z}\right)}^2{\cos }^3\alpha \cos \theta (3.4)$

Taking eq. (3.1) into eq. (3.4) yields

$E=L\frac{\pi }{4}{\left(\frac{d}{\lambda }\right)}^2{\cos }^4\alpha (3.5)$

The image irradiance E is proportional to scene radiance L and the diameter of the lens d, and is inversely proportional to the focus of the lens λ.

When the object is fixed, L can be considered as a constant, so can λ and d. If the camera moves, the change of the irradiance will be reflected on the angle α, that is,

$\begin{array}{l}\Delta E=L\frac{\pi }{4}{\left(\frac{d}{\lambda }\right)}^2\left[{\cos }^4\alpha -{\cos }^4(\alpha +\Delta \alpha )\right]\\ =L\frac{\pi }{4}{\left(\frac{d}{\lambda }\right)}^2\left[{\cos }^2\alpha +{\cos }^2(\alpha +\Delta \alpha )\right]\left[{\cos }^2\alpha -{\cos }^2(\alpha +\Delta \alpha )\right]\\ =L\frac{\pi }{4}{\left(\frac{d}{\lambda }\right)}^2\left[{\cos }^2\alpha +{\cos }^2(\alpha +\Delta \alpha )\right]\left[\cos \alpha +\cos (\alpha +\Delta \alpha )\right]\left[\cos \alpha -\cos (\alpha +\Delta \alpha )\right]\\ =-2L\frac{\pi }{4}{\left(\frac{d}{\lambda }\right)}^2\left[{\cos }^2\alpha +{\cos }^2(\alpha +\Delta \alpha )\right]\left[\cos \alpha +\cos (\alpha +\Delta \alpha )\right]\left[\sin (\alpha +\frac{\pi }{2})\sin (\Delta \alpha /2)\right]\\ (3.6)\end{array}$

As Δα is quite small, the above equation becomes

$\Delta E\approx -L\frac{\pi }{2}{\left(\frac{d}{f}\right)}^2\left[4{\cos }^2\alpha \cos \alpha \sin \alpha \times \Delta \alpha /2\right]=-L\pi {\left(\frac{d}{f}\right)}^2{\cos }^2\alpha \sin (2\alpha )\times \Delta \alpha (3.7)$

The angular velocity of the camera is

$v=\frac{\Delta \alpha }{\Delta t}=\frac{\Delta E/\Delta E}{-L\pi {\left(\frac{d}{f}\right)}^2{\cos }^2\alpha \sin (2\alpha )}(3.8)$

which is propositional to the change rate of irradiance ΔE/Δt.

3.1.1.2Bi-directional Reflectance Distribution Function

Consider the coordinate system shown in Figure 3.2, where N is the normal vector of the surface patch, OR is an arbitrary reference line, and the direction of a ray I can be represented by an angle θ (polar angle, between I and N) and angle ϕ (azimuth, between a perpendicular projection of the ray onto the surface and OR on the surface).

Using Figure 3.2, one ray falling on the surface can be specified by the direction (θi, ϕi) and one ray toward the viewer can be specified by the direction (θe, ϕe), as shown in Figure 3.3.

The bi-directional reflectance distribution function (BRDF) can be denoted f(θi, ϕi; θe, ϕe). This function represents how bright a surface appears when it is viewed from one direction while the light falls on it from another. It is the ratio of the radiance δL(θe, ϕe) to the irradiance δE(θi, ϕi), given by

$f({\theta }_i,{\varphi }_i;{\theta }_e,{\varphi }_e)=\frac{\text{δ}L({\theta }_e,{\varphi }_e)}{\text{δ}E({\theta }_i,{\varphi }_{\text{i}})}(3.9)$

Consider now the extended source (see Section 2.3 of Volume I in this book set). In Figure 3.4, the solid angle, corresponding to a patch of size δθi in the polar angle and size δϕi in the azimuth angle, is $\delta \omega =\sin {\theta }_l\delta {\theta }_l\delta {\varphi }_i.$ Denote Eo(δi, ϕi) as the radiance per unit solid angle coming from the direction (θi, ϕi), the radiance from the patch is Eo(θi, ϕi) sin ${\theta }_i\delta {\theta }_i\delta {\varphi }_i$ and the total irradiance of the surface is

$E=\underset{-\pi }{\overset{\pi }{{\displaystyle \int }}}\underset{0}{\overset{\pi /2}{{\displaystyle \int }}}{E}_{\text{o}}({\theta }_i,{\phi }_{\text{i}})\sin {\theta }_i\cos {\theta }_i\text{d}{\theta }_i\text{d}{\phi }_i(3.10)$

To obtain the radiance for the whole surface, the products of the BRDF and the irradiance from all possible directions are integrated. This gives the radiance in the direction toward the viewer by

$L({\theta }_e,{\varphi }_e)=\underset{-\pi }{\overset{\pi }{{\displaystyle \int }}}\underset{0}{\overset{\pi /2}{{\displaystyle \int }}}f({\theta }_i,{\varphi }_{\text{i}};{\theta }_e,{\varphi }_e)E_{\text{o}}({\theta }_i,{\varphi }_i)\sin {\theta }_i\cos {\theta }_i\text{d}{\theta }_i\text{d}{\varphi }_i(3.11)$

3.1.2Surface Reflectance Properties

Consider two extreme cases: An ideal scatter surface and an ideal specular reflection surface. Horn (1986).

An ideal scatter surface is also called a Lambertian surface. It appears equally bright from all viewing directions and reflects all incident light. From this definition. its BRDF must be a constant. Integrating the radiance of the surface over all directions and equating to the total radiance. the total irradiance is given by

$\underset{-\pi }{\overset{\pi }{{\displaystyle \int }}}\underset{0}{\overset{\pi /2}{{\displaystyle \int }}}f({\theta }_i,{\varphi }_i;{\theta }_e,{\varphi }_e)E({\theta }_i,{\varphi }_i)\cos {\theta }_i\sin {\theta }_e\cos {\theta }_e\text{d}{\theta }_e\text{d}{\varphi }_e=E({\theta }_i,{\varphi }_{\text{i}})\cos {\theta }_i(3.12)$

Multiplying both sides by cos θi to convert them to a direction N, the BRDF of ideal scatter surface is

$f({\theta }_{\text{i}},{\varphi }_i;{\theta }_e,{\varphi }_e)=1/\pi (3.13)$

For an ideal scatter surface, the radiance L and the irradiance E have the relation given by

$L=E/\pi (3.14)$

Suppose that an ideal Lambertian surface is illuminated by a point source of a radiance E, which is located in the direction (θs, ϕs) and has the radiance

$E({\theta }_{\text{i}},{\varphi }_i)=E\frac{\delta ({\theta }_i-{\theta }_{\text{S}})\delta ({\varphi }_i-{\varphi }_{\text{S}})}{\sin {\theta }_i}(3.15)$

It is then derived from eq. (3.13) that

$L=\frac{1}{\text{π}}E\cos {\theta }_i{\theta }_i\ge 0(3.16)$

Equation (3.16) is called Lambert’s law.

If an ideal Lambert surface is under a uniform radiance E, then

$L=\underset{-\pi }{\overset{\pi }{{\displaystyle \int }}}\underset{0}{\overset{\pi /2}{{\displaystyle \int }}}\frac{E}{\pi }\sin {\theta }_i\cos {\theta }_i\text{d}{\theta }_i\text{d}{\varphi }_i=E(3.17)$

Equation (3.17) shows that the radiance of the patch is the same as that of the source.

An ideal specular reflection surface can reflect all the light arriving from the direction (θi, ϕi) into the direction (θe, ϕe), as shown in Figure 3.5. The BRDF is proportional to the product of the two impulses δ(θe − θi) and δ(ϕe − ϕi − π) with a factor k.

The total radiance is obtained by integrating over all emittance directions of the surface, given by

$\underset{-\pi }{\overset{\pi }{{\displaystyle \int }}}\underset{0}{\overset{\pi /2}{{\displaystyle \int }}}k\delta ({\theta }_e-{\theta }_i)\delta ({\varphi }_e-{\varphi }_{\text{i}}-\pi )\sin {\theta }_e\cos {\theta }_e\text{d}{\theta }_e\text{d}{\varphi }_e=k\sin {\theta }_{\text{i}}\cos {\theta }_i=1(3.18)$

The BRDF is then solved as

$f({\theta }_i,{\varphi }_i;{\theta }_e,{\varphi }_e)=\frac{\text{δ}({\theta }_e-{\theta }_i)\text{δ}({\varphi }_e-{\varphi }_i-\text{π})}{\sin {\theta }_{\text{i}}\cos {\theta }_i}(3.19)$

For an extended source, taking eq. (3.19) into eq. (3.11) yields

3.1.3Surface Orientation

A smooth surface has a tangent plane at every point. The direction of the tangent plane can be used to represent the direction of the plane at this point. The surface normal can be used to specify the orientation of this plane. In practice, the coordinate system is chosen such that one axis is lined up with the optical axis of the imaging system while the other two axes are parallel to the image plane, as shown in Figure 3.6. A surface is then described in terms of its perpendicular distance −z from some reference planes parallel to the image plane.

The surface normal can be found by taking the cross-product of any two (nonparallel) lines in the tangent plane, as shown in Figure 3.7.

Taking a small step δx in the x-direction from a given point (x, y), the change in z is δz = δx × ∂z/∂x + e, where e represents a higher-order term. Denote the first-order partial derivatives of z with respect to x and y, p and q, respectively. Look at Figure 3.7. The change along the z-direction is pδx when δx is taken in the x-direction and the change along the z-direction is qδy when δy is taken in the y-direction. The former can be written as [δx 0 pδx]T, which is parallel to rx = [1 0 p]T, and the latter can be written as [δx 0 pδx]T, which is parallel to ry= [0 1 q]T. The surface normal (it points toward the viewer) is then given by

$N=r_x\times r_y={[\begin{array}{ccc}1& 0& p\end{array}]}^{\text{T}}\times {[\begin{array}{ccc}0& 1& q\end{array}]}^{\text{T}}={[\begin{array}{ccc}-p& -q& 1\end{array}]}^{\text{T}}(3.21)$

A unit vector is obtained by

$\widehat{N}=\frac{N}{\left|N\right|}=\frac{{[-p-q1]}^T}{\sqrt{1+p^2+q^2}}.(3.22)$

The angle θe between the surface normal and the direction to the lens is obtained by taking the dot-product of the (unit) surface normal vector and the (unit) view vector (from the object to the lens) [0 0 1] T

$\widehat{N}•\widehat{V}=\cos {\theta }_e=\frac{1}{\sqrt{1+p^2+q^2}}.(3.23)$

3.1.4Reflectance Map and Image Irradiance Equation

Now consider the link between the pixel gray scale (image brightness) and the pixel gradient (surface orientation).

3.1.4.1Reflectance Map

When a point source of a radiance E illuminates a Lambertian surface, the scene radiance is

$L=\frac{1}{\pi }E\text{cos}{\theta }_i{\theta }_i\ge \text{O}(3.24)$

where θi is the angle between the surface normal vector [−p − q 1]T and the direction vector toward the source [−ps − qs 1]T. The dot-product is given by

$\cos {\theta }_i=\frac{1+p_{s}p+q_{s}q}{\sqrt{1+p^2+q^2}\sqrt{1+p_{\text{s}}^2+q_{\text{s}}^2}}(3.25)$

Taking eq. (3.25) into eq. (3.24) gives the relation between the object brightness and surface orientation. Such a relation function is called the reflectance map and is denoted R(p, q).

For the Lambertian surface illuminated by a single distant point source, its reflectance map is

$R(p,q)=\frac{1+p_{s}p+q_{s}q}{\sqrt{1+p^2+q^2}\sqrt{1+p_{\text{s}}^2+q_{\text{s}}^2}}(3.26)$

The reflectance map can be represented by a contour map in the gradient space. According to eq. (3.26), contours of constant brightness are nested conic sections in the PQ plane, since R(p, q) = c implies ${\left(1+p_{s}p+q_{s}q\right)}^2=c^2\left(1+p^2+q^2\right)\left(1+p_s^2+q_s^2\right).$ The maximum of R(p, q) is achieved at (ps, qs).

Example 3.1 Illustration of the reflectance map of the Lambertian surface.

Three examples of reflectance maps of the Lambertian surface are shown in Figure 3.8. Figure 3.8(a) shows the case of ps = 0, qs = 0 (nested circles). Figure 3.8(b) shows the case of ps ≠ 0, qs = 0 (ellipse or hyperbola). Figure 3.8(c) shows the case of ps ≠ 0, qs ≠ 0 (hyperbola).

Different from the Lambertian surface, a surface that emits radiation equally in all directions can be called a homogeneous emission surface. Such a surface appears brighter when viewed obliquely. This is because the visible surface is reduced when the radiance of the same power comes from a foreshortened area. In this case, the brightness depends on the inverse of the cosine of the emittance angle. Since the radiance is proportional to cos θi/cos θe, and cos θe = 1/(1 + p2 + q2)1/2, so

$R(p,q)=\frac{1+p_{s}p+q_{s}q}{\sqrt{1+p_{\text{s}}^2+q_{\text{s}}^2}}(3.27)$

The contours with constant brightness are now parallel lines, because from R(p, q) = c, $\left(1+p_{s}p+q_{s}q\right)=c\left(1+p_s^2+q_s^2\right)$ will be obtained. These lines are orthogonal to the direction of (ps, qs).

Example 3.2 The reflectance map of a homogeneous emission surface.

Figure 3.9 illustrates an example of the reflectance map of a homogeneous emission surface. Here, ps/qs = 1/2, and the contour lines have a slope of 2.

3.1.4.2Image Constraint Equation

The irradiance of a point in an image, E(x, y), is propositional to the radiance at the corresponding point on the object in the scene. Suppose that the surface gradient at this point is (p, q) and the radiance there is R(p, q). Setting the constant of proportionality to one yields

$E(x,y)=R(p,q)(3.28)$

This is the image brightness constraint equation or image irradiance equation. It indicates that the gray-level value I(x, y) at the pixel (x, y) depends on its reflectance property R(p, q).

Consider a sphere with a Lambertian surface illuminated by a point source at the same place as the viewer (see Figure 3.10).

As there are θe = θi and (ps, qs) = (0, 0), the relation between the radiance and gradient is given by

$R(p,q)=\frac{1}{\sqrt{1+p^2+q^2}}(3.29)$

If the center of the sphere is on the optical axis, its surface equation is

$z=z_0+\sqrt{r^2-(x^2+y^2)}{x}^2+y^2⩽r^2(3.30)$

where r is the radius of the sphere and −z0 is the distance from the center of the sphere to the lens.

Since p = −x/(z − z0) and q = −y/(z − z0), it has then (1 + p2 + q2)1/2 = r/(z − z0), and finally

$E(x,y)=R(p,q)=\sqrt{1-\frac{x^2+y^2}{r^2}}(3.31)$

According to eq. (3.31), the brightness falls off in image, from its maximum at the center to zero at the edge. This conclusion can also be obtained by considering the source emission direction S, the viewing direction V, and the surface normal N. When people see such a change in brightness, a conclusion maybe made is that this image is captured from a round, probably spherical, object.

3.1.5Solution for Photometric Stereo

Given an image, it is expected that the shape of the object in image can be recovered. There is a unique mapping from the surface orientation to the radiance. However, the inverse mapping is not unique, as the same brightness can be originated from many surface orientations. A contour of constant R(p, q) connects such a set of orientations in the reflectance map.

In some cases, the surface orientation can be determined with the help of particular points where the brightness attains its maximum or minimum. According to eq. (3.26), for a Lambertian surface, R(p, q) = 1 only when (p, q) = (ps, qs). Therefore, the surface orientation can be uniquely defined by the surface brightness. While in general cases, the mapping from the image brightness to the surface orientation is not unique. This is because in every space location the brightness has only one degree of freedom, while the orientation has two degrees of freedom (two gradients).

To recover the surface orientation, some additional information should be introduced. To determine two unknowns, p and q, two equations are required. Two images, taken with different lighting conditions as shown in Figure 3.11, yield the following two equations

$\begin{array}{l}{R}_1(p,q)=E_1\\ R_2(p,q)=E_2(3.32)\end{array}$

If these two equations are linearly independent, a unique solution for p and q is possible.

Example 3.3 Solution of photometric stereo.

Suppose that

$R_1(p,q)=\sqrt{\frac{1+p_{1}p+q_{1}q}{r_1}}\text{and}{R}_2(p,q)=\sqrt{\frac{1+p_{2}p+q_{2}q}{r_2}}$

where

$r_1=\sqrt{1+p_1^2+q_1^2}\text{and}{r}_2=\sqrt{1+p_2^2+q_2^2}$

Provided p1/q1 ≠ p2/q2, p and q can be solved from eq. (3.32) as

$p=\frac{(E_1^2{r}_1-1)q_2-(E_2^2{r}_2-1)q_1}{p_1{q}_2-q_1{p}_2}\text{and}q=\frac{(E_2^2{r}_2-1)p_1-(E_1^2{r}_1-1)p_2}{p_1{q}_2-q_1{p}_2}$