2.1.1.1World Coordinate System

World coordinate system is also known as the real-world coordinate system XYZ, which represents the absolute coordinates of the objective world (it is also called objective coordinate system). General 3-D scenes are using this coordinate system for representation.

2.1.1.2Camera Coordinate System

Camera coordinate system is the coordinate system xyz, which is developed by taking the camera as center (origin) and the camera optical axis as z axis.

2.1.1.3Image Plane Coordinate System

Image plane coordinate system is the coordinate system x′ y′ at the imaging plane of camera. Generally, the xy plane is taken as parallel to the plane of the camera coordinate system, as well as the x-axis and the x′-axis, y-axis and the y′-axis coincides, respectively, so that the origin of the image plane is on the optical axis of the camera.

2.1.1.4Computer Image Coordinate System

Computer image coordinate system is the coordinate system MN that is inside the computer for representing image. The final image is stored by a memory within the computer, so the projection coordinates on image plane needs to be converted to computer image coordinate system.

According to the above mutual relations of several different coordinate systems, several different (camera) imaging model can be formed. In the following, the step-by-step descriptions from simple to complex are provided.

2.1.2.1Projection Model

A model of image capturing by a camera with perspective projection is shown in Figure 2.1. Three coordinate systems are involved: The world coordinate system (X, Y, Z), the camera coordinate system (x, y, z), and the image coordinate system (x′, y′). Here, suppose the camera coordinate system is aligned with the world coordinate system, and the image coordinate system is aligned with the camera coordinate system. This camera position is known as the normal position. The focal length of the camera lens is λ. A 3-D point W(X, Y, Z) is projected onto the point (x, y) = (x′, y′) at the image plane.

According to the properties of similar triangles, the relationship between a 3-D point in the world space and its 2-D projection at the image plane is established by

$\frac{x}{\lambda }=-\frac{X}{Z-\lambda }=\frac{X}{\lambda -Z}\left(2.1\right)$

$\frac{y}{\lambda }=-\frac{Y}{Z-\lambda }=\frac{Y}{\lambda -Z}\left(2.2\right)$

where the negative signs in front of X and Y indicate the inversion of the image point with respect to the space point. The coordinates of the image point are obtained by

$x=\frac{\lambda X}{\lambda -Z}\left(2.3\right)$

$y=\frac{\lambda Y}{\lambda -Z}\left(2.4\right)$

The above projection maps a line in 3-D space onto a line at the image plane (with the exception that the line to be projected is perpendicular to the image plane). A rectangle in 3-D space could be any quadrangle (determined by four points) at the image plane, after the projection. Therefore, such a projection is referred to as a four-point mapping. The transformations in eq. (2.3) and eq. (2.4) are nonlinear as the coordinate values are involved in the division.

Example 2.1 Telecentric imaging and supercentric imaging

In general standard optical imaging system, the light beams are converged. It has a significant adverse effect for the optical measurement; see Figure 2.2(a). If the position of the object is changing, the image becomes larger when the object is close to the lens and becomes smaller when the object is away from the lens. The depth of the object could not be obtained from the image, unless the object is placed on the known distance, otherwise the measurement error is inevitable.

If the aperture position is moved to the convergence point of parallel light (F2) a telecentric imaging system can be obtained, as shown in Figure 2.2(b). In this case, the main ray (the light beam passing through the center of aperture) is parallel to the optical axis in the object space and the small change of object position would not change the size of the object image. Of course, the farther away the object is from the focus position, the stronger the object is blurred. However, the center of blur disk does not change the position. The disadvantage of telecentric imaging is that the diameter of telecentric lens should reach at least the size of the object to be imaged. Thus, using a telecentric imaging is very expensive for very big object.

If the aperture is put even closer to the image plane than the convergence point of parallel light beams, as in Figure 2.2(c), the main ray becomes the converging lines of space. In contrast to the standard imaging of Figure 2.2(a), the farther the object, the bigger the object looks. This imaging technique is called supercentric imaging. One of its characteristics is that the surface parallel with the optical axis can be seen. In Figure 2.3, three imaging techniques for viewing a thin-walled cylinder along the optical axis are illustrated. The standard imaging can see the cross section and the inner wall, the telecentric imaging can only see the cross section, while the supercentric imaging can see both the cross section and an outer wall (Jähne, 2004).

2.1.2.2Homogeneous Coordinates

To express these transformations in the form of a linear matrix, which is more convenient for manipulation, the homogeneous coordinates are often used.

The homogeneous coordinates of a point in an XYZ coordinate system are defined as (kX, kY, kZ, k), where k is an arbitrary but nonzero constant. For example, a point in an XYZ coordinate system can be expressed as a vector:

$\mathbf{W}={\left[\begin{array}{ccc}X& Y& Z\end{array}\right]}^{\text{T}}\left(2.5\right)$

The vector in the homogeneous coordinates is

${\mathbf{W}}_{\text{h}}={\left[\begin{array}{cccc}kX& kY& kZ& k\end{array}\right]}^{\text{T}}\left(2.6\right)$

By defining the matrix of perspective projection as

$\mathbf{P}=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& -1/\lambda & 1\end{array}\right]\left(2.7\right)$

The perspective projection of Wh is given by

$c_{\text{h}}=\mathbf{P}{\mathbf{W}}_{\text{h}}=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& -1/\lambda & 1\end{array}\right]\left[\begin{array}{c}kX\\ kY\\ kZ\\ k\end{array}\right]=\left[\begin{array}{c}kX\\ kY\\ kZ\\ -kZ/\lambda +k\end{array}\right]\left(2.8\right)$

The elements of ch are the camera coordinates in the homogeneous coordinates. They can be converted to Cartesian coordinates by dividing the first three elements of ch by the fourth one

$\mathbf{c}={\left[\begin{array}{ccc}x& y& z\end{array}\right]}^{\text{T}}={\left[\begin{array}{ccc}\frac{\lambda X}{\lambda -Z}& \frac{\lambda Y}{\lambda -Z}& \frac{\lambda Z}{\lambda -Z}\end{array}\right]}^{\text{T}}\left(2.9\right)$

The first two elements of c are just the image coordinates of a 3-D space point (X, Y, Z) after projection.

2.1.2.3Inverse Projection

Inverse projection is used to determine the coordinates of a 3-D point according to its projection on a 2-D image. Following eq. (2.8), this projection can be expressed by

${\mathbf{W}}_{\text{h}}={\mathbf{P}}^{-1}{c}_{\text{h}}\left(2.10\right)$

where the inverse projection matrix P–1 is

${\mathbf{P}}^{-1}=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 1/\lambda & 1\end{array}\right]\left(2.11\right)$

Now consider a 2-D image point (x′, y′, 0), which can be represented by a homogeneous form as

$c_{\text{h}}={\left[\begin{array}{cccc}k{x}^{\prime }& k{y}^{\prime }& 0& k\end{array}\right]}^{\text{T}}\left(2.12\right)$

By integrating eq. (2.12) into eq. (2.10), the coordinates of a 3-D point in homogeneous form are

${\mathbf{W}}_{\text{h}}={\left[\begin{array}{cccc}k{x}^{\prime }& k{y}^{\prime }& 0& k\end{array}\right]}^{\text{T}}\left(2.13\right)$

The corresponding coordinates in a Cartesian system are

$\mathbf{W}={\left[\begin{array}{ccc}X& Y& Z\end{array}\right]}^{\text{T}}={\left[\begin{array}{ccc}{x}^{\prime }& y^{\prime }& 0\end{array}\right]}^{\text{T}}\left(2.14\right)$

It is clear that the Z coordinate value of a 3-D point could not be uniquely determined by the image point (x′, y′), as Z is always zero for any 3-D point. In other words, the image point (x′, y′) now corresponds to all points lying on the line that pass through (x′, y′, 0) and (0, 0, *), as shown in Figure 2.1.

Solving X and Y from eq. (2.3) and eq. (2.4) yields

$X=\frac{x^{\prime }}{\lambda }\left(\lambda -Z\right)\left(2.15\right)$

$Y=\frac{y^{\prime }}{\lambda }\left(\lambda -Z\right)\left(2.16\right)$

It is clear that recovering a 3-D point from its projection on an image by inverse perspective transformation is impossible unless its Z coordinate value is known.

2.1.3.1Orthogonal Projection

In the orthogonal projection, the Z-coordinate of the 3-D spatial point is not considered (the information for object distance are lost), which is equivalent to mapping the 3-D spatial point directly/vertically to the image plane in the direction of the optical axis of the camera. Figure 2.4 shows a bar-shaped object projected onto the Y-axis (corresponding to the YZ profile in Figure 2.1) in both orthogonal projection and perspective projection.

The result of orthographic projection shows the true scale of the cross section of the object, while the result of the perspective projection is related to the object distance d. The orthographic projection can be seen as the particular perspective projection in which the focal length λ is infinite, so the projection transformation matrix can be written as

$\mathbf{P}=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\left(2.17\right)$

2.1.3.2Weak Perspective Projection

Weak perspective projection (WPP) is an approximation to the perspective projection. In the weak perspective projection, the first step is to make an orthogonal projection to the plane where the mass center of object is located and is parallel to the image plane; the second step is to make a perspective projection to the image plane. The perspective projection of step 2 here can be achieved by means of equal scaling within the image plane. Let the perspective scaling factor be s, then

$s=\lambda /d\left(2.18\right)$

is the ration of the focal length over the object distance. By considering s, the projection matrix of the weak perspective can be written as

$\mathbf{P}=\left[\begin{array}{cccc}s& 0& 0& 0\\ 0& s& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\left(2.19\right)$

The orthogonal projection in the weak perspective projection does not consider the Z-coordinate of the 3-D space point but also changes the relative position of each point in the projection plane (corresponding to the change of size). This can make a greater impact on object points that have relatively large distance with the optical axis of the camera. It can be proved that the error on image is the first-order infinitesimal of the error on object.

2.1.3.3Paraperspective Projection

Paraperspective projection is also a projection method located between the orthogonal projection and perspective projection (Dean et al., 1995). A schematic of the paraperspective projection is shown in Figure 2.5. In the figure, the world coordinate system coincides with the camera coordinate system, and the focal length of the camera is λ. The image plane intersects the Z-axis perpendicularly at point (0, 0, λ). Point C is the center of mass of the object S, and the distance from the origin in Z direction is d.

Given a projection plane at Z = d and parallel to the image plane, the process of paraperspective projection has two steps:

(1)Given a particular object S (one of the objects in scene), projecting it parallelly to the projection plane, where each projection line is parallel to the line OC (not necessarily perpendicular to the projection plane).

(2)The above projection result on the projection plane is then projected onto the image plane. The projection result on the image plane will be reduced to λ/d (the same as in the weak perspective projection) because the projection plane and the image plane are parallel.

The first step described above takes into account the effects of the pose and perspective, which preserves the relative position of the object points in the projection plane. The second step takes into account the effects of distance and other positions. It can be shown that the image error is the second-order infinitesimal of the object error, so the parallel perspective projection is closer to the perspective projection than the weak perspective projection.

Example 2.2 Comparison of projection modes

The perspective projection mode and its three approximate projection modes are shown together in Figure 2.6 for comparison.

2.1.4.1Model Conversion

In the above discussion, the camera coordinate system is supposed to be aligned with the world coordinate system. In a real situation, this assumption is not always applicable. A more general case is depicted in Figure 2.7. In Figure 2.7, the camera coordinate system is separated from the world coordinate system, but the image coordinate system is still aligned with the camera coordinate system. The offset of the center of the camera (the origin of the camera coordinate system) from the origin of the world coordinate system is denoted D = [Dx, Dy, Dz]. The camera has been panned by an angle (the angle between x and X) and tilted by an angle α (the angle between z and Z).

The above general camera model can be generated from the basic model shown in Figure 2.1, using the following consecutive steps:

(1)Displace the camera center from the origin of the world coordinate system.

(2)Pan the x-axis by an angle γ (around the z-axis).

(3)Tilt the z-axis by an angle α (around the x-axis).

Moving the camera with respect to the world coordinate system is equivalent to inversely moving the world coordinate system with respect to the camera. The first step of the displacement of the origin of the world coordinate system to the camera center can be accomplished by the following translation transformation matrix

$\mathbf{T}=\left[\begin{array}{cccc}1& 0& 0& -D_x\\ 0& 1& 0& -D_y\\ 0& 0& 1& -D_z\\ 0& 0& 0& 1\end{array}\right]\left(2.20\right)$

In other words, a point (Dx, Dy, Dz) whose homogeneous form is Dh will be located at the origin of the world coordinate system after the transformation of TDh.

Further, consider the coincident of coordinate systems. The pan angle γ is the angle between the x- and X-axes, which are parallel in original situation. The rotation about the x-axis by the angle γ can be achieved by rotating the camera around the z-axis with the angle γ counterclockwise, where the rotation transformation matrix is

${\mathbf{R}}_{\gamma }=\left[\begin{array}{cccc}\cos \gamma & \sin \gamma & 0& 0\\ -\sin \gamma & \cos \gamma & 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\left(2.21\right)$

Similarly, the tilt angle α is the angle between the z- and Z-axes. The rotation about z-axis by the angle α can be achieved by rotating the camera around the x-axis with the angle α counterclockwise, where the rotation transformation matrix is

${\mathbf{R}}_{\alpha }=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& \cos \alpha & \sin \alpha & 0\\ 0& -\sin \alpha & \cos \alpha & 0\\ 0& 0& 0& 1\end{array}\right]\left(2.22\right)$

The above two rotation matrices can be cascaded into a single matrix

$\mathbf{R}={\mathbf{R}}_{\alpha }{\mathbf{R}}_{\gamma }=\left[\begin{array}{cccc}\cos \gamma & \sin \gamma & 0& 0\\ -\sin \gamma \cos \alpha & \cos \alpha \cos \gamma & \sin \alpha & 0\\ \sin \alpha \sin \gamma & -\sin \alpha \cos \gamma & \cos \alpha & 0\\ 0& 0& 0& 1\end{array}\right]\left(2.23\right)$

2.1.4.2Perspective Projection

According to the above discussion, a 3-D world point in its homogeneous form Wh after perspective projection becomes

$c_{\text{h}}=\mathbf{P}\mathbf{R}\mathbf{T}{\mathbf{W}}_{\text{h}}\left(2.24\right)$

where P is defined in eq. (2.7).

By dividing the first two elements of ch by its fourth element, the Cartesian coordinates (x, y) for a 3-D world point W(X, Y, Z) can be obtained by

$x=\lambda \frac{\left(X-D_x\right)\cos \gamma +\left(Y-D_y\right)\text{sin }\gamma }{-\left(X-D_x\right)\sin \alpha \sin \gamma +\left(Y-D_y\right)\sin \alpha \cos \gamma -\left(Z-D_z\right)\cos \alpha +\lambda }\left(2.25\right)$

$y=\lambda \frac{-\left(X-D_x\right)\sin \gamma \cos \alpha +\left(Y-D_y\right)\text{cos}\alpha \text{cos }\gamma +\left(Z-D_z\right)\sin \alpha }{-\left(X-D_x\right)\sin \alpha \sin \gamma +\left(Y-D_y\right)\sin \alpha \cos \gamma -\left(Z-D_z\right)\cos \alpha +\lambda }\left(2.26\right)$

Example 2.3 Computing image-plane coordinates

Suppose a camera is mounted in a position for surveillance as shown in Figure 2.8. The center of the camera is located at (0, 0, 1), the focal length of the camera is 0.05 m, the pan angle is 135°, and the tilt angle is also 135°. The task is to determine the image coordinates for space point W (1, 1, 0).

According to the above discussion, it has to find the procedure that moves the camera from the normal position in Figure 2.1 to the particular position in Figure 2.8. The three required steps are depicted in Figure 2.9. Figure 2.9(a) shows the normal position of the camera. The first step is to displace the camera out of the origin, as shown in Figure 2.9(b). The second step is to pan the camera by the angle γ around the z-axis, as shown in Figure 2.9(c). The third step is to rotate the camera around the x-axis and tilt it by the angle α relative to the z-axis, as shown in Figure 2.9(d).

Putting all parameters into eq. (2.22) and eq. (2.23), the image coordinate values of the 3-D point W(1, 1, 0) are x = 0 m and y = –0.008837488 m.