14.1 Introduction

The concept of signal space has its roots in the mathematical theory of inner product spaces known as Hilbert spaces (Stakgold, 1967). Many books on linear systems touch on the subject of signal spaces in the context of Fourier series and transforms (Ziemer et al., 1998). The applications of signal space concepts in communication theory find their power in the representation of signal detection and estimation problems in geometrical terms, which provides much insight into signaling techniques and communication system design. The first person to have apparently exploited the power of signal space concept in communication theory was the Russian Kotel'nikov (1968), who presented his doctoral dissertation in January, 1947. Wozencraft and Jacobs (1965) expanded on this approach and their work is still today widely referenced. Arthurs and Dym (1962) made use of signal space concepts in the performance analysis of several digital modulation schemes. A one-chapter summary of the use of signal space methods in signal detection and estimation is provided in Ziemer and Tranter (2009). Another application of signal space concepts is in signal and image compression. Wavelet theory (Rioul and Vetterli, 1991) is currently finding use in these application areas. Finally, the application of signal space concepts to nonlinear filtering is discussed. In the next section, the fundamentals of generalized vector spaces are summarized, followed by an overview of several applications to signal representations.

14.2 Fundamentals

A linear space or vector space (signal space) (Stakgold, 1967) is a collection of elements (called vectors) x, y, z, …, for which the following axioms are satisfied:

1. To every pair of vectors x and y there corresponds a vector x + y with the properties:

   1. x + y = y + x

   2. x + (y + z) = (x + y) + z

   3. There exists a unique element 0 such that x + 0 = x, for every x

   4. To every x, there exists a unique vector labeled −x such that x + (−x) = 0

2. To all vectors, x and y, and all numbers (scalars), α and β, the following commutative and associative rules hold:

   1. α(βx) = (αβ)x

   2. (α + β)x = αx + βx

   3. α(x + y) = αx + αy

   4. 1x = x where 1 is the identity element

A vector is said to be a linear combination of the vectors x₁, x₂, …, x_k in a vector space if there exist numbers (not all 0) α₁, α₂, …, α_k such that

The vectors x₁, x₂, …, x_k are said to be linearly dependent (or form a dependent set) if there exist numbers α₁, α₂, …, α_k not all zero, such that

If Equation 14.2 can be satisfied only for α₁ = α₂ = … = α_k = 0, the vectors are linearly independent.

One is tempted to use the infinite-sum version of Equation 14.2 in defining the notion of independence for an infinite set of vectors. This is not true in general; one needs the notion of convergence, which is based on the concept of distance between vectors.

With the idea of linear independence firmly in mind, the concept of dimension of a vector space readily follows. A vector space is n-dimensional if it possesses a set of n independent vectors, but every set of n + 1 vectors is a dependent set. If for every positive integer k, a set of k independent vectors in the space can be found, the space is said to be infinite dimensional. A basis for a vector space means a finite set of vectors e₁, e₂, …, e_n with the following attributes:

1. They are linearly independent

2. Every vector x in the space can be written as a linear combination of the basis vectors; that is

   $$x={\displaystyle \sum _{i=1}^n{\xi }_i{e}_i}\left(14.3\right)$$

It can be proved that the representation (14.3) is unique and that if the space is n-dimensional, any set of n independent vectors e₁, e₂, …, e_n forms a basis.

The next concept to be developed is that of a metric space. In addition to the addition of vectors and multiplication of vectors by scalars, as is true of ordinary three-dimensional vectors, it is important to have the notions of length and direction of a vector imposed. In other words, a metric structure must be added to the algebraic structure already defined. A collection of elements x, y, z, … in a space will be called a metric space if to each pair of elements x, y there corresponds a real number satisfying the properties:

1. d(x, y) = d(y, x)

2. d(x, y) ≥ 0 with equality if and only if x = y

3. d(x, z) ≤ d(x, y)+ d(y, z) (called the triangle inequality)

The function d(x, y) is called a metric (or distance function). Note that the definition of d(x, y) does not require that the elements be vectors; there may not be any way of adding elements or multiplying them by scalars as required for a vector space.

With the definition of a metric, one can now discuss the idea of convergence of a sequence {x_k} of elements in the space. Note that d(x, x_k) is a sequence of real numbers. Therefore, it is sensible to write that

if the sequence of numbers d(x, x_k) converges to 0 in the ordinary sense of convergence of sequences of real numbers. If

the sequence {x_k} is said to be a Cauchy sequence. It can be shown that if a sequence {x_k} converges, it is a Cauchy sequence. The converse is not necessarily true, for the limit may have carelessly been excluded from the space. If the converse is true, then the metric space is said to be complete.

The next vector space concept to be defined is that of length or norm of a vector. A normed vector space (or linear space) is a vector space in which a real-valued function ‖x‖ (known as the norm of x) is defined, with the properties

1. ‖x‖ ≥ 0 with equality if and only if x = 0.

2. ‖α x‖ = ‖α‖ ‖x‖.

3. ‖x₁ + x₂‖ ≤ ‖x₁‖ + ‖x₂‖.

A normed vector space is automatically a metric space if the metric is defined as

which is called the natural metric for the space. A normed vector space may be viewed either as a linear space, a metric space, or both. Its elements may be interpreted as vectors or points, a case in point being ordinary three-dimensional geometry wherein we can identify points as vectors emanating from the origin to the point in question.

The structure of a normed vector space will now be refined further with the definition of the notion of angle between two vectors. In particular, it will be possible to tell whether two vectors are perpendicular. The notion of angle between two vectors will be obtained by defining the inner product (also known as a scalar or dot product). In general, an inner product in a vector space is a complex-valued function of ordered pairs x, y with the properties

1. 〈x, y〉 = 〈y, x〉* (the asterisk denotes complex conjugate)

2. 〈α x, y〉 = α 〈x, y〉

3. 〈x₁ + x₂, y〉 = 〈x₁, y〉 + 〈x₂, y〉

4. 〈x, x〉 ≥ 0 with equality if and only if x = 0

From the first two properties, it follows that

Also, Schwarz's inequality can be proved and is given by

with equality if and only if x = αy (or x = 0 or y = 0). The real, nonnegative quantity 〈x, x〉^1/2 satisfies all the properties of a norm. Therefore, it is adopted as the definition of the norm, and Schwartz's inequality assumes the form

The natural metric in the space is given by Equation 14.6. An inner product space, which is complete in its natural metric, is called a Hilbert space.

An additional requirement that can be imposed on a Hilbert space is separability, which, roughly speaking, restricts the number of elements in the space. A Hilbert space H is separable if there exists a countable (i.e., can be put in one-to-one correspondence with the positive integers) set of elements (f₁, f₂, …, f_n, …) whose finite linear combinations are such that for any element f in H and ∊ > 0 there exist an index N and constants α₁, α₂, …, α_N such that

The set (f₁, f₂, …, f_n, …) is called a spanning set. The discussions from here on are limited to separable Hilbert spaces.

Any finite-dimensional Hilbert space E_n is separable. In fact, there exists a set of n vectors (f₁, f₂, …, f_n) such that each vector x in E_n has the representation

It can be shown that the spaces consisting of square-integrable functions on the intervals [a, b], (−∞, b], [a, ∞), and (−∞, ∞) are all separable, where a and b are finite, and spanning sets exist for each of these spaces. For example, a spanning set for the space of square-integrable functions on [a,b] is the set (1,t,t², … ), which is clearly countable. For such function spaces, convergence is in the sense of convergence in the mean defined as

Similarly, the ideas of independence and basis sets apply to Hilbert spaces in infinite-dimensional form.

It is necessary to distinguish between the concepts of a basis and a spanning set consisting of independent vectors. As ∊ is reduced in Equation 14.15, it is expected that N must be increased, and it may also be necessary to change the previously found coefficients α₁, α₂, …, α_N. Hence, there might not exist a fixed sequence of constants ξ₁, ξ₂, …, ξ_n, … with the property

as would be required if the set {f_k} were a basis. For example, on the space of square-integrable functions on [−1,1], the independent set f₀ = 1, f₁ = t, f₂ = t², … is a spanning set, but not a basis, since there are many square-integrable functions on [−1,1] that cannot be expanded in a series like Equation 14.18 (an example is |t|). Odd as it may seem at first, it is possible if the powers of t in this spanning set are regrouped into the set of polynomials known as the Legendre polynomials, P_k (t).

Two vectors x, y are orthogonal or perpendicular if 〈x, y〉 = 0. A finite or countably infinite set of vectors {φ₁, φ₂, …, φ_k, …} is said to be an orthogonal set if 〈φ_i, φ_j〉 = 0, i ≠ j. A proper orthogonal set is an orthogonal set none of whose elements is the zero vector. A proper orthogonal set is an independent set. A set is orthonormal if

An important concept is that of a linear manifold in a Hilbert space. A set M is said to be a linear manifold if, for x and y belonging to M, so does αx + βy for arbitrary complex numbers α and β; thus, M is itself a linear space. If a linear manifold is a closed set, it is called a closed linear manifold (i.e., every Cauchy sequence has a limit in the space) and is itself a Hilbert space. In three-dimensional Euclidean space, linear manifolds are simply lines and planes containing the origin. In a finite-dimensional space, every linear manifold is necessarily closed.

Let M be a linear manifold, closed or not. Consider the set M^⊥ of all vectors which are orthogonal to every vector in M. It is a linear manifold that can be shown to be closed. If M is closed, M and M^⊥ are known as orthogonal complements. Given a linear manifold M, each vector in the space can be decomposed in a unique manner as a sum x_p + z, where x_p is in M and z is in M^⊥.

Given an infinite orthonormal set {φ₁, φ₂, …, φ_k, …} in the space of all square-integrable functions on the interval [a,b], let {a_n} be a sequence of complex numbers. The Riesz–Fischer theorem tells us how to represent an element in the space and states:

1. If

   $${\displaystyle \sum _{n=1}^{\infty }{\left|a_n\right|}^2}\left(14.20\right)$$

   diverges, then

   $${\displaystyle \sum _{n=1}^{\infty }{a}_n{\varphi }_n}\left(14.21\right)$$

   diverges.

2. If Equation 14.20 converges, then Equation 14.21 also converges to some element g in the space and

   $$a_n=\langle g,{\varphi }_n\rangle \left(14.22\right)$$

The next question that arises is how to construct an orthonormal set from an independent set {e₁, e₂, …, e_n}. A way of doing this is known as the Gram–Schmidt procedure. The construction is as follows:

1. Pick a vector from the set {e₁, e₂, …, e_n}, say e₁. Let

   $${\varphi }_1=\frac{e_1}{{\langle e_1,e_1\rangle }^{1/2}}\left(14.23\right)$$

2. Remove from a second vector in the set, say e₂, its projection on φ₁. This yields

   $$g_2=e_2-\langle e_2,{\varphi }_1\rangle {\varphi }_1\left(14.24\right)$$

   The vector g₂ is a linear combination of e₁ and e₂, and is orthogonal to φ₁. To normalize it, form

   $${\varphi }_2=\frac{g_2}{{\langle g_2,g_2\rangle }^{1/2}}\left(14.25\right)$$

3. Pick another vector from the set, say e₃, and form

   $$g_3=e_3-\langle e_3,{\varphi }_2\rangle {\varphi }_2-\langle e_3,{\varphi }_1\rangle {\varphi }_1\left(14.26\right)$$

   Normalize g₃ in a manner similar to that used for g₂.

4. Continue until all vectors in the set {e_k} have been used.

Note that the sets {e_k}, {g_k}, and {φ_k} all generate the same linear manifold. (Also, if {e_k} is not an independent set, the construction will result in 0 at one or more steps.)

A basis consisting of orthonormal vectors is known as an orthonormal basis. If the basis vectors are not normalized, it is simply an orthogonal basis.

It is next desired to approximate an arbitrary vector x in a Hilbert space in terms of a linear combination of the independent set {e₁, e₂, …, e_k}, where k ≤ n if the space is an n-dimensional Euclidean space, and k is an arbitrary integer if the space is infinite dimensional. First, the orthonormal set {φ_k} is constructed from {e_k}. The unique, best approximation to x is the Fourier sum

which is geometrically the projection of x onto the linear manifold generated by {φ_k}, or equivalently, the sum of the projections along the individual axes defined by φ₁, φ₂, …, φ_k. The square of the distance between x and its projection is

Since the left-hand side is nonnegative, Equation 14.29 gives Bessel's inequality, which is

A convenient feature of the Fourier sum is as follows: If another vector φ_k+1 is added to the orthonormal approximating set of vectors, the best approximation now becomes

Thus, an additional term is added to the series expansion without changing previously computed coefficients, which makes the extension to a countably infinite orthonormal approximating set simple to envision. In the case of an infinite orthonormal approximating set, Bessel's inequality (14.30) now has an infinite limit on the sum. Does the approximating sum (14.31) converge to x? The answer is that convergence can be guaranteed only if the set {φ_k} is extensive enough, that is, if it is a basis or a complete orthonormal set. In such cases, Equation 14.30 becomes an equality. In fact, a number of equivalent criteria can be stated to determine whether an orthonormal set {φ_k} is a basis or not (Stakgold, 1967). These are

1. In finite n-dimensional Euclidean space, {φ_k} has exactly n elements for completeness

2. For every x in the space of square-integrable functions

   $$x={\displaystyle \sum _i\langle x,{\varphi }_i\rangle }{\varphi }_i\left(14.32\right)$$

3. For every x in the space of square-integrable functions

   $${\Vert x\Vert }^2={\displaystyle \sum _i^{}{\left|\langle x,{\varphi }_i\rangle \right|}^2}\left(14.33\right)$$

   (known as Parseval's equality)

4. The only x in the space of square-integrable functions for which all the Fourier coefficients vanish is the 0 function

5. There exists no function φ(t) in the space of square-integrable functions such that {φ, φ₁, φ₂, …, φ_k, …} is an orthonormal set

Examples of complete orthonormal sets of trigonometric functions over the interval [0,T] are as follows: (1) The complex exponentials with frequencies equal to the harmonics of the fundamental frequency ω₀ = 2π/T, or

The factor T^1/2 in the denominator is necessary to normalize the functions and is often not included in the definition of the complex exponential Fourier series. (2) The sines and cosines with frequencies equal to harmonics of the fundamental frequency ω₀ = 2 π/T or

Note that if any function is left out of these sets, the basis is incomplete.

14.3 Application of Signal Space Representation to Signal Detection

The M-ary signal detection problem is as follows: given M signals, s₁(t), s₂(t), …, s_M(t), defined over 0 ≤ t ≤ T. One is chosen at random and sent each T-second interval through a channel that adds white, Gaussian noise of power spectral density N₀/2 to it. The challenge is to design a receiver that will decide which signal was sent through the channel during each T-second interval with minimum probability of making an error.

An approach to this problem, as expanded upon in greater detail by Wozencraft and Jacobs (1965, Chapter 4) and Ziemer and Tranter (2009, Chapter 10), is to construct a linear manifold, called the signal space, using the Gram–Schmidt procedure on the M signals. Suppose that this results in the orthonormal basis set {φ₁, φ₂, …, φ_K} where K ≤ M (the number of basis functions may be less than the number of signals because the signals are not necessarily linearly independent). The received signal plus noise is represented in this signal space as vectors with coordinates (note that they depend on the signal transmitted)

where

The numbers Z_ij are components of vectors referred to as the signal vectors, and the space of all signal vectors is called the observation space. They may be produced by a bank of matched filters or correlators, with one filter matched to each orthonormal function.

An apparent problem with this approach is that not all possible noise waveforms added to the signal can be represented as vectors in this K-dimensional observation space. The part of the noise that is represented is

where

In terms of Hilbert space terminology, Equation 14.38 is the projection of the noise waveform onto the observation space (i.e., a linear manifold). The unrepresented part of the noise is

and is the part of the noise that must be represented in the orthogonal complement of the observation space. The question is whether the decision process will be harmed by ignoring this part of the noise. It can be shown that n_⊥(t) is uncorrelated with n_‖(t). Thus, since n(t) is Gaussian they are statistically independent and n_⊥(t) has no bearing on the decision process; nothing is lost by ignoring n_⊥(t).

The decision process can be shown to reduce to choosing that signal s_l (t) minimizing the distance to the data vector; that is,

where

and

Thus, the signal detection problem is reduced to a geometrical one, where the observation space is subdivided into decision regions in order to make a decision. Each decision region is constructed to ensure that each observation point included in it is closer to the chosen signal point than to any other.

14.4 Application of Signal Space Representation to Parameter Estimation

The procedure used in applying signal space concepts to estimation is similar to that used for signal detection. Consider the observed waveform consisting of additive signal and noise of the form

where A is a parameter to be estimated and the noise is white as before. Let {φ₁, φ₂, …, φ_k, …} be a complete orthonormal basis set. The observed waveform can be represented as

where

and N_j is defined by Equation 14.39. Hence, an estimate can be made on the basis of the set of coefficients

or on the basis of a vector in the signal space with these coordinates. A reasonable criterion for estimating A is to maximize the likelihood ratio, or a monotonic function thereof. Its logarithm, for n(t) Gaussian, can be shown to reduce to

In the limit as K → ∞, this becomes

A necessary condition for the value of A that maximizes Equation 14.49 is

The value of A that maximizes Equation 14.49, denoted $Â$, is called the maximum likelihood estimate.

14.5 Miscellaneous Applications

References

Arthurs, E. and Dym, H. 1962. On the optimum detection of digital signals in the presence of white gaussian noise—A geometric approach and a study of three basic data transmission systems. IRE Trans. Commun. Syst., CS-10(Dec.): 336–372.

Benedetto, S. and Biglieri, E. 1983. Nonlinear equalization of digital satellite channels. IEEE J. Sel. Areas in Commun., SAC-1(Jan.): 57–62.

Haykin, S. 1996. Adaptive Filter Theory, 3rd ed. Englewood Cliffs, NJ: Prentice Hall.

Kotel'nikov, V.A. 1968. The Theory of Optimum Noise Immunity (trans. R.A. Silverman), Dover, New York.

Mathews, V. 1991. Adaptive polynomial filters. IEEE Signal Proc. Mag., 8(Jul.): 10–26.

Papoulis, A. and Pillai, S. Unnikrishna. 2001. Probability, Random Variables, and Stochastic Processes, 3rd ed., New York: McGraw–Hill.

Rioul, O. and Vetterli, M. 1991. Wavelets in signal processing. IEEE Signal Proc. Mag., 8(Oct.):14–38.

Stakgold, I. 1967. Boundary Value Problems of Mathematical Physics, Vol. 1, London: Macmillan, Collier–Macmillan Ltd.

Wozencraft, J.M. and Jacobs, I.M. 1965. Principles of Communication Engineering, New York: Wiley. (Available from Prospect Heights, IL: Waveland Press.)

Ziemer, R.E., Tranter, W.H., and Fannin, D.R. 1998. Signals and Systems: Continuous and Discrete, 4th ed. New York: Macmillan.

Ziemer, R.E. and Tranter, W.H. 2009. Principles of Communications: Systems, Modulation, and Noise, 6th ed. Boston, MA: Houghton Mifflin.

Further Reading

Very readable expositions of signal space concepts as applied to signal detection and estimation are found in Chapter 4 of the classic book by Wozencraft and Jacobs. The paper by Arthurs and Dym listed in the references is also worth reading. A treatment of signal space concepts and applications to signal detection and parameter estimation is given in Chapter 10 of the book by Ziemer and Tranter. For the mathematical theory behind signal space concepts, the book by Stakgold is recommended. For those interested in wavelet transforms and their relationship to signal spaces, the April 1992 issue of the IEEE Signal Processing Magazine provides a tutorial article on wavelet transforms. A tutorial treatment on adaptive Volterra filters is provided in the paper by Mathews.