8.2.1. Contents of the RAW format

The native RAW format needs to include the elements which differentiate different sensors. It generally includes the following elements:

• 1) in the metadata field:
  • - a header including the order of bytes (little-endian or big-endian⁵), the name of the image and the address of the raw data;
  • - information describing the sensor: number of pixels, type of color mastering (using a Bayer matrix or other matrix types, see section 5.4.2.2), the arrangement of photosites⁶ and colorimetric profile, expressing the composition of the chromatic filters;
  • - information describing the image: date, camera type, lens type, exposure time, aperture, focal, selected settings etc., and sometimes geographic position;
  • - the setting elements required for white balance using sensor measurements or user settings;
  • - the dynamic of the signal (between 8 and 14 bits) and the parameters of the electronic sequence used to correct this signal (dark current or measures allowing determination of the dark current, bias at zero, amplification factors (or parameters allowing these factors to be calculated using exposure time, ISO sensitivity, and the energy measurement)).
• 2) in the image field:
  • - a thumbnail to accompany the image and represent it each time the memory is consulted;
  • - a reduced JPEG image used to obtain a representation of reasonable size, in certain cases;
  • - the image data, presented according to sensor geometry (with RGB channels interwoven in a single image plane) (see Figure 8.2).

As many metadata elements are shared by all images, typical formats for these elements are used (for example Exif); these are discussed below. However, the native format file also contains sensor-specific information (particularly in relation to the precise parameters governing electronic elements: thermal noise, positions of dead pixels, non-homogeneity in gains, corrections for pixel vignetting) which manufacturers do not always wish to make public. For this reason, the metadata of native files is sometimes encrypted in order to prevent them from being read by software outside of the manufacturer’s control.

Finally, note that certain manufacturers use native formats in which image files undergo lossless coding (Lempel-Ziv type).

As we have stated, the native format is bulky, and this represents one of its major drawbacks. The header file takes up a few thousand bytes, generally negligible in relation to the size of the image file. For an image with n pixels, each pixel is represented by a single signal in one of the R, G or B channels, and this signal is generally coded using 2 bytes. Without compression, the whole file will therefore take up around 2n bytes. Using a lossless compression algorithm, we can obtain a reduction of the order of 2; a RAW file takes up around n bytes⁷. This compression, if selected by the manufacturer, will be applied automatically. Native image files generally take up between 10 and 50 Mb, around ten times more than the corresponding JPEG file.

8.2.2. Advantages of the native format

The quality of the signal captured by a camera is best preserved using the native RAW format. Moreover, the accompanying metadata provides all of the elements required for efficient image processing.

This metadata allows us to carry out all of the operations needed to reconstruct an image signal over 8 or 16 bits for any applications, even those which are demanding in terms of quality, such as printing, publication, photocomposition, video integration, etc. JPEG type output is still possible for everyday uses, but if volume is not an issue, formats with a higher dynamic, which better respect the fine details of images, are often preferable: TIFF, DNG, PNG, etc.

Use of the native format is essential in many advanced applications, such as those described in Chapter 10: high dynamic rendering, panoramas, and increasing resolution and focus.

It is often better to use separate computer software for reconstruction processes. These programs, whether free or commercial, give us the opportunity to carry out post-processing as required by specific applications. Post-processing is more effective if the source signal is used than if the signal has already been processed for general usage.

8.2.3. Drawbacks of the native format

Whilst native formats provide the most faithful rendering of images, they are also subject to a number of faults:

• – the volume of these images is often prohibitive; for general use, the storage space required is multiplied by a factor of between 5 and 10;
• – the size of the image reduces the speed at which successive photographs can be taken, as the flow rate through the bus connecting the sensor to the memory card is limited;
• – the time taken to display or navigate through image databases increase accordingly;
• – many widespread visualization programs cannot process native files;
• – the lack of standardization in native formats leads to uncertainty regarding the future accessibility of images stored in this way. This is a serious issue, as rapid restructuring by camera manufacturers results in the elimination of major players, who protected their reading and writing software using encrypted coding. Moreover, technological developments have also resulted in the evolution of stored information, and manufacturers may create new formats in order to satisfy requirements not met by their existing format.

8.2.4. Standardization of native formats

8.3.1. The XMP standard

An ISO standard (16684 - 1: 2012) has been created regarding the description of these data, based on the XMP (extensible metadata platform) profile, proposed by Adobe. In theory, this format allows conservation of existing metadata alongside an XMP profile; in practice, however, conversion of all metadata into XMP format is highly recommended. Metadata is grouped into mutually independent “packets”, which may potentially be processed by separate applications. Users may create their own packets and develop new applications to use these packets, but care is required to avoid affecting other packets.

The resource description framework (RDF) language, popularized by the semantic web, forms the basis for packet description, and XML (extensible markup language) syntax is generally used.

The XMP format is widely used in artistic production, photocomposition, the creation of synthetic images and virtual reality. It has been used in a number of commercial developments, in professional and semi-professional processing tools, and has been progressively introduced into photographic treatment toolkits. However, users are generally unfamiliar with this format and rarely possess the tools required to use it themselves, unlike Exif files, which are much more accessible.

8.3.2. The Exif metadata format

Exif (exchangeable image file format) was designed as a complete sound and image exchange format for audiovisual use, and made up of two parts: one part compressed data, and the other part metadata. Nowadays, only the metadata file accompanying an image is generally used. For the signal compression aspect, JPEG, TIFF or PNG are generally preferred (but the Exif exchange standard includes these compression modes in a list of recommended algorithms). For metadata, however, elements of the Exif file form the basis for a number of other standards: TIFF, JPEG, PNG, etc.

Created by Japanese manufacturers in 1995, the Exif format is used in camera hardware, and, while it has not been standardized or maintained by a specific organization (the latest version was published in 2010), it still constitutes a de facto standard. Unfortunately, however, it has tended to drift away from initial principles, losing its universal nature.

The metadata part of Exif (now referred to simply as Exif) is broadly inspired by the structure of TIFF files. The file is nested within a JPEG or native image file, and contains both public data for general usage and manufacturer-specific data, which is not intended for use outside of image programs. Unfortunately, while image handling programs generally recognize Exif files within images and use certain data from these files, the file is not always retransmitted in derivative images, or is only partially transmitted, leading to a dilution of Exif information and de facto incompatibility. The Exif file can be read easily using simple character sequence recognition tools (for example using PERL), and a wide range of applications for decoding these files are available online.

8.4.1. General lossless coding algorithms

These coding techniques make use of duplications between strings of bits, without considering the origin of these strings. General forms are included in computer operating systems [GUI 03a].

• – Huffman coding, which produces variable length code, associates short code words with frequently-recurring messages (in this case, the message is the value of a pixel or set of pixels). The best results are obtained for independent messages. In cases where messages are correlated, the Huffman code only exploits this correlation by grouping messages into super-messages. These dependencies are not modeled, meaning that the method is not particularly efficient for images. However, it is particularly useful for coding graphics and drawings, producing significant gains in association with a low level of complexity. Huffman coding by run-lengths or zones is therefore used in a number of standards (including TIFF);
• – arithmetic coding does not separate images into pixels, but codes the bit flux of an image using a single arithmetic representation suitable for use with a variety of signal states. Arithmetic coding is not used in image processing, as there is no satisfactory symbol appearance model;
• – Lempel–Ziv coding (codes LZ77 and LZ78) uses a sliding window, which moves across the image, and a dictionary, which retains traces of elementary strings from their first occurrence. Later occurrences are replaced by the address of this first example in the dictionary. LZ is now routinely used to code very large files with high levels of duplication, such as TIFF files.
• – hybrid coding types use back references, as in the Lempel–Ziv method, along with shorter representations for more frequent strings, constructed using prefix trees. These codes, used in a number of generic formats such as Deflate and zip, are extremely widespread;
• – more specific coding types have been developed for use in imaging, making explicit use of two-dimensional (2D) and hierarchical dependencies in images; details of these coding types are given in [PRO 03]. These methods are used by standards such as JBIG (highly suited to progressive document transmission in telescopy) and coders such as Ebcot, which features in the JPEG 2000 standard and will be discussed further in section 8.6.2.

All of these coding methods are asymptotically very close to the Shannon entropy limit for random sequences. The size of the headers involved means that they are not worth using for short messages, but image files rarely fall into this category. These methods are not particularly efficient for images, as duplication statistics are generally relatively low in a pixel flux representation. This duplication is concealed within the image structure, something which hierarchical and JPEG-type coding methods are designed to exploit.

8.4.2. Lossless JPEG coding

Specific algorithms have been developed for image coding, based on the specific two-dimensional (2D) properties of images. They have enjoyed a certain level of success in areas such as medical imaging, where image degradation must be avoided at all costs. They are also widely used in graphics, where scanning images are used in conjunction with text and drawings [GUI 03b].

The most widespread approaches are based on predicting the value of a pixel X at position (i, j) using neighbors for which values have already been transmitted and are known to the receiver, specifically those denoted A(i, j − 1), B(i − 1, j − 1), C(i − 1, j − 1) and D(i − 1, j + 1) (Figure 8.3, left). The difference between the predicted and actual values is often very low, and often follows a Gaussian distribution, which may be conveniently coded using the Huffman approach.

The lossless JPEG standard, published in 1993, proposes an algorithm for the prediction of X conditional on the presence of a horizontal or vertical contour. There are eight possible situations in this case, leading to eight different decisions. The remainder is coded using the Huffman method. Compression rates of up to 3 have been obtained in medical imaging using this standard, but results are often lower in natural imaging, and lossless JPEG has not been particularly successful in this area.

A second type of lossless JPEG coder has since been proposed. JPEG-LS is based on a typical prediction algorithm, LOCO-1, which counts configurations using a predictor known as the median edge detector. There are 365 possible configurations, and predictions are made using the hypothesis that the error follows a Laplace distribution. Golomb-Rice coding is then used to compress the transmitted sequence (this type of coding is suitable for messages with a high probability of low values). It is optimal for geometric sequences and performs better than lossless JPEG, with a compression rate of up to 3 for natural images (a comparison of coder performances is given in [GUI 03b]). JPEG-LS has been standardized under the reference ISO-14495-1.

The JPEG 2000 standard concerns another lossless coding method (sometimes noted JPEG 2000_R, with R reflecting the fact that the method is reversible). This method uses biorthogonal 5/3 wavelet coding (Le Gall wavelets with a low-pass filter using 5 rational coefficients and a high-pass filter with 3 rational coefficients). This third form of lossless JPEG coding presents the advantage of being scalable¹¹, progressive¹² and compatible with the image handling facilities of JPEG 2000 (see section 8.6.2). Unlike JPEG-LS, it uses the same principle for lossy and lossless coding, but it is slower, and the implementation is more complex. In terms of compression, it does not perform better than JPEG-LS.

8.5.1. The PNG format

The PNG (portable network graphics) format was designed for graphic coding in scanning (rather than vector) mode and for lossless compression. It was developed as an alternative to GIF (graphics interchange format), created by CompuServe, which was not free to access for a considerable period of time (the relevant patents are now in the public domain).

The PNG format offers full support for color images coded over 24 bits in RGB (unlike GIF, which only codes images over 8 bits), alongside graphics coded using palettes of 24 or 32 bits. It is also able to handle transparency information associated with images (via channel α, which varies from 0 to 1, where 0 = opaque and 1 = completely transparent). However, unlike GIF, PNG is not able to support animated graphics. For images, PNG conversion must occur after the application of corrections (demosaicing, white balancing, etc.) and improvement processes (filtering, enhancement, etc.). Treatment information and details contained in the Exif file are not transmitted, and are therefore lost.

PNG has been subject to ISO standardization (ISO — 15948: 2004).

PNG files contain a header, for identification purposes, followed by a certain number of chunks, some compulsory, other optional, used in order to read the image. Chunks are identified by a 4-symbol code, and protected by an error protection code. The compulsory elements are:

• – the IHDR chunk, including the image width, the number of lines and the number of bits per pixel;
• – the PLTE chunk, including the color palette for graphic images;
• – the IDAT chunk, containing the image data itself, in one or more sections;
• – the IEND chunk, marking the end of the file.

Optional chunks may then be used to provide additional information concerning the image background, color space, white corrections, gamma corrections, the homothety factor for display purposes, and even information relating to stereoscopic viewing. PNG has been progressively extended to include color management operations in the post-production phase, i.e. control of the reproduction sequence from the image to the final product.

PNG is widely used in the graphic arts, enabling transparent and economical handling of graphics or binary texts, with graphics defined using up to 16 bits per channel over 4 channels (images in true color + channel α). The main attractions of this format do not, therefore, reside in its compression capacity. However, PNG images are compressed using a combination of predictive coding (particularly useful for color blocks, which have high levels of duplication) and a Deflate coder. The predictive coder is chosen by the user from a list of five options (with possible values of 0 (no prediction), A, B, (A + B)/2, A or B or C depending on the closest value of A + B — C, (see diagram in Figure 8.3)).

8.5.2. The TIFF format

The TIFF format has already been mentioned in connection with the EP extension, which allows it to handle native formats. In this section, we will consider TIFF itself. Owned by Adobe, TIFF was specifically designed for scanned documents used in graphic arts, which evolved progressively from handling binary documents to handling high-definition color documents. Intended as a “container” format for use in conserving documents of varying forms (vector graphics, text, scanned imaged coded with or without loss, etc.), TIFF is an ideal format for archiving and file exchange, guaranteeing the quality of source documents used with other files to create a final document.

TIFF files are made up of blocks of code, identified by tags, where each block specifies data content. Tags are coded on 16 bits. Above 32,000, tags are assigned to companies or organizations for collective requirements. Tags from 65 000 to 65 535 may be specified by users for private applications. There are therefore a number of variations of TIFF, not all of which are fully compatible. The basic version available in 2015, version 6.0, was created over 20 years ago; this version indicates essential points which must be respected in order to ensure inter-system compatibility. TIFF allows very different structures to be contained within a single image: subfiles, bands or tiles, each with their own specific characteristics to enable coexistence.

Algorithms used to read TIFF files must be able to decode lossless Huffman coding by range and JPEG coding by DCT, and, in some cases, other compression methods (such as LZW and wavelet JPEG), as specific flags and tags are used for these compression types.

TIFF recognizes color images created using RGB, YCbCr or CIE la*b*, coded on 24 bits, or of the CMYK type, coded on 32 bits (see chapter 5). Extensions have been developed for images using more than 8 bits per channel and for very large images.

As a general rule, the TIFF file associated with an image is relatively large. In the case of an image of n pixels in sRVB (3 bytes per pixel), a TIFF file using exactly 1 byte per channel will take up 3n bytes before lossless compression, and around half this value after compression; however, compression is only carried out if requested by the user, making the method slower.

8.5.3. The GIF format

The GIF format is also lossless, and also a commercial product, maintained by CompuServe. While designed for use with graphics, it may also be used to transport images under certain conditions.

GIF is an 8 bit coding method for graphics coded using a color palette, using range-based Lempel-Ziv coding. Black and white images are therefore transmitted with little alteration using a transparent palette (although a certain number of gray levels are often reserved for minimum graphic representation), but with a low compression rate. Color images must be highly quantized (in which case lossless compression is impossible) or split into sections, each with its own palette, limited to 256 colors. Performance in terms of compression is lower in the latter case. Another solution uses the dithering technique, which replaces small uniform blocks of pixels with a mixture of pixels taken from the palette.

8.6.1. JPEG

In the 1980s, a group of researchers began work on the creation of an effective standard for lossy compression. This goal was attained in the 1990s, and JPEG (joint photographic expert group) format was standardized in 1992, under the name ISO/IEC 10918-1. The standard leaves software developers a considerable amount of freedom, and variations of the coding program exist. The only constraint is that decoding programs must be able to read a prescribed set of test images [GUI 03b].

8.6.1.1. Principles of JPEG encoding

JPEG encoding is a six-step process, involving:

• 1) a color space change to convert the signal, generally presented in the RGB space, into a luminance/chrominance space (generally YCrCb);
• 2) subsampling of chrominance components using a variable factor, up to 4 in x and y, but usually 2 (this step uses the fact that the eye is less sensitive to fluctuations in color than to fluctuations in gray levels);
• 3) division of the image into fixed-size blocks: 8 × 8 pixels for luminance images, 16 × 16 pixels for chrominance images;
• 4) transformation of these blocks using an operator which “concentrates” the energy of each block into a small number of coefficients, in this case the DCT (discrete cosine transform);
• 5) quantization of these coefficients in order to reduce their entropy;
• 6) transmission of the reordered coefficients by zigzag scanning of the window, followed by Lempel-Ziv coding of the bit chain.

Each of these steps is reversed during the decoding process, with the exception of steps 2 and 5, which involve irreversible information loss.

The direct cosine transform, defined by the matrix in Table 8.1, is a variation of the Fourier transform, which is better suited to real, positive signals, such as images, than the Fourier transform itself [AHM 74]. The DCT is a 2D, exact and reversible transformation of a scalar signal, which creates a real signal with as many samples as the input signal.

Let J(i,j) be a window of N × N pixels (where N = 8, for example) of the channel being transmitted (the luminance or chrominance channel of the original image). The DCT K_J of J is given by:

[8.1]

where and C(i, i ≠ 0) = 1.

The reverse transformation:

[8.2]

allows us to retrieve the exact value J(i, j), as long as K_J(k, l) has not been subjected to quantization.

Variables k and l are spatial frequencies, as in the case of a Fourier transform. The pair (k = 0, l = 0) has a frequency of zero (i.e. the mean value of J(i, j) for the window to within factor N). As k increases, the horizontal frequencies increase. As l increases, so do the vertical frequencies, and all pair values (k, l) where k ≠ 0, l = 0 corresponds to a frequency which is inclined in relation to the axes 0_x and 0_y.

The term K_J(0, 0), known as the continuous component, is always much larger than the others. It is treated separately in the DCT, and often transmitted without alterations in order to provide exact, if not precise, information regarding the value of the block.

	0	1	2	3	4	5	6	7
0	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
1	0.981	0.831	0.556	0.195	-0.195	-0.556	-0.831	-0.981
2	0.924	0.383	-0.383	-0.924	-0.924	-0.383	0.383	0.924
3	0.831	-0.195	-0.981	-0.556	0.556	0.981	0.195	-0.831
4	0.707	-0.707	-0.707	0.707	0.707	-0.707	-0.707	0.707
5	0.556	-0.981	0.195	0.831	-0.831	-0.195	0.981	-0.556
6	0.383	-0.924	0.924	-0.383	-0.383	0.924	-0.924	0.383
7	0.195	-0.556	0.831	-0.981	0.981	-0.831	0.556	-0.195

The other terms are subject to quantization, to a greater or lesser extent depending on the desired compression rate. We begin by defining a quantizer q(k,l) and an approximation of coefficient K_J(k, l) using such that:

[8.3]

where denotes the default rounded integer value of x.

The set of coefficients q(k, l) thus forms a quantization matrix which is applied to the whole image. This matrix is transmitted with the associated coefficients in order to enable reconstruction of the image. Coders differ in the way in which they use chrominance subsampling and in the choice of quantization matrices.

As a general rule, high frequency terms can be shown to contain very little energy, and these terms are therefore of limited importance in terms of the perceived quality of the image. These terms are set to zero during quantization (see Figure 8.4 and Table 8.1).

	0	1	2	3	4	5	6	7		0	1	2	3	4	5	6	7
0	1450	65	0	0	0	0	0	0		980	100	0	0	0	0	0	0
1	0	0	0	0	0	0	0	0		0	0	60	200	0	0	0	0
2	0	0	0	0	0	0	0	0		0	0	0	0	0	0	0	0
3	0	0	0	0	0	0	0	0		0	0	0	0	0	0	0	0
4	0	0	0	0	0	0	0	0		0	0	0	0	0	0	0	0
5	131	0	0	0	0	0	0	0		0	0	0	0	0	0	0	0
6	0	0	0	0	0	0	0	0		0	0	0	0	0	0	0	0
7	0	0	0	0	0	0	0	0		0	0	0	0	0	0	0	0

The coefficients are then transmitted by zigzag scanning, which places frequencies in increasing 2D order (as far as possible) after the continuous component (see Figure 8.3, right). After a short period, the coefficient string will only contain zeros, which will be compressed effectively using either Liv-Zempel or range-based Huffman techniques.

JPEG images are decompressed by reversing all of the steps described above. Coefficient quantization can highlight faults in individual blocks which are characteristic of the JPEG format. Certain reconstruction algorithms therefore include a low-pass filtering stage which reduces these faults. The subsampling of chromatic components may also require an additional interpolation stage, which may also have damaging effects on the image.

8.6.2. JPEG 2000

The JPEG 2000 standard is recognized by the ISO under the name ISO/IEC 15444-1, and is a relatively general image exchange standard, offering a variety of functions (archiving, internet and video). In this context, we will focus on the aspects relating to the image format.

JPEG 2000 makes use of the fact that wavelet composition offers better performances than the DCT in terms of image compression: this method provides a better quality image than DCT with the same compression rate, and higher compression rates for the same image quality [BAR 03a].

JPEG 2000 is also scalable.

The format also uses a highly developed arithmetic encoder, EBCOT (embedded block coding with optimized truncation) which makes use of the 2D tree structure of the wavelet decomposition [TAU 00].

Finally, JPEG 2000 gives users the possibility to describe image content at a higher level of the pyramid; this is useful in facilitating intelligent use of image collections, and for optimized transmission over new-generation networks. To do this, the format makes use of the progressive transmission capabilities offered by wavelet scalability, via a codestream transmission structure, and the ability to define regions of interest (ROI) which are transmitted first and with higher quality levels. It explicitly accounts for the possible presence of text or graphics in compound structures. The format is also designed for application to very large images (up to 2³² pixels per line), images with a large number of channels (up to 2¹⁴) and images with high-dynamic pixels (up to 38 bits per sample).

As with many other standards, JPEG 2000 is open and does not explicitly specify a compression mode, leaving developers the freedom to make improvements over time. The encoder is obliged to produce a binary string which conforms to the standard, but any strategy may be used to do this. Decoders must simply prove their ability to decode the example files included with the standard.

As a transmission standard, JPEG 2000 is highly suited to professional requirements. It has had less success with the general public; its requirements in terms of calculation power and decoding software are higher than those of JPEG, but this is of minor importance compared to the effects of inertia in the general market, where users are not overly concerned with obtaining high quality/flux ratios.

8.6.2.1. Principles of JPEG 2000

Like JPEG, JPEG 2000 is able to transform color images created using the RGB space into a luminance/chrominance space (YUV or Y C_RC_B). Two types of transformations are available, one reversible, the other irreversible (but more effective in terms of compact representation). The chromatic components are then processed separately.

JPEG 2000 then operates either on the image as a whole, or on tiles, identically-sized sections of the image (with the exception of the edge tiles). These tiles are decomposed hierarchically by a bank of 2D, bi-orthogonal wavelet filters [BAR 03a].

The standard allows the use of two types of filters¹³:

• 1) Le Gall 5/3 rational coefficient filters, as discussed in section 8.4; these filters only operate using rounded integers, and may be used for lossy or lossless coding, depending on whether all of the non-null coefficients are transmitted (using the EBCOT method, described below),
• 2) Daubechies 9/7 real coefficient filters¹⁴ (see Table 8.3), which can only be used for lossy coding, as calculations are carried out using real numbers with a finite level of precision. These wavelets perform better than rational wavelets in the context of lossy coding.

These filters are bi-orthogonal¹⁵, and defined by a low-pass filter used to calculate images with increasingly low resolutions, starting from the original level of the tile, and a high-pass filter, used to define the contents of details characterizing the specific level in question. These details are contained within the coefficients of the wavelet decomposition: a_n(i, j).

Filtering can be carried out in two ways. The first method, using convolution, is that generally used in signal processing as a whole. The second method, which constitutes a major advantage of the hierarchical wavelet approach, involves the iterative application of two filters, one low-pass, the other high-pass; these filters are identical for all levels of the pyramid. This is known as the lifting process. Using this process (see Figure 8.6), the odd coefficients of a decomposition are expressed using the value of the even coefficients with the application of a corrective term. This corrective term constitutes the specific detail image for the level in question [SWE 96].

The basic wavelet transformation is one-dimensional (1D); it is thus applied once to the lines and once to the columns of the area of interest. This results in the production of 4 images (see Figure 8.7):

• – an image with high frequencies in both lines and columns;
• – an image with high frequencies in the lines and low frequencies in the columns;
• – an image with high frequencies in the columns and low frequencies in the lines;
• – an image with low frequencies in both lines and columns.

In a carefully-chosen wavelet base, these coefficients present a number of interesting statistical properties; they are highly concentrated, and can be efficiently modeled using a generalized Gaussian law, allowing efficient quantization:

[8.4]

Quantization tables are defined for each resolution level, and calculated in such a way as to provide a wide range of compression levels.

These quantized coefficients, transformed into bit planes and grouped into subbands, are then transmitted to the encoder. As we have seen, the EBCOT coder uses a high-performance contextual encoding process involving a wavelet coefficient tree, predicting each bit using the same bit from the previous resolution level and its immediate neighbors, starting with those bits with the heaviest weighting. The encoder processes subblocks, which are relatively small (with a maximum 64 × 64); the precise size of these subblocks is defined by the user.

The power of the coding method is based on the domain of hierarchical analysis of the wavelet decomposition. We will consider neighborhoods of 6 or 27 connections within a three-dimensional (3D) pyramid of resolution levels. The majority pixels (identified by the bit from the previous level) are not encoded; the others are coded using their sign and deviation from the previous level, if this is significant. Finally, we consider situations where there is no dominant configuration. The resulting tree structure is encoded using an algebraic code.

order of filter	0	± 1	± 2	± 3	± 4
analysis: low-pass filter	0.602949	0.266864	-0.078223	-0.016864	0.026749
analysis: high-pass filter	0.557543	0.295636	-0.028772	-0.045636	0
synthesis: low-pass filter	0.602949	-0.266864	-0.078223	0.016864	0.026749
synthesis: high-pass filter	0.557543	-0.295636	-0.028772	0.045636	0

8.8.1. Video encoding and standardization

Standardization work has been carried out by the ITU¹⁶, producing recommendations H.120, H.261, H.263 and H.264. The H263 standard was the first to include an inter-image mode; H264 serves as the basis for MPEG-4.

These recommendations have also been used outside of the field of telecommunications, for applications such as archiving, video games and private usage. The moving picture expert group (MPEG) committee, a joint venture by the ISO and the IEC, has worked on creating consensus solutions for the purposes of standardization [NIC 03].

The first recommendation, MPEG-1, concerned low flow rate transmissions for wired transmissions in the period preceding ADSL (using packets of 64 kbit/s, giving a maximum rate of around 1.5 Mbit/s). This very low rate was only suitable for video transmission of very small images (352 × 288 pixels per image at a rate of 25 image/s). MPEG-1 was widely used in the context of video-CD archiving.

The MPEG-2 standard was designed for television, and specifically for high quality transmission (contribution coding). It is particularly suitable for Hertz transmission, whether via cable or ADSL. The standard allows transmissions of between 2 and 6 Mbit/s for ordinary video and of 15 to 20 Mbit/s for HDTV¹⁷.

MPEG-4 is a multimedia transmission standard, part of which deals specifically with video images. It is particularly concerned with low-rate transmission (around 2 Mbit/s) and uses more elaborate compression mechanisms than those found in MPEG-2; these mechanisms are suitable for use with highly compressed files (although a significant reduction in quality is to be expected). MPEG-4 makes use of the scalability properties included in recommendation H.263. It forms the basis for easy communications with the field of computer science, particularly with regard to copyright protection.

Developments MPEG-7, MPEG-21 and MPEG-x (with x taking a value from A to V) are designed for multimedia applications of video sequences, and define specific functions for archiving, database searches, production (particularly the integration of a variety of sources), interaction, diffusion and content protection.

8.8.2. MPEG coding

8.8.2.1. Mechanisms used in MPEG

The video coding standards contained in MPEG-2 and MPEG-4 are currently used in many cameras. Architectures designed for vector-based treatment and integrated into the camera are used to carry out weighty calculation processes, using many of the elements of these standards [NIC 03, PER 02].

We will begin by considering the implementation of MPEG-2. A color image, generally acquired in a RGB space, is converted into a {luminance, chrominance} space, and the chrominance channels are generally subsampled using a factor of 2 for both lines and columns, as in the case of JPEG.

Intra-image coding is carried out by subdividing the image into groups of blocks (GOB), constructed from macroblocks, MB. A macroblock is itself made up of 4 blocks of 8 × 8 pixels of luminance and, generally, 1 chrominance block (chroma subsampling encoding, 4 : 2 : 0) or 4 blocks (in the case of 4 : 4 : 4 encoding, without subsampling). The binary string is therefore made up of an image layer (I), made up of layers of groups of blocks (GOB), themselves made up of macroblock (MOB) layers. This macroblock layer typically includes the 4 luminance blocks followed by 2 chrominance blocks, one for dominant R, one for dominant B.

These blocks are coded using a discrete cosine transformation (DCT) quantizied, then subjected to entropic encoding (by range length, or, better, arithmetic coding) (Figure 8.8).

There are two possible modes of inter-image encoding:

• 1) predictive mode (mode P): one image from the flux is taken as a reference and transmitted with intra-image encoding (image I). The new image (P), to be transmitted later, is compared to I in order to determine the movement affecting each pixel. This movement prediction stage operates using block matching, and determines a movement vector (Δx in x and Δy in y) for each macroblock. The difference between the macroblock in image P and that in image I, offset by {Δx, Δy}, constitutes an error macroblock, which is encoded using the DCT. The image is transmitted by sending the address of the reference macroblock in I, the movement vector {Δx, Δy}, and the error block, quantized and entropically compressed.
• 2) bidirectional interpolated mode (mode B): using this mode, the image flux is interpolated using both an earlier image I and a later image P

to reconstruct intermediate images noted B (see Figure 8.9) using a very small quantity of additional transmission space (only elements essential for positioning the macroblock in B are added, in cases where this cannot be deduced by simple interpolation of that of P). However, this operating mode does result in a delay in image reconstruction; for this reason, it is mostly used for archiving purposes, where real time operation is not essential, or in applications where a very low level of latency will be tolerated.

This coding mode generates significant block effects in the case of high compression rates. A number of improvements have been proposed to eliminate this issue: multiplication of movement vectors for each macroblock, block layering when identifying movement, a posteriori filtering, etc.

In MPEG-4, most of the functions of MPEG-2 described above have been improved: better prediction of DCT coefficients, better movement prediction, resynchronization of the data flux, variable length entropic coding, error correction, etc. Part 10 of the standard introduces a new object, the VOP (video object plane), a box surrounding an object of interest. A VOP may be subjected to a specific compression mode different to that used for the rest of the image. A VOP is made up of a number of whole macroblocks, and will be identified in the data flux. For example, a participant in a television program, speaking against a fixed background, may be placed into a VOP with high definition and a high refresh frequency, while the studio background will be coded at a very low rate. VOPs, generally described on an individual basis by users, are not currently used in most cameras (which tend to focus on part 2 of the MPEG-4 recommendations). However, certain cameras now include very similar functions (for example higher-quality coding in a rectangular zone defined by facial recognition).

The use of MPEG-4 coders in cameras and their ability to encode high definition videos (HDTV format¹⁸, in 2015, with 4k format available on certain prototypes for durations of a few seconds) represents a major development in camera technology, as this application currently determines the peak processing power, defines the processor and signal processing architecture, the data bus and the buffer memory.