The ability to retrieve and manipulate data and obtain information from a rich-media database is, of course, a long-term essential objective for information systems. However, this objective is very challenging. All humans are naturally good at understanding multimedia information, recalling audio and visual images, and linking complex information together. The way in which the human visual system in the brain is able to do this is not well understood although clearly this involves a number of transformations to the data received by the sensory systems in the brain. In contrast, people have to spend years learning how to deal with text, spoken and written. However, text is much easier for computers to deal with because it can be structured to a much greater extent than other media. This is the reason why metadata is very important in media retrieval. Therefore, it is not surprising that achieving content-based retrieval is difficult in practice and requires an effective user interface as well as advanced database capabilities. It also only works for restricted sets of images and with tightly specified requirements.

Oracle interMedia provides both a framework to handle the multimedia data objects in Oracle DBMS and a set of tools for content-based image retrieval (CBIR) functionalities. CBIR for video and audio is less well developed. CBIR capabilities can be explored through the StillImage object type introduced in Chapter 3.

A primary benefit of using content-based retrieval (CBR) is reduced time and effort required to obtain media-based information but it also allows us to develop new application areas. It is possible to take the simplest approach and annotate the image object with text metadata based on a human being’s perception of the image but there are still no standards for developing image descriptions. Photographic images tend not to be self-identifying since the information usually available for text objects, such as title, author, abstract, list of contents, etc., is often not available, although the new digital camera standard will increasingly provide this kind of metadata.

Content-based retrieval is being developed to try to overcome some of these difficulties since very large sets of images are being developed. CBR implies a system with the ability to search based on the user’s association of an impression of the media object. Progress in this technology has been limited, especially with audio, but we can still look forward to being able to hum a tune and then obtain from the computer a sample and all the details of the artist, etc. Therefore, for the time being this chapter will focus on image retrieval using the StillImage object type.

The similarity between two images arises because they have similar appearances. This may be the result of several factors, such as color, texture, and orientation. Therefore, one approach is to break down an image into these factors and compare the factors separately.

An image can then be represented by a collection of low-level features, such as color, texture, shape, and spatial relationships (i.e., location information) that can be extracted from the image. These features are then used during query processing to compute the similarity between a sample image provided or selected by a user and the images in a database. The result set consists of a set of similar images rather than a specific result and would require the end user to review and probably refine the output. This is one of the reasons why the design of the user interface is crucial. The system then processes the information contained in image data and creates abstractions of its content in terms of visual attributes.

interMedia will support ABR, TBR, and CBR but as we already know these are difficult to achieve. Therefore, the best strategy is usually to combine these approaches in designing an application. We can specify text columns to describe the semantic significance of the image (e.g., that the pictured automobile won a particular award, or that its engine has six or eight cylinders) and use the StillImage object type to permit content-based queries based on intrinsic attributes of the image (for example, how closely its color and shape match a picture of a specific automobile).

When frequent adding and updating of images is needed in massive databases, it is often not practical to require manual entry of all attributes that might be needed for ABR queries, so content-based retrieval provides increased flexibility and practical value. CBIR is also useful in providing the ability to query attributes such as texture or shape that are difficult to represent using text metadata. A feature is an attribute derived from transforming the original visual object by using an image analysis algorithm. A feature characterizes a specific property of the image such as its color. Usually a feature is represented by a set of numbers often called a feature vector.

The visual query mode (Chapter 11) involves matching the input image to preextracted features of real or idealized objects. Query processing using a visual mode is based on matching a vector of the sample image’s features with those of other images in the database. The methods used depend on the type of image. Many query operations deal solely with this abstraction rather than with the image itself. Thus, every image inserted into the database needs to be analyzed first, and a compact representation of its content stored in a feature vector.

The preextracted features can be held in the database using the StillImage object type. The purpose of the feature extraction is to extract a set of numerical features that remove redundancy from the image and reduces its dimension. A well-selected feature set will have useful information that defines the information that discriminated the image from its collection.

The objective is to use very simple computations on low-level features of an image, such as color, to recognize the actual content. These features are used to compute the similarity between a picture selected by a user and the images in a database. Table 13.1 lists some of the features extracted. The most commonly used features for CBIR are shape, color, and texture.

These object types offer the database designer an alternative to defining image-related attributes in columns separate from the image. The database designer can create a specialized composite data type that combines an SI.StillImage object and the appropriate text, numeric, and date attributes. Oracle interMedia not only allows the media data, like images, to be stored in the database table, but also allows the features of the images to be stored inside the database table as well.

For example, the feature vector of an image can be extracted by segmenting the image into regions based on color. Each region can be associated with color, texture, and shape information. One way to improve its retrieval is to crop images so that background noise is removed and the object of interest occupies the main part of the image.

It is useful to consider the following visual attributes:

Images in the database can be retrieved using the visual–visual mode queries by matching them with a comparison image. The comparison image can be any image inside or outside the current database, a sketch drawn by the user, an algorithmically generated image, and so forth. The comparison is not based on an exact match as in the case of SQL traditional data types. Instead the similarity of the images is evaluated for each of the three visual attributes (color, location, texture). A very difficult issue in image retrieval using visual mode is determining the relevance of each image in a result set to the query image. Ideally, the process would not miss any relevant images and not return any irrelevant ones. The quality of the retrieval process in terms of text retrieval is well established and can be measured in terms of two metrics, namely recall and precision. Recall is a measure of how well the retrieval engine performs in finding relevant objects. The recall of a system is defined as how much of the information that ideally could have been retrieved or extracted by the system was in fact retrieved or extracted by the system.

Precision is a measure of how well the engine performs in not returning irrelevant images. If every image is relevant the precision is 100%. The precision of a system is defined as how much of the information that ideally could have been retrieved or extracted by the system belongs to the information that should have been retrieved or extracted by the system. The retrieval engine must balance recall against precision. High precision means few unwanted media objects are retrieved, while low recall means many relevant objects are missed.

We can measure the similarity for each visual attribute of an image by calculating the score or distance between the two images with respect to that attribute. It is convenient to define the score to range from 0.00 (no difference) to 100.0 (maximum possible difference). Thus, the more similar two images are with respect to a particular visual attribute, the smaller the score will be for that attribute.

As an example of how distance could be used, assume that the dots in Figure 13.2 represent scores for three images with respect to two visual attributes, such as color and texture, plotted along the x-axis and y-axis of a graph. Suppose the user specifies Image 1 as the example and we want to locate similar images (recall) that are as close as possible to the sample image (precision). For matching, Image 1 is the comparison image, and Images 2 and 3 are each being compared with Image 1. We can see that Images 2 and 3 are both roughly equidistant from Image 1, but with respect to the color attribute plotted on the x-axis, the distance between Image 1 and Image 2 is relatively small (e.g., 15), whereas the distance between Image 1 and Image 3 is much greater (e.g., 50). If the color attribute is given more weight, then the fact that the two distance values differ by a great deal will probably be very important in determining whether or not Image 2 and Image 3 match Image 1. However, if color is minimized and the texture attribute is emphasized instead, then Image 3 will match Image 1 better than Image 2 matches Image 1.

Figure 13.2. Similarity over visual attributes.

If we measured the distance just between Image 1 and Image 2 on three visual attributes we would get the result shown in Table 13.2. In this theoretical example, the two images are most similar with respect to texture (distance = 5) and most different with respect to positional color (distance = 50).

In Figure 13.3 there are two very different images but we can see that some areas of color and texture in both images are similar by viewing the histograms.

Figure 13.3. Images and color histograms.

The result of using the different visual attributes as the basis for comparison is shown in Table 13.3. Looking at the distance score we can see that the images are considered most similar in terms of texture but most different in terms of color. The table gives information about the distance when the visual attributes are given different weightings. Before we consider how these results were obtained we need to deal with the concept of weightings.

Each weight value reflects how sensitive the matching process for a given attribute should be to the degree of similarity or dissimilarity between two images. For example, if we want color to be completely ignored in matching, we would assign a weight of 0.0 to color; in this case, any similarity or difference between the color of the two images is totally irrelevant in matching. In Table 13.3 the value for texture is 14.8712 where color is set at 0.0. On the other hand, if color is extremely important, we would assign it a weight greater than any of the other attributes; this will cause any similarity or dissimilarity between the two images with respect to color to contribute greatly to whether or not the two images match, so a value of 26.9861 is shown in Table 13.3. These results suggest the images are closest in terms of texture and furthest apart in terms of position.

When all attributes were set at 0.25 or 1.0 (i.e., equal weighting) the score (i.e., similarity) of the two images in Figure 13.3 is 23.2217. How have these similarity values been derived?

Assume that for the matching process, the following weights have been assigned to each visual attribute:

The weights are supplied in the range of 0.0 to 1.0. Within this range, a weight of 1.0 indicates the strongest emphasis, and a weight of 0.0 means the attribute should be ignored. The values you supply are automatically normalized such that the weights total 1.0, still maintaining the ratios supplied. In this example, the weights specified meant that normalization was not necessary.

The following formula is used to calculate the weighted sum of the distances, which is used to determine the degree of similarity between two images:

weighted_sum = color_weight    * color_distance +
               texture_weight  * texture_distance +
               position_weight    * position_distance

The degree of similarity between two images in this case is computed as:

0.7* color_distance + 0.2* texture_distance +
0.1* position_distance

Suppose we obtain distance scores of 15 for color, 5 for texture, and 50 for position, then using these values, the overall distance can be calculated as:

(0.7*15 + 0.2*5 + 0.1*50) = (10.5 + 1.0 + 5.0) = 16.5

To illustrate the effect of different weights in this case, assume that the weights for color and shape were reversed. In this case, the degree of similarity between two images is computed as:

0.1*color_distance + 0.2*texture_distance + 0.7*position_distance

That is:

(0.1*15 + 0.2*5 + 0.7*50) = (1.5 + 1.0 + 35.0) = 37.5

In this second case, the images are considered to be less similar than in the first case, because the overall score (37.5) is greater than in the first case (16.5).

The SI_StillImage object types form a complex interrelated group that exploit many of the features of object-relational database theory. The SI_StillImage object type can be used to store actual image data together with standard image metadata, such as height, width, format, etc. There are six object types that represent image features that could be used for CBIR:

In addition, there is an object type, SI_FeatureList, that represents an image as a composite feature based on the above with associated feature weights.

There are three object types that represent image features that could be used for CBIR object types for the color values, the average color feature, and the color histogram. As you can imagine this abundance of object types makes CBIR complex (see Table 13.4).

A public synonym has been created for each StillImage object type so that we do not need to specify the ORDSYS schema name when we use these object types in SQL procedures and functions, for example, SI_MKAvgClr(). In addition, there is a set of views accessible to the public that give useful information about these object types. It makes good sense to check these out before deciding to employ the SI_StillImage object types for CBIR.

In Chapter 3 we introduced the following table:

CREATE TABLE photos_SI
(id          NUMBER PRIMARY KEY,
                    description VARCHAR2(40) NOT NULL,
                    location    VARCHAR2(40),
                    image       SI_StillImage,
                    thumb       SI_StillImage)

The SI_StillImage has a number of methods that are useful for image processing that we can illustrate with this table.

SI_ClearFeatures() should be used when we know we do not want to do image matching as clearing the image features can improve performance for other operations because the image features, such as color histograms and texture, are synchronized with changes to the image.

SI_InitFeatures() is a related method that extracts image features and caches them in the SI_StillImage object. The method only needs to be called once as afterward every time an image is processed, new image features will be automatically extracted.

SI_RetainFeatures checks whether the features have been extracted.

SI_Content() returns the BLOB stored in the content_SI attribute of the SI_StillImage object.

CREATE OR REPLACE PROCEDURE still_content
AS
myimage  SI_StillImage;
photo     BLOB;
BEGIN
   SELECT image INTO myimage FROM photos_SI
   WHERE id = 1235;
   photo := myimage.SI_Content;
END;

SI_SetContent() updates the content of an SI_StillImage object as well as setting the value of its attributes (content_SI, contentLength, height_SI, width_SI).

CREATE OR REPLACE PROCEDURE still_setcontent
AS
   Newlob  blob;
   fils   BFILE := BFILENAME('PHOTO_DIR','raven.bmp'),
   myimage   SI_StillImage;
BEGIN
--use temporary BLOB
   DBMS_LOB.CREATETEMPORARY(newlob, TRUE);
   DBMS_LOB.fileopen(fils, DBMS_LOB.file_readonly);
   DBMS_LOB.LOADFROMFILE(newlob, fils,
DBMS_LOB.GETLENGTH(fils));
   DBMS_LOB.FILECLOSE(fils);
--select row for update
 SELECT image INTO myimage FROM photos_SI
 WHERE id = 1235 FOR UPDATE;
               myimage.SI_SetContent(newlob);
--update table
    UPDATE photos_si SET
 Image = myimage WHERE id = 1235;
   DBMS_LOB.FREETEMPORARY(newlob);
   COMMIT;
END;

CREATE OR REPLACE PROCEDURE still_averageColor
AS
myimage     SI_StillImage;
myAvgColor  SI_AverageColor;

BEGIN
    SELECT image INTO myimage FROM photos_SI
    WHERE id = 1235;
    myAvgColor := NEW SI_AverageColor(myimage);
END;

CREATE OR REPLACE PROCEDURE match_averageColor
AS
score        DOUBLE PRECISION;
myimage      SI_StillImage;
other_image  SI_StillImage;
myAvgColor   SI_AverageColor;

BEGIN
   SELECT image INTO myimage FROM photos_SI
   WHERE id = 1234;
   myAvgColor := NEW SI_AverageColor(myimage);
SELECT image INTO other_image FROM photos_SI
   WHERE id = 1235;
   Score := myAvgColor.SI_Score(other_image);
   DBMS_OUTPUT.PUT_LINE('Score is ' || score);
END;

SQL> execute score_ColorHistogram;
Score is 31.173184375

For Color Matching—Using SI_ColorHistogram Object Type

Using the color histogram should give a more reliable result than just using average color values. This object type has two attributes:

• SI_ColorsList as an array of colors (check SI_VALUES view for maximum size)

• SI_FrequenciesList as an array of color frequencies

However, it is recommended to use the methods rather than the attributes of this object type. We can use the SI_ColorHistogram(image) method to generate a color histogram for the image specified, provided the image is on the supported list. First we need to alter the photos_SI table to add a column to hold the color histogram when it’s been generated.

ALTER TABLE photos_SI ADD(COLOR_HISTOGRAM SI_ColorHistogram);

CREATE OR REPLACE PROCEDURE store_ColorHistogram
AS
myimage     SI_StillImage;
myColorHist SI_ColorHistogram;
BEGIN
 SELECT image INTO myimage FROM photos_SI
 WHERE id = 1235 FOR UPDATE;
 myColorHist:= NEW SI_ColorHistogram(myimage);
 UPDATE photos_SI SET color_histogram = myColorHist
WHERE id = 1235;
COMMIT;
DBMS_OUTPUT.PUT_LINE('Update completed'),
END;

Now that we have the color histogram of the image we can use this in matching using another object method called SI_Score(image) from the SI_ColorHistogram object type. This will compare two color histograms and return a value between 0, identical, and 100, no match. Note that this method will return NULL if the value of the image.SI_Content is NULL. We can also use SI_ScoreByClrHstgr() as an SQL function that accepts a color histogram object as the argument.

CREATE OR REPLACE PROCEDURE score_ColorHistogram
AS
myimage       SI_StillImage;
myColorHist   SI_ColorHistogram;
other_image   SI_StillImage;
score         DOUBLE PRECISION;
BEGIN
 SELECT image INTO myimage FROM photos_SI
 WHERE id = 1236;
 myColorHist:= NEW SI_ColorHistogram(myimage);
SELECT image INTO other_image FROM photos_SI
 WHERE id = 1235;
myColorHist := NEW SI_ColorHistogram(myimage);
score := SI_ScoreByClrHstgr(myColorHist,other_image);
DBMS_OUTPUT.PUT_LINE('Score is' || score);
END;

This time when we compare the same images as before (the raven and the robin) using the color histograms we get a slightly different result:

SQL> execute score_ColorHistogram;
Score is31.173184375

We can also use this method with color histogram data that has already been stored in the attributes of the image as is illustrated in the next procedure.

CREATE OR REPLACE PROCEDURE match_ColorHistogram
AS
myimage      SI_StillImage;
myColorHist  SI_ColorHistogram;
other_image  SI_StillImage;
score        DOUBLE PRECISION;
BEGIN
 SELECT color_histogram INTO myColorHist FROM photos_SI
 WHERE id = 1235;
SELECT image INTO other_image FROM photos_SI
 WHERE id = 1236;
score := SI_ScoreByClrHstgr(myColorHist,other_image);
DBMS_OUTPUT.PUT_LINE('Score is '|| score);
END;

SQL> execute match_ColorHistogram;
Score is 31.173184375

The score can range from 0 to 100. A value of 0 means that the color histogram of the specified image and the SI_ColorHistogram object instance are identical. A NULL value is returned if any one of the following is true:

• The value of the SI_ColorHistogram object instance is NULL.

• The value of the specified image is NULL.

• The value of image.SI_Content is NULL.

• The value of the color histogram feature is not supported for the format of the specified image.

But if it executes without generating a score? The score is NULL.

As we may want to use the other features and store them, we can modify the photos_SI table to add columns to hold the four feature types as follows:

ALTER TABLE photos_SI ADD (average_color   SI_AverageColor,
                  Positional_Color  SI_PositionalColor,
                  Texture           SI_Texture,
                  Feature_List      SI_FeatureList);

The facility that is most related to the ORDImageSignature object type is SI_FeatureList. This specifies an object containing up to four of the basic features and associated weights. The weight specifies the importance of the feature during image matching. Each weight can have a value between 0.0 and 1.0, where 0.0 indicates the feature should not be considered. Thus, the object type has four attributes corresponding to average color, color histogram, texture, and positional color each with a DOUBLE PRECISION weight. However, we usually use the object methods rather than the attributes. There is a constructor method called SI_FeatureList() with an equivalent SQL function called SI_MKFtrList() with a list of input parameters for the four features and weights that can be used as follows:

CREATE OR REPLACE PROCEDURE Still_features
AS
myimage       SI_StillImage;
myColorHist   SI_ColorHistogram; -- WEIGHTING 0.35
myAvgColor    SI_AverageColor; -- WEIGHTING 0.25
myPosColor    SI_PositionalColor; -- WEIGHTING 0.10
myTexture     SI_Texture; -- WEIGHTING 0.50
myFeatures    SI_FeatureList;
other_image   SI_StillImage;

BEGIN
    SELECT image INTO myimage FROM photos_SI
    WHERE id = 1235 FOR UPDATE;
    myColorHist :=SI_ColorHistogram(myimage);
myAvgColor :=  SI_AverageColor(myimage);
myPosColor :=  SI_PositionalColor(myimage);
myTexture  :=  SI_Texture(myimage);
myFeatures :=  NEW SI_FeatureList(myAvgColor, 0.25,
myColorHist, 0.35, myPosColor, 0.10,
myTexture, 0.50);

UPDATE photos_SI SET feature_list = myFeatures
    WHERE id = 1235;
COMMIT;
END;

If we want to use the SQL function we would change the myFeatures assignment statement to

myFeatures := SI_MkFtrList(myAvgColor, 0.25,
myColorHist, 0.35, myPosColor, 0.10, myTexture, 0.50)

For image matching we would use the SI_Score() method for the SI_FeatureList. This operator determines and returns the score of a specified image compared with a given SI_FeatureList value. The lower the returned score value, the better the image is characterized by the SI_FeatureList object used for scoring the image. The return score value is computed as follows.

Let n be the number of non-NULL feature attributes of the FeatureList object to which you are applying the method. For i ranging from 1 to n, let f_i be the feature attribute and W_i be the value of the corresponding feature weight. The result is the sum of f_i.SI_Score(image) * W_i divided by the sum of W_i. The process by which the score value is determined can also be described by the following expression:

A DOUBLE PRECISION value between 0 and 100 is returned. A value of 0 means that the image is identical to the FeatureList object. A value of 100 means that the image is completely different from the FeatureList object.

CREATE OR REPLACE PROCEDURE Score_features
AS
myimage     SI_StillImage;
myFeatures  SI_FeatureList;
score       DOUBLE PRECISION;

BEGIN
    SELECT feature_list INTO myfeatures FROM photos_SI
    WHERE id = 1235;
SELECT image INTO myimage FROM photos_SI
    WHERE id = 1235;
Score := myFeatures.SI_Score(myimage);
DBMS_OUTPUT.PUT_LINE( 'Score is ' || score);
END;

SQL> execute Score_features;
Score is 0

Internal helper types are created in the ORDSYS schema and do not have public synonyms. The internal helper types are the following.

colorsList

The syntax for this internal helper type is:

CREATE OR REPLACE TYPE colorsList AS VARRAY(100) OF SI_Color;

This internal helper type is used to specify the SI_ColorsList attribute of the SI_ColorHistogram object type as described.

colorFrequenciesList

The syntax for this internal helper type is:

CREATE OR REPLACE TYPE colorFrequenciesList AS VARRAY(100) OF
DOUBLE PRECISION;

This internal helper type is used to specify the SI_FrequenciesList attribute of the SI_ColorHistogram object type as described.

colorPositions

The syntax for this internal helper type is:

CREATE OR REPLACE TYPE colorPositions AS VARRAY(9) OF
SI_Color;

This internal helper type is used to specify the SI_ColorPositions attribute of the SI_PositionalColor object type as described.

textureEncoding

The syntax for this internal helper type is:

CREATE OR REPLACE TYPE textureEncoding AS VARRAY(5) of DOUBLE
PRECISION;

This internal helper type is used to specify the SI_TextureEncoding attribute of the SI_Texture object type as described.

The rich semantic nature of multimedia data creates difficulties for information retrieval that require the application of advanced techniques. Although queries can be carried out based on attributes and text annotation, the goal is to achieve content-based retrieval. In this chapter we introduced several techniques that can be used to support multimedia queries.

In present-day image applications the main process involved is to identify various features (color, texture, shape, etc.) of all the images in the database and to use these features to classify the images into groups. This has led to the development of a framework for image retrieval that we have seen several examples of in the practical exercises. It can be summarized as follows.