5.1 Introduction

In many cases, we can face with a task of collecting data of different nature obtained (registered, recorded, and measured) in different time slots from sources of different types. For example, long‐life medical observation of a patient is such a case: data about a patient's health status can be obtained from a wide range of medical equipment and tools that can be both digital (e.g. a magnetic imaging system) and nondigital (e.g. a mercury thermometer), as well as these datasets can be registered both in paper notes such as the so‐called “patient's medical card” and by using electronic medical documenting systems. The next task after data collection is data synchronization. This task is a part of a wider task of data analysis of all collected multimodal datasets in order to get a whole picture on the object (subject, process, and event) status, behavior, further states, etc.

One of the possible ways toward solving the task of multimodal data analysis is presenting data sets describing the same object as a complex data structure called an aggregate introduced in the Algebraic System of Aggregates (ASA) [1,2].

Formally, an aggregate A is a tuple of arbitrary tuples, whose elements belong to certain sets [1,2]:

(5.1)

where {A} is a tuple of sets M_j and 〈A〉 is a tuple of element tuples corresponding to the tuple of sets ().

If multimodal data are defined (registered, recorded, and measured) in terms of time, an aggregate must have a time value sequence as the first tuple. Such timewise aggregate is called a multi‐image [1,2]. The multi‐image is a data structure that describes an object in such complete way as it is enabled by the available data. Thus, a multi‐image is a nonempty aggregate formalized as:

(5.2)

where T is a set of time values; τ ≥ n_i, i ∈ [1, …, N].

For example, if there are three tuples of the values measured simultaneously, namely, systolic blood pressure (sp), diastolic blood pressure (dp), and heart rate (hr), as well as there is a tuple of time moments when these values are being measured, then we can present all the measurements as one complex data structure – a multi‐image I:

However, the composition of multimodal data in one multi‐image becomes a nontrivial task if the data values have been measured in different time moments. To process such data sequences, we need to synchronize them at first. This task is especially important in cases when multimodal data are being collected for a relatively long time from sources of different types such as remote sensors, paper archives, cloud storages, etc.

In this chapter, we discuss operations on multimodal data and relations between them, which enable complex representation of multiple multimodal data sequences obtained with respect to time in order to compose multi‐images and process them. We pay special attention to multimodal data synchronization by using both crisp and fuzzy approaches.

5.2 Basic Statements of ASA

The ASA is an algebraic system [3–6] that consists of sets (M, F, R), where M is a nonempty set (carrier), whose elements are the elements of the system, F is a set of operations, and R is a set of relations.

As mentioned above, the carrier of ASA is an arbitrary set of specific structures called aggregates. Tuple elements in an aggregate A can have both crisp values a_i and fuzzy values .

A tuple element can be empty ∅; a tuple may be empty as well. For example, in Eq. (5.3), the second tuple is empty.

(5.3)

If an aggregate consists of only empty tuples, it is called an empty aggregate:

An aggregate that does not include any component is called a null‐aggregate: A_∅ = 〚∅ ∣ 〈∅〉〛. The null‐aggregate plays a role of a neutral element in ASA.

A tuple element can be undefined, and its notation is _. A tuple may be undefined as well. If an aggregate consists of undefined tuples, it is called an undefined aggregate: The practical meaning of an undefined aggregate is that we can predefine a data structure (aggregate) even if data sequences are currently unavailable, e.g. if remote sensors are off and, thus, data are not received yet; however, we know their type and can predefine the aggregate sets and tuples.

The aggregate length ∣ A∣ is the quantity of tuples in it. The length of the empty aggregate is ∣ A∣ = N. The length of the undefined aggregate is also ∣ A∣ = N. At the same time, the length of the null‐aggregate is ∣ A_∅∣ = 0.

The sequence order of sets and corresponding tuples in an aggregate defines how operations on the aggregate will be fulfilled. In this regard, two aggregates, A₁ and A₂, can be compatible, A₁ ≑ A₂, quasi‐compatible, A₁ ≐ A₂, incompatible, A₁ ≗ A₂, or hiddenly compatible, A₁(≑)A₂.

• Aggregates A₁ and A₂ are compatible if {A₁} ≡ {A₂}, i.e. both aggregates have the same set of sets and the sequence order of the sets in both aggregates is the same.
• Aggregates A₁ and A₂ are quasi‐compatible if {A₁} ≢ {A₂} and {A₁} ∩ {A₂} ≠ ∅, i.e. the type and sequence order of these aggregates coincide partly.
• Aggregates A₁ and A₂ are incompatible if {A₁} ∩ {A₂} = ∅.
• Aggregates A₁ and A₂ are hiddenly compatible if {A₁} ≢ {A₂}, ∣ A₁∣ = ∣ A₂∣ = N, and ∀M_j ⊂ {A_k}, j = [1, … , N], k = [1, 2], i.e. both aggregates have the same set of sets, but the order of these sets differs.

For example, let the sets be defined in the following way:

• M_t = [35.0, …, 39.9] is a set of temperature values (°C);
• M_hr = [50, …, 110] is a set of heart rate values (bpm);
• M_sp = [80, …, 190] is a set of systolic pressure values (mmHg);
• M_dp = [55, …, 100] is a set of diastolic pressure values (mmHg).

Let us consider the following aggregates whose elements belong to these sets:

• A₁ = 〚M_t, M_hr ∣ 〈36.4,36.1,36.3,36.2,36.5,36.3〉, 〈75,76,74,73,75,75〉〛;
• A₂ = 〚M_t, M_hr ∣ 〈36.5,36.5,36.8,36.6,36.3,36.4,37.0,36.5〉, 〈74,81,76,93,97,97,96〉〛;
• A₃ = 〚M_sp, M_dp ∣ 〈185,166,175,166,171,152〉, 〈76,73,74,73,71,76〉〛;
• A₄ = 〚M_t, M_sp ∣ 〈36.5,36.5,36.8,36.6,36.3,36.4〉, 〈177,159,174,155,167,150〉〛;
• A₅ = 〚M_hr, M_t ∣ 〈74,81,76,93,97〉, 〈36.5,36.5,36.3,36.4,37.0,36.5〉〛.

Then, we can conclude that

1. – A₁ ≑ A₂ because the first tuple in both A₁ and A₂ belongs to set M_t and the second tuple in both A₁ and A₂ belongs to set M_hr.
2. – A₁ ≗ A₃ because the first tuple of A₁ belongs to set M_t, whereas the first tuple of A₃ belongs to M_sp, as well as the second tuple of A₁ belongs to set M_hr, whereas the second tuple of A₃ belongs to M_dp.
3. – A₁ ≐ A₄ because the first tuple of both A₁ and A₄ belongs to set M_t, but at the same time, the second tuple of A₁ belongs to set M_hr, whereas the second tuple of A₄ belongs to M_sp.
4. – A₁(≑)A₅ because now they are incompatible, but they will become compatible if we change the order of tuples in one of them (e.g. in A₅): . It can be fulfilled by applying an ordering operation to the elements of this tuple.

5.3 Operations on Aggregates and Multi‐images

Operations on aggregates include arithmetical operations, logical operations, and ordering operations. Arithmetical operations include elementwise addition, scalar addition, elementwise subtraction, scalar subtraction, elementwise multiplication, scalar multiplication, elementwise division, and scalar division. The logical operations on aggregates are union; intersection; difference; symmetric difference; and exclusive intersection [1].

The union of two aggregates A₁ and A₂ is aggregate B, which includes components of both aggregates and is formed according to the following rule:

1. If aggregates A₁ and A₂ are such as A₁ ≑ A₂ and
   
   

   then the elements of i‐tuple of A₂ are added at the end of i‐tuple of A₁:

   (5.4)
2. If aggregates A₁ and A₂ are such as A₁ ≗ A₂ and
   
   

   then the tuples of A₂ are added at the end of the tuple of tuples of A₁ and the corresponding sets of A₂ are added at the end of the set sequence of A₁:

   (5.5)
3. If aggregates A₁ and A₂ are such as A₁ ≐ A₂ and
   
   

   then the elements of i‐tuple of A₂ are added at the end of i‐tuple of A₁ if the elements of these tuples belong to the same set; otherwise, the rule for incompatible aggregates is applied:

   (5.6)

The intersection of two aggregates A₁ and A₂ is aggregate B, which includes only common components of both aggregates and is formed according to the following rule:

1. If aggregates A₁ and A₂ are such as A₁ ≑ A₂ and
   
   

   then B includes the elements of both aggregates, which are common for them, in every tuple:

   (5.7)

   where

   
   
   

   It means that if we go through elements of from its first element to its last element and compare each next element (1 ≤ l_i ≤ l, where l is the total number of elements in ) with elements of , then is the first element of , which is also present in , and is the last element of , which is also present in .

   Because we find intersection of and , we also look for and . To find them, we go through elements of from its first element to its last element and compare each next element (1 ≤ r_j ≤ r, where r is the total number of elements in ) with elements of , then is the first element of , which is also present in , and is the last element of , which is also present in . Finally, the result of intersection of and is the tuple with elements .

   For example, if and , then and . At the same time, as well as . Finally, .

2. If aggregates A₁ and A₂ are such as A₁ ≗ A₂, then B = A₁ ∩ A₂ = 〚∅ ∣ 〈∅〉〛 = A_∅.
3. If aggregates A₁ and A₂ are such as A₁ ≐ A₂ and
   
   

   then B includes elements of both aggregates, which are common for them, only in tuples of common sets; thus, the number of sets shortens. Only considering the common tuples, we proceed as in the first case:

   (5.8)

The difference of two aggregates A₁ and A₂ is aggregate B, which includes only components present in A₁ and absent in A₂; it is formed according to the following rule:

1. If aggregates A₁ and A₂ are such as A₁ ≑ A₂ and
   
   

   then B includes the elements of A₁, which are absent in A₂, in every tuple:

   (5.9)

   where

   
   
   
2. If aggregates A₁ and A₂ are such as A₁ ≗ A₂, then B = A₁A₂ = A₁.
3. If aggregates A₁ and A₂ are such as A₁ ≐ A₂ and
   
   

   then B includes the elements of A₁, which are absent in A₂, in tuples of common sets and all tuples of sets defined only in A₁:

   (5.10)

   where

   
   

The symmetric difference of two aggregates A₁ and A₂ is aggregate B, which includes both components present in A₁ and absent in A₂ and components present in A₂ and absent in A₁; it is formed according to the following rule:

1. If aggregates A₁ and A₂ are such as A₁ ≑ A₂ and
   
   

   then B includes the elements of A₁, which are absent in A₂, and elements of A₂, which are absent in A₁, in every tuple:

   (5.11)

   where

   
   
   
2. If aggregates A₁ and A₂ are such as A₁ ≗ A₂ and
   
   

   then B is equal to the union of A₁ and A₂:

   (5.12)
3. If aggregates A₁ and A₂ are such as A₁ ≐ A₂ and
   
   

   then B includes elements of A₁, which are absent in A₂, and elements of A₂, which are absent in A₁, in tuples of common sets, all tuples of sets defined only in A₁, and all tuples of sets defined only in A₂:

   (5.13)

   where

   
   

The exclusive intersection of two aggregates A₁ and A₂ is aggregate B, which includes only components of A₁ common for both aggregates and is formed according to the following rule:

1. If aggregates A₁ and A₂ are such as A₁ ≑ A₂ and
   
   

   then B includes the elements of A₁, which are common for both A₁ and A₂, in every tuple:

   (5.14)

   where

   
   
   
2. If aggregates A₁ and A₂ are such as A₁ ≗ A₂, then B = A₁ ¬ A₂ = 〚∅ ∣ 〈∅〉〛 = A_∅.
3. If aggregates A₁ and A₂ are such as A₁ ≐ A₂ and
   
   

   then B includes the elements of A₁, which are common for both A₁ and A₂, only in tuples of common sets; thus, the number of sets shortens:

   (5.15)

   where

   
   

The difference between exclusive intersection and intersection is that the result of intersection of two aggregates A₁ and A₂ is the aggregate that includes common components of both aggregates. At the same time, the result of exclusive intersection is the aggregate that includes components of A₁, which are also present in A₂, but it does not include any components of A₂. For example, if we have two compatible aggregates A₁ and A₂ such as:

Then, intersection gives us A₁ ∩ A₂ = 〚M_t, M_hr ∣ 〈36.4,36.3,36.5,36.3,36.5,36.5,36.3,36.4,36.5〉, 〈76,74,74,76〉〛. At the same time, exclusive intersection is resulted in A₁ ¬ A₂ = 〚M_t, M_hr ∣ 〈36.4,36.3,36.5,36.3〉, 〈76, 74〉〛.

If we have two quasi‐compatible aggregates A₁ and A₃, where A₃ is defined as:

Then, we get a result in a similar way but only for the first tuples because they both belong to the same set M_t:

The logical operations in ASA are noncommutative because the sequence order is important in tuples; this property distinguishes them from the logical operations on sets.

Ordering operations include sets ordering, ascending sorting, descending sorting, singling, extraction, and insertion [2].

The sets ordering operation reorders an aggregate according to a template aggregate. The template aggregate can be arbitrary, undefined, or empty.

Let the aggregate A be defined as:

(5.16)

Besides, let the template aggregate A_tem be defined as:

(5.17)

Then, the result of sets ordering operation on the aggregates A and A_tem is aggregate B defined as:

(5.18)

The sets ordering operation can also be applied to two arbitrary aggregates A₁ and A₂, where A₂ is used as a template. The practical meaning of such operation is that we can reorder one aggregate (A₁) according to another aggregate (A₂). Thus, if A₁(≑)A₂, these aggregates become compatible as a result of sets ordering operation and we can further handle them in the same way, e.g. we can compare the first tuple of A₁ with the first tuple of A₂ and they will be comparable – which was not possible before reordering because the “old” first tuple of A₁ and the first tuple of A₂ belonged to different sets.

The result of sets ordering operation depends on the aggregates' compatibility:

1. – If A₁ ≑ A₂, then sets ordering operation is resulted in aggregate B, which is equal to A₁:
   
2. – If A₁ ≐ A₂ and there is no hidden compatibility between A₁ and A₂, then the result of sets ordering operation is empty aggregate:

   B = A₁ ⊨ A₂ = 〚{A₂} ∣ 〈∅〉〛.

3. – If A₁ ≗ A₂ and there is no hidden compatibility between A₁ and A₂, then the result of sets ordering operation is aggregate B:
   (5.19)

   where , , , 〈B〉 ≢ 〈A₁〉.

4. – If A₁(≑)A₂, then the result of sets ordering operation is aggregate B such that B ≑ A₂ and 〈B〉 ≡ 〈A₁〉:
   (5.20)

   where .

For example, if we have aggregates A₁, A₂, and A₃ such as A₁ ≗ A₂, A₂(≑) A₃, which are defined in the following way:

Then, we can obtain the following results of sets ordering operation:

The sorting operations are ascending sorting and descending sorting. These operations enable reordering of all tuples according to new – sorted – elements order (ascending or descending) of a certain tuple (called a primary tuple) among all tuples of the aggregate.

Let and ∃k such as 1 < k < N, k ≠ 2 and n₁ > n_k > n_N, n₂ = n_k. Then, the result of ascending sorting operation of A₁ according to the elements of tuple is aggregate B such as:

(5.21)

where and n = n_k if n_j ≥ n_k or n = n_j if n_j < n_k.

The result of descending sorting operation of A₁ according to the elements of tuple is aggregate B such as:

(5.22)

If k = 1, k = 2, or k = N, the sorting operation is fulfilled by the same principle.

If n₁ = n₂ = ⋯ = n_k = ⋯ = n_N, the result of ascending sorting operation is:

(5.23)

and the result of descending sorting operation is:

(5.24)

A variant of the sorting operations is the sorting operations with appending (ascending sorting with appending and descending sorting with appending), which allow to increase the length of shorter tuples according to the length of the primary tuple by adding a value x (x ∈ [∅, _, q], q ∈ M_j, 1 ≤ j ≤ N) either to the end or to the beginning of the shorter tuples.

The result of ascending sorting with appending at the end of the shorter tuples of aggregate A₁ is aggregate B:

(5.25)

where .

The result of ascending sorting with appending at the beginning of the shorter tuples of the aggregate A₁ is aggregate B:

(5.26)

where .

The result of descending sorting with appending at the end of the shorter tuples of aggregate A₁ is aggregate B:

(5.27)

The result of descending sorting with appending at the beginning of the shorter tuples of the aggregate A₁ is aggregate B:

(5.28)

If aggregate A₁is defined as:

(5.29)

where 1 ≤ k ≤ N, and let ∃m_l, ∀ l such as , then the result of singling operation on aggregate A₁ by the tuple is aggregate B such as:

(5.30)

If there is aggregate A₁ defined as:

(5.31)

then the result of extraction operation of the element from aggregate A₁ is aggregate B such as:

(5.32)

The operation of conditional extraction is defined on the assumption of a given condition such as or . For example: .

If there is aggregate A₁ defined as:

(5.33)

where 1 ≤ m ≤ N and let ∃d such as , then the result of insertion operation of d to aggregate A₁ can be obtained by using two equivalent ways defined in Eqs. (5.34) and (5.35).

(5.34)

(5.35)

The operation of conditional insertion is defined on the assumption of a given condition such as or . For example, .

The ordering operations in ASA allow us to reorder both elements in tuples and tuples in aggregates (multi‐images).

5.4 Relations and Digital Intervals

Relations in ASA include relations between tuple elements, relations between tuples, and relations between aggregates. Relations between tuple elements are is greater, is less, is equal, proceeds, and succeeds. The first three relations are based on element value and the last two relations concern elements position in a tuple. Naturally, elements must belong to the same tuple. Relations between tuples enable the following types of tuple comparison: arithmetical comparison, frequency comparison, and interval comparison. Arithmetical comparison is elementwise and based on the following relations: is strictly greater, is majority‐vote greater, is strictly less, is majority‐vote less, is strictly equal, and is majority‐vote equal.

Frequency comparison is based on the following relations: is thicker, is rarer, and is equally frequent. Interval comparison is based on the following temporal relations: coincides with, is before, is after, meets, is met by, overlaps, is overlapped by, during, contains, starts, is started by, finishes, and is finished by.

Relations between tuples of aggregates are identical to relations between single tuples defined above: relations of arithmetical comparison, relations of frequency comparison, and relations of interval comparison. However, the possibility of their application depends on the aggregates' compatibility: relations between tuples can be applied only to compatible and quasi‐compatible aggregates. Hiddenly compatible aggregates must be first transformed to compatible ones [2] and then a relation between tuples can be applied to them.

Because a multi‐image, by definition, is an aggregate, all types of relations defined in ASA can be applied to multi‐images as well.

A time tuple in a multi‐image, which is defined by (5.2), can be considered as an interval. However, in contrast with a classical interval, the time tuple consists of a finite number of discrete values. To show the difference between the time tuple and a classical interval, let us first discuss a classical approach of interval‐based temporal logic.

One of the pioneering works related to interval processing is [7], where Allen presented interval algebra and interval‐based temporal logic, including relations between intervals. If X and Y are intervals such as X = [x⁻, x⁺] and Y = [y⁻, y⁺], the relations between them are defined as shown in Table 5.1.

In [8], Allen and Hayes axiomatized a theory of time in terms of intervals and the single relation meet. They extended Allen's interval‐based theory by formally defining the beginnings and endings of intervals, which have properties normally associated with points. The authors distinguished between these point‐like objects and the concept of a moment as hypothesized in discrete time models.

In these and other related works [9–13], the relations of Allen's interval‐based theory are to be applied to an interval [x⁻, x⁺], where x⁻ ≤ x⁺. However, in ASA, we operate with discrete values and we cannot use Allen's interval‐based theory approaches directly. For example, when we measure data for further composing of a multi‐image, we operate with discrete time values, which being considered all together can be defined as a discrete time interval as it consists of specific discrete time points when the data values have been obtained. Thus, we need to make a transition from an interval [7–13] to a discrete interval.

Let us define a discrete interval as a tuple , whose elements are unique discrete values such as that either t_i < t_i + 1 or t_i > t_i + 1 is true for all pairs (t_i, t_i + 1), ∀i ∈ [1…n − 1], t_i ∈ ℝ. Thus, a discrete interval is a strictly increasing or decreasing finite sequence in ℝ.

Then, discrete intervals can be a subject of relations similar to those introduced by Allan in [7]. These temporal relations between two discrete intervals are coincides with, is before, is after, meets, is met by, overlaps, is overlapped by, during, contains, starts, is started by, finishes, and is finished by. Let us define these relations. Note that for consistency with works on Allen's interval algebra, we refer to in this research; hereinafter, we use notion for relations between digital intervals similar to that used in Allen's interval algebra (e.g. e_d).

For two discrete intervals and , we define:

1. – The relation coincides with as:
   (5.36)
2. – The relation is before as:
   (5.37)
3. – The relation is after as:
   (5.38)
4. – The relation meets as:
   (5.39)
5. – The relation is met by as:
   (5.40)
6. – The relation overlaps as:
   (5.41)
7. – The relation is overlapped by as:
   (5.42)
8. – The relation during as:
   (5.43)
9. – The relation contains as:
   (5.44)
10. – The relation starts as:
    (5.45)
11. – The relation is started by as:
    (5.46)
12. – The relation finishes as:
    (5.47)
13. – The relation is finished by as:
    (5.48)

Note that since and are discrete intervals, there is no requirement that and (1 < i < n₁, 1 < j < n₂) must coincide in such relations as , , , , , , , , and . For example, if we have two tuples and as depicted in Figure 5.1.

Then, coincides with because and .

In this work, we also introduce the relation between, which can be applied to any number of discrete intervals:

1. – means that a discrete interval is between the discrete values α and β;
2. – means that both discrete intervals and are between the discrete values α and β;
3. – means that all discrete intervals ( j ∈ [1…n]) are between the discrete values α and β.

Thus, the relation between for two tuples and can be defined as follows:

(5.49)

where α and β are given values, α, β ∈ T; T is the time value set.

The relation between for two discrete intervals can be considered as a generalized form of the relation coincides with, because the relation between (5.49) is equivalent to the relation coincides with (5.36) if both and .

5.5 Data Synchronization

Because a multi‐image is a digital representation of a real‐world object (process, phenomenon, event, subject), data tuples of different modalities, which are obtained from different sources (sensors, cloud storages, local storages, computing resources, etc.), need to be synchronized between each other to enable constructing the proper multi‐image as well as to compare and analyze several multi‐images in the same processing procedure. Mathematical models of data synchronization will enable further development of algorithms as well as software for multimodal data processing.

Let us consider two multi‐images I₁ and I₂, which present a state of the same object of study and are defined as:

(5.50)

We assume that one characteristic of this object has been measured in time moments , and as a result, we obtained the data tuple ; another characteristic has been measured in time moments , and as a result, we obtained the data tuple . Since I₁ and I₂ describe different sides of behavior of the same object, our task is to compose a joint multi‐image I that consists of the joint time tuple and the synchronized data tuples and . The tuple includes values of and empty elements and is formed according to the rule:

(5.51)

Similarly, the tuple includes values of and empty elements and is formed according to the rule:

(5.52)

In general, the joint multi‐image I is formed as a result of three operations, viz., union defined by (5.6), sorting defined by (5.21), and singling defined by (5.30):

(5.53)

The union enables consolidation of two multi‐images I₁ and I₂ in one multi‐image , but if time tuples in I₁ and I₂, i.e. and , include the same values, then will include duplicated elements and, thus, the synchronization of data tuples and will be incorrect. To avoid it, we must sort by time tuple values by using sorting operation and, next, remove duplicated elements in the joint time tuple by using singling operation.

Let us consider the following practical task as an example. We assume that two parameters, viz., a temperature and an erythrocyte sedimentation rate, have being measured for the same patient during four‐week health status monitoring. Because these measurements are of different nature and fulfilled by different hospital units, we obtain two multi‐images, even if measurements of both types were obtained at the same days of a month:

where

• M_t = [36.0, …, 39.9] is a set of temperature values (°C);
• M_esr = [2, …, 20] is a set of erythrocyte sedimentation rate values (mm/h);
• T = [1, …, 31] is a set of time values (day of a month).

If we try to apply only union operation to I₁ and I₂, we obtain wrong synchronization because data of different types do not correspond each other: .

To correct it, we need to sort in ascending order of time values and then remove duplicated time values:

Now, all data correspond each other, e.g. we can see that on the 16th day of monitoring, the patient had the body temperature 37.3 °C and the erythrocyte sedimentation rate in his blood test was 15 mm/h.

In this example, we considered the particular case when time values fully coincide with ; however, in a general case, time tuples can be connected with any temporal relation defined by (5.36)–(5.49). Thus, to find time values t_j satisfying the condition in (5.51) and/or (5.52), we need to analyze interval relations between time value tuples and and compose the joint time value tuple , which includes all time values from both and but do not duplicate them if some of time values in and coincide. To do this, we need to compare value intervals of and .

The simplest case is when the measurements of both and data tuples have been fulfilled simultaneously (hereinafter, we suppose that the data values are obtained from sensors as a result of measuring certain parameters of a physical process; however, the data can also be obtained as a result of modeling, processing, prediction, simulation, etc.). It means that and are connected by the relation coincides with. In general, the relation coincides with between two tuples and is defined in Eq. (5.36). This relation enables many different subcases depending on a number of values in each data tuple. Let us consider all possible subcases (Figures 5.2–5.4).

The first group of subcases for is when n₁ is an even value (n₁ mod 2 = 0) and , i.e. is equally frequent to ; it means that (i.e. n₁ = n₂).

If , ∀i ∈ [1…n₁], then n = n₁ and the tuple values of the multi‐image I as introduced in (5.2) are defined in (5.54) and depicted in Figure 5.2a.

(5.54)

The case when and can be illustrated by Figure 5.2b. Such mutual alignment of the time moments , , , and , when data values , , , and have been measured, requires synchronization of tuple elements which can be stated as:

(5.55)

It means that in time moment , we have both data values and measured (consider the values at the left column above as well as the graphical elements at the left side of Figure 5.2b); in time moment , we have only measured (consider the central column as well as the central part of Figure 5.2b); in time moment , we have only measured (consider the right column as well as the right side of Figure 5.2b).

The rule for j index calculation is , where i ∈ [2…(n₁ − 2)] and i is an even value (i mod 2 = 0). The total number of elements in each of the joint tuples, viz., , , and , is . If we consider the first and the last values of each joint tuple when and are connected with relation coincides with, we obtain the following:

(5.56)

Similarly, if and , i mod 2 = 0, , and , then the tuple value synchronization can be illustrated by Figure 5.2c. This case requires data synchronization defined in (5.57).

(5.57)

If and (Figure 5.2d), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then , , and the tuple values of the multi‐image I are as follows:

(5.58)

If and (Figure 5.2e), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I can be obtained by using (5.59).

(5.59)

If , , and (Figure 5.2f), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I can be defined as:

(5.60)

If , , and (Figure 5.2g), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then , , and the tuple values of the multi‐image I are as follows:

(5.61)

If (Figure 5.2h), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I are defined in (5.62).

(5.62)

If and (Figure 5.2i), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I are as follows:

(5.63)

If and (Figure 5.2j), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then , , and the tuple values of the multi‐image I are defined in (5.64).

(5.64)

If , , and (Figure 5.2k), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I can be defined as follows:

(5.65)

If (Figure 5.2l), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then , , and the tuple values of the multi‐image I are as follows:

(5.66)

If (Figure 5.2m), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I are defined in (5.67).

(5.67)

The second group of subcases for is when n₁ is an odd value (n₁ mod 2 ≠ 0) and , i.e. is equally frequent to ; it means that (i.e. n₁ = n₂). Mutual alignment of i‐elements and (i + 1)‐elements in both data sequences for these subcases is similar to the subcases of the first group and, therefore, it is also illustrated in Figure 5.2. However, since n₁ is an odd value, it complicates data synchronization because we need to analyze mutual alignment of and as well.

If (Figure 5.2a), ∀i ∈ [1…n₁], then the tuple values can be defined in the same way as it has been set in (5.54).

If , (Figure 5.2b), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0 and , then , and the tuple values of the multi‐image I can be defined as follows:

(5.68)

If , (Figure 5.2b), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I are defined in (5.69).

(5.69)

If , (Figure 5.2b), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I are as follows:

(5.70)

If , (Figure 5.2c), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I are defined as:

(5.71)

If , (Figure 5.2c), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I are stated in (5.72).

(5.72)

If , (Figure 5.2c), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I are defined in (5.73).

(5.73)

If , (Figure 5.2d), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I are as follows:

(5.74)

If , (Figure 5.2d), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I can be defined as follows:

(5.75)

If , (Figure 5.2d), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I can be stated as follows:

(5.76)

If , (Figure 5.2e), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2n₁ − 3, and the tuple values of the multi‐image I are as follows:

(5.77)

If , (Figure 5.2e), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then  j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I are defined in (5.78).

(5.78)

If , (Figure 5.2e), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then  j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I are as follows:

(5.79)

If , , (Figure 5.2f), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2n₁ − 3, and the tuple values of the multi‐image I are defined in (5.80).

(5.80)

If , , (Figure 5.2f), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I can be obtained as follows:

(5.81)

If , , (Figure 5.2f), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I are defined in (5.82).

(5.82)

If , , (Figure 5.2g), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I are as follows:

(5.83)

If , , (Figure 5.2g), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I can be obtained by using (5.84).

(5.84)

If , , (Figure 5.2g), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I are as follows:

(5.85)

If (Figure 5.2h), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0 and , then j = 2i, n = 2n₁ − 3, and the tuple values of the multi‐image I can be found as defined in (5.86).

(5.86)

If (Figure 5.2h), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I are stated as follows:

(5.87)

If (Figure 5.2h), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0 and , then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I can be defined as:

(5.88)

If , (Figure 5.2i), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2n₁ − 3, and the tuple values of the multi‐image I are as follows:

(5.89)

If , (Figure 5.2i), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I are defined as:

(5.90)

If , (Figure 5.2i), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I are stated in (5.91).

(5.91)

If , (Figure 5.2j), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I are defined in (5.92).

(5.92)

If , (Figure 5.2j), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I are as follows:

(5.93)

If , (Figure 5.2j), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I can be defined as:

(5.94)

If , , (Figure 5.2k), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2n₁ − 3, and the tuple values of the multi‐image I are defined in (5.95).

(5.95)

If , , (Figure 5.2k), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I are as follows:

(5.96)

If , , (Figure 5.2k), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I can be obtained as follows:

(5.97)

If (Figure 5.2l), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0 and , then , , and the tuple values of the multi‐image I are defined in (5.98).

(5.98)

If (Figure 5.2l), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I are as follows:

(5.99)

If (Figure 5.2l), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then , , and the tuple values of the multi‐image I can be obtained by using (5.100).

(5.100)

If (Figure 5.2m), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2n₁ − 3, and the tuple values of the multi‐image I are as follows:

(5.101)

If (Figure 5.2m), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, and , then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I can be found as defined in (5.102).

(5.102)

If (Figure 5.2m), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0 and , then j = 2i, n = 2(n₁ − 1), and the tuple values of the multi‐image I are as follows:

(5.103)

The third group of subcases (Figure 5.3) for is when , i.e. is rarer than ; it means that . Because ratio R between and can be various, let us assume that for every i‐element in , there are two elements in , except and , which coincide with and , respectively (Figure 5.3). It means that n₂ mod 2 = 0 and .

If (Figure 5.3a), ∀i ∈ [2…(n₂ − 2)] such as i mod 2 = 0, then n = n₂ and the tuple values of the multi‐image I can be defined as:

(5.104)

If (Figure 5.3b), ∀i ∈ [2…(n₂ − 2)] such as i mod 2 = 0, then , , and the tuple values of the multi‐image I are as follows:

(5.105)

If (Figure 5.3c), ∀i ∈ [2…(n₂ − 2)] such as i mod 2 = 0, then n = n₂ and the tuple values of the multi‐image I are defined in (5.106).

(5.106)

If (Figure 5.3d), ∀i ∈ [2…(n₂ − 2)] such as i mod 2 = 0, then , , and the tuple values of the multi‐image I are as follows:

(5.107)

If and (Figure 5.3e), ∀i ∈ [2…(n₂ − 2)] such as i mod 2 = 0, then , , and the tuple values of the multi‐image I are defined as:

(5.108)

The forth group of subcases for is when , i.e. is thicker than ; it means that . Similar to the third group (), ratio R between and can vary. Thus, we assume that for every i‐element in , there are two elements in , except and , which coincide with and , respectively (Figure 5.4). It means that n₁ mod 2 = 0 and .

If (Figure 5.4a), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then n = n₁ and the tuple values of the multi‐image I can be defined as:

(5.109)

If (Figure 5.4b), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then , , and the tuple values of the multi‐image I are as follows:

(5.110)

If and (Figure 5.4c), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then , , and the tuple values of the multi‐image I are defined in (5.111).

(5.111)

If (Figure 5.4d), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then , , and the tuple values of the multi‐image I are as follows:

(5.112)

If (Figure 5.4e), ∀i ∈ [2…(n₁ − 2)] such as i mod 2 = 0, then n = n₁ and the tuple values of the multi‐image I are defined in (5.113).

(5.113)

Thus, we presented all possible subcases for the relation coincides with.

The case, which is more general than defined above, is when the same number of measurements of both and have been fulfilled in the same period of time from the moment α to the moment β but not simultaneously. It means that and are connected by the relation between defined in (5.49). The difference between the relations coincides with and between consists in defining the first and the last values of tuples in a multi‐image. The first values of tuples are to be defined based on comparison of and :

(5.114)

The last values of tuples are to be defined based on comparison of and :

(5.115)

The rest of tuple elements are to be defined in the same way as stated in (5.54)–(5.113).

The case, when measuring of forestalls measuring of , means that and are connected by the relation is before defined in (5.37). Thus, if , then ∀i₁ ∈ [1…n₁], ∀i₂ ∈ [1…n₂]:

(5.116)

The case, when measuring of succeeds measuring of , means that and are connected by the relation is after defined in (5.38). Thus, if , then ∀i₁ ∈ [1…n₁], ∀i₂ ∈ [1…n₂]:

(5.117)

The case, when measuring of forestalls measuring of , but measuring of the last value of coincides with measuring of the first value of , means that and are connected by the relation meets defined in (5.39). Thus, if then ∀i₁ ∈ [1…(n₁ − 1)], ∀i₂ ∈ [2…n₂]:

(5.118)

The case, when measuring of succeeds measuring of , but measuring of the first value of coincides with measuring of the last value of , means that and are connected by the relation is met by defined in (5.40). Thus, if , then ∀i₁ ∈ [2…n₁], ∀i₂ ∈ [1…(n₂ − 1)]:

(5.119)

The case, when measuring of forestalls measuring of , but measuring of K last values of coincides with measuring of K first values of , means that and are connected by the relation overlaps defined in (5.41). Thus, if , then ∀i₁ ∈ [1…(n₁ − K)], ∀i₂ ∈ [(K + 1)…n₂], ∀k ∈ [1…K]:

(5.120)

The case, when measuring of succeeds measuring of , but measuring of K first values of coincides with measuring of K last values of , means that and are connected by the relation is overlapped by defined in (5.42). Thus, if , then ∀i₁ ∈ [(K + 1)…n₁], ∀i₂ ∈ [1…(n₂ − K)], and ∀k ∈ [1…K]:

(5.121)

The case, when measuring of starts later and at the same time it finishes earlier than measuring of does, means that and are connected by the relation during defined in (5.43). Thus, if , therefore, such as , , and . Then, ∀ k₁ ∈ [1…K₁], ∀ k₂ ∈ [K₂…n₂] such as and , we can define the first values (), , , and the last values (), , of the resulting tuples , , and as defined in (5.122). The rest of tuple elements (, , ) of the resulting tuples , , and can be obtained accordingly to subcases defined in (5.54)–(5.115). Then, the length of the resulting tuple is n = K₁ + m + n₂ − K₂ + 1, where m is the length of the tuple obtained as a result of synchronization of tuples and .

(5.122)

The case, when measuring of starts earlier and at the same time it finishes later than measuring of does, means that and are connected by the relation contains defined in (5.44). Thus, if , therefore, such as , , and . Then, ∀ k₁ ∈ [1…K₁] and ∀ k₂ ∈ [K₂…n₁] such as and , we can define the first values (), , and and the last values (), , and of the resulting tuples , , and as defined in (5.123). The rest of tuple elements (, , ) of the resulting tuples , , and can be obtained accordingly to subcases defined in (5.54)–(5.115). Then, the length of the resulting tuple is n = K₁ + m + n₁ − K₂ + 1, where m is the length of the tuple obtained as a result of synchronization of tuples and .

(5.123)

The case, when measuring of both and start simultaneously, but measuring of finishes earlier than measuring of does, means that and are connected by the relation starts defined in (5.45). Thus, if , therefore, such as and . Then, ∀k₂ ∈ [K…n₂] such as , we can define the first value t₁(), , and and the last values (), , and of the resulting tuples , , and as defined in (5.124). The rest of tuple elements (, , , and ∀k₁ ∈ [2…m], where m is the length of the tuple obtained as a result of synchronization of tuples and ) of the resulting tuples , , and can be obtained accordingly to subcases defined in (5.54)–(5.115). Then, the length of the resulting tuple is n = m + n₂ − K + 1.

(5.124)

The case, when measuring of both and start simultaneously, but measuring of finishes later than measuring of does, means that and are connected by the relation is started by defined in (5.46). Thus, if , therefore, such as and . Then, ∀k₂ ∈ [K…n₁] such as , we can define the first value t₁(), , , and the last values (), , and of the resulting tuples , , and as defined in (5.125). The rest of tuple elements (, , , and ∀ k₁ ∈ [2…m], where m is the length of the tuple obtained as a result of synchronization of tuples and ) of the resulting tuples , , and can be obtained accordingly to subcases defined in (5.54)–(5.115). Then, the length of the resulting tuple is n = m + n₁ − K + 1.

(5.125)

The case, when measuring of both and finish simultaneously, but measuring of starts later than measuring of does, means that and are connected by the relation finishes defined in (5.47). Thus, if , therefore, such as and . Then, ∀ k₁ ∈ [1…K], such as , we can define the first values (), , and and the last value t_n (), , and of the resulting tuples , , and as defined in (5.126). The rest of tuple elements (, , , and ∀k₂ ∈ [(K + 1)…(n − 1)], where the length of the resulting tuple is n = K + m; m is the length of the tuple obtained as a result of synchronization of tuples and ) of the resulting tuples , , and can be obtained accordingly to subcases defined in (5.54)–(5.115).

(5.126)

The case, when measuring of both and finish simultaneously, but measuring of starts earlier than measuring of does, means that and are connected by the relation is finished by defined in (5.48). Thus, if , therefore, such as and . Then, ∀k₁ ∈ [1…K], such as , we can define the first values (), , and and the last value t_n (), , and of the resulting tuples , , and as defined in (5.127). The rest of tuple elements (, , and , ∀k₂ ∈ [(K + 1)…(n − 1)], where the length of the resulting tuple is n = K + m; m is the length of the tuple obtained as a result of synchronization of tuples and ) of the resulting tuples , , and can be obtained accordingly to subcases defined in (5.54)–(5.115).

(5.127)

As one can see, in the case of crisp time value processing, there is a great number of cases for data synchronization. Thus, in applications where the crisp time values are not strictly required, it is reasonable to use fuzzy synchronization.

5.6 Fuzzy Synchronization

For fuzzy data synchronization of multi‐images, we need fuzzy equivalents of temporal relations defined by (5.36)–(5.49). There is a number of works on this topic, in particular, [11,14–18]. In this research, we apply the approach formulated in [11,18]. Let us present it in detail.

The degree to which the beginning of is approximately α(α ∈ ℝ) time units before the beginning of , where β (β ≥ 0) indicates a degree of tolerance, is a fuzzification generalization of strict inequality ; it is defined as follows:

(5.128)

where the operation ∘_T means composition of fuzzy relations as defined in (5.129); the operation ⊳_I means superproduct as defined in (5.130) according to [19]; T_w is Łukasiewicz t‐norm defined in (5.131); is a Łukasiewicz implicator defined in (5.132); is the fuzzy relation between time points and defined in (5.133).

The sup‐T composition of fuzzy relations is defined as:

(5.129)

where R₁ is a fuzzy relation from a set X to a set Y; R₂ is a fuzzy relation from a set Y to a set Z; x ∈ X, y ∈ Y, z ∈ Z.

The I‐superproduct of fuzzy relations is defined as:

(5.130)

The Łukasiewicz triangular norm and its corresponding implicator are given as:

(5.131)

where a, b ∈ [0, 1].

(5.132)

The fuzzy relation models a metric constraint between time points and is given as:

(5.133)

where α is the distance (in time units) between and and β is the degree of tolerance.

The degree to which the beginning of is approximately α time units before or the same time as the beginning of is a generalization of the nonstrict inequality ; it is defined as follows:

(5.134)

The degree to which the end of is approximately α time units before the end of is a generalization of the strict inequality ; it is defined as follows:

(5.135)

where the operation ⊲_I means subproduct as defined in (5.136) according to [19].

(5.136)

The degree to which the end of is approximately α time units before or the same time as the end of is a generalization of the nonstrict inequality ; it is defined as follows:

(5.137)

The degree to which the beginning of is approximately α time units before the end of is a generalization of the strict inequality ; it is defined as follows:

(5.138)

The degree to which the beginning of is approximately α time units before or the same time as the end of is a generalization of the nonstrict inequality ; it is defined as follows:

(5.139)

The degree to which the end of is approximately α time units before the beginning of is a generalization of the strict inequality ; it is defined as follows:

(5.140)

The degree to which the end of is approximately α time units before or the same time as the beginning of is a generalization of the nonstrict inequality ; it is defined as follows:

(5.141)

As one can see in Section 5.5, crisp data synchronization in the case when is quite complicated because of many subcases. We can simplify data synchronization if we consider the degree of closeness between time moments and . Such degree can be estimated as . Let us prove that if Δt is a minimal distance between and (we assume Δt as a time measurement error), then and, thus, and can be considered as one fuzzy moment of time .

According to (5.140), . From expressions (5.130) and (5.136), we obtain the following:

(5.142)

Taking into account (5.132), we have:

(5.143)

Similarly,

(5.144)

Thus, we can rewrite (5.142) as follows:

(5.145)

Then, we have from (5.133) that:

(5.146)

Since , we have in (5.146) because Δt is a time measurement error, i.e. it is the least possible distance between and .

Thus, and therefore . From practical point of view, it means that if there is a time measurement error in two time value sequences and (actually, it can happen in many cases if the measurements are taken simultaneously but by different executors, e.g. atmosphere pressure is being measured in one research unit and the sun's radiation is being measured by another research unit), we may neglect this time measurement error.

Now let the distance between time moments and exceed Δt and be equal to α. Then, in (5.146), we have ; thus, and therefore . Thus, the conclusion is that we again can consider these two time moments and as one fuzzy moment of time .

Let us consider different cases (Figure 5.5) for α between neighboring values in and when in order to simplify data value synchronization defined by (5.54)–(5.113). Let α₁ be a distance between and , let α₂ be a distance between and , and let α₃ be a distance between and . Besides, let α be a minimal distance between such and that we can consider them as one vague time moment to be presented as one fuzzy value in the time tuple of a multi‐image.

If both α₁ < α and α₃ < α are true (Figure 5.5a), then regardless of the value of α₂, the data synchronization can be simplified to (5.147).

(5.147)

If both α₁ < α and α₃ > α are true (Figure 5.5b), then regardless of the value of α₂, the data synchronization can be simplified to (5.148).

(5.148)

If both α₁ > α and α₂ < α are true (Figure 5.5c), then regardless of the value of α₃, the data synchronization can be simplified to (5.149).

(5.149)

If either α₁ > α > α₂ > α₃ or both α₁ > α > α₃ and α₂ > α (Figure 5.5d) is true, the data synchronization can be simplified to (5.150).

(5.150)

Thus, fuzzy synchronization allows us to simplify synchronization of data values measured at neighboring moments of time when the time difference is small enough that we can neglect this difference. We considered as the degree of closeness between time moments and but other relation degrees defined in can be employed depending on a specific condition of the data synchronization task.

When the data sequences have been synchronized, we can compose a multi‐image of the object of study and process this multi‐image (which is the complex representation of all information we know about this object) employing the logical operations, ordering operations, arithmetical operations, and relations, which are defined in ASA, according to a specific logic and by using specific methods of solving a given task related to the object of study.

5.7 Conclusion

Multimodal data can be obtained in a wide range of applications in industry, medicine, engineering, etc., as a result of measurement, computing, modeling, prediction, and other data processing. The majority of approaches consider multimodal data sequences as separate ones like temperature data sequence, pressure data sequence, etc. However, complex representation of all multimodal data sequences as a sole mathematical object can be conducive to more effective solving of tasks related to multimodal data analysis and processing. In this chapter, we presented the apparatus of ASA that enables such complex representation and processing of multimodal data in the form of aggregates and multi‐images.

Multi‐images data synchronization is an important task because it ensures correct interpretation of multimodal data and their correlation in terms of time. Multimodal data synchronization can be fulfilled by using relations defined in ASA. After synchronization, we obtain the multi‐image, which is a complete timewise data structure, ready for further complex data analysis in such tasks as data modeling, prediction, classification, clustering, and other tasks. Further data processing can be fulfilled by using a domain‐specific program language such as ASAMPL [20].

References

1. 1 Dychka, I. and Sulema, Y. (2019). Ordering operations in algebraic system of aggregates for multi‐image data processing. Research Bulletin of the National Technical University of Ukraine “Kyiv Polytechnic Institute” 1: 15–23.
2. 2 Dychka, I. and Sulema, Y. (2018). Logical operations in algebraic system of aggregates for multimodal data representation and processing. Research Bulletin of the National Technical University of Ukraine “Kyiv Polytechnic Institute” 6: 44–52.
3. 3 Maltsev, A.I. (1970). Algebraic Systems. Nauka.
4. 4 Fraenkel, A.A., Bar‐Hillel, Y., Levy, A. et al. (1973). Foundations of Set Theory. Elsevier.
5. 5 Kulik, B.A., Zuenko, A.A., and Fridman, A.Y. (2010). Algebraic Approach to Intellectual Processing of Data and Knowledge. Izdatelstvo Politekhnicheskogo Universiteta.
6. 6 Petrovsky, A.B. (2003). Space of Sets and Multi‐sets. Editorial URSS.
7. 7 Allen, J.F. (1983). Maintaining knowledge about temporal intervals. Communications of the ACM 26: 832–843. https://doi.org/10.1145/182.358434.
8. 8 Allen, J.F. and Hayes, P.J. (1989). Moments and points in an interval‐based temporal logic. Computational Intelligence 5 (3): 225–238. https://doi.org/10.1111/j.1467-8640.1989.tb00329.x.
9. 9 Nebel, B. and Bürckert, H.‐J. (1995). Reasoning about temporal relations: a maximal tractable subclass of Allen's interval algebra. Journal of the ACM 42 (1): 43–66.
10. 10 Allen, J.F. and Ferguson, G. (1997). Actions and events in interval temporal logic. In: Spatial and Temporal Reasoning, 205–245. Springer https://doi.org/10.1007/978-0-585-28322-7_7.
11. 11 Schockaert, S., De Cock, M., and Kerre, E. (2010). Reasoning About Fuzzy Temporal and Spatial Information from the Web, Intelligent Information Systems, vol. 3. World Scientific.
12. 12 Bozzelli, L. et al. (2016). Interval vs. point temporal logic model checking: an expressiveness comparison. Proceedings of the 36th IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS 2016), Chennai, India (13–15 December 2016). Chennai: Chennai Mathematical Institute.
13. 13 Grüninger, M. and Li, Z. (2017). The time ontology of Allen's interval algebra. Proceedings of 24th International Symposium on Temporal Representation and Reasoning (TIME 2017), Article No. 16, 16–18 October 2017, Mons, Belgium. LIPICS Schloss Dagstuhl – Leibniz‐Zentrum für Informatik GmbH, Dagstuhl Publishing, Saarbrücken/Wadern,Germany.
14. 14 Guesgen, H.W., Hertzberg, J. and Philpott, A. (1994). Towards implementing fuzzy Allen relations. Proceedings of the ECAI‐94 Workshop on Spatial and Temporal Reasoning. August 8–12 Amsterdam, The Netherlands.
15. 15 Bodenhofer, U. A new approach to fuzzy orderings. Tatra Mountains Mathematical Publications 16: 21–29.
16. 16 Badaloni, S. and Giacomin, M. (2000). A fuzzy extension of Allen's interval algebra. Lecture Notes in Artificial Intelligence 1792: 155–165.
17. 17 Dubois, D., Hadj Ali, A., and Prade, H. (2003). Fuzziness and uncertainty intemporal reasoning. Journal of Universal Computer Science 9: 1168–1194.
18. 18 Schockaert, S., De Cock, M., and Kerre, E. (2008). Fuzzifying Allen's temporal interval relations. IEEE Transactions on Fuzzy Systems 16 (2): 517–533.
19. 19 Bandler, W. and Kohout, L.J. (1980). Fuzzy relational products as a tool for analysis and synthesis of the behaviour of complex natural and artificial systems. In: Fuzzy Sets: Theory and Application to Policy Analysis and Information Systems (eds. S.K. Wang and P.P. Chang), 341–367. New York: Plenum.
20. 20 Sulema, Y. (2018). ASAMPL: programming language for mulsemedia data processing based on algebraic system of aggregates. In: Interactive Mobile Communication Technologies and Learning. IMCL 2017, Advances in Intelligent Systems and Computing, vol. 725, 431–442. Springer https://doi.org/10.1007/978-3-319-75175-7_43.