2.5.1Central tendency: Mean, median and mode

x1, x2, . . . , xN is the set of N observed values or observations for X. The values here may also be referred to as the dataset (forD). If the observations were plotted, where would most of the values fall? This would elicit an idea of the data central tendency. Measures of central tendency comprise the mean, median, mode and midrange [33–39].

The most effective and common numeric measure of the “center” of a set of data is the (arithmetic) mean.

Definition 2.5.1.1. The arithmetic mean of this set of values is given as follows:

$\overline{x}=\frac{{\displaystyle \sum {}_{i=1}^N{x}_i}}{N}=\frac{x_1+x_2+\dots +x_N}{N}\left(2.1\right)$

Example 2.4 Suppose that we have the following 10 values X = 3.5, 3.5, 4, {2.5, 4, 4.5, 5, 5, 6, 6.5}. Using eq. (2.1), the mean is given as:

$\overline{x}=\frac{2.5+3.5+3.5+4+4+4.5+5+5+6+6.5}{10}=\frac{44.5}{10}=4.45$

Sometimes each value xi in a set may be linked with a weight wi for i = 1, . . . , N.. Here the weights denote the significance or frequency of occurrence according to their values in the relevant order. In this case, we can have a look at the following definition.

Definition 2.5.1.2. Weighted arithmetic mean or weighted average

$\overline{x}=\frac{{\displaystyle \sum {}_{i=1}^N{w}_i{x}_i}}{{\displaystyle \sum {}_{i=1}^N{w}_i}}=\frac{w_1{x}_1+w_2{x}_2+\dots +w_N{x}_N}{w_1+w_2+\dots +w_N}\left(2.2\right)$

The mean is the only useful quantity for the description of a dataset, but it does not always offer the optimal way of measuring the center of the data. A significant problem related to the mean is its sensitivity to outlier values, also known as extreme values. It is possible to see cases in which even small extreme values can corrupt the mean. For instance, the mean income for bank employees may be significantly raised by a manager whose salary is extremely high. Similarly, the mean score of a class in a college course in a test could be pulled down slightly by a few very low scores to counterbalance the effect by a small number of outliers. In order to avoid such problems, sometimes the trimmed mean is used.

Definition 2.5.1.3. Trimmed mean is defined as the mean obtained after one chops off the values that lie at the high and low extremes. For example, it is possible to arrange the values observed for credits and remove the top and bottom 3% prior to calculating the mean. It should be noted that one is supposed to avoid trimming a big portion (such as 30%) at both ends; such a procedure may bring about a loss of valuable information.

For skewed or asymmetric data, median is known to be a better measure of the center of data.

Definition 2.5.1.4. Median is defined as the middle value in a set of ordered data values. It is the value that separates the higher half of a dataset from the lower half. In the fields of statistics and probability, the median often signify numeric data, and, it is possible to extend the concept of ordinal data. Assume that a given dataset of N values for X attribute is arranged in growing order. If N is odd, the median is the value lying at the middle position of the ordered set. If N is even, the median is not exceptional.. It is given by the two values that lie in the middle. If X is a numeric attribute, the median is considered by rule and handled as the mean of the values that lie in the very middle [33–39].

Example 2.5 Median. By using the data of Example 2.4 we can see that the observations are already ordered but even then the median is not unique. This means that the median may be any value that lie in the very middle between values 4 and 4.5. According to this example, it is within the sixth and seventh values in the list. It is a rule to assign the average of the two middle values as the median, which gives us: $\frac{4+4.5}{2}=\frac{8.5}{2}=\mathrm{4.25.}$ From this, the median is 4.25.

In another scenario, let us say that only the first 11 values are in the list. With an odd number of values, the median is the middle most value. In the list, it is the sixth value with a value of 4.

The median is computationally difficult to calculate when there is a large number of observations. For numeric attributes, Boolean value can be calculated easily. Data may be grouped in intervals based on their xi data values and the frequency (number of data values) of each interval may be known. For example, the individuals whose EDSS score is between 2 and 8.5. The interval that contains the median frequency is the median interval. In this case, we can estimate the median of the whole dataset (the median personal credit) by interpolation by using the following formula:

$\text{median}={\text{L}}_{\text{1}}+\left(\frac{N/2-{\left({\displaystyle \sum \text{freq}}\right)}_1}{{\text{freq}}_{\text{medium}}}\right)\text{width}\left(2.3\right)$

where L1 is the lower boundary of the median interval, N is the number of values in the whole dataset, ${\left({\displaystyle \sum \text{freq}}\right)}_1$ is the frequencies’ sum that belongs to the intervals that are lower than the median interval, freqmedian is the median interval’s frequency and width i refers to the width of the median interval.

Median is the value that falls in the middle when the values are ranked from largest to smallest for ungrouped data. Median is the value that splits the array into two.

Equation 2.4(a) and (b) is applied depending on whether the number of elements in an array is even or odd.

$x_{\text{median}}=x_{\left(\frac{N}{2}\right)}+x_{\left(\frac{N+1}{2}\right)}N\text{is}\text{even}\left(2.4\text{a}\right)$

$x_{\text{median}}=x_{\left(\frac{N+1}{2}\right)}N\text{is}\text{odd }\left(2.4\text{b}\right)$

The mode is another measure of central tendency.

Definition 2.4.1.5. The mode is the value that occurs most frequently in the set. For this reason, it can be identified for qualitative and quantitative attributes. The greatest frequency may correspond to a number of different values, which brings about more than one mode. Datasets with one, two and three modes are called unimodal, bimodal and trimodal, respectively. Overall, a dataset that has two or more modes is multimodal. At the other extreme of the spectrum, if each data value is seen once only, we can say that there is no mode at all [7, 33–39].

Example 2.6 Mode. The data from Example 2.5 are bimodal. The two modes are 3.5, 4 and 5.

The following empirical relation is obtained for unimodal numeric data that are skewed moderately or happen to be asymmetrical [40].

$\text{mean}-\text{mode}\approx \text{3}\times \left(\text{mean}-\text{median}\right)\left(2.5\right)$

This suggests that the mode for unimodal frequency curves, moderately skewed, can be estimated easily if we know the mean and median value.

We can also use the midrange for the assessment of the central tendency of a numeric dataset. Midrange is known to be the average of the largest and smallest values in the set. This measurement is convenient and easy kind when the SQL aggregate functions, max() and min(), are used [34].

Example 2.7: Midrange. The midrange of the data provided in Example 2.4 is $\frac{2.5+6.5}{2}=4.5$

In a unimodal frequency curve with perfect symmetric data distribution, the mean, median and mode are all at the same center value (Figure 2.15(a)). Data in most applications in real life are not symmetric. Instead, they may be positively skewed (the mode occurs at a value smaller than the median (see Figure 2.15(b)), or negatively skewed (the mode occurs at a value greater than the median (see Figure 2.15(c)) [35–36].

2.5.2Spread of data

The spread or dispersion of numeric data is an important measure to be assessed in statistics. The most common measures of dispersion are range, quantiles, quartiles, percentiles and the interquartile range. Box plot is also another useful tool to identify outliers. Variance and standard deviation are the other measures that show the spreading of a data distribution.

Let x :x1, x2, . . . , xN be a set of observations for numeric attribute. The range is the difference between the largest (max()) and smallest (min()) values.

$\text{range:max}\left(\right)-\min \left(\right)\left(2.6\right)$

Quantiles are points that are taken at regular intervals of a data spread or distribution. It is found by dividing the interval into equal size sequential sets.

The quartiles indicate a distribution’s center, spread and shape. The first quartile, as denoted by Q1, is the 25th percentile. It cuts off the lowest 25% of the data. The third quartile, as denoted by Q3, is the 75th percentile. It cuts off the highest 25% or the lowest 75% of the data. The second quartile is known to be the 50th percentile (Q2, Median). It is also the median, which yields the center of the data distribution [7, 33–39].

In order to find the middle half of the data, the distance between the first and third quartiles is seen as a simple measure of spread, which presents the range that is covered. Interquartile range (IQR) is the distance defined as follows:

$\text{IQR}\text{=}{Q}_3-Q_1\left(2.7\right)$

Boxplot is a common technique used for the visualization of a distribution (see Figure 2.16). The ends of the box are at the quartiles. The box length is the interquartile range. A line within the box marks the median. Two lines are called whiskers, and these are outside the box extending to the minimum and maximum observations.

Example 2.8: Let us calculate the mean, median, interquartile range and min() and max() values of the numerical array X = [1, 2, − 3, 4, 5, − 6, 2, − 3, 5]. Let us also draw its boxplot.

The mean calculation of the numerical array X = [1, 2, − 3, 4, 5, − 6, 2, − 3, 5] is based on eq. (2.1):

Median calculation is based on eq. (2.4(a))

Range calculation is based on eq. (2.6).

Let us rank the numerical array X = [1, 2, − 3, 4, 5, − 6, 2, − 3, 5] from small to large values:

interquartile range: (IQR) = max() − min()=5− ( − 6) = 11

max() value: 5

min() value: −6

Since boxplot is a plot of the distribution, so you may give a simple figure (see Figure 2.17) to show the boxplot for a simple distribution.

It is possible to make use of boxplots while making comparisons of several sets of compatible data. A method to calculate about boxplots is by making the computation in O(nlogn) time.

Example 2.9 Boxplot. Figure 2.18 presents the boxplots for the EDSS score and lesion counts that belong to the MR images of the patients within a given time interval and are given in Table 2.10. The initial value for EDSS is seen as 0 and for the third quartile the value for this is 5.

The figure presents the boxplot for the same data with the maximum whisker length specified as 5.0 times the interquartile range. Data points beyond the whiskers are represented using +.

2.5.3Measures of data dispersion

These measures in a data distribution are spread out.

Let x1, x2, . . ., xN be the values of a numeric attribute X, and the variance of N observations is given by the following equation:

${\sigma }^2=\frac{1}{N}{\displaystyle \sum {}_{i=1}^N}{\left(x_i-\tilde{x}\right)}^2\left(\frac{1}{N}{\displaystyle \sum {}_{i=1}^N{x}_i^2}\right)-{\overline{\left(x\right)}}^2\left(2.8\right)$

Here, x̄ is the mean value of the observations. The standard deviation σof the observations is the square root of the variance σ2. A low standard deviation is a proof that the data observations are very close to the mean, and, a high standard deviation shows that the data are spread out over a large range of values [40–42].

Example 2.10 By using eq. (2.1) we found in Example 2.4 that the mean is x̄ = 4.45. In order to determine the variance and standard deviation of the data from that example, we set N = 10 and use eq. (2.8).

The basic properties of the standard deviation are as follows:

It measures how the values spread around the mean.

σ = 0 when there is no dispersion, which means that all observations share the same value.

For this reason, the standard deviation is considered to be a sound indicator of the dataset dispersion [43].

2.5.4Graphic displays

The graphic displays of basic statistical descriptions include histograms, scatter plots, quantile plot and quantile–quantile plot.

Histograms are extensively used in all areas where statistics is used. It displays the distribution of a given attribute X. If X is nominal, a vertical bar is drawn for each known value of this attribute X. The height of the bar shows the frequency (how many times it occurs) of the X. The graph is also known as bar chart. Histograms are preferred options when X is numeric. The subranges are defined as buckets or bins, which are separate sections for data distribution for X attribute. The width of the bar corresponds to the range of a bucket. The buckets have equal width.

Scatter plot is another effective graphical method to determine a possible relationship trend between two attributes that are numeric in nature. In order to form a scatter plot, each pair of values is handled as a pair of coordinates, and plotted as points in the plane. Figure 2.19 illustrates a scatter plot for the set of data in Table 2.22.

The scatter plot is a suitable method to see clusters of points and outliers for bivariate data. It is also useful to visualize some correlation.

Definition 2.5.4.1. Two attributes, X, and Y, are said to be correlated (more information in Chapter 3) if the variability of one attribute has some influence on the variability of the other.

Correlations can be positive, negative or null. In this case, the two attributes are said uncorrelated. Figure 2.15 provides some examples of positive and negative correlations. If the pattern of the plotted points is from lower left to upper right, this is an indication that values of X go up when the values of Y go up as well. is from lower left to upper right, this is an indication that values of X go up when the values of Y go up as well. As it is known, this kind of pattern suggests a positive correlation (Figure 2.21 (a)). If the pattern of plotted points is from upper left to lower right, with an increase in values of X the values of Y go down. A negative correlation is seen in Figure 2.21 (b). Figure 2.22 Figure 2.24 indicates some cases in which no correlation relationship between the two attributes exists in the datasets given [42, 43].

Quantile plot: A quantile plot is an effective and straightforward way to know about a univariate data distribution. It displays all the data for the given attribute. It also plots quantile information. Let xi, for i = 1 to N, be the data arranged in an increasing order, so we see x1 as the smallest observation and xN is the largest for numeric attribute X. Each observation, xi, is coupled with a percentage, fi. This shows that on average fi × 100% of the data are lower than the value, xi. The word “on average” is used here since a value with exactly a fraction, fi, of the data below xi may not be found here. As explained earlier, the 0.25 percentile corresponds to quartile Q1, the 0.50 percentile is the median and the 0.75 percentile is Q3.

Let

$f_i=\frac{i-0.5}{N}\left(2.9\right)$

The numbers go up in equal steps of 1/N, varying from 1/2N (as slightly above 0) to 1 − 1/2N (as slightly below 1). xi is plotted against fi on a quantile plot. This enables us to compare different distributions based on their quantiles. For instance, the quantile plots of sales data for two different time periods, their Q1, median, Q3 and other fi values can be compared [44].

Example 2.11 Quantile plot. Figure 2.23 indicates a quantile for the time to import (days) and Time to exports (days) data for Table 2.8.

Quantile–quantile plot, also known as q–q plot, plots the quantiles of one univariate distribution against the corresponding quantiles of another. It is a powerful visualization tool because it enables the user to view if there is a shift going from one distribution to the other.

Let us assume that we have two sets of observations for the attribute or variable interest (income and deduction) that are obtained from two dissimilar branch locations. x1, . . . , xN is the data from the first branch, and y1, . . . , yM is the data from the second one. Each dataset is arranged in an increasing order. If M= N, then this is the case when the number of points in each set is the same. In this case, yi is plotted against xi where yi and xi are both (i − 0.5)=Nquantiles of their datasets, respectively. If MtN (to exemplify, the second branch has fewer observations compared to the first), there can be only M points on the q–q plot. At this point, yi is the (i − 0.5) = M quantile of y [45].

Example 2.12 Table 2.15 is based on economy dataset. The histogram , scatter plot and q–q plot are derived from the time to import (days) and time to exports (days) that belong to Italy dataset (economy data).

Histograms are extensively used. Yet, they might not prove to be as effective as the quantile plot, q–q plot and boxplot methods while one is making comparisons in the groups of univariate observations.

In Table 2.24, Italy dataset (economy data) sets an example of time to import (days) and time to exports (days), normalized with max–min normalization.

Based on the data derived from Italy economy, q–q plot was drawn for the time to import (days) and time to export (days) values as shown in Figure 2.23 for Table 2.24.

Table 2.24: Italy dataset (economy data) presents an example of time to import (days) and time to exports (days).

Figure 2.23 shows q–q plot graph of the attributes pertaining to time to import (days)–time to export (days) attributes as listed in Table 2.8. Based on Figure 2.23, while x = 0 for time to import (days) attribute, y = 0 between the period of 1960–2015. When x = 0, in y = 0.5, an increase has been observed in time to export (days). While time to import (days) attribute is seen within this interval, time to export (days) value has remained constant at 0.