10.2.1 Regularization methods

Let Z_t = [Z_1,t, Z_2,t, …, Z_m,t]^′, t = 1, 2, …, n, be a zero‐mean m‐dimensional time series with n observations. It is well known that the least squares method can be used to fit the VAR(p) model by minimizing

(10.3)

where ‖‖₂ is Euclidean (L²) norm of a vector. In practice, more compactly, with data Z_t = [Z_1,t, Z_2,t, …, Z_m,t]^′, t = 1, 2, …, n, we can present the VAR(p) model in Eq. (10.2) in the matrix form,

(10.4)

where

and

So, minimizing Eq. (10.3) is equivalent to

(10.5)

where ‖‖_F is the Frobenius norm of the matrix.

For a VAR model in high‐dimensional setting, many regularization methods have been developed, which assume sparse structures on coefficient matrices Φ_k and use regularization procedure to estimate parameters. These methods include the Lasso (Least Absolute Shrinkage and Selection Operator) method, the lag‐weighted lasso method, and the hierarchical vector autoregression method, among others.

10.2.1.1 The lasso method

One of the most commonly used regularization methods is the lasso method proposed by Tibshirani (1996) and extended to the vector time series setting by Hsu et al. (2008). Formally, the estimation procedure for the VAR model is through

(10.6)

where the second term is the regularization through L₁ penalty with λ being its control parameter. λ can be determined by cross‐validation. The lasso method does not impose any special assumption on the relationship of lag orders and tends to over select the lag order p of the VAR model. This leads us to the development of some modified methods.

10.2.1.2 The lag‐weighted lasso method

Song and Bickel (2011) proposed a method that incorporates the lag‐weighted lasso (lasso and group lasso structures) approach for the high‐dimensional VAR model. They placed group lasso penalties introduced by Yuan and Lin (2006) on the off‐diagonal terms and lasso penalties on the diagonal terms. More specifically, if we denote Φ(j, − j) as the vector composed of off‐diagonal elements {ϕ_j,i}_i ≠ j, and Φ_k(j, j) as the jthdiagonal element of Φ_k, then the regularization for Φ_k is

(10.7)

where W(−j) = diag(w₁, …, w_j − 1, w_j + 1, …, w_m), an (m − 1) × (m − 1) diagonal matrix with w_j being the positive real‐valued weight associated with the jth variable for 1 ≤ j ≤ m, which is chosen to be the standard deviation of Z_j,t. λ is the control parameter that controls the extent to which other lags are less informative than its own lags. The first term of Eq. (10.7) is the group lasso penalty, the second term is the lasso penalty, and they impose regularization on other lags and its own lags, respectively. Let 0 < α < 1 and (k)^α be the other control parameter for different regularization for different lags; the estimation procedure is based on

(10.8)

10.2.1.3 The hierarchical vector autoregression (HVAR) method

More recently, Nicholson et al. (2016) proposed the HVAR method for high‐dimensional time series. Particularly, they assume various predefined sparse assumptions on the coefficient matrices of the VAR model. Let Φ_k(i) be the ith row of the coefficient matrix Φ_k and Φ_k(i, j) be the (i, j)th element of the coefficient matrix Φ_k. To express their model, we denote

and

Consider the m × m matrix of elementwise coefficient lags L as

(10.9)

where we let L_i,j = 0 if Φ_k(i,j) = 0 for all k = 1, …, p. Thus, each L_i,j denotes the maximal coefficient lag for the (i, j)th component, meaning L_i,j is the smallest k such that Φ_{k + 1 : p}(i, j) = 0.

The method includes three types of sparse structures for coefficient matrices of the VAR model. They are (i) the componentwise structure, which allows each of the m marginal equations from Eq. (10.2) to have its own maximal of lag orders, but requires all components within each equation to share the same maximal lag orders, such that L_i,j = L_i for i = 1, …, m; (ii) the own‐other structure, which assumes a series' own lags are more informative than lags from other series and emphasizes the importance of diagonal elements of the coefficient matrices Φ_k, such that for i ≠ j and for i = 1, …, m, and (iii) the elementwise structure, which places no stipulated relationship.

The parameter estimation is based on a convex optimization algorithm. For the componentwise structure, the parameters are estimated through

(10.10)

where again λ is the control parameter controlling sparsity such that bigger λ means for more i and for smaller k. This means that if then for all k^′ > k. For the own‐other structure, the objective function is

(10.11)

where D is a vector concatenating Φ_k(i, − i) = {Φ_k(i, j) : j ≠ i}_{(m − 1) × 1} and Φ_{(k + 1) : p}(i). The additional second penalty allows coefficient matrices to be sparse such that the influence of component i itself may be nonzero at lag k even though the influence of other components is zero at that lag. This indicates that for all k^′ > k., implies , and implies . Finally, for (iii) the elementwise structure, the objective function is given by

(10.12)

This structure in Eq. (10.12) indicates that each of the components of coefficient matrix can have its own maximum lags. Thus, this is the most flexible structure proposed by Nicholson et al. (2016), which performs well if L_i,j differs for all i and j, but would be suboptimal if L_i,j = L_i. The HVAR Method can be fitted by using R package BigVAR in the CRAN, which is a network of FTP and web servers around the world that store identical, up‐to‐date, versions of code and documentation for R.

10.2.2 The space–time AR (STAR) model

Similar to the regularization methods that control the values of parameters, when modeling time series associated with spaces or locations, it is very likely that many elements of Φ_k are not significantly different from zero for pairs of locations that are spatially far away and uncorrelated given information from other locations. Thus, a model incorporating spatial information is not only helpful for parameter estimation, but also for dimension reduction and forecasting.

For a zero‐mean stationary spatial time series, the space–time autoregressive moving average model is defined by

(10.13)

where the zeros of lie outside the unit circle, a_t is a Gaussian vector white noise process with zero‐mean vector 0, and covariance matrix structure

(10.14)

and ε is an m × m symmetric positive definite matrix. The model becomes a space–time autoregressive model when q = 0. The STAR models were first introduced by Cliff and Ord (1975) and further extended to STARMA models by Pfeifer and Deutsch (1980a, b, c). Since a stationary model can be approximated by an autoregressive model, because of its easier interpretation, the most widely used STARMA models in practice are models,

(10.15)

where Z_t is a zero‐mean stationary spatial time series or a proper differenced and transformed series of a nonstationary spatial time series.

The spatial information is introduced to the model by weighting matrices . Suppose that there are a total of m locations and we let Z_t = [Z_1,t, Z_2,t, …, Z_m,t]^′ be the vector of times series of these m locations. Based on the spatial orders, with respect to the time series at location i, we will assign weights related to this location, such that they are nonzero only when the location j is the ℓth order neighbor of location i, and the sum of these weights is equal to 1. In other words, with respect to location i, we have where

Combining these weights, for all m locations, we have the spatial weight matrix for the neighborhood, which is an m × m matrix with being nonzero if and only if locations i and j are in the same ℓth order neighbor and each row sums to 1. The weight can be chosen to reflect physical properties such as border length or distance of neighboring locations. One can also assign equal weights to all of the locations of the same spatial order. Clearly, W⁽⁰⁾ = I, an identity matrix, because each location is its own zeroth order neighbor.

It should be noted that the space–time autoregressive moving average (STARMA) model is a special case of the VARMA model,

(10.16)

where and For more detailed discussion of models related to both time and space, we refer readers to Chapter 8.

10.2.3 The model‐based cluster method

Clustering or cluster analysis is a methodology that has been used by researchers to group data into homogeneous groups for a long time and may have originated in the fields of anthropology and psychology. There are many methods of clustering, including subjective observation and various distance methods for similarity. Earlier works include Tryon (1939), Cattell (1943), Ward (1963), Macqueen (1967), McLachlan and Basford (1988), among others. We will discuss these further in the last section. These methods were extended to the model‐based cluster approach with an associated probability distribution by researchers including Banfield and Raftery (1993), Fraley and Raftery (2002), Wang and Zhou (2008), Scrucca (2010), and others. More recently, Wang et al. (2013) introduced a robust model‐based clustering method for forecasting high‐dimensional time series, and in this section, we will use their approach as an illustration. Let p_h be the probability a time series belongs to cluster h. The method first groups multiple time series into H mutually exclusive clusters, and assumes that each mean adjusted time series in a given cluster follows the same AR(p) model. Thus, for the ith time series that is in cluster h, we have,

(10.17)

where h = 1, 2, …, H, the ε_i,t are i. i. d. N(0, 1) random variables, independent across time and series. Let be the vector of all parameters in cluster h, and Θ = (θ₁, …, θ_H, η), where η = (p₁, …, p_H). The estimation procedure is accomplished through the Bayesian Markov Chain and Monte Carlo method.

10.2.4 The factor analysis

In previous sections, we have mainly reviewed methods that are based on the VAR model but with different model constraints and estimation procedures. However, there exist many other models for multivariate time series analysis, such as the transfer function model (Box et al. 2015), the state space model (Kalman 1960), and canonical correlation analysis (Box and Tiao 1977). More recently, Stock and Watson (2002a, b) introduced the factor model for dimension reduction and forecasting. The dynamic orthogonal component analysis was proposed by Matteson and Tsay (2011). In this review section, we will concentrate on the factor model.

The factor model is also called the diffusion index approach and can be written as

(10.18)

where F_t = (F_1,t, F_2,t, …, F_k,t)^′ is a (k × 1) vector of factors at time t, L = [ℓ_i,j] is a (m × k) loading matrix, ℓ_i,j is the loading of the ith variable on the jth factor, i = 1, 2, …, m, j = 1, 2, …, k, and ε_t = (ε_1,t, …, ε_m,t)^′ is a (m × 1) vector of noises with E(ε_t) = 0, and Cov(ε_t) = Σ. Let Z_{i,t + ℓ} be ith component of Z_t + ℓ, once values of factors are obtained, we can build a forecast equation for the ℓ − step ahead forecast, such that

(10.19)

where β = (β₁, …, β_k)^′ denotes the coefficient vector and ε_{i,t + ℓ} is a sequence of uncorrelated zero‐mean random variables. Note that the Eq. (10.19) can be further extended to:

(10.20)

where X_i,t is a m × 1 vector of lagged values of Z_{i,t + ℓ} and/or other observed variables. We follow the approach proposed by Bai and Ng (2002) plus the penalty term k[(m + n)/mn] log[mn/(m + n)] to select the number of factors in our simulation studies and empirical examples. Other methods or penalties as described in Bai and Ng (2002) can also be used although this is beyond the scope of this chapter.

10.4.1 Scenario 1

In scenario 1, we assume Φ₁ to be a diagonal matrix with the diagonal elements generated from uniform distribution U(0.2, 0.4). This is a very simple case in which there is no interdependence between each individual time series, and the AR coefficients for each series are similar. Thus, a simple model based on the univariate AR model for each time series would possibly produce fairly reasonable fitting and forecasts.

Table 10.1 displays the MSFEs and corresponding standard deviations of two‐region aggregation. The smallest MSFE in each category are in boldface to facilitate presentation. It appears that the VAR method has much larger MSFE compared to all other methods when n = 100. This is due to large parameter estimation errors when n is relatively small. As the series length n increases to 500, the MSFE of the VAR method approaches other methods. Although all methods except the VAR method perform similarly in terms of MSFE, the proposed method produces the smallest MSFE in most cases. Further, it seems that all regularization methods produce similar MSFEs. Table 10.2 presents the MSFEs and their standard deviations of total aggregation. The results in Table 10.2 are nearly consistent with results in Table 10.1 and the proposed method is still among one of the best methods.

10.4.2 Scenario 2

In scenario 2, the coefficient matrix Φ₁ is generated from a “band” matrix pattern shown in Figure 10.1 where purple points correspond to nonzero entries and white areas correspond to zero entries. The nonzero diagonal entries of Φ₁ are fixed to be 0.3 and the nonzero off‐diagonal elements are fixed at 0.1. This coefficient structure indicates that each time series depends largely on its own past, and weakly depends on other series that are close. Tables 10.3 and 10.4 present the MSFEs of two different aggregation schemes. Again, the MSFEs and standard deviations of VAR method are much larger than all other methods when series length n = 100. For n = 100, the proposed method outperforms all other methods. For n = 500, the proposed method outperforms all other methods when ℓ = 1 and 2, and the factor model performs best when ℓ = 3. Among all regularization methods, the HVAR‐OO produces relative smaller MSFEs when n = 100. This is because HVAR‐OO assumes the diagonal elements to be more informative that are close to the true coefficient matrix structure. HVAR‐E has smaller MSFEs than other regularization methods when n = 500. This is due to its flexible structure assumption.

10.4.3 Scenario 3

In the scenario 3, the coefficient matrix Φ₁ is generated from a “cluster” matrix pattern. We set the diagonal elements of Φ₁ to be 0.3. Then, we randomly select 2m elements from off‐diagonal elements of Φ₁ and assign each of them with the value 0.1 (see Figure 10.2). Tables 10.5 and 10.6 show the MSFEs and the corresponding standard deviations. In this more complicated simulation setting, the AR model and the model‐based cluster method have very large MSFEs for both of the sample sizes we consider. This is because they largely ignore the interdependences between each time series. For both n = 100 and 500, the proposed method outperforms other methods in terms of MSFEs.

10.5.1 The macroeconomic time series

We compare MSFEs of different methods and assess the effectiveness of the proposed method through the collection of time series of U.S. macroeconomic indicators. The data is collected from Stock and Watson (2009) and Koop (2013). The full data list contains 168 quarterly macroeconomic variables from Quarter 1 of 1959 to Quarter 4 of 2007, representing information about many aspects of the US economy. We retrieve 61 time series from the full dataset. Readers who are interested in this data can view their papers for further details. Variables which are originally at a monthly frequency are transformed to quarterly by taking average of three months in a quarter. Seasonally adjustments are made if necessary, leading to a dataset with m = 61 and n = 196. Time series are transformed to stationary using the suggestion of Stock and Watson (2009) with n = 195 remaining, and listed as WW10 in the Data Appendix. Table 10.7 contains brief descriptions of each variable, and the aggregation group they belong to, along with a transformation code, where 1 = first differencing of log of variables, and 2 = second differencing of log of variables.

Those 61 time series can be aggregated into three main macroeconomic measures by their nature: gross domestic product (GDP), industrial production index (IPS), and constant elasticity of substitution (CES).

The main interest of this section is on accurately forecasting three aggregate variables: GDP, IPS, and CES, since they are important measurements of the US economy. In this application, we used data from Quarter 1 of 1959 to Quarter 3 of 1992 for model fitting, and then compute the rolling out‐of‐sample one‐step‐ahead forecasts, starting from Quarter 4 of 1992 to Quarter 4 of 2007. The MSFEs of the univariate AR, VAR, lasso, lag‐weighted lasso, HVAR‐C, HVAR‐OO, HVAR‐E, the factor model, the model‐based cluster, and the proposed method are compared in this application and the results given in Table 10.8.

The three regularization methods, including HVAR‐C, HVAR‐OO, and HVAR‐E, perform the best as their MSFEs are below 0.7. The proposed method performs close to those three methods and has smaller MSFEs than all other methods. The benchmark univariate AR method outperforms the VAR, the factor model, and the model‐based cluster, but does not perform as well as the proposed aggregation method.

10.5.2 The yearly U.S. STD data

In this section, we provide an illustration using a spatial time series introduced in Chapter 8. Recall that the data set contains yearly sexually transmitted disease STD morbidity rates reported to the National Center for HIV/AIDS, viral Hepatitis, STD, and TB Prevention (NCHHSTP), Center for HIV, and Centers for Disease Control and Prevention (CDC) from 1984 to 2014. The dataset was retrieved from the CDC's website and includes 50 states plus DC. The rates per 100,000 persons are calculated as the incidence of STD reports, divided by the population, and multiplied by 100,000.

For the analysis, we standardized each time series and removed data from the following states, Montana, North Dakota, South Dakota, Vermont, Wyoming, Alaska, and Hawaii, due to missing data. Hence, the dimension of data is m = 44 and n = 29. The data is listed as WW8c in the Data Appendix and shown in Figure 10.3. In modeling STD data, researchers are interested in forecasting aggregate data based on nine Morbidity and Mortality Weekly Report (MMWR) regions or four STD regions as shown in Figure 10.4.

We used the first 24 observations for model fitting, and the rest of the observations for evaluating the forecasting performance. The MSFEs averaged across the lags are reported. Methods considered include: the univariate AR, VAR, lasso, lag‐weighted lasso, HVAR‐C, HVAR‐OO, HVAR‐E, the factor model, the model‐based cluster, and the proposed method. In addition, we also add the STAR model for comparison in this application, as it is one of most naturally considered models for spatial time series analysis, which can also be reviewed as a dimension reduction method.

The MSFEs when forecasting the STD at nine MMWR regions are presented in Table 10.9. The VAR fails to estimate the parameters. STAR(2_{2, 1}) performs the best in this case, followed by the model‐based cluster, the factor model, the proposed method, and the univariate AR. All regularization methods have much larger MSFEs.

Table 10.10 displays the MSFEs when forecasting STDs in four STD regions. Again, the VAR fails to estimate the parameters due to the number of parameters to estimate is bigger than the number of observations. The proposed method performs the best among all methods. The STAR(2_{2, 1}) model has second smallest MSFEs. The factor model and the model‐based cluster also perform reasonably well. The MSFE for the univariate AR is in the middle range. Again, all regularization methods do not perform well.

We have analyzed many procedures in this section and summarized their estimation results in Tables 10.8–10.10. In supporting the results in these tables and for readers' reference, the outputs of the analysis and estimation from various procedures are provided in this chapter's appendix.

10.6.1 More on clustering

Clustering is a method of grouping a set of elements or a complicated data set into some homogeneous groups. Other than subjective observation, a very nature objective method of clustering is based on a distance and similarity measure. In time domain, given two r‐dimensional observations, X = (x₁, x₂, …, x_r) and Y = (y₁, y₂, …, y_r). Some commonly used distance and similarity measures include the following.

1. Euclidian distance:
   
2. Minkowski distance:
   
3. Canberra distance:
   
4. Kullback–Leibler (KL) distance:

Statistically, we see that a good distance and similarity measure is a distribution‐based or a model‐based approach briefly introduced in Section 10.2.3, where an associated probability distribution is used in clustering. In terms of frequency domain spectral density distributions, given two spectra, f_i(ω_k) and f_j(ω_k), 0 < ω_k < 1/2, the most commonly used similarity measure of spectrum is the KL distance, defined as

(10.26)

and its symmetric form as the quasi‐distance D(f_i, f_j) = KL(f_i, f_j) + KL(f_j, f_i). It turns out that

(10.27)

which is the distance we will use in the frequency domain.

Based on a distance measure, we have hierarchical and nonhierarchical clustering methods.

Given a set of N elements, the hierarchical clustering method includes the following steps:

• Step 1. Begin with N clusters by assuming each of the elements is an individual cluster. The distances between clusters are the same as the distances between elements.
• Step 2. Combine the two elements that are most similar with minimum distance and form a new cluster. This leads to one less cluster.
• Step 3. Compute distances between the newly formed cluster and other remaining clusters.
• Step 4. Repeat steps 2 and 3 a total (N − 1) times so that all become a single cluster after the algorithm terminates, with the results shown in a dendrogram.

It should be noted that step 3 can be done in different ways or algorithms. The first way is “single linkage” in which the minimum distance between each element of newly formed cluster and other clusters is computed. The second way is “complete linkage” in which the maximum distance between each element of newly formed cluster and other clusters is computed. The third way is “average linkage” in which the average distance between each element of newly formed cluster and other clusters is computed. The fourth way is “median linkage” in which the median distance between each element is used as introduced by Tibshirani et al. (2001).

In terms of nonhierarchical methods, the most popular approach is the K‐means method, which assigns each item to the cluster having the nearest centroid (mean) or assigned center point.

In terms of deciding an optimal number of clusters, we will use the total within sum of squares (TWSS), which measures total intra‐cluster variation and we would like it to be as small as possible. We will illustrate its use through an empirical example in the next section.

10.6.2 Forming aggregate data through both time domain and frequency domain clustering

In modeling the U.S. yearly STD data, we created aggregate data based on MMWR regions and four STD regions. We can also form the aggregate groups using clustering as illustrated in the following.

10.6.2.2 Example of frequency domain clustering

Next, we will show that the clustering of the STD data can also be done through frequency domain spectral matrices. Specifically, we will use the kernel smoothing method discussed in Chapter 9 to obtain the estimates of spectral matrices. We choose the Daniell smoothing window with bandwidth m = 5. The resulting individual spectra for 44 elements are presented in Figure 10.5.

10.6.2.2.1 Clustering using similarity measures

Based on the KL distance and its quasi‐distance form, D(f_i, f_j), we compute TWSS, which measures total intra‐cluster variation. Figure 10.6 is the plot for TWSS across different numbers of clusters from the STD data set. The location of a bend (knee) in the plot is generally considered to be an indicator of the appropriate number of clusters. It shows that five or six clusters seem to be a good choice.

10.6.2.2.2 Clustering by subjective observation

The spectrum matrices in Figure 10.5 clearly indicate a group or cluster pattern. Intuitively, the spectrum of each state can be roughly categorized into the following clusters, which are highly consistent with the MMWR regions. Based on the similarity of spectrums, we can group them in the following six clusters given in Table 10.12.

Cluster	Number of states	States
1	3	ME, NH, RI These states are in the North East region.
2	4	CT, NJ, NY, PA These states are in the Middle Atlantic region.
3	12	MA, DE, DC, MD, VA, AL, FL, GA, MS, SC, NC, TN Most of these states are in the South Atlantic and East South Central regions.
4	14	IL, IN, KY, MI, MN, OH, WV, WI, KS, MO, IA, TX, LA, AR These states are in the East North Central, Western North Central, and West South Central regions.
5	8	OK, NE, AZ, NM, NV, ID, UT, CO These states are in the West South Central and Mountain regions.
6	3	CA, OR, WA These states are in the Pacific region.

10.6.2.2.3 Hierarchical clustering

The dendrograms of the four algorithms for the STD data are shown in Figures 10.7–10.10.

10.6.2.2.4 Nonhierarchical clustering using the K‐means method

The Partitioning Around Medoids (PAM) is a robust version of the K‐means clustering algorithm. It begins with a selection K objects as medoids, and then we minimize the sum of the dissimilarities of the observations to their closest cluster. We apply PAM to the spectrums of STD data set with K = 4, and obtain the following result in Table 10.13 and its cluster plot in Figure 10.11.

Cluster	Number of states	States
1	17	CT, NY, DC, PA, AL, FL, GA, MS, TN, IL, MI, OH, AR, LA, MO, CA, WA
2	9	ME, NH, NM, OK, NE, CO, UT, AZ, NV
3	13	MA, RI, DE, VA, WV, KY, NC, SC, IN, MN, WI, TX, IA
4	5	NJ, MD, KS, ID, OR

It turns out that the clustering patterns found by the model‐based clustering and the PAM method are very similar. Specifically, the cluster 1 of the model‐based clustering method is almost identical to the cluster 1 of PAM; the cluster 2 of the model‐based clustering method corresponds to the clusters 3 and 4 of PAM method; and cluster 3 of the model‐based clustering method is close to the cluster 2 of the PAM method.

For more discussion and examples of clustering, we refer readers to Tibshirani et al. (2001), Johnson and Wichern (2007), and Izenman (2008), among others.

10.6.3 The specification of aggregate matrix and its associated aggregate dimension

Big data and high‐dimensional problems are everywhere in the age of fast computers and the internet. To address this, we propose aggregation as a dimension reduction method. It is very natural and simple to use, and its performance in forecasting is supported by both simulation and empirical examples as a good method for dimension reduction.

The aggregation matrix A and its associated s can occur in many different specifications. Even based on practical considerations, we can specify different forms of A and s. By choosing s = 1, A becomes a row vector, and any m‐dimensional multivariate time series Z_t will aggregate to become a univariate time series Y_t. Most of the time, the result of aggregation is meaningful. For example, sales data can be specified in terms of regions or kinds (categories). With regard to housing sales of the 3144 US counties, this data can be aggregated into housing sales for each of the 50 states, into the housing sales of the four regions (East, West, North, and South), and further into the total housing sales of the entire country. We can also specify A and s based on data‐driven considerations, which we will leave to our readers to try.

10.6.4 Be aware of other forms of aggregation

It is important to note that aggregation as a tool for high dimension reduction that we have suggested is contemporal aggregation. Other forms of aggregation include temporal aggregation for a flow variable such as industrial production and a systematic sampling for a stock variable such as the price of a given commodity. However, these techniques are not useful for high dimension reduction, in fact, they will aggravate the problem, since temporal aggregation and systematic sampling lead to very serious information loss as shown in Wei (2006, chapter 20). In fact, limitations of temporal aggregation and systematic sampling have been studied by many researchers including Amemiya and Wu (1972), Brewer (1973), Tiao and Wei (1976), Weiss (1984), Stram and Wei (1986), Lütkepohl (1987), Teles and Wei (2000, 2002), Breitung and Swanson (2002), Teles et al. (1999, 2008), Lee and Wei (2017), and many more. An interesting related topic is how to recover the information loss of temporal aggregation and systematic sampling. One can use either temporal disaggregation or bootstrap, which we will not discuss in this book and refer readers to Wei and Stram (1990), Meyer and Kreiss (2015), Kim and Wei (2018), and Tewes (2018), among others.

Before closing this chapter, we want to point out that high dimension is an important issue in multivariate time series analysis. We introduce a simple and useful method to reduce dimension. For more discussion and applications on high‐dimensional problems, we refer readers to Chen and Qin (2010), Huang et al. (2010), Flamm et al. (2013), Cho and Fryzlewicz (2015), Giraud (2015), Hung et al. (2016), Lee et al. (2016), Gao et al. (2017), Huang and Chiang (2017), and Wood et al. (2017), among others.

The details of parameter estimation results for the two empirical examples in Section 10.5 are given next.

10.A.1 Further details of the macroeconomic time series

Since we used out‐of‐sample, the estimation results shown in this section is based on in‐sample data from quarter 1 of 1959 to Quarter 4 of 2007.

10.A.1.1 VAR(1)

Based on the AIC criteria, the VAR(1) model was fitted to the data. The least square estimation was used and the estimated coefficient matrix is

It appears that the matrix is not sparse and is noisy. We will compare the estimated coefficient matrix with estimates from other regularization methods in following sections.

10.A.1.2 Lasso

The maximum allowed lag order is set to be 4. The estimates of coefficient matrices are

The estimates of those matrices are much sparser compared to the estimates from the VAR, using least squares.

10.A.1.3 Componentwise

The maximum allowed lag order is set to be 4. The estimates of coefficient matrices are

10.A.1.4 Own‐other

The maximum allowed lag order is set to be 4. The estimates of coefficient matrices are

10.A.1.5 Elementwise

The maximum allowed lag order is set to be 4. The estimates of coefficient matrices are

10.A.1.6 The factor model

In this section, we show the equation of forecasting the first component Z_1,t, which is GDP252. Based on Bai and Ng (2002), two factors are selected. The first factors F_1,t and the second factor F_2,t are shown in Figure 10.A.1.

Suppose we want to obtain ℓ − step ahead forecast for the first component, Z_1,t, of those time series by using factors F_t. The equation is:

where X_1,t is the first lag of Z_1,t. The estimated parameters are presented in Table 10.A.1.

Estimate
	0.0005	−0.0001	0.7599

10.A.1.7 The model‐based cluster

The maximum number of clusters allowed is set to be 4 and the AR(1) model is fitted for each cluster. The individual time series is assigned to cluster i if its probability of being in cluster i is the highest. Table 10.A.2 presents probabilities of each time series being in specific clusters (first four columns) and the finally assigned cluster for each time series (the last column). Table 10.A.2 shows that no time series is within cluster 4 based on the estimated probability. Moreover, most of “GDPXXX” time series are assigned to cluster 1, most of “CESXXX” time series are in cluster 2, and most of “IPSXXX” time series are in cluster 2 and 3. The models of cluster 1, 2, and 3 can be written as

where the estimated parameters are shown in Table 10.A.3.

	Cluster 1	Cluster 2	Cluster 3	Cluster 4	Cluster Assigned
GDP252	0.505	0.441	0.053	0.002	1
GDP253	0.001	0.000	0.704	0.295	3
GDP254	0.499	0.501	0.000	0.000	2
GDP255	0.559	0.098	0.328	0.015	1
GDP256	0.016	0.000	0.626	0.357	3
GDP257	0.540	0.093	0.352	0.015	1
GDP258	0.554	0.097	0.342	0.006	1
GDP259	0.214	0.042	0.677	0.067	3
GDP260	0.162	0.039	0.727	0.072	3
GDP261	0.629	0.208	0.161	0.002	1
GDP266	0.095	0.000	0.653	0.251	3
GDP267	0.558	0.299	0.143	0.000	1
GDP268	0.500	0.300	0.186	0.015	1
GDP269	0.556	0.302	0.142	0.000	1
GDP271	0.690	0.310	0.000	0.000	1
GDP272	0.502	0.297	0.186	0.015	1
GDP274	0.559	0.098	0.328	0.015	1
GDP275	0.559	0.098	0.328	0.015	1
GDP276	0.559	0.098	0.328	0.015	1
GDP277	0.557	0.098	0.328	0.017	1
GDP278	0.558	0.299	0.143	0.000	1
GDP279	0.502	0.298	0.186	0.015	1
GDP280	0.557	0.219	0.215	0.009	1
GDP281	0.511	0.349	0.128	0.013	1
GDP282	0.484	0.038	0.373	0.104	1
GDP284	0.559	0.098	0.328	0.015	1
GDP285	0.136	0.000	0.634	0.230	3
GDP286	0.745	0.110	0.143	0.002	1
GDP287	0.838	0.049	0.036	0.077	1
GDP288	0.860	0.135	0.002	0.002	1
GDP289	0.860	0.135	0.002	0.002	1
GDP290	0.860	0.135	0.002	0.002	1
GDP291	0.499	0.501	0.000	0.000	2
GDP292	0.499	0.501	0.000	0.000	2
IPS11	0.499	0.501	0.000	0.000	2
IPS299	0.499	0.501	0.000	0.000	2
IPS12	0.860	0.135	0.002	0.002	1
IPS13	0.042	0.000	0.684	0.273	3
IPS18	0.860	0.135	0.002	0.002	1
IPS25	0.499	0.501	0.000	0.000	1
IPS32	0.061	0.000	0.694	0.245	3
IPS34	0.068	0.002	0.708	0.222	3
IPS38	0.499	0.501	0.000	0.000	2
IPS43	0.499	0.501	0.000	0.000	2
IPS307	0.000	0.000	0.680	0.320	3
IPS306	0.010	0.000	0.730	0.260	3
CES275	0.499	0.501	0.000	0.000	2
CES277	0.499	0.501	0.000	0.000	2
CES278	0.499	0.501	0.000	0.000	2
CES003	0.499	0.501	0.000	0.000	2
CES006	0.196	0.016	0.627	0.161	3
CES011	0.499	0.501	0.000	0.000	2
CES015	0.499	0.501	0.000	0.000	2
CES017	0.499	0.501	0.000	0.000	2
CES033	0.499	0.501	0.000	0.000	2
CES046	0.499	0.501	0.000	0.000	2
CES048	0.499	0.501	0.000	0.000	2
CES049	0.499	0.501	0.000	0.000	2
CES053	0.499	0.501	0.000	0.000	2
CES088	0.499	0.501	0.000	0.000	2
CES140	0.499	0.501	0.000	0.000	2

Estimates
	0.398	0.033	0.541	0.032	0.155	0.039

10.A.1.8 The proposed method

The 61 time series are aggregated into trivariate time series, GDP, IPS, and CES. We fit a VAR(p) model,

The best fitted model is p = 2 based on AIC. Estimates of coefficient matrices along with their standard errors (in parenthesis) are shown next:

10.A.2 Further details of the STD time series

10.A.2.1 VAR

Since the number of dimension m = 44 is much greater than the number of observations n = 29, the ordinary least square estimation without any restrictions failed for this model.

10.A.2.2 Lasso

10.A.2.3 Componentwise

10.A.2.4 Own‐other

The maximum allowed lag order is set to be 4.

10.A.2.5 Elementwise

The maximum allowed lag order is set to be 4.

10.A.2.6 The STAR model

Since the STD data are collected from different states, which are spatially related, the space–time modeling can be used. Based on AIC criteria, the STAR(2_{2, 1}) model is used and can be written as

The estimates are presented in Table 10.A.4.

Estimates
	0.074	0.286	0.239	−0.126	0.065

Moreover, the weighting matrices, W⁽⁰⁾, W⁽¹⁾, and W⁽²⁾, are specified by using spatial relationship between states and are shown next:

10.A.2.7 The factor model

In this section, we show the equation for forecasting the first component Z_1,t, which is the STD rate in Connecticut. Six factors are selected, and the estimated factor scores are shown in Figure 10.A.2.

To compute the ℓ − step ahead forecast for the first component, Z_{1,t + ℓ}, we will use the following model equation, which is:

where X_1,t is the first lag of Z_1,t. The estimated parameters are presented in Table 10.A.5.

Estimate
	0.054	0.232	−0.056	−0.100	−0.208	−0.203	0.234

10.A.2.8 The model‐based cluster

The maximum number of clusters allowed is set to be 4 and the AR(1) model is fitted for each cluster. Each time series is assigned to cluster i if its probability of being in cluster i is the highest. Probabilities of each time series being in specific clusters (first four columns) and the finally assigned cluster for each time series (the last column) are presented in Table 10.A.6. It shows that no time series was assigned to cluster 1. The models of cluster 1, 2, 3, and 4 can be written as

where the estimated parameters are shown in Table 10.A.7.

	Cluster 1	Cluster 2	Cluster 3	Cluster 4	Assigned Cluster
CT	0.02	0.62	0.00	0.36	2
ME	0.05	0.00	0.94	0.01	3
MA	0.23	0.12	0.14	0.51	4
NH	0.10	0.00	0.85	0.05	3
RI	0.13	0.00	0.83	0.04	3
NJ	0.10	0.36	0.01	0.53	4
NY	0.02	0.66	0.00	0.32	2
DE	0.21	0.16	0.08	0.55	4
DC	0.02	0.61	0.00	0.36	2
MD	0.18	0.22	0.03	0.57	4
PA	0.02	0.63	0.00	0.35	2
VA	0.23	0.12	0.13	0.52	4
WV	0.04	0.00	0.96	0.00	3
AL	0.07	0.42	0.00	0.50	4
FL	0.01	0.70	0.00	0.29	2
GA	0.01	0.68	0.00	0.31	2
KY	0.23	0.03	0.48	0.26	3
MS	0.03	0.55	0.00	0.42	2
NC	0.07	0.43	0.00	0.49	4
SC	0.17	0.23	0.03	0.57	4
TN	0.11	0.34	0.01	0.54	4
IL	0.03	0.56	0.00	0.40	2
IN	0.19	0.19	0.06	0.56	4
MI	0.03	0.59	0.00	0.38	2
MN	0.13	0.22	0.34	0.31	3
OH	0.01	0.76	0.00	0.23	2
WI	0.07	0.43	0.00	0.50	4
AR	0.03	0.56	0.00	0.41	2
LA	0.01	0.74	0.00	0.26	2
NM	0.04	0.00	0.96	0.00	3
OK	0.07	0.00	0.92	0.01	3
TX	0.12	0.31	0.01	0.55	4
IA	0.17	0.22	0.14	0.47	4
KS	0.18	0.20	0.06	0.56	4
MO	0.02	0.64	0.00	0.34	2
NE	0.06	0.00	0.92	0.02	3
CO	0.16	0.02	0.68	0.15	3
UT	0.12	0.01	0.78	0.09	3
AZ	0.07	0.00	0.92	0.01	3
CA	0.05	0.49	0.00	0.45	2
NV	0.04	0.00	0.95	0.00	3
ID	0.24	0.09	0.22	0.45	4
OR	0.17	0.23	0.04	0.56	4
WA	0.02	0.68	0.00	0.31	2

Estimates
	0.121	1.032	0.518	0.929	−0.246	0.989	0.349	0.999

10.A.2.9 The proposed method

The 44 time series are aggregated into MMWR region (9 time series) and fit a VAR(p) model, such that

The best fitted model is p = 1 based AIC. The estimate of the coefficient matrix and the associated standard error (in parenthesis) are shown next.

If we aggregate the 44 time series into STD region (4 time series) and fit a VAR(p) model, the best fitted model is p = 4 with the following estimates of the associated coefficient matrices.

Projects

1. Find a high‐dimensional time series data set in a social science field, perform its principal component and factor analyses, and complete your analysis with a written report and attached software code.
2. Use the aggregation method to build an aggregate model for the data set from Project 1 and complete your analysis with a written report and attached software code.
3. Perform your analyses in Projects 1 and 2 with (n − 5) observations, then compare the results in terms of their next five‐step forecasts.
4. Find a high‐dimensional time series data set in a natural science‐related field, and perform its analysis with a written report and associated software code.
5. Find a real high‐dimensional vector time series data set of your interest, use various dimension reduction methods to analyze your data set, make comparisons, and complete your analysis with a written report and associated software code.