MUSCAT: Multi-Scale Spatio-Temporal Learning with Application to Climate Modeling

1 Introduction

The spatio-temporal data obtained from climate and environmental sciences are often available at multiple spatial resolutions. For example, Table ?? shows examples of climate data generated by different global and regional climate models, which can have varying spatial scales, from tens to several hundred kilometers. Such multi-scale data provide useful information that can aid scientists in understanding the variability of the climate system in order to predict its future behavior. However, since the data at different scales are potentially correlated with each other, concatenating them together into a single feature vector may not be an effective strategy for building robust prediction models. Utilizing only the data from the finest resolution is also not a viable approach as the data at coarser resolutions may capture broad-scale effects that cannot be easily discerned from the finest resolution data. Finding an effective way to integrate the multi-scale data into a prediction modeling framework is thus a challenge that needs to be addressed [Miller et al., 2015].

For applications such as climate modeling, in addition to the multi-scale nature of the data, the predictions must be made at multiple locations. Although a model can be trained to fit the training data at each location independently, the output predictions may not be spatially coherent as the models may not preserve the spatial autocorrelation of the data. Furthermore, the amount of training data may vary from one location to another, making it difficult to obtain accurate models for locations with limited training data. Recent works have demonstrated the advantages of applying multi-task learning to such multi-location prediction problems [Xu et al., 2016a][Xu et al., 2016b][Gonçalves et al., 2016][Gonçalves et al., 2017][Yu et al., 2015]. These approaches consider the spatial relationship between different locations to jointly train the models. However, none of them are designed to handle multi-scale data. Furthermore, these approaches are mostly developed for batch learning algorithms, making it harder to scale them up to larger spatio-temporal datasets.

Finally, model interpretability is also important for many spatio-temporal applications. For example, climate scientists are interested to understand the major driving factors that influence the climate variability at various locations. Some of the factors, such as the El Niño phenomenon, are well-known to the scientists, but there may be other broad-scale patterns governing the variability of the data. Thus, it would be useful to develop a framework that can shed light on these patterns in addition to generating accurate predictions.

To address these challenges, this paper presents a novel framework called MUSCAT (MUlti-SCAle Spatio-Temporal Learning) for the predictive modeling of multi-scale spatiotemporal data. MUSCAT represents the spatio-temporal data at each scale as a 3-dimensional tensor. The tensors are then jointly decomposed into a set of shared latent factors, representing the various patterns that can help summarize the variability observed in the spatio-temporal data. MUSCAT employs an ensemble of spatial and temporal prediction models to make its predictions, where the spatial models are trained to fit the climate response variable against the shared spatial latent factors of the tensors whereas the temporal models are trained to fit the respective temporal latent factors. More importantly, the multi-scale tensor decomposition and the fitting of spatial and temporal prediction models are performed simultaneously in a unified learning framework. As this can be computationally expensive due to the massive size of the spatio-temporal data, an incremental learning algorithm is proposed. The algorithm enables the latent factors and model parameters to be iteratively learned over space and time, thereby avoiding the need to rebuild the models from scratch each time there is new data available.

In short, the main contributions of this work are:

1. A novel framework called MUSCAT is proposed for the predictive modeling of multi-scale spatio-temporal data. The framework allows the latent spatial and temporal patterns shared by the multiple scales to be simultaneously derived using a multi-tensor decomposition approach. MUSCAT also trains an ensemble of spatial and temporal prediction models, whose outputs will be aggregated when making predictions for a new location or for a future time period.
2. To improve its scalability, an incremental learning algorithm over space and time is proposed.
3. Experiments performed on real-world data from the United States Historical Climate Network (USHCN) demonstrates the superiority of MUSCAT compared to other competing algorithms.

2 RelatedWorks

This section reviews some of the previous works related to this research. In recent years, multi-task learning has been proven to be effective at learning models for predicting multiple tasks jointly by taking into account the relationships among the tasks [Caruana, 1997]. The success of multi-task learning for spatio-temporal data has also been been demonstrated in [Xu et al., 2017][Xu et al., 2016a][Xu et al., 2016b][Zhao et al., 2015][Yu et al., 2015]. However, none of these approaches are designed for multi-scale data.

The term multi-scale learning has been used rather loosely in the literature to describe different classes of methods. For example, in traditional machine learning, it has referred to techniques based on multiple kernels [Bellocchio et al., 2012], multi-covariance matrices [Walder et al., 2008] or multi-basis functions [Nounou and Nounou, 2010], none of which are designed to handle multi-scale data. Instead, they were developed to extract multi-scale features from the given data. A closer related area is in deep learning, where multiscale modeling approaches have been developed for computer vision applications. Here, the multi-scale data refer to different resolutions of an image [Bertasius et al., 2015; Zhao and Du, 2016]. Other multi-scale learning approaches, such as those proposed in [Neverova et al., 2014] and [Eigen et al., 2014] do not consider the relationships between data at different scales.

Tensor decomposition has been widely used to explore the latent features of multi-dimensional data [Kolda and Bader, 2009]. This includes previous works on coupled tensor decomposition [Acar et al., 2011; Ermiş et al., 2015] for the joint analysis of data from multiple sources. However, such methods are mostly designed for batch learning, and thus, cannot efficiently handle the dynamic growth of data in different dimensions (e.g., spatial and temporal). As an alternative, incremental or online tensor decomposition approaches have been developed in recent years [Sun et al., 2008; Zhou et al., 2016]. These methods are unsupervised, and thus, are not as effective compared to the MUSCAT framework proposed in this paper. More recently, Yu, et al. [Yu et al., 2015] also presented an online supervised tensor decomposition method. Unlike MUSCAT, the method employed a tensor to represent the model parameters instead of the data. As will be shown in our experiments, the strategy used by MUSCAT to represent data as tensors instead of model parameters as tensors is more effective for modeling the multi-scale spatio-temporal data investigated in this study.

3 ProposedMUSCAT Framework

MUSCAT is a supervised learning framework that jointly performs a coupled tensor decomposition on a multi-scale spatiotemporal dataset and fits the derived latent factors to the response variable of interest at multiple locations. This section presents the details of our proposed framework.

3.1 Preliminaries

Let D = (X (1) , ...,X (L) ,Y) be a multi-scale spatiotemporal data set, where X (l) ∈ ℜS×T×dl denote the spatiotemporal tensor of predictor variables at the l-th scale, Y ∈ ℜS×T denote the time series of the response variable for all locations, S is the number of locations, T is the length of time series, and dl is the number of predictor variables associated with the l-th scale.

Inspired by previous work on discovering climate indices using SVD [Steinbach et al., 2003], we employ tensor decomposition to extract broad-scale patterns from the spatio-temporal data. Specifically, the following CANDECOMP/PARAFAC (CP) decomposition [Kolda and Bader, 2009] is used to decompose a 3rd-order tensor X into its corresponding latent factors, A, B, and C:

where ak, bk and ck are column vectors corresponding to the k-th columns of matrices A, B, and C, and ◦ denotes the outer product operation.

3.2 Supervised Tensor Decomposition for Multi-Scale Data

For multi-scale spatio-temporal data, the tensor decomposition can be performed at each scale l as follows:

where A ∈ ℜS×K denote the spatial latent factors and B ∈ ℜT×K denote the temporal latent factors. Our framework assumes that the spatial and temporal latent factors A and B are invariant across all scales. The normalization factor vol(l) = 2(S×T ×dl) ensures that the decomposition at each scale does not depend on its tensor size. Finally, Ωd(A, B, C) is the regularization term used to enforce the sparsity of the latent factors:

Although the latent factors derived via multi-tensor decomposition may capture the spatial and temporal variabilities of the data, they may not be well-suited for predictive modeling applications. To overcome this problem, MUSCAT incorporates the multi-tensor decomposition process directly into a supervised learning framework. Specifically, it assumes that the prediction for location s at time t is given by the weighted average of the predictions generated from models trained on data at different scales, i.e., ˆ ys,t = ∑Ll αlˆ y(l) s,t , where αl is the weight associated with the prediction function of the l-th scale. The weights are assumed to form a probability simplex that satisfies αl ≥ 0 and ∑Ll αl = 1.

Furthermore, the prediction model at a given scale l is given by an ensemble of spatial and temporal models:

where x(l) s,t denotes the feature vector for location s and time t at scale l, w(l) k and v(l) k are parameters of the spatial and temporal prediction models for the k-th latent factor, while As,k and Bt,k are the scalar coefficients for the k-th spatial and temporal latent factors associated with location s and time t, respectively. Note that we enforce the constraint that the spatial and temporal latent factors are invariant across all scales, i.e., ∀l : A(l) = A, B(l) = B.

Our framework to simultaneously uncover the latent factors and derive the parameters for the spatial and temporal prediction models can be formalized as follows:

where W, V and C denote the set of W(l) , V(l) and C(l) for l = 1, ..., L respectively for notation simplicity, and ˆY(l) corresponds to the predictions generated by the spatio-temporal models at scale l, as shown in Equation (2). We also use ∥[{W(l) ,V(l) ,C(l) },A, B]∥1 to denote the ℓ1 norm regularization term for W(l) , V(l) , A, B and C(l) , respectively.

3.3 Incremental Learning

Optimizing Eq. (3) can be expensive for large spatiotemporal data. The problem is further exacerbated by the fact that the data may grow over time. Learning the model from scratch whenever there are new data available is not a feasible solution. In this section, we present an efficient algorithm to learn the model parameters and latent factors incrementally over space or time without requiring the memory of old data. Let Dnew = (X new,Y new) denote the set of new observations and ˜Π = {˜α, ˜W, ˜V, ˜C, ˜A, ˜ B} be the previous set of model parameters and latent factors estimated by the algorithm. For incremental learning, the model parameters and latent factors are updated by optimizing the following objec-

tive function:

where F(Π; Dnew) is given by Eq.(3) and

is a regularization term to ensure smoothness of the model by controlling the amount of information to be retained from the previous model. Our formulation enables the update formula to be applied when augmented with data from new locations or when data for a new time period become available. We consider the former as incremental learning over space and the latter as incremental learning over time.

Incremental Learning over Space

Let T be the current time step and S be the number of locations with training data at time T . Without loss of generality, we assume that the model is updated with new data one location at a time. Furthermore, the historical data for the new location is assumed to be available from time t0 to T. If the location has only one historical observation, then t0 = T. For brevity, we denote the multi-scale tensor data for the new location as X (l) S+1 ∈ ℜ1×(T-t0+1)×dl for l = 1, ...,L, and d = ∑Ll=1 dl. With the addition of the new data, the spatial latent factors need to be updated from its previous matrix ˜AS ∈ ℜS×K to AS+1 ∈ ℜ(S+1)×K = [AS; aTS+1 ]. We further assume that the spatial latent factors for other locations are unaffected by the addition of the new location, i.e., AS = ˜AS. However, the latent factors for other modes of the tensor (B and C(l) ) as well as the parameters of the prediction models (αl, W(l) and V(l) ) can be affected by the addition of the new data.

for incremental learning over space is:

where aS+1 is a column vector that represents the spatial latent factors for the new location. The smoothness parameter η1 determines the extent to which the previous model parameters should be retained.

An alternating minimization strategy is used to solve the optimization problem, and each subproblem is solved by the proximal gradient descent method [Parikh and Boyd, 2014]. The parameters are updated iteratively by calculating the gradient on the smooth part of the objective function, and then apply the soft-thresholding operator to determine its next value. The step size can be found using a line search algorithm. The details of the gradient calculation is omitted in this paper due to the space limitation.

Incremental Learning over Time

MUSCAT performs incremental learning over time when there are new observations available at the new timestamp. Let S be the number of locations and T be the current time. We denote the newly acquired data for all S stations at time T + 1 as {X (1) T +1 , X (2) T+1 , · · · ,X (L) T+1 }, where each X (l) T+1 ∈ ℜS×1×dl corresponds to the data at scale l and d = ∑Ll=1 dl. We use this information to update the temporal latent factors from ˜BT ∈ ℜT×K to BT+1 ∈ ℜ(T+1)×K = [BT ; bTT+1 ]. Similar to the strategy for incremental learning over space, we assume the new data for time T + 1 does not affect previous temporal latent factors: BT = ˜BT .

Incremental learning over time is implemented via the following two steps. First, we learn the temporal latent factor bT+1 based on the predictor variables for the new time period before the target variable is observed. Next, the parameters and latent factors for other modes are updated when the target variable for the new time period is observed for all the locations.

Step 1: Updating the temporal latent factor bT+1. The objective function for updating bT+1 is:

Note that A and {C(l) } correspond to the matrices obtained from the previous update. The predictions for the target variable are performed using bT+1 and other model parameters from previous update.

Step 2: Updating model parameters and latent factors after observing target variable. After observing the true values of the target variable at time T +1, the model parameters and other latent factors are updated by minimizing the following objective function:

4 Experimental Evaluation

This section describes the extensive experiments performed to demonstrate the effectiveness of MUSCAT when applied to a multi-scale climate dataset.

4.1 Climate Data

The climate data used in our experiments has three spatial resolutions. At the finest scale, monthly climate data are obtained for more than 300 weather stations from the United States Historical Climatology Network (USHCN) 1 . Four variables-maximum (tmax), minimum (tmin), mean (tmean) temperature and precipitation (prcp)-are selected as response variables for our prediction task. We train a prediction model for each response variable separately.

We use two gridded climate datasets, NARR and NCEP reanalysis, to create the predictor variables. NARR 2 , which stands for the North American regional reanalysis dataset, has a spatial resolution of 0.3◦ (32 km) whereas NCEP reanalysis 3 has a coarser resolution of 2.5◦ . Nine variables fromNARR (acpcp, air.2m, dlwrf, dswrf, lftx4, prate, prmsl, pr_wtr, and rhum) along with seven variables from NCEP reanalysis (cprat.sfc, dlwrf.sfc, dswrf.sfc, prate.sfc, tmax.2m, tmin.2m, and lftx.sfc) are chosen as predictor variables.

The monthly climate data span a 30-year period between January 1985 to November 2015. Table 2 shows the number of stations and grid cells, which may vary for each response variable since we discard stations with missing values. The time series for the predictor and response variables are deseasonalized (by subtracting each monthly value from the mean of its corresponding month) and subsequently standardized.

1 http://cdiac.ornl.gov/epubs/ndp/ushcn/ushcn.html

2 https://www.esrl.noaa.gov/psd/data/gridded/data.narr.html

3 http://www.esrl.noaa.gov/psd/data/gridded/data.ncep. reanalysis.derived.html

4.2 Experimental Setup

The 30-year climate dataset is divided into 3 partitions. We first incrementally build the models using training data from the first 10 years (1985-1994) and then apply the models to validation data from the next 10 years (1995-2004). After tuning the model hyperparameters using the validation set, we apply the chosen models to data from the last 10 years (20052015), which serve as our test set. Note that the number of weather stations associated with the training, validation, and test sets may vary depending on when each station was introduced into the incremental learning process. Initially, we randomly choose 100 weather stations as our starting locations. At each step of the incremental learning process, we randomly add either a new station or a new time period to update the latent factors and the ensemble of prediction models.

We used the mean absolute error (MAE) metric to evaluate the performance of various algorithms:

Note that the MAE is calculated for each station s starting from its corresponding test period ts. For example, if a station was introduced into the incremental learning formulation during the training or validation period, then its MAE is calculated for the entire 10-year test period. However, if the station was first introduced during the testing period, its MAE is computed starting from the time it was introduced. The incremental learning process is repeated 10 times, each time with a different initialization and random ordering of weather stations. Results are reported based on the average performance over the 10 trials.

4.3 Baseline Algorithm

We compare the performance of MUSCAT against the following three incremental learning algorithms.

1. STL (Single Task Learning): A linear model is incrementally trained at each location using the stochastic gradient descent algorithm. When a new weather station is introduced during the incremental learning process, its model parameters are initialized randomly and updated as new observation data become available.
2. ALTO: This is variation of the spatio-temporal multitask learning algorithm proposed in [Yu et al., 2015] which was designed to build models for multiple response variables simultaneously.
3. WISDOM: This is a recent spatio-temporal multi-task learning approach [Xu et al., 2016b] that applies tensor decomposition on the data but does not distinguish between variables from different scales.

4.4 Experimental Results

We designed our experiments to (1) compare the performance of MUSCAT against the baseline methods, (2) assess the value of using multi-scale data, and (3) analyze the significance of the latent factors.

Comparison against Baseline Methods

Table 3 shows the mean and standard deviation of MAE after 10 trials for each method. The results suggest that WISDOM and MUSCAT significantly outperform STL and ALTO in all four datasets, which suggests the advantages of using a multitask learning approach based on tensor decomposition. In addition, MUSCAT outperforms WISDOM on all four datasets in this study, which shows the benefits of using our approach to factorize the multi-scale tensors jointly, instead of factorizing a single tensor with concatenated features from all scales, which is the approach used in WISDOM. A more detailed analysis given in Table 5 shows that MUSCAT outperforms all other competing methods in at least 70% of the stations for all 4 response variables.

Value of Multi-scale Data

To determine the value of using multi-scale data, we consider the following two variations of MUSCAT: (1) MUSCATS1, which uses only predictor variables from NCEP, and (2) MUSCAT-S2, which uses only predictor variables from NARR. Once again, the results shown in Table 4 suggest that MUSCAT outperforms WISDOM, MUSCAT-S1, and MUSCAT-S2 on all four datasets. For temperature predictions, WISDOM is superior than MUSCAT-S1 and MUSCATS2, which shows the importance of incorporating data from all scales into the learning framework. However, for precipitation prediction, MUSCAT-S2 outperforms WISDOM, which indicates that augmenting the predictor variables from the coarsest scale may degrade the model performance. Nonetheless, MUSCAT still achieves the lowest MAE because it can learn the appropriate weight (α) for combining the predictions from NARR and NCEP reanalysis datasets. To measure the relative influence of data at different scales on the performance of MUSCAT, we examine the mean values of the parameters α1 and α2 over the 10 trials. The results given in Table 6 suggest that α2, which is the weight associated with the finer-level predictors, has a consistently higher weight than α1, the weight associated with coarser-level predictors on all four datasets. This shows that the finer-level predictors have higher impact on the prediction than coarser-level predictors.

4.5 Analysis of Latent Factors

One of the advantages of using MUSCAT is that the latent factors can be used to identify the spatial patterns of the data. To illustrate this, Figure 1 shows the spatial distribution of the latent factors for precipitation data. For each spatial latent factor, we plot its top 20% most influential stations on the map. The results suggest that the latent factors exhibit some spatially coherent patterns. For example, the first latent factor is more dominant on the eastern part of the United States, while the third and fifth latent factors are more influential on the northwest part of the country.

We also examine the temporal latent factors derived by MUSCAT by computing their pairwise absolute correlation against some of the well-known climate indices, such as AOI, NAO, WPI, QBO, PDO, and SOI (details of these climate indices are shown in Table 7). The results are shown in Figure 2. The correlation values are not that high, which is not surprising as the study region is limited to the United States only.

5 Conclusion

This paper presents a multi-scale multi-task learning framework for geospatio-temporal data by employing a supervised multi-tensor decomposition approach. The framework enables the multi-scale relationships to be harnessed by enforcing a constraint on the consistency between the spatial and temporal latent factors derived from the multi-scale geospatio-temporal data. An incremental learning algorithm over space and time is then proposed to efficiently learn the weights of the model. Experiments performed on a real-world multi-scale climate dataset demonstrate the effectiveness of proposed method compared to several baseline algorithms.