--- title: "Multivariate Times Series Classification Using Multichannel CNN" authors: - "YongKyung Oh" doi: "10.24963/ijcai.2022/835" url: "https://ijcai.org/proceedings/2022/835" pdf: "https://ijcai.org/proceedings/2022/0835.pdf" venue: "Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence" year: 2022 pages: "5865-5866" track: "Doctoral Consortium" keywords: - "Machine Learning (ML): General" --- # Multivariate Times Series Classification Using Multichannel CNN YongKyung Oh *Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence*, pages 5865-5866. DOI: [10.24963/ijcai.2022/835](https://doi.org/10.24963/ijcai.2022/835) ## Abstract Multivariate time series classification is an important and demanding task in sequence data mining. We focus on the multichannel representation of the time series and its corresponding convolutional neural network (CNN) classifier. The proposed method transforms multivariate time series into multichannel analogous image and it is fed into a pretrained multichannel CNN with transfer learning. To verify the efficacy of the proposed method, we compared it with recent deep learning-based time series classification models on five datasets with small amounts of training data. The results indicate that the proposed method provides improved performance on average compared with the other methods when incorporated with transfer learning. ## Full text ## 1 Introduction Multivariate time series classification (MTSC) has attracted considerable attention in the literature because it is widely applied in diverse areas such as health monitoring, motion recognition, gas mixture classification, and bioinformatics. Consequently, numerous algorithms have been developed for this task [Bagnall et al., 2018; Ismail Fawaz et al., 2019]. In this study, we focused on deep learning-based MTSC models that utilize convolutional neural networks (CNNs). Oh and Kim [2021] introduced multichannel representation of multivariate time series and its corresponding CNN architecture for gas mixture classification. Their proposed approach transforms multivariate time series into multichannel analogous images and utilizes well-established computer vision models pretrained by a large dataset, e.g. ImageNet. In this study, we expand the multichannel CNN classifier to a different dataset and compare it with recent deep learning methods for MTSC. ## 2 Problem Statement Suppose that we observe N classes belonging to the given dataset C = {c1, . . . , cnc } as follows: where nc denotes the number of classes. Let Xi = (xmt) ∈ RM×T denote a multivariate time series, where T is the number of time steps and M is the number of features. To predict the classes of Xi, we must construct a classifier F: F can be trained by minimizing the discrepancy between the predicted class ˆ yi and the ground-truth class yi. Here, we adopt a positive loss function L to reflect this discrepancy. ## 3 Methodology Figure 1: Image Transformation of RacketSports dataset ![Image](https://ijcai.org/proceedings/2022/fulltext/835/image_000000_5481ba2c59bc692615d9b3c94dae60aa32ff046bcbf5ff6d343876dda3936a9d.png) In general, Xi is not an appropriate form for 2D CNN model. Thus, we transform the original time series into an analogous image matrix before applying a CNN. The core concept is to transform the original time series Xi into an analog image matrix, as described by Oh and Kim [2021]. It generates M grayscale images using a preprocessing function. For example, the RacketSports dataset consists of six features that can be transformed into a six-dimensional analogous image matrix, as shown in Figure 1. After converting the multivariate time series into an image, the ResNet50 model is modified with a multichannel input. The input multivariate time series data are then converted into an analog image matrix and passed through corresponding channels. Subsequently, we assign individual time series to individual channels and apply a convolutional operation. In the case of the RacketSports dataset, the number of input channels is the same as the number of features, that is, six (= M). The final fully connected layer has four (= N) nodes for each class. In addition, we use the ImageNet pretrained weight for transfer learning. In previous studies, CNN models have shown promising results for MTSC [Ismail Fawaz et al., 2019]. Given an input multivariate time series, a convolutional layer consists of sliding filters over the time series, thereby enabling the network to extract nonlinear discriminant features that are time invariant and useful for classification. Thus, the proposed multichannel CNN approach enables the extraction of the nonlinear characteristics of multivariate time series. Table 1: Experiment results for five UEA datasets | Model | EP - WF1 | EP - Rank | EC - WF1 | EC - Rank | RS - WF1 | RS - Rank | SWJ - WF1 | SWJ - Rank | UWG - WF1 | UWG - Rank | Average Rank | |------------------|------------|-------------|------------|-------------|------------|-------------|-------------|--------------|-------------|--------------|----------------| | MLP | 0.274 | 7 | 0.168 | 7 | 0.243 | 7 | 0.275 | 4 | 0.247 | 7 | 6.4 | | FCN | 0.965 | 3 | 0.246 | 3 | 0.853 | 1 | 0.255 | 5 | 0.815 | 1 | 2.6 | | ResNet | 0.967 | 2 | 0.259 | 2 | 0.850 | 2 | 0.221 | 6 | 0.624 | 5 | 3.4 | | Time-CNN | 0.506 | 5 | 0.206 | 6 | 0.631 | 5 | 0.334 | 1 | 0.801 | 2 | 3.8 | | InceptionTime | 0.814 | 4 | 0.226 | 4 | 0.807 | 3 | 0.281 | 2 | 0.716 | 4 | 3.4 | | Proposed (False) | 0.681 | 6 | 0.213 | 5 | 0.419 | 6 | 0.167 | 7 | 0.262 | 6 | 6.0 | | Proposed (True) | 0.987 | 1 | 0.271 | 1 | 0.764 | 4 | 0.280 | 3 | 0.789 | 3 | 2.4 | ## 4 Experiments We compared five conventional deep learning-based time series classification methods to the proposed method. Wang et al. [2017] proposed three baseline models: multi-layer perceptron (MLP), fully convolutional neural network (FCN), and residual neural network (ResNet). The time-CNN approach proposed by Zhao et al. [2017] comprises two consecutive convolutional layers with a sigmoid activation function. Ismail Fawaz et al. [2020] introduced InceptionTime, which is inspired by the Inception architecture. Benchmark methods were implemented using the Python libraries sktime-dl 1 . We validated the proposed approach using datasets from the University of East Anglia (UEA) multivariate time series classification database [Bagnall et al., 2018]. We utilized five different MTSC datasets with small amounts of training data, as presented in Table 2: Epilepsy (EP), EthanolConcentration (EC), RacketSports (RS), StandWalkJump (SWJ), and UWaveGestureLibrary (UWG). Table 2: Characteristics of the five datasets | Problem | EP | EC | RS | SWJ | UWG | |--------------------------------------------------|------|---------|------------|-----------|-----------------| | Train size Test size # of # of Time # of Classes | 137 | 261 3 4 | 151 6 30 4 | 12 15 4 3 | 120 320 3 315 8 | | | 138 | 263 | 152 | | | | Features (M) | 3 | | | | | | steps (T) | 206 | 1751 | | 2500 | | | (N) | 4 | | | | | We compared the weighted F1 score for five-fold experiments and the average rank for each problem. In Table 1, the average of the five-fold scores is shown, and the ranks are compared. Proposed (False) is ResNet50 with random initialization, and Proposed (True) is ResNet50 with the ImageNet pretrained model. The proposed method with transfer learning is superior on EP and EC problems but underperforms on RS, SWJ, and UWG data. On average, the proposed approach achieved the highest ranking. As expected, transfer learning improved the classification performance, even with a pretrained weight from a totally different task. 1 https://github.com/sktime/sktime-dl ## 5 Conclusion We propose image transformation and CNN approach for MTSC. The proposed method exhibited better classification performance than previous approaches on the selected five datasets. Hence, it can be applied to different MTSC datasets. However, the proposed approach was validated using only five multivariate time series. Thus, future work will involve conducting more experiments on different MTSC datasets. Also, the proposed approach requires computation of the transformed image. The computational costs and inference times should be investigated in future studies. ## Acknowledgments This research paper is a part of my doctoral dissertation. I would like to express my heartfelt gratitude to Prof. Sungil Kim, my Ph.D. advisor, for his unwavering support, patience, encouragement, passion, and vast knowledge. ## References 1. [Bagnall et al., 2018] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075, 2018. 2. [Ismail Fawaz et al., 2019] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4):917-963, 2019. 3. [Ismail Fawaz et al., 2020] Hassan Ismail Fawaz, Benjamin Lu- cas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre- Alain Muller, and Franc ¸ois Petitjean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowl- edge Discovery, 34(6):1936-1962, 2020. 4. [Oh and Kim, 2021] YongKyung Oh and Sungil Kim. Multi- channel convolution neural network for gas mixture classifica- tion. In 2021 International Conference on Data Mining Work- shops (ICDMW), pages 1094-1095. IEEE, 2021. 5. [Wang et al., 2017] Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint con- ference on neural networks (IJCNN), pages 1578-1585. IEEE, 2017. 6. [Zhao et al., 2017] Bendong Zhao, Huanzhang Lu, Shangfeng Chen, Junliang Liu, and Dongya Wu. Convolutional neural net- works for time series classification. Journal of Systems Engineer- ing and Electronics, 28(1):162-169, 2017.