Core-Structures-Guided Multi-Modal Classification Neural Architecture Search

1 Introduction

A satisfied multi-modal classification (MMC) model is located in the space determined by multiple features and fusion operators [Liang et al., 2021; Liang et al., 2024; Yin et al., 2022]. According to the professional knowledge, experts can directly provide a solution such as [Xu et al., 2024; Han et al., 2023; Liang et al., 2022a; Guo et al., 2022]. Recently, some exciting results indicate that search-based techniques are able to obtain better solution than human designers. The random search strategy shown in Figure 1(a) may be feasible for querying all fusion architectures in a small search space. However, when the search space is larger, the strategy becomes computationally impractical.

∗Corresponding Author

Later, researchers turned their attention to neural architecture search (NAS), proposing various MMC based on NAS methods (MMC-NAS) such as EDF [Liang et al., 2021], BMNAS [Yin et al., 2022] and DC-NAS [Liang et al., 2024]. As illustrated in Figure 1(b), these methods first randomly initialize some structures, and then use the directed search strategies such as evolutionary algorithm, reinforcement learning or gradient-based learning for the best multi-modal fusion architecture search. The existing MMC-NAS methods have achieved a significant success. However, the ever-expanding number of multi-modal features and fusion operations has resulted in an increasingly vast search space. Consequently, the training and evaluation processes become time-consuming, and the entire search process continues to face substantial challenges in terms of optimization and efficiency.

To improve search performance and mitigate challenges posed by large-scale search spaces, as illustrated in Figure 1(c), we propose a core-structures-guided multi-modal classification architecture search method (CSG-NAS). In comparison to existing methods, CSG-NAS rapidly narrows down the entire search space to a high-quality subspace and searches for the optimal fusion structure within this subspace. This avoids evaluating numerous underperforming structures, directly improving the performance of MMC-NAS. The motivation behind this method is that optimal multi-modal fusion architectures often consist of structures composed of superior features and fusion operators, which we refer to as core structures. By basing our search on these core structures, we can quickly locate regions with optimal multi-modal fusion architectures, thereby reducing the entire search space to a subspace with advanced performance. Subsequently, we conduct local search within these core structures to find the optimal multi-modal fusion architectures. To further improve search efficiency, we introduce a knowledge inheritance strategy on top of the evolutionary algorithm. In the crossbreeding and mutation of offspring, many methods typically only consider the encoding perspective, neglecting the weight parameters learned by the parents. We leverage these weight parameters to further improve the efficiency of the evolutionary process.

Our research has been validated on multiple multi-modal datasets, showcasing optimal performance in terms of accuracy and efficiency. Specifically, our contributions include:

• The optimal MMC architecture is definitely located in the subspace that is determined by high-quality features and fusion operators. Based on the finding, we give the definition of the core structures for the first time and propose a core-structures-guided multi-modal classification architecture search method, which can rapidly identify the optimal fusion architecture in a vast search space.
• The similar MMC architectures exhibit similar performance. With this idea, we design an innovative adaptive knowledge inheritance strategy for the evolutionary algorithm, which facilitates the sharing and reutilization of knowledge within the population, enhancing the learning capabilities.
• Extensive experimental comparisons across multiple multi-modal tasks demonstrate that CSG-NAS achieves competitive performance in terms of reduced search time and model parameter count compared to state-ofthe-art multi-modal feature fusion methods.

2 RelatedWork

Multi-Modal Fusion: It fuses the relevant information from different modalities, achieving better performance [Jiang et al., 2024; Liang et al., 2022b].In context of deep neural networks, multi-modal fusion techniques are generally classified into three types: early fusion, late fusion, and intermediate layer fusion. Early fusion combines low-level features, late fusion combines high-level features such as predictions from the output layer of the network. Additionally, research on intermediate feature fusion indicates its benefits for learning gains and can enhance later fusion to improve performance. Therefore, some studies propose fusion at multiple intermediate layers. For example, CentralNet [Vielzeuf et al., 2019] and MMTM [Vaezi Joze et al., 2020] connect latent representations at each layer and pass them as auxiliary information into deeper layers. However, such approaches significantly increase the parameters of multi-modal fusion models.

Neural Architecture Search: NAS [Liu et al., 2019] has been introduced to automate the design of neural models, aiming to discover efficient architectures with competitive performance. The initial approach [Zoph et al., 2018] iteratively generated and trained candidate architectures with a reinforcement learning controller, but incurred high computational costs. Subsequent research adopted methods such as genetic algorithms [Real et al., 2019], Bayesian optimization [Mendoza et al., 2016], and predictors [Wei et al., 2023].The introduction of supernetworks was a significant milestone; for example, SMASH [Brock et al., 2018] utilized a one-shot network, training multiple candidate architectures simultaneously through shared weights, significantly reducing training time. To address limitations in exploratory performance, some methods like OFA [Cai et al., 2020] employed novel path sampling and optimization techniques. Overall, one-shot networks have proven to be efficient, achieving state-of-theart NAS performance.

NAS-based Multi-Modal Fusion: The integration of NAS into multi-modal learning has attracted significant interest. Evolutionary algorithms like EDF [Liang et al., 2021] and DC-NAS [Liang et al., 2024] maintained a set of architectures by generating new multi-modal fusion architectures through genetic operations such as mutation and crossover. Sequential model-based optimization algorithms are used in MFAS to search for given single-modal to multi-modal fusion architectures. Differentiable methods such as MMIF [Peng et al., 2020], 3D-CDC-NAS2 [Yu et al., 2021], and BM-NAS [Yin et al., 2022] optimize shared weights and architectural parameters to significantly reduce computational resource requirements and improve its search efficiency. However, they still require a considerable amount of time when dealing with complex and extensive multi-modal fusion search spaces.

3 Methods

In this paper, we propose a core structures-guided neural architecture search (CSG-NAS) for finding the optimal multimodal fusion architecture. CSG-NAS consists of two steps: (1) core structure search (CSS) and (2) core structures-guided optimal fusion architecture search. The main framework of CSG-NAS is shown in Figure 2.

To avoid confusion, we provide precise definitions for certain terms here. A population consists of individuals, where each individual corresponds to a MMC model encoded in the form of a tree. All representations extracted from different modalities are collectively referred to as features. The space composed of high-quality features and fusion operators is termed the core structure search space, while the remaining constitutes the non-core structure search space.

3.1 Retionality

Let S be the entire search space of multi-modal fusion, one partitions it into the architecture set with better performance S1 and the architecture set with worse performance S2 via one partition strategy. Our goal is to focus on searching the small portion of space (called the shrunk space in this paper) where S1 resides and eliminate the space where S2 is located. This can be achieved using a space shrunk method that can remove architectures with relatively poorer performance from the search space. There are two obvious advantages when one search algorithm works on the shrunk search space than S that 1) the shrunk search space with smaller cardinality leads to an improved training efficiency; 2) the higher proportion of optimal fusion architectures allows the search algorithm to concentrate training and exploration in more favorable regions of the search space. This leads to a better performance ranking. Therefore, our objective is to use the algorithm to locate the space where S1 resides and allocate computational resources to enhance the efficiency of the algorithmic search, while disregarding the space where S2 is situated. In this paper, we achieve this objective by proposing the Core Structures Search algorithm (CSS) detailed in Section 3.2.

3.2 CSS: Core Structure Search

By examining previously advanced multi-modal NAS methods such as MFAS [Perez Rua et al., 2019], EDF [Liang et al., 2021], BM-NAS [Yin et al., 2022] and DC-NAS [Liang et al., 2024], it is observed that the optimal multi-modal fusion architectures often contain certain structures that consist of the features and fusion operators with good performance, which we refer to as core structures. Hence, it can be assumed that the core structures are located in the subspace that is determined by the high-quality features and fusion operators. The subspace is referred to as core structure search space.

Based on the assumption, we can narrow down the entire search space to the core structure search space by evaluating the performance of each feature and fusion operator at a relatively low cost. Specifically, given n features denoted as M1, M2, ..., Mn and a single-modal classifier f. And then we obtain each feature Mi classification performance by passing it f, and select the first k1 higher-performance features to the high-quality feature set M 1 . Given m fusion operators denoted as F1, F2, ..., Fm and a multi-modal classifier h.

And then, we obtain the classification performance of each fusion operator Fi by replacing the fusion way of h with Fi, and select the first k2 higher-performance fusion operators to high-quality fusion operator set F 1 .

The space composed of M 1 and F 1 is termed the core structure search space. Next, in the core structure search space we employ the proposed enhanced evolutionary algorithm ENAS1 for the search. The final result after ENAS1 iterations is referred to as the core structures. The detailed process of the evolutionary algorithm can be found in Section 3.4. The remaining features and fusion operators are put into two sets M 2 and F 2 , i.e., M 2 = M-M 1 and F 2 = F -F 1 . Both of them will be used in CSG-OFSS in Section 3.3.

3.3 CSG-OFSS: Core Structures-Guided Optimal Fusion Architectures Search

Due to the fact that optimal multi-modal fusion architectures often contain core structures, we can leverage the core structures obtained in the first-stage search to rapidly determine a high-quality search sub-space containing the optimal multimodal fusion architecture. This subspace is comprised of the neighborhood surrounding the core structures.

The core structures identified through the ENAS1 search become the focus in this stage, where we concentrate on exploring the neighborhoods of these core structures to find the optimal multi-modal fusion architecture. Initially, the neighborhood of a core structure refers to the multi-modal fusion architectures formed by continuously adding substructures composed of the remaining features M 2 and fusion operator set F 2 to the core structures. To obtain the optimal multimodal fusion architecture, we employ ENAS2 to search the neighborhood of each core structure. Through the evolutionary algorithm, we can adaptively evaluate the fusion architectures around each core structure, achieving the goal of searching the neighborhood. The detailed process of the evolutionary algorithm can be found in Section 3.4.

3.4 Search Strategy

Two armed evolutionary algorithms ENAS1 and ENAS2 with our proposed adaptive knowledge inheritance (AKI) are used as search strategy to core structures search and the optimal multi-modal fusion architecture, respectively. AKI will be detailed in Section 3.6.

Initially, we initialize a population from the core structure search space and iteratively discover the core structures through the search process. Subsequently, evolutionary algorithms are utilized to search within the neighborhoods of these structures. Specifically, we treat the core structures as a new population and focus on expanding individuals in this population using mutation operations during the iterative search process. This expansion is aimed at exploring the surrounding neighborhoods. For instance, when an individual undergoes a mutation operation, we generate a substructure from the feature set M 2 and the fusion operator set F 2 , and expand the individual through the addition operation of the mutation, as illustrated in the lower right corner of Figure 2. The key steps of CSG-NAS include population initialization, fitness evaluation, offspring generation, and selection.

Population Initialization: Generate a population P consisting of K individuals randomly from the core structure search space.

Fitness Evaluation: Each individual is decoded into a multi-modal classification model, with detailed decoding processes outlined in Section 3.5. Subsequently, the model is trained and evaluated using the corresponding dataset to obtain the fitness value for each individual.

Crossover and Mutation: The entire algorithm is divided into two stages: ENAS1 for CSS and ENAS2 for CSGOFSS. Each stage has different crossover and mutation rates to meet its specific search objectives.

ENAS1: Crossover rate is r = r1, mutation rate is r = r2 In this stage, following the traditional EA principles, the probability of crossover operation is significantly higher than the probability of mutation operation. The objective is to search for core structures, and the mutation operation generates a substructure from the core structure search space, replacing a specific substructure within the individual.

ENAS2: Crossover rate is r = r1/9, mutation rate is r = 4r2. In contrast to the ENAS1, ENAS2 employs an extremely low crossover rate and a very high mutation rate. The goal is to explore the neighborhood around core structures; hence, the probability of mutation operation is high. The mutation operation generates a substructure from the non-core structure space and adds it to the individual, searching the neighborhood around the core structures. The low crossover rate aims to avoid disrupting the core structures themselves.

Selection: Binary tournament selection (BTS) [Liang et al., 2021] is used.

(a) Conventional crossover operator

(b)The novel crossover operator employs knowledge exchange for weight inheritance

Figure 3: The difference between original crossover and our one

3.5 Encoding and Decoding of Multi-Modal Classification Models

Each individual p within the population is encoded as a binary tree, where the leaf nodes denote features, and the branch nodes represent fusion operators. In this paper, fusion operators include concatenation, addition, multiplication, Maximize, and average. Detailed definitions can be found in work [Liang et al., 2021]. For each individual, if the binary tree contains k features, then it must contain k - 1 fusion operators. Each individual corresponds to a multi-modal classification model. The binary tree can be decoded into a multimodal classification model through the following steps: 1) Channel the modality features, represented by the leaf nodes of the individual encoding tree, into fully connected layers (FC) for feature alignment to facilitate feature fusion; 2) Conduct feature fusion based on the fusion operators represented by the branch nodes; 3) Direct the fused features through FC and Softmax layers for the final prediction output.

3.6 AKI: Adaptive Knowledge Inheritance

In traditional evolutionary NAS, offspring are typically generated through crossover and mutation of the encoding structures, without considering the weights learned by the parent during training, as illustrated in Figure 3. However, in addition to the network structure, the initialization weights also play a crucial role in the performance of convolutional neural networks (CNNs). Well-known initialization methods, such as Xavier initializer and Kaiming initializer, are often used in neural architecture search methods. However, these initialization methods do not fully leverage the knowledge acquired from training CNNs.

Based on the analysis, we introduce a novel breeding strategy based on knowledge inheritance. Its core idea is that similar structures often demonstrate comparable performance. In other words, the more similarity exists between the structure of offspring and parent, the closer their performance tends to be. This perspective has been validated in numerous studies. Consequently, we determine whether to inherit knowledge from the parent by calculating the similarity between offspring and parent, specifically the weight knowledge acquired by the parent during training. When the similarity surpasses a threshold s, it needs to inherit the parent's knowledge, thereby facilitating the learning process of the offspring. This approach allows some offspring to avoid training from scratch and instead fine-tune on the weights learned by the parent, leading to significant performance improvements and enhancing the efficiency of the entire search process.

To quantify the similarity between parent and offspring, the concept of tree edit distance [Gopal et al., 2023] is introduced, representing the minimum number of changes of transforming one individual into another. The calculation is specifically performed using a dynamic programming algorithm. This strategy aims to improve learning performance and efficiency by sharing and reusing the weights learned by the parent. During the breeding process, in addition to manipulating the network structure, this strategy also transfers the weights learned by the parent to the offspring, which helps the offspring network to converge and learn faster while fully utilizing valuable information obtained in the parent network.

4 Experiments

4.1 Multi-Modal Datasets

Our method is implemented using TensorFlow 2.0.3. The computational environment consists of Ubuntu 16.04.4, 512GB DDR4 RDIMM, 2X 40-Core Intel Xeon CPU E52698 v4 @ 2.20GHz, and NVIDIA Tesla P100. The used GPU configuration in this paper is the same as the MFAS.

We validated five popular multi-modal datasets: (1) ChemBook-10k (CB) [Liang et al., 2021] dataset, designed for chemical structure image recognition in patent retrieval studies, which contains 100,000 chemical structure images distributed into 10,000 categories. (2) NUS-WIDE-128 (NUS) [Tang et al., 2017] dataset, which contains 43,800 images divided into 128 categories. We chose a subset of 10 categories totalling 23,438 images from this dataset. (3) MMIMDB [Arevalo et al., 2017] dataset for the multi-label film genre classification task, which contains a total of 23 categories. The dataset is divided into a training set of 15,552 films, a validation set of 2,608 films, and a test set of 7,799 films. (4) NTU RGB-D [Shahroudy et al., 2016] dataset for multimodal action recognition task containing 60 categories. The training, validation and test sets include 23,760, 2,519 and 16,558 samples, respectively. (5) EgoGesture [Zhang et al., 2018] dataset for multimodal gesture recognition task containing 83 categories. The training set of this dataset includes 14,416 samples, the validation set includes 4,768 samples, and the test set includes 4,977 samples.

4.2 ComparisonMethods

To validate the effectiveness and efficiency of the CSGNAS, we compared it with several state-of-the-art algorithms. These peer competitors can be broadly categorized based on whether the architecture is manually designed. The first category is MMC whose fusion architectures are designed by human experts, including MBL [Kim et al., 2017], MFB [Yu et al., 2018], TFN [Zadeh et al., 2017], LMF [Liu et al., 2018], PTP [Hou et al., 2019], TMC [Han et al., 2023], TMOA [Liu et al., 2022], AWDR [Yang et al., 2019], RAMC [Jiang et al., 2022], Maxout MLP [Goodfellow et al., 2013] , VGG Transfer [Simonyan and Zisserman, 2015], GMU [Arevalo et al., 2017], CentralNet [Vielzeuf et al., 2019], Inflated ResNet50 [Baradel et al., 2018], Co-occurrence [Li et al., 2018], MMTM [Vaezi Joze et al., 2020], VGG-16 + LSTM [Yang and Tian, 2014], C3D + LSTM + RSTTM [Molchanov et al., 2016], I3D [Carreira and Zisserman, 2017], ResNext-101 [K¨op¨ukl¨u et al., 2019], and MTUT [Gupta et al., 2019]. The second category is NAS-based MMC methods including EDF [Liang et al., 2021], MFAS [Perez Rua et al., 2019], BMNAS [Yin et al., 2022], 3D-CDC-NAS2 [Yu et al., 2021] and DC-NAS [Liang et al., 2024].

4.3 Performance Comparison

Results on CB and NUS. To mitigate the randomness introduced by data splitting and network initialization, each dataset is divided into training and testing sets via 5-fold cross-validation.

From Figure 4, the performance of each feature and fusion operation can be observed. Following the aforementioned algorithm, we first select high-quality features and fusion operations to form the core structure search space. For example, for the CB dataset, we chose features M1, M3, M4, M7, M8, and fusion operations F1, F2, F5. For the NUS dataset, we selected features M2, M4, M6, and fusion operations F1, F2, F5. We then search for the core structures and, based on the core structures, use a local algorithm to search for the optimal multi-modal fusion architecture.

To thoroughly demonstrate the advancement of MMCNAS, our experimental settings follow the EDF [Liang et al., 2021]. We compared MMC-NAS with some advanced multimodal fusion operators and existing advanced multi-modal fusion methods. From the results in Table 1, we can conclude that, compared to advanced fusion operators, our use of basic fusion operators with our search strategy achieves a significant lead. Among multi-modal methods, except for EDF and DC-NAS, all others are non-NAS methods. It is evident that the performance of MMC-NAS methods is superior to manual selection. Due to our ability to rapidly locate core structures, locally search for optimal fusion architectures, and utilize a knowledge inheritance mechanism, our performance surpasses EDF and DC-NAS.

Results on MM-IMDB. To ensure a fair comparison with other explicit multi-modal fusion methods, we adopted the same neural network backbone models as BM-NAS and DCNAS to extract various modality features, with the weighted F1 score as the evaluation metric. The parameters are set as follows: the population size N is 20, the number of population iterations T is 8, the dimension of the fusion vector FD is 128, and modality features are repeatable. According to Table 2, the weighted F1 score of CSG-NAS is better than the existing multi-modal classification methods, surpassing the state-of-the-art MFAS, BM-NAS, and DC-NAS by 1.96%, 1.22%, and 0.42%, respectively.

Results on NTU RGB-D. Followed the data preprocessing pipelines of BM-NAS and DC-NAS to ensure the fairness of the experimental results. Specifically, Inflated ResNet-50 [Baradel et al., 2018] and Co-occurrence [Li et al., 2018] are adopted as the feature extractor for the skeleton and video modality, respectively, extracting eight features denoted as skeletoni and videoi, where i = 1, 2, 3, 4. Moreover, the population size is 28, iteration times is 8, fusion modality dimension is 64, and modalities are not reused. In Table 3, our method achieved a cross-subject accuracy of 91.12%, outperforming recent methods using video and skeleton modalities. By examining the optimal architectures obtained by MFAS, BM-NAS and DC-NAS, it can be seen that they both contain high-quality features, such as skeleton4 and video4, validating the necessity of searching from the core structure space.

Results on EgoGesture. The settings of the BM-NAS method are followed where ResNeXt-101 [K¨op¨ukl¨u et al., 2019] as the backbone network for RGB and depth video modalities. CSG-NAS is compared with various singlemodal and multi-modal methods. The experimental settings for CSG-NAS included a population size of 28, 10 iterations, no modal reuse, and a fusion dimension of 32. Table 4 presents the experimental results on the EgoGesture dataset. Compared to other unimodal/multimodal methods, CSG-NAS achieves state-of-the-art fusion performance, indicating that CSG-NAS can effectively enhance gesture recognition performance on the EgoGesture dataset.

4.4 Search Efficiency Comparison

The goal of this section is to compare CSG-NAS with several powerful MMC baseline methods, including MFAS, EDF, BM-NAS, DC-NAS, and MMTM, focusing primarily on search efficiency and performance to demonstrate its advanced capabilities. The research results are comprehensively summarized in Table 5. From the table, it can be observed that on five complex datasets, CSG-NAS finds the optimal fusion architecture in the shortest time. For example, on NUS and CB, we outperform EDF and DC-NAS in terms of performance and achieve nearly four times the efficiency of the EDF method and twice that of the state-of-the-art DC-NAS method. On NTU RGB-D and EgoGesture, we achieve significant advantages in both performance and efficiency, with the time consumption for searching the optimal fusion model reduced by almost ten times compared to the state-of-the-art BM-NAS and twice that of DC-NAS. This is attributed to our core structures-guided neural architecture search method, which significantly narrows down the search space, effectively avoiding the evaluation of a large number of poorly performing models, and focusing on assessing models with superior performance.

4.5 Ablation Study

To provide a more in-depth analysis for the proposed CSGNAS, we conducted ablation experiments on the CB dataset.

Effectiveness of Core Structures (CS) and Knowledge Inheritance (KI): To delve into the impact of CS and KI on CSG-NAS, we conducted a performance analysis in three scenarios of CSG-NAS. From Table 6, we can draw the following conclusions: Compared to searching the entire space, using core structures to guide neural architecture search allows achieving the same performance level in a shorter time. Looking at the standard deviations of CSG-NAS1 and CSGNAS2, CSG-NAS2 exhibits higher stability, indicating consistent discovery of the optimal fusion architecture. On the other hand, CSG-NAS1 shows larger fluctuations and may occasionally converge to suboptimal solutions. The results clearly demonstrate that the use of knowledge inheritance architectures can effectively enhance search efficiency and improve performance. For instance, the time was reduced from 36 hours to 25 hours and the performance increased from 88.57% to 89.20%.

Analysis of Core Structures Selection Strategies: To investigate the impact of core structures selection on subsequent search, four experiments were conducted. The first one experiments involved searching for core structures using high-quality features and fusion operators, while the next three experiments randomly selected features and fusion operators for core structures search. The experimental results in Figure 5 clearly indicate that searching for core structures using high-quality features and fusion operations leads to significantly superior outcomes compared to the case of randomly selecting features and fusion operations.

Knowledge Inheritance: To investigate the impact of knowledge inheritance on CSG-NAS, we conducted five experiments including no inheritance, complete inheritance, random inheritance, and inheritance occurs based on empirical observation when similarity is greater than 0.5 and less than 0.5. The results in Table 7 show that inheriting from offspring with a similarity greater than 0.5 leads to optimal performance. This indicates that determining knowledge inheritance based on the similarity between parent and offspring has a significant positive impact on CSG-NAS.

5 Conclusion

This paper has investigated a rapid and adaptive method for searching multi-modal fusion architectures, utilizing a core structure to swiftly narrow down the entire search space to a compact subset with state-of-the-art performance. The core structures have been employed for local search in its vicinity to identify the optimal multi-modal fusion architecture. Additionally, a novel knowledge inheritance strategy has been proposed to further enhance performance. Extensive experiments have validated CSG-NAS's advantages. The use of CSG-NAS holds the promise of significantly alleviating the challenges posed by vast multi-modal fusion search spaces and remarkably improved search efficiency.

Contribution Statement

Pinhan Fu and Xinyan Liang contributed equally to this work.

Acknowledgments

This work was supported by National Natural Science Foundation of China (Nos. 62136005, 62306171, 62376281, 62106132, 62306170, 62376146), the National Key Research and Development Program of China (No. 2021ZD0112400), the Science and Technology Major Project of Shanxi (No. 202201020101006), and Young Scientists Fund of the Natural Science Foundation of Shanxi (No. 202203021222183).