Beyond Vision: A Multimodal Dataset and Framework for Pest Recognition via Plant Electrophysiological Signals

1 Introduction

In the realm of modern precision agriculture, pest recognition serves as a pivotal cornerstone for sustainable crop protection [Zhou et al., 2024]. Timely identification enables earlystage intervention, significantly mitigating crop yield losses. Prevailing monitoring paradigms predominantly rely on the [Wadhwa and Malik, 2024]. Pest benchmark datasets ranging from the early IP102 [Wu et al., 2019] to the recent AP162 [Wang et al., 2025] have significantly advanced the application of pest detection models such as the YOLO series [Wang et al., 2024b] and Transformer variants [Fu et al., 2024] in field detection. However, the visual modality faces inherent physical limitations: complex illumination, leaf occlusion, and high concealment of tiny pests severely degrade recognition robustness, coupled with the prohibitive computational overheads of high-dimensional image data preclude all-weather continuous monitoring [de la Parte et al., 2024]. More critically, vision-based monitoring typically exhibits inherent temporal latency, as it often detects infestation only after pest maturation or damage escalation [Wadhwa and Malik, 2024]. This observational lag precludes proactive early warning during the larval feeding stage, thus calling for a novel monitoring paradigm.

In light of this, leveraging plant electrophysiological signals-a novel temporal modality characterized by low dimensionality, high sensitivity, and illumination insensitivity-emerges as a promising avenue to surmount the bottlenecks. Recent research indicates that systemic electrical signals, including Action Potentials (APs), Variation Potentials (VPs) and System Potentials (SPs) carry specific physiological response information induced by biotic stress [Li et al., 2025; Farmer et al., 2020]. Unlike occlusion-prone visual cues, the distinct electrophysiological variations within these signals provide unambiguous internal evidence of pest feeding behaviors (Fig. 1). Nevertheless, leveraging these electrophysiological signatures for fine-grained pest identification remains largely unexplored. Effectively harnessing these plant bio-signals is non-trivial. It necessitates overcoming a series of hierarchical obstacles, ranging from biological domain incompatibility and temporal misalignment to a counter-intuitive amplitude-semantic disparity.

The first challenge lies in algorithmic incompatibility. While signal processing is well-established for human physiology (e.g., EEG, ECG), directly transferring these stateof-the-art algorithms [Mohammadi Foumani et al., 2024] to plant electrophysiology faces fundamental challenges. This is primarily because existing time-series modeling methods typically assume that signals are generated by relatively stable or task-consistent mechanisms. For instance, human signals, regulated by the central nervous system and cardiac homeostasis, typically exhibit cyclostationary waveforms. However, during insect herbivory, the generation mechanisms of plant electrophysiological signals evolve non-linearly involving stimulus response, tissue damage, and physiological alteration [Farmer et al., 2020], leading to drastic structural distribution drifts. Such drifts do not originate from observational noise or external environmental changes, making them difficult to mitigate through traditional normalization or domain adaptation. To tackle this challenge, we introduce a Generation-State Encoder (GSE) that explicitly models these latent generation states to condition downstream feature learning.

The second challenge arises from the severe temporal mismatch between the hysteresis of plant physiological responses and the intermittency of insect behavior. Unlike rapid animal bio-signals, plant electrophysiological dynamics operate on a much slower timescale. Coupled with the stochastic nature of insect feeding, this characteristic results in continuous acquisitions containing substantial invalid data. To balance signal validity with real-time monitoring, we adopt an Event-triggered Windowing Strategy, extracting fixed-length segments of 16 seconds exclusively during feeding events. Although this strategy eliminates non-informative intervals, it leaves the intrinsic structural disparity within the bio-signals unresolved.

The third challenge stems from this disparity, manifesting as a "Semantic Inversion" where signal amplitude diverges from semantic significance. Plant bio-signals inherently comprise high-frequency APs and slow-varying VPs/SPs. Specifically, physiological patterns induced by herbivory are predominantly encoded within the low-amplitude VPs/SPs [Li et al., 2025; Mousavi et al., 2013]. However, these critical cues are easily overshadowed by the co-existing, high-amplitude APs. In this context, the conspicuous APs act as disruptive semantic interference, masking the robust biological features necessary for precise identification. Consequently, directly applying standard normalization causes feature collapse, as sparse AP outliers compress the effective dynamic range of VPs/SPs, severely impairing the model's ability to identify invasive biotic stressors. To mitigate this amplitude-frequency imbalance, we propose a Morphological Semantic Decoupling (MSD) strategy. This strategy is designed to selectively attenuate high-amplitude AP interference while accentuating the feature representation of slow-wave components, thereby recovering the effective dynamic range without compromising critical semantic information.

We summarize our contributions as follows:

• To the best of our knowledge, we release the HerbivoryInduced Plant Bio-signal Multimodal (HIPB-MM) dataset, the first dataset featuring synchronized video and plant electrophysiological signals specifically for insect herbivory, bridging the data gap and establishing a benchmark for future multimodal research.
• We propose the Herbivory-Induced Physiological Sensing (HIPS) framework for plant electrophysiological signals, integrating a MSD strategy to preserve robust slowwave semantics and GSE to model evolving states.
• Experimental results uncover the evolutionary patterns of herbivory-induced signals in relation to damage severity, providing a crucial empirical foundation for future research on invasive plant stress.

2 RelatedWork

2.1 Pest Recognition and Benchmarks

Existing pest recognition benchmarks focus on the visual modality. Early works released small-scale datasets, containing fewer than 1,000 samples. Subsequently, larger-scale datasets emerged; notably, IP102 [Wu et al., 2019] comprises 75,222 images across 102 classes, establishing a significant benchmark. Recently, research has pivoted towards finegrained classification and higher annotation quality [Wang et al., 2025]. These datasets drove algorithmic evolution. Early pest monitoring relied on analyzing environmental factors [Lee and Yun, 2023]. With the advent of deep learning, CNN-based approaches became the dominant paradigm, leveraging transfer learning and data augmentation to mitigate data scarcity [Ayan, 2024]. To address the limitations of standard CNNs in complex scenes, recent studies leverage Vision Transformers (ViTs) for global feature capture [Fu et al., 2024] and optimized YOLO models for real-time, small-object detection [Wang et al., 2024b]. Specialized networks have also been proposed for camouflaged and few-shot tasks [Bi et al., 2021; Wei et al., 2025].

Recent datasets like APTV-99 [Wang et al., 2023] and visual-text fusion methods [Duan et al., 2023] incorporate textual knowledge to assist recognition, yet they remain confined to static imagery. To address this, we propose the HIPBMM dataset and the HIPS framework, which synchronize dynamic video with plant electrophysiological signals to transcend the limitations of static imagery, establishing the first multimodal benchmark for pest classification.

2.2 Plant Electrophysiological Signals

Recently, deep learning has significantly expanded the boundaries of plant-based sensing, evolving from basic environmental perception to the sophisticated monitoring of plant physiological responses under various biotic and abiotic stresses [Zarbakhsh et al., 2025]. These advancements, coupled with the application of Automated Machine Learning (AutoML) [Aust et al., 2025], have validated the viability of plants as intelligent biosensors. Furthermore, to address the poor generalization caused by pronounced inter-individual variability, researchers have successfully introduced domain adaptation techniques to enhance cross-individual recognition accuracy via feature alignment [Liu et al., 2025a]. However, species-level pest identification via electrophysiological signals remains unexplored. To this end, we propose the HIPB-MM dataset and the HIPS framework, establishing a solid baseline for multimodal pest classification, which is essential for reducing chemical pesticide usage in sustainable agriculture.

3 The HIPB-MMDataset

3.1 Data Collection

To acquire synchronized leaf electrophysiological time-series data and insect-herbivory video, a dedicated dual-view imaging platform was developed (Fig. 2a). Within this setup, three-week-old Arabidopsis thaliana are positioned such that the target leaf extends into a transparent chamber, restricting larval feeding to the designated field of view. The recording electrode contacted the petiole via a KCl/agar salt bridge, while the ground electrode was inserted into the soil to maintain a stable reference potential. All electrophysiological signals were acquired using a Keithley 2401 source-measure unit. Both electrodes were prepared following a standard protocol [Mousavi et al., 2014].

Video and electrophysiological signal acquisitions were initiated simultaneously, with the trial commencing once the baseline reached a stable state. Subsequently, a secondinstar larva was introduced to the target leaf. As the larva engaged in continuous feeding, the resulting tissue damage elicited bioelectrical activity. Each trial spanned 45-85 minutes, yielding synchronized multimodal datasets of dualview videos and electrophysiological time-series. The larvae selected for this study comprised three common lepidopteran pests: Helicoverpa armigera, Spodoptera exigua, and Plutella xylostella (Fig. 3). The experimental site, data acquisition equipment, and theoretical physiological guidance were provided by our interdisciplinary partners (Fig. 2b,c). This collaboration ensured biological consistency in signal interpretation and dataset fidelity.

3.2 Data Pre-processing

Due to the temporal mismatch between the hysteresis of plant physiological responses and the intermittency of insect behavior, direct utilization of long-duration raw signals introduces substantial redundancy. To construct a high-quality benchmark, we implement a Herbivory-Event-triggered Windowing Strategy for data cleaning and segmentation, strictly adhering to the "multimodal consistency" principle. Under this protocol, a valid "Insect Feeding Event" requires simultaneous cross-modal verification: i) visual confirmation through rhythmic larval mandibular movements or explicit leaf area loss; and ii) electrophysiological correlation evidenced by significant non-stationary fluctuations in the plant signal. Applying these criteria, valid feeding intervals were extracted as short-duration segments, while ambiguous or fragmented samples were excluded. Ultimately, we constructed a dataset of 4,023 strictly aligned segments, with the category distribution detailed in Tab. 1.

3.3 Dataset Highlights

The HIPB-MM dataset features synchronously acquired plant electrophysiological signals and dynamic videos, manually screened to isolate valid feeding-induced segments. Unlike traditional static benchmarks, the HIPB-MM dataset retains complex real-world visual challenges-such as occlusions and underside feeding-to ensure robustness without relying on idealized imagery (Fig. 3). The dataset's rare synchronized dual-modality data facilitates the capture of finegrained behavioral cues, advancing research in plant-insect interactions and signal-based identification. Both aligned feeding segments and raw complete datasets are released to support multi-scale research in multimodal learning and biosignal modeling. By enabling early detection of pest infestations, this resource supports targeted intervention for sustainable agriculture, while extending the benchmark to abiotic and mechanical-stress conditions remains future work.

4 Method

4.1 Framework Overview

We formulate the task as a signal-centric multimodal classification problem, learning a mapping Fθ : (xs, xv) → Y from synchronized electrophysiological signals xs and auxiliary video context xv to target pest categories Y. The HIPS framework, illustrated in (Fig. 4), is designed as an asymmetric dual-branch architecture that prioritizes the intrinsic dynamics of plant electrophysiological signals while incorporating visual evidence for grounded calibration.

At its core, a primary signal branch is dedicated to decoding the complex biopotential variations; it first recovers stable slow-wave semantics via MSD and subsequently infers latent physiological regimes through a GSE. This process is facilitated by a supplementary visual branch that extracts pest behavioral dynamics and leaf morphological integrity to provide explicit physical priors. By integrating these heterogeneous representations through a conditional modulation mechanism, HIPS enables the robust inverse inference of pest species from non-stationary bioelectrical signatures.

4.2 Morphological Semantic Decoupling (MSD)

To mitigate "Semantic Inversion," MSD employs a robust trend-residual decomposition. We first isolate the physiological trend v = Gσ(xs) and residual r = xs - v. Then, we compute a robust threshold τ via Median Absolute Deviation (MAD) to clip AP outliers and reconstruct the signal y:

where λ and α are the robustness factor and reconstruction weight, respectively. This strategy effectively recovers slowwave semantics for state-aware encoding.

4.3 Generation-State Encoder (GSE)

Drawing on modern multi-scale paradigms, we initialize the signal embedding hs by fusing handcrafted multi-scale statistical and spectral features with deep window-based dynamics. Building upon this representation, to mitigate structural distribution drifts arising from progressive tissue damage, the GSE (Fig. 5) models the signal generation mechanism as a latent physiological condition. For a given input hs, the module infers a state vector zstate that characterizes the underlying generation regime. Based on this inferred state, we employ Feature-wise Linear Modulation (FiLM) to adaptively calibrate the feature manifold:

˜hs = FiLM(hs, zstate) = (1+γ(zstate))⊙hs+β(zstate), (3) where γ(·) and β(·) denote learnable affine transformations initialized to zero. This mechanism aligns signal distributions across varying stages of herbivory, ensuring that extracted features remain invariant to evolving tissue properties.

4.4 Dual-StreamVisual Context Encoding

The visual branch extracts explicit physical references through two specialized streams. The behavioral motion stream utilizes self-supervised VideoMAE [Tong et al., 2022] to capture ethological patterns-including locomotion, feeding, and quiescence-providing a semantic anchor for transient biopotential bursts. Concurrently, the leaf state stream employs pre-trained ResNet18 [He et al., 2016] to map tissue morphological integrity into structural references that counteract damage-induced drifts. Following latent compression, these high-dimensional cues are projected into compact embeddings (˜hm, ˜zl), which collectively guide the cross-modal fusion to resolve the ambiguity of implicit biological states.

4.5 Optimization Objectives

HIPS is optimized via a joint multi-task objective that facilitates robust classification while regularizing the feature manifold and cross-modal consistency.

Multi-branch Supervision. To mitigate class imbalance and background dominance, we apply a focal loss Ltask across the signal (s), video (v), and fusion (f) branches:

The fusion head hf is obtained by modulating the calibrated signal ˜hs with the behavioral context hm via Adaptive Normalization, which dynamically recalibrates signal features using visual cues.

Geometric Shaping and Distillation. To refine the representation space, a global contrastive loss Lcon enforces intraclass compactness and inter-class separability:

Furthermore, we introduce a state distillation loss Lstate to align the signal-derived state zstate with the visually grounded leaf integrity ˜zl via cosine distance:

This anchors implicit physiological inferences to explicit physical evidence of tissue damage.

Interaction Constraints. To prevent the signal branch from collapsing into the visual branch, Linter penalizes feature redundancy and enforces semantic consistency between the video branch Pv and signal branch Ps:

The first term minimizes the Frobenius norm of the crosscorrelation matrix, while the second term distills knowledge from the robust fusion head back to the unimodal branch.

Total Objective. The entire architecture is trained end-toend by minimizing:

where {λi} are hyperparameters balancing supervision and regularization.

5 Experiments

5.1 Experimental Setup

Baseline Methods. To comprehensively evaluate the performance of our framework, we compare it against four categories of time-series analysis methods: (1) Conventional DL: We select TCN [Bai et al., 2018], ConvLSTM [Shi et al., 2015], ResNet-TS [Wang et al., 2017], and InceptionTime [Ismail Fawaz et al., 2020] as standard deep learning benchmarks. (2) Non-Attention TS: We employ TimesNet [Wu et al., 2022], DLinear [Zeng et al., 2023], TimeMixer [Wang et al., 2024a], LightTS [Campos et al., 2023], and DisMS-TS [Liu et al., 2025b] to represent recent CNN or MLP-based architectures. (3) Attention-based TS: The comparison covers standard Transformer [Vaswani et al., 2017], FEDformer [Zhou et al., 2022], Informer [Zhou et al., 2021], Non-stationary Transformer [Liu et al., 2022], iTransformer [Liu et al., 2024], and ShapeFormer [Le et al., 2024]. (4) Foundation / SSM TS: We further incorporate emerging foundation and state-space models, specifically Peri-midFormer [Wu et al., 2024], MOMENT (evaluated in Frozen) [Goswami et al., 2024], and TSCMamba [Ahamed and Cheng, 2025]. Additionally, we report results for HIPS in both Signal-only and Visually-Guided settings to verify the efficacy of the fusion strategy.

Implementation Details. All experiments were implemented using PyTorch on a single NVIDIA RTX A6000 (48GB) GPU. To ensure reproducibility, we fixed the random seed at 42 and applied stratified sampling to split the HIPBMM dataset into training and testing sets with an 8:2 ratio. Following the established protocol, raw electrophysiological signals are segmented into 16-second windows without normalization, while video inputs are formatted as 128-frame clips. During training, we utilized the Adam optimizer with a batch size of 16. The initial learning rate was set to 5×10-4 and adjusted via a ReduceLROnPlateau scheduler based on validation loss. Evaluation metrics include Accuracy, MacroF1, Macro-Recall, Macro-AUC, MCC, and Cohen's Kappa.

5.2 Performance Evaluation

Extensive benchmarking on the HIPB-MM dataset against 18 baselines demonstrates the superiority of our framework. The Visually-Guided HIPS achieves state-of-the-art performance with 69.81% accuracy, outperforming the runner-up TimesNet by 7.2%. Notably, despite employing diverse mechanisms, the majority of advanced architectures-including TSCMamba and ShapeFormer-plateau below a 63% accuracy ceiling. This bottleneck indicates that generic sequence modeling struggles to disentangle the stochastic nonstationarity inherent in plant bio-signals. Similarly, foundation models such as MOMENT yield a suboptimal accuracy of 55.9% due to severe domain shifts. Remarkably, our Signal-Only variant achieves a competitive accuracy of

66.34%. This result surpasses all baseline methods and validates the efficacy of MSD and GSE in capturing discriminative semantics independently.

5.3 Ablation Studies

Effect of Structural Components. As shown in Tab. 3, HIPS performance improves incrementally with the integration of proposed modules. Specifically, MSD secures steady gains over the baseline, while GSE further enhances results, indicating their complementary roles in physiological modeling. Notably, Fig. 6 demonstrates that MSD consistently elevates accuracy across the majority of tested architectures, with particularly significant improvements in ResNet-TS and HIPS(Visually-Guided). These results confirm the broad efficacy of MSD in recovering robust physiological semantics, whereas marginal performance decays in a few specific models suggest potential architectural mismatches.

Sensitivity Analysis of Loss Terms. Fig. 7 illustrates the sensitivity of HIPS to loss weights λstate and λinter with λcon = 1. The proposed method consistently outperforms the 66.83% baseline, reaching a peak accuracy of 69.81%. Crucially, the removal of either λstate or λinter results in a marked performance decay compared to the optimal configuration. This trend underscores that both optimization objectives are essential and mutually reinforcing, validating the necessity of our joint training strategy.

Sensitivity to Window Length. To investigate the impact of the observation window on model robustness, we analyzed synchronized signal-video window lengths (Tab. 4).

Accuracy

The Visually-Guided accuracy peaks at 16s, striking the optimal balance between information density and stationarity. In comparison, the Full-Fusion model saturates to 100% at 32s. This anomaly betrays a severe visual bias, where the model bypasses physiological learning for visual shortcuts. This fragility is corroborated in our subsequent robustness analysis, where such visually-dependent models suffer catastrophic collapse under environmental degradation, validating the necessity of our signal-centric design.

5.4 In-depth Analysis of Visual Guidance

Cross-Modal Transfer Efficiency. Table 5 reveals a stark contrast: standard fusion achieves perfect accuracy on joint inputs yet collapses on signal-only inputs. Notably, even the HIPS backbone plummets to 27.83%. Knowledge Distillation offers partial recovery (∼53%) but still trails the 69.81% benchmark set by our Visually-Guided strategy. This confirms that robust physiological learning demands explicit physical alignment rather than generic probability mimicry.

Robustness against Environmental Degradation. We evaluate inference stability by injecting Gaussian noise into visual embeddings (Fig. 8) as an approximation of degraded visual conditions. With increasing noise, the Full-Fusion performance drops significantly, while our Visually-Guided mode remains stable. This failure exposes the high redundancy in visual data; relying on it makes the system fragile when visual quality degrades. This also implies the Full-Fusion model is unreliable, despite the perfect accuracy at 32s (Tab. 4).

6 Conclusion

In this work, we introduce plant electrophysiological signals as a complementary modality to overcome unimodal visual bottlenecks in complex agricultural environments. We present HIPB-MM, the first fine-grained multimodal benchmark coupling plant electrophysiology and video for pest stress recognition, alongside the HIPS framework. By integrating Morphological Semantic Decoupling with a Generation-State Encoder, HIPS synergizes physical priors and latent state dynamics. Experiments show that HIPS consistently outperforms existing methods in fine-grained classification, establishing a new performance benchmark.

Acknowledgments

The research was supported in part by the Key Research and Development Program in Xinjiang Uygur Autonomous Region (2025B04019-001), the China NSFC Grant (No. 62372307, No. U2001207), Guangdong Basic and Applied Basic Research Foundation (No. 2024A1515011691), Shenzhen Science and Technology Program (No. RCYX20231211090129039), Shenzhen Science and Technology Foundation (No. JCYJ20230808105906014), Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things (No. 2023B1212010007), the Project of DEGP (No. 2023KCXTD042), the National Natural Science Foundation of China (No. 22304186). Tao Wei and Chuchu Qin are corresponding authors.