Diffract and Conquer: Hyperspectral Imaging from Any RGB Camera via Optical Encoding and Learning

1 Introduction

Computer vision technologies are widespread in various monitoring and inspection tasks, particularly in application domains such as precise agriculture [Arad et al., 2020; Kakani et al., 2020] or automated waste sorting [Kiyokawa et al., 2022]. In practice, these systems rely on RGB or RGB-D technical vision cameras due to their affordability and ease of integration, while hyperspectral imaging technologies have gained attention only in recent years [Kar and Dhal, 2025; Benelli et al., 2020; Tamin et al., 2023]. By capturing dense spectral features at each spatial pixel of an image, hyperspectral imaging allows precise material properties and color analysis. Hyperspectral data are conventionally captured using industrial scanning hyperspectral cameras based on whiskbroom or pushbroom imaging techniques. Such systems are often expensive and require a time-consuming acquisition procedure, which limits their applications in dynamic or realtime scenarios [Mukhtar et al., 2025].

Snapshot hyperspectral imaging is an alternative that aims to capture the hyperspectral data in a single exposure, significantly reducing acquisition time and motion artifacts. A common strategy for snapshot systems relies on spatial-spectral encoding via optical elements as prisms [Arce et al., 2013] or diffractive optical elements (DOEs) [Shi et al., 2024]. Such optical encoding produces hyperspectrally mixed measurements, making software reconstruction a necessary component of the HSI registration pipeline. Recovering dense spectral information from these measurements is an ill-posed inverse problem [Mukhtar et al., 2025]. A simple example of this paradigm is HSI reconstruction from RGB images [Arad and Timofte, 2022] or Coded-aperture snapshot spectral system (CASSI) images [Meng et al., 2020], where a neural network reconstructs the high-dimensional spectra from compressed observations.

The spectral resolution of RGB sensor equipped with Bayer color filter array (CFA) can be enhanced with DOEbased optical elements. Initially, the a Bayer CFA usage constrains optical efficiency [Shi et al., 2024] and spectral fidelity. However, these limitations can be mitigated by moving the spectral sensitivity enhancement to the DOEs configuration process. Extending this idea, we propose a snapshot hyperspectral imaging system based on a 4×4 matrix of harmonic diffractive lenses (HDLs).

In contrast to single-wavelength DOEs, our HDL focuses three wavelengths, featuring spectral profiles tailored to Bayer filter transmittance. To reconstruct the high-fidelity spectral signals from the captured snapshots, we introduce a compact network inspired by Kolmogorov-Arnold Networks, enabling efficient and accurate spectral reconstruction. The overall system is illustrated in the Figure 1. In summary, the main contributions of our work are as follows:

• We introduce a snapshot hyperspectral imaging system recovering 31 spectral bands (400-700 nm) via a 4×4 HDLs matrix. The architecture exploits multi-order diffraction of HDLs to acquire 48 raw channels on a standard Bayer RGB sensor, followed by a neural network that mitigates optical aberrations for high-fidelity spectral reconstruction.
• We propose a learned reconstruction approach inspired by Kolmogorov-Arnold Networks (KANs). Distinct from mentioned, our approach incorporates a learnable basis of Gaussian primitives. The parameters of these primitives are generated by a hypernetwork for each HSI pixel. Extensive experiments demonstrate that our design outperforms state-of-the-art methods by up to 10 dB in PSNR, while maintaining a compact size of 70K parameters.
• We develop and validate a fully functional prototype of the proposed snapshot hyperspectral camera. The prototype is experimentally evaluated under real-world imaging conditions and benchmarked against Specim IQ, an

industrial-grade scanning hyperspectral camera. The proposed system demonstrates comparable spectral reconstruction accuracy and spatial registration quality.

2 RelatedWorks

Optical Solutions for Snapshot Hyperspectral Imaging. Snapshot HSI, essential for dynamic scene analysis, is generally categorized into two paradigms following the taxonomy in [Mukhtar et al., 2025]. The first paradigm focuses on encoding spatial-spectral information via compressive sensing. While originating from the CASSI concept, this field remains active [Zhao et al., 2023; Ma et al., 2024], with recent architectures integrating periodic [Arguello et al., 2021] or random Fabry-Perot masks [Yako et al., 2023]. Alternative encoding mechanisms exploit dispersion via prisms [Baek et al., 2017], engineer spectrally varying PSFs using diffractive elements [Li et al., 2022; Xu et al., 2023], or employ multi-channel diffractive designs [Shi et al., 2024]. Although these approaches facilitate compact optics, they require solving ill-posed inverse problems heavily reliant on scene priors such as spectral smoothness [Baek et al., 2017; Shi et al., 2024; Mukhtar et al., 2025]. Conversely, the second paradigm utilizes spatial multiplexing to map the 3D datacube directly onto a 2D sensor. Prominent designs employ microlens arrays (MLA) with discrete filters or interferometers [Mu et al., 2019; Hubold et al., 2021; Picone et al., 2023; Wang et al., 2024], or utilize fiber-optic remapping [Lu et al., 2023]. While the pixel-to-voxel isomorphism of this approach eliminates the computational overhead of iterative reconstruction [Mukhtar et al., 2025], existing counterparts are hindered by significant hardware complexity. Specifically, [Mu et al., 2019] is rendered economically impractical by stringent filter array requirements, whereas [Hubold et al., 2021], utilizing a linear filter and tilted optics, is constrained by manufacturing intricacies and spectral broadening artifacts.

We propose a novel architecture synergizing a harmonic lens system with standard Bayer color filters and a neural reconstruction backbone. By integrating the CMV4000 sensor, our system achieves a spatial resolution of 450 × 450 pixels, effectively circumventing the resolution bottlenecks of existing spatial multiplexing hyperspectrometers and broadening their application scope.

Learned Hyperspectral Reconstruction techniques. Modern hyperspectral reconstruction (HSR) from RGB is software-only alternative to costly hyperspectral-imaging hardware, enabling spectral recovery using standard RGB sensors [Arad and Timofte, 2022]. The problem is inherently ill-posed due to spectral information loss during RGB formation. The data-driven methods have been widely adopted to learn the mapping from RGB to spectral space. Modern state-of-the-art (SoTA) methods employ transformer-based deep neural networks to achieve high-quality reconstruction [Cai et al., 2022c; Wu et al., 2024; Yang et al., 2024; Yang et al., 2026].

Beyond RGB inputs, similar learning-based reconstruction approaches have been developed for CASSI measurements [Meng et al., 2020; Cai et al., 2022a; Zhao et al., 2023;

Ma et al., 2024; Meng et al., 2025]. Although these systems provide richer measurements than RGB images, the captured signals remain strongly compressed, and their reconstruction pipelines commonly adopt transformer-based network architectures [Yao et al., 2024; Cai et al., 2022b; Li et al., 2024]. In contrast, snapshot hyperspectral systems based on DOE arrays capture more discriminative spectral information. These optical advancements enable SoTA results using simpler dense CNN-based architectures, eliminating the need for computationally demanding self-attention mechanisms while achieving superior reconstruction accuracy [Li et al., 2022; Arguello et al., 2021; Shi et al., 2024].

3 Snapshot Hyperspectral Camera with Harmonic Diffractive Optics and Gaussian Primitives

Figure 2 illustrates the optical architecture of the imaging spectrometer utilizing a harmonic lens matrix. The corresponding hardware implementation, featuring a 4×4 lens array, is depicted in Figure 3.

A harmonic lens differs from a conventional diffraction lens by a significantly higher microrelief h [Sweeney and Sommargren, 1995]:

where m is an order of harmonic lens, showing how many times the microrelief height of a harmonic lens is greater than the microrelief height of a diffraction lens at wavelength λ0. In this case, several wavelengths arise for a harmonic lens λκ [Sweeney and Sommargren, 1995]:

where k is an integer.

We demonstrate an imaging spectrometer where specific microrelief heights align three diffraction harmonics with standard RGB transmission windows.

Consequently, a 4×4 harmonic lens array yields 48 independent image channels. The optical performance of this HDL system is evaluated via sequential simulation of individual components. We formulate the image formation process for a specific HDL indexed by d (d = 1, ..., 16). The optical system focuses the incident wavefront from the scene onto a local region of the sensor plane equipped with a Bayer filter array. Integrating the spectral sensitivity of the photodetector Tc (λ) (where c = 1, 2, 3 corresponds to the R,G,B channels), the captured image Id,c (x, y) is expressed as follows:

where PSFd (x, y, λ) denotes the spatially and spectrally variant point spread function corresponding to the d-th HDL, and I (x, y, λ) represents the spectral radiance of the scene. The integration domain ∆λc is determined by the spectral bandwidth of the respective Bayer color filters c.

To compute the point spread function PSFd of the d-th HDL, we consider an incident spherical wave Uin (x1, y1, λ) originating from a point source located at a distance z = 2f, where f denotes the lens focal length. Consequently, by incorporating the complex transmission function, the output complex amplitude is obtained as follows [Doskolovich et al., 2020]:

where n is the wavelength-dependent refractive index, and hd (x2, y2) defines the geometry of the microrelief.

Leveraging (4), the PSF at distance z is defined as:

Considering the stochastic nature of the signal due to photon and electronic noise, and based on [Shi et al., 2024], the sensor readout is modeled as:

where Np constitutes the Poisson noise component, and Ng represents the Gaussian noise term. Eqs. (3) - (6) describe the formation of the d × c spectral bands Id,c.

3.1 Harmonic Lens Wavelength Optimization

In our 4×4 snapshot configuration, the aggregate spectral sampling is determined by the union of harmonics produced by all HDL elements. We address the visible spectral range of 400-700 nm, discretized into |Λ| = 31 bands with a uniform step of 10 nm.

A target wavelength λt ∈ Λ is considered covered by an HDL with height h and maximum order m, as defined in Eq. (1), if there exists an integer k ≤ m such that |λt -λk(h)| < δλ, where λk(h) = (n - 1)h/k.

Candidate microrelief heights are generated via an inverse design approach: λt and admissible k, we compute

then restrict h to [hmin, hmax] and quantize by ∆h.

We select at most D = 16 HDLs by solving a budgeted maximum-coverage problem:

We use a greedy maximum-coverage procedure (ties: smaller M, then smaller h);

3.2 Generated Gaussian Primitives Image Reconstruction Framework

Spectral Projector and Aggregator Blocks

Unlike prior methods that reconstruct full hyperspectral images (HSI) from a small number of channels [Shi et al., 2024] or from RGB inputs [Cai et al., 2022c], our proposed optical design acquires the full spectral range directly, eliminating the need for spectral super-resolution. Consequently, the reconstruction objective is formulated purely as an image enhancement task. The cmKAN framework [Nikonorov et al., 2025], which employs pixel-wise parameter generation for Kolmogorov-Arnold Networks (KAN), has demonstrated superior performance in RGB image enhancement. It leverages the spline capabilities of Kolmogorov-Arnold Networks (KANs) to model the color matching between source and target distributions. Specifically, it uses a hypernetwork that generates spatially varying weight maps to control the nonlinear splines of a KAN, enabling accurate color matching. The number of parameters generated per-pixel for M input channels is defined as:

where k, and G denote spline order and group size for the KAN spline basis, respectively. According to (9), processing RGB data requires 30 parameters per channel, totaling 90 parameters. However, extending (9) to HSI with 31 channels results in 310 per output channel, and 9,610 parameters in total, making cmKAN impractical for HSI applications.

To address this problem, we introduce a Spectral Projector block (Fig. 4), which functions as a sliding window along the spectral dimension to capture local dependencies within a three-channel receptive field. Furthermore, it incorporates a mechanism to compensate for the chromatic aberration inherent to HDLs. By modeling aberration as energy dispersion controlled by a PSF, we approximate the energy redistribution using a non-linear modulation term. Specifically, the feature refinement is defined as x+x·conv2D(x)2 , where the squared convolution term acts as an attention-like gate to correct PSFinduced artifacts.

The Spectral Aggregator block then projects the latent features back to the original HSI spectral space to restore energy balance. As shown in Fig. 4, a Feed-Forward Network (FFN) block [Cai et al., 2022c] serves as a bottleneck, reducing dimensionality from N to 3 prior to rearrangement. By integrating these projection blocks into the cmKANlight backbone [Nikonorov et al., 2025], we establish our baseline architecture, cmKAN++, specifically adapted for HSI reconstruction.

Gaussian Primitives Layer

Recent studies indicate that reducing the spline parameter count in KAN layers and employing Gaussian approximations for the spline bases [Li, 2024; Ta, 2024] improve HSI analysis performance [Firsov et al., 2024]. Motivated by these results and appealing to the Kolmogorov-Arnold Theorem in the form of Sprecher's generalization [Braun and Griebel, 2009], we propose a Gaussian Primitives Layer, which models the output channels cj , 0 ≤ j < N as a linear combination of parameterized 1D Gaussian functions:

where dj, aij, bij are layer parameters and si, 0 ≤ i < M are input channels. Unlike fastKAN, the Gaussian primitives in (10) are parameterized to adapt to the local data context. The layer processes M input channels generated by the Spectral Projector. The number of output channels N is a hyperparameter that can be tuned to optimize performance; in our work, we evaluated N ∈ 3, 5, 7, 9, 11. For each channel, the parameter count is given by:

For our range of N, this configuration results in at most 63 parameters, which is significantly fewer than the 310 parameters required by (9). Consequently, replacing traditional regular B-spline bases with learnable, non-regular Gaussian-based basis allows the Gaussian Primitives Layer to adapt to local image structure using fewer basis functions, thereby achieving substantial parameter efficiency.

Generated Gaussian Primitives Framework

We implement the Generated Gaussian Primitives (GGP) layer using a hypernetwork to generate spatially varying parameter maps that control the Gaussian bases in (10). The generator hypernetwork incorporates Illumination Estimator and Color Transformer blocks adopted from cmKANlight and redesigned specifically for Gaussian basis. To reduce memory consumption, we employ independent generators for each output channel.

By integrating the GGP Layer with a Spectral Projector and a Spectral Aggregator - which maps the N Gaussian layer outputs back to the original M spectral channels - we arrive at the proposed Generated Gaussian Primitives Image Restoration (GGPIR) architecture (see Fig. 4). The model is then defined as:

where Sproj and Saggr are the spectral projector and aggregator, GPL is the layer of Generated Gaussian Primitives, G is the generator block with parameters θ, and X, ˆ Y are the input and output hyperspectral images.

4 Computational Experiments

In this section, we provide a detailed analysis and evaluation of the proposed optical design and our reconstruction framework on four publicly available datasets.

4.1 Evaluation on Synthetic Data

To validate the proposed system, we utilized 4 diverse HSI datasets: NTIRE 2022 [Arad and Timofte, 2022]

(950 images), CAVE [Yasuma et al., 2010] (32 images), ICVL [Arad and Ben-Shahar, 2016] (400 images), and CZ HSDB [Chakrabarti and Zickler, 2011] (50 images). As a recent study [Fu et al., 2025] indicates that reconstruction methods are sensitive to scene content and HS device variations, therefore we evaluate each method within a single dataset without cross-dataset validation. We follow the official split for NTIRE and use a 4:1 training-to-testing ratio for the remaining datasets.

It is important to highlight that we compare various HSI estimation methods, including both software-only and opticsdependent approaches. This allows us to highlight the question "How precisely can we reconstruct spectral information using a similar sensor, but different (including apparatus) methods?". The first group consists of methods that reconstruct HSIs from RGB inputs, thus relying on a simple and accessible imaging system. For comparison, we selected SoTA neural network models from different years: MST++ [Cai et al., 2022c], MSFN [Wu et al., 2024], CESST [Yang et al., 2024]. The methods in the second group jointly utilize optical hardware and computational reconstruction, typically based on DOE-arrays: SCCD [Arguello et al., 2021], QDO [Li et al., 2022], and the method proposed in [Shi et al., 2024].

For the software-only approaches group, we use RGB images, provided by the dataset authors, as inputs. Since CZ HSDB does not include corresponding RGB images, we rendered RGB responses from the HSIs to sRGB color space using XYZ spectral sensitivities, following a procedure similar to other datasets. To generate the input images for the second group, we utilize the optical system simulators released by the respective authors.

For the evaluation of the proposed approach, which utilizes the combination of hardware equipment and neural reconstruction network, we employed a synthetic data generation pipeline simulating our hyperspectral camera model accordingly (3)-(6). This simulation was used for each dataset, as detailed explicitly in the previous section. The lens configuration was optimized as described in Section 3.2 to produce 31 spectral channels approximating a uniform spanning of 400700 nm with a 10 nm stride. The simulation parameters were set as follows: 20 mm focal length, 40 mm object/image distance, and 2.8 mm aperture. Subsequently, layer-wise normalization was applied strictly to the training subset.

To measure different aspects of reconstructed images, we employ 3 evaluation metrics: Spectral Angle Mapper (SAM) [Kruse et al., 1993] to estimate the accuracy of the reconstructed spectral shape, PSNR to measure the fidelity of reconstructed intensity values, and Structural Similarity Index Measure (SSIM) [Wang et al., 2004] to evaluate the spatial consistency of the reconstructed HSI.

The quantitative comparisons are summarized in the Table 1. On large-scale datasets such as NTIRE 2022, ICVL, HARVARD, the GGPIR achieves the highest reconstruction fidelity in terms of PSNR, outperforming the second-best method by at least 5 dB. At the same time, our approach shows competitive results in terms of SAM and SSIM, indicating good spectral accuracy and structural consistency. These results demonstrate the advantage of the proposed approach over both other software-only and hardware-software co-design methods.

On smaller and diverse datasets, such as CAVE, the performance gap is less pronounced. This reduced margin may be attributed to the tendency of software-only methods, which contain at least 1.5 M parameters, to overfit when trained on limited data. In contrast, the proposed HDLs-based optical system allows the use of a compact reconstruction network with less than 100 K trainable parameters, while still maintaining competitive reconstruction accuracy.

4.2 Assessment of the Optical Scheme

We conducted a quantitative evaluation of the proposed optical design on the CAVE and NTIRE datasets (Table 2), comparing the proposed 4×4 system featuring an optimized height profile against a variant with random height distribution (in the 2.0-4.5 µm range) as well as the 5×5 and 6×6 configurations. The 3×3 configuration was omitted due to insufficient spectral sampling.

Experiments demonstrated that a random lens-height distribution consistently degrades the resulting quality. This is attributed to the spectral mismatch between lens harmonics and target wavelengths (400-700 nm with a 10 nm step). Furthermore, transitioning to a 5×5 or 6×6 array with a fixed sensor size reduces the aperture of individual lenses, which significantly degrades image contrast.

4.3 Evaluation of the Reconstruction Framework

To validate the generalization ability of the proposed methods, we utilize three datasets as sources of HSIs: NTIRE 2022, ICVL, CAVE (see Table 3). To simulate corresponding input images we used image formation model of the proposed imaging system (3)-(6). As a baseline, we used cmKAN++ [Nikonorov et al., 2025] with the additional first layer - a projector reducing the data dimensionality from 31 channels to 3, along with the additional last layer - aggregator expanding the data dimensionality back to 31 channels, as described in Section 3.3. The proposed GGPIR architecture has been investigated in various configurations differing in the number of Gaussian layers: N=3 (S3), N=5 (S5), N=7 (S7) and N=9 (S9), and also a single configuration (M7) utilizing the 3D Gaussians instead of the summation of 1D Gaussians. Increasing the number of Gaussian layers, we get a higher capacity, however the number of network parameters increases proportionally: S3 contains 47K parameters, S7 and S11 contain 71K and 95K parameters respectively (i.e. 6K per each Gaussian layer), while the cmKAN++ contains 76K parameters. The optimal number of layers differs for each dataset: we get S7 as an optimal design for ICVL (leading to a 3.86 dB improvement in PSNR over the cmKAN++) and for CAVE (1.15dB improvement in PSNR), while the optimal design for NTIRE 2022 is S9 (2.79dB improvement in PSNR). Using 3D Gaussians (M7) does not bring any benefits; the PSNR and SSIM values are quite close to those obtained for S7. On the CAVE-HSI dataset the proposed architecture is inferior to the MST++ in terms of PSNR (35.58dB for S7 vs 36.36dB), while the S7 contains more than 22 times fewer trainable parameters (71K vs 1.6M). One extra architecture (SC) has been investigated: SC is based on S7 with the only block 1+x2 replaced with the block 1 + xσ(x) in the projector and aggregator (Figure 4), where σ(·) is a sigmoid function.

Notably, our main competitor is the Transformer-based MST++ baseline. However, despite using orders of magnitude fewer parameters, GGPIR achieves better reconstruction quality: as shown in Table 3, GGPIRS9 reaches 38.47 dB / 0.968 on NTIRE'22 with only 83K parameters, outperforming MST++ (37.88 dB / 0.96) that uses 1.6M parameters. This highlights the strong parameter-efficiency of the proposed GGPIR framework.

4.4 Qualitative Assessment

The proposed experimental setup was evaluated against ground truth data from a Specim IQ camera. Comparisons of reconstructed RGB imagery and spectral profiles (Figure 5) reveal a close alignment between our acquisition and reconstruction and the reference across all 31 spectral bands. The system achieved exposure times of 15 ms and 200 ms for halogen and RGB-LED light sources, respectively. These exposure times are comparable to those of the Specim IQ camera, confirming the feasibility of our snapshot implementation.

5 Conclusion

We presented an easy-to-implement snapshot hyperspectral imaging system that captures 31 spectral bands using a plugand-play optical adapter for conventional RGB cameras. The system combines an array of 16 harmonic diffractive lenses with a lightweight neural reconstruction model, enabling hyperspectral imaging without dedicated, less accessible devices. Each diffractive lens exploits multi-order diffraction to encode multiple spectral channels into RGB measurements. For reconstruction, we proposed GGPIR, a Gaussianprimitive-based framework with fewer than 70k parameters, which achieves accuracy comparable to or better than stateof-the-art models. Experimental results demonstrate consistent improvements of at least 5 dB over RGB-only reconstruction and prior optical adapter-based methods. Our results highlight the effectiveness of jointly designing diffractive optical encoding and neural reconstruction for practical hyperspectral imaging. The source code and supplementary is available at https://github.com/Liptee/Diffract-and-conquerhsi.

Acknowledgments

This work was supported by the Ministry of Economic Development of the Russian Federation (agreement identifier 000000C313925P3U0002, grant No 139-15-2025-003 dated 16.04.2025).