One4Many-StablePacker: An Efficient Deep Reinforcement Learning Framework for the 3D Bin Packing Problem

1 Introduction

The three-dimensional bin packing problem (3D-BPP) is a classic combinatorial optimization problem (COP) aiming to pack a set of items with known dimensions into a bin to maximize space utilization while satisfying geometric feasibility constraints, such as non-overlapping and fully enclosed placements [Fowler et al., 1981]. Existing approaches, including exact and heuristic algorithms [Chen et al., 1995; Martello et al., 2000; Wu et al., 2010; Crainic et al., 2009], face scalability and quality limitations when applied to large instances [Wu et al., 2023]. Recent efforts have applied deep reinforcement learning (DRL) to develop packing solutions from data or interactions, demonstrating superior empirical performance compared to heuristic rules [Zhang et al., 2021]. However, most DRL-based studies focus solely on maximizing volume utilization, neglecting stability-related constraints that are critical for real-world logistics, such as weight stability (preventing heavy-over-light stacking) and base support (ensuring sufficient contact area) [Zhou et al., 2024]. These constraints, essential for generating feasible solutions, remain underexplored in prior work [Wu et al., 2023]. Moreover, most existing DRL models are designed for fixed bin sizes (see Table 1), requiring retraining when bin dimensions vary, which limits their practical deployment in logistics with diverse bin specifications [Xiong et al., 2024].

∗Corresponding authors.

To address these limitations, we propose the One4ManyStablePacker (O4M-SP) framework. The term "One4Many" denotes the model's capability to generalize across bins with varying dimensions-including those unseen during training-without requiring retraining. Meanwhile, "Stable" highlights the explicit integration of stability constraints to ensure real-world feasibility. Simultaneously enforcing stability constraints while requiring cross-dimension generalization significantly increases problem complexity. Furthermore, existing DRL methods frequently struggle with singleobjective reward functions and are prone to premature policy entropy collapse [Wang et al., 2025b]. To mitigate these issues, we develop a new and efficient DRL-based framework. Specifically, we construct a weighted reward function that combines the loading rate with a novel height difference metric, and we implement clipped policy gradient optimization with policy drifting to promote exploration at critical decision nodes.

The main contributions are summarized as follows. First, we propose O4M-SP, the first DRL framework for offline 3DBPP that simultaneously addresses stability constraints and generalizes across diverse bin dimensions in a single training process. Second, we design a new weighted reward function to mitigate the limitations of single-reward functions, and conduct policy entropy control at critical decision nodes to promote exploration and enhance policy quality. Finally, we demonstrate that O4M-SP achieves superior packing performance over baseline methods and confirm that O4M-SP effectively enforces practical stability constraints.

2 RelatedWork

2.1 DRL for solving COPs

Combinatorial optimization is a key research topic in operations research and computer science [Schrijver and others, 2003], playing essential roles in path planning [Veres and Moussa, 2019; Li et al., 2024; Li et al., 2025], resource allocation [Huang et al., 2025], and task scheduling [Dolgui et al., 2019]. Current methods for solving COPs include traditional algorithms, deep reinforcement learning (DRL), and hybrid approaches. Due to their NP-hard nature, exact methods ensure optimality but are computationally inefficient, while heuristics improve speed yet face limitations in largescale, real-time scenarios. DRL has recently demonstrated advantages in solving COPs by modeling discrete optimization tasks as sequential decision-making processes [Wu et al., 2024], enabling efficient strategies to be learned through interactions between policy networks and the environment. Specifically, neural networks generate strategies trained via supervised learning [Fu et al., 2021] or reinforcement learning [Bello et al., 2016]. Early work focused on classic graph optimization problems. Khalil et al. [2017] integrated graph embeddings with Q-learning to achieve near-optimal solutions and strong generalization for minimum vertex cover and maximum cut problems. Later, Luo et al. [2023] introduced a Light Encoder and Heavy Decoder (LEHD) Transformer that dynamically captures node relationships, achieving effective generalization from small TSP and CVRP instances to solve large-scale problem instances with up to 1000 nodes. Regarding hybrid methods, Feng and Yang [2025] proposed the Suboptimal-demonstration-guided Reinforcement Learning framework, which outperforms prior learningbased methods for mixed integer linear programming problems in both branching quality and training efficiency.

2.2 DRL for solving 3D-BPP

Recent advances in DRL have demonstrated strong potential for solving 3D-BPP. In Table 1, we summarize recent learning-based approaches for solving offline 3D-BPP.

To our knowledge, Hu et al. [2017] first introduced a pointer network with policy gradient to minimize bin surface area for a novel variant of 3D-BPP, laying the groundwork for sequential decision frameworks for 3D-BPP. Subsequent efforts focused on model and algorithmic improvements. The approach in Duan et al. [2019] applied the more stable PPO algorithm, whereas Laterre et al. [2018] integrated deep neural networks (DNNs) with Monte Carlo Tree Search (MCTS) and introduced center-of-mass support constraints. In a related line of work, search-based methods such as the Packing Configuration Tree (PCT) have been proposed to explore continuous action spaces for bin packing tasks [Zhao et al., 2025]. With the rise of convolutional neural networks (CNNs), height map-based state representations became prevalent [Goyal and Deng, 2020; Jiang et al., 2021a; Jiang et al., 2021b; Zhao et al., 2024]. Height maps provide a compact encoding of space occupancy and, when paired with CNNs, deliver strong performance for fixed-size bins. However, the fixed input structure of CNNs restricts their generalization to variable-sized bins, limiting practical applicability. Recent studies have aimed to address this limitation: Zhang et al. [2021] leverage self-attention to model inter-item and item-bin relationships directly, demonstrating promise in handling variable-sized bins. The approach in Hu et al. [2020] combines CNNs (height maps) with attention mechanisms (for item features), introducing a stability ratio objective function and "convex hull checks", marking an early effort to integrate spatial efficiency with stability. For a comprehensive overview of the evolution of DRL in bin packing problems, readers are referred to Dahmani et al. [2025].

Despite the importance of stability in real-world logistics, our survey reveals that existing DRL-based offline 3DBPP research has largely neglected this constraint. As detailed in Table 1, exceptions are limited: Laterre et al. [2018] introduced center-of-mass support constraints, while Hu et al. [2020] implemented a simplified stability check through convex hull checks. In contrast, physical stability remains a central requirement in the adjacent field of robotic packing [Zhao et al., 2022; Zhao et al., 2023]. Notably, progress has been made in online 3D-BPP, where bottom support constraints have been considered and modeled [Yang et al., 2023; Zhao et al., 2021a; Zhou et al., 2024]. These studies offer valuable precedents for integrating complex yet practical stability constraints into offline 3D-BPP.

3 Methodology

3.1 Problem Statement and Formulation

In this subsection, we formulate 3D-BPP with stability constraints. The problem studied aims to find a non-overlapping placement of N cuboid items into a bin. Each item i has dimensions (li, wi, hi), where li, wi, and hi represent its length, width, and height, respectively. Similarly, the bin has fixed dimensions (L, W, H), where the maximum possible height is HN = ∑Ni=1 max(li,wi, hi). The objective is to maximize space utilization η:

HN is the minimal bin height required to enclose all items. The X, Y , and Z axes correspond to the bin's length, width, and height, respectively. All items must be axis-aligned, with edges parallel to the axes, and may be placed with any face as the base. Beyond these geometric constraints, we incorporate stability constraints (Figure 2(b)), comprising two main parts:

Support Constraints: To ensure physical stability, a placed item must be adequately supported by the items beneath it. Specifically, we compute the convex hull of the contact areas formed between the new item's base and the supporting items. The constraint is satisfied if and only if the geometric center of the item's bottom surface lies within this convex hull; otherwise, the placement is considered unstable.

Weight Constraints: These enforces that the vertical load on any item does not exceed its bearing capacity. For a new item j, the weight contribution Gji transmitted to a supporting item i is calculated based on the contact area ratio. The placement is feasible only if G(i) cur +Gji ≤ rwGi for all supporting items, where G(i) cur is the existing load on item i and rw is the maximum load-bearing ratio relative to the item's own weight Gi.

We formulate the above 3D-BPP as a Markov Decision Process (MDP), which is a tuple < S, A, P, R, γ >.

State: At time step t, the environment state is defined as st = {sbin, svalid items}. First, for the bin state sbin, existing methods have limitations: height maps [Zhao et al., 2024] and weighted 3D voxel grids [Yang et al., 2023], which rely on CNNs, limit the model's ability to generalize across diverse bin dimensions; the item list representation [Zhao et al., 2021b] not fully capture remaining space; and the Empty Maximal Space (EMS)-based PG representation [Xiong et al., 2024] captures all available space but provides a weak representation of packed items. To address these issues, we propose a hybrid state representation that combines the strengths of these approaches. At time step t, with n items packed, sbin is represented as an (n+2)×7 matrix: The first row contains the bin's attributes. The second row contains the attributes of the current space. The subsequent n rows represent the attributes of the packed items. A right-handed coordinate system is used with the front-left-bottom (FLB) vertex of the bin as the origin; columns 1-3 of the matrix represent the FLB coordinates, with the bin's coordinates being (0, 0, 0), columns 4-6 represent the length, width, and height, and column 7 represents the weight, with the weights of the bin and the current space being 0. The current space (highlighted in orange in Figure 2(a)) is selected from all available spaces (highlighted in blue S1 to S5 in Figure 2(a)) generated by the Empty Maximal Space (EMS) method using a screening heuristic. The screening heuristic is as follows: The loading sequence follows an XYZ order (prioritizing the X-axis, then the Y-axis, and finally the Z-axis) to maximize floor and space utilization. In our implementation, all available spaces are pushed onto a stack. We evaluate the stability of the space at the top of the stack; if valid, it is selected as the current target space. If unstable, the top space is popped from the stack, and the next top is evaluated.

Second, the representation of the valid items state svalid items is closely tied to the stability check. At time step t, if no available space exists, no valid item or valid action is available. Otherwise, the stack containing available spaces is checked as follows: the top space is evaluated, and the remaining items and their rotations are examined to identify valid orientations that can be placed in the space. If no valid orientations exist, the top space is popped from the stack, and the next top space is evaluated; if valid orientations are found, a stability check is performed on the candidate actions. Actions that pass the stability check are included in svalid items, and the top space in the stack is selected as the current space. At time step t, with k types of valid items, the state svalid items is represented as a k × 5 matrix: columns 1-3 represent the length, width, and height of each item, column 4 represents the remaining quantity of each item, and column 5 represents the weight of each item.

Action: Given the state st, action at selects a valid item and its rotation, placeing it at the front-left-bottom (FLB) corner of the current target space. The dimension of the action space depends on the number of valid items and their feasible rotations.

State Transition: State transitions are deterministic. Specifically, given the state st, action at is sampled according to the policy π(·|st), transitioning deterministically to the state st+1.

Reward: An effective stepwise reward provides meaningful intermediate signals to guide policy exploration. Existing methods employ mainly a single reward, such as volume change [Li et al., 2022], volume gap reduction [Li et al., 2020], loading-rate change [Zhang and Shuai, 2024], or gap-ratio reduction [Zhao et al., 2024], which may lead policies to converge prematurely to local optima. In this work, we introduce a new weighted reward (WR):

where rLR t denotes the loading rate reward (Eq. 3), rHD t denotes the height difference reward for the packing layout (Eq. 4), and α1, α1 are hyperparameters.

In Equations (3)-(4), Ht denotes the maximum height of all packed items at step t, and Ht ′ denote the second maximum height of all packed items. We normalize the difference between Ht and Ht ′ , assigning rewards to smaller height difference metrics to encourage flatter packing (See Figure 1).

We theoretically validate the proposed reward mechanism. Detailed proofs are provided in our GitHub project 1 . First, we establish that the stepwise loading rate reward rLR t forms a telescoping sum, meaning the cumulative reward exactly equals the final global utilization η. This property ensures that maximizing the stepwise reward is equivalent to maximizing the global packing objective. Next, regarding the height difference reward rHD t , we employ Potential-Based Reward Shaping (PBRS) [Ng et al., 1999] with the potential function Φ(s) representing surface flatness:

where H(1) and H(2) denote the maximum and secondmaximum heights, respectively. Since Φ(s) depends solely on the state, this shaping term guarantees policy invariance, accelerating convergence by encouraging flat packing surfaces without altering the optimal policy π∗ . Finally, we demonstrate that minimizing height variance maximizes the measure of the feasible action space. By explicitly penalizing height variance, the agent is guided to reduce surface fragmentation, thereby preserving larger continuous support areas for subsequent placements.

3.2 Network Architecture

The network architecture significantly influences the learning and generalization capabilities of DRL algorithms. To address limitations of existing approaches, we propose a novel state representation that enables model generalization across bins of varied dimensions and introduce a stability checker module to ensure adherence to physical stability constraints. The model adopts an actor-critic framework, as shown in Figure 2. The packing environment provides the bin state, valid item candidates, and rotation information, which are encoded into 256-dimensional embeddings. These embeddings are processed by the feature extractor, actor, and critic networks to produce action probabilities and value estimates, respectively. The network architecture comprises the following core components:

1 https://github.com/chaojihetao/3dbpp

The final solution Weighted reward

Figure 1: The loading-rate-only reward induces greedy vertical stacking, leading to surface fragmentation and restricted placement options (top). In contrast, the weighted reward (WR) promotes surface flatness, preserving a larger feasible action space and preventing premature convergence to local optima (bottom).

Feature Extractor: The feature extractor consists of eight identical encoder blocks, each comprising a self-attention layer, a cross-attention layer, and a feed-forward network (FFN), followed by residual connections and layer normalization. The self-attention layer integrates information from the bin, current space, and placed items to produce an updated state vector Snew. Along with Svalid items, it is fed into and processed by the cross-attention layer to enable interfeature interactions. The FFN in each block performs nonlinear transformations on the attention outputs, enhancing feature expressiveness.

Actor and Critic Networks: Both are two-layer multilayer perceptrons (MLPs) with GELU activation functions. The actor network processes state and rotation features separately, then combines them multiplicatively to produce a probability distribution over actions. The critic network directly estimates the state value from the packing state features.

3.3 TrainingMethod

A core challenge in training DRL neural networks lies in the exploration-exploitation trade-off [Sutton, 1988]. Policy entropy, an indicator of exploration capability reflecting action diversity, typically decreases as model performance improves [Cui et al., 2025].

To address this, we propose a tailored entropy-control scheme integrated into PPO for 3D-BPP. First, we note that prior studies identify critical decision nodes where policy entropy decreases significantly, narrowing the action space [Wang et al., 2025b]. Accordingly, we selectively apply entropy control at critical decision nodes during packing to restrict high-confidence decisions and enhance the model's ability to escape local optima. We observe that in 3D-BPP, entropy consumption is most pronounced during the initial steps, with entropy curves for varying item quantities shown in Figure 3.

Next, as indicated by Cui et al. [2025], when policy parameters are updated from θk to θk+1 , the change in policy entropy H for a given state s follows the pattern presented in Equation (6).

where ∆H = H(πk+1 θ | s) -H(πk θ | s) and H(πk θ | s) = - ∑a πk θ (a | s) logπk θ (a | s) represents the policy entropy at step k for state s, A(s, a) denotes the advantage of action a, and Cova∼πk θ (·|s)(·, ·) denotes the covariance calculated for all possible actions a under the policy πk θ (· | s) at step k. It reveals that when the covariance at step k is high, forward exploration leads to significant policy entropy loss, implying that high-covariance nodes are critical decision nodes.

Based on these findings, we conduct policy entropy control on the above two types of critical decision nodes. For nodes Nhigh cov with high covariance, we extract a small portion ϕ · Nhigh cov and apply the clipped policy gradient optimization proposed in Cui et al. [2025]:

This approach enables the model to explore highcovariance nodes while restricting certain nodes to perform only forward propagation, effectively mitigating the neural network's overfitting and preserving policy entropy. However, for the nodes Nfirst step corresponding to initial item placements, clipping may adversely affect the model's convergence. Hence, we apply a tailored policy drifting method, similar to Liu et al. [2025].

Jp(θ) denotes the policy loss term, which includes a penalty Ppolicy = |log( πθ(at|st) πθold (at|st) )|. The method mitigates policy deviation during initial item placement, weakens backpropagation intensity, and avoids premature convergence to local optima, thereby slowing policy entropy loss.

4 Experiments

4.1 Experimental setup

We implement O4M-SP on an Ubuntu 24.04 Linux server equipped with dual AMD EPYC 9554 64-Core Processors, 768GB memory, and an NVIDIA L40S GPU with 48GB memory. Model training takes about 37 hours. Training consists of 10,000 iterations, using the Adam with Weight Decay (AdamW) optimizer [Loshchilov and Hutter, 2019] with a batch size of 256. The learning rate schedule employs a warm-up phase (20 epochs) followed by cosine annealing, with an initial learning rate of 10-4 and a minimum learning rate of 10-6 . Performance is evaluated every 20 iterations on an independent validation set of 10,000 samples, and the best model is saved. Training incorporates an early stopping mechanism with a patience value of 100 evaluations. The source code and data generation details are publicly available at: https://github.com/chaojihetao/3dbpp.

4.2 Baseline

To evaluate O4M-SP, we select representative, publicly available algorithms from the literature as baselines. Table 2 reports the average loading rates and their variances for 10,000 test instances after 10,000 training iterations. All experiments are conducted with a fixed bin size of 100 × 100 × H, evaluating each method's performance on instances with 10, 16, 20, and 30 items. To investigate the effect of training sets with varied bin dimensions, we train O4M-SP on two datasets: one with fixed bin size (100 × 100 × H), denoted O4M-SP(S), and another with bin dimensions independently drawn from a uniform distribution over the interval [100, 450], denoted O4M-SP(M). Following Zhao et al. [2024], all DRL-based methods, including ours, decode 128 samples per instance and retain the best solution.

Experimental results demonstrate that O4M-SP consistently outperforms all baseline methods across varied problem instances. Notably, O4M-SP(S) and O4M-SP(M) exhibit distinct strengths depending on instance complexity. On smaller instances (10 and 16 items), O4M-SP(M) significantly outperforms both baseline methods and O4M-SP(S), indicating that small-scale packing problems are less sensitive to bin dimensions. Training with random dimensions enhances the model's generalization ability, mitigating overfitting to fixed bin sizes. Conversely, O4M-SP(S) achieves superior performance on larger instances (20 and 30 items), suggesting that sensitivity to bin dimensions increases with problem scale. Hence, specialized training on fixed bin sizes performs better on larger problems.

4.3 Generalization

To evaluate O4M-SP's generalization ability, we conduct two additional experiments. First, we test varied item quantities. Similar to the data generation method in Zhao et al. [2024], where the bin size is fixed at 100×100×H with 10 items per instance, we construct the training set and test it on benchmark sets with varied sizes (10, 16, 20, and 30 items). As shown in Table 4, the model trained on the training set with 10 items generalizes well. Compared to the models in Zhao et al. [2024] trained on test sets with 10, 16, 20, and 30 items, O4M-SP improves loading rates by 2.41%, 0.90%, and 0.88% on the 10-, 16-, and 30-item test sets, respectively. O4M-SP performs slightly worse (1.09%) on the 20-item test set.

Second, we evaluate the generalization capability of O4MSP on bin dimensions not encountered during training. The experimental setup comprises three fixed-size test sets (B1 30, B2 30, B3 30) with bin dimensions (150, 230, 180), (176, 153, 203), and (298, 103, 159), alongside a mixed test set (BM M) featuring randomized bin geometries and item quantities. Since existing learning-based methods typically fail to generalize to unseen bin dimensions without retraining, we compare O4M-SP against two heuristic algorithms: Greedy and GRASP.Additionally, we employ MCTS (configured with an intensive search time of one hour per instance) as a high-performance baseline to quantify the performance gap. As summarized in Table 3, O4M-SP demonstrates robust zero-shot generalization capability. It consistently outperforms both the Greedy and Grasp algorithms by a significant margin across all test scenarios. While the loading rate of O4M-SP is 5.18% lower than that of MCTS, it offers superior computational efficiency: O4M-SP solves all instances in minutes, whereas MCTS requires nearly an hour per instance.

4.4 Ablation study

To assess the contributions of key components in O4M-SP, namely the weighted reward (WR) and entropy control (EC), we conduct ablation studies using three variants: O4M-SP, O4M-SP without EC, and O4M-SP without WR. A multidimensional test dataset is constructed, comprising bins with fixed dimensions (S1: 100 × 100 ×H; S2: 300 × 150 ×H) and bins with dimensions randomly drawn from a uniform distribution over [100, 450]. Bin height H is set to the sum of the maximum dimensions (length, width, or height) of each items. Item quantities are either fixed (suffix 10, 30) or randomly generated (suffix M). This yields seven test sets: S1 10, S1 30, S2 10, S2 30, M 10, M 30, and M M. The ablation study results are summarized in Table 5. The policy entropy curves for each model during training are presented in Figure 4.

As illustrated in Table 5, the O4M-SP model consistently achieves superior performance across all test scenarios. Specifically, the average loading rate of the complete model surpasses that of O4M-SP w/o EC and O4M-SP w/o WR by 4.87% and 5.72%, respectively, validating the distinct contribution of each component. Figure 4 further corroborates these findings. The average entropy curves (left) demonstrate that the EC module effectively sustains higher policy entropy a higher level of policy entropy, preventing premature convergence and maintaining exploration capability. Consequently, as shown in the loading rate curves (right), the full O4M-SP model demonstrates accelerated convergence and attains a superior final loading rate compared to its counterparts.

4.5 Handling Stability Constraints

To validate O4M-SP's ability to handle practical constraints, we conduct a case study using seven item types (see Figure 5). The bin has fixed length and width dimensions of 100, while the height is set to the maximum item height multiplied by the total number of items. We test four scenarios based on the definitions provided in Section 3: the support constraints mandate that an item's bottom center lies within the supporting convex hull, while the weight constraints ensure the load on any item does not exceed its load-bearing capacity. As illustrated in Figure 5, Scenario 1 (unconstrained) achieves the highest loading rate of 84.24%, followed by Scenario 2 (support only) at 83.36%, Scenario 3 (weight only) at 82.49%, and Scenario 4 (both constraints) at 80.80%. Visual inspection reveals that Scenarios 1 and 2 violate weight limits, as heavy items (yellow) are stacked atop lighter ones. Conversely, Scenarios 3 and 4 satisfy weight constraints by placing heavy items at the bottom or stacking them on identical items. Similarly, Scenarios 2 and 4 successfully enforce support constraints, ensuring sufficient base contact for all items. In contrast, Scenarios 1 and 3 exhibit support failures, characterized by item centers falling outside the underlying contact convex hull. These results confirmO4M-SP's effectiveness in strictly enforcing stability constraints, underscoring its suitability for practical industrial applications.

5 Conclusion

We present a novel deep reinforcement learning framework, One4Many-StablePacker (O4M-SP), which integrates complex yet practical stability constraints to enhance realworld applicability. A core advantage is its "train once, apply broadly" capability, which enables generalization across diverse bin dimensions Extensive experiments demonstrate O4M-SP's superior performance over existing methods across various problem instances. Future research could extend the framework to address irregularly shaped items and refined load distribution models for multi-support configurations, thus enabling more comprehensive applications.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No. 72192823) and the Zhejiang Key Laboratory of Decision Intelligence (Grant No. 2025E10006). H. Ma (hongma@zju.edu.cn) and W. Zhou (larryzhou@zju.edu.cn) are the corresponding authors.