ClothPPO: A Proximal Policy Optimization Enhancing Framework for Robotic Cloth Manipulation with Observation-Aligned Action Spaces

1 Introduction

With the continuous technological advancements in the field of embodied intelligence [Gupta et al., 2021], emerging application scenarios for robots are gradually coming to the fore, such as service robots. The manipulation of flexible objects is a fundamental and common skill in human domestic labor, including activities such as unfolding and tidying. A robot capable of manipulating clothes could alleviate the burden of household chores and significantly enhance the practicality of robots in our lives.

Compared to rigid body manipulation [Zeng et al., 2020], predicting morphological changes in fabrics through visual inputs is very difficult due to their high dimensionality and complex dynamics [Seita et al., 2021; Lin et al., 2022] in cloth manipulation. The reliance on supervised learning for such tasks [Xiong et al., 2023] demands an extensive collection of labeled training data, which can be a time-intensive and arduous process. [Matas et al., 2018] proposes to use selfsupervised learning to deal with the manual labeling problem. However, supervised learning models often exhibit limited generalization capabilities [Nair et al., 2017]. They tend to excel in conditions similar to their training environments but may falter in new or unforeseen scenarios. This characteristic becomes particularly problematic in robotic manipulation.

∗ corresponding auther

The emergence of Deep Reinforcement Learning (DRL) [Franc¸ois-Lavet et al., 2018] offers a new paradigm to achieve more generality and adaptability. Previous methods [Ha and Song, 2021; Canberk et al., 2022; Avigal et al., 2022] employ a value function to predict the unfolded coverage in the task of folding clothes and achieve very impressive results. By employing supervised learning on the coverage area values directly in the observation [Wu et al., 2020a], these methods successfully build the robot's ability to increase the performance of unfolding garments. However, these methods inherit self-supervision from the DRL framework but do not use multi-step reward, still highly rely on the accuracy of the value function. This makes these methods inherit the problems of supervised learning.

On the other hand, policy-based reinforcement methods [Chen* et al., 2023] have shown mediocre performance in practice. Traditional action commands in policy-based DRL are typically represented by low-dimensional discrete or continuous vectors. For instance, robot actions are often controlled by specifying directions (forward, backward, left, right) and movement distances. This results in an inherently discrete low-dimensional action space, compared with the high-dimensional nature of visual input space. [Wu et al., 2020b] employ CNN-based networks and MLPs to map the visual inputs to this low-dimensional action space. As the essential complexity of the garment's dynamics, this mapping potentially leads to significant information loss, which hinders the learning of policy. However, the dual-pose action directly within the pixel space meets another problem an exponential escalation in the number of feasible actions. The pixels in the image as a potential point of action. Considering an image dimension of 256, the resulting potential action choices would exponentially rise to the magnitude of 1010 , which is impossible to solve.

In this paper, we propose ClothPPO to fully take advantage of reinforcement learning for cloth unfolding tasks. To this end, an observation alignment policy is introduced to map the image directly to a high-dimension action distribution similar to the spatial action maps [Wu et al., 2020a]. To address the information loss from images to low-dimensional actions and directly connect image pixels with robot actions, we first to use spatial action maps in policy-based RL to model policy. This makes our proposed policy have enough capability to capture complex dynamics of fabric changes during manipulation. In addition, inspired by the success of reinforcement learning from human feedback [Ouyang et al., 2022] on large language model, a two-stage learning framework with PPO [Schulman et al., 2017] is proposed to solve the large action space learning problem [Ramamurthy et al., 2022] in policy-based RL. At first, we offline pre-train a UNet-based policy in a self-supervised learning fashion. Then an actorcritic architecture [Schulman et al., 2017] is employed with PPO to further online fine-tune the proposed policy.

We have the following three contributions. 1)Our method generates actions by sampling from a distribution-modeled policy, encouraging thorough exploration of the action space without relying on accurate estimations. Our work is the first to use a policy-based RL approach to directly select robot actions in the huge pixel space aligned with observation. 2) The vast action space makes the exploration process of the RL inefficient and difficult to learn. Inspired by the training process of ChatGPT, we bridge value-based and policy-based approaches, using the value-based method for initialization and employing PPO to enhance the pre-trained model's effectiveness in addressing the issue of large action spaces in RL. 3) We apply this method to the complex task of robotic cloth unfolding which achieves SOTA results on this difficult practical application problem, and contribute an environment based on our method to the robotics community.

2 RelatedWork

Research in the domain of flexible cloth manipulation has extensively focused on vision-based methodologies to orchestrate manipulative actions for an array of tasks, encompassing folding and unfolding. In the nascent stages of robotic garment folding, prevalent practices entailed initially unfurling the cloth into predefined, antecedently known states, subsequently engaging in the automated folding via procedural or rudimentary rule-based heuristic techniques [Triantafyllou et al., 2016]. The perception module mainly uses CNN [Avigal et al., 2022; Hu et al., 2023], Transformer [Mo et al., 2022; Chen et al., 2024] and other network structures.

In robotics cloth manipulation, the primitives used for manipulation are categorized into two types: quasi-static and dynamic. Previous work has actively explored the determination of parameters for predefined operational primitives [Mulero et al., 2023]. Dynamic methods calculate high-speed dynamic throwing actions directly from overhead images, which can smooth a garment to 80% coverage within an average of three actions [Ha and Song, 2021].

Current mainstream approaches are divided into modelbased learning, which models the environment's dynamics while neglecting policy learning, and model-free learning, which derives cloth manipulation strategies from images through supervised or reinforcement learning. The latter faces challenges in achieving high robustness due to ambiguities in 2D observational data, insufficiently flexible kinematic structures, and complex environmental interactions. Some papers address this issue by learning from demonstrations [Xiong et al., 2023]. Demonstrations can be obtained from scripted action sequences based on heuristic experts [Doumanoglou et al., 2016] or cloth descriptors [Ganapathi et al., 2021]. [Avigal et al., 2022] engage in selfsupervised learning augmented by minimal expert demonstrations to execute cloth smoothing and folding with an assortment of manipulation primitives, achieving a folding success rate as high as 93% within 120 seconds from a randomized initial configuration. However, this method is trained directly in the real world, has weak generalization, requires costly manual demonstration annotations, and has high training costs. Another learning approach is model-free reinforcement learning, [Canberk et al., 2022] introduces a selfsupervised reward-based approach for learning cloth normalization and alignment, diminishing the complexity of specific task-oriented downstream policies. However, due to the cloth's potential for various wrinkle structures, it is challenging to generalize to different target situations. There is also work primarily based on models, suggesting methods for modeling the dynamics of flexible objects for manipulation. [Lin et al., 2022] employing a particle-based GNN dynamic model informed by point cloud observations. Another study [Huang et al., 2023] proposes estimating the real-world cloth 3D mesh under action conditions. Yet, learning accurate 3D representations of real flexible objects within an endto-end framework remains difficult, the effectiveness of such methods is heavily reliant on the performance of 3D perceptual dynamics modeling.

3 ProblemFormulation

3.1 Markov Decision Process

The task of cloth unfolding presents a complex challenge, where robots must interact with fabric over discrete time steps t to maximize coverage area. This necessitates devising an effective manipulation strategy to explore the problem of learning robotic actions at from visual observations ot. A key difficulty lies in the inability to fully observe the cloth's state in real-world scenarios. It relies on partial visual inputs, creating a gap between perception and complete state awareness. Therefore, we model cloth manipulation as a Partially Observable Markov Decision Process (POMDP) [Sondik, 1971]. A POMDP is defined by the tuple (S, A, O, P, O, R, γ), where S denotes the state space of all possible configurations of the cloth. A represents the action space, comprising all the manipulative actions a robot can perform on the cloth. O is the observation space, reflecting the visual observations ot available to the robot at time t. P(s′ |s, a) is the state transition probability function that provides the probability of moving to state s′ from state s after taking action at. O(o|s′ , a) is the observation function that gives the probability of receiving an observation o after taking an action at and ending up in state s′ . R(s, a) is the reward function that assigns a numeric reward for taking an action at in state s, guiding the robot to maximize the cloth's coverage area. γ ∈ [0, 1] is the discount factor that values the importance of immediate versus future rewards. Suppose the strategy parameter is θ and our optimization objective is to maximize the average return. A reward is rt = r(ot, at, ot+1). Given the sequence tuples of (ot, at, rt). The optimization objective can be expressed as J(θ) = E[R], where J(θ) is the objective function of the strategy and R = ∑Tt=0 γ t rt:

It is to find a policy πθ(at|ot) that maximizes the expected sum of discounted future rewards. This involves calculating an expected value that balances immediate and future rewards. To increase the probabilities of taking actions in highreward trajectories. In this way, during the training process, the policy gradually adjusts its parameters to improve its performance.

3.2 Definition of Observation and Action

In the task of clothing unfolding, the agent is assigned to grasp the points on the cloth and unfold it into a flat state. This task can be accomplished through a series of dual-pose motion primitives, such as pick and place. Given the original header observation, policy decides the optimal actions at = (ap1 , ap2 ) ∈ A. Observation is an RGBD image ot ∈ RW×H×C , where H and W are the height and width dimensions of the image. The function that maps observations to actions is defined as:

where p(ap1 |ot) and p(ap2 |ot, ap1 ) are the probabilities of these actions given the observation ot. Since observations can be projected into a 2D-pixel space, with the third dimension held at a fixed depth, it treats this as a two-dimensional action. This constrains ap1 and ap2 to two points in pixel coordinate.

The dual-pose joint action model, which incorporates both direct and visual input modalities, is formalized as a multivariate discrete scalar action space. This space comprises four variables amalgamated within the pixel space. However, the two-step operational action space involved in making decisions directly at the image pixel level is huge.

4 Method

As shown in Fig.1, the training process of our proposed method consists of two phases, our contributions focus on the second phase:

Self Supervised Pre-training. The model is founded on self-supervised principles and adopts a regression methodology for individual pixel values Q using Mean Squared Error (MSE) loss. However, its inherently step-by-step prediction approach presents a key limitation: a noticeable absence of long-term guidance.

PPO Training. In the second phase, we use the PPO to fine-tune the pre-trained model. In this phase, we introduce a long-term reward mechanism to improve the model's performance in goal-oriented tasks. The PPO iteratively optimizes the model's policy to better adapt to the goal task and guides the model's learning process through reward signals. With these two phases of training, our goal is to enable the model to generate excellent grasping actions in grasping tasks and show higher performance in goal-oriented tasks. This twostage training strategy improves the generalization ability and task adaptability of the model.

4.1 Observation Alignment Policy

Redefine Action to Reduce the Action Space. As formulated Eq. 2, the dual-pose joint action comprises four variables amalgamated within the pixel space. We redefine the action space of the above PDMDP as another multivariate discrete action space and re-describe the joint action using the starting point and the line segment originating from it. This effectively reduces the dimensionality of the action from a larger two-step pixel space action to a one-step pixel 106 action space, while ensuring the correlation between the two poses. The action at time t can be presented as

where each element represents a different dimension of the action space, x, y represents the set of action positions in pixel coordinate, n is the number of a set of angles Φ = {ϕi|ϕi = 360◦ ∗i n , i = 1, 2, ..., n}. m represents the number of a set of movement distances D = {di|di = b-a m ∗ i + a, i = 1, 2, ..., m}, where b, a represent the max and min movement distance. z ∈ Φ×D denotes the combination of an angle and a movement distance.

The line segments on the clothes explain this definition in Fig.2. The black and purple line segments represent two different actions.

Rotate and Scale Observation. For intuitive understanding, we refer to the setting of spatial action maps [Wu et al., 2020a] to rotate and scale the observation to align action space with the visual input, x, y, z is the position of the 3D map. The corresponding joint action is determined by the index of the pixel in the 3D map, so a single pixel can correspond to an action. Spatial action maps facilitate the network's ability to learn and predict the Q value for each point, guided by the characteristics of the state representation. They are used to predict the value function Q(st, at), with the map value representing the expected reward for choosing a particular action. We use it as spatial policy maps to model the policy itself, which represents the preference for choosing various possible actions in a given state. The value in our map is the probability of opting for a particular action. As far as we know, ours is the first work to use it in policy-based RL.

4.2 Sampling Action fromOBAP

Flatten 3D Policy Maps to Categorical Distribution. Spatial policy maps output from the network indicate the probability of each potential action within the environment. To sample an action at according to the policy πθ(·|ot), the multi-dimensional spatial policy maps must be converted into a one-dimensional logits vector, which represents the log probabilities before normalization. This transformation process is referred to as flattening.

When employing a categorical action spaces [Tutz, 1991], the output from the network can be transformed into a discrete probability distribution through a softmax operation on the logits vector, denoted as zt:

where at is the action taken at time t and k iterates over all possible actions.

Spatial Policy Mask. Sampling actions directly on the existing spatial policy maps can lead to many invalid actions (e.g., futile grasps or actions exceeding designated limits). Since the map denotes the probability of selecting an action, the corresponding mask represents regions with non-zero selection probabilities, visually equivalent to clothing parts in the observation. Due to the different rotations and scalings of each layer, the shape of the cloth mask is also different (Fig. 2). For the mask, these clothing areas are assigned true values while other areas are marked as false values, which are displayed in black in the simulation. We have added a different mask to each layer of the spatial policy maps. For each state observation ot at time t, the applicable action space is obtained by excluding the invalid actions identified by the mask Mt. As indicated by the equation below, we have incorporated a mask into the action space on top of PPO to filter out futile actions:

where Mt(at) is the masking function at time t that returns 1 for valid actions and 0 for invalid actions.

As shown in the red area in Fig. 2, if the observation is scaled to smaller, from an action perspective, the line segment will be relatively longer, and the movement will be further. From the perspective of observation, there are fewer pixels left after being masked. Then its proportion in the distribution will become smaller after flattening. In other words, violent action will be a small probability to sample, and vice versa (blue area). The benefit of this Spatial Policy Mask(SPM) is the probability of actions of different granularities being sampled can be controlled by the size of the mask corresponding to the scaling, thus enabling a more efficient search through the action space for the optimal set of movements.

In this way, we get our Observation Alignment Policy(OBAP). To account for constraints in the action space, mask vector mt is applied element-wise to the logits before the softmax operation. This mask has binary entries where 1 indicates a valid action and 0 is an invalid one. The probability of selecting any specific action at from the masked action space is given by ˜ p(at|ot), which respects the feasibility of actions as determined by the constraints encoded in the mask mt .

where at is the action taken at time t and k iterates over all possible actions, zt is the logits vector which is transformed into a discrete probability distribution through a softmax operation.

4.3 Proximal Policy Optimization Finetuning

By employing the PPO [Schulman et al., 2017], we have augmented the efficacy of the supervised policy in cloth manipulation tasks.

Throughout the training, an action from the action space is sampled at each timestep. Specifically, we sample an action at according to the policy πθ(at|ot) derived from the current model parameters θ. The core of the PPO update is the clipped surrogate objective, which is used to modify the parameters θ of the policy network to maximize the expected return. Given a set of trajectories (rollouts), collected under the current policy πθold , PPO updates the policy parameters by optimizing the following loss function:

where ˆ At = rt+γVt(ot+1)-Vt(ot), is an estimator of the advantage function at timestep t, pt(θ) denotes the probability ratio pt(θ) = πθ(at|ot) πθold (at|ot) , and ϵ is a hyperparameter, typically a small constant like 0.1 or 0.2, that defines the clipping range to avoid overly large policy updates. The advantage function estimator ˆ At quantifies the benefit of taking action at in ot over the expected value of the policy at that time. It is calculated using a value function V (ot). E[·] is taken over the distribution of actions and states encountered under the current policy πθold .

Model Architecture. As shown in Fig. 1, the PPO phase fine-tuned the OBAP in simulation via an actor-critic architecture [Schulman et al., 2017].

Actor. OBAP selects actions based on the current observation. The goal of the actor is to maximize the cumulative reward. OBAP accepts RGBD observations ot as input and outputs per-step action at. OBAP gets the next observation ot by interacting with the environment with at and optimizes its parameters based on the multi-step reward to enhance the policy's performance. In the initial phase of PPO training, OBAP is a copy of the pre-train model. As the PPO finetunes proceed, the parameters of OBAP are iteratively updated through the policy gradient (see Eq. 7).

Critic. The critic is the value function V (ot) (see Eq. 7) estimator that evaluates the performance of OBAP. The label of the critic is calculated by muti-step rewards R(ot, at) (see Eq.1). The critic accepts ot as input and outputs an estimate of the corresponding R(ot, at). Critic obtains reward signals by interacting with the environment and uses these signals to update the parameters of the value function. The network consists of a pre-trained U-Net combined one-layer MLP, that is updated using MSE loss for optimization.

4.4 Training Details

Reward Design. In addressing the cloth unfolding task, our objective is to maximize the cloth's coverage over the workspace, referred to as Coverage. To effectively guide the agent toward achieving higher coverage, we formulate a reward function that integrates both sparse and dense rewards. The reward at each time step rt, is determined by assessing changes in coverage. This change is calculated as the ratio of the difference in current and previous coverage to the maximum possible coverage. The agent will receive a sparse reward of rt = -1 if the coverage decreases, and a dense reward of rt = ∆Coverage × 20 if the coverage increases. Additionally, achieving a Coverage greater than 0.9 triggers a sparse reward of 5, further motivating the agent towards optimal cloth manipulation.

Reward Scaling. Given the presence of abrupt, sparse rewards in the reward distribution, overly large numbers can cause instability during network training. Therefore, we have scaled the rewards to act as normalization, stabilizing the training. The scale is realized by dividing the reward rt by the standard deviation of the reward sequence RS. This is mathematically reflected in the following equation:

where ˜ rt represents the normalized reward at time t, and σ(RS) denotes the standard deviation of the reward sequence RS. By applying this operation, the rewards retain their relative differences in value but are expressed in measures of standard deviations from the mean of the reward distribution.

5 Experiments Settings

Environment and Tasks. Referring to the setting of [Canberk et al., 2022], the simulation environment is built on top of PyFleX [Li et al., 2018] bindings with Nvidia FleX bindings provided by SoftGym [Lin et al., 2020]. We randomly sample the initial state of the cloth from a subset of Cloth3D [Bertiche et al., 2020], which contains filtered meshes of various garments. We only trained on the T-shirt task and tested on five different tasks of 200 wrinkled clothing with low coverage. To evaluate our policy, we load tasks from the testing task datasets and then run the policy for 10 steps or until the cloth is out of observation or 10 steps end.

Metrics. In the simulation, the workspace is discretized into a grid. Each cell on the grid is either covered by cloth or not. We define a coverage metric Csim, which is the sum of the areas of all grid cells covered by the cloth: Csim = ∑Ni=1 Ai, where N is the number of cells covered by the cloth and Ai is the area of the i-th cell. To evaluate the cloth manipulation performance, we compute the coverage percentage Cpct, which is the ratio of Csim to the total area Aflat designated for the task, multiplied by 100: Cpct = Csim Aflat × 100%. Final coverage measures the coverage at the end of an episode and delta coverage is final coverage minus initial coverage. Percent positive is the proportion of actions that increased coverage.

Cloth Action Gym. We build a reinforcement learning environment Cloth Action Gym that is easy to use and extend by wrapping an existing simulation with the OpenAI Gym [Brockman et al., 2016] API. We also apply Tianshou [Weng et al., 2022]'s PPO algorithm to implement the above method. In this way, others can quickly modify the existing task environments and high-level action primitives to suit their research goals, and use any reinforcement learning frameworks that are compatible with our environment.

Baselines. We compare our method with the following baselines on various categories of clothing: Flingbot [Ha and Song, 2021], which utilizes the dynamic dual-arm Fling action primitive, leveraging coverage as a label in its value map. We employ the original model and action primitive configurations provided by Flingbot, which is trained on rectangular fabric of varying sizes. Experiments are conducted on multi-category clothing tasks. PPO means the PPO algorithm without pre-training model. Cloth Funnels [Canberk et al., 2022], a multi-action primitive method for dual-arm robots using canonicalized-alignment labels that requires shape priors. Cloth Funnels-P, Cloth Funnels, which solely employs the Pick-Place action in line with our action, is also the pretrained model of ours ClothPPO.

6 Results

Comparing with Baselines. Table.1 shows the performance comparison of various methods applied to five different types of clothing tasks. Each entry in the table shows three numbers separated by a slash representing three different performance metrics. Apart from the T-shirt Task, ClothPPO has attained superior results across all other metrics. This is because the shape of prior labels related to clothing categories used by Cloth Funnels is particularly prominent in this Task. But our ClothPPO still performs better on other types of cloth tasks than the method that requires the use of additional robotic arms to operate Fling action primitives. The most compelling evidence of its efficacy is the significant improvement it has shown when compared to the pre-training model. This clearly illustrates the effectiveness of our approach. This makes perfect sense considering our rewards are directly tied to enhance these three metrics.

Generalize to Unseen Cloth Types. The reward system in ClothFunnels relies on the alignment of cloth shapes, thereby requiring knowledge of the fabric's geometry beforehand. Consequently, optimal outcomes for different garment shapes necessitate distinct training models. However, in practical terms, this approach is highly inefficient. Conversely, ClothPPO's requisite reward is based on the coverage of the cloth, which is a universal metric applicable to all clothing categories, thereby obviating the need for training individual models for each distinct class of garments. Our experiments demonstrate that ClothPPO, even when exclusively trained on the Shirt task, exhibits excellent performance and adaptability across multiple types of clothing. Notably, ClothPPO's ability to retain high-performance metrics across varying garment types indicates a substantial capacity for generalization. This is crucial for practical applications wherein robots may be tasked to handle a wide array of garments.

Comparing Performance Before and After PPO Training. Fig. 3 demonstrates a comparison of performance on a consistent task set by both the pre-training model and ClothPPO. The presence of robotic grippers highlighted by red lines, suggests an interaction between the robotic manipulator and the cloth. Within the same task, ClothPPO displays a more significant improvement in coverage at every step. In the final stage of the pre-train model, the coverage stagnates due to the action being void. The pre-training model approaches reliance on the argmax action selection strategy, thereby limiting its exploration of other potential actions. In contrast, ClothPPO manifests a broader scope for action exploration.

Comparing ClothPPO to PPO Trained from Scratch. Fig.5 provides a comparative visualization of ClothPPO and PPO, the latter of which has been trained from scratch without pre-training. The red line, representing ClothPPO, consistently surpasses the green line, symbolizing PPO From Scratch. ClothPPO not only commences with a higher initial coverage but also exhibits an evident upward progression, indicating the productive impact of pretraining on the learning process. Conversely, the PPO From Scratch method struggles to exhibit significant advancements throughout the training. These findings suggest that employing pretraining with PPO offers a remarkable advantage for learning efficiency and end performance in challenging cloth unfolding tasks when compared to starting from scratch. Pretraining gives the model a head start and leads to more robust learning.

Reward Ablation. As Fig. 4, it is evident that different rewards influence the learning curve and the speed at which the agent achieves high rewards. The loss/entropy graph indicates the variability and uncertainty in the model's predictions, with lower values suggesting better model stability. Finally, the best coverage graph illustrates the highest coverage achieved by the agent, serving as a direct measure of the performance for each reward function. The threshold achievement reward shows a trend towards higher coverage, indicating a potentially more robust learning outcome compared to the other rewards. This could be because the rewards for over achievement have a large variance. This causes the model to struggle more and undergo severe fluctuations. On the other hand, the rewards for immediate termination are excessively challenging, making it difficult for the model to learn.

Reward Scale. As shown in Fig.6, our results indicate that the reward scale can effectively facilitate the convergence of critic networks. The blue line, which represents the scenario with reward normalization, shows a significantly smoother descent and lower variance in critic loss compared to the orange line, which indicates the scenario without reward normalization. Reward normalization leads to a more stable training process, as indicated by the less volatile and more consistent decline in loss. The normalized reward approach can converge faster to a lower loss value than the nonnormalized approach. Non-normalized reward settings experience sharp increases in losses, suggesting that wildly changing sparse rewards can produce large prediction errors in the critic model.

7 Conclusion

Our research introduces a novel framework ClothPPO that harnessing the potential of using PPO to enhance the pretrained model. This approach is aimed to surmount the initial barriers often faced in high-dimensional control tasks and solved the problem of large grasping action space in the task of folding clothes by robots. By optimizing and updating the strategy, the method can increase the robot's surface area for clothing unfolding under the soft-body manipulation task. Furthermore, our approach is scalable and holds the potential for extension to a broader range of tasks in the future.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (62102152) and sponsored by the Special Fund for Graduate International Conferences of East China Normal University.