MORL-CA: Dynamic Multi-Objective Reinforcement Learning for Chlor-Alkali Process Optimization Under Time-Varying Conditions

1 Introduction

Artificial intelligence (AI) is increasingly reshaping chemical engineering by integrating data, domain knowledge, and automation into intelligent process systems [Chakraborty et al., 2025]. Recent advances, including large language models and generative AI, have enabled data-driven process modeling, flowsheet design, and safety analysis, accelerating the transition toward adaptive and autonomous chemical manufacturing [Schweidtmann, 2024; Mann et al., 2025].

Chlor-alkali production is a foundational chemical process that supplies essential chemicals such as caustic soda, chlorine, and hydrogen to numerous downstream industries [Kermeli and Worrell, 2025]. Figure 1, adapted from [Li et al., 2021], illustrates the core working process of the electrolyzer, the key equipment for modern ion-exchange-membrane-based chlor-alkali production. Chlor-alkali production is highly energy-intensive, and each ton of alkali consumes about 2100-2600 kW·h of electricity. Therefore, energy conservation is a top priority regarding the economic benefits of chlor-alkali plants. To reserve energy, chlor-alkali production requires coordinated adjustment of multiple parameters (e.g., current, voltage, electrolyte flow, and temperature) to balance conflicting objectives: maximizing yield and current efficiency while minimizing energy consumption and extending equipment life, under strict safety and operational constraints [Ghanbarzadeh et al., 2024].

Traditional optimization approaches, including gradientbased methods and evolutionary multi-objective algorithms, typically assume fixed operating conditions and require repeated re-optimization when the environment changes [Lee et al., 2024; Silva et al., 2024; Zhu et al., 2024]. Such static formulations are poorly suited to the dynamic nature of modern chlor-alkali production, whose optimal operating strategies must evolve over time due to objective environmental change factors like membrane aging, electrode degradation, and power supply fluctuations, as well as subjective environmental change factors such as total current limitations. In contrast, reinforcement learning (RL) learns with environmental changes via dynamic policy updates, exploration-exploitation balance, and lifelong learning, proving adaptive in process optimization across fields like chemicals. [Bloor et al., 2025; Fortela et al., 2025; Devarakonda et al., 2025].

Despite this potential, RL-based multi-objective optimization for chlor-alkali process remains largely underexplored. Existing studies [Mirzazadeh et al., 2008; Kaveh et al., 2008; Ghanbarzadeh et al., 2024] primarily rely on heuristic algorithms or single-objective learning methods that fail to capture the high-dimensional, multi-objective, and time-varying characteristics of chlor-alkali production.

To address these challenges, this paper proposes MORLCA, a multi-objective RL framework for dynamic process optimization in chlor-alkali production. MORL-CA facilitates continuous policy adaptation to dynamic environments via continual online learning.

The main contributions are summarized as follows:

• Dynamic Pareto-Tracking Formulation for Industrial Process Optimization. We formulate chlor-alkali process parameter optimization as a dynamic multiobjective sequential decision problem, aiming to continuously track a time-varying Pareto set rather than repeatedly solving static optimizations.
• State-Aware Adaptive Objective Weighting for Dynamic MORL. We introduce a state-conditioned adaptive objective weighting mechanism within a multicritic actor-critic framework, enabling localized Paretoimproving policy updates under non-stationary conditions.
• An Offline-to-Online MORL Framework for LowCost Industrial Adaptation. We propose an industrialoriented MORL framework that combines offline pretraining with constrained online refinement, achieving rapid adaptation to environmental changes with limited online interactions.

2 RelatedWork

2.1 Multi-Objective Reinforcement Learning

Multi-objective RL (MORL) extends classical RL to vectorvalued rewards and aims to learn policies that balance conflicting objectives [Rodr´ıguez Soto et al., 2024]. Most MORL methods rely on scalarization or utility-based formulations to encode preferences, enabling either preference-conditioned policies or online-adjusted objective weights [Rodr´ıguez Soto et al., 2024; Shi et al., 2025].

To support gradient-based optimization, actor-critic architectures with multiple critics or policy heads have been widely adopted, while closely related multi-agent formulations assign individual objectives to separate agents and coordinate their decisions [Lu et al., 2020; Shen et al., 2022; Geng et al., 2025]. More recently, adaptive-weight (dynamicreward) mechanisms have been proposed to adjust preferences online, improving robustness under time-varying operating conditions [Zhang and Ou, 2025].

In industrial and chemical engineering applications, MORL is often combined with surrogate modeling and hybrid offline-online training to leverage historical data while ensuring safe online adaptation [Lin et al., 2024; Lu et al., 2024].

Building on these ideas, MORL-CA integrates a multicritic architecture with an adaptive-weight network to address continuous multi-objective chlor-alkali production process optimization.

2.2 Reinforcement Learning for Sequential Decision Optimization

RL has been increasingly used to reformulate optimization problems as sequential decision processes [Goldie et al., 2024]. Existing approaches can be broadly categorized into constructive methods, which build solutions step by step [Liu et al., 2024; Hottung et al., 2025], and update-based methods, which learn iterative improvement operators or parameter update policies [Goldie et al., 2024].

In chemical and process optimization, control-informed RL and offline-online deployment strategies have shown effectiveness in handling safety constraints and time-varying dynamics [Devarakonda et al., 2025; Bloor et al., 2025].

However, sequential decision formulations for chlor-alkali process parameter tuning remain largely unexplored. MORLCA addresses this gap by modeling parameter adjustment as an update-based sequential decision problem, enabling continuous refinement toward the Pareto front.

2.3 Adaptation to Non-Stationary and Safety-Constrained Settings

In dynamic chemical production (e.g., chlor-alkali), operating conditions drift due to changing current limits, equipment aging, etc. Under such non-stationarity, heuristic methods like NSGA-II are static and cannot adapt online. Multi-objective Bayesian optimization (MOBO) [Rodemann and Augustin, 2024] assumes stationarity and requires costly GP retraining to track shifts. Thus, both are less suitable for long-running industrial processes with evolving conditions.

Adaptation under non-stationary dynamics has become a central topic in RL, with meta-RL, context-conditioned policies, and continual learning enabling rapid response to distribution shifts [Marin Moreno et al., 2024; Xu and Zhu, 2024; Jackson et al., 2024]. In safety-critical industrial settings, a hybrid paradigm combining offline pretraining with conservative online fine-tuning is commonly adopted to balance safety, data efficiency, and adaptability [Chen and Luo, 2025; Zheng et al., 2024].

Motivated by these advances, MORL-CA adopts a hybrid offline-online learning framework, where historical data are used to initialize a stable policy and online interaction enables continuous adaptation to non-stationary multi-objective operating conditions.

3 The MORL-CA Framework

3.1 ProblemDefinition

We consider a multi-objective process optimization problem in an industrial chlor-alkali electrolyzer system. The primary goal is to find an optimal Pareto set that dynamically coordinates electrolyzers to achieve competing objectives, modeling it as a sequential decision-making problem for multiobjective RL.

Decision Variables and Observation Space

The system consists of M parallel electrolytic cells operating under shared electrical and operational constraints. For each cell m ∈ {1, ...,M}, there are:

• K controllable parameters, including cell current, cathode outlet temperature, etc;
• L uncontrollable but observable parameters, including product caustic concentration and product caustic temperature.

Let xm ∈ RK , z ∈ RL denote the controllable parameters of cell m and the shared uncontrollable parameters, respectively. The full system observation vector is defined as

where X denotes the feasible observation space bounded by operational limits:

Optimization Objectives

Without loss of generality, we assume there are three objectives. The objectives are modeled as continuous functions of system states:

where f1 and f2 are to be maximized, while f3 is to be minimized. To be handled uniformly as a minimization problem, we define

Given these objectives, we derive a Pareto-optimal set P ∗ with robust convergence and diversity:

Moreover, the solution must adapt iteratively: as the operating environment changes, the method should swiftly update P ∗ to reflect current constraints and conditions.

Constraints

The system is subject to a global electrical constraint on total operating current:

where Im is the operating current of cell m, and Imax the maximum total current allowed.

Equipment safety and operational feasibility impose additional constraints on the bounded domain X .

Pareto front

Reformulating as a Sequential Decision Problem

To address the time-varying nature of chlor-alkali production, the multi-objective optimization is reformulated as a continuous sequential decision process.

Starting at x0 ∈ X, the agent incrementally adjusts. At step t, it observes xt, takes action δt ∈ RMK , and then:

subject to feasibility and operational constraints.

After a finite horizon T , the final operating point is:

with the objective of guiding xT towards the Pareto-optimal set, as depicted in Figure 2. The adaptive objective weighting network influences xt's change direction.

This method converts static multi-objective optimization into a dynamic continuous-control problem, ideally suited for RL, which excels in learning adaptive policies through sequential interaction, aligning well with continuous decision spaces and time-varying industrial settings.

3.2 Framework Overview

The MORL-CA framework consists of three modules: population initialization with offline policy pretraining, online RL, and inference and solution refinement.

First, offline pretraining leverages historical data to populate an experience buffer and establish a stable initial policy.

Second, online training initializes the environment with a Latin Hypercube Sampling (LHS)-generated population. The RL agent interacts continuously, selecting actions, storing transitions, and updating its policy. A training episode covers the entire population, allowing progressive policy adaptation.

Finally, in inference, NSGA-II generates initial candidates, which the trained RL policy refines through fixed decision steps toward the Pareto front. Adjustment trajectories are aggregated, pruned using SPEA2-based density estimation, and sorted to yield the final solution set.

Figure 3 illustrates the modules and their interactions.

3.3 Population Initialization and Offline Learning

Population initialization is customized for training and inference to balance exploration and convergence. During online training, we sample the initial population with LHS across the feasible parameter domain; LHS's space-filling property ensures extensive, uniform coverage and yields diverse experiences for policy learning.

Prior to online interaction, an offline pretraining stage leverages historical production data to populate an experience buffer and initialize an initial action-selection policy. This pretraining step stabilizes early learning and facilitates online adaptation.

During inference, initialization favors convergence: when a Pareto set from the immediately prior environment is available, it serves as the seed for the initial population; only at system startup, when no such prior set is available, do we generate the initial population with NSGA-II. This strategy reduces required refinement steps and accelerates convergence under incremental environment changes.

3.4 Reinforcement-Learning Agent Architecture

We utilize a weighted adaptive actor-critic framework, adapted from Soft Actor-Critic (SAC), and extend it to incorporate multiple critics for multi-objective optimization. Given n objectives, the environment yields a reward vector at time t:

We employ n critics Qθi (s, a) (with parameters θ) and their corresponding target critics Q¯ θi . The stochastic policy πϕ(a|s) includes an entropy term regulated by α > 0 (where α = exp(η) and η is optimized).

Environment Creation

Since direct interaction with the production plant is impractical, a data-driven surrogate environment is established for training, offering state transitions and multi-objective rewards.

State, Action and Reward

Our definitions of state, action, and reward in RL are as follows:

• State st: Measured production parameters (e.g., current density, cell voltage, temperature, flow rates).
• Action at: Incremental adjustments to controllable parameters.
• Reward: Vector rt (see Equation (9)), with each component representing an optimization objective (e.g., yield, energy, current efficiency).

Multi-Critic Target and Critic Loss

For a sampled transition (st, at, rt, st+1), the action at+1 ∼ πϕ(·|st+1) is sampled and its log-probability log πϕ(at+1|st+1) is evaluated. Let α = exp(η) denote the entropy temperature, we define the soft Bellman backup for the i-th objective as

The corresponding TD-target is then given by

Stacking these targets yields ˆQt ∈ Rn . Each critic Qθi is trained to regress ˆQt,i with mean squared error:

State-Aware Adaptive Objective Weighting

To adapt RL to multi-objective optimization, we equip the actor with a state-aware adaptive objective weighting (AOW) network hψ : S → Rn , producing weights:

where wi(s) reflects the emphasis on objective i at state s. This enables each state to "exploit its strengths": states excelling in certain objectives receive higher corresponding weights, fostering focused decisions and faster convergence to the nearest Pareto region.

The AOW is learned incrementally via supervised regression. For each sampled state s, we compute normalized objectives ˆf(s), derive goodness scores g(s) = 1 - ˆf(s), and establish target weights w∗ (s) = softmax(g(s)), training hψ to regress w∗ (s).

Actor Update

Given per-objective critics Qi(s, a), we define an aggregated state-action value by linearly combining the critics with the state-dependent weights from the AOW:

where wi(s) come from hψ(s). The actor maximizes this aggregated value with an entropy regularizer. Formally, the actor loss is:

and the entropy coefficient α = exp(η) is adapted by minimizing

where ¯H is a target entropy. This formulation follows the standard maximum entropy reinforcement learning framework, adapted here to the multi-objective setting with state-aware preferences.

3.5 Online Learning and Inference

Following offline pretraining on historical data, the agent proceeds online, interacting with the live plant (or its surrogate) and accumulates transitions (s, a, r, s′ ) into an online replay buffer. Periodically, the agent refines its policy via actorcritic gradient updates on mini-batches randomly sampled from the buffer, maintaining stability inherited from offline initialization while adapting to gradual or sudden environmental changes.

Inference combines population initialization with agentguided refinement. An initial population of N candidates is refined by executing the learned policy for a predetermined, consistent number of sequential actions. For a candidate xi, the refinement trajectory proceeds as follows:

and the pooled candidate set comprises a total of S = N × steps solutions. The set is purified via: (1) non-dominated sorting to rank solutions based on their optimality and (2) SPEA2-style density pruning that favors solutions in sparse regions [Zaferani et al., 2024]. The resulted solutions constitute the final Pareto front approximation, which is then presented to the decision-makers.

This online-inference cycle facilitates ongoing policy improvement by leveraging deployment data, simultaneously generating a concise yet diverse Pareto set. This is achieved without the necessity for exhaustive re-optimization following every alteration in the environment.

4 Experiments and Results

4.1 Experimental Setup and Dynamic Protocol

We build a surrogate environment using a real industrial dataset from 8 electrolytic cells, each with 7 controllable inputs (current, cathode outlet temperature, acid dosing flow, anolyte flow, catholyte flow, brine feed flow, and catholyte header level) and 2 non-controllable observables (product caustic concentration and product caustic temperature), totaling 8 × 7 + 2 = 58 inputs. An MLP regression model maps these inputs to three objectives, serving as the interactive surrogate for RL training and inference.

The three competing objectives are: maximize daily caustic soda production (f1(x)), maximize caustic current efficiency (f2(x)), and minimize specific energy use (f3(x)).

To simulate dynamics, we vary the total-current limit Imax from 86 kA to 119 kA. This reflects industrial chlor-alkali practice, where current density is adjusted to electricity prices and grid constraints-a structured, operationally meaningful shift rather than random noise.

For each new current limit, the policy is adapted with 3 online episodes. The Pareto set is generated by NSGA-II (86 kA cold-start) or seeded from the previous Pareto set, then refined and pruned to at most 1,000 solutions.

This protocol embodies a practical workflow: swift, incremental online adjustments followed by efficient inference, avoiding full re-optimization after each change.

4.2 Correlation Analysis of State-Aware AOW with the Objective

Before comparative experiments, we verify the AOW network's correctness and efficacy to ensure it accurately reflects local objective dominance. We perform a targeted correlation analysis by randomly sampling 1,000 states from each of three distinct objective-space regions (region1 , region2 , region3 ), where ˆ f1(x), ˆ f2(x), and ˆ f3(x) are locally dominant, respectively. Each sampled state x is processed by the trained AOW network to yield w(x) = [w1, w2, w3]⊤, from which per-region averages wi are computed.

Figure 4 presents the mean weight vectors per region, revealing that states from the ˆ f1-dominant region exhibit significantly higher w1 values (similarly for w2 and w3). This indicates that the AOW network prioritizes the locally dominant objective, fostering swift convergence toward the nearest Pareto region. Such empirical evidence underscores the AOW network's state-aware weighting mechanism.

4.3 Baselines

We compare MORL-CA against representative state-of-theart evolutionary and RL baselines multi-objective problems: C3M [Sun et al., 2022], c-DPEA [Ming et al., 2021], CMDEIPCM [Liang et al., 2022], IDBEA [Sun et al., 2020], ToP [Liu and Wang, 2019], MADDPG [Wang et al., 2022], and SAC [Haarnoja et al., 2018].

All heuristic baselines are run on PlateEMO [Tian et al., 2017], with a fixed population size of 1,000 and default hyperparameters for all algorithms.

For RL baselines like MADDPG and SAC, we employ the same online adaptation as MORL-CA: agents interact with the updated environment for three episodes to refine their policies. Notably, SAC, being single-objective, uses a scalarized reward with equal weights for all objectives.

4.4 Evaluation

Metrics

We evaluate the quality of Pareto set using two domainstandard metrics: Hypervolume (HV) [Lu et al., 2024] and Inverted Generational Distance (IGD) [Wang et al., 2022].

To qualify performance stability across environmental changes, we introduce a smoothness metric for the time series of evaluation metrics. Let y = [y1, ..., yT ] be the metric sequence (HV or IGD) across T constraint scenarios. We define the smoothness metric, mean absolute first difference (MAFD), as:

A lower MAFD indicates a smoother metric curve, indicating superior dynamic adaptation. We also present the average ranking across scenarios for HV and IGD, where a lower rank is preferable.

Reference Pareto front

When optimizing parameters in the chlor-alkali process, the Pareto front lacks an analytical solution. For each scenario k, we construct an empirical reference front by aggregating solutions from all methodologies and subsequently retaining only the non-dominated points:

where S (m) k is the solution set of algorithmm under scenario k, and NDS(·) is the non-dominated sort operator. IGD and HV are computed with respect to Rk.

4.5 Results Overview

Figures 5a and 5b show HV and IGD respectively for all algorithms across varying current limits. Table 1 summarizes each method's average HV and IGD ranks, along with their MAFD values.

MORL-CA leads with the best average rankings in both IGD and HV across tests. It also records the lowest MAFD for both metrics, indicating smaller performance fluctuations as the total-current limit changes. These findings demonstrate MORL-CA's superior Pareto quality and its stable performance amid environmental shifts.

4.6 Ablation Study

We conduct two ablation studies to quantify the impacts of (1) the online learning module and (2) the density-estimation pruning used in inference. Results are summarized in Figure 6 and Table 2.

Online Learning Ablation

We disable online updates, relying solely on a pretrained policy for inference. Comparing HV and IGD against full MORL-CA under current-limit sequences, Figure 6 reveals that the algorithm with online learning (W) outperforms its counterpart without (W/o), showing smaller indicator fluctuations across environments. This confirms that brief online adaptation significantly improves both Pareto quality and stability under changing constraints.

(a) HV performance of each algorithm.

(b) IGD performance of each algorithm.

Figure 6: Ablation experiment results of online learning module.

Density-Estimation Pruning Ablation

We evaluate MORL-CA with and without SPEA2-style density-estimation pruning (W vs. W/o DE). With DE, we retain 1,000 solutions per scenario; without DE, all candidates are kept. For each scenario, we monitor average HV, IGD, and solution-set size, then compute the relative change rate for each metric after enabling DE using the following formula:

Table 2 shows that density-estimation pruning greatly reduces the solution-set size, which simplifies decision-making and speeds up metric computation. However, it also degrades HV and IGD, indicating a clear trade-off. In practice, the pruning strength should be adjusted according to specific requirements.

4.7 Analysis and Discussion

The experimental results can be summarized as follows:

• Convergence and Diversity: MORL-CA's multi-critic SAC framework with adaptive weight aggregation produces Pareto sets with superior convergence (lower IGD) and better diversity (higher HV) compared to classical MOEAs and baseline RL methods.
• Dynamic Adaptation: Upon environmental changes (new Imax), MORL-CA quickly updates with few online episodes and lightweight inference, recovering high-quality Pareto sets with smaller metric fluctuations.

In real chlor-alkali plants, environmental changes are moderate but frequent. MORL-CA's workflow-online policy adaptation plus fast re-inference from prior Pareto setsrequires far fewer environment evaluations than restarting a MOEA each time, leading to a much lower wall-clock cost for adapting to new constraints. This makes it ideal for near-real-time production adaptation.

5 Conclusion and FutureWork

This paper presents MORL-CA, a dynamic multi-objective RL framework for process optimization in ion-exchange membrane-based chlor-alkali production. By integrating state-aware adaptive objective weighting with offline-toonline learning, MORL-CA continuously adapts to timevarying conditions and facilitates localized Pareto-improving updates without repeated optimization. Experiments conducted in an environment constructed from real industrial data demonstrate its effectiveness in tracking evolving Pareto trade-offs and achieving superior solutions under dynamic environments. The framework code is publicly available at https://github.com/hoshinojyunn/morl-ca-framework.

While all validation is performed in a realistic data-driven surrogate environment due to industrial safety constraints, the proposed MORL-CA framework is designed for practical deployment. Future work will implement conservative, constraint-guaranteed online fine-tuning for physical electrolyzers in closed-loop operation within physical chlor-alkali plants. In addition, although this study focuses on chlor-alkali production, the framework can be generalized to a broader class of dynamic multi-objective industrial control tasks. Future research will also investigate theoretical guarantees and extensions to other industrial domains.

Acknowledgements

This work was supported in part by the CUG Enterprise Cooperation Project (No. 20251961419).