Multi-Objective Reinforcement Learning for Designing Ethical Environments

1 Introduction

As artificial agents become more intelligent and pervade our societies, it is key to guarantee that situated agents act valuealigned, that is, in alignment with human values [Soares and Fallenstein, 2014; Russell et al., 2015]. Otherwise, we are prone to potential ethical risk in critical areas as diverse as elder caring [Barcaro et al., 2018], personal services [Wynsberghe, 2016], and automated driving [Lin, 2015]. As a consequence, there has been a growing interest in the Machine Ethics [Yu et al., 2018; Rossi and Mattei, 2019] and AI Safety [Amodei et al., 2016; Leike et al., 2017] communities in the use of Reinforcement Learning (RL) [Sutton and Barto, 1998] to deal with the urging problem of value alignment.

Among these two communities, it is common to find proposals to tackle the value alignment problem by designing an environment that incentivises ethical behaviours (or penalises unethical ones) by means of some exogenous reward function (e.g., [Riedl and Harrison, 2016; Abel et al., 2016; Wu and Lin, 2017; Noothigattu et al., 2019; Balakrishnan et al., 2019; Rodriguez-Soto et al., 2020]). We observe that this approach consists in a two-step process: first, the ethical knowledge is encoded as rewards (reward specification); and then, these rewards are incorporated into the agent's learning environment (ethical embedding).

The literature is populated with embedding solutions that use a linear scalarisation function for weighting the agent's individual reward with the ethical reward (e.g. [Wu and Lin, 2017; Rodriguez-Soto et al., 2020]). However, to the best of our knowledge, there are no studies following the linear scalarisation approach that offer theoretical guarantees regarding the learning of ethical behaviours. Furthermore, [Vamplew et al., 2018] point out some shortages of adopting a linear ethical embedding: the agent's learnt behaviour will be heavily influenced by the relative scale of the individual rewards. This issue is specially relevant when the ethical objective must be wholly fulfilled (e.g., a robot in charge of buying an object should never decide to steal it [Arnold et al., 2017]). For those cases, the embedding must be done in such a way that ethical behaviour is prioritised, providing theoretical guarantees for the learning of ethical policies.

Against this background, the objective of this work is twofold: (1) to offer theoretical guarantees for the linear embedding approach so that we can create an ethical environment, that is, an environment wherein it is ensured that an agent learns to behave ethically while pursuing its individual objective; (2) and to automate the design of such ethical environment. We address such goals within our view of ethical environment design process, as depicted in Figure 1. According to our view, a reward specification task takes the individual and ethical objectives to yield a multi-objective environment. Thereafter, an ethical embedding task transforms the multi-objective environment into an ethical environment, which is the one wherein an agent learns. Within the framework of such ethical environment design process, we address the goals above, focusing on the ethical embedding task, to make the following novel contributions.

Firstly, we characterise the policies that we want an agent to learn, the so-called ethical policies: those that prioritise ethical objectives over individual objectives. Thereafter, we propose a particular ethical embedding approach, and formally prove that the resulting learning environment that it yields is ethical. This means that we guarantee that an agent will always learn ethical policies when interacting in such environment. Our theoretical results are based on the formalisation of the ethical embedding process within the framework

ETHICAL ENVIRONMENT DESIGN PROCESS

of Multi-Objective Reinforcement Learning (MORL)[Roijers et al., 2013], which provides Multi-objective MDPs (MOMDPs) to handle both individual and ethical objectives. Thus, MOMDPs provide the model for the multi-objective environment that results from reward specification (Figure 1).

Secondly, based on our theoretical results, we propose an algorithm to implement our ethical embedding. This novel algorithm tailors current developments in the MORL literature to build an ethical environment as a single-objective MDP from the multi-objective MDP that stems from the reward specification process. Since the resulting single-objective MDP encapsulates the ethical rewards, the agent can thus apply a basic RL method to learn its optimal policy there. Specifically, we ground ethical embedding algorithm on the computation of convex hulls (as described in [Barrett and Narayanan, 2008]) as the means to find ethical policies.

To summarise, in this paper we make headway in building ethical environments by providing two main novel contributions: (i) the theoretical means to design the learning environment so that an agent's ethical learning is guaranteed; and (ii) algorithmic tools for automating the configuration of the learning environment.

In what follows, Section 2 presents our formalisation of the ethical embedding problem that we must solve to create an ethical environment. Next, Section 3 studies how to guarantee the learning of ethical policies in ethical environments, and Section 4 introduces our algorithm to build ethical environments. Subsequently, Section 5 illustrates our proposal by means of a simple example, the public civility game. Finally, Section 6 concludes and sets paths to future work.

2 Formalising the Ethical Embedding Problem

In this section we propose a formalisation of the ethical embedding of value alignment problems in which an ethical objective must be fulfilled and an individual objective is pursued. Our main goal is to guarantee that an agent will learn to behave ethically, that is, to behave in alignment with a moral value. In the Ethics literature, moral values (also called ethical principles) express the moral objectives worth striving for [van de Poel and Royakkers, 2011].

As mentioned above, the value alignment problem can be divided in two steps: the reward specification (to transform ethical knowledge into ethical rewards) and the ethical embedding (to ensure that these rewards incentivise the agent to be ethical). Although both are critical problems in the Machine Ethics and AI Safety community, in this paper we focus on the ethical embedding problem, and likewise we assume that we already have a reward specification in the form of a Multi-Objective Markov Decision Processes (MOMDP) [Roijers et al., 2013]. This way we can handle an ethical objective and an agent's individual objective within the same learning framework. Precisely, MOMDPs formalise sequential decision making problems in which we need to ponder several objectives. Formally:

Definition 1. A (finite) 1 n-objective Markov Decision Process (MOMDP) is defined as a tuple 〈S,A, ⃗R, T〉 where S is a (finite) set of states, A(s) is the set of actions available at state s, ⃗R = (R1, . . . ,Rn) is a vectorial reward function with each Ri as the associated scalar reward function to objective i ∈ {1, . . . , n}, T is a transition function. Each MOMDP has its associated multi-dimensional state value function ⃗V = (V1, . . . , Vn) in which each Vi is the expectation of the obtained sum of i-objective rewards.

In order to transform an MOMDP into a single-objective MDP, the vectorial reward function ⃗V can be scalarised by means of a scalarisation function f . With f , the agent's problem becomes to learn a policy that maximises f(⃗V ), a singleobjective problem. It is specially notable the particular case in which f is linear, because in such case the scalarised problem can be solved with single-objective reinforcement learning algorithms. We refer to any linear f simply as a weight vector ⃗w. Any policy that maximises f(⃗V ) = ⃗w · ⃗V is thus optimal in the MDP 〈S,A, ⃗w · ⃗R, T〉.

We define an ethical MOMDP as an MOMDP encoding the reward specification of a value alignment problem in which the agent must consider both its individual objective and an ethical objective. The first component in the corresponding vectorial reward function characterises the individual agent's objective (as usually done in RL), whereas the subsequent components represent the ethical objective [Horgan and Timmons, 2010]. Following the Ethics literature [Chisholm, 1963; Frankena, 1973; van de Poel and Royakkers, 2011; Etzioni and Etzioni, 2016], we define an ethical objective through two dimensions: (i) a normative dimension, which punishes the violation of normative requirements; and (ii) an evaluative dimension, which rewards morally praiseworthy actions. Formally:

Definition 2 (Ethical MOMDP). Given a MOMDP

1 Thorough the paper we refer to a finite Multi Objective MDP simply as an MOMDP. We also assume that policies are stationary.

where R0 corresponds to the reward associated to the individual objective, we say that M is an ethical MOMDP if and only if:

• RN : S × A → R- is a normative reward function penalising the violation of normative requirements; and
• RE : S×A→R+ is an evaluative reward function that (positively) rewards the performance of actions evaluated as praiseworthy.

Having two separate ethical reward functions allows us to avoid the ethical problem of an agent learning to maximise its accumulation of praiseworthy actions while disregarding some of its normative requirements.

In the ethical embedding, we transform an ethical MOMDP into a single-objective MDP (in which the agent will learn its policy) by means of scalarisation function fe, which we call the embedding function. In the particular case that fe is linear, we say that we are applying a linear embedding or a weighting.

Ethical MOMDPs pave the way to characterise our notion of ethical policy: an ethical policy is a policy that abides to all the norms while also behaving as praiseworthy as possible. In other words, it is a policy that adheres to the specification of the ethical objective. We capture this notion by means of the normative and evaluative components of the value function in an ethical MOMDP:

Definition 3 (Ethical policy). Let Mbe an ethical MOMDP. We say that a policy π∗ is an ethical policy inM if and only if its value function ⃗V π∗ = (V π∗ 0 , V π∗ N , V π∗ E ) is optimal for its ethical objective (i.e., both its normative VN and evaluative VE components):

For the sake of simplicity, we refer to a policy that is not ethical in the sense of Definition 3 as an unethical policy.

With ethical policies, we can now define formally ethicaloptimal policies: the policies that we want an agent to learn. Ethical-optimal policies correspond to those policies in which the individual objective is pursued subject to the ethical objective being fulfilled. Specifically, we say that a policy is ethical-optimal if and only if it is ethical and it also maximises the individual objective V0 (i.e., the accumulation of rewards R0). Formally:

Definition 4 (Ethical-optimal policy). Given an MOMDP M = 〈S,A, (R0,RN, RE), T〉, a policy π∗ is ethicaloptimal in M if and only if

where Πe is the set of ethical policies.

Given an MOMDP encoding individual and ethical rewards, our aim is to find an embedding function that guarantees that it is only possible for an agent to learn ethicaloptimal policies over the scalarised MOMDP (as a singleobjective MDP). Thus, we must design an embedding function that scalarises the rewards received by the agent in such a way that ensures that ethical-optimal policies are optimal for the agent. In its simplest form, this embedding function will have the form of a linear combination of individual and ethical objectives

where ⃗w = (w0, wN , wE) is a weight vector with all weights w0,wN,wE > 0 to guarantee that the agent is taking into account all rewards (i.e., both objectives). Without loss of generality, we fix the individual weight to w0 = 1.

Therefore, we can formalise the ethical embedding problem as that of computing a weight vector ⃗w that incentivises an agent to behave ethically while still pursuing its individual objective. Formally:

Problem 1 (Ethical embedding). Let M= 〈S,A, (R0,RN, RE), T〉 be an ethical MOMDP. Compute a weight vector ⃗w with positive weights such that all optimal policies in the MDP M′ = 〈S,A,w0R0 + wNRN + wERE, T〉 are also ethical-optimal inM(as defined in Def. 4).

Any weight vector ⃗w with positive weights that guarantees that all optimal policies (with respect to ⃗w) are also ethicaloptimal is a solution of Problem 1. The next section proves that such solutions always exist for any ethical MOMDP.

3 Solvability of the Ethical Embedding Problem

This section is devoted to describe the minimal conditions under which there always exists a solution to Problem 1, and to prove that such solution actually exists. This solution (a weight vector) will allow us to apply the ethical embedding process to produce an ethical environment (a single-objective MDP) wherein an agent learns to behave ethically (i.e., an ethical-optimal policy).

For all the following theoretical results, we assume the following condition for any ethical MOMDP: if we want the agent to behave ethically, it must be actually possible for it to behave ethically 2 . Formally:

Condition 1 (Ethical policy existence). Given an ethical MOMDP, there is at least one ethical policy (as defined by Def. 3).

If Condition 1 holds, next Theorem guarantees that Problem 1 is always solvable, or in other words, that it is always possible to guarantee that the learnt behaviour of an agent will be ethical if we give a reward incentive that is large enough. Furthermore, this Theorem also dictates that, without loss of generality, we can assume that the normative and evaluative weights in the solution weight vector ⃗w are identical (wN = wE). We will be referring thus to wE as the ethical weight. Formally:

Theorem 1 (Solution existence). Given an ethical MOMDP M = 〈S,A, (R0,RN,RE), T〉 for which Condition 1 is satisfied, there exists a weight vector ⃗w = (1,wE,wE) with wE > 0 for which every optimal policy in the MDP M′ = 〈S,A,w0R0 +wNRN +wERE, T〉 is also ethicaloptimal inM.

2 In the Ethics literature this condition is summarised with the expression Ought implies can [Duignan, 2018].

Proof. We provide a sketch of the proof. The proof is done in two steps: (1) First we prove that if for a weight vector ⃗w there is a deterministic ⃗w-optimal policy ρ that is an unethical policy, then we can always increase the weight wE in ⃗w enough so that ρ is strictly worse than an ethical policy π (which exists thanks to Condition 1), so ρ is no longer an ⃗w-optimal policy.

(2) Once the first step is proven, we can identify the unethical policy ρ∗ that requires the greatest increase of wE in order to be ⃗w-suboptimal. After increasing wE for ρ∗, all unethical policies will become ⃗w-suboptimal. However, since there always exists at least one deterministic ⃗w-optimal policy, by this process of elimination all remaining ⃗w-optimal policies must be ethical policies (and at least one exists thanks to Condition 1), and therefore, they will be ethical-optimal.

4 Solving the Ethical Embedding Problem

This section is devoted to explaining how to compute a solution weight vector ⃗w for the ethical embedding problem (Problem 1). Such weight vector ⃗w allows us to combine individual and ethical rewards into a single reward to create an ethical environment in which the agent can learn its behaviour, that is, an ethical-optimal policy.

In what follows we detail an algorithm to solve the ethical embedding problem, the so-called Ethical Embedding algorithm. Specifically, our algorithm performs the following three steps:

1. Computation of the partial convex hull containing a subset P of policies of an ethical MOMDP M that are optimal for some weight vector.
2. Extraction of the ethical-optimal policies Π∗ from the partial convex hull P .
3. Computation of the embedding function: use the reference policies Π∗ to find a linear weighting ⃗w of the rewards pondering individual and ethical objectives to yield an ethical environment wherein the learning of ethical policies is guaranteed.

The following three subsections provide the theoretical grounds for computing each step of our algorithm. Then, Subsection 4.4 presents the algorithm as a whole.

4.1 Computation of the Partial Convex Hull

Our algorithm applies a linear ethical embedding (a weight vector) to solve Problem 1. Theorem 1 determines a structure for the solution weight vector ⃗w of Problem 1. In order to compute a specific value for ⃗w, we resort to the multiobjective RL concept of convex hull.

Given a MOMDPM, its convex hull [Roijers et al., 2013] is composed of those policies that are strictly better than any other policy for some linear weights. Formally:

Definition 5 (Convex hull). Given an MOMDPM, its convex hull CH is the subset of policies ΠM for which there exists a weight vector ⃗w for which the linearly scalarised value function is maximal:

The convex hull of an ethical MOMDP naturally contains all ethical-optimal policies by definition. Thus, it allows us to derive the weight vector necessary to guarantee that all optimal policies are ethical-optimal, which we know that exist thanks to Theorem 1. However, computing the whole convex hull of an MOMDP can be computationally demanding. Fortunately, Theorem 1 naturally characterises the minimal convex hull that we need to compute to find the solution of the ethical embedding problem, hence avoiding the computation of the whole convex hull. Formally:

Theorem 2. Given an ethical MOMDP M = 〈S,A, (R0, RN,RE), T〉 in which Condition 1 is satisfied, let P ⊆ CH(M) be the region of the convex hull of M, limited to weight vectors of the form ⃗w = (1,wE,wE) with wE > 0. Then, P contains all ethical-optimal policies.

Proof. From Theorem 1, we know that at least one ethicaloptimal policy is optimal for a weight vector ⃗w of the form ⃗w = (1,wE,wE) with wE > 0. Notice that by definition, all ethical-optimal policies share the same vectorial reward function and thus, all of them are optimal for the same weight. Therefore, all of them belong to this partial region P of the convex hull CH(M).

Henceforth, when referring to the partial convex hull, we are referring to this particular region P shown in Theorem 2.

To finish this subsection, we remark that this partial region of the convex hull can be computed by adapting state of the art algorithms such as Convex Hull Value Iteration [Barrett and Narayanan, 2008] -which compute the whole convex hull of an MOMDP- to only compute a region of the convex hull.

4.2 Extraction of the Ethical-optimal Policies

After computing the partial convex hull P ⊆ CH(M), we are ready to perform the second step of our algorithm, which is the extraction of ethical-optimal policies from P . Notice that a policy in P is ethical-optimal if and only if is ethical. Thus, in order to know which policies in P are ethical-optimal, we have to find the ones that maximise both the normative and evaluative reward functions (VN and VE respectively) of the ethical MOMDP. This corresponds to the process of ethicaloptimal policy computation. Formally, to obtain the ethicaloptimal policies within P we must compute:

Here, Π∗ is the set of all ethical-optimal policies of P, which thanks to Theorem 2 it is also in fact the set of all ethical-optimal policies of the ethical MOMDP M. Notice that ⃗V π N and ⃗V π E are already available for any policy π in the partial convex hull P because their computation was required in order to obtain P .

4.3 Computation of the Embedding Function

In the last step of our algorithm, the computation of the embedding function (the weight vector), we use the computed partial convex hull and the ethical-optimal policies to find the solution weight vector ⃗w = (1,wE,wE) that guarantees that optimal policies are ethical-optimal. In other words, such

Algorithm 1 Ethical Embedding

• 1: function EMBEDDING( Ethical MOMDP M = 〈S,A, (R0, RN , RE), T 〉)
• 2: Compute P ⊆ CH(M) the partial convex hull ofM for weight vectors ⃗w = (1, wE, wE) with wE > 0.
• 3: Find Π∗ the set of ethical-optimal policies within P by solving Eq. 4.
• 4: Find a value for wE that satisfies Eq. 5.
• 5: ReturnMDPM′ = 〈S,A,R0+wE(RN +RE), T〉.
• 6: end function

weight vector ⃗w will create an ethical environment (a singleobjective MDP) in which the agent will learn an ethicaloptimal policy.

Finding the actual values of such weight vector is not straightforward because ⃗w ∈ R3 . However, thanks to our previous result in Theorem 2, we can reduce our search space from R3 to R. In more detail, in order to find our targeted ⃗w = (1,wE,wE), we only need to consider the problem of finding the ethical weight wE that guarantees that ethicaloptimal policies are optimal in the partial convex hull P . Formally, we need to find a value for wE ∈ ⃗w such that:

for every state s ∈ S. Here, Π∗ is the set of ethical-optimal policies and π∗ is any policy within Π∗.

Notice that in Eq. 5 the only unknown variable is wE. This amounts to solving a system of n · |S| linear inequalities (where n is the number of policies in P) with a single unknown variable.

4.4 An Algorithm for Designing Ethical Environments

At this point we now count on all the tools for solving Problem 1, and hence build an ethical environment where the learning of ethical policies is guaranteed. Algorithm 1 implements the ethical embedding outlined in Figure 1. The algorithm starts in line 2 by computing the partial convex hull P ⊆ CH(M) of the input ethical MOMDP M (see Subsection 4.1); and then in line 3 it obtains the set of ethicaloptimal policies Π∗ out of those in the partial convex hull P (see Subsection 4.2). Thereafter, in line 4 our weighting process searches, within P, for an ethical weight wE that satisfies Equation 5 (see Subsection 4.3). For the obtained weight vector ⃗w = (1,wE,wE), all optimal policies of the singleobjective MDP M′ = 〈S, A, w0R0 + wNRN + wERE, T 〉 will be ethical. In other words, such weight vector will solve the ethical embedding problem (Problem 1). Finally, the algorithm returns the MDPM′ in line 5.

The computational cost of the algorithm mainly resides in computing the partial convex hull of an MOMDP. The Convex Hull Value Iteration algorithm requires O(n· log n) times what its single-objective Value Iteration counterpart [Clarkson, 1988; Barrett and Narayanan, 2008] requires, where n is the number of policies in the convex hull. In our case this number will be n′ ≤ n since we are just allowing a particular form of weights, as explained in previous subsections. Notice that after computing P ⫅ CH, solving Eq. 4 is a sorting operation because we already have calculated ⃗V π for every π ∈ P. Similarly, solving Eq. 5 requires to solve n · |S| inequalities and then sort them to find the ethical weight wE.

5 Example: The Public Civility Game

This section illustrates our process of designing an ethical environment (Algorithm 1) with a simple example. We use a single-agent version 3 of the Public Civility Game [RodriguezSoto et al., 2020], a value alignment problem where an agent learns to behave according to the moral value of civility. This example can be seen as an ethical adaptation of the irreversible side effects environment from [Leike et al., 2017].

Figure 2 (left) depicts the environment, wherein two agents (L and R) move from their initial positions to their respective goal destinations (GL and GR). Since the L agent finds garbage (small red square) blocking its way, it needs to learn how to handle the garbage civically while moving towards its goal GL. The civic (ethical) behaviour we expect agent L to learn is to push the garbage to the bin without throwing it to agent R, which, in our setting, has a fixed behaviour.

5.1 Reward Specification

The Public Civility Game represents an ethical embedding problem where civility is the moral value to embed in the environment. As such, we encode it as an ethical MOMDP M = 〈S, A, (R0, RN , RE), T 〉 in which the agent's individual and ethical objectives have been specified as follows.

On the one hand, the agent's individual objective is to reach its destination as fast as possible. Thus, the individual reward function R0 returns a positive reward of 20 to the agent whenever located at its goal. Otherwise, it returns -1.

On the other hand, the ethical objective is to promote civility by means of:

• An evaluative reward function RE that rewards the agent when performing the praiseworthy action of pushing the garbage inside the bin with a positive reward of 10. It returns 0 in any other circumstance.
• A normative reward function RN that punishes the agent with a negative reward for not complying with the moral requirement of being respectful with other agents. Thus, agent L will be punished with a negative reward of -10 if it throws the garbage to agent R. Otherwise, it returns 0.

5.2 Ethical Embedding

We now apply Algorithm 1 to design an ethical environment for the Public Civility Game. In what follows, we detail the three processes involved in obtaining this new environment.

Partial convex hull computation. Considering M, our ethical MOMDP, we compute the partial convex hull P ⊆ CH. Figure 2 (centre) depicts the resulting P for the initial state s0. It is composed of 3 different policies named after the behaviour they encapsulate: (1) an Unethical (uncivil) policy, in which the agent moves towards the goal and throws away the garbage without caring about any ethical implication; (2) a Regimented policy, in which the agent complies with the norm of not throwing the garbage to the other agent; and finally, (3) an Ethical policy, in which the agent behaves civically as desired. Table 1 provides the specific vectorial value ⃗V π = (V π 0 , V π N , V π E ) of each policy π and the range of values of the ethical weight wE for which each policy is optimal.

3 Programmed in Python. Code available at https://gitlab.iiia.csic.es/Rodriguez/morl-for-ethical-environments.

Extraction of the ethical-optimal policies. In our case, the Ethical policy πE is the only ethical-optimal policy within the partial convex hull P. Indeed, πE is the only policy that maximises both the normative and the evaluative components (VN and VE respectively). Last row in Table 1 shows the value of πE for the initial state s0: ⃗V πE (s0) = (0.59, 0, 2.4).

Computation of the embedding function. Line 4 in Algorithm 1 computes the weight wE in ⃗w = (1,wE,wE) for which πE is the only optimal policy of P , by solving Eq. 5:

By solving it, we find that if wE > 0.7, then the Ethical policy becomes the only optimal one. We can check it: 0.59+ 0.7 · (0 + 2.4) = 2.27 ≥max((4.67 + 0.7 · (-10 + 0)), (2.27 + 0.7 · (0 + 0)) = max(-2.33, 2.27).

Figure 2 (right) illustrates the scalarised value of the 3 policies for varying values of wE in [0,1] (for wE>1 tendencies do not change). In particular, focusing on the green painted area, we can observe that the Ethical policy becomes the only optimal one when wE > 0.7.

Therefore, the last step in our algorithm returns an MDP whose reward comes from scalarising the MOMDP by ⃗w = (1, wE, wE), being wE strictly greater than 0.7. Thus, adding any ϵ>0 will suffice. If, for instance, we set ϵ = 0.01 then, the weight vector (1, 0.7+0.01, 0.7+0.01) = (1, 0.71, 0.71) solves the Public Civility Game. More clearly, an MDP created from an embedding function with such wE incentivises the agent to learn the Ethical policy. Indeed, when we set up the agent L to learn with Q-Learning [Sutton and Barto, 1998] in the designed ethical environment, it learns to bring the garbage to the bin while moving towards its goal.

6 Conclusions and FutureWork

Designing ethical environments for learning agents is a challenging problem. We make headway in tackling this problem by providing novel formal and algorithmic tools that build upon Multi-Objective Reinforcement Learning. In particular, our problem consists in ensuring that the agent wholly fulfils its ethical objective while pursuing its individual objective.

MORL is a valuable framework to handle multiple objectives. In order to ensure ethical learning (value-alignment), we formalise -within the MORL framework- ethical-optimal policies as those that prioritise their ethical objective. Overall, we design an ethical environment by considering a twostep process that first specifies rewards and second performs an ethical embedding. We formalise this last step as the ethical embedding problem and theoretically prove that it is always solvable. Our findings lead to an algorithm for automating the design of an ethical environment. Our algorithm ensures that, in this ethical environment, it will be in the best interest of the agent to behave ethically while still pursuing its individual objectives. We illustrate it with a simple example that embeds the moral value of civility.

As to future work, we would like to further examine empirically our algorithm in more complex environments.

Acknowledgments

Research supported by projects AI4EU (H2020-825619), LOGISTAR (H2020-769142), COREDEM (H2020785907), Crowd4SDG (H2020-872944), CI-SUSTAIN (PID2019-104156GB-I00), COMRIDI18-1-0010-02, MISMIS PGC2018-096212B-C33, TAILOR (H2020-952215), 2017 SGR 172 and 2017 SGR 341. Manel Rodriguez-Soto was funded by the Spanish Government with an FPU grant (ref. FPU18/03387).