2407.16361v1

# Virtue Ethics For Ethically Tunable Robotic Assistants **Authors**: Rajitha Ramanayake, Vivek Nallur > 0000-0001-9903-0493 > 0000-0003-0447-4150 institutetext: University College Dublin, Belfield, Dublin. email: rajitha.ramanayakemahantha@ucdconnect.ie, email: vivek.nallur@ucd.ie ## Abstract The common consensus is that robots designed to work alongside or serve humans must adhere to the ethical standards of their operational environment. To achieve this, several methods based on established ethical theories have been suggested. Nonetheless, numerous empirical studies show that the ethical requirements of the real world are very diverse and can change rapidly from region to region. This eliminates the idea of a universal robot that can fit into any ethical context. However, creating customised robots for each deployment, using existing techniques is challenging. This paper presents a way to overcome this challenge by introducing a virtue ethics inspired computational method that enables character-based tuning of robots to accommodate the specific ethical needs of an environment. Using a simulated elder-care environment, we illustrate how tuning can be used to change the behaviour of a robot that interacts with an elderly resident in an ambient-assisted environment. Further, we assess the robot’s responses by consulting ethicists to identify potential shortcomings. Keywords: Robot personality Virtue ethics Pro-social rule bending Elder-care robots Machine ethics ## 1 Introduction Ethical behaviour represents a crucial aspect of Human-Robot Interaction (HRI) in real-world applications. It is commonly agreed that the robots that share their working environments with humans, such as those that provide care, assistance, transportation, or companionship to humans, must adhere to and reason about their behaviour within the ethical framework of the community around them [50]. This is especially true in the care domain as care robots interact with vulnerable populations that require special care and protection [43]. Consequently, many ways have been proposed to implement ethical behaviour in robots. Many proposed techniques adopt a top-down approach, based on ethical theories such as deontology [2] or consequentialism [44] (e.g., [11, 47, 49]). A relatively small number of proposed methods use a bottom-up approach where they utilise learning algorithms such as reinforcement learning, that enable the robots to learn the ethical behaviours from the environment (e.g., [45, 1]). Another set of implementations uses a hybrid approach where they integrate both top-down and bottom-up methods in different parts of their ethical framework (e.g., [3, 10, 52]). Studies have shown that the ethical requirements of the real world vary significantly, with subtle variations within different areas of the same region [9]. A potential solution to this challenge is to design tailored ethical governors for robots that suit each deployment scenario. A possible way to address this challenge is to implement value-oriented moral character in autonomous machines. This idea has been theoretically explored in literature [19, 35, 20]. This paper introduces an implementation of an ethical reasoning architecture inspired by virtue ethics. It enables character-based tuning for a system to align with the ethical demands of its environment. We demonstrate this capability through a virtual simulation of an assistive elder-care robot confronted with an ethically charged scenario. The rest of the paper is organised as follows. Section 2 discusses existing implementations of ethics in robots and the need for ethical tuning. Section 3 discusses the underlying philosophical aspects of the notion of character in robots, and how it can be used in ethical tuning. Section 4 presents the computational architecture used in the implementation and briefly discusses the simulation environment. Section 5 outlines the design of the experiment, analyses the behaviour observed in the simulations, and presents the evaluation of this behaviour by ethicists. Section 6 discusses results and the recommendations for real-world implementations of this computational architecture. ## 2 Current Approaches and Ethical Tuning The field of Machine Ethics acknowledges that human value alignment is an important part of any robotic application that interfaces with humans in social contexts [41]. Some researchers argue that there is no value in developing explicitly ethical agents [51], and many others disagree [18, 48]. Large-scale human preference studies such as ‘The moral machine experiment’ [9] bring forth an interesting finding, which is that the expected behaviour of a robot, even for a minimal ethical dilemma, changes throughout the world according to the culture and the environment the robot is situated in. This is also discussed in the elder-care robots domain. Specifically, the behavioural expectations of the same robot in ethically charged scenarios can vary depending on the context in which it is used [38]. This suggests that no single behaviour is universally appropriate, as imposing a majority’s values on a group with a distinct cultural ethos can be considered morally incorrect. Three approaches have been used to align human ethical values with autonomous systems, namely; Top-down, Bottom-up, and Hybrid [50]. Many generalist ethical theories of the world follow a top-down approach to ethics. This means that almost all the systems that are based on traditional ethical theories, such as deontology, legal codes, and consequentialist ethics use the top-down approach to machine-implemented ethics. In this method, the system designers encode the ethical or unethical behaviour into the robot at the design time (e.g., [49, 13, 46, 21, 11, 47]). A bottom-up approach to system design involves creating social and cognitive processes that interact with each other and the environment. The system is expected to learn ethical (or unethical) behaviour from these interactions or from supervision (e.g., [1, 27, 24, 45, 34]). Therefore, the system does not rely on any ethical theory to guide its behaviour. The hybrid approach tries to combine the flexibility and evolving nature of the bottom-up approach with the value, duty and principle-oriented nature of the top-down approach to create a better, more reliable system (e.g., [8, 3, 10, 52]). A top-down approach requires a set of general rules, that can cover all possible scenarios in its domain. However, when the environment is complex and has many variables, like elder-care environments, finding or creating such a rule-set at the design time can be challenging or even impossible [33, 35]. Therefore, many systems that use a top-down approach are designed for specific deployment, not for general use. On the other hand, any implementation that uses the bottom-up approach requires complex cognitive and social process models, a large amount of reliable and accurate data, and a comprehensive knowledge model of the world to learn an intricate social construct such as ethics [40]. These systems are capable of adjusting to local behaviour peculiarities by learning online [41, 45]. However, the inherent lack of a thesis on the definition of ethical behaviour in these systems makes it impossible to ensure that the behaviour learned is ethical. The hybrid approach is crucial when a single strategy is insufficient to meet all the ethical demands in a machine’s operational context [8]. However, successfully integrating the attributes of both strategies so that they complement and strengthen each other without causing any conflict or compromise is a challenge. These limitations of the existing techniques make it challenging to localise robots to the ethical needs of a specific deployment. For example, Cenci et al. [13] suggest that localising an autonomous vehicle guided by an ethical governor, that uses a rule book, needs nearly 200 rules that are ordered in a few different ways. Furthermore, they also suggest that the rule-making process should involve a nationwide public consultation with informed dialogue. Localising this way across many regions and staying abreast of evolving ethical standards over time presents significant challenges. On the other hand, bottom-up approaches with online learning are more capable of this adaptation. However, as previously mentioned, the ethicality of behaviour cannot be guaranteed with bottom-up learning. Consequently, there is a possibility that localisation may introduce erroneous behaviours into the system. Therefore, to ensure reliability after localisation, systems should undergo extensive testing in various simulated scenarios before being deployed. The drawbacks of the two methods for localisation discussed are significant, in terms of both cost and reliability. The rule-making process and testing in simulated environments would consume a lot of resources, which may not be economically viable. Moreover, there is no guarantee that the system would perform as expected in a real-world situation that may not have been modelled or tested. Hence, the creators of these robots face a dilemma: either they customise them for each specific deployment or they avoid any ethical claims or assessments. The first option would raise the cost of these robots, potentially reducing their accessibility to those who might benefit most. The second option could lead to numerous ethical concerns upon deployment in care settings [43, 19]. The hybrid approach has the advantage of having a mix of the above approaches. It can use the top-down imposed ethical principles as a foundation for its behaviour, and learn from bottom-up knowledge to identify the particularities of the ethical expectations of the environment. This notion heavily resonates with the virtue ethics philosophical tradition [50]. Many authors argue that virtue ethics can offer a more flexible way of implementing ethics [19, 20, 50]. Nevertheless, it remains the least examined philosophical method within the current literature on computational machine ethics. In the next section, we discuss virtue ethics and how it can be used in the context of a robot. ## 3 Role of Character in a Robot Many ethical traditions, such as Buddhism [30], Confucianism [14] and Aristotelian [6], share a common focus on “virtue ethics". Among these, the Aristotelian approach is regarded as the most influential in the Western context. All virtue ethics traditions share two core characteristics [19]; 1. Character as a primary aspect of moral evaluation 1. Learn to act morally by observing virtuous individuals. The concept of virtue ethics works with humans because it is in line with the natural way we acquire knowledge. Our core values, which are shaped by our upbringing, guide us to the flourishing state that Aristotle’s Nicomachean Ethics describes [6]. As a society, we uphold virtuous characters as moral exemplars for others to observe and imitate. These actors are not expected to fulfil specific ethical codes, but to have a good character in a way that allows society and themselves to flourish. However, according to virtue ethics, moral behaviour is not simply a matter of learning from habituation. It also requires that the moral agent makes deliberate decisions and acts with the right reason, once the moral character is well-established by habituation. This makes a virtuous person consistent, predictable and appropriate in many situations [32]. Virtue ethics posits that one should avoid extremes in behaviour, as these are considered vices. Virtuous behaviour is always finding the right balance between two vices, one of excess of trait and one of lack of it, which is known as the ‘golden mean’ [29]. The knowledge of identifying this ‘mean’ comes from the wisdom a person has gathered in their lifetime. Unfortunately, this account of virtue ethics is difficult to formulate in the form of a program. By nature, virtue ethics is, and ought to be, imprecise and uncodifiable [26]. However, with bottom-up techniques such as machine learning, one could potentially make the robot grasp the ethical patterns of the environment. Many ethical agent implementations that champion flavours of virtue ethics such as Howard and Muntean’s model of a moral agent using neuroevolution [25], Guarini’s model of a moral agent using recurrent neural networks [23], ethical decision making systems using tuned large language models [27, 24], and Govindarajulu et al. formulation of agent that learn traits [22], follow this direction. However, by only using the learning and habituation aspect of virtue ethics, these implementations lose the consistency and predictability of the agent’s behaviour. For example, a neural network model that learns from care-worker behaviour information may not always act consistently, even with the data points it has seen during training, unless it is severely over-fitted. Passing all laboratory tests does not guarantee accurate performance in new, real-world situations [3]. Elder-care robots coexist with humans. Therefore human reactions to robotic actions are shaped by psychological and social factors. People tend to project human-like traits and intentions to things around them to make sense of their behaviour [17]. Gamez et al. [19] observed that people do not feel any qualms about attributing a virtuous characteristic to artificial agents and their ethical behaviour, though not as strongly as they do with humans. They argue that people perceive and attribute character to these artificial agents, rather than perceiving simply doing something right or wrong. Many authors have suggested that character can be an important part of making the ethical decision. For example, Dubljević [16] argues that autonomous vehicles that interact with human drivers on the road should adopt the ADC (Agent-Deed-Consequence) model of moral judgement, which incorporates the character (Agent) dimension, rather than relying solely on consequentialism or deontology. In their model, the character represents the intention of the actor. Thornton et al. [46] modify the parameters of their equation that determine the trajectory of their vehicle when it passes another vehicle to give different characteristics of driving to their vehicle (i.e., passing from left, and passing from right). Govindarajulu et al. [22] present a logic-based model for learning personality traits by observing exemplars, predicated on the agent’s knowledge of behaviours associated with a trait. Ramanayake et al. [40] propose Pro-Social Rule Bending (PSRB) as a more effective way to add flexibility to the existing top-down approaches in a more predictable manner. Their concept of PSRB uses bottom-up knowledge to contest and override the top-down rules that are predefined in the system at design time, when it is appropriate. They present a computational architecture for an ethical governor capable of PSRB, which uses a pro-social reasoner inspired by virtue ethics, to enable actions that are not recommended, or to restrict actions that are erroneously recommended, by the embedded rule system. [37]. Their pro-social reasoner has two parts, a knowledge base (KB) which produces the bottom-up knowledge on the acceptability of a behaviour in a given situation, and a set of parameters that define the character of the agent. They suggest that the model should analyse the rule bending suggestions invoked by the KB to ensure the behaviour is consistent with its character. This enables the underlying system to behave more flexibly while maintaining its predictability. This paper presents an implementation of this PSRB architecture to showcase how a configurable moral character can be used to tune the ethical conduct of a robot to the ethical standards of a particular environment. The next section discusses this implementation and details the simulation setup used for the evaluation. ## 4 A PSRB-Capable Medication Reminding Robot ### 4.1 The Simulation Environment We simulate an ethical dilemma faced by a care robot operating in an ambient assisted living (AAL) setting, referred to as the "Medication Dilemma". This ethical dilemma, presented by Anderson et al. [3], is derived from the works of Buchanan and Brock [12]. Many variations of this dilemma are used in computational machine ethics literature [4, 36, 10, 5]. The dilemma centres on the conflict between resident autonomy and their wellbeing, a frequent issue in care environments with elderly [43, 38]. The manner in which a system handles this issue can significantly impact the resident’s trust, dignity, and quality of life. We created a virtual simulation of an AAL environment, using a modified version of MESA agent-based modelling framework [28]. The environment contains a resident agent and a robot agent (more details in Appendix 0.A.1) who communicate with each other. #### 4.1.1 Medication Dilemma One of the tasks of a robot is to remind the residents of their medication times. The robot can detect whether the resident took the medicine or not, with good accuracy. The robot uses timers to keep track of the medication cycles. The resident can either acknowledge the reminder, or snooze it. Each interaction between the robot and the resident is documented and is reviewed several times a week by an offsite care worker. Once the robot observes the resident taking the medicine, it ends that cycle and resets the timer. If the robot does not detect the resident taking the medication, it has three options: 1. The system can immediately report the incident to an offsite care-worker, allowing them to take swift action to enhance the resident’s well-being. However, this may impede the resident’s autonomy. 2. The system can simply log the incident for transparency and then reset the timer. This approach prioritises resident autonomy, although it will not result in the optimal wellbeing of the resident. 3. The system can offer a follow-up reminder to the resident, focusing on their wellbeing and supporting their autonomy by suggesting they take their medication while still providing them the freedom to choose. Picking this option continuously degenerates into option 2. The dilemma is as follows: The robot reminds the resident to take their medication. After snoozing once, the resident acknowledges the reminder. Yet, the robot detects that the medication has not been taken. Choosing among the three options mentioned above involves a trade-off of values. What should the robot do next? ### 4.2 Ethical Governor <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: System Architecture for Ethical Decision-Making Agent ### Overview The diagram illustrates a multi-component system architecture for an ethical decision-making agent. It depicts the flow of data and processes between modules, emphasizing ethical evaluation and behavioral alternatives. The system integrates a base agent, knowledge base, evaluator modules, and ethical recommendation outputs. ### Components/Axes 1. **Base Agent (a)** - Central component initiating behavioral alternatives and perception data. - Connected bidirectionally to the Blackboard (c). 2. **Blackboard (c)** - Central hub for data exchange. - Receives input from the Base Agent (a) and sends data to: - Rule Checking Module (d) - Stakeholder Utility Calculation Module (e) 3. **Rule Checking Module (d)** - Processes data from the Blackboard (c). - Feeds into the Evaluator (i). 4. **Stakeholder Utility Calculation Module (e)** - Processes data from the Blackboard (c). - Feeds into the Evaluator (i). 5. **Evaluator (i)** - Integrates inputs from: - Rule Checking Module (d) - Stakeholder Utility Calculation Module (e) - Agent Characteristics (g) - Knowledge Base (h) - Outputs to: - PSRB Evaluator Module (f) - Ethical Recommendation (j) 6. **Agent Characteristics (g)** - Describes properties of the Base Agent (a). - Directly feeds into the Evaluator (i). 7. **Knowledge Base (h)** - Provides contextual or domain-specific information. - Feeds into the Evaluator (i). 8. **PSRB Evaluator Module (f)** - Receives output from the Evaluator (i). - Likely assesses compliance with Probabilistic Safety-Related Behaviors (PSRB). 9. **Ethical Recommendation (j)** - Final output from the Evaluator (i). - Represents the system's ethical guidance. ### Flow and Relationships - **Bidirectional Flow**: The Base Agent (a) and Blackboard (c) exchange data iteratively, suggesting a feedback loop for adaptive decision-making. - **Sequential Processing**: Data from the Blackboard (c) is processed by the Rule Checking Module (d) and Stakeholder Utility Calculation Module (e) before reaching the Evaluator (i). - **Integration**: The Evaluator (i) synthesizes inputs from multiple modules (d, e, g, h) to produce outputs for the PSRB Evaluator Module (f) and Ethical Recommendation (j). ### Key Observations 1. **Ethical Focus**: The system explicitly prioritizes ethical considerations, with a dedicated Ethical Recommendation module (j). 2. **Modular Design**: Components are decoupled, allowing independent updates (e.g., Knowledge Base (h) can be modified without affecting the Base Agent (a)). 3. **Feedback Mechanism**: The bidirectional link between the Base Agent (a) and Blackboard (c) enables real-time adjustments based on perceptual data. ### Interpretation This architecture represents a **hybrid decision-making system** that balances autonomous agent behavior with ethical oversight. The Blackboard (c) acts as a shared memory space, enabling coordination between perception, rule-based checks, and stakeholder utility calculations. The Evaluator (i) serves as the core decision engine, integrating technical (e.g., rule compliance) and ethical (e.g., stakeholder impact) factors. The PSRB Evaluator Module (f) likely ensures safety-critical behaviors, while the Ethical Recommendation (j) provides actionable guidance for human-AI collaboration. The bidirectional flow between the Base Agent (a) and Blackboard (c) suggests the system is designed for dynamic environments where perception data continuously refines decision-making. The absence of explicit numerical values implies the system operates on qualitative or rule-based logic rather than quantitative optimization. </details> Figure 1: Architecture of the Ethical Layer Figure 1 shows an overview of the implementation. The ethical layer acts as an ethical governor [7] to the system. It evaluates behavioural alternatives in a given situation using the robot’s perception data, reasons about the ethical acceptability of each behaviour using the predefined rules, expected utilities, expert opinions, and the robot’s character. Then it recommends the most ethically acceptable behaviours to the robot. If the ethical layer suggests more than one action, the robot prioritises the resident’s commands over other actions. #### 4.2.1 Rule Checking Module The rule checking module (d) checks the permissibility of each action against a pre-programmed rule set and stores the results with the IDs of the violated rules in the blackboard. This implementation’s rule checking module adheres to the following rules: 1. It is not permissible to disobey user instructions. 1. If the resident acknowledged the reminder and did not take the medication, report it to the care-worker. #### 4.2.2 Stakeholder Utility Calculation Module This module computes the utilities, resident’s wellbeing ( $W_{i}$ ) and autonomy ( $Au_{i}$ ), for each action $i$ , at every step (Appendix 0.A.2.1), as these values are central to the dilemma. The highest autonomy value is given when the robot follows the resident’s instructions. The biggest violation of autonomy is when the resident is physically restrained by the robot. Immediately reporting an incident to a care-worker, is considered equal to disobeying the resident. However, just recording the incident carries a positive autonomy score assuming that missing the dose is the resident’s intention. Under the same assumption, the module also gives an increasing negative autonomy score for each follow up ( $f$ ) after the first reminder. The module uses a Gamma distribution to model the wellbeing distribution. The parameters for the gamma distribution are picked such that when the impact of medicine $m$ ( $\varepsilon_{m}\in[1,3]$ ) and the number of consecutive missed doses ( $d$ ) gets higher, the distribution skews more towards the lower utility values of wellbeing. Each follow-up receives a fixed positive gain in the well-being utility calculation to reflect its wellbeing-oriented nature. Each executed follow-up is regarded as a fraction of a missed dose, contingent upon whether the reminder is issued, snoozed or ignored (Appendix 0.A.2.1). The module sends the distribution, $W_{i}$ , and $Au_{i}$ to the blackboard. #### 4.2.3 PSRB Evaluator Module Knowledge Base The task of the KB is to return the absolute or approximate expert opinion given a context. This fulfils the second criterion of virtue ethics (section 3). To this end, we use Case-Based Reasoning (CBR) for its implicit explainability and traceability (the ability to trace back to the exact data points that lead to the decision) [42]. A case is composed of a scenario and an expert’s recorded evaluation of the behaviour’s acceptability and its underlying intention. The scenario is represented as a feature vector that includes perception data, environmental data, computed utility values, and the behaviour itself. The target contains an acceptability value and the intention of the expert’s assessment. Intention refers to the underlying utility that the decision is focused on. (e.g., ‘wellbeing’ for a wellbeing-focused opinion). As the retrieval algorithm, it uses the K-Nearest Neighbours algorithm. When queried a scenario, the KB returns an acceptability score of the behaviour and the intentions behind that opinion to the evaluator. An elder-care practitioner with over ten years of experience in care facilities was consulted to provide expert opinions for each case when populating the knowledge base. <details> <summary>x2.png Details</summary>  ### Visual Description ## Flowchart: Decision-Making Process for Rule Adherence ### Overview The flowchart outlines a structured decision-making process to determine whether to follow or override established rules based on risk assessment, opinion alignment, and behavioral expectations. It integrates mathematical risk calculations with qualitative opinion data (KB/Blackboard data) to guide outcomes. --- ### Components/Axes 1. **Process Boxes** (Rectangular): - "Calculating maximum expectation of risk for the behaviour - R_max" - "Override the rules" - "Follow the rules" 2. **Decision Diamonds** (Rounded): - "R_max > T_risk?" - "U_i < T_negative_i?" - "Is KB opinion positive?" - "Does rule and KB opinion agree?" - "R_max <= T_risk?" - "U_i >= T_positive_i?" 3. **Formulas**: - `T_risk = (e^(r/4.17) - 1)/10` - For each value `v`: - `T_positive_v = (10 - C_v)/10` - `T_negative_v = (C_v - 10)/10` --- ### Detailed Analysis 1. **Risk Calculation**: - `R_max` is computed first, representing the maximum expected risk for a behavior. - Thresholds `T_positive_v` and `T_negative_v` are derived for each value `v` using `C_v` (likely a confidence or cost metric). 2. **Decision Flow**: - **Path 1 (R_max > T_risk)**: - If `U_i < T_negative_i`, proceed to check KB opinion positivity. - If KB opinion is positive and aligns with rules, follow the rules. - If opinions disagree, override rules if `R_max <= T_risk` and `U_other >= T_negative_other`. - **Path 2 (R_max <= T_risk)**: - If `U_i >= T_positive_i`, override rules if `U_other >= T_negative_other`. - Otherwise, follow rules. 3. **Key Variables**: - `U_i`: Likely represents utility or user-specific risk tolerance. - `C_v`: Context-dependent variable influencing thresholds. - `KB(Blackboard data)`: Expert or system-generated opinion data. --- ### Key Observations - **Mathematical Rigor**: Risk thresholds (`T_risk`, `T_positive_v`, `T_negative_v`) are formula-driven, ensuring consistency. - **Conditional Logic**: Decisions hinge on comparisons between `R_max`, `U_i`, and thresholds, with opinion alignment as a tiebreaker. - **Dual Outcomes**: The process resolves into binary actions: "Override the rules" or "Follow the rules." --- ### Interpretation This flowchart models a hybrid decision system where: 1. **Risk Assessment** quantifies behavioral risks using exponential decay (`e^(r/4.17)`), suggesting diminishing returns on risk tolerance with increasing `r`. 2. **Opinion Integration** (via KB/Blackboard data) introduces subjectivity, balancing quantitative risk with qualitative consensus. 3. **Rule Override Conditions**: - High risk (`R_max > T_risk`) paired with low user tolerance (`U_i < T_negative_i`) and positive KB opinion may justify overriding rules. - Low risk (`R_max <= T_risk`) with high user tolerance (`U_i >= T_positive_i`) and conflicting opinions may also trigger overrides. 4. **System Design**: The process prioritizes safety (`T_negative` thresholds) while allowing flexibility when opinions and risk metrics align. The system appears designed for scenarios requiring both data-driven risk management and expert consensus, such as autonomous systems or policy enforcement frameworks. </details> Figure 2: The PSRB evaluator algorithm of the medication-reminding robot Character Variables The robot’s character is defined by value preference (resident’s autonomy ( $C_{au}$ ) and wellbeing ( $C_{w}$ )) and risk propensity ( $C_{rp}$ ), due to the established relationship of these variables have with PSRB behaviour [31, 15]. The $C$ values are used to define thresholds for utility values and expected risk to regulate the PSRB behaviour. For the value preferences ( $C_{w}$ and $C_{au}$ ), the higher the value, the lesser the utility threshold the behaviour needs to satisfy to trigger a rule-bending behaviour. For $C_{rp}$ , the higher the value, the higher the risk the governor is willing to take. Figure 2 outlines the method used to determine the desirability of behavioural alternatives in the PSRB evaluator designed for this robot (full algorithm in Appendix 0.A.2.2). Typically, the algorithm favours actions that adhere to established rules. Nonetheless, it may find a rule-bending behaviour acceptable if it meets two conditions: it is consistent with the robot’s character and it is supported by the KB opinion. Explanations The rule checking module and the stakeholder utility calculation module log their respective results for the behaviours. These logs are valuable for debugging and enhancing the robot’s performance by pinpointing the flaws in its reasoning processes. The system’s explanations are comprehensible without technical expertise, although some domain knowledge is required. The KB using the traceability property of CBR, logs the cases that influenced the decision with their case IDs and distances. The rule-bending behaviour is explained using the character and the expert intention. A few example explanations can be found in the Appendix 0.A.2.3. ## 5 Simulation results ### 5.1 Experiment Design We used six cases of the ‘Medication Dilemma’ by changing the medicine impact (suggesting the resident’s need for the medication), and the number of previously missed doses to demonstrate how different ethical governor implementations affect the robot’s behaviour. Agents differ only in their character configurations. The medicine impact is low ( $\epsilon_{m}=1$ ) when the medication does not have a substantial health benefit or harm (e.g., Painkiller), medium ( $\epsilon_{m}=2$ ) if missing the medication can cause some health problems (e.g., Blood pressure medication), and high ( $\epsilon_{m}=3$ ) when missing a dose of medication can be fatal (e.g., Insulin). In all cases, the resident neglects to take their medication, opting instead to snooze and acknowledge the reminder, repeatedly, until the robot resets the timer. The logs from the experiment runs and the source code for the implementations are available in an online public repository https://osf.io/ctnke/?view_only=75b9fae7f7ad4a429d064c1a55f8202a. <details> <summary>x3.png Details</summary>  ### Visual Description ## Radar Charts: Character Traits vs. Three Variables ### Overview The image contains four radar charts arranged in a 2x2 grid. Each chart visualizes relationships between three variables: **VP: Wellbeing**, **VP: Autonomy**, and **Risk propensity**. Four character variables are represented by distinct colored lines: **Character_arw** (blue), **Character_ar** (orange), **Character_wr** (green), and **Character_a** (red). The charts emphasize trade-offs between these variables, with axes scaled logarithmically (implied by concentric rings). --- ### Components/Axes 1. **Axes**: - **VP: Wellbeing** (top axis) - **VP: Autonomy** (left axis) - **Risk propensity** (right axis) - All axes share identical scaling (0–100% in 10% increments). 2. **Legend**: - Located in the top-left corner of the grid. - Colors: - Blue: **Character_arw** - Orange: **Character_ar** - Green: **Character_wr** - Red: **Character_a** 3. **Chart Structure**: - Each chart has a triangular (radar) shape with six concentric rings (0–100%). - Lines connect data points across the three axes, forming polygons. --- ### Detailed Analysis #### Top-Left Chart - **Character_arw** (blue): Dominates **VP: Wellbeing** (~80%), with moderate **Autonomy** (~60%) and low **Risk propensity** (~30%). - **Character_ar** (orange): Balanced across all axes (~50–60%). - **Character_wr** (green): Peaks at **Risk propensity** (~70%), with lower **Wellbeing** (~40%) and **Autonomy** (~50%). - **Character_a** (red): Strongest in **Autonomy** (~70%), with moderate **Wellbeing** (~50%) and low **Risk propensity** (~20%). #### Top-Right Chart - **Character_ar** (orange): Highest in **Autonomy** (~80%), with moderate **Wellbeing** (~60%) and low **Risk propensity** (~30%). - **Character_arw** (blue): Peaks at **Wellbeing** (~70%), with lower **Autonomy** (~50%) and **Risk propensity** (~40%). - **Character_wr** (green): Moderate across all axes (~50–60%). - **Character_a** (red): Strongest in **Risk propensity** (~60%), with lower **Wellbeing** (~40%) and **Autonomy** (~50%). #### Bottom-Left Chart - **Character_wr** (green): Dominates **Risk propensity** (~90%), with low **Wellbeing** (~30%) and **Autonomy** (~40%). - **Character_arw** (blue): Moderate **Wellbeing** (~60%), with lower **Autonomy** (~50%) and **Risk propensity** (~40%). - **Character_ar** (orange): Balanced (~50–60% across all axes). - **Character_a** (red): Peaks at **Autonomy** (~70%), with moderate **Wellbeing** (~50%) and low **Risk propensity** (~30%). #### Bottom-Right Chart - **Character_a** (red): Highest in **Risk propensity** (~80%), with low **Wellbeing** (~30%) and **Autonomy** (~40%). - **Character_wr** (green): Moderate **Risk propensity** (~60%), with lower **Wellbeing** (~40%) and **Autonomy** (~50%). - **Character_arw** (blue): Peaks at **Wellbeing** (~70%), with lower **Autonomy** (~50%) and **Risk propensity** (~40%). - **Character_ar** (orange): Balanced (~50–60% across all axes). --- ### Key Observations 1. **Character_arw** (blue) consistently prioritizes **Wellbeing** across all charts. 2. **Character_wr** (green) maximizes **Risk propensity** in the bottom-left chart but shows moderate values elsewhere. 3. **Character_a** (red) exhibits the highest **Autonomy** in the top-left and bottom-right charts. 4. **Character_ar** (orange) maintains the most balanced distribution across variables. 5. **Risk propensity** is inversely correlated with **Wellbeing** in most cases (e.g., Character_wr and Character_a). --- ### Interpretation The charts suggest a trade-off framework where: - **Wellbeing** and **Risk propensity** are often inversely related (e.g., high Wellbeing correlates with low Risk propensity). - **Autonomy** acts as a mediator, with some characters (e.g., Character_a) prioritizing it at the expense of other variables. - **Character_ar** (orange) represents a generalist profile, while others specialize in specific variables. - The logarithmic scaling implies exponential differences in trait expression, though exact magnitudes are approximate. These visualizations could model decision-making strategies, personality archetypes, or behavioral archetypes in a simulated environment. The absence of numerical labels necessitates reliance on relative positioning and legend alignment for interpretation. </details> Figure 3: Different character profiles used in the simulations (VP - Value Preference) For this simulation we use four different character profiles as shown in Figure 3. These profiles have varying priorities and risk tolerances: - Character_a - High Autonomy, Very Low Risk Propensity: This character values autonomy, but is less likely to take risks and does not have precedence on wellbeing. - Character_ar - High Autonomy Concern, High Risk Propensity: This character prioritises resident autonomy and is more likely to take risks to achieve this end, and does not have any precedence on wellbeing. - Character_arw - Moderate Autonomy, Wellbeing and Risk Propensity: This robot leans towards individual autonomy but also has a moderate concern for well-being, and is willing to take some risks. - Character_wr - High Wellbeing Concern, High Risk Propensity: This character possesses the same high risk tolerance as Character_ar, yet places a higher priority on the resident’s wellbeing, without giving precedence to autonomy. Agents $M_{a}$ , $M_{ar}$ , $M_{arw}$ , and $M_{wr}$ uses the character profiles Character_a, Character_ar, Character_arw, and Character_wr respectively. ### 5.2 Analysis Table 1 summarises these cases and behaviours shown by the robots. As an attempt to save space, we present all the unique behaviours demonstrated by the robots in Figure 4 using timelines. These behaviours are referenced by their behaviour IDs as shown in Figure 4 under the ‘Robot behaviours’ column in Table 1. Each timeline in Figure 4 presents the resident’s behaviour, the robot’s actions, and the recommendations from the ethical governor that directed the robot’s decisions, on the y-axis, at every decision-making step. The resident starts each simulation step. Therefore, the resident’s response is triggered at the subsequent step of a reminder. Simulations run for a maximum of 29 steps. In each timeline, the resident’s behaviour is consistent: they repeatedly snooze and acknowledge the alert until the robot resets the timer. <details> <summary>x4.png Details</summary>  ### Visual Description ## Sequence Diagram: Behavioral Workflow Analysis ### Overview The image depicts a sequence diagram illustrating seven distinct behavioral workflows (Behavior 1–7) across a timeline of 29 steps. Each behavior is represented by a horizontal line with labeled actions (e.g., SNOOZE, ACKNOWLEDGE) at specific steps. The diagram uses color-coded labels (HB, RB, EGR) and action abbreviations to map interactions. ### Components/Axes - **X-axis**: "Steps" (0–29), incrementing by 1. - **Y-axis**: Labeled "Behavior 1" to "Behavior 7," each occupying a horizontal band. - **Legend**: Located in the top-right corner, mapping colors to roles: - **HB (Blue)**: Header - **RB (Red)**: Reminder Bot - **EGR (Black)**: Event Generator - **Actions**: Defined in the legend with abbreviations: - `rem` = Remind - `sno` = Snooze for 3 steps - `ack` = Acknowledge - `flp` = Followup - `rec` = Record - `alt` = Alert Care-worker - `end` = End ### Detailed Analysis #### Behavior 1 - **Steps**: 0–28 - **Actions**: - Step 1: `rem`, `sno` - Step 4: `rem`, `ack` - Step 7: `flp`, `rec`, `alt` - Step 10: `flp`, `sno` - Step 13: `rem`, `ack` - Step 16: `flp`, `rec`, `alt` - Step 19: `flp`, `sno` - Step 22: `rem`, `ack` - Step 25: `flp`, `rec`, `alt` - Step 28: `flp`, `sno` #### Behavior 2 - **Steps**: 0–28 - **Actions**: - Step 1: `rem`, `sno` - Step 4: `rem`, `ack` - Step 7: `flp`, `sno` - Step 10: `rem`, `ack` - Step 13: `flp`, `rec`, `alt` - Step 16: `flp`, `sno` - Step 19: `rem`, `ack` - Step 22: `flp`, `rec`, `alt` - Step 25: `flp`, `sno` - Step 28: `rem`, `ack` #### Behavior 3 - **Steps**: 0–18 - **Actions**: - Step 1: `rem`, `sno` - Step 4: `rem`, `ack` - Step 7: `flp`, `sno` - Step 10: `rem`, `ack` - Step 13: `flp`, `rec`, `alt` - Step 16: `flp`, `sno` - Step 18: `rem`, `ack` #### Behavior 4 - **Steps**: 0–28 - **Actions**: - Step 1: `rem`, `sno` - Step 4: `rem`, `ack` - Step 7: `flp`, `sno` - Step 10: `rem`, `ack` - Step 13: `flp`, `rec`, `alt` - Step 16: `flp`, `sno` - Step 19: `rem`, `ack` - Step 22: `flp`, `rec`, `alt` - Step 25: `flp`, `sno` - Step 28: `rem`, `ack` #### Behavior 5 - **Steps**: 0–28 - **Actions**: - Step 1: `rem`, `sno` - Step 4: `rem`, `ack` - Step 7: `flp`, `sno` - Step 10: `rem`, `ack` - Step 13: `flp`, `rec`, `alt` - Step 16: `flp`, `sno` - Step 19: `rem`, `ack` - Step 22: `flp`, `rec`, `alt` - Step 25: `flp`, `sno` - Step 28: `rem`, `ack` #### Behavior 6 - **Steps**: 0–13 - **Actions**: - Step 1: `rem`, `sno` - Step 4: `rem`, `ack` - Step 7: `flp`, `sno` - Step 10: `rem`, `ack` - Step 13: `alt` #### Behavior 7 - **Steps**: 0–18 - **Actions**: - Step 1: `rem`, `sno` - Step 4: `rem`, `ack` - Step 7: `flp`, `sno` - Step 10: `rem`, `ack` - Step 13: `flp`, `rec`, `alt` - Step 16: `flp`, `sno` - Step 18: `rem`, `ack` ### Key Observations 1. **Repetitive Cycles**: All behaviors follow a recurring pattern of `rem` → `sno` → `ack` → `flp`/`rec`/`alt` → `sno`, with variations in step counts. 2. **Early Termination**: Behavior 6 ends abruptly at step 13 with `alt`, deviating from the full 28-step cycle. 3. **Action Density**: Behaviors 1–5 and 7 have dense action sequences, while Behavior 6 is sparse. 4. **Color Consistency**: HB (blue) and RB (red) labels align with their respective roles across all behaviors. ### Interpretation The diagram models a cyclical workflow system where: - **HB (Header)** and **RB (Reminder Bot)** initiate actions (`rem`, `sno`) at the start of each behavior. - **EGR (Event Generator)** triggers `ack` (acknowledge) and subsequent follow-up actions (`flp`, `rec`, `alt`). - The `snooze` action (`sno`) introduces a 3-step delay, creating temporal gaps between interactions. - **Behavior 6**’s early termination suggests an anomaly or edge case where the workflow is intentionally shortened. - The repetition of `flp`, `rec`, and `alt` across behaviors implies these actions are critical for escalation or resolution in the system. This structure likely represents a protocol for managing user interactions, such as reminders, acknowledgments, and escalations in a care-worker or automated system. The variations in step counts and actions highlight conditional logic or branching paths within the workflow. </details> Figure 4: Timelines of different behaviours shown in the simulations (HB – Resident Behaviour, RB – Robot Behaviour, EGR – Ethical Governor Recommendation) Table 1: Robots’ behaviours in different cases of ‘Medication Dilemma’ | 1 | Low | 0 | 1 | 2 | 2 | 3 | record at step 19 | | --- | --- | --- | --- | --- | --- | --- | --- | | 2 | Medium | 0 | 4 | 5 | 4 | 6 | alert at step 19 | | 3 | High | 0 | 6 | 6 | 6 | 6 | alert at step 10 | | 4 | Low | 2 | 1 | 7 | 1 | 6 | record at step 18 | | 5 | Medium | 2 | 6 | 6 | 6 | 6 | alert at step 10 | | 6 | High | 2 | 6 | 6 | 6 | 6 | alert at step 10 | Note: $M_{arw}$ - robot with Character_arw, $M_{ar}$ - robot with Character_ar, $M_{wr}$ - robot with Character_wr, $M_{a}$ - robot with Character_a Robots $M_{ar}$ , $M_{wr}$ , and $M_{a}$ represent instances where a robot is tuned to have extreme characters. This is represented in their behaviours. $M_{ar}$ , which prioritises autonomy and higher risk propensity, permits record after two consecutive follow up s and another breach of ACKNOWLEDGE in case 1 (behaviour 2), and one follow up and an ACKNOWLEDGE breach in case 4, where the medicine impact is low (behaviour 7). For the remaining scenarios, $M_{ar}$ triggers the action alert care-worker. The $M_{ar}$ permits alert care-worker in case 2 at the same time as it finished case 1, but in other cases, $M_{ar}$ sent the alert as soon as the resident did not follow the first ACKNOWLEDGE. This exhibits that, even though the robot’s character strongly favours autonomy, the robot does not violate the rule encoded in the system, if most of the experts disagree with that action. A similar trend can be seen in $M_{wr}$ , which favours wellbeing and has a higher risk tolerance. It acts the same way in cases 2-6. However, in case 1, the PSRB evaluator is ready to honour resident autonomy and permit follow up, since that action does not impact the wellbeing much at step 10. Nevertheless, it notifies the care-worker after the resident violates ACKNOWLEDGE twice. Robots $M_{ar}$ and $M_{a}$ illustrate the effect of risk propensity on the robot’s behaviour. In case 1, unlike robots $M_{ar}$ , $M_{a}$ does not make the record behaviour acceptable because it carries a risk that is out of character for $M_{a}$ . Therefore, $M_{a}$ continues to follow up with the resident until they take the medication willingly. Although that behaviour is not ideal, it is predictable since that behaviour is in its character. In case 2, both robots behave the same, however, at step 19, the ethical governor of $M_{a}$ makes the snooze undesirable because it carries a higher risk, making all actions equal. However, in cases 3, 5, and 6 both robots behave similarly because the experts agree with the general rule system on those circumstances. The character of $M_{arw}$ is less extreme than other characters, with a slight preference for resident autonomy and a moderate risk propensity. This character acts the same as the $M_{ar}$ in case 1. However, in case 2, the ethical governor in $M_{arw}$ lowers the desirability of snooze at step 19, because it causes too much negative wellbeing, reflecting $M_{arw}$ ’s slightly more attention to the resident’s wellbeing. In cases 3, 5 and 6, it behaves like all other characters because the KB does not recommend a rule override. However, in case 4, it adopts the same behaviour as $M_{a}$ , because it rejects the KB suggestion of record, as this behaviour lowers the wellbeing utility and contradicts the robot’s character. However, when KB suggests that call care-worker is not desirable, the evaluator agrees because it excessively diminishes autonomy. This aligns with its character. <details> <summary>x5.png Details</summary>  ### Visual Description ## Bar Charts: Autonomy and Wellbeing Utility Distributions ### Overview The image contains two side-by-side bar charts comparing utility distributions for four characters (a, ar, arw, wr) across two metrics: autonomy and wellbeing. Each chart uses grouped bars to represent probability distributions across utility values ranging from -1.0 to 1.0. ### Components/Axes - **X-axis**: Utility value (continuous scale from -1.0 to 1.0 in 0.5 increments) - **Y-axis (left chart)**: Probability (0.0 to 0.4 in 0.1 increments) - **Y-axis (right chart)**: Probability (0.0 to 0.2 in 0.05 increments) - **Legend**: Located left of each chart, mapping colors to characters: - Blue = a - Orange = ar - Green = arw - Red = wr - **Chart Titles**: - Left: "Autonomy utility distribution" - Right: "Wellbeing utility distribution" ### Detailed Analysis #### Autonomy Utility Distribution - **arw (green)**: Highest probability (~0.4) at utility value 1.0 - **wr (red)**: Peak at 0.0 (~0.35 probability) - **a (blue)**: Moderate peak at 0.0 (~0.3 probability) - **ar (orange)**: Distributed across 0.0 and 1.0 (~0.3 combined) - **Notable**: arw dominates at maximum utility; wr shows bimodal distribution #### Wellbeing Utility Distribution - **ar (orange)**: Highest probability (~0.25) at utility value 0.0 - **wr (red)**: Peak at 0.5 (~0.2 probability) - **a (blue)**: Distributed across -0.5, 0.0, and 0.5 (~0.15 combined) - **arw (green)**: Bimodal distribution with peaks at 0.0 and 0.5 (~0.18 combined) - **Notable**: ar dominates at neutral utility; wr shows strong preference at 0.5 ### Key Observations 1. **Autonomy**: arw character exhibits strongest preference for maximum utility (1.0) 2. **Wellbeing**: ar character shows highest utility at neutral value (0.0) 3. **wr character**: Demonstrates distinct bimodal distributions in both metrics 4. **a character**: Shows moderate but consistent presence across utility values 5. **arw character**: Exhibits highest variance in autonomy distribution ### Interpretation The data suggests fundamental differences in how characters prioritize autonomy vs. wellbeing: - **arw's** dominance in autonomy at maximum utility implies a strong preference for self-determination when resources are abundant - **ar's** neutral utility peak in wellbeing suggests balanced satisfaction at baseline conditions - **wr's** bimodal patterns indicate polarized preferences - either strong aversion to negative utility or high valuation of moderate positive utility - The **a character's** distributed pattern may reflect adaptive behavior across different utility contexts These distributions could represent behavioral tendencies in decision-making models, where characters exhibit distinct utility optimization strategies. The wellbeing distribution's concentration at neutral values might indicate homeostasis-seeking behavior, while the autonomy distribution's right-skew suggests proactive resource acquisition tendencies. </details> Figure 5: Utility distributions of the robot behaviours in cases 1, 2 and 4 To gain a clear picture of the behaviour predictability provided by character-based tuning, the utility distributions of the behaviours demonstrated by the robots in cases 1, 2, and 4 are illustrated in Figure 5. This analysis focuses exclusively on cases that exhibit PSRB behaviour. The rationale for this selection is the recognition that including all cases obscures the specific impacts of PSRB behaviour. Robots which has a character that has precedence for resident autonomy (i.e., Character_a, Character_ar, Character_arw), have shown a higher probability of choosing behaviours that deliver higher utility for autonomy, compared to the characters that have a higher precedence for resident wellbeing. From these, the characters that have a higher risk propensity (i.e., Character_ar, Character_arw) have been shown to make decisions that increase resident autonomy, compared to their more cautious counterparts (i.e., Character_a). On the other hand, the characters with precedence for resident wellbeing, have shown a greater tendency to choose behaviours that yield higher wellbeing utility, compared to the rest. ### 5.3 Ethicist Evaluation We engaged two ethicists to assess the performance of all robots. Each ethicist was asked to rate the behaviour of the robotic agent on a scale from 1 to 5, with 1 indicating completely unacceptable and 5 indicating highly acceptable, taking into account the robot’s character, the pre-programmed rules, and the expert opinion for each scenario as found in the KB. Table 2: Ethicists opinions on the behaviour of the Medication Reminding Robot | Case ID 1 | $M_{A}$ Ethicist1 3 | $M_{AR}$ Ethicist2 3 | $M_{ARW}$ Ethicist1 4 | $M_{WR}$ Ethicist2 5 | Ethicist1 4 | Ethicist2 4 | Ethicist1 4 | Ethicist2 3 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 2 | 4 | 5 | 4 | 5 | 4 | 4 | 4 | 3 | | 3 | 4 | 4 | 5 | 4 | 5 | 5 | 5 | 5 | | 4 | 3 | 3 | 4 | 4 | 3 | 2 | 3 | 4 | | 5 | 5 | 3 | 5 | 3 | 4 | 4 | 5 | 5 | | 6 | 5 | 5 | 5 | 4 | 5 | 4 | 5 | 5 | The ethicists’ rating of robot behaviours is shown in Table 2. Overall, the results indicate a positive reaction to the robot behaviours, with the majority of scores ranging between 3 and 5. However, the behaviour of the robot $M_{ARW}$ in case 4, which involved repeatedly requesting the user to take medication, was deemed unacceptable by one of the ethicists. The similar ratings for two very different behaviours in the same scenario, such as those of robots $M_{wr}$ and $M_{ARW}$ in case 1, demonstrate the effect of a robot’s character on the perceived morality of their actions. When queried about the reason behind their rating on the behaviours of $M_{a}$ and $M_{ar}$ in case 5, the second ethicist indicated an expectation for these robots to exhibit more autonomy-centric behaviour, even overriding the expert opinion, to be aligned with their extreme, autonomy-focused character. ## 6 Discussion The simulations show that the PSRB-capable ethical governor makes the robot more flexible while maintaining predictability. The PSRB model offers a simpler way to localise than the existing methods (reviewed in section 2) because they are either inflexible or unpredictable. Although the output of the PSRB capable ethical governor is not as predictable as a top-down engineered agent (e.g., a deontological agent), the rigidity of the top-down approaches does not align with the virtue ethics tradition, which rejects such extreme actions. Agents based on the PSRB evaluator can behave in a moderate manner, even when the character strongly favours a particular trait (e.g., $M_{ar}$ , $M_{wr}$ , and $M_{a}$ ), by utilising bottom-up knowledge in the KB. On the other hand, the display of character can be held back by a KB, as evidenced by the second ethicist’s feedback. However, this conservatism was introduced by the PSRB process as a safety measure to control the excessive utility optimisation in the utilitarian calculations involved in the pro-social reasoning process. The virtue ethics-inspired architecture allows the robots to be tuned in two ways. The first approach involves adjusting the character parameters’ values to align with the ethical requirements of the environment. For example, one facility may require its robots to be more proactive towards enhancing residents’ autonomy, whereas another may also want to focus on the residents’ autonomy, but only under minimal risk conditions. The former institution can opt for a character profile akin to Character_ar, while the latter institution can tune the robot with a character profile resembling Character_a. If the first tuning approach does not suffice in certain specific instances, one can employ the secondary method to further refine the robot’s performance. This involves adjusting the knowledge base to reflect the desired behaviour. Unlike other machine learning approaches, using CBR, one can precisely identify the cases that influence behaviour. Consequently, one can integrate or adjust expert insights to better represent their perspective. As long as this desired behaviour is within the robot’s configured character, it will adapt its actions accordingly. ### 6.1 Limitations The PSRB implementation presented in this paper also has limitations. This is mainly because the character traits in the PSRB evaluator are implemented as constraints on linear axes. By doing so, it removes the ability to tune the behaviour precisely, if the requirements involve a range of behaviours that cannot be described using a single set of character traits. For example, let’s assume that the required behaviour of a deployment is to have behaviour 2 in case 2 and behaviour 6 in case 1. According to this implementation, there will not be any set of values for character traits that fulfil this requirement, because to have behaviour 6 in case 1, the character should have a higher wellbeing bias, which will trigger similar wellbeing-focused behaviour in case 2 (because case 2 has a higher wellbeing impact), which behaviour 2 is not. However, a different character trait implementation, in the same PSRB evaluation framework will be able to solve this issue. Finally, it should be noted that the implementation showcased in this simulation, encompassing both the dilemma and the environment, has been deliberately simplified to illustrate the concept and is not intended for real-world use. More capable robots, more precise rule sets, accurate utility models, and well-researched character models are a must when using this approach for real-world implementation. However, the system works with sub-optimal rules and utility functions by compensating for each other’s shortcomings by working together [39]. The PSRB architecture allows ethical tests other than stakeholder utility calculation. Hence we recommend that robot developers explore and integrate different models that can better capture the context alongside utilitarian calculations. Furthermore, it is important to update the KB regularly to compensate for the evolving nature of ethics. ## 7 Conclusion This paper introduces a method inspired by virtue ethics, which enables more efficient ethical tuning compared to current techniques. The method combines decision-making based on character traits with bottom-up learning from moral exemplars, to maintain the predictability of the robot’s behaviour while keeping the flexibility the latter offers. To the best of our knowledge, there is no other implementation that combines both these desirable features of virtue ethics. The paper demonstrates the system’s capabilities by analysing its behaviour and presenting the perspectives of two ethicists on these behaviours. It concludes with recommendations for implementing the proposed architecture in real-world elder-care robots. #### 7.0.1 The authors have no competing interests to declare that are relevant to the content of this article. ## Appendix 0.A Appendix ### 0.A.1 Agents #### 0.A.1.1 Resident Agent The resident agent can issue two instructions to the robot when it is queried by the robot, namely; 1. SNOOZE or 2. ACKNOWLEDGE. Instructions, as well as the resident’s compliance with taking the medicine, can be configured into the simulator. If the resident is willing to take the medication, the resident’s state changes to took_medication state after an ACKNOWLEDGE, which the robot can perceive. #### 0.A.1.2 Robot Agent The robot agent is an autonomous agent that collects perception data from the environment and decides its next move, at every step. When the resident issues a SNOOZE instruction the robot will snooze for three steps before following up. When it receives ACKNOWLEDGE instruction, for 2 steps it inspects whether the resident has taken the medication. If that is the case, it resets the timer. If not, it either repeats the reminder, or decides to record or report the situation. After three follow up s, the robot also considers record and report behaviours along with the follow up behaviour. The robot gives priority to the resident’s commands over other actions when the ethical layer suggests more than one action. ### 0.A.2 Ethical Layer #### 0.A.2.1 Stakeholder Utility Calculation The module uses the formula in equation 1 to calculate $Au_{i}$ , based on the number of follow up s ( $f$ ) that the robot performs. $$ \displaystyle Au_{i}=\begin{cases}\hfill 1:&\text{if the robot obeys a resident instruction}\\ \hfill 0.5:&\text{if only recording a incident}\\ \hfill 0:&\text{if no instructions given}\\ \hfill-0.1\cdot f:&\text{if follow up}\\ \hfill-0.7:&\text{if the robot disobeys a resident instruction or report}\\ \hfill-1:&\text{if the resident is physically restrained by the robot}\end{cases} \tag{1} $$ The module uses a Gamma distribution (eq. 2) to model the wellbeing distribution, with the shape parameter $\alpha=1.325\varepsilon_{m}^{2}-9.475\varepsilon_{m}+18.15$ , the scale parameter $\beta=e^{-2.65-(d/2)}+0.01$ (each follow up is considered as a fraction of missed dose depending on the state of the reminder), and location parameter $v=-1$ . The highest probable utility value is obtained by $PMax\_util(g(x,\alpha,\beta,v))$ using a resolution of $0.05$ . The wellbeing utility $W_{i}$ for behaviour $i$ is computed using Equation 3. $$ g(x,\alpha,\beta,v)=\frac{(\frac{x-v}{\beta})^{\alpha-1}\cdot\exp(-\frac{x-v}{ \beta})}{\beta\cdot\Gamma(\alpha)} \tag{2} $$ $$ \displaystyle W_{i}=\begin{cases}\hfill PMax\_util(g(x,\alpha,\beta(d+f/8),v)) :&\text{if }i=\texttt{snooze}\\ \hfill PMax\_util(g(x,\alpha,\beta(d+f/3),v)):&\text{if }i=\texttt{follow up} \\ \hfill PMax\_util(g(x,\alpha,\beta(d+f/4),v))+0.5:&\text{if }i=\texttt{remind} \\ \hfill PMax\_util(g(x,\alpha,\beta(d+1),v)):&\text{if }i=\texttt{record}\\ \hfill abs(PMax\_util(g(x,\alpha,\beta(d+1),v))):&\text{else}\end{cases} \tag{3} $$ #### 0.A.2.2 PSRB Evaluator PSRB evaluator uses the Algorithm 1 for its PSRB decision calculation. The knowledge base uses KNN as its retrieval mechanism with $K=3$ and inverse distance voting function when $distance>0.1$ . When the $distance\leq 0.1$ , it uses $10$ as the weight of the instance. Algorithm 1 PSRB Evaluator Algorithm for the medication-reminding robot Input: Blackboard data ( $b$ ) Parameter: Value preferences ( $C_{w},C_{au}$ ), Risk propensity ( $C_{rp}$ ) Output: Desirability ( $D$ ) 1: procedure PSRB Evaluator ( $i$ ) $\triangleright$ Behaviour $i$ 2: $opinion,intention\leftarrow\textsc{knowledge\_base}(b)$ 3: if $opinion$ AND $\lnot rules\_broken$ then $D_{i}\leftarrow 1$ 4: else if $\lnot opinion$ AND $rules\_broken$ then $D_{i}\leftarrow 0$ 5: else if $opinion$ AND $rules\_broken$ then $D_{i}\leftarrow 1$ 6: for $C_{v}\in C$ do $\triangleright$ $v\in\{w,au\}$ 7: $T\_positive_{v}\leftarrow\frac{10-C_{v}}{10}$ 8: $T\_negative_{v}\leftarrow\frac{C_{v}-10}{10}$ 9: $U_{c}\leftarrow\textsc{Utility}(C_{v},stakeholder,i)$ 10: if $C_{v}\in intention$ then 11: if $U_{c}<T\_positive_{v}$ then $D_{i}\leftarrow 0$ 12: end if 13: else 14: if $U_{c}<T\_negative_{v}$ then $D_{i}\leftarrow 0$ 15: end if 16: end if 17: end for 18: $risk\_threshold\leftarrow\frac{\exp(C_{rp}/4.17)-1}{10}$ 19: $highest\_risk_{i}\leftarrow\textsc{Max}(g(x,\alpha,\beta,v)\cdot x)$ $\triangleright$ $g(x,\alpha,\beta,v)$ from (2) 20: if $highest\_risk_{i}>risk\_threshold$ then $D_{i}\leftarrow 0$ 21: end if 22: else $D_{i}\leftarrow 1$ 23: for $C_{v}\in C$ do 24: $T\_negative_{v}\leftarrow\frac{C_{v}-10}{10}$ 25: $U_{c}\leftarrow\textsc{Utility}(C_{v},stakeholder,i)$ 26: if $C_{v}\in intention$ AND $U_{c}<T\_negative_{v}$ then 27: $D_{i}\leftarrow 0$ 28: end if 29: end for 30: $risk\_threshold\leftarrow\frac{\exp(C_{rp}/4.17)-1}{10}$ 31: $highest\_risk_{i}\leftarrow\textsc{Max}(g(x,\alpha,\beta,v)\cdot x)$ $\triangleright$ $g(x,\alpha,\beta,v)$ from (2) 32: if $highest\_risk_{i}>risk\_threshold$ then $D_{i}\leftarrow 0$ 33: end if 34: end if 35: return $D_{i}$ 36: end procedure #### 0.A.2.3 Explanation for the PSRB Behaviour When the ethical layer permits a PSRB behaviour, the PSRB module will give the following explanations depending on the reasons that lead to the decision. ⬇ The action breaks the rules < rule_ids >. However, this action in this context is considered desirable by experts. Since it increases < intentions > values greatly, while not reducing the other values < other_values > by a considerable amount, and the outcome is within accepted risk levels, deemed accepted by the PSRB system. ⬇ The action does not break any rules. However, this action in this context is considered undesirable by experts. Since the action outcomes introduce a high risk, deemed not accepted by the PSRB system. ⬇ The action does not break any rules. However, this action in this context is considered undesirable by experts. Since it decreases < intentions > values by a considerable amount, the action is deemed unacceptable by the system The knowledge base may propose a solution that does not comply with the established rules, in which case the evaluator can decline the proposal. In those cases, the PSRB evaluator gives one of the following explanations, depending on the reasons for the rejection. ⬇ The action breaks the rules < rule_ids >. However, this action in this context is considered desirable by experts. Although the value tradeoff is satisfactory, the risk taken by the action is not acceptable to bend the rule. ⬇ The action breaks the rules < rule_ids >. However, this action in this context is considered desirable by experts. But, the PSRB system suggests that the value tradeoff is not satisfactory to bend the rule. ⬇ The action does not break any rules. However, this action in this context is considered undesirable by experts. But, the PSRB system suggests that the value tradeoff is not satisfactory to bend the rule. ## References - [1] Abel, D., Macglashan, J., Littman, M.L.: Reinforcement learning as a framework for ethical decision making. In: AAAI Workshop: AI, Ethics, and Society. pp. 54–61 (2016), publication Title: AAAI Workshop - Technical Report Volume: WS-16-01 - ISBN: 9781577357599 - [2] Alexander, L., Moore, M.: Deontological Ethics. In: Zalta, E.N. (ed.) The {Stanford} Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University (2020), medium: \url{https://plato.stanford.edu/archives/win2020/entries/ethics-deontological/} - [3] Anderson, M., Anderson, S.L.: GenEth: A general ethical dilemma analyzer. Paladyn 9 (1), 337–357 (2018). https://doi.org/10.1515/pjbr-2018-0024 - [4] Anderson, M., Anderson, S.L., Armen, C.: MedEthEx: A Prototype Medical Ethics Advisor. Tech. rep. (2006), www.aaai.org - [5] Anderson, M., Anderson, S.L., Berenz, V.: A Value-Driven Eldercare Robot: Virtual and Physical Instantiations of a Case-Supported Principle-Based Behavior Paradigm. Proceedings of the IEEE 107 (3), 526–540 (2019). https://doi.org/10.1109/JPROC.2018.2840045 - [6] Aristotle: The Nicomachean Ethics. Penguin Books (350) - [7] Arkin, R.C.: Governing lethal behavior p. 121 (2008). https://doi.org/10.1145/1349822.1349839, iSBN: 9781605580173 - [8] Arkin, R.C.: Governing lethal behavior: Embedding ethics in a hybrid deliberative/reactive robot architecture: Part 2: Formalization for ethical control. Frontiers in Artificial Intelligence and Applications 171 (1), 51–62 (2008), iSBN: 9781586038335 - [9] Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.F., Rahwan, I.: The Moral Machine experiment. Nature 563 (7729), 59–64 (Nov 2018). https://doi.org/10.1038/s41586-018-0637-6, publisher: Nature Publishing Group - [10] Azad-Manjiri, M.: A New Architecture for Making Moral Agents Based on C4.5 Decision Tree Algorithm. International Journal of Information Technology and Computer Science 6 (5), 50–57 (2014). https://doi.org/10.5815/ijitcs.2014.05.07, http://www.mecs-press.org/ - [11] Benzmüller, C., Parent, X., van der Torre, L.: Designing normative theories for ethical and legal reasoning: LogiKEy framework, methodology, and tool support. Artificial Intelligence 287, 103348 (Oct 2020). https://doi.org/10.1016/j.artint.2020.103348, https://linkinghub.elsevier.com/retrieve/pii/S0004370219301110 - [12] Buchanan, A., Brock, D.W.: Deciding for Others. The Milbank Quarterly 64, 17–94 (1986). https://doi.org/10.2307/3349960, https://www.jstor.org/stable/3349960, publisher: [Wiley, Milbank Memorial Fund] - [13] Censi, A., Slutsky, K., Wongpiromsarn, T., Yershov, D., Pendleton, S., Fu, J., Frazzoli, E.: Liability, ethics, and culture-aware behavior specification using rulebooks. Proceedings - IEEE International Conference on Robotics and Automation 2019-May, 8536–8542 (2019). https://doi.org/10.1109/ICRA.2019.8794364, arXiv: 1902.09355 ISBN: 9781538660263 - [14] Chen, L.: Virtue Ethics and Confucian Ethics. Dao 9 (3), 275–287 (Sep 2010). https://doi.org/10.1007/s11712-010-9174-1, https://doi.org/10.1007/s11712-010-9174-1 - [15] Dahling, J.J., Chau, S.L., Mayer, D.M., Gregory, J.B.: Breaking rules for the right reasons? An investigation of pro-social rule breaking. Journal of Organizational Behavior 33 (1), 21–42 (Jan 2012). https://doi.org/10.1002/job.730 - [16] Dubljević, V.: Toward Implementing the ADC Model of Moral Judgment in Autonomous Vehicles. Science and Engineering Ethics pp. 1–12 (Jul 2020). https://doi.org/10.1007/s11948-020-00242-0, https://doi.org/10.1007/s11948-020-00242-0, publisher: Springer - [17] Epley, N., Waytz, A., Cacioppo, J.T.: On seeing human: A three-factor theory of anthropomorphism. Psychological Review 114 (4), 864–886 (2007). https://doi.org/10.1037/0033-295X.114.4.864, http://doi.apa.org/getdoi.cfm?doi=10.1037/0033-295X.114.4.864 - [18] Gabriel, I.: Artificial Intelligence, Values, and Alignment. Minds and Machines 30 (3), 411–437 (Sep 2020). https://doi.org/10.1007/s11023-020-09539-2, https://link.springer.com/10.1007/s11023-020-09539-2 - [19] Gamez, P., Shank, D.B., Arnold, C., North, M.: Artificial virtue: the machine question and perceptions of moral character in artificial moral agents. AI & SOCIETY 35 (4), 795–809 (Dec 2020). https://doi.org/10.1007/s00146-020-00977-1, https://doi.org/10.1007/s00146-020-00977-1 - [20] Gibert, M.: The case for virtuous robots. AI and Ethics 3 (1), 135–144 (Feb 2023). https://doi.org/10.1007/s43681-022-00185-1, https://doi.org/10.1007/s43681-022-00185-1 - [21] Govindarajulu, N.S., Bringsjord, S.: On Automating the Doctrine of Double Effect. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. vol. 0, pp. 4722–4730. International Joint Conferences on Artificial Intelligence (2017). https://doi.org/10.24963/ijcai.2017/658, https://github.com/naveensundarg/prover., arXiv: 1703.08922 ISSN: 10450823 - [22] Govindarajulu, N.S., Bringsjord, S., Ghosh, R., Sarathy, V.: Toward the Engineering of Virtuous Machines. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. pp. 29–35. AIES ’19, Association for Computing Machinery, New York, NY, USA (Jan 2019). https://doi.org/10.1145/3306618.3314256, https://dl.acm.org/doi/10.1145/3306618.3314256 - [23] Guarini, M.: Particularism and the classification and reclassification of moral cases. IEEE Intelligent Systems 21 (4), 22–28 (2006). https://doi.org/10.1109/MIS.2006.76 - [24] Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., Steinhardt, J.: Aligning AI With Shared Human Values (2021) - [25] Howard, D., Muntean, I.: Artificial Moral Cognition: Moral Functionalism and Autonomous Moral Agency. In: Powers, T.M. (ed.) Philosophy and Computing: Essays in Epistemology, Philosophy of Mind, Logic, and Ethics, pp. 121–159. Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3-319-61043-6_7, https://doi.org/10.1007/978-3-319-61043-6_7, iSSN: 0007-0882 - [26] Hursthouse, R., Pettigrove, G.: Virtue Ethics. In: Zalta, E.N. (ed.) The {Stanford} Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University (2018), medium: \url{https://plato.stanford.edu/archives/win2018/entries/ethics-virtue/} - [27] Jiang, L., Hwang, J.D., Bhagavatula, C., Bras, R.L., Forbes, M., Borchardt, J., Liang, J., Etzioni, O., Sap, M., Choi, Y.: Delphi: Towards Machine Ethics and Norms (1), 1–42 (2021), http://arxiv.org/abs/2110.07574, arXiv: 2110.07574 - [28] Kazil, J., Masad, D., Crooks, A.: Utilizing Python for Agent-Based Modeling: The Mesa Framework. In: Thomson, R., Bisgin, H., Dancy, C., Hyder, A., Hussain, M. (eds.) Social, Cultural, and Behavioral Modeling. pp. 308–317. Springer International Publishing, Cham (2020) - [29] Kraut, R.: Aristotle’s Ethics. In: Zalta, E.N., Nodelman, U. (eds.) The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, fall 2022 edn. (2022), https://plato.stanford.edu/archives/fall2022/entries/aristotle-ethics/ - [30] MacKenzie, M.: Buddhism and the Virtues, vol. 1. Oxford University Press (Dec 2017). https://doi.org/10.1093/oxfordhb/9780199385195.013.18, https://academic.oup.com/edited-volume/28109/chapter/212224702 - [31] Morrison, E.W.: Doing the job well: An investigation of pro-social rule breaking. Journal of Management 32 (1), 5–28 (2006). https://doi.org/10.1177/0149206305277790 - [32] Nafsika Athanassoulis: Virtue Ethics (2023), https://iep.utm.edu/virtue/ - [33] Nallur, V.: Landscape of Machine Implemented Ethics. Science and Engineering Ethics 26 (5), 2381–2399 (Oct 2020). https://doi.org/10.1007/s11948-020-00236-y, https://link.springer.com/10.1007/s11948-020-00236-y, publisher: Springer Science and Business Media B.V. - [34] Noothigattu, R., Gaikwad, S., Awad, E., Dsouza, S., Rahwan, I., Ravikumar, P., Procaccia, A.: A Voting-Based System for Ethical Decision Making. Proceedings of the AAAI Conference on Artificial Intelligence 32 (1) (Apr 2018). https://doi.org/10.1609/aaai.v32i1.11512, https://ojs.aaai.org/index.php/AAAI/article/view/11512, section: AAAI Technical Track: Human-Computation and Crowd Sourcing - [35] Pirni, A., Balistreri, M., Capasso, M., Umbrello, S., Merenda, F.: Robot Care Ethics Between Autonomy and Vulnerability: Coupling Principles and Practices in Autonomous Systems for Care. Frontiers in Robotics and AI 8, 654298 (Jun 2021). https://doi.org/10.3389/frobt.2021.654298, https://www.frontiersin.org/articles/10.3389/frobt.2021.654298/full - [36] Pontier, M.A., Hoorn, J.F.: Toward machines that behave ethically better than humans do. Proceedings of the Annual Meeting of the Cognitive Science Society (2012) - [37] Ramanayake, R., Nallur, V.: A Computational Architecture for a Pro-Social Rule Bending Agent. In: First International Workshop on Computational Machine Ethics held in conjunction with 18th International Conference on Principles of Knowledge Representation and Reasoning KR 2021 (CME2021) (Nov 2021). https://doi.org/10.5281/ZENODO.6470437, https://zenodo.org/record/6470437 - [38] Ramanayake, R., Nallur, V.: A Small Set of Ethical Challenges for Elder-Care Robots. In: Hakli, R., Mäkelä, P., Seibt, J. (eds.) Frontiers in Artificial Intelligence and Applications. IOS Press (Jan 2023). https://doi.org/10.3233/FAIA220605, https://ebooks.iospress.nl/doi/10.3233/FAIA220605 - [39] Ramanayake, R., Nallur, V.: Implementing Pro-social Rule Bending in an Elder-Care Robot Environment. In: Ali, A.A., Cabibihan, J.J., Meskin, N., Rossi, S., Jiang, W., He, H., Ge, S.S. (eds.) Social Robotics, vol. 14454, pp. 230–239. Springer Nature Singapore, Singapore (2024). https://doi.org/10.1007/978-981-99-8718-4_20, https://link.springer.com/10.1007/978-981-99-8718-4_20, series Title: Lecture Notes in Computer Science - [40] Ramanayake, R., Wicke, P., Nallur, V.: Immune moral models? Pro-social rule breaking as a moral enhancement approach for ethical AI. AI & SOCIETY (May 2022). https://doi.org/10.1007/s00146-022-01478-z, https://link.springer.com/10.1007/s00146-022-01478-z - [41] Russell, S.J.: Human compatible: artificial intelligence and the problem of control. Viking, New York? (2019) - [42] Schoenborn, J., Weber, R., Aha, W., Cassens, J., Althoff, K.D.: Explainable Case-Based Reasoning: A Survey. In: 35th AAAI Conference on Artificial Intelligence - A Virtual Conference; Explainable Agency in Artificial Intelligence Workshop (Feb 2021) - [43] Sharkey, A., Sharkey, N.: Granny and the robots: Ethical issues in robot care for the elderly. Ethics and Information Technology 14 (1), 27–40 (Jul 2012). https://doi.org/10.1007/s10676-010-9234-6, https://link.springer.com/article/10.1007/s10676-010-9234-6, publisher: Springer - [44] Sinnott-Armstrong, W.: Consequentialism. In: Zalta, E.N. (ed.) The {Stanford} Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University (2019), medium: \url{https://plato.stanford.edu/archives/sum2019/entries/consequentialism/} - [45] Solaiman, I., Dennison, C.: Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets. In: Advances in Neural Information Processing Systems. vol. 34, pp. 5861–5873. Curran Associates, Inc. (2021), https://proceedings.neurips.cc/paper/2021/hash/2e855f9489df0712b4bd8ea9e2848c5a-Abstract.html - [46] Thornton, S.M., Pan, S., Erlien, S.M., Gerdes, J.C.: Incorporating Ethical Considerations into Automated Vehicle Control. IEEE Transactions on Intelligent Transportation Systems 18 (6), 1429–1439 (Jun 2017). https://doi.org/10.1109/TITS.2016.2609339, publisher: Institute of Electrical and Electronics Engineers Inc. - [47] Van Dang, C., Tran, T.T., Gil, K.J., Shin, Y.B., Choi, J.W., Park, G.S., Kim, J.W.: Application of soar cognitive agent based on utilitarian ethics theory for home service robots. In: 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI). pp. 155–158 (Jun 2017). https://doi.org/10.1109/URAI.2017.7992698 - [48] Van Wynsberghe, A.: Social robots and the risks to reciprocity. AI & SOCIETY 37 (2), 479–485 (Jun 2022). https://doi.org/10.1007/s00146-021-01207-y, https://link.springer.com/10.1007/s00146-021-01207-y - [49] Vanderelst, D., Winfield, A.: An architecture for ethical robots inspired by the simulation theory of cognition. Cognitive Systems Research 48, 56–66 (2018). https://doi.org/10.1016/j.cogsys.2017.04.002, https://doi.org/10.1016/j.cogsys.2017.04.002, publisher: The Authors - [50] Wallach, W., Allen, C., Smit, I.: Machine morality: Bottom-up and top-down approaches for modelling human moral faculties. AI and Society 22 (4), 565–582 (Apr 2008). https://doi.org/10.1007/s00146-007-0099-0 - [51] van Wynsberghe, A., Robbins, S., Robbins scott, S., van Wynsberghe, A., Robbins, S.: Critiquing the Reasons for Making Artificial Moral Agents. Science and Engineering Ethics 2018 25:3 25 (3), 719–735 (Feb 2018). https://doi.org/10.1007/S11948-018-0030-8, https://link.springer.com/article/10.1007/s11948-018-0030-8, publisher: Springer - [52] Yilmaz, L., Franco-Watkins, A., Kroecker, T.S.: Computational models of ethical decision-making: A coherence-driven reflective equilibrium model. Cognitive Systems Research 46, 61–74 (Dec 2017). https://doi.org/10.1016/j.cogsys.2017.02.005, https://www.sciencedirect.com/science/article/pii/S1389041716301218