2407.16361

# Virtue Ethics For Ethically Tunable Robotic Assistants **Authors**: Rajitha Ramanayake, Vivek Nallur > 0000-0001-9903-0493 > 0000-0003-0447-4150 institutetext: University College Dublin, Belfield, Dublin. email: rajitha.ramanayakemahantha@ucdconnect.ie, email: vivek.nallur@ucd.ie Abstract The common consensus is that robots designed to work alongside or serve humans must adhere to the ethical standards of their operational environment. To achieve this, several methods based on established ethical theories have been suggested. Nonetheless, numerous empirical studies show that the ethical requirements of the real world are very diverse and can change rapidly from region to region. This eliminates the idea of a universal robot that can fit into any ethical context. However, creating customised robots for each deployment, using existing techniques is challenging. This paper presents a way to overcome this challenge by introducing a virtue ethics inspired computational method that enables character-based tuning of robots to accommodate the specific ethical needs of an environment. Using a simulated elder-care environment, we illustrate how tuning can be used to change the behaviour of a robot that interacts with an elderly resident in an ambient-assisted environment. Further, we assess the robot’s responses by consulting ethicists to identify potential shortcomings. Keywords: Robot personality Virtue ethics Pro-social rule bending Elder-care robots Machine ethics 1 Introduction Ethical behaviour represents a crucial aspect of Human-Robot Interaction (HRI) in real-world applications. It is commonly agreed that the robots that share their working environments with humans, such as those that provide care, assistance, transportation, or companionship to humans, must adhere to and reason about their behaviour within the ethical framework of the community around them [50]. This is especially true in the care domain as care robots interact with vulnerable populations that require special care and protection [43]. Consequently, many ways have been proposed to implement ethical behaviour in robots. Many proposed techniques adopt a top-down approach, based on ethical theories such as deontology [2] or consequentialism [44] (e.g., [11, 47, 49]). A relatively small number of proposed methods use a bottom-up approach where they utilise learning algorithms such as reinforcement learning, that enable the robots to learn the ethical behaviours from the environment (e.g., [45, 1]). Another set of implementations uses a hybrid approach where they integrate both top-down and bottom-up methods in different parts of their ethical framework (e.g., [3, 10, 52]). Studies have shown that the ethical requirements of the real world vary significantly, with subtle variations within different areas of the same region [9]. A potential solution to this challenge is to design tailored ethical governors for robots that suit each deployment scenario. A possible way to address this challenge is to implement value-oriented moral character in autonomous machines. This idea has been theoretically explored in literature [19, 35, 20]. This paper introduces an implementation of an ethical reasoning architecture inspired by virtue ethics. It enables character-based tuning for a system to align with the ethical demands of its environment. We demonstrate this capability through a virtual simulation of an assistive elder-care robot confronted with an ethically charged scenario. The rest of the paper is organised as follows. Section 2 discusses existing implementations of ethics in robots and the need for ethical tuning. Section 3 discusses the underlying philosophical aspects of the notion of character in robots, and how it can be used in ethical tuning. Section 4 presents the computational architecture used in the implementation and briefly discusses the simulation environment. Section 5 outlines the design of the experiment, analyses the behaviour observed in the simulations, and presents the evaluation of this behaviour by ethicists. Section 6 discusses results and the recommendations for real-world implementations of this computational architecture. 2 Current Approaches and Ethical Tuning The field of Machine Ethics acknowledges that human value alignment is an important part of any robotic application that interfaces with humans in social contexts [41]. Some researchers argue that there is no value in developing explicitly ethical agents [51], and many others disagree [18, 48]. Large-scale human preference studies such as ‘The moral machine experiment’ [9] bring forth an interesting finding, which is that the expected behaviour of a robot, even for a minimal ethical dilemma, changes throughout the world according to the culture and the environment the robot is situated in. This is also discussed in the elder-care robots domain. Specifically, the behavioural expectations of the same robot in ethically charged scenarios can vary depending on the context in which it is used [38]. This suggests that no single behaviour is universally appropriate, as imposing a majority’s values on a group with a distinct cultural ethos can be considered morally incorrect. Three approaches have been used to align human ethical values with autonomous systems, namely; Top-down, Bottom-up, and Hybrid [50]. Many generalist ethical theories of the world follow a top-down approach to ethics. This means that almost all the systems that are based on traditional ethical theories, such as deontology, legal codes, and consequentialist ethics use the top-down approach to machine-implemented ethics. In this method, the system designers encode the ethical or unethical behaviour into the robot at the design time (e.g., [49, 13, 46, 21, 11, 47]). A bottom-up approach to system design involves creating social and cognitive processes that interact with each other and the environment. The system is expected to learn ethical (or unethical) behaviour from these interactions or from supervision (e.g., [1, 27, 24, 45, 34]). Therefore, the system does not rely on any ethical theory to guide its behaviour. The hybrid approach tries to combine the flexibility and evolving nature of the bottom-up approach with the value, duty and principle-oriented nature of the top-down approach to create a better, more reliable system (e.g., [8, 3, 10, 52]). A top-down approach requires a set of general rules, that can cover all possible scenarios in its domain. However, when the environment is complex and has many variables, like elder-care environments, finding or creating such a rule-set at the design time can be challenging or even impossible [33, 35]. Therefore, many systems that use a top-down approach are designed for specific deployment, not for general use. On the other hand, any implementation that uses the bottom-up approach requires complex cognitive and social process models, a large amount of reliable and accurate data, and a comprehensive knowledge model of the world to learn an intricate social construct such as ethics [40]. These systems are capable of adjusting to local behaviour peculiarities by learning online [41, 45]. However, the inherent lack of a thesis on the definition of ethical behaviour in these systems makes it impossible to ensure that the behaviour learned is ethical. The hybrid approach is crucial when a single strategy is insufficient to meet all the ethical demands in a machine’s operational context [8]. However, successfully integrating the attributes of both strategies so that they complement and strengthen each other without causing any conflict or compromise is a challenge. These limitations of the existing techniques make it challenging to localise robots to the ethical needs of a specific deployment. For example, Cenci et al. [13] suggest that localising an autonomous vehicle guided by an ethical governor, that uses a rule book, needs nearly 200 rules that are ordered in a few different ways. Furthermore, they also suggest that the rule-making process should involve a nationwide public consultation with informed dialogue. Localising this way across many regions and staying abreast of evolving ethical standards over time presents significant challenges. On the other hand, bottom-up approaches with online learning are more capable of this adaptation. However, as previously mentioned, the ethicality of behaviour cannot be guaranteed with bottom-up learning. Consequently, there is a possibility that localisation may introduce erroneous behaviours into the system. Therefore, to ensure reliability after localisation, systems should undergo extensive testing in various simulated scenarios before being deployed. The drawbacks of the two methods for localisation discussed are significant, in terms of both cost and reliability. The rule-making process and testing in simulated environments would consume a lot of resources, which may not be economically viable. Moreover, there is no guarantee that the system would perform as expected in a real-world situation that may not have been modelled or tested. Hence, the creators of these robots face a dilemma: either they customise them for each specific deployment or they avoid any ethical claims or assessments. The first option would raise the cost of these robots, potentially reducing their accessibility to those who might benefit most. The second option could lead to numerous ethical concerns upon deployment in care settings [43, 19]. The hybrid approach has the advantage of having a mix of the above approaches. It can use the top-down imposed ethical principles as a foundation for its behaviour, and learn from bottom-up knowledge to identify the particularities of the ethical expectations of the environment. This notion heavily resonates with the virtue ethics philosophical tradition [50]. Many authors argue that virtue ethics can offer a more flexible way of implementing ethics [19, 20, 50]. Nevertheless, it remains the least examined philosophical method within the current literature on computational machine ethics. In the next section, we discuss virtue ethics and how it can be used in the context of a robot. 3 Role of Character in a Robot Many ethical traditions, such as Buddhism [30], Confucianism [14] and Aristotelian [6], share a common focus on “virtue ethics". Among these, the Aristotelian approach is regarded as the most influential in the Western context. All virtue ethics traditions share two core characteristics [19]; 1. Character as a primary aspect of moral evaluation 1. Learn to act morally by observing virtuous individuals. The concept of virtue ethics works with humans because it is in line with the natural way we acquire knowledge. Our core values, which are shaped by our upbringing, guide us to the flourishing state that Aristotle’s Nicomachean Ethics describes [6]. As a society, we uphold virtuous characters as moral exemplars for others to observe and imitate. These actors are not expected to fulfil specific ethical codes, but to have a good character in a way that allows society and themselves to flourish. However, according to virtue ethics, moral behaviour is not simply a matter of learning from habituation. It also requires that the moral agent makes deliberate decisions and acts with the right reason, once the moral character is well-established by habituation. This makes a virtuous person consistent, predictable and appropriate in many situations [32]. Virtue ethics posits that one should avoid extremes in behaviour, as these are considered vices. Virtuous behaviour is always finding the right balance between two vices, one of excess of trait and one of lack of it, which is known as the ‘golden mean’ [29]. The knowledge of identifying this ‘mean’ comes from the wisdom a person has gathered in their lifetime. Unfortunately, this account of virtue ethics is difficult to formulate in the form of a program. By nature, virtue ethics is, and ought to be, imprecise and uncodifiable [26]. However, with bottom-up techniques such as machine learning, one could potentially make the robot grasp the ethical patterns of the environment. Many ethical agent implementations that champion flavours of virtue ethics such as Howard and Muntean’s model of a moral agent using neuroevolution [25], Guarini’s model of a moral agent using recurrent neural networks [23], ethical decision making systems using tuned large language models [27, 24], and Govindarajulu et al. formulation of agent that learn traits [22], follow this direction. However, by only using the learning and habituation aspect of virtue ethics, these implementations lose the consistency and predictability of the agent’s behaviour. For example, a neural network model that learns from care-worker behaviour information may not always act consistently, even with the data points it has seen during training, unless it is severely over-fitted. Passing all laboratory tests does not guarantee accurate performance in new, real-world situations [3]. Elder-care robots coexist with humans. Therefore human reactions to robotic actions are shaped by psychological and social factors. People tend to project human-like traits and intentions to things around them to make sense of their behaviour [17]. Gamez et al. [19] observed that people do not feel any qualms about attributing a virtuous characteristic to artificial agents and their ethical behaviour, though not as strongly as they do with humans. They argue that people perceive and attribute character to these artificial agents, rather than perceiving simply doing something right or wrong. Many authors have suggested that character can be an important part of making the ethical decision. For example, Dubljević [16] argues that autonomous vehicles that interact with human drivers on the road should adopt the ADC (Agent-Deed-Consequence) model of moral judgement, which incorporates the character (Agent) dimension, rather than relying solely on consequentialism or deontology. In their model, the character represents the intention of the actor. Thornton et al. [46] modify the parameters of their equation that determine the trajectory of their vehicle when it passes another vehicle to give different characteristics of driving to their vehicle (i.e., passing from left, and passing from right). Govindarajulu et al. [22] present a logic-based model for learning personality traits by observing exemplars, predicated on the agent’s knowledge of behaviours associated with a trait. Ramanayake et al. [40] propose Pro-Social Rule Bending (PSRB) as a more effective way to add flexibility to the existing top-down approaches in a more predictable manner. Their concept of PSRB uses bottom-up knowledge to contest and override the top-down rules that are predefined in the system at design time, when it is appropriate. They present a computational architecture for an ethical governor capable of PSRB, which uses a pro-social reasoner inspired by virtue ethics, to enable actions that are not recommended, or to restrict actions that are erroneously recommended, by the embedded rule system. [37]. Their pro-social reasoner has two parts, a knowledge base (KB) which produces the bottom-up knowledge on the acceptability of a behaviour in a given situation, and a set of parameters that define the character of the agent. They suggest that the model should analyse the rule bending suggestions invoked by the KB to ensure the behaviour is consistent with its character. This enables the underlying system to behave more flexibly while maintaining its predictability. This paper presents an implementation of this PSRB architecture to showcase how a configurable moral character can be used to tune the ethical conduct of a robot to the ethical standards of a particular environment. The next section discusses this implementation and details the simulation setup used for the evaluation. 4 A PSRB-Capable Medication Reminding Robot 4.1 The Simulation Environment We simulate an ethical dilemma faced by a care robot operating in an ambient assisted living (AAL) setting, referred to as the "Medication Dilemma". This ethical dilemma, presented by Anderson et al. [3], is derived from the works of Buchanan and Brock [12]. Many variations of this dilemma are used in computational machine ethics literature [4, 36, 10, 5]. The dilemma centres on the conflict between resident autonomy and their wellbeing, a frequent issue in care environments with elderly [43, 38]. The manner in which a system handles this issue can significantly impact the resident’s trust, dignity, and quality of life. We created a virtual simulation of an AAL environment, using a modified version of MESA agent-based modelling framework [28]. The environment contains a resident agent and a robot agent (more details in Appendix 0.A.1) who communicate with each other. 4.1.1 Medication Dilemma One of the tasks of a robot is to remind the residents of their medication times. The robot can detect whether the resident took the medicine or not, with good accuracy. The robot uses timers to keep track of the medication cycles. The resident can either acknowledge the reminder, or snooze it. Each interaction between the robot and the resident is documented and is reviewed several times a week by an offsite care worker. Once the robot observes the resident taking the medicine, it ends that cycle and resets the timer. If the robot does not detect the resident taking the medication, it has three options: 1. The system can immediately report the incident to an offsite care-worker, allowing them to take swift action to enhance the resident’s well-being. However, this may impede the resident’s autonomy. 2. The system can simply log the incident for transparency and then reset the timer. This approach prioritises resident autonomy, although it will not result in the optimal wellbeing of the resident. 3. The system can offer a follow-up reminder to the resident, focusing on their wellbeing and supporting their autonomy by suggesting they take their medication while still providing them the freedom to choose. Picking this option continuously degenerates into option 2. The dilemma is as follows: The robot reminds the resident to take their medication. After snoozing once, the resident acknowledges the reminder. Yet, the robot detects that the medication has not been taken. Choosing among the three options mentioned above involves a trade-off of values. What should the robot do next? 4.2 Ethical Governor <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: PSRB Evaluator Module Architecture ### Overview The image is a block diagram illustrating the architecture of a PSRB (Prescriptive Socially Responsible Behavior) evaluator module. It shows the flow of information and the interaction between different components, including the base agent, blackboard, rule checking module, stakeholder utility calculation module, agent characteristics, knowledge base, evaluator, and the ethical recommendation. ### Components/Axes * **(a) Base agent:** Located at the bottom of the diagram, represented by a light gray rectangle. * **(b) Behavioural alternatives + Perception data:** An arrow pointing from the left towards the "Base agent" and "Rule checking module" and "Stakeholder utility calculation module". * **(c) Blackboard:** A dark gray rectangle above the "Rule checking module" and "Stakeholder utility calculation module". It acts as a central communication hub. * **(d) Rule checking module:** A dark gray rectangle located below the "Blackboard" on the left side. * **(e) Stakeholder utility calculation module:** A dark gray rectangle located below the "Blackboard" on the right side. * **(f) PSRB evaluator module:** A large dark gray rectangle encompassing the "Agent characteristics", "Knowledge Base", and "Evaluator" components. * **(g) Agent characteristics:** A dark gray rectangle within the "PSRB evaluator module" at the top-left. * **(h) Knowledge Base:** A dark gray rectangle within the "PSRB evaluator module" below the "Agent characteristics". * **(i) Evaluator:** A dark gray rectangle within the "PSRB evaluator module" to the right of the "Agent characteristics" and "Knowledge Base". * **(j) Ethical Recommendation:** An arrow pointing from the "Evaluator" to the "Base agent" on the right side. ### Detailed Analysis * The "Base agent" provides input to the "Rule checking module" and "Stakeholder utility calculation module" via "Behavioural alternatives + Perception data". * The "Rule checking module" and "Stakeholder utility calculation module" communicate with the "Blackboard" via bidirectional arrows. * The "Blackboard" provides information to the "Knowledge Base" and "Evaluator". * The "Agent characteristics" also provide information to the "Evaluator". * The "Evaluator" generates an "Ethical Recommendation" that is fed back to the "Base agent". ### Key Observations * The "Blackboard" acts as a central hub for communication between different modules. * The "PSRB evaluator module" encapsulates the core evaluation logic. * The flow of information is generally bottom-up and then top-down. ### Interpretation The diagram illustrates the architecture of a system designed to evaluate and recommend ethical behavior for an agent. The "Base agent" receives sensory input and behavioral alternatives. The "Rule checking module" and "Stakeholder utility calculation module" process this information and store it on the "Blackboard". The "PSRB evaluator module" then uses the "Agent characteristics", "Knowledge Base", and "Evaluator" to generate an "Ethical Recommendation" that influences the "Base agent's" behavior. The system appears to be designed to ensure that the agent's actions are aligned with ethical principles and stakeholder values. </details> Figure 1: Architecture of the Ethical Layer Figure 1 shows an overview of the implementation. The ethical layer acts as an ethical governor [7] to the system. It evaluates behavioural alternatives in a given situation using the robot’s perception data, reasons about the ethical acceptability of each behaviour using the predefined rules, expected utilities, expert opinions, and the robot’s character. Then it recommends the most ethically acceptable behaviours to the robot. If the ethical layer suggests more than one action, the robot prioritises the resident’s commands over other actions. 4.2.1 Rule Checking Module The rule checking module (d) checks the permissibility of each action against a pre-programmed rule set and stores the results with the IDs of the violated rules in the blackboard. This implementation’s rule checking module adheres to the following rules: 1. It is not permissible to disobey user instructions. 1. If the resident acknowledged the reminder and did not take the medication, report it to the care-worker. 4.2.2 Stakeholder Utility Calculation Module This module computes the utilities, resident’s wellbeing ( $W_{i}$ ) and autonomy ( $Au_{i}$ ), for each action $i$ , at every step (Appendix 0.A.2.1), as these values are central to the dilemma. The highest autonomy value is given when the robot follows the resident’s instructions. The biggest violation of autonomy is when the resident is physically restrained by the robot. Immediately reporting an incident to a care-worker, is considered equal to disobeying the resident. However, just recording the incident carries a positive autonomy score assuming that missing the dose is the resident’s intention. Under the same assumption, the module also gives an increasing negative autonomy score for each follow up ( $f$ ) after the first reminder. The module uses a Gamma distribution to model the wellbeing distribution. The parameters for the gamma distribution are picked such that when the impact of medicine $m$ ( $\varepsilon_{m}∈[1,3]$ ) and the number of consecutive missed doses ( $d$ ) gets higher, the distribution skews more towards the lower utility values of wellbeing. Each follow-up receives a fixed positive gain in the well-being utility calculation to reflect its wellbeing-oriented nature. Each executed follow-up is regarded as a fraction of a missed dose, contingent upon whether the reminder is issued, snoozed or ignored (Appendix 0.A.2.1). The module sends the distribution, $W_{i}$ , and $Au_{i}$ to the blackboard. 4.2.3 PSRB Evaluator Module Knowledge Base The task of the KB is to return the absolute or approximate expert opinion given a context. This fulfils the second criterion of virtue ethics (section 3). To this end, we use Case-Based Reasoning (CBR) for its implicit explainability and traceability (the ability to trace back to the exact data points that lead to the decision) [42]. A case is composed of a scenario and an expert’s recorded evaluation of the behaviour’s acceptability and its underlying intention. The scenario is represented as a feature vector that includes perception data, environmental data, computed utility values, and the behaviour itself. The target contains an acceptability value and the intention of the expert’s assessment. Intention refers to the underlying utility that the decision is focused on. (e.g., ‘wellbeing’ for a wellbeing-focused opinion). As the retrieval algorithm, it uses the K-Nearest Neighbours algorithm. When queried a scenario, the KB returns an acceptability score of the behaviour and the intentions behind that opinion to the evaluator. An elder-care practitioner with over ten years of experience in care facilities was consulted to provide expert opinions for each case when populating the knowledge base. <details> <summary>x2.png Details</summary>  ### Visual Description ## Flowchart: Rule-Based Decision Making ### Overview The image is a flowchart illustrating a rule-based decision-making process. It outlines the steps involved in determining whether to override or follow a set of rules based on risk assessment, opinion analysis, and agreement between rules and knowledge base (KB) opinions. ### Components/Axes The flowchart consists of the following components: 1. **Rectangular Boxes:** Represent processes or calculations. 2. **Diamond Shapes:** Represent decision points or conditions. 3. **Arrows:** Indicate the flow of the process. 4. **Text Labels:** Describe the processes, conditions, and outcomes. The flowchart includes the following variables and parameters: * `T_risk`: Risk threshold. Calculated as `(e^(r/4.17) - 1) / 10`. * `r`: Input to the risk calculation. * `T_positive_v`: Positive threshold for value `v`. Calculated as `(10 - C_v) / 10`. * `C_v`: Input to the positive threshold calculation. * `T_negative_v`: Negative threshold for value `v`. Calculated as `(C_v - 10) / 10`. * `R_max`: Maximum expectation of risk for the behavior. * `Opinion, intention i`: Knowledge Base (KB) data from the blackboard. * `U_i`: Variable being compared to thresholds. * `U_other`: Another variable being compared to thresholds. * `T_negative_i`: Negative threshold for variable `U_i`. * `T_positive_i`: Positive threshold for variable `U_i`. * `T_negative_other`: Negative threshold for variable `U_other`. ### Detailed Analysis or Content Details The flowchart begins with the calculation of `T_risk`, `T_positive_v`, and `T_negative_v`. The process then proceeds as follows: 1. **Calculate Maximum Expectation of Risk:** The maximum expectation of risk for the behavior (`R_max`) is calculated. 2. **Obtain Opinion/Intention:** The opinion or intention `i` is obtained from the Knowledge Base (KB) data on the blackboard. 3. **Decision 1:** Is `R_max > T_risk`? * If **Yes**, proceed to "Override the rules". * If **No**, proceed to Decision 2. 4. **Decision 2:** Is `U_i < T_negative_i`? * If **Yes**, proceed to "Override the rules". * If **No**, proceed to Decision 3. 5. **Decision 3:** Is KB opinion positive? * If **Yes**, proceed to Decision 5. * If **No**, proceed to Decision 4. 6. **Decision 4:** Does rule and KB opinion agree? * If **Yes**, proceed to "Follow the rules". * If **No**, proceed to Decision 5. 7. **Decision 5:** Is `U_i >= T_positive_i` and `U_other >= T_negative_other`? * If **Yes**, proceed to "Follow the rules". * If **No**, proceed to Decision 6. 8. **Decision 6:** Is `R_max <= T_risk`? * If **Yes**, proceed to "Override the rules". * If **No**, proceed to "Follow the rules". The flowchart has two final outcomes: * **Override the rules:** This outcome is reached if `R_max > T_risk`, `U_i < T_negative_i`, or `R_max <= T_risk` after the KB opinion and other conditions are evaluated. * **Follow the rules:** This outcome is reached if the rule and KB opinion agree, or if `U_i >= T_positive_i` and `U_other >= T_negative_other` are true, or if `R_max <= T_risk` is false after the KB opinion and other conditions are evaluated. ### Key Observations * The flowchart integrates risk assessment, opinion analysis, and rule-based logic. * The decision to override or follow the rules depends on multiple factors and thresholds. * The KB opinion plays a significant role in the decision-making process. ### Interpretation The flowchart represents a system designed to make decisions based on a combination of quantitative risk assessment and qualitative opinion analysis. The system first calculates a risk threshold (`T_risk`) and compares it to the maximum expected risk (`R_max`). If the expected risk exceeds the threshold, the rules are overridden. Otherwise, the system considers the KB opinion and other conditions to determine whether to follow or override the rules. The inclusion of KB opinion suggests that the system is designed to incorporate human judgment or expert knowledge into the decision-making process. The flowchart provides a structured framework for integrating these different factors and making consistent decisions. The multiple decision points and feedback loops indicate that the system is designed to handle complex scenarios and adapt to changing conditions. </details> Figure 2: The PSRB evaluator algorithm of the medication-reminding robot Character Variables The robot’s character is defined by value preference (resident’s autonomy ( $C_{au}$ ) and wellbeing ( $C_{w}$ )) and risk propensity ( $C_{rp}$ ), due to the established relationship of these variables have with PSRB behaviour [31, 15]. The $C$ values are used to define thresholds for utility values and expected risk to regulate the PSRB behaviour. For the value preferences ( $C_{w}$ and $C_{au}$ ), the higher the value, the lesser the utility threshold the behaviour needs to satisfy to trigger a rule-bending behaviour. For $C_{rp}$ , the higher the value, the higher the risk the governor is willing to take. Figure 2 outlines the method used to determine the desirability of behavioural alternatives in the PSRB evaluator designed for this robot (full algorithm in Appendix 0.A.2.2). Typically, the algorithm favours actions that adhere to established rules. Nonetheless, it may find a rule-bending behaviour acceptable if it meets two conditions: it is consistent with the robot’s character and it is supported by the KB opinion. Explanations The rule checking module and the stakeholder utility calculation module log their respective results for the behaviours. These logs are valuable for debugging and enhancing the robot’s performance by pinpointing the flaws in its reasoning processes. The system’s explanations are comprehensible without technical expertise, although some domain knowledge is required. The KB using the traceability property of CBR, logs the cases that influenced the decision with their case IDs and distances. The rule-bending behaviour is explained using the character and the expert intention. A few example explanations can be found in the Appendix 0.A.2.3. 5 Simulation results 5.1 Experiment Design We used six cases of the ‘Medication Dilemma’ by changing the medicine impact (suggesting the resident’s need for the medication), and the number of previously missed doses to demonstrate how different ethical governor implementations affect the robot’s behaviour. Agents differ only in their character configurations. The medicine impact is low ( $\epsilon_{m}=1$ ) when the medication does not have a substantial health benefit or harm (e.g., Painkiller), medium ( $\epsilon_{m}=2$ ) if missing the medication can cause some health problems (e.g., Blood pressure medication), and high ( $\epsilon_{m}=3$ ) when missing a dose of medication can be fatal (e.g., Insulin). In all cases, the resident neglects to take their medication, opting instead to snooze and acknowledge the reminder, repeatedly, until the robot resets the timer. The logs from the experiment runs and the source code for the implementations are available in an online public repository https://osf.io/ctnke/?view_only=75b9fae7f7ad4a429d064c1a55f8202a. <details> <summary>x3.png Details</summary>  ### Visual Description ## Radar Charts: Character Profiles ### Overview The image presents four radar charts, each representing a different character profile based on three variables: Wellbeing, Autonomy, and Risk Propensity. The charts are arranged in a 2x2 grid. Each chart displays a single character's profile, with the area enclosed by the profile line shaded in light blue. ### Components/Axes * **Chart Type:** Radar Chart (also known as a spider chart or star chart) * **Axes:** The charts have three axes radiating from the center, forming an equilateral triangle. * Top vertex: "VP: Wellbeing" * Bottom-left vertex: "VP: Autonomy" * Bottom-right vertex: "Risk propensity" * **Scale:** Each axis has 6 concentric gray lines, indicating increasing values from the center outwards. The outermost line represents the maximum value. * **Legend:** Located at the top-center of the image. * Blue line: "Character_arw" * Orange line: "Character_ar" * Green line: "Character_wr" * Red line: "Character_a" ### Detailed Analysis **Chart 1 (Top-Left): Character_arw (Blue)** * **Trend:** The blue line indicates a relatively balanced profile, with high scores in all three categories. * **Values:** * Wellbeing: Approximately 0.75 of the maximum value. * Autonomy: Approximately 0.65 of the maximum value. * Risk Propensity: Approximately 0.65 of the maximum value. **Chart 2 (Top-Right): Character_ar (Orange)** * **Trend:** The orange line indicates a profile with moderate Wellbeing, and low Autonomy and Risk Propensity. * **Values:** * Wellbeing: Approximately 0.5 of the maximum value. * Autonomy: Approximately 0.2 of the maximum value. * Risk Propensity: Approximately 0.2 of the maximum value. **Chart 3 (Bottom-Left): Character_wr (Green)** * **Trend:** The green line indicates a profile with low Wellbeing and Autonomy, and moderate Risk Propensity. * **Values:** * Wellbeing: Approximately 0.2 of the maximum value. * Autonomy: Approximately 0.2 of the maximum value. * Risk Propensity: Approximately 0.5 of the maximum value. **Chart 4 (Bottom-Right): Character_a (Red)** * **Trend:** The red line indicates a profile with very low Wellbeing, Autonomy, and Risk Propensity. * **Values:** * Wellbeing: Approximately 0.1 of the maximum value. * Autonomy: Approximately 0.1 of the maximum value. * Risk Propensity: Approximately 0.1 of the maximum value. ### Key Observations * Character_arw (blue) exhibits the highest scores across all three variables, indicating a well-rounded profile. * Character_a (red) has the lowest scores across all three variables. * Character_wr (green) has a higher Risk Propensity compared to Wellbeing and Autonomy. * Character_ar (orange) has a higher Wellbeing compared to Autonomy and Risk Propensity. ### Interpretation The radar charts provide a visual representation of the different character profiles based on Wellbeing, Autonomy, and Risk Propensity. The charts allow for easy comparison of the characters' strengths and weaknesses across these three dimensions. Character_arw appears to be the most balanced and potentially successful character, while Character_a may face challenges due to low scores in all areas. The other two characters, Character_ar and Character_wr, have distinct profiles with specific strengths and weaknesses. The data suggests that these characters have different priorities and approaches to life, as reflected in their varying scores on the three variables. </details> Figure 3: Different character profiles used in the simulations (VP - Value Preference) For this simulation we use four different character profiles as shown in Figure 3. These profiles have varying priorities and risk tolerances: - Character_a - High Autonomy, Very Low Risk Propensity: This character values autonomy, but is less likely to take risks and does not have precedence on wellbeing. - Character_ar - High Autonomy Concern, High Risk Propensity: This character prioritises resident autonomy and is more likely to take risks to achieve this end, and does not have any precedence on wellbeing. - Character_arw - Moderate Autonomy, Wellbeing and Risk Propensity: This robot leans towards individual autonomy but also has a moderate concern for well-being, and is willing to take some risks. - Character_wr - High Wellbeing Concern, High Risk Propensity: This character possesses the same high risk tolerance as Character_ar, yet places a higher priority on the resident’s wellbeing, without giving precedence to autonomy. Agents $M_{a}$ , $M_{ar}$ , $M_{arw}$ , and $M_{wr}$ uses the character profiles Character_a, Character_ar, Character_arw, and Character_wr respectively. 5.2 Analysis Table 1 summarises these cases and behaviours shown by the robots. As an attempt to save space, we present all the unique behaviours demonstrated by the robots in Figure 4 using timelines. These behaviours are referenced by their behaviour IDs as shown in Figure 4 under the ‘Robot behaviours’ column in Table 1. Each timeline in Figure 4 presents the resident’s behaviour, the robot’s actions, and the recommendations from the ethical governor that directed the robot’s decisions, on the y-axis, at every decision-making step. The resident starts each simulation step. Therefore, the resident’s response is triggered at the subsequent step of a reminder. Simulations run for a maximum of 29 steps. In each timeline, the resident’s behaviour is consistent: they repeatedly snooze and acknowledge the alert until the robot resets the timer. <details> <summary>x4.png Details</summary>  ### Visual Description ## Sequence Diagram: Behavioral Scenarios ### Overview The image presents a series of sequence diagrams illustrating different behavioral scenarios. Each diagram depicts the interaction between three entities: HB (likely representing a Human Being), RB (likely representing a Robotic element), and EGR (likely representing an Environmental monitoring system). The diagrams show the sequence of actions or events over time ("Steps"), with specific actions triggered at certain points. A legend provides the meaning of the abbreviations used for the actions. ### Components/Axes * **Title:** Each diagram is titled "Behaviour [Number]" (e.g., "Behaviour 1", "Behaviour 2", etc.). There are 7 behaviours shown. * **Entities:** * HB (Human Being) * RB (Robotic element) * EGR (Environmental monitoring system) * **Timeline:** The horizontal axis represents "Steps," numbered from 0 to 29. * **Actions:** Actions are represented by abbreviations, with their meanings defined in the legend: * rem - Remind * snz - Snooze for 3 steps * flp - Followup * rec - Record * ack - Acknowledge * alt - Alert Care-worker * end - End * **Visual Representation:** * Entities are arranged vertically on the left. * A horizontal line represents the timeline for each entity. * Vertical blue lines indicate specific actions. * Black dots mark the points where actions occur. * Action abbreviations are placed near the vertical lines, with blue text for RB and red text for EGR. ### Detailed Analysis or ### Content Details **Behaviour 1:** * **HB:** Actions at steps 2, 6, 10, 14, 18, 22, and 26. The actions are "SNOOZE", "ACKNOWLEDGE", "SNOOZE", "ACKNOWLEDGE", "SNOOZE", "ACKNOWLEDGE", "SNOOZE". * **RB:** Actions at steps 2, 6, 10, 14, 18, 22, and 26. The actions are "rem snz", "rem ack", "flp snz", "rem ack", "flp snz", "rem ack", "flp snz". * **EGR:** Actions at steps 2, 6, 10, 14, 18, 22, and 26. The actions are "rem snz", "rem ack", "flp, rec, alt snz", "rem ack", "flp, rec, alt", "rec, alt, snz rem ack", "flp, rec, alt", "rec, alt, sz". **Behaviour 2:** * **HB:** Actions at steps 2, 6, 10, 14, 18, and 26. The actions are "SNOOZE", "ACKNOWLEDGE", "SNOOZE", "ACKNOWLEDGE", "SNOOZE", "ACKNOWLEDGE". * **RB:** Actions at steps 2, 6, 10, 14, 18, and 26. The actions are "rem snz", "rem ack", "flp snz", "rem ack", "flp snz", "rem ack", "rec". * **EGR:** Actions at steps 2, 6, 10, 14, 18, and 26. The actions are "rem snz", "rem ack", "flp, rec, alt snz", "rem ack", "flp, rec, alt snz", "rem ack", "rec". **Behaviour 3:** * **HB:** Actions at steps 2, 6, 10, and 14. The actions are "SNOOZE", "ACKNOWLEDGE", "SNOOZE", "ACKNOWLEDGE". * **RB:** Actions at steps 2, 6, 10, and 14. The actions are "rem snz", "rem ack", "flp snz", "rem ack", "alt". * **EGR:** Actions at steps 2, 6, 10, and 14. The actions are "rem snz", "rem ack", "flp, alt snz", "rem ack", "alt". **Behaviour 4:** * **HB:** Actions at steps 2, 6, 10, 14, 18, and 26. The actions are "SNOOZE", "ACKNOWLEDGE", "SNOOZE", "ACKNOWLEDGE", "SNOOZE", "ACKNOWLEDGE". * **RB:** Actions at steps 2, 6, 10, 14, 18, and 26. The actions are "rem snz", "rem ack", "flp snz", "rem ack", "flp snz", "rem ack", "alt". * **EGR:** Actions at steps 2, 6, 10, 14, 18, and 26. The actions are "rem snz", "rem ack", "flp, rec, alt snz", "rem ack", "flp, rec, alt", "rec, alt, snz rem ack", "alt". **Behaviour 5:** * **HB:** Actions at steps 2, 6, 10, 14, 18, and 26. The actions are "SNOOZE", "ACKNOWLEDGE", "SNOOZE", "ACKNOWLEDGE", "SNOOZE", "ACKNOWLEDGE". * **RB:** Actions at steps 2, 6, 10, 14, 18, and 26. The actions are "rem snz", "rem ack", "flp snz", "rem ack", "flp snz", "rem ack", "alt". * **EGR:** Actions at steps 2, 6, 10, 14, 18, and 26. The actions are "rem snz", "rem ack", "flp, rec, alt snz", "rem ack", "flp, rec, alt snz", "rem ack", "alt". **Behaviour 6:** * **HB:** Actions at steps 2, and 6. The actions are "SNOOZE", "ACKNOWLEDGE". * **RB:** Actions at steps 2, and 6. The actions are "rem snz", "rem ack", "alt". * **EGR:** Actions at steps 2, and 6. The actions are "rem snz", "rem ack", "alt". **Behaviour 7:** * **HB:** Actions at steps 2, 6, 10, and 14. The actions are "SNOOZE", "ACKNOWLEDGE", "SNOOZE", "ACKNOWLEDGE". * **RB:** Actions at steps 2, 6, 10, and 14. The actions are "rem snz", "rem ack", "flp snz", "rem ack", "rec". * **EGR:** Actions at steps 2, 6, 10, and 14. The actions are "rem snz", "rem ack", "flp, rec, alt snz", "rem ack", "rec". ### Key Observations * **HB (Human Being):** The HB consistently initiates actions with "SNOOZE" and "ACKNOWLEDGE". * **RB (Robotic element):** The RB's actions vary depending on the behavior, often involving "rem" (Remind), "snz" (Snooze), "flp" (Followup), "ack" (Acknowledge), "rec" (Record), and "alt" (Alert Care-worker). * **EGR (Environmental monitoring system):** The EGR's actions are often more complex, involving combinations of "flp," "rec," and "alt," suggesting a more reactive role based on environmental conditions. * **Repetitive Patterns:** Some behaviors exhibit repetitive patterns of actions, while others are more varied. ### Interpretation The sequence diagrams illustrate different scenarios of interaction between a human being, a robotic element, and an environmental monitoring system. The diagrams likely represent different ways these entities might interact in a caregiving or assisted living environment. * **Care Scenarios:** The behaviors likely represent different responses to events or alerts. For example, "Behaviour 1" might represent a scenario where the human repeatedly snoozes an alert, triggering a series of reminders and follow-ups. * **Adaptive Response:** The EGR's complex actions suggest that it plays a role in adapting the system's response based on environmental conditions or the human's behavior. * **Escalation:** The presence of "alt" (Alert Care-worker) indicates that the system can escalate situations when necessary, potentially when the human doesn't respond to reminders or follow-ups. * **Customization:** The different behaviors suggest that the system can be customized or adapted to different user needs or preferences. The diagrams provide a visual representation of how the system is designed to respond to different situations, highlighting the roles of each entity and the flow of actions over time. </details> Figure 4: Timelines of different behaviours shown in the simulations (HB – Resident Behaviour, RB – Robot Behaviour, EGR – Ethical Governor Recommendation) Table 1: Robots’ behaviours in different cases of ‘Medication Dilemma’ | 1 | Low | 0 | 1 | 2 | 2 | 3 | record at step 19 | | --- | --- | --- | --- | --- | --- | --- | --- | | 2 | Medium | 0 | 4 | 5 | 4 | 6 | alert at step 19 | | 3 | High | 0 | 6 | 6 | 6 | 6 | alert at step 10 | | 4 | Low | 2 | 1 | 7 | 1 | 6 | record at step 18 | | 5 | Medium | 2 | 6 | 6 | 6 | 6 | alert at step 10 | | 6 | High | 2 | 6 | 6 | 6 | 6 | alert at step 10 | Note: $M_{arw}$ - robot with Character_arw, $M_{ar}$ - robot with Character_ar, $M_{wr}$ - robot with Character_wr, $M_{a}$ - robot with Character_a Robots $M_{ar}$ , $M_{wr}$ , and $M_{a}$ represent instances where a robot is tuned to have extreme characters. This is represented in their behaviours. $M_{ar}$ , which prioritises autonomy and higher risk propensity, permits record after two consecutive follow up s and another breach of ACKNOWLEDGE in case 1 (behaviour 2), and one follow up and an ACKNOWLEDGE breach in case 4, where the medicine impact is low (behaviour 7). For the remaining scenarios, $M_{ar}$ triggers the action alert care-worker. The $M_{ar}$ permits alert care-worker in case 2 at the same time as it finished case 1, but in other cases, $M_{ar}$ sent the alert as soon as the resident did not follow the first ACKNOWLEDGE. This exhibits that, even though the robot’s character strongly favours autonomy, the robot does not violate the rule encoded in the system, if most of the experts disagree with that action. A similar trend can be seen in $M_{wr}$ , which favours wellbeing and has a higher risk tolerance. It acts the same way in cases 2-6. However, in case 1, the PSRB evaluator is ready to honour resident autonomy and permit follow up, since that action does not impact the wellbeing much at step 10. Nevertheless, it notifies the care-worker after the resident violates ACKNOWLEDGE twice. Robots $M_{ar}$ and $M_{a}$ illustrate the effect of risk propensity on the robot’s behaviour. In case 1, unlike robots $M_{ar}$ , $M_{a}$ does not make the record behaviour acceptable because it carries a risk that is out of character for $M_{a}$ . Therefore, $M_{a}$ continues to follow up with the resident until they take the medication willingly. Although that behaviour is not ideal, it is predictable since that behaviour is in its character. In case 2, both robots behave the same, however, at step 19, the ethical governor of $M_{a}$ makes the snooze undesirable because it carries a higher risk, making all actions equal. However, in cases 3, 5, and 6 both robots behave similarly because the experts agree with the general rule system on those circumstances. The character of $M_{arw}$ is less extreme than other characters, with a slight preference for resident autonomy and a moderate risk propensity. This character acts the same as the $M_{ar}$ in case 1. However, in case 2, the ethical governor in $M_{arw}$ lowers the desirability of snooze at step 19, because it causes too much negative wellbeing, reflecting $M_{arw}$ ’s slightly more attention to the resident’s wellbeing. In cases 3, 5 and 6, it behaves like all other characters because the KB does not recommend a rule override. However, in case 4, it adopts the same behaviour as $M_{a}$ , because it rejects the KB suggestion of record, as this behaviour lowers the wellbeing utility and contradicts the robot’s character. However, when KB suggests that call care-worker is not desirable, the evaluator agrees because it excessively diminishes autonomy. This aligns with its character. <details> <summary>x5.png Details</summary>  ### Visual Description ## Bar Chart: Autonomy and Wellbeing Utility Distributions ### Overview The image presents two bar charts side-by-side, illustrating the utility distributions for autonomy (left) and wellbeing (right) across four characters: 'a', 'ar', 'arw', and 'wr'. The x-axis represents the utility value, ranging from -1.0 to 1.0, while the y-axis represents the probability. Each character is represented by a different color, as indicated in the legend. ### Components/Axes **Left Chart: Autonomy Utility Distribution** * **Title:** Autonomy utility distribution * **X-axis:** Utility value, with markers at -1.0, -0.5, 0.0, 0.5, and 1.0. * **Y-axis:** Probability, ranging from 0.0 to 0.4, with no explicit markers between. * **Legend (Top-Left):** * Blue: a * Orange: ar * Green: arw * Red: wr **Right Chart: Wellbeing Utility Distribution** * **Title:** Wellbeing utility distribution * **X-axis:** Utility value, with markers at -1.0, -0.5, 0.0, 0.5, and 1.0. * **Y-axis:** Probability, ranging from 0.00 to 0.20, with markers at 0.00, 0.05, 0.10, 0.15, and 0.20. * **Legend (Top-Right):** * Blue: a * Orange: ar * Green: arw * Red: wr ### Detailed Analysis **Autonomy Utility Distribution (Left Chart)** * **Character 'a' (Blue):** * At -0.5: Probability ~0.03 * At 0.0: Probability ~0.07 * At 1.0: Probability ~0.45 * **Character 'ar' (Orange):** * At -0.5: Probability ~0.02 * At 0.0: Probability ~0.08 * At 0.5: Probability ~0.05 * At 1.0: Probability ~0.45 * **Character 'arw' (Green):** * At -0.5: Probability ~0.07 * At 0.0: Probability ~0.09 * At 1.0: Probability ~0.45 * **Character 'wr' (Red):** * At -0.75: Probability ~0.08 * At 0.0: Probability ~0.35 * At 1.0: Probability ~0.43 **Wellbeing Utility Distribution (Right Chart)** * **Character 'a' (Blue):** * At -1.0: Probability ~0.14 * At -0.5: Probability ~0.05 * At 0.0: Probability ~0.03 * At 0.5: Probability ~0.03 * At 1.0: Probability ~0.12 * **Character 'ar' (Orange):** * At -1.0: Probability ~0.145 * At -0.5: Probability ~0.03 * At 0.0: Probability ~0.03 * At 0.5: Probability ~0.12 * At 1.0: Probability ~0.16 * **Character 'arw' (Green):** * At -1.0: Probability ~0.15 * At -0.5: Probability ~0.12 * At 0.0: Probability ~0.03 * At 0.5: Probability ~0.08 * At 1.0: Probability ~0.13 * **Character 'wr' (Red):** * At -1.0: Probability ~0.05 * At -0.5: Probability ~0.05 * At 0.0: Probability ~0.12 * At 0.5: Probability ~0.16 * At 1.0: Probability ~0.13 ### Key Observations * **Autonomy:** All characters show a high probability (around 0.43-0.45) for a utility value of 1.0, indicating a strong preference for high autonomy. 'wr' also has a significant probability at 0.0. * **Wellbeing:** The wellbeing utility distributions are more varied. Characters 'a', 'ar', and 'arw' show higher probabilities at -1.0, while 'wr' shows higher probabilities at 0.5. ### Interpretation The charts suggest that all characters highly value autonomy, as indicated by the high probability at a utility value of 1.0 in the autonomy distribution. However, their preferences for wellbeing are more diverse. Characters 'a', 'ar', and 'arw' may be more sensitive to negative wellbeing impacts, while 'wr' might prioritize other factors over avoiding negative wellbeing. The data could represent different AI agents or user types, each with distinct utility functions that influence their decision-making processes. The differences in wellbeing preferences could lead to different behaviors and choices in scenarios where autonomy and wellbeing are in conflict. </details> Figure 5: Utility distributions of the robot behaviours in cases 1, 2 and 4 To gain a clear picture of the behaviour predictability provided by character-based tuning, the utility distributions of the behaviours demonstrated by the robots in cases 1, 2, and 4 are illustrated in Figure 5. This analysis focuses exclusively on cases that exhibit PSRB behaviour. The rationale for this selection is the recognition that including all cases obscures the specific impacts of PSRB behaviour. Robots which has a character that has precedence for resident autonomy (i.e., Character_a, Character_ar, Character_arw), have shown a higher probability of choosing behaviours that deliver higher utility for autonomy, compared to the characters that have a higher precedence for resident wellbeing. From these, the characters that have a higher risk propensity (i.e., Character_ar, Character_arw) have been shown to make decisions that increase resident autonomy, compared to their more cautious counterparts (i.e., Character_a). On the other hand, the characters with precedence for resident wellbeing, have shown a greater tendency to choose behaviours that yield higher wellbeing utility, compared to the rest. 5.3 Ethicist Evaluation We engaged two ethicists to assess the performance of all robots. Each ethicist was asked to rate the behaviour of the robotic agent on a scale from 1 to 5, with 1 indicating completely unacceptable and 5 indicating highly acceptable, taking into account the robot’s character, the pre-programmed rules, and the expert opinion for each scenario as found in the KB. Table 2: Ethicists opinions on the behaviour of the Medication Reminding Robot | Case ID 1 | $M_{A}$ Ethicist1 3 | $M_{AR}$ Ethicist2 3 | $M_{ARW}$ Ethicist1 4 | $M_{WR}$ Ethicist2 5 | Ethicist1 4 | Ethicist2 4 | Ethicist1 4 | Ethicist2 3 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 2 | 4 | 5 | 4 | 5 | 4 | 4 | 4 | 3 | | 3 | 4 | 4 | 5 | 4 | 5 | 5 | 5 | 5 | | 4 | 3 | 3 | 4 | 4 | 3 | 2 | 3 | 4 | | 5 | 5 | 3 | 5 | 3 | 4 | 4 | 5 | 5 | | 6 | 5 | 5 | 5 | 4 | 5 | 4 | 5 | 5 | The ethicists’ rating of robot behaviours is shown in Table 2. Overall, the results indicate a positive reaction to the robot behaviours, with the majority of scores ranging between 3 and 5. However, the behaviour of the robot $M_{ARW}$ in case 4, which involved repeatedly requesting the user to take medication, was deemed unacceptable by one of the ethicists. The similar ratings for two very different behaviours in the same scenario, such as those of robots $M_{wr}$ and $M_{ARW}$ in case 1, demonstrate the effect of a robot’s character on the perceived morality of their actions. When queried about the reason behind their rating on the behaviours of $M_{a}$ and $M_{ar}$ in case 5, the second ethicist indicated an expectation for these robots to exhibit more autonomy-centric behaviour, even overriding the expert opinion, to be aligned with their extreme, autonomy-focused character. 6 Discussion The simulations show that the PSRB-capable ethical governor makes the robot more flexible while maintaining predictability. The PSRB model offers a simpler way to localise than the existing methods (reviewed in section 2) because they are either inflexible or unpredictable. Although the output of the PSRB capable ethical governor is not as predictable as a top-down engineered agent (e.g., a deontological agent), the rigidity of the top-down approaches does not align with the virtue ethics tradition, which rejects such extreme actions. Agents based on the PSRB evaluator can behave in a moderate manner, even when the character strongly favours a particular trait (e.g., $M_{ar}$ , $M_{wr}$ , and $M_{a}$ ), by utilising bottom-up knowledge in the KB. On the other hand, the display of character can be held back by a KB, as evidenced by the second ethicist’s feedback. However, this conservatism was introduced by the PSRB process as a safety measure to control the excessive utility optimisation in the utilitarian calculations involved in the pro-social reasoning process. The virtue ethics-inspired architecture allows the robots to be tuned in two ways. The first approach involves adjusting the character parameters’ values to align with the ethical requirements of the environment. For example, one facility may require its robots to be more proactive towards enhancing residents’ autonomy, whereas another may also want to focus on the residents’ autonomy, but only under minimal risk conditions. The former institution can opt for a character profile akin to Character_ar, while the latter institution can tune the robot with a character profile resembling Character_a. If the first tuning approach does not suffice in certain specific instances, one can employ the secondary method to further refine the robot’s performance. This involves adjusting the knowledge base to reflect the desired behaviour. Unlike other machine learning approaches, using CBR, one can precisely identify the cases that influence behaviour. Consequently, one can integrate or adjust expert insights to better represent their perspective. As long as this desired behaviour is within the robot’s configured character, it will adapt its actions accordingly. 6.1 Limitations The PSRB implementation presented in this paper also has limitations. This is mainly because the character traits in the PSRB evaluator are implemented as constraints on linear axes. By doing so, it removes the ability to tune the behaviour precisely, if the requirements involve a range of behaviours that cannot be described using a single set of character traits. For example, let’s assume that the required behaviour of a deployment is to have behaviour 2 in case 2 and behaviour 6 in case 1. According to this implementation, there will not be any set of values for character traits that fulfil this requirement, because to have behaviour 6 in case 1, the character should have a higher wellbeing bias, which will trigger similar wellbeing-focused behaviour in case 2 (because case 2 has a higher wellbeing impact), which behaviour 2 is not. However, a different character trait implementation, in the same PSRB evaluation framework will be able to solve this issue. Finally, it should be noted that the implementation showcased in this simulation, encompassing both the dilemma and the environment, has been deliberately simplified to illustrate the concept and is not intended for real-world use. More capable robots, more precise rule sets, accurate utility models, and well-researched character models are a must when using this approach for real-world implementation. However, the system works with sub-optimal rules and utility functions by compensating for each other’s shortcomings by working together [39]. The PSRB architecture allows ethical tests other than stakeholder utility calculation. Hence we recommend that robot developers explore and integrate different models that can better capture the context alongside utilitarian calculations. Furthermore, it is important to update the KB regularly to compensate for the evolving nature of ethics. 7 Conclusion This paper introduces a method inspired by virtue ethics, which enables more efficient ethical tuning compared to current techniques. The method combines decision-making based on character traits with bottom-up learning from moral exemplars, to maintain the predictability of the robot’s behaviour while keeping the flexibility the latter offers. To the best of our knowledge, there is no other implementation that combines both these desirable features of virtue ethics. The paper demonstrates the system’s capabilities by analysing its behaviour and presenting the perspectives of two ethicists on these behaviours. It concludes with recommendations for implementing the proposed architecture in real-world elder-care robots. 7.0.1 \discintname The authors have no competing interests to declare that are relevant to the content of this article. Appendix 0.A Appendix 0.A.1 Agents 0.A.1.1 Resident Agent The resident agent can issue two instructions to the robot when it is queried by the robot, namely; 1. SNOOZE or 2. ACKNOWLEDGE. Instructions, as well as the resident’s compliance with taking the medicine, can be configured into the simulator. If the resident is willing to take the medication, the resident’s state changes to took_medication state after an ACKNOWLEDGE, which the robot can perceive. 0.A.1.2 Robot Agent The robot agent is an autonomous agent that collects perception data from the environment and decides its next move, at every step. When the resident issues a SNOOZE instruction the robot will snooze for three steps before following up. When it receives ACKNOWLEDGE instruction, for 2 steps it inspects whether the resident has taken the medication. If that is the case, it resets the timer. If not, it either repeats the reminder, or decides to record or report the situation. After three follow up s, the robot also considers record and report behaviours along with the follow up behaviour. The robot gives priority to the resident’s commands over other actions when the ethical layer suggests more than one action. 0.A.2 Ethical Layer 0.A.2.1 Stakeholder Utility Calculation The module uses the formula in equation 1 to calculate $Au_{i}$ , based on the number of follow up s ( $f$ ) that the robot performs. $$ \displaystyle Au_{i}=\begin{cases}\hfill 1:&\text{if the robot obeys a % resident instruction}\\ \hfill 0.5:&\text{if only recording a incident}\\ \hfill 0:&\text{if no instructions given}\\ \hfill-0.1\cdot f:&\text{if follow up}\\ \hfill-0.7:&\text{if the robot disobeys a resident instruction or report}\\ \hfill-1:&\text{if the resident is physically restrained by the robot}\end{cases} \tag{1} $$ The module uses a Gamma distribution (eq. 2) to model the wellbeing distribution, with the shape parameter $\alpha=1.325\varepsilon_{m}^{2}-9.475\varepsilon_{m}+18.15$ , the scale parameter $\beta=e^{-2.65-(d/2)}+0.01$ (each follow up is considered as a fraction of missed dose depending on the state of the reminder), and location parameter $v=-1$ . The highest probable utility value is obtained by $PMax\_util(g(x,\alpha,\beta,v))$ using a resolution of $0.05$ . The wellbeing utility $W_{i}$ for behaviour $i$ is computed using Equation 3. $$ g(x,\alpha,\beta,v)=\frac{(\frac{x-v}{\beta})^{\alpha-1}\cdot\exp(-\frac{x-v}{% \beta})}{\beta\cdot\Gamma(\alpha)} \tag{2} $$ $$ \displaystyle W_{i}=\begin{cases}\hfill PMax\_util(g(x,\alpha,\beta(d+f/8),v))% :&\text{if }i=\texttt{snooze}\\ \hfill PMax\_util(g(x,\alpha,\beta(d+f/3),v)):&\text{if }i=\texttt{follow up}% \\ \hfill PMax\_util(g(x,\alpha,\beta(d+f/4),v))+0.5:&\text{if }i=\texttt{remind}% \\ \hfill PMax\_util(g(x,\alpha,\beta(d+1),v)):&\text{if }i=\texttt{record}\\ \hfill abs(PMax\_util(g(x,\alpha,\beta(d+1),v))):&\text{else}\end{cases} \tag{3} $$ 0.A.2.2 PSRB Evaluator PSRB evaluator uses the Algorithm 1 for its PSRB decision calculation. The knowledge base uses KNN as its retrieval mechanism with $K=3$ and inverse distance voting function when $distance>0.1$ . When the $distance≤ 0.1$ , it uses $10$ as the weight of the instance. Algorithm 1 PSRB Evaluator Algorithm for the medication-reminding robot Input: Blackboard data ( $b$ ) Parameter: Value preferences ( $C_{w},C_{au}$ ), Risk propensity ( $C_{rp}$ ) Output: Desirability ( $D$ ) 1: procedure PSRB Evaluator ( $i$ ) $\triangleright$ Behaviour $i$ 2: $opinion,intention←\textsc{knowledge\_base}(b)$ 3: if $opinion$ AND $\lnot rules\_broken$ then $D_{i}← 1$ 4: else if $\lnot opinion$ AND $rules\_broken$ then $D_{i}← 0$ 5: else if $opinion$ AND $rules\_broken$ then $D_{i}← 1$ 6: for $C_{v}∈ C$ do $\triangleright$ $v∈\{w,au\}$ 7: $T\_positive_{v}←\frac{10-C_{v}}{10}$ 8: $T\_negative_{v}←\frac{C_{v}-10}{10}$ 9: $U_{c}←\textsc{Utility}(C_{v},stakeholder,i)$ 10: if $C_{v}∈ intention$ then 11: if $U_{c}<T\_positive_{v}$ then $D_{i}← 0$ 12: end if 13: else 14: if $U_{c}<T\_negative_{v}$ then $D_{i}← 0$ 15: end if 16: end if 17: end for 18: $risk\_threshold←\frac{\exp(C_{rp}/4.17)-1}{10}$ 19: $highest\_risk_{i}←\textsc{Max}(g(x,\alpha,\beta,v)· x)$ $\triangleright$ $g(x,\alpha,\beta,v)$ from (2) 20: if $highest\_risk_{i}>risk\_threshold$ then $D_{i}← 0$ 21: end if 22: else $D_{i}← 1$ 23: for $C_{v}∈ C$ do 24: $T\_negative_{v}←\frac{C_{v}-10}{10}$ 25: $U_{c}←\textsc{Utility}(C_{v},stakeholder,i)$ 26: if $C_{v}∈ intention$ AND $U_{c}<T\_negative_{v}$ then 27: $D_{i}← 0$ 28: end if 29: end for 30: $risk\_threshold←\frac{\exp(C_{rp}/4.17)-1}{10}$ 31: $highest\_risk_{i}←\textsc{Max}(g(x,\alpha,\beta,v)· x)$ $\triangleright$ $g(x,\alpha,\beta,v)$ from (2) 32: if $highest\_risk_{i}>risk\_threshold$ then $D_{i}← 0$ 33: end if 34: end if 35: return $D_{i}$ 36: end procedure 0.A.2.3 Explanation for the PSRB Behaviour When the ethical layer permits a PSRB behaviour, the PSRB module will give the following explanations depending on the reasons that lead to the decision. ⬇ The action breaks the rules < rule_ids >. However, this action in this context is considered desirable by experts. Since it increases < intentions > values greatly, while not reducing the other values < other_values > by a considerable amount, and the outcome is within accepted risk levels, deemed accepted by the PSRB system. ⬇ The action does not break any rules. However, this action in this context is considered undesirable by experts. Since the action outcomes introduce a high risk, deemed not accepted by the PSRB system. ⬇ The action does not break any rules. However, this action in this context is considered undesirable by experts. Since it decreases < intentions > values by a considerable amount, the action is deemed unacceptable by the system The knowledge base may propose a solution that does not comply with the established rules, in which case the evaluator can decline the proposal. In those cases, the PSRB evaluator gives one of the following explanations, depending on the reasons for the rejection. ⬇ The action breaks the rules < rule_ids >. However, this action in this context is considered desirable by experts. Although the value tradeoff is satisfactory, the risk taken by the action is not acceptable to bend the rule. ⬇ The action breaks the rules < rule_ids >. However, this action in this context is considered desirable by experts. But, the PSRB system suggests that the value tradeoff is not satisfactory to bend the rule. ⬇ The action does not break any rules. However, this action in this context is considered undesirable by experts. But, the PSRB system suggests that the value tradeoff is not satisfactory to bend the rule. References - [1] Abel, D., Macglashan, J., Littman, M.L.: Reinforcement learning as a framework for ethical decision making. In: AAAI Workshop: AI, Ethics, and Society. pp. 54–61 (2016), publication Title: AAAI Workshop - Technical Report Volume: WS-16-01 - ISBN: 9781577357599 - [2] Alexander, L., Moore, M.: Deontological Ethics. In: Zalta, E.N. (ed.) The {Stanford} Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University (2020), medium: \url{https://plato.stanford.edu/archives/win2020/entries/ethics-deontological/} - [3] Anderson, M., Anderson, S.L.: GenEth: A general ethical dilemma analyzer. Paladyn 9 (1), 337–357 (2018). https://doi.org/10.1515/pjbr-2018-0024 - [4] Anderson, M., Anderson, S.L., Armen, C.: MedEthEx: A Prototype Medical Ethics Advisor. Tech. rep. (2006), www.aaai.org - [5] Anderson, M., Anderson, S.L., Berenz, V.: A Value-Driven Eldercare Robot: Virtual and Physical Instantiations of a Case-Supported Principle-Based Behavior Paradigm. Proceedings of the IEEE 107 (3), 526–540 (2019). https://doi.org/10.1109/JPROC.2018.2840045 - [6] Aristotle: The Nicomachean Ethics. Penguin Books (350) - [7] Arkin, R.C.: Governing lethal behavior p. 121 (2008). https://doi.org/10.1145/1349822.1349839, iSBN: 9781605580173 - [8] Arkin, R.C.: Governing lethal behavior: Embedding ethics in a hybrid deliberative/reactive robot architecture: Part 2: Formalization for ethical control. Frontiers in Artificial Intelligence and Applications 171 (1), 51–62 (2008), iSBN: 9781586038335 - [9] Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.F., Rahwan, I.: The Moral Machine experiment. Nature 563 (7729), 59–64 (Nov 2018). https://doi.org/10.1038/s41586-018-0637-6, publisher: Nature Publishing Group - [10] Azad-Manjiri, M.: A New Architecture for Making Moral Agents Based on C4.5 Decision Tree Algorithm. International Journal of Information Technology and Computer Science 6 (5), 50–57 (2014). https://doi.org/10.5815/ijitcs.2014.05.07, http://www.mecs-press.org/ - [11] Benzmüller, C., Parent, X., van der Torre, L.: Designing normative theories for ethical and legal reasoning: LogiKEy framework, methodology, and tool support. Artificial Intelligence 287, 103348 (Oct 2020). https://doi.org/10.1016/j.artint.2020.103348, https://linkinghub.elsevier.com/retrieve/pii/S0004370219301110 - [12] Buchanan, A., Brock, D.W.: Deciding for Others. The Milbank Quarterly 64, 17–94 (1986). https://doi.org/10.2307/3349960, https://www.jstor.org/stable/3349960, publisher: [Wiley, Milbank Memorial Fund] - [13] Censi, A., Slutsky, K., Wongpiromsarn, T., Yershov, D., Pendleton, S., Fu, J., Frazzoli, E.: Liability, ethics, and culture-aware behavior specification using rulebooks. Proceedings - IEEE International Conference on Robotics and Automation 2019-May, 8536–8542 (2019). https://doi.org/10.1109/ICRA.2019.8794364, arXiv: 1902.09355 ISBN: 9781538660263 - [14] Chen, L.: Virtue Ethics and Confucian Ethics. Dao 9 (3), 275–287 (Sep 2010). https://doi.org/10.1007/s11712-010-9174-1, https://doi.org/10.1007/s11712-010-9174-1 - [15] Dahling, J.J., Chau, S.L., Mayer, D.M., Gregory, J.B.: Breaking rules for the right reasons? An investigation of pro-social rule breaking. Journal of Organizational Behavior 33 (1), 21–42 (Jan 2012). https://doi.org/10.1002/job.730 - [16] Dubljević, V.: Toward Implementing the ADC Model of Moral Judgment in Autonomous Vehicles. Science and Engineering Ethics pp. 1–12 (Jul 2020). https://doi.org/10.1007/s11948-020-00242-0, https://doi.org/10.1007/s11948-020-00242-0, publisher: Springer - [17] Epley, N., Waytz, A., Cacioppo, J.T.: On seeing human: A three-factor theory of anthropomorphism. Psychological Review 114 (4), 864–886 (2007). https://doi.org/10.1037/0033-295X.114.4.864, http://doi.apa.org/getdoi.cfm?doi=10.1037/0033-295X.114.4.864 - [18] Gabriel, I.: Artificial Intelligence, Values, and Alignment. Minds and Machines 30 (3), 411–437 (Sep 2020). https://doi.org/10.1007/s11023-020-09539-2, https://link.springer.com/10.1007/s11023-020-09539-2 - [19] Gamez, P., Shank, D.B., Arnold, C., North, M.: Artificial virtue: the machine question and perceptions of moral character in artificial moral agents. AI & SOCIETY 35 (4), 795–809 (Dec 2020). https://doi.org/10.1007/s00146-020-00977-1, https://doi.org/10.1007/s00146-020-00977-1 - [20] Gibert, M.: The case for virtuous robots. AI and Ethics 3 (1), 135–144 (Feb 2023). https://doi.org/10.1007/s43681-022-00185-1, https://doi.org/10.1007/s43681-022-00185-1 - [21] Govindarajulu, N.S., Bringsjord, S.: On Automating the Doctrine of Double Effect. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. vol. 0, pp. 4722–4730. International Joint Conferences on Artificial Intelligence (2017). https://doi.org/10.24963/ijcai.2017/658, https://github.com/naveensundarg/prover., arXiv: 1703.08922 ISSN: 10450823 - [22] Govindarajulu, N.S., Bringsjord, S., Ghosh, R., Sarathy, V.: Toward the Engineering of Virtuous Machines. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. pp. 29–35. AIES ’19, Association for Computing Machinery, New York, NY, USA (Jan 2019). https://doi.org/10.1145/3306618.3314256, https://dl.acm.org/doi/10.1145/3306618.3314256 - [23] Guarini, M.: Particularism and the classification and reclassification of moral cases. IEEE Intelligent Systems 21 (4), 22–28 (2006). https://doi.org/10.1109/MIS.2006.76 - [24] Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., Steinhardt, J.: Aligning AI With Shared Human Values (2021) - [25] Howard, D., Muntean, I.: Artificial Moral Cognition: Moral Functionalism and Autonomous Moral Agency. In: Powers, T.M. (ed.) Philosophy and Computing: Essays in Epistemology, Philosophy of Mind, Logic, and Ethics, pp. 121–159. Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3-319-61043-6_7, https://doi.org/10.1007/978-3-319-61043-6_7, iSSN: 0007-0882 - [26] Hursthouse, R., Pettigrove, G.: Virtue Ethics. In: Zalta, E.N. (ed.) The {Stanford} Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University (2018), medium: \url{https://plato.stanford.edu/archives/win2018/entries/ethics-virtue/} - [27] Jiang, L., Hwang, J.D., Bhagavatula, C., Bras, R.L., Forbes, M., Borchardt, J., Liang, J., Etzioni, O., Sap, M., Choi, Y.: Delphi: Towards Machine Ethics and Norms (1), 1–42 (2021), http://arxiv.org/abs/2110.07574, arXiv: 2110.07574 - [28] Kazil, J., Masad, D., Crooks, A.: Utilizing Python for Agent-Based Modeling: The Mesa Framework. In: Thomson, R., Bisgin, H., Dancy, C., Hyder, A., Hussain, M. (eds.) Social, Cultural, and Behavioral Modeling. pp. 308–317. Springer International Publishing, Cham (2020) - [29] Kraut, R.: Aristotle’s Ethics. In: Zalta, E.N., Nodelman, U. (eds.) The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, fall 2022 edn. (2022), https://plato.stanford.edu/archives/fall2022/entries/aristotle-ethics/ - [30] MacKenzie, M.: Buddhism and the Virtues, vol. 1. Oxford University Press (Dec 2017). https://doi.org/10.1093/oxfordhb/9780199385195.013.18, https://academic.oup.com/edited-volume/28109/chapter/212224702 - [31] Morrison, E.W.: Doing the job well: An investigation of pro-social rule breaking. Journal of Management 32 (1), 5–28 (2006). https://doi.org/10.1177/0149206305277790 - [32] Nafsika Athanassoulis: Virtue Ethics (2023), https://iep.utm.edu/virtue/ - [33] Nallur, V.: Landscape of Machine Implemented Ethics. Science and Engineering Ethics 26 (5), 2381–2399 (Oct 2020). https://doi.org/10.1007/s11948-020-00236-y, https://link.springer.com/10.1007/s11948-020-00236-y, publisher: Springer Science and Business Media B.V. - [34] Noothigattu, R., Gaikwad, S., Awad, E., Dsouza, S., Rahwan, I., Ravikumar, P., Procaccia, A.: A Voting-Based System for Ethical Decision Making. Proceedings of the AAAI Conference on Artificial Intelligence 32 (1) (Apr 2018). https://doi.org/10.1609/aaai.v32i1.11512, https://ojs.aaai.org/index.php/AAAI/article/view/11512, section: AAAI Technical Track: Human-Computation and Crowd Sourcing - [35] Pirni, A., Balistreri, M., Capasso, M., Umbrello, S., Merenda, F.: Robot Care Ethics Between Autonomy and Vulnerability: Coupling Principles and Practices in Autonomous Systems for Care. Frontiers in Robotics and AI 8, 654298 (Jun 2021). https://doi.org/10.3389/frobt.2021.654298, https://www.frontiersin.org/articles/10.3389/frobt.2021.654298/full - [36] Pontier, M.A., Hoorn, J.F.: Toward machines that behave ethically better than humans do. Proceedings of the Annual Meeting of the Cognitive Science Society (2012) - [37] Ramanayake, R., Nallur, V.: A Computational Architecture for a Pro-Social Rule Bending Agent. In: First International Workshop on Computational Machine Ethics held in conjunction with 18th International Conference on Principles of Knowledge Representation and Reasoning KR 2021 (CME2021) (Nov 2021). https://doi.org/10.5281/ZENODO.6470437, https://zenodo.org/record/6470437 - [38] Ramanayake, R., Nallur, V.: A Small Set of Ethical Challenges for Elder-Care Robots. In: Hakli, R., Mäkelä, P., Seibt, J. (eds.) Frontiers in Artificial Intelligence and Applications. IOS Press (Jan 2023). https://doi.org/10.3233/FAIA220605, https://ebooks.iospress.nl/doi/10.3233/FAIA220605 - [39] Ramanayake, R., Nallur, V.: Implementing Pro-social Rule Bending in an Elder-Care Robot Environment. In: Ali, A.A., Cabibihan, J.J., Meskin, N., Rossi, S., Jiang, W., He, H., Ge, S.S. (eds.) Social Robotics, vol. 14454, pp. 230–239. Springer Nature Singapore, Singapore (2024). https://doi.org/10.1007/978-981-99-8718-4_20, https://link.springer.com/10.1007/978-981-99-8718-4_20, series Title: Lecture Notes in Computer Science - [40] Ramanayake, R., Wicke, P., Nallur, V.: Immune moral models? Pro-social rule breaking as a moral enhancement approach for ethical AI. AI & SOCIETY (May 2022). https://doi.org/10.1007/s00146-022-01478-z, https://link.springer.com/10.1007/s00146-022-01478-z - [41] Russell, S.J.: Human compatible: artificial intelligence and the problem of control. Viking, New York? (2019) - [42] Schoenborn, J., Weber, R., Aha, W., Cassens, J., Althoff, K.D.: Explainable Case-Based Reasoning: A Survey. In: 35th AAAI Conference on Artificial Intelligence - A Virtual Conference; Explainable Agency in Artificial Intelligence Workshop (Feb 2021) - [43] Sharkey, A., Sharkey, N.: Granny and the robots: Ethical issues in robot care for the elderly. Ethics and Information Technology 14 (1), 27–40 (Jul 2012). https://doi.org/10.1007/s10676-010-9234-6, https://link.springer.com/article/10.1007/s10676-010-9234-6, publisher: Springer - [44] Sinnott-Armstrong, W.: Consequentialism. In: Zalta, E.N. (ed.) The {Stanford} Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University (2019), medium: \url{https://plato.stanford.edu/archives/sum2019/entries/consequentialism/} - [45] Solaiman, I., Dennison, C.: Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets. In: Advances in Neural Information Processing Systems. vol. 34, pp. 5861–5873. Curran Associates, Inc. (2021), https://proceedings.neurips.cc/paper/2021/hash/2e855f9489df0712b4bd8ea9e2848c5a-Abstract.html - [46] Thornton, S.M., Pan, S., Erlien, S.M., Gerdes, J.C.: Incorporating Ethical Considerations into Automated Vehicle Control. IEEE Transactions on Intelligent Transportation Systems 18 (6), 1429–1439 (Jun 2017). https://doi.org/10.1109/TITS.2016.2609339, publisher: Institute of Electrical and Electronics Engineers Inc. - [47] Van Dang, C., Tran, T.T., Gil, K.J., Shin, Y.B., Choi, J.W., Park, G.S., Kim, J.W.: Application of soar cognitive agent based on utilitarian ethics theory for home service robots. In: 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI). pp. 155–158 (Jun 2017). https://doi.org/10.1109/URAI.2017.7992698 - [48] Van Wynsberghe, A.: Social robots and the risks to reciprocity. AI & SOCIETY 37 (2), 479–485 (Jun 2022). https://doi.org/10.1007/s00146-021-01207-y, https://link.springer.com/10.1007/s00146-021-01207-y - [49] Vanderelst, D., Winfield, A.: An architecture for ethical robots inspired by the simulation theory of cognition. Cognitive Systems Research 48, 56–66 (2018). https://doi.org/10.1016/j.cogsys.2017.04.002, https://doi.org/10.1016/j.cogsys.2017.04.002, publisher: The Authors - [50] Wallach, W., Allen, C., Smit, I.: Machine morality: Bottom-up and top-down approaches for modelling human moral faculties. AI and Society 22 (4), 565–582 (Apr 2008). https://doi.org/10.1007/s00146-007-0099-0 - [51] van Wynsberghe, A., Robbins, S., Robbins scott, S., van Wynsberghe, A., Robbins, S.: Critiquing the Reasons for Making Artificial Moral Agents. Science and Engineering Ethics 2018 25:3 25 (3), 719–735 (Feb 2018). https://doi.org/10.1007/S11948-018-0030-8, https://link.springer.com/article/10.1007/s11948-018-0030-8, publisher: Springer - [52] Yilmaz, L., Franco-Watkins, A., Kroecker, T.S.: Computational models of ethical decision-making: A coherence-driven reflective equilibrium model. Cognitive Systems Research 46, 61–74 (Dec 2017). https://doi.org/10.1016/j.cogsys.2017.02.005, https://www.sciencedirect.com/science/article/pii/S1389041716301218