2312.12559

# Beyond Fairness: Alternative Moral Dimensions for Assessing Algorithms and Designing Systems **Authors**: - Kimi Wenzel (Human-Computer Interaction Institute) - Pittsburgh, PA - Geoff Kaufman (Human-Computer Interaction Institute) - Pittsburgh, PA - Laura Dabbish (Human-Computer Interaction Institute) - Pittsburgh, PA Abstract The ethics of artificial intelligence (AI) systems has risen as an imminent concern across scholarly communities. This concern has propagated a great interest in algorithmic fairness. Large research agendas are now devoted to increasing algorithmic fairness, assessing algorithmic fairness, and understanding human perceptions of fairness. We argue that there is an overreliance on fairness as a single dimension of morality, which comes at the expense of other important human values. Drawing from moral psychology, we present five moral dimensions that go beyond fairness, and suggest three ways these alternative dimensions may contribute to ethical AI development. 1 Introduction Evaluations of morality in algorithmic systems typically take a monist approach, optimizing for a single dimension of morality, namely, fairness (Mitchell et al., 2021). In contrast, models of moral reasoning have proposed a multitude of moral pillars, in addition to fairness, that guide human judgment and behavior. The following paper describes factors contributing to the positioning of fairness as a central focus in the evaluation of algorithms, provides a review of alternative moral values derived from psychological theories, and finally makes a case for how expanding our descriptive understanding of morality can enhance the ethics of technologists and help design algorithmic tools with a broad user acceptance. 2 The Reign of Algorithmic Fairness Why has fairness prevailed over other moral dimensions? There are three factors we describe here: (1) the prioritization of Western ideals, (2) the context of deployment, and (3) the mathematical “messiness” See Graham et al.’s (p. 57) summary of William James’s work from 1909 (Graham et al., 2013). of morality. We also describe unique problems that arise for each factor, respectively. 2.1 Prioritization of Western Ideals Fairness is a highly weighted virtue in the West (Sambasivan et al., 2021; Velasquez et al., 1990; Dator et al., 2006). Many non-Western cultures also value fairness; however, in such contexts fairness often coexists with many other virtues (Shweder et al., 2013; Dator et al., 2006; Graham et al., 2011), which in turn affects how “fair” algorithmic systems are deployed and accepted (Sambasivan et al., 2021). Thus, when evaluating the descriptive morality of algorithmic systems, it is important then to not focus solely on fairness, as has been the precedent for several years now (Starke et al., 2022), but to evaluate multiple dimensions of morality. 2.2 Context of Deployment While fairness as it pertains to mathematics and algorithms has been studied for centuries, its recent popularity has largely been towards the development of risk assessments for legal systems (Ochigame, 2020; Eubanks, 2018). This legal context, even in the absence of algorithmic tools, aims to provide fair outcomes. Thus, when building tools for these types of risk assessments, optimizing for algorithmic fairness can appear logical. This simplistic framing however, can promote broader contextual unawareness and ultimately fails to acknowledge the social, environmental, and economic structures that contribute to unfair and unethical outcomes (Hoffmann, 2019; Selbst et al., 2019). Beyond basic contextual awareness, another problem has arisen regarding how the context of fairness applications itself has changed. Fairness’s scholarly boom has led it to be adapted and applied to various domains, sometimes with insufficient reasoning justifying its application. For example, fairness may have inappropriately superceded liberty in the context of biometrics (Valdivia et al., 2022) and superceded harm in automatic speech recognition (Wenzel et al., 2023). The latter example demonstrated how fair and equal error rates in voice assistant interactions can lead to disparate mental health outcomes across races, demonstrating how championing fairness can fail to the extent that it ignores the pre-existing social landscape (i.e. racism), and it can also fail to the extent that it undermines other values (i.e. care). (See Table 1 for definitions.) 2.3 Mathematical “Messiness” of Morality Fairness, in spite of the difficulties associated with defining and operationalizing it (Mitchell et al., 2021), may be perceived as more compatible with mathematics. In his article about fairness and political philosophy, Binns describes how “‘fairness’ as used in the fair machine learning community is best understood as a placeholder term for a variety of normative egalitarian considerations” (Binns, 2018). This egalitarian interpretation held by computer scientists points to how fairness may be, in its most essential form, composed of equality conditions. For example, a model for facial recognition may be considered fair if the error rates are equal across all races. Other moral dimensions have fewer a priori mappings to equations and thus are more difficult to operationalize, perhaps due to the social intuition involved in moral judgements (Haidt, 2001). Adding to the complexity, embracing moral pluralism would require not only translating moral values into optimization problems, but would also require that all these moral values are respected and upheld. Considering how pluralistic applications of mathematical fairness alone are impossible (Kleinberg et al., 2016), there is probable concern that pluralistic applications of morality more broadly may pose similar problems. 3 Taking a Step Forward Built upon years of prior work, the Moral Foundations Theory (MFT) differentiates various facets of morality, namely care, fairness, loyalty, authority, and sanctity (Graham et al., 2013), with some research arguing to include other dimensions such as liberty (Haidt, 2012); see Table 1. MFT is perhaps most known for explaining differences in moral judgement between conservatives and liberals in the United States (Haidt, 2012; Haidt and Graham, 2007; Graham et al., 2009); however, it has been adapted for non-WEIRD countries (i.e. non- Western, educated, industrialized, rich, democratic countries) (Doğruyol et al., 2019; Atari et al., 2022; Matsuo et al., 2019) and has been studied in contexts as diverse as the life aspirations of Mongolian youth (Bespalov et al., 2017), nuclear disarmament in South Africa (Das, 2022), and Islamic business in Russia (Penikas et al., 2021). We propose using MFT to reframe research on AI ethics. Understanding human perceptions of morality, beyond fairness, may (a) help us understand user adoption of algorithmic systems, (b) help us understand how to best build interventions for technologists creating systems with algorithmic power, and (c) orient systems away from technosolutionism and toward society-in-the-loop methods. Table 1: Six moral foundations, adapted from (Grizzard and Ahn, 2017; Haidt, 2012) | Care | Alleviate physical or mental suffering | | --- | --- | | Fairness | Foster equality | | Loyalty | Act loyal to one’s ingroup | | Authority | Respect relationships within hierarchies | | Sanctity | Avoid physical, mental, or spiritual contamination | | Liberty | Protect freedom; Keep dominant agents “in-check” | a. Promote User Adoption Aligning a product’s ethical values with users’ ethical values using MFT may help boost user acceptance (Telkamp and Anderson, 2022). Thus, MFT may not only help improve the ethical integrity of algorithmic systems, but may also help improve user satisfaction more broadly. Building upon this, MFT may also help explain differences in perceptions of AI across different cultures. For example, might higher valuations of authority, loyality, or sanctity among Indians, vs. Americans and Europeans, (Renner, 2014) explain the observed aspiration toward AI among Indian nationals (Sambasivan et al., 2021)? Such insights may help design more culturally sensitive algorithmic systems. We encourage further research be done to evaluate how existing algorithmic systems may be perceived by users, based on their MFT orientations. b. Build Interventions MFT has been shown to help facilitate collective action (Milesi and Alberici, 2018). Thus, it may be worth investigating how we may harness MFT to help facilitate collective action toward ethical goals in computing. Respect for individuals’ MFT orientations can also facilitate better cross-cultural collaboration (Romig et al., 2018), so there is sufficient reason to believe studying diverse users’ MFT orientations could assist researchers’ efforts in creating technologies for cooperative work. c. Facilitate a Social Contract The mathematical difficulties (Sec 2.3) associated with deploying diverse MFT foundations may actually be beneficial to the extent that they discourage technosolutionism: If a set of moral foundations cannot be readily optimized in an algorithmic system, designers and developers may consider alternative solutions that integrate human feedback and oversight into the product lifecycle. Such solutions may harness human-in-the-loop (HITL) and human-AI complementarity methods, which have been recommended by researchers for AI tools used in education (Holstein et al., 2019), social services (Kawakami et al., 2022; Fogliato et al., 2022), and healthcare (Sivaraman et al., 2023). These methods afford users the ability to integrate their own expertise and moral oversight into algorithmic decisions. Going a step past HITL methods, we encourage a society-in-the-loop (SITL) approach. SITL builds upon HITL by integrating a social contract into the cycle (Rahwan, 2018). Importantly, the social contract includes groups with conflicting interests and values. Translating these values to MFT foundations, and integrating MFT foundations into the SITL method may support a more comprehensive social contract and a more inclusive values system. 4 Limitations As we work to embed a diverse set of morals into emerging technologies, we must think critically about whose values will be prioritized. Importantly, MFT is a descriptive theory that makes sense of people’s moral judgements – it is not a normative theory intended to guide human behavior. Should we be designing technologies to match users’ moral orientations, so as to best promote user acceptance? Should we leverage persuasive design techniques to re-orient individuals towards a normative ethic? We must approach this research agenda with great care and intentionality, knowing that appealing too far towards users’ moral biases can cause harm (Ranalli and Malcom, 2023), and trying to persuade users can sometimes accelerate abuse (Hunter and Evans, 2016). To this point, we stress that any integration of MFT should be culturally situated. While MFT alone does not have norm embeddings, it may still inform Norm Sensitive Design, and help adapt value-sensitive methodologies to diverse cultural contexts (Martin et al., 2023). Furthermore, we also advocate for full integration of diverse and historically marginalized technologists into the AI product lifecycle. Importantly, this integration goes beyond simply having marginalized identities represented in teams, but also ensuring that they are actively engaged and uplifted (Smith-Doerr et al., 2017). Adequate integration helps ensure that algorithmic systems respect pluralistic values. 5 Conclusion As algorithms are increasingly deployed in both high-stakes and everyday consumer contexts, it’s imperative that they are not exclusively optimized for fairness, but rather, are designed to uphold the ethical integrity of our diverse and morally pluralistic society. In this paper, we have outlined three reasons driving fairness’s popularity over other moral values (prioritization of Western ideals, the context of legal risk assessments, mathematical challenges) and describe challenges associated with each reason. We then introduced Haidt’s Moral Foundation’s Theory and described how it may be adapted to computing research by promoting user adoption, informing ethics interventions, and assisting in the development of important social contracts. Lastly, we stress the value of diversity and cultural awareness in the development of AI systems. Going forward, we encourage research studying human perceptions of fairness in algorithmic systems expands to include human perceptions of morality more broadly. If a core tenent of responsible AI is to “align system goals with human values” (Dignum, 2019), we urge technologists to continually question whose human values they are aligning to, and whose values are being ignored. References - (1) - Atari et al. (2022) Mohammad Atari, Jonathan Haidt, Jesse Graham, Sena Koleva, Sean T Stevens, and Morteza Dehghani. 2022. Morality beyond the WEIRD: How the nomological network of morality varies across cultures. (2022). - Bespalov et al. (2017) Alexander Bespalov, Marina Prudnikova, Bavuu Nyamdorj, and Mikhail Vlasov. 2017. Life aspirations, values and moral foundations in Mongolian youth. Journal of Moral Education 46, 3 (2017), 258–271. - Binns (2018) Reuben Binns. 2018. Fairness in machine learning: Lessons from political philosophy. In Conference on fairness, accountability and transparency. PMLR, 149–159. - Das (2022) Arunjana Das. 2022. Role of moral foundations in the nuclear disarmament of South Africa. Scientia Militaria: South African Journal of Military Studies 50, 1 (2022), 91–119. - Dator et al. (2006) Jim Dator, D Pratt, and Y Seo. 2006. What Is Fairness. Fairness, Globalization, and Public Institutions (2006), 19. - Dignum (2019) Virginia Dignum. 2019. Responsible artificial intelligence: how to develop and use AI in a responsible way. Vol. 2156. Springer. - Doğruyol et al. (2019) Burak Doğruyol, Sinan Alper, and Onurcan Yilmaz. 2019. The five-factor model of the moral foundations theory is stable across WEIRD and non-WEIRD cultures. Personality and Individual Differences 151 (2019), 109547. - Eubanks (2018) Virginia Eubanks. 2018. Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin’s Press. - Fogliato et al. (2022) Riccardo Fogliato, Maria De-Arteaga, and Alexandra Chouldechova. 2022. A case for humans-in-the-loop: Decisions in the presence of misestimated algorithmic scores. Available at SSRN (2022). - Graham et al. (2013) Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P Wojcik, and Peter H Ditto. 2013. Moral foundations theory: The pragmatic validity of moral pluralism. In Advances in experimental social psychology. Vol. 47. Elsevier, 55–130. - Graham et al. (2009) Jesse Graham, Jonathan Haidt, and Brian A Nosek. 2009. Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology 96, 5 (2009), 1029. - Graham et al. (2011) Jesse Graham, Brian A Nosek, Jonathan Haidt, Ravi Iyer, Spassena Koleva, and Peter H Ditto. 2011. Mapping the moral domain. Journal of personality and social psychology 101, 2 (2011), 366. - Grizzard and Ahn (2017) Matthew Grizzard and Changhyun Ahn. 2017. Morality & personality: perfect and deviant selves. Avatar, Assembled: The Social and Technical Anatomy of Digital Bodies. New York, NY: Peter Lang (2017). - Haidt (2001) Jonathan Haidt. 2001. The emotional dog and its rational tail: a social intuitionist approach to moral judgment. Psychological review 108, 4 (2001), 814. - Haidt (2012) Jonathan Haidt. 2012. The righteous mind: Why good people are divided by politics and religion. Vintage. - Haidt and Graham (2007) Jonathan Haidt and Jesse Graham. 2007. When morality opposes justice: Conservatives have moral intuitions that liberals may not recognize. Social justice research 20, 1 (2007), 98–116. - Hoffmann (2019) Anna Lauren Hoffmann. 2019. Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse. Information, Communication & Society 22, 7 (2019), 900–915. - Holstein et al. (2019) Kenneth Holstein, Bruce M McLaren, and Vincent Aleven. 2019. Designing for complementarity: Teacher and student needs for orchestration support in AI-enhanced classrooms. In Artificial Intelligence in Education: 20th International Conference, AIED 2019, Chicago, IL, USA, June 25-29, 2019, Proceedings, Part I 20. Springer, 157–171. - Hunter and Evans (2016) David Hunter and Nicholas Evans. 2016. Facebook emotional contagion experiment controversy. , 2–3 pages. - Kawakami et al. (2022) Anna Kawakami, Venkatesh Sivaraman, Hao-Fei Cheng, Logan Stapleton, Yanghuidi Cheng, Diana Qing, Adam Perer, Zhiwei Steven Wu, Haiyi Zhu, and Kenneth Holstein. 2022. Improving human-AI partnerships in child welfare: understanding worker practices, challenges, and desires for algorithmic decision support. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–18. - Kleinberg et al. (2016) Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807 (2016). - Martin et al. (2023) Diana Adela Martin, Rockwell F Clancy, Qin Zhu, and Gunter Bombaerts. 2023. Why do we need Norm Sensitive Design? A WEIRD critique of value sensitive approaches to design. (2023). - Matsuo et al. (2019) Akiko Matsuo, Kazutoshi Sasahara, Yasuhiro Taguchi, and Minoru Karasawa. 2019. Development and validation of the Japanese moral foundations dictionary. PloS one 14, 3 (2019), e0213343. - Milesi and Alberici (2018) Patrizia Milesi and Augusta Isabella Alberici. 2018. Pluralistic morality and collective action: The role of moral foundations. Group Processes & Intergroup Relations 21, 2 (2018), 235–256. - Mitchell et al. (2021) Shira Mitchell, Eric Potash, Solon Barocas, Alexander D’Amour, and Kristian Lum. 2021. Algorithmic fairness: Choices, assumptions, and definitions. Annual Review of Statistics and Its Application 8 (2021), 141–163. - Ochigame (2020) Rodrigo Ochigame. 2020. The Long History of Algorithmic Fairness. Phenomenal World (2020). - Penikas et al. (2021) Henry Penikas, EA Fedorova, AR Nevredinov, and SM Druchok. 2021. Textual analysis of moral components in Islamic and Non-Islamic business in Russia. In 2021 International Conference on Sustainable Islamic Business and Finance. IEEE, 140–143. - Rahwan (2018) Iyad Rahwan. 2018. Society-in-the-loop: programming the algorithmic social contract. Ethics and information technology 20, 1 (2018), 5–14. - Ranalli and Malcom (2023) Christopher Ranalli and Finlay Malcom. 2023. What’s so bad about echo chambers? Inquiry (2023), 1–43. - Renner (2014) Walter Renner. 2014. Globalization and Indian youth: Findings from moral foundations theory. In Current issues of science and research in the global world: Proceedings of the International Conference on Current Issues of Science and Research in the Global World, Vienna, Austria; 27. 7. - Romig et al. (2018) Charles A Romig, Virginia T Holeman, and Jill Duba Sauerheber. 2018. Using moral foundations theory to enhance multicultural competency. Counseling and Values 63, 2 (2018), 180–193. - Sambasivan et al. (2021) Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021. Re-imagining algorithmic fairness in india and beyond. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 315–328. - Selbst et al. (2019) Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the conference on fairness, accountability, and transparency. 59–68. - Shweder et al. (2013) Richard A Shweder, Nancy C Much, Manamohan Mahapatra, and Lawrence Park. 2013. The “big three” of morality (autonomy, community, divinity) and the “big three” explanations of suffering. In Morality and health. Routledge, 119–169. - Sivaraman et al. (2023) Venkatesh Sivaraman, Leigh A Bukowski, Joel Levin, Jeremy M Kahn, and Adam Perer. 2023. Ignore, trust, or negotiate: understanding clinician acceptance of AI-based treatment recommendations in health care. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–18. - Smith-Doerr et al. (2017) Laurel Smith-Doerr, Sharla N Alegria, and Timothy Sacco. 2017. How diversity matters in the US science and engineering workforce: A critical review considering integration in teams, fields, and organizational contexts. Engaging Science, Technology, and Society 3 (2017), 139–153. - Starke et al. (2022) Christopher Starke, Janine Baleis, Birte Keller, and Frank Marcinkowski. 2022. Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature. Big Data & Society 9, 2 (2022), 20539517221115189. - Telkamp and Anderson (2022) Jake B Telkamp and Marc H Anderson. 2022. The implications of diverse human moral foundations for assessing the ethicality of Artificial Intelligence. Journal of Business Ethics 178, 4 (2022), 961–976. - Valdivia et al. (2022) Ana Valdivia, Júlia Corbera Serrajòrdia, and Aneta Swianiewicz. 2022. There is an elephant in the room: Towards a critique on the use of fairness in biometrics. AI and Ethics (2022), 1–16. - Velasquez et al. (1990) Manuel Velasquez, Claire Andre, T Shanks, and Michael J Meyer. 1990. Justice and fairness. Issues in Ethics 3, 2 (1990), 1–3. - Wenzel et al. (2023) Kimi Wenzel, Nitya Devireddy, Cam Davison, and Geoff Kaufman. 2023. Can Voice Assistants Be Microaggressors? Cross-Race Psychological Responses to Failures of Automatic Speech Recognition. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–14.