## On the Opportunities and Risks of
## Foundation Models
Rishi Bommasani* Drew A. Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S. Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill Erik Brynjolfsson Shyamal Buch Dallas Card Rodrigo Castellon Niladri Chatterji Annie Chen Kathleen Creel Jared Quincy Davis Dorottya Demszky Chris Donahue Moussa Doumbouya Esin Durmus Stefano Ermon John Etchemendy Kawin Ethayarajh Li Fei-Fei Chelsea Finn Trevor Gale Lauren Gillespie Karan Goel Noah Goodman Shelby Grossman Neel Guha Tatsunori Hashimoto Peter Henderson John Hewitt Daniel E. Ho Jenny Hong Kyle Hsu Jing Huang Thomas Icard Saahil Jain Dan Jurafsky Pratyusha Kalluri Siddharth Karamcheti Geoff Keeling Fereshte Khani Omar Khattab Pang Wei Koh Mark Krass Ranjay Krishna Rohith Kuditipudi Ananya Kumar Faisal Ladhak Mina Lee Tony Lee Jure Leskovec Isabelle Levent Xiang Lisa Li Xuechen Li Tengyu Ma Ali Malik Christopher D. Manning Suvir Mirchandani Eric Mitchell Zanele Munyikwa Suraj Nair Avanika Narayan Deepak Narayanan Ben Newman Allen Nie Juan Carlos Niebles Hamed Nilforoshan Julian Nyarko Giray Ogut Laurel Orr Isabel Papadimitriou Joon Sung Park Chris Piech Eva Portelance Christopher Potts Aditi Raghunathan Rob Reich Hongyu Ren Frieda Rong Yusuf Roohani Camilo Ruiz Jack Ryan Christopher Ré Dorsa Sadigh Shiori Sagawa Keshav Santhanam Andy Shih Krishnan Srinivasan Alex Tamkin Rohan Taori Armin W. Thomas Florian Tramèr Rose E. Wang William Wang Bohan Wu Jiajun Wu Yuhuai Wu Sang Michael Xie Michihiro Yasunaga Jiaxuan You Matei Zaharia Michael Zhang Tianyi Zhang Xikun Zhang Yuhui Zhang Lucia Zheng Kaitlyn Zhou Percy Liang* 1
Center for Research on Foundation Models (CRFM) Stanford Institute for Human-Centered Artificial Intelligence (HAI) Stanford University
AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) trained on broad data (generally using self-supervision at scale) that can be adapted to a wide range of downstream tasks. We call these models foundation models to underscore their critically central yet incomplete character. This report provides a thorough account of the opportunities and risks of foundation models, ranging from their capabilities (e.g., language, vision, robotic manipulation, reasoning, human interaction) and technical principles (e.g., model architectures, training procedures, data, systems, security, evaluation, theory) to their applications (e.g., law, healthcare, education) and societal impact (e.g., inequity, misuse, economic and environmental impact, legal and ethical considerations). Though foundation models are based on standard deep learning and transfer learning, their scale results in new emergent capabilities, and their effectiveness across so many tasks incentivizes homogenization. Homogenization provides powerful leverage but demands caution, as the defects of the foundation model are inherited by all the adapted models downstream. Despite the impending widespread deployment of foundation models, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties. To tackle these questions, we believe much of the critical research on foundation models will require deep interdisciplinary collaboration commensurate with their fundamentally sociotechnical nature.
1 Corresponding author: pliang@cs.stanford.edu
## Contents
| Contents | Contents | 2 |
|----------------------------|---------------------------------------------------|---------|
| 1 | Introduction | 3 |
| 1.1 | Emergence and homogenization | 3 |
| 1.2 | Social impact and the foundation models ecosystem | 7 |
| 1.3 | The future of foundation models | 9 |
| 1.4 | Overview of this report | 12 |
| 2 | Capabilities | 21 |
| 2.1 | Language | 22 |
| 2.2 | Vision | 28 |
| 2.3 | Robotics | 34 |
| 2.4 | Reasoning and search | 40 |
| 2.5 | Interaction | 44 |
| 2.6 | Philosophy of understanding | 48 |
| 3 | Applications | 53 |
| 3.1 | Healthcare and biomedicine | 54 |
| 3.2 | Law | 59 |
| 3.3 | Education | 67 |
| 4 | Technology | 73 |
| 4.1 | Modeling | 74 |
| 4.2 | Training | 81 |
| 4.3 | Adaptation | 85 |
| 4.4 | Evaluation | 91 |
| 4.5 | Systems | 97 |
| 4.6 | Data | 101 |
| 4.7 | Security and privacy | 105 |
| 4.8 | Robustness to distribution shifts | 109 |
| 4.9 | AI safety and alignment | 114 |
| 4.10 | Theory | 118 |
| 4.11 | Interpretability | 123 |
| 5 | Society | 129 |
| 5.1 | Inequity and fairness | 130 |
| 5.2 | Misuse | 136 |
| 5.3 | Environment | 140 |
| 5.4 | Legality | 146 |
| 5.5 | Economics | 149 |
| 5.6 | Ethics of scale | 152 |
| 6 | Conclusion | 161 |
| Acknowledgments References | Acknowledgments References | 161 161 |
## 1 INTRODUCTION
This report investigates an emerging paradigm for building artificial intelligence (AI) systems based on a general class of models which we term foundation models . 2 A foundation model is any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks; current examples include BERT [Devlin et al. 2019], GPT-3 [Brown et al. 2020], and CLIP [Radford et al. 2021]. From a technological point of view, foundation models are not new - they are based on deep neural networks and self-supervised learning, both of which have existed for decades. However, the sheer scale and scope of foundation models from the last few years have stretched our imagination of what is possible; for example, GPT-3 has 175 billion parameters and can be adapted via natural language prompts to do a passable job on a wide range of tasks despite not being trained explicitly to do many of those tasks [Brown et al. 2020]. At the same time, existing foundation models have the potential to accentuate harms, and their characteristics are in general poorly understood. Given their impending widespread deployment, they have become a topic of intense scrutiny [Bender et al. 2021].
## 1.1 Emergence and homogenization
The significance of foundation models can be summarized by two words: emergence and homogenization . Emergence means that the behavior of a system is implicitly induced rather than explicitly constructed; it is both the source of scientific excitement and anxiety about unanticipated consequences. Homogenization indicates the consolidation of methodologies for building machine learning systems across a wide range of applications; it provides strong leverage towards many tasks but also creates single points of failure. To better appreciate emergence and homogenization, let us reflect on their rise in AI research over the last 30 years.
Fig. 1. The story of AI has been one of increasing emergence and homogenization . With the introduction of machine learning, how a task is performed emerges (is inferred automatically) from examples; with deep learning, the high-level features used for prediction emerge; and with foundation models, even advanced functionalities such as in-context learning emerge. At the same time, machine learning homogenizes learning algorithms (e.g., logistic regression), deep learning homogenizes model architectures (e.g., Convolutional Neural Networks), and foundation models homogenizes the model itself (e.g., GPT-3).
<details>
<summary>Image 1 Details</summary>
### Visual Description
\n
## Diagram: Evolution of Machine Learning
### Overview
The image is a horizontal diagram illustrating the evolution of machine learning, progressing from Machine Learning to Deep Learning and finally to Foundation Models. It depicts a timeline-like progression with associated concepts and characteristics. There is no quantitative data present; it's a conceptual representation.
### Components/Axes
The diagram consists of three main sections, each representing a stage in the evolution:
* **Machine Learning:** (Light Green) - Associated with "how" and "learning algorithms". An icon depicting a bar graph with a clock overlay is present.
* **Deep Learning:** (Purple) - Associated with "features" and "architectures". An icon representing a neural network is present.
* **Foundation Models:** (Pink) - Associated with "functionalities" and "models". An icon representing a complex, spherical structure is present.
A horizontal arrow indicates the direction of progression.
On the left side, text labels indicate:
* "Emergence of..." (Red)
* "Homogenization of..." (Orange)
### Detailed Analysis or Content Details
The diagram presents a qualitative progression.
* **Machine Learning:** This stage is linked to the concept of "how" things are learned and the specific "learning algorithms" used.
* **Deep Learning:** This stage builds upon Machine Learning and focuses on "features" and the underlying "architectures" of the models.
* **Foundation Models:** This stage represents the latest evolution, emphasizing "functionalities" and the "models" themselves.
The arrow indicates a temporal progression from left to right. The icons visually represent the complexity increasing with each stage.
### Key Observations
The diagram highlights a shift in focus as machine learning evolves. Initially, the emphasis is on *how* learning happens (algorithms). Then, the focus shifts to the internal structure and characteristics of the models (features, architectures). Finally, the emphasis is on what the models *can do* (functionalities, models).
### Interpretation
The diagram suggests a narrative of increasing abstraction and capability in machine learning. The progression from "learning algorithms" to "functionalities" implies a move from explicitly programmed methods to more generalized and adaptable systems. The "Emergence of..." and "Homogenization of..." labels suggest that each stage brings about new capabilities and a convergence of techniques. The diagram doesn't provide specific data, but it conveys a clear conceptual understanding of the evolution of the field. The increasing complexity of the icons visually reinforces the idea of growing sophistication. The diagram is a high-level overview and doesn't delve into the specifics of each stage. It's a conceptual map rather than a data-driven analysis.
</details>
Machine learning. Most AI systems today are powered by machine learning, where predictive models are trained on historical data and used to make future predictions. The rise of machine learning within AI started in the 1990s, representing a marked shift from the way AI systems were built previously: rather than specifying how to solve a task, a learning algorithm would induce it based on data - i.e., the how emerges from the dynamics of learning. Machine learning also
2 Wechose the term foundation models to capture the unfinished yet important status of these models - see §1.1.1: naming for further discussion of the name.
represented a step towards homogenization: a wide range of applications could now be powered by a single generic learning algorithm such as logistic regression.
Despite the ubiquity of machine learning within AI, semantically complex tasks in natural language processing (NLP) and computer vision such as question answering or object recognition, where the inputs are sentences or images, still required domain experts to perform 'feature engineering' - that is, writing domain-specific logic to convert raw data into higher-level features (e.g., SIFT [Lowe 1999] in computer vision) that were more suitable for popular machine learning methods.
Deep learning. Around 2010, a revival of deep neural networks under the moniker of deep learning [LeCun et al. 2015] started gaining traction in the field of machine learning. Deep learning was fueled by larger datasets, more computation (notably, the availability of GPUs), and greater audacity. Deep neural networks would be trained on the raw inputs (e.g., pixels), and higher-level features would emerge through training (a process dubbed 'representation learning'). This led to massive performance gains on standard benchmarks, for example, in the seminal work of AlexNet [Krizhevsky et al. 2012] on the ImageNet dataset [Deng et al. 2009]. Deep learning also reflected a further shift towards homogenization: rather than having bespoke feature engineering pipelines for each application, the same deep neural network architecture could be used for many applications.
Foundation models. Foundation models have taken shape most strongly in NLP, so we focus our story there for the moment. That said, much as deep learning was popularized in computer vision but exists beyond it, we understand foundation models as a general paradigm of AI, rather than specific to NLP in any way. By the end of 2018, the field of NLP was about to undergo another seismic change, marking the beginning of the era of foundation models. On a technical level, foundation models are enabled by transfer learning [Thrun 1998] and scale . The idea of transfer learning is to take the 'knowledge' learned from one task (e.g., object recognition in images) and apply it to another task (e.g., activity recognition in videos). Within deep learning, pretraining is the dominant approach to transfer learning: a model is trained on a surrogate task (often just as a means to an end) and then adapted to the downstream task of interest via fine-tuning .
Transfer learning is what makes foundation models possible, but scale is what makes them powerful. Scale required three ingredients: (i) improvements in computer hardware - e.g., GPU throughput and memory have increased 10 × over the last four years (§4.5: systems); (ii) the development of the Transformer model architecture [Vaswani et al. 2017] that leverages the parallelism of the hardware to train much more expressive models than before (§4.1: modeling); and (iii) the availability of much more training data.
The importance of the availability of data and the ability to harness it cannot be underestimated. Transfer learning with annotated datasets has been common practice for at least a decade, for example, pretraining on the ImageNet dataset [Deng et al. 2009] for image classification in the computer vision community. However, the non-trivial cost of annotation imposes a practical limit on the benefits of pretraining.
In self-supervised learning on the other hand, the pretraining task is derived automatically from unannotated data. 3 For example, the masked language modeling task used to train BERT [Devlin et al. 2019] is to predict a missing word in a sentence given its surrounding context (e.g., I like sprouts ). Self-supervised tasks are not only more scalable, only depending on unlabeled data, but they are designed to force the model to predict parts of the inputs, making them richer and potentially more useful than models trained on a more limited label space.
3 Interestingly, self-supervised learning was dominant in the early days of deep learning [Hinton et al. 2006], but was for a decade largely overtaken by pure supervised learning as labeled datasets became larger.
There had been considerable progress in self-supervised learning dating back to word embeddings [Turian et al. 2010; Mikolov et al. 2013; Pennington et al. 2014], which associated each word with a context-independent vector, provided the basis for a wide range of NLP models. Shortly thereafter, self-supervised learning based on autoregressive language modeling (predict the next word given the previous words) [Dai and Le 2015] became popular. This produced models that represented words in context, such as GPT [Radford et al. 2018], ELMo [Peters et al. 2018], and ULMFiT [Howard and Ruder 2018]. 4
The next wave of developments in self-supervised learning - BERT [Devlin et al. 2019] GPT-2 [Radford et al. 2019], RoBERTa [Liu et al. 2019], T5 [Raffel et al. 2019], BART [Lewis et al. 2020a] quickly followed, embracing the Transformer architecture, incorporating more powerful deep bidirectional encoders of sentences, and scaling up to larger models and datasets.
While one can view this last wave of technical developments purely through the lens of selfsupervised learning, there was a sociological inflection point around the introduction of BERT. Before 2019, self-supervised learning with language models was essentially a subarea in NLP, which progressed in parallel to other developments in NLP. After 2019, self-supervised learning with language models became more of a substrate of NLP, as using BERT has become the norm. The acceptance that a single model could be useful for such a wide range of tasks marks the beginning of the era of foundation models.
Foundation models have led to an unprecedented level of homogenization : Almost all state-ofthe-art NLP models are now adapted from one of a few foundation models, such as BERT, RoBERTa, BART, T5, etc. While this homogenization produces extremely high leverage (any improvements in the foundation models can lead to immediate benefits across all of NLP), it is also a liability; all AI systems might inherit the same problematic biases of a few foundation models [Bolukbasi et al. 2016; Caliskan et al. 2017; Abid et al. 2021, inter alia ]) - see §5.1: fairness, §5.6: ethics for further discussion.
Weare also beginning to see a homogenization across research communities. For example, similar Transformer-based sequence modeling approaches are now applied to text [Devlin et al. 2019; Radford et al. 2019; Raffel et al. 2019], images [Dosovitskiy et al. 2020; Chen et al. 2020d], speech [Liu et al. 2020d], tabular data [Yin et al. 2020], protein sequences [Rives et al. 2021], organic molecules [Rothchild et al. 2021], and reinforcement learning [Chen et al. 2021b; Janner et al. 2021]. These examples point to a possible future where we have a unified set of tools for developing foundation models across a wide range of modalities [Tamkin et al. 2021b].
Besides the homogenization of approaches, we also see the homogenization of actual models across research communities in the form of multimodal models - e.g., foundation models trained on language and vision data [Luo et al. 2020; Kim et al. 2021a; Cho et al. 2021; Ramesh et al. 2021; Radford et al. 2021]. Data is naturally multimodal in some domains-e.g., medical images, structured data, clinical text in healthcare (§3.1: healthcare). Thus, multimodal foundation models are a natural way of fusing all the relevant information about a domain, and adapting to tasks that also span multiple modes (Figure 2).
Foundation models have also led to surprising emergence which results from scale. For example, GPT-3 [Brown et al. 2020], with 175 billion parameters compared to GPT-2's 1.5 billion, permits in-context learning , in which the language model can be adapted to a downstream task simply by providing it with a prompt (a natural language description of the task), an emergent property that was neither specifically trained for nor anticipated to arise.
4 The prescient work of Collobert and Weston [2008] is related: they trained on a scalable task akin to masked language modeling jointly with downstream tasks, rather than producing a single foundation model that can be adapted after the fact to downstream tasks.
Fig. 2. A foundation model can centralize the information from all the data from various modalities. This one model can then be adapted to a wide range of downstream tasks.
<details>
<summary>Image 2 Details</summary>
### Visual Description
\n
## Diagram: Foundation Model Training and Tasks
### Overview
This diagram illustrates the process of training a Foundation Model using various data types and adapting it to perform different tasks. The diagram is segmented into three main regions: "Data" (left), "Foundation Model" (center), and "Tasks" (right). Arrows indicate the flow of information from data to the model during training and from the model to tasks during adaptation.
### Components/Axes
The diagram consists of the following components:
* **Data:** Includes Text (represented by a document icon), Images (represented by a picture frame icon), Speech (represented by a sound wave icon), Structured Data (represented by a table icon), and 3D Signals (represented by a radar/signal icon).
* **Foundation Model:** A central sphere labeled "Foundation Model".
* **Tasks:** Includes Question Answering (represented by a question mark and flag icon), Sentiment Analysis (represented by a smiley face icon), Information Extraction (represented by a magnifying glass icon), Image Captioning (represented by a picture and text bubble icon), Object Recognition (represented by a cube icon), and Instruction Following (represented by a computer screen and cursor icon).
* **Arrows:** Indicate the flow of data during "Training" (from Data to Foundation Model) and "Adaptation" (from Foundation Model to Tasks).
* **Color Coding:** Each task is associated with a unique color. These colors are used to trace the adaptation path from the Foundation Model to each specific task.
### Detailed Analysis or Content Details
The diagram shows a flow of information.
1. **Data Input:** Five data types feed into the Foundation Model during the "Training" phase.
2. **Foundation Model:** The central "Foundation Model" receives data from all five sources.
3. **Adaptation to Tasks:** The Foundation Model then adapts to perform six different tasks. Each task is linked to the Foundation Model via a curved arrow, colored to match the task's icon.
* **Question Answering:** (Orange) - The arrow originates from the Foundation Model and leads to the Question Answering task.
* **Sentiment Analysis:** (Red) - The arrow originates from the Foundation Model and leads to the Sentiment Analysis task.
* **Information Extraction:** (Green) - The arrow originates from the Foundation Model and leads to the Information Extraction task.
* **Image Captioning:** (Blue) - The arrow originates from the Foundation Model and leads to the Image Captioning task.
* **Object Recognition:** (Purple) - The arrow originates from the Foundation Model and leads to the Object Recognition task.
* **Instruction Following:** (Pink) - The arrow originates from the Foundation Model and leads to the Instruction Following task.
### Key Observations
The diagram emphasizes the versatility of Foundation Models. A single model can be trained on diverse data types and then adapted to perform a wide range of downstream tasks. The color-coding effectively illustrates the connection between the Foundation Model and each specific task. The diagram does not provide any quantitative data or specific performance metrics.
### Interpretation
The diagram represents a common paradigm in modern machine learning, particularly with the rise of large language models and multimodal models. The "Foundation Model" represents a pre-trained model with a broad understanding of data, which can then be fine-tuned or adapted for specific applications. The diagram highlights the benefits of this approach:
* **Data Efficiency:** Training a single Foundation Model on diverse data can reduce the need for task-specific training data.
* **Transfer Learning:** Knowledge gained during pre-training can be transferred to new tasks, improving performance and reducing training time.
* **Generalization:** Foundation Models are designed to generalize well to unseen data and tasks.
The diagram suggests a pipeline where raw data is transformed into a generalized representation within the Foundation Model, and then specialized representations are created for specific tasks. The use of different icons for each data type and task emphasizes the diversity of applications. The diagram is conceptual and does not provide details about the model architecture, training process, or adaptation techniques. It serves as a high-level overview of the Foundation Model paradigm.
</details>
Homogenization and emergence interact in a potentially unsettling way. Homogenization could potentially provide enormous gains for many domains where task-specific data is quite limited - see the opportunities presented in several such domains (e.g., §3.1: healthcare, §3.2: law, §3.3: education); on the other hand, any flaws in the model are blindly inherited by all adapted models (§5.1: fairness, §5.6: ethics). Since the power of foundation models comes from their emergent qualities rather than their explicit construction, existing foundation models are hard to understand (§4.4: evaluation, §4.10: theory, §4.11: interpretability) and they have unexpected failure modes (§4.7: security, §4.8: robustness). Since emergence generates substantial uncertainty over the capabilities and flaws of foundation models, aggressive homogenization through these models is risky business. Derisking is the central challenge in the further development of foundation models from an ethical (§5.6: ethics) and AI safety (§4.9: ai-safety) perspective.
## 1.1.1 Naming.
We introduce the term foundation models to fill a void in describing the paradigm shift we are witnessing; we briefly recount some of our reasoning for this decision. Existing terms (e.g., pretrained model , self-supervised model ) partially capture the technical dimension of these models, but fail to capture the significance of the paradigm shift in an accessible manner for those beyond machine learning. In particular, foundation model designates a model class that are distinctive in their sociological impact and how they have conferred a broad shift in AI research and deployment. In contrast, forms of pretraining and self-supervision that technically foreshadowed foundation models fail to clarify the shift in practices we hope to highlight.
Fig. 3. Before reasoning about the social impact of foundation models, it is important to understand that they are part of a broader ecosystem that stretches from data creation to deployment. At both ends, we highlight the role of people as the ultimate source of data into training of a foundation model, but also as the downstream recipients of any benefits and harms. Thoughtful data curation and adaptation should be part of the responsible development of any AI system. Finally, note that the deployment of adapted foundation models is a decision separate from their construction, which could be for research.
<details>
<summary>Image 3 Details</summary>
### Visual Description
\n
## Diagram: AI Lifecycle - Data Creation to Deployment
### Overview
The image depicts a diagram illustrating the lifecycle of an Artificial Intelligence (AI) system, from initial data creation through to deployment. The process is segmented into five stages: Data Creation, Data Curation, Training, Adaptation, and Deployment. Each stage is visually represented with icons and arrows indicating the flow of information. The diagram emphasizes the human element in both data creation and the final deployment phase.
### Components/Axes
The diagram is divided into five distinct, vertically oriented sections, each with a header indicating the stage of the AI lifecycle. There are no explicit axes in the traditional sense, but the flow is clearly left-to-right. The sections are:
* **Data Creation:** Represented by a light blue background.
* **Data Curation:** Represented by a purple background.
* **Training:** Represented by a yellow background.
* **Adaptation:** Represented by an orange background.
* **Deployment:** Represented by a pink background.
Each section contains icons representing the activities within that stage. Arrows connect the stages, showing the progression of the AI lifecycle.
### Detailed Analysis or Content Details
**1. Data Creation (Light Blue):**
This stage shows a series of diverse human figures (approximately 7 individuals) on the left side. Each figure is associated with a different data source icon, including:
* Documents/Reports
* Financial Reports
* Lists/Checklists
* Emails
* Books
* Graphs/Charts
* Computer/Code
The arrows point from each human figure/data source combination towards the Data Curation stage.
**2. Data Curation (Purple):**
This stage features three cylindrical shapes, colored in shades of purple (dark purple, medium purple, and a lighter purple). These likely represent data storage or databases. Arrows originate from the Data Creation stage and converge on these cylinders.
**3. Training (Yellow):**
This stage contains two spherical, network-like structures. The top sphere is predominantly blue and green, while the bottom sphere is pink and purple. Arrows originate from the Data Curation stage and point towards these spheres, indicating the data is used for training.
**4. Adaptation (Orange):**
This stage shows two open box icons. The top box is multicolored (green, yellow, and blue), while the bottom box is pink and purple. Arrows originate from the Training stage and point towards these boxes, suggesting the AI model is being adapted. There is a bidirectional arrow between the two spheres in the Training stage and the boxes in the Adaptation stage, indicating a feedback loop.
**5. Deployment (Pink):**
This stage features a series of human figures (approximately 6 individuals) on the right side. Arrows originate from the Adaptation stage and point towards these figures, indicating the AI model is being deployed to end-users.
### Key Observations
* The diagram emphasizes the iterative nature of the AI lifecycle, particularly between the Training and Adaptation stages.
* The diversity of data sources in the Data Creation stage suggests a broad range of input data.
* The human representation in both the Data Creation and Deployment stages highlights the importance of human involvement in the AI lifecycle.
* The color scheme is consistent throughout the diagram, with each stage having a distinct color.
### Interpretation
The diagram illustrates a simplified, high-level overview of the AI lifecycle. It demonstrates the flow of data from its creation through curation, training, adaptation, and ultimately, deployment to end-users. The feedback loop between Training and Adaptation suggests a continuous improvement process, where the model is refined based on its performance. The inclusion of diverse human figures in the Data Creation and Deployment stages underscores the importance of considering human factors and ensuring inclusivity in AI development. The diagram doesn't provide specific data points or numerical values, but rather focuses on the conceptual flow and key stages of the AI lifecycle. It serves as a visual aid for understanding the overall process and the relationships between different components. The diagram is a conceptual illustration and does not contain factual data.
</details>
Additionally, while many of the iconic foundation models at the time of writing are language models, the term language model is simply too narrow for our purpose: as we describe, the scope of foundation models goes well beyond language. We also considered terms such as general-purpose model and multi-purpose model that capture the important aspect that these models can serve multiple downstream tasks, but both fail to capture their unfinished character and the need for adaptation. Terms such as task-agnostic model would capture the manner of training, but fail to capture the significant implication to downstream applications.
We chose the new term foundation models to identify the models and the emerging paradigm that are the subject of this report. In particular, the word 'foundation' specifies the role these models play: a foundation model is itself incomplete but serves as the common basis from which many task-specific models are built via adaptation. We also chose the term 'foundation" to connote the significance of architectural stability, safety, and security: poorly-constructed foundations are a recipe for disaster and well-executed foundations are a reliable bedrock for future applications. At present, we emphasize that we do not fully understand the nature or quality of the foundation that foundation models provide; we cannot characterize whether the foundation is trustworthy or not. Thus, this is a critical problem for researchers, foundation model providers, application developers who rely on foundation models, policymakers, and society at large to address.
## 1.2 Social impact and the foundation models ecosystem
Foundation models are scientifically interesting due to their impressive performance and capabilities, but what makes them critical to study is the fact that they are quickly being integrated into realworld deployments of AI systems with far-reaching consequences on people. For example, Google search, which boasts 4 billion users, now depends on foundation models like BERT [Devlin et al. 2019] as one of its signals. 5
5 https://blog.google/products/search/search-language-understanding-bert/
We must thus pause and ask: What is the nature of this social impact? In this report, we address many aspects of this question: the potential exacerbation of social inequities (§5.1: fairness), the economic impact due to increased capabilities (§5.5: economics), the environmental impact due to increased computation demands (§5.3: environment), potential concerns of amplifying disinformation (§5.2: misuse), legal ramifications due to powerful generative capabilities (§5.4: legality), ethical issues resulting from homogenization, and the broader political economy in which foundation models are developed and deployed (§5.6: ethics). Given the protean nature of foundation models and their unmapped capabilities, how can we responsibly anticipate and address the ethical and societal considerations they raise? A recurring theme is that it is easier to reason about the social impact of specific systems deployed to specific users than it is to reason about the social impact of foundation models, which could be adapted to any number of unforeseen downstream systems.
Before attempting to answer these questions, we need to lay some groundwork. First, let us distinguish between research on foundation models and deployment of foundation models. Most of what is publicly known is foundation models research - through academic papers, demonstrations, and progress on leaderboards. While the production of knowledge can play a vital role in shaping the future, the direct social impact is through the actual deployment of these models, which is governed by proprietary practices on often private data. Sometimes the deployment is through new products - e.g., GitHub's Copilot 6 based on OpenAI's Codex model [Chen et al. 2021f], but often, it is through upgrades to existing products (e.g., Google search using BERT). Research models are often not extensively tested and might have unknown failure modes; warning labels should be placed on research models that are not fit to deploy. On the other hand, deployed foundation models that actually affect people's lives should be subject to much more rigorous testing and auditing.
To further understand the research and deployment of foundation models, we must zoom out and consider the full ecosystem that these foundation models inhabit, from data creation to actual deployment. It is important to note that the foundation model is only one component (though an increasingly important component) of an AI system. Simplifying, we can think about the ecosystem of a foundation model in terms of sequence of stages, extending the training and adaptation stages from before. 7 Appropriately, as we're interested in social impact, people occupy both ends of the pipeline. This ecosystem view allows us to see that different questions about foundation models (e.g., whether a foundation model is ethical) should actually be answered with respect to different stages.
- (1) Data creation : Data creation is fundamentally a human-centric process: all data is created by people and most data is at least implicitly about people. Sometimes data is created by people for other people in the form of emails, articles, photos, etc., and sometimes it is a measurement of people (e.g., genomic data) or a measurement of the environment people live in (e.g., satellite images). It is important to note that all data has an owner and is created with a purpose (where that purpose may or may not include training a foundation model).
- (2) Data curation : Data is then curated into datasets. There is no single natural distribution of data; even the most permissive Internet crawl requires some selection and post-filtering. Ensuring data relevance and quality while respecting legal and ethical constraints is critical but challenging. While this is recognized in industry, it is underappreciated in AI research (§4.6: data).
6 https://copilot.github.com/
7 In practice, the end of the pipeline is followed by monitoring, and feedback is used to readjust the previous stages.
- (3) Training : Training foundation models on these curated datasets 8 is the celebrated centerpiece in AI research, though it is only one of many stages.
- (4) Adaptation : In the context of machine learning research, adaptation is about creating a new model based on the foundation model that performs some task (e.g., document summarization). For deployment, adaptation is about creating a system, which requires potentially many different modules, custom rules (e.g., restrictions on the output space) or classifiers (e.g., for toxicity classification), and combination with other complementary signals (e.g., a question answering model's generated answers would be validated against relevant documents). For example, a problematic model capable of generating toxic content might be tolerable if appropriate precautions are taken downstream. The extra application-specific logic is crucial for mitigating harms.
- (5) Deployment : The direct social impact of an AI system occurs when it is deployed to people. Though we would not want to deploy potentially harmful foundation models trained on questionable data, there might still be value in permitting them in research to advance scientific understanding, though one must still exercise caution. More generally, it is standard practice in large-scale deployments to conduct gradual releases, where deployment happens to an increasing fraction of users; this can partially mitigate any potential harms.
While this report is about foundation models, it is important to note that many of the impacts come from decisions made in other stages in the pipeline, and thoughtful monitoring and intervention is needed at every stage. While large organizations might own the entire pipeline, each stage could be performed by a different organization, e.g., a company which specializes in creating custom foundation models for various domains that application-developers can use.
Think ecosystem, act model. While the social impact depends on the whole ecosystem, it is still important to be able to reason about the social implications of a foundation model, given that many researchers' and practitioners' purview is restricted to the training stage. This is difficult because foundation models are unfinished intermediate objects that can be adapted to many downstream applications, sometimes by an entirely different entity for unforeseen purposes. What we need are two things: (i) surrogate metrics for a representative set of potential downstream evaluation (§4.4: evaluation), and (ii) a commitment to documenting these metrics [Mitchell et al. 2019] similar to data sheets for materials such as metals and plastics, which can be adapted to many downstream use cases.
Characterizing the potential downstream social impact of foundation models is challenging and demands a deep understanding of both the technological ecosystem and of society. One cannot fully assess the harms (§5.1: fairness) of a foundation model without recognizing how it will be deployed, and one cannot just define automatic metrics without considering the rich social and historical context.
## 1.3 The future of foundation models
Foundation models have demonstrated raw potential, but we are still in the early days. Despite their deployment into the real world, these models are very much research prototypes that are poorly understood. Even the professional norms - what Robert Merton calls the ethos of science [Merton 1979] - around foundation models are underdeveloped. For example, there is lack of agreement on basic questions such as when models are 'safe' to release or how the community should react in response to methodological misconduct. Given that the future of foundation models is thus filled with uncertainty, a big question is: who will determine this future?
8 A foundation model (e.g., Codex) can also be trained with another model (e.g., GPT-3) as a starting point.
Disciplinary diversity. The technology behind foundation models is based on decades of research in machine learning, optimization, NLP, computer vision, and other fields. These technical contributions have come from both academia and industrial research labs. However, research on building foundation models themselves has occurred almost exclusively in industry - big tech companies such as Google, Facebook, Microsoft, or Huawei, or startups such as OpenAI or AI21 Labs, though AI2 is a notable exception [Peters et al. 2018; Zellers et al. 2019b].
The furious pace of technological progress and the entrenchment due to centralization raise powerful concerns that demand the attention of humanists and social scientists in addition to technologists. We should not rely on post-hoc audits of ethical and social consequences, conducted only after the technical architecture and deployment decisions have been made. We instead need to infuse social considerations and ethical design deeply into the technological development of foundation models and their surrounding ecosystem from the start. Academic institutions are unique in that they host the widest set of disciplines under one roof, thus bringing together computer scientists, social scientists, economists, ethicists, legal scholars, etc. Given the importance of disciplinary diversity in understanding and solving problems that combine technical, ethical, legal, social, and political dimensions [Hong and Page 2004; Solomon 2006; Steel et al. 2018], we therefore see academia as playing a crucial role in developing foundation models in such a way to promote their social benefit and mitigate their social harms, as well as determining the contexts under which actions in each of the stages of the ecosystem (§1.2: ecosystem) ranging from data curation to deployment should be strictly prohibited.
Incentives. The political economy in which foundations models are designed, developed, and deployed provides an inevitable incentive structure for decision-making at every stage. How people and institutions respond to incentives is an elementary lesson of economics. Market-driven commercial incentives can align well with social benefit: making foundation models more accurate, reliable, safe, and efficient while searching for a wide variety of potential use cases can produce a great deal of social utility. However, commercial incentives can also lead to market failures and underinvestment in domains where shareholders are unable to capture the value of innovation. Just as the pharmaceutical industry has little incentive to devote significant resources to the research and development of malaria treatments, because poor people cannot afford medications, 9 the tech industry has little incentive to devote significant resources to technologies designed for improving the condition of poor and marginalized people [Reich et al. 2021]. What's more, commercial incentives can lead companies to ignore social externalities [Acemoglu 2021; Reich et al. 2021] such as the technological displacement of labor, the health of an informational ecosystem required for democracy, the environmental cost of computing resources, and the profit-driven sale of technologies to non-democratic regimes. Finally, there is little incentive for any given company to create an open, decentralized ecosystem for developing foundation models that encourages broad participation.
In contrast, the long-standing and deeply-seated research mission of universities is the production and dissemination of knowledge and creation of global public goods [Kerr 2001; Rhoten and Calhoun 2011; Nussbaum 2010]. We believe that academia is distinctively positioned to shape the development of foundation models to ensure that we capture directions with potentially large social benefit that might not otherwise be prioritized by industry.
Loss in accessibility. Unfortunately, academia has not been able to participate in the fullest way possible due to the loss in accessibility. One of the often overlooked effects of the deep learning revolution was the increase in reproducibility and open science: it increasingly became the norm
9 See https://www.gatesfoundation.org/about/our-role.
to publicly release code and datasets, and packages such as TensorFlow [Abadi et al. 2016] and PyTorch [Paszke et al. 2019] made it much easier for people to collaborate and build off of each other's work. Initiatives like the ML Reproducibility Challenge 10 as well as reproducibility checklists adopted by major conferences [Pineau et al. 2020], alongside platforms like CodaLab Worksheets 11 helped advance community standards for reproducibility. This resulted in a surge in technological innovation and progress.
Foundation models start to roll back this positive trend. Some models (e.g., GPT-3) are not released at all (only API access to a limited pool of people). Even datasets (e.g., for GPT-2) are not released. While trained models may be available (e.g., BERT), the actual training of foundation models is unavailable to the vast majority of AI researchers, due to the much higher computational cost and the complex engineering requirements.
Some meaningful research can still be done by training smaller models within reach of an academic budget, and indeed the surprisingly regularity predicted by scaling laws [Kaplan et al. 2020] make this a viable strategy for cases where the differences due to scale are quantitative (e.g., accuracy goes up). However, due to the emergent nature of these foundation models, some functionalities like in-context learning have only been demonstrated in models of sufficient size, so scale is needed to even ask the right questions.
It is also possible to productively study pre-existing models that have been released; indeed, this has led to a large subcommunity within NLP for probing these models [Rogers et al. 2020; Manning et al. 2020]. Having access to existing models can be useful for powering downstream applications or identifying defects (e.g., bias), but this might not be enough for us to design better architectures or training objectives for foundation models that can fix these defects (e.g., mitigate the bias). It is worth reflecting on how much of NLP research today is based on BERT, a particular (and somewhat arbitrary) foundation model. Given the need to infuse social awareness and ethical design into the construction of these models, it is possible that we need to build foundation models that look quite different from what exists today. This will demand intense experimentation at scale.
Community efforts such as EleutherAI 12 and Hugging Face's BigScience project 13 are attempting to train large foundation models, but the gap between the private models that industry can train and the ones that are open to the community will likely remain large if not grow. Further, today startups (OpenAI, Anthropic, AI21 Labs, etc.) are much more well-resourced than academia and can therefore still afford to train the largest foundation models (e.g., OpenAI's GPT-3). However, big tech companies are on a completely different level in terms of resources, especially in terms of the infrastructure, users, and data that come from their market position. The fundamental centralizing nature of foundation models means that the barrier to entry for developing them will continue to rise, so that even startups, despite their agility, will find it difficult to compete, a trend that is reflected in the development of search engines [Radinsky 2015].
One way to close the resource gap is for the government to invest in public infrastructure. We can look to Big Science projects such as the Hubble Space Telescope and the Large Hadron Collider as inspiration, where substantial investment made possible fundamental scientific discoveries which wouldn't have been possible. One can imagine a similar infrastructure for computing, from which academic research on foundation models would greatly benefit. In the US, the nascent National Research Cloud initiative 14 is a step in this direction.
10 https://paperswithcode.com/rc2020
11 https://worksheets.codalab.org/
12 https://www.eleuther.ai/
13 https://bigscience.huggingface.co/
14 https://hai.stanford.edu/policy/national-research-cloud
Another complementary approach is to rely on volunteer computing, in which any of the billions of computing devices (nodes) can connect to a central server and contribute computation. The Folding@home project has successfully implemented this approach for simulating protein dynamics [Beberg et al. 2009]. Recently, the Learning@home project is attempting to harness volunteer computing for training foundation models [Ryabinin and Gusev 2020]. The high latency connections between nodes and the high bandwidth requirements for training foundation models make this an open technical challenge.
Summary. There are tremendous economic incentives to push the capabilities and scale of foundation models, so we anticipate steady technological progress over the coming years. But the suitability of a technology relying largely on emergent behavior for widespread deployment to people is unclear. What is clear that we need to be cautious, and that now is the time to establish the professional norms that will enable the responsible research and deployment of foundation models. Academia and industry need to collaborate on this: industry ultimately makes concrete decisions about how foundation models will be deployed, but we should also lean on academia, with its disciplinary diversity and non-commercial incentives around knowledge production and social benefit, to provide distinctive guidance on the development and deployment of foundation models that is both technically and ethically grounded.
## 1.4 Overview of this report
In March 2021, we created an informal community at Stanford University of students, faculty, and researchers interested in some aspect of foundation models. 15 From the very beginning, the community included not just AI researchers, but those eager to apply foundation models to their domain (e.g., healthcare and law), as well as those who were interested in societal concerns (e.g., ethics and economics). As discussions progressed, we noticed that there were many gaps in mutual understanding - how the technology worked, how industry develops foundation models, how to think about the ethical concerns, etc., and existing literature only covered bits and pieces. We wanted to therefore provide a fuller picture of foundation models, identify opportunities and risks, and establish a constructive vision for the future responsible development of foundation models.
The writing of this report was an experiment: we had over 100 people from different backgrounds come together to write a single report covering a wide range of aspects of foundation models. A large part of this report is a survey of existing work, but through many discussions, we have unified it in one report to highlight all the interdisciplinary connections.
Structure. The report is divided into 26 sections, each discussing one aspect of foundation models. The sections are grouped into four parts: capabilities (§2: capabilities), applications (§3: applications), technology (§4: technology), and society (§5: society), although there are many connections across sections. These connections highlight an integrated approach in which the technologies and capabilities are developed in a way that is sensitive to real societal concerns, while being inspired by and grounded out in applications.
While we have sought to capture most of the important topics surrounding foundation models, this report will inevitably be incomplete, especially as the field evolves quickly. For example, many applications (e.g., natural sciences, music, finance, agriculture) are not included, though they are as likely to be affected as the applications we have chosen to discuss. It would also be interesting to
15 This community led to the founding of the Center for Research on Foundation Models (CRFM) , a new interdisciplinary initiative at the Stanford Institute for Human-Centered AI (HAI).
Fig. 4. This report is divided into four parts: capabilities, applications, technology, and society, where each part contains a set of sections, and each section covers one aspect of foundation models.
<details>
<summary>Image 4 Details</summary>
### Visual Description
\n
## Diagram: Paper Roadmap
### Overview
The image presents a roadmap, visually organized into four sections: Capabilities, Applications, Technology, and Society. Each section contains a series of icons representing different concepts, accompanied by labels and version numbers (e.g., 2.1, 3.2). The overall layout is a vertical progression, suggesting a sequence of development or consideration. The title "Paper Roadmap" is positioned at the top of the image.
### Components/Axes
The diagram is structured into four main sections, each with a heading and a set of icons. There are no traditional axes in the chart sense, but the sections themselves act as categories. Each icon is labeled with a descriptive term and a numerical identifier.
* **Section 2: Capabilities** - Includes Language (2.1), Vision (2.2), Robotics (2.3), Reasoning (2.4), Interaction (2.5), and Philosophy (2.6).
* **Section 3: Applications** - Includes Healthcare (3.1), Law (3.2), and Education (3.3).
* **Section 4: Technology** - Includes Modeling (4.1), Training (4.2), Adaptation (4.3), Evaluation (4.4), Systems (4.5), Data (4.6), Security (4.7), Robustness (4.8), AI Safety & Alignment (4.9), Theory (4.10), and Interpretability (4.11).
* **Section 5: Society** - Includes Inequity (5.1), Misuse (5.2), Environment (5.3), Legality (5.4), Economics (5.5), and Ethics (5.6).
### Detailed Analysis or Content Details
The diagram consists of icons arranged horizontally within each section. Each icon is paired with a label and a version number. The version numbers appear to indicate a progression or prioritization within each category.
* **Language (2.1):** Icon of a book.
* **Vision (2.2):** Icon of an eye.
* **Robotics (2.3):** Icon of a robot.
* **Reasoning (2.4):** Icon of a gear/cog.
* **Interaction (2.5):** Icon of two people.
* **Philosophy (2.6):** Icon of a head.
* **Healthcare (3.1):** Icon of a person with a cross.
* **Law (3.2):** Icon of scales of justice.
* **Education (3.3):** Icon of a graduation cap.
* **Modeling (4.1):** Icon of a geometric shape.
* **Training (4.2):** Icon of a microphone.
* **Adaptation (4.3):** Icon of a chameleon.
* **Evaluation (4.4):** Icon of a checklist.
* **Systems (4.5):** Icon of a globe with connections.
* **Data (4.6):** Icon of a document.
* **Security (4.7):** Icon of a shield with a lock.
* **Robustness (4.8):** Icon of a pillar.
* **AI Safety & Alignment (4.9):** Icon of binary code.
* **Theory (4.10):** Icon of a funnel.
* **Interpretability (4.11):** Icon of a magnifying glass.
* **Inequity (5.1):** Icon of broken pieces.
* **Misuse (5.2):** Icon of a hand with a warning sign.
* **Environment (5.3):** Icon of a tree.
* **Legality (5.4):** Icon of a gavel.
* **Economics (5.5):** Icon of a coin.
* **Ethics (5.6):** Icon of a person.
### Key Observations
The diagram presents a holistic view of considerations related to "Paper Roadmap," encompassing capabilities, applications, technology, and societal impacts. The version numbers suggest a potential order of development or importance. The visual arrangement is clean and organized, making it easy to grasp the different components.
### Interpretation
This "Paper Roadmap" appears to outline the key areas of focus for a research or development project, likely related to Artificial Intelligence (AI) given the inclusion of "AI Safety & Alignment." The roadmap progresses from foundational capabilities (Language, Vision) to practical applications (Healthcare, Law) and the underlying technology (Modeling, Training) before concluding with societal considerations (Inequity, Ethics). The version numbers could represent a phased approach to development, with lower numbers indicating earlier stages. The inclusion of societal impacts as the final section highlights the importance of responsible AI development. The roadmap is not presenting data in the traditional sense, but rather a conceptual framework for organizing thoughts and priorities. It's a high-level overview, and each category would likely require further detailed exploration.
</details>
study how foundation models relate to research in neuroscience, cognitive science, and psychology to explain intelligence and aid efforts in computational social science to understand society.
Author Contributions Percy Liang initiated and conceptualized the framing and structure of the overall report. He and Rishi Bommasani worked together to lead the decentralized writing effort and provided guidance on individual sections. Drew A. Hudson created all the figures in the report, discussing their structure and content with the authors of each section. Each of the 26 sections of this report was written by a subset of authors, whose names are listed at the beginning of each section. There were, however, many discussions that spanned multiple sections, so the actual contributions to each section generally came from a broader set. Finally, we note that not all the views expressed in this report are held by all the authors.
## 1.4.1 Overview of capabilities.
Foundation models acquire various capabilities that can power applications. We have chosen to discuss five potential capabilities: the ability to process different modalities (e.g., language, vision), to affect the physical world (robotics), to perform reasoning, and to interact with humans (interaction). Finally, we conclude with a philosophical discussion of potential limits on their capabilities.
§2.1: Language. NLP as a field has blazed the trail for foundation models. While these models dominate standard benchmarks, there is a clear gap between the capabilities these models acquire currently and those that characterize language as a complex system for human communication and thought. In response to this, we emphasize the full range of linguistic variation (e.g., different styles, dialects, languages), which poses an opportunity and challenge given some variants are data-limited. Further, child language acquisition is more sample efficient than the training of foundation models; we examine how signals beyond text and grounding may help to bridge this gap. Both of these characteristics of language provide clear directions for future foundation models research.
§2.2: Vision. Computer vision led the adoption of deep learning in AI [Russakovsky et al. 2015], demonstrating that models pretrained on large annotated datasets can transfer to numerous downstream settings. Now, pretraining on web-scale raw data instead of curated datasets, foundation models are on the rise in computer vision [e.g., Radford et al. 2021]. These models have shown promising results for standard tasks in the field, like image classification and object detection, and training on multimodal and embodied data beyond images may enable progress on significant challenges (e.g., 3D geometric and physical understanding, commonsense reasoning). We also discuss some of the key challenges in modeling (e.g., the ability to scale effectively to videos) and evaluation (e.g., the measurement of higher-order capabilities) along with the applications (e.g., ambient intelligence for healthcare) and societal considerations (e.g., surveillance) that will determine the impact of foundation models for computer vision going forward.
§2.3: Robotics. A longstanding goal of robotics research is to develop 'generalist' robots capable of performing myriad tasks across physically diverse environments. Unlike language and vision, which have led the way with foundation models both due to the abundance of raw data to train these models on and the availability of virtual applications to apply these models to, robotics faces fundamental challenges due to being anchored to the physical world. The principal challenge in developing new types of foundation models for robotics - different in nature than their language and vision counterparts - is acquiring sufficient data of the right form that is conducive to learning: we explore how plentiful data (e.g., generic videos of humans, amongst others) that is not specific to particular environments and across modalities (e.g., language, vision) may help to bridge this gap. These new robotic foundation models could allow for easier task specification and learning , ushering in new applications (e.g., better robotic assistance for household tasks) and heightening the importance of robustness and safety (e.g., formal safety evaluation).
§2.4: Reasoning and search. Reasoning and search problems such as theorem proving and program synthesis have been long-standing challenges in AI. The combinatorial search space renders traditional search-based methods intractable. However, humans are known to operate intuitively even in the most mathematical of domains [Lakoff and Núñez 2000], and indeed existing work such as AlphaGo have already shown that deep neural networks can be effective in guiding the search space. But humans also transfer knowledge across tasks, facilitating much more efficient adaptation and the ability to reason more abstractly. Foundation models offer the possibility of closing this gap: their multi-purpose nature along with their strong generative and multimodal capabilities offer new leverage for controlling the combinatorial explosion inherent to search.
§2.5: Interaction. Foundation models show clear potential to transform the developer and user experience for AI systems: foundation models lower the difficulty threshold for prototyping and building AI applications due to their sample efficiency in adaptation, and raise the ceiling for novel user interaction due to their multimodal and generative capabilities. This provides a synergy we encourage going forward: developers can provide applications that better fit the user's needs and values , while introducing far more dynamic forms of interaction and opportunities for feedback .
§2.6: Philosophy of understanding. What could a foundation model come to understand about the data it is trained on? Focusing on the case of natural language, we identify different positions on the nature of understanding and explore their relevance for our central question. Our tentative conclusion is that skepticism about the capacity of future foundation models to understand natural language may be premature, especially where the models are trained on multi-modal data.
## 1.4.2 Overview of applications.
At present, foundation model research is largely confined to computer science and AI, with the impact of foundation models and the applications they support largely being centered in the tech industry. Moving forward, foundation models present clear potential to transform and extend the reach of AI across many sectors beyond the tech industry, suggesting a more pervasive effect on people's lives. While there is a multitude of applications and domains to consider, we we have chosen three applications - healthcare, law, and education - because they represent foundational pillars of our society. For foundation models to significantly contribute to these application domains, models will require specific capabilities (§2: capabilities) as well as technical innovation (§4: technology) to account for the unique considerations in each domain. Further, since these domains are critical to societal function (§5: society), applying foundation models in these domains requires engaging with deeply sociotechnical matters such as those those pertaining to data (§4.6: data), privacy (§4.7: security), interpretability (§4.11: interpretability), fairness (§5.1: fairness) and ethics (§5.6: ethics).
§3.1: Healthcare and biomedicine. Healthcare tasks (e.g., patient care via disease treatment) and biomedical research (e.g., scientific discovery of new therapies) require expert knowledge that is limited and expensive. Foundation models present clear opportunities in these domains due to the abundance of data across many modalities (e.g., images, text, molecules) to train foundation models, as well as the value of improved sample efficiency in adaptation due to the cost of expert time and knowledge. Further, foundation models may allow for improved interface design (§2.5: interaction) for both healthcare providers and patients to interact with AI systems, and their generative capabilities suggest potential for open-ended research problems like drug discovery. Simultaneously, they come with clear risks (e.g., exacerbating historical biases in medical datasets and trials). To responsibly unlock this potential requires engaging deeply with the sociotechnical
matters of data sources and privacy as well as model interpretability and explainability, alongside effective regulation of the use of foundation models for both healthcare and biomedicine.
§3.2: Law. Legal applications require that attorneys read and produce long coherent narratives that incorporate shifting contexts and decipher ambiguous legal standards. Foundation models may provide benefits in this domain: ample data exists in the form of legal documents and their generative capabilities are well-suited to the manygenerative tasks required in law , but significant improvements are required for foundation models to be able to reliably reason over various sources of information to generate truthful long-form documents. As is the care in healthcare (§3.1: healthcare), the sample efficiency of adaptation for foundation models is of heightened value given the costs of expert time and knowledge in the legal domain, which may allow for the re-allocation of expertise towards pressing problems of justice and government service. The responsible development of foundation models for law will require specific consideration of privacy, and highlights core limitations of existing foundation models that will require fundamental advances with respect to provenance for their behavior and guarantees for the factuality of their generation.
§3.3: Education. Education is a complex and subtle domain; effective teaching involves reasoning about student cognition and should reflect the learning goals of students. The nature of foundation models presents promise here that has yet to be realized in the sphere of AI for education: while certain many streams of data in education are individually too limited to train foundation models, the ability to leverage relevant data from outside the domain (e.g., the Internet) and make use of data across multiple modalities (e.g., textbooks, mathematical formula, diagrams, video-based tutorials) jointly offers hope for foundation models that are broadly applicable to educational tasks. If foundation models lead to a significant improvement in education-relevant capabilities, there is clear potential for new applications that align with the open-ended generative (e.g., problem generation) and interactive (e.g., feedback to teachers) aspects of foundation models; the sample efficient adaptation of foundation models suggests greater ability for adaptive and personalized learning . In this event, renewed consideration is required of hallmarks of applying technology to education (e.g., student privacy), along with certain concerns becoming more critical (e.g., inequity in access to technology in education, technology-aided plagiarism).
## 1.4.3 Overview of technology.
Now we discuss the technology behind building better model architectures, training and adaptation procedures, and of course scaling up the systems. One crucial but often overlooked topic is data where does it come from and what is its composition? In addition, we want foundation models to be robust to distribution shifts and secure against attackers. Finally, we wish to understand why foundation models work from both a mathematical perspective as well as an empirical perspective.
§4.1: Modeling. What structural properties give rise to a foundation model? In the modeling section, we explore the underlying architectures behind foundation models and identify 5 key attributes. First, we start by discussing expressivity of the computational model to capture and assimilate real-world information, and scalability - to adeptly handle large quantities of highdimensional data. These properties are successfully realized by existing architectures such as the transformer network [Vaswani et al. 2017] that underpins most foundation models to date. We then proceed to attributes may be essential for the next generation of models, including: multimodallity -to consume, process and potentially produce content from different sources and domains, memory capacity to effectively store and retrieve the acquired knowledge, and finally, compositionality , to foster successful generalization to novel settings and environments. We believe that realizing the
full potential envisioned for foundation models will hinge on modelling advances to fulfill these desiderata.
§4.2: Training. Training objectives mathematically specify how models should learn and acquire capabilities from their training data. The current status quo for training foundation models involves modality-specific objectives (e.g., masked language modeling [Devlin et al. 2019] for text and SimCLR [Chen et al. 2020c] for images) that are often chosen heuristically. We envision that future training objectives for foundation models will reflect two changes: principled selection derived from systematic evidence and evaluation (§4.4: evaluation), and domain-generality to provide rich, scalable, and unified training signal across data sources and modalities. We also discuss important design trade-offs, including generative vs discriminative training, the choice of input data representation, and the potential of future training objectives that involve explicit representations of goals.
§4.3: Adaptation. Foundation models are intermediary assets; they are unfinished and generally should not be used directly, instead requiring adaptation for specific downstream tasks. The de facto approach for adaptation has been fine-tuning, with recent work suggesting that lightweight fine-tuning alternatives and prompting-based methods may achieve favorable accuracy-efficiency tradeoffs. Moving forward, we envision a more expansive view of adaptation that goes beyond just specializing foundation models to perform the task of interest: adaptation will alleviate deficiencies of stand-alone foundation models (e.g., temporal adaptation to reflect changes over time in the world) or introduce constraints (e.g., GDPR compliance relating to the right to be forgotten ; §4.7: security); this broader perspective on adaptation coincides with a need for new evaluation protocols (§4.4: evaluation) that systematically evaluate adaptation methods while controlling for resources (e.g., runtime, memory) and access requirements involved in adaptation.
§4.4: Evaluation. Evaluation offers context to foundation models by providing a means to track progress, understand models, and document their capabilities and biases. Foundation models challenge the ability of standard evaluation paradigms in machine learning to achieve these goals since they are one step removed from specific tasks. To envision new paradigms in evaluation that suit foundation models, we discuss (a) evaluating foundation models directly to measure their inherent capabilities and inform how foundation models are trained, (b) evaluating task-specific models by controlling for adaptation resources and access , and (c) broader evaluation design to provide richer context beyond measures of accuracy (e.g., robustness (§4.8: robustness), fairness (§5.1: fairness), efficiency (§4.5: systems), environmental impact (§5.3: environment)). Reform of evaluation practices will allow for evaluation that adequately serves both the diverse goals and stakeholders involved in the foundation model paradigm.
§4.5: Systems. While the training data (§4.6: data) determines the theoretical information available for foundation models, and model architectures (§4.1: modeling) and training objectives (§4.2: training) determine how much of this information can be extracted, computer systems determine what is practically achievable. Systems are a key bottleneck for scaling in terms of data and model size, both of which appear to reliably track with improvements in capabilities. To ensure that we can train the next generation of foundation models efficiently (with respect to time and cost), we will require the co-design of algorithms, models, software, and hardware. This co-design is already starting to happen to in various forms, from carefully tuned parallelism strategies to new architectures such as retrieval-based and mixture-of-expert models. Beyond training, we consider what will be required to deploy applications on top of foundation models (e.g., efficient inference).
§4.6: Data. Data is the lifeblood of foundation models; the training data of these models largely determines what these capabilities these models can acquire. The centrality of data is not unique to foundation models; recent calls for data-centric AI [Press 2021; Ré 2021] indicate the pervasive importance of managing, understanding, and documenting data used to train machine learning models. For foundation models specifically, the current modus operandi is for training data to be selected using unspecified or unclear principles with a general lack of transparency regarding the nature of training data. We believe an alternative approach is needed to re-imagine the data ecosystem surrounding foundation models: we draw upon work on data visualization and management to propose a data hub for foundation models. We articulate how this proposal relates to many of the relevant data-centric considerations for foundation models: selection, curation, documentation, access, visualization and inspection, quality assessment, and legal regulation.
§4.7: Security and privacy. Security and privacy for foundation models is largely uncharted at present. Fundamentally, foundation models are a high-leverage single point of failure , making them a prime target for attack: existing work demonstrates a variety of security vulnerabilities (e.g., adversarial triggers to generate undesirable outputs) or privacy risks (e.g., memorization of training data) for these models. Further, the generality of foundation models compounds these concerns, intensifying the risk for function creep or dual use (i.e., use for unintended purposes). For security, we view foundation models as akin to operating systems in traditional software systems; we discuss steps towards secure foundation models which, if achieved, would provide a strong abstraction layer to build upon for reliable ML applications. For privacy, by leveraging knowledge transfer from public data, foundation models may enable more sample efficient adaptation to sensitive data distributions, i.e., privacy-preserving applications may incur less degradation in accuracy when built using foundation models.
§4.8: Robustness to distribution shifts. A major limitation of standard machine learning is that it produces models that are not robust to distribution shifts , where the training distribution does not match the test distribution (for the downstream task). Existing work shows that adapting a foundation model trained on a broad range of unlabeled data improves the robustness of adapted models across a wide variety of shifts. This opens a new set of promising directions for improving training and adaptation of foundation models for robustness. However, we do not believe that foundation models are a panacea for robustness - challenges such as extrapolation across time and spurious correlations are not likely to be fully addressed.
§4.9: AI safety and alignment. Ensuring foundation models are reliable (§4.5: systems), robust (§4.8: robustness), and interpretable (§4.11: interpretability) is increasingly important when considering the potential real-world applications of these models. In addition to critical and immediate considerations, we also consider the relationship between foundation models and larger-scale risks, hazards, and harms that have the potential for increased relevance as model capabilities continue to advance. For example, we consider the importance of aligning foundation models such that they are not deployed with misspecified goals or values . We also discuss the relevance of forecasting the emergent behaviors of foundation models (e.g., the ability to deceive or plan strategically), which may complicate attempts to adapt them to particular tasks, and may require new approaches for interpretability (§4.11: interpretability) or evaluation (§4.4: evaluation).
§4.10: Theory. Learning theory provides a broad foundation for the variety of contexts encountered in applied machine learning; theory offers both understanding, principles, and guarantees to complement empirical findings. At present, the study of foundation models is largely empirical: the theory of standard supervised learning, while relatively mature, is inadequate to fully explain foundation models. Specifically, the discrepancy between the training phase and the adaptation
phase within the foundation model regime pinpoints the insufficiency of existing theory, since these phases correspond to (potentially) completely different tasks and data distributions. Nevertheless, we endeavor that advances in theory to address this discrepancy, even in simple, limited settings, will provide useful insights.
§4.11: Interpretability. Interpretability provides clarity to foundation models: the opacity of the deep neural networks that underpin foundation models, alongside the expected ubiquity of foundation models, heightens the need to understand these models and their capabilities. Interpretability methods at present generally are designed for interpreting and explaining the behavior of task-specific models; the nature of foundation models (i.e., the wide array of tasks these models are beneficial for and the unexpected emergent properties they acquire) introduces new challenges for interpretability research. To frame the discussion of interpretability for foundation models, we propose the one model-many models paradigm, which aims to determine the extent to which the one model (the foundation model) and its many models (its adapted derivatives) share decision-making building blocks. In addition to interpreting the decision-making components involved, we further discuss explainability in the context of foundation models (e.g., the validity of post hoc explanations generated by models) as well as the mechanisms that drive model behavior (which may clarify the extent to which understanding foundation models can extend to understanding their adapted derivatives). Given the critical role we ascribe interpretability in the study of foundation models, we conclude with an assessment of the societal impact of interpretability and non-interpretability.
## 1.4.4 Overview of society.
We believe the rapid development of foundation models, adapted and deployed to various applications, will have wide-ranging consequences on the health of societies. What makes these models so exciting and also so troubling is their task agnosticity. Societal impact is easier (but still non-trivial) to understand and reason about when we talk about specific systems deployed to users, but how can we take into account the societal impact of all possible systems and use cases when developing foundation models?
§5.1: Inequity and fairness. In many contexts, machine learning has been shown to contribute to, and potentially amplify, societal inequity. Foundation models may extend this trend, i.e., furthering the unjust treatment of people who have been historically discriminated against. However, understanding the relationship between inequity and foundation models requires reckoning with the abstraction of foundation models; foundation models are intermediary assets that are adapted for applications that impact users. Therefore, we delineate intrinsic biases , i.e., properties in foundation models that portend harm, and extrinsic harms , i.e., harms arising in the context of specific applications built using foundation models. We taxonomize various sources (e.g., training data, lack of diversity among foundation model developers, the broader sociotechnical context) that give rise to these biases and harms, emphasizing the importance, and technical difficulty, of source tracing to understand ethical and legal responsibility. We do not view unfairness as inevitable in the foundation model paradigm: to address unfair outcomes that arise from foundation models, we dually consider proactive interventions (e.g., technical methods like counterfactual data augmentation) and reactive recourse (e.g., mechanisms for feedback propagation and attribution of moral/legal responsibility).
§5.2: Misuse. We define foundation model misuse as the use of foundation models as they are technically intended (e.g., to generate language or video), but with the goal of causing societal harm (e.g., to generate disinformation, to develop deepfakes for harassment). We argue that advances in foundation models will result in higher-quality machine-generated content that will be easier to
create and personalize for misuse purposes. For example, disinformation actors may use them to quickly generate collections of articles targeted across different demographic groups (e.g., nationality, political party, religion, etc.). While these new capabilities may limit existing human detection methods for harmful content (e.g., tracking similar text across different sources), foundation models may themselves provide promising potential as automated misuse detectors.
§5.3: Environment. Foundation models are the byproducts of computationally expensive training regimes, with the existing trajectory favoring even more intensive models; the energy required for this training coincides with the release of more carbon into the atmosphere and the degradation of the environment. At present, current discussion centers these enormous single-time training costs and the potential to amortize these costs across repeated use. We seek to clarify these discussions by identifying assumptions that shape the calculus of environmental impact for foundation models. Further, we envision that the ecosystem surrounding foundation models requires a multi-faceted approach: (a) more compute-efficient models, hardware, and energy grids all may mitigate the carbon burden of these models, (b) environmental cost should be a clear factor that informs how foundation models are evaluated (§4.4: evaluation), such that foundation models can be more comprehensively juxtaposed with more environment-friendly baselines, and (c) the cost-benefit analysis surrounding environmental impact necessitates greater documentation and measurement across the community.
§5.4: Legality. Foundation models rest on tenuous legal footings at present; how the law bears on both the development and use of these models is largely unclear. Legal and regulatory frameworks for foundation models specifically, alongside those for AI technology more generally, will be needed to influence, constrain, and even foster practices in research, development, and deployment. Centering on the legal landscape of the United States, where existing consideration of algorithmic tools remains broadly uncertain, we highlight the pertinent issues of liability for model predictions and protections from model behavior. With respect to both issues, we describe how legal standards will need to be advanced to address these given the intermediary status of foundation models (as opposed to that of user-facing task-specific models).
§5.5: Economics. Foundation models are likely to have substantial economic impact due to their novel capabilities and potential applications in a wide variety of industries and occupations. We consider the implications of the development and use of foundation models for the future of the US and global economy with a focus on productivity, wage inequality, and concentration of ownership.
§5.6: Ethics of scale. In addition to running the risk of increasing inequity, as discussed in §5.1: fairness, the widespread adoption of foundation models poses other ethical, political and social concerns. We discuss ethical issues related to the scale of application of foundation models, such as homogenization and the concentration of power, as well as the norms and release strategies appropriate to address them.
## 2 CAPABILITIES
Foundation models acquire capabilities, some that surprisingly emerge from their learning process, that power downstream applications (§3: applications). Specifically, we discuss linguistic (§2.1: language) and visual (§2.2: vision) capabilities alongside the ability to affect the physical world (§2.3: robotics), perform reasoning and search (§2.4: reasoning), and interact with humans (§2.5: interaction). In addition, we discuss how self-supervision (the technical approach used to learn most current foundation models) philosophically relates to the ability to understand (§2.6: philosophy).
## 2.1 Language
Authors: Isabel Papadimitriou, Christopher D. Manning
## 2.1.1 The nature of human language.
Language is the basis of most human communication and interaction. However, it is not just a means for humans to achieve shared goals: language is central to human thought, to how social and emotional relations are formed, to how we identify ourselves socially and personally, and to how humans record knowledge and develop societal intelligence. Spoken or signed languages arise in every human society, and the languages of the world are both incredibly diverse in the ways that they express and structure the information they convey, while also exhibiting surprising concordance in the richness of what makes a language [Comrie 1989]. Languages are remarkably complex yet efficient systems, acquired consistently by children in a short amount of time, and which evolve and encompass the changing needs and conditions of linguistic communities. Due to this centrality of language in human activities, language understanding and generation is a critical element of research in artificial intelligence. Natural language processing (NLP) is the subfield of artificial intelligence concerned with language and, together with the related fields of automatic speech recognition (ASR) and text-to-speech (TTS), has the goal of giving computers the ability to understand and generate human language in much the same way human beings can.
To date in 2021, NLP has been the field most profoundly affected by foundation models. The first generation of foundation models showcased an impressive variety of linguistic abilities, as well as a surprising amount of adaptability to a large range of linguistic situations. Since the introduction of the early foundation models ELMo [Peters et al. 2018] and BERT [Devlin et al. 2019] in 2018, the field of NLP has become largely centered around using and understanding foundation models. The field has shifted to using foundation models as the primary tool, moving towards more generalized language learning as a central approach and goal. In this section, we go over the recent successes of foundation models in NLP, detail how foundation models have changed the overall process and mentality for training machine learning models for language, and discuss some of the theoretical and practical challenges facing foundation models as they are applied to a broader set of languages and more realistic and complex linguistic situations.
## 2.1.2 Impact of foundation models on NLP.
Foundation models have had a huge impact on the field of NLP, and are now central to most NLP systems and research. On a first level, many foundation models are skilled language generators: for example, Clark et al. [2021] demonstrate that non-experts have difficulty distinguishing short-form English text that was written by GPT-3 from that written by humans. However, the feature of foundation models that has been most impactful in NLP is not their raw generation abilities but their surprising generality and adaptability : a single foundation model can be adapted in different ways in order to achieve many linguistic tasks.
The field of NLP has historically focused on defining and engineering systems for challenging linguistic tasks, with the vision that models that are good at these tasks will lead to competent language systems for downstream applications. NLP tasks include classification tasks for a whole sentence or document (e.g., sentiment classification, like predicting whether a movie review is positive or negative), sequence labeling tasks, in which we classify each word or phrase in a sentence or document (e.g., predicting if each word is a verb or a noun, or which spans of words refer to a person or an organization), span relation classification , (e.g., relation extraction or parsing, like whether a person and location are linked by a 'current residence' relation, or a verb and a noun by a 'subject-verb' relation) and generation tasks , producing new text that is conditioned
Fig. 5. Only a tiny percentage of the world's languages are currently represented in foundation models. There are over 6,000 languages in the world, with estimates varying due to the inherent uncertainty of what constitutes a separate language [Nordhoff and Hammarström 2011]. This map shows the languages of the world, with each dot representing one language and its color indicating the top-level language family. Data is from Glottolog [Hammarström et al. 2021]. We label a few of the languages on the map as examples.
<details>
<summary>Image 5 Details</summary>
### Visual Description
\n
## Map: World Languages
### Overview
The image is a world map displaying the geographical distribution of various languages. Each language is represented by a colored dot or cluster of dots, and a label indicating the language name is placed near its primary area of usage. The map appears to be illustrative, showing approximate locations rather than precise linguistic boundaries.
### Components/Axes
The map itself serves as the primary component. There are no explicit axes, but the geographical coordinates (latitude and longitude) implicitly define the spatial arrangement. The languages are represented by different colored dots, and each language has a corresponding label.
### Detailed Analysis or Content Details
The following languages are identified on the map, along with their approximate locations:
* **English** (Pink): Primarily located in North America, the United Kingdom, Australia, and New Zealand.
* **Español** (Pink): Predominantly in Spain and South America.
* **Français** (Pink): Mainly in France, Canada, and parts of Africa.
* **Deutsch** (Yellow): Concentrated in Germany and surrounding areas.
* **Italiano** (Yellow): Primarily in Italy.
* **Русский** (Yellow): Predominantly in Russia and surrounding areas.
* **Türkçe** (Yellow): Mainly in Turkey.
* **العربية** (Yellow): Predominantly in the Middle East and North Africa. (Arabic)
* **中文** (Yellow): Primarily in China. (Chinese)
* **日本語** (Yellow): Primarily in Japan. (Japanese)
* **हिन्दी** (Yellow): Primarily in India. (Hindi)
* **Português** (Green): Predominantly in Portugal, Brazil, and parts of Africa.
* **Sranantongo** (Green): Located in Suriname.
* **Nahuatl** (Green): Located in Mexico.
* **Quechua** (Green): Located in the Andes region of South America.
* **Aymar aru** (Green): Located in the Andes region of South America.
* **Edé Yorùbá** (Green): Located in West Africa.
* **Oromo** (Green): Located in Ethiopia and surrounding areas.
* **Gikuyu** (Green): Located in Kenya.
* **isiZulu** (Green): Located in South Africa.
* **Jawa** (Purple): Located in Indonesia.
* **Yolngu** (Purple): Located in Australia.
* **Māori** (Purple): Located in New Zealand.
* **Reo Tahiti** (Purple): Located in French Polynesia.
* **Kiribati** (Purple): Located in Kiribati.
* **Olelo Hawai'i** (Purple): Located in Hawaii.
* **Wichi** (Pink): Located in Argentina.
* **Avafe'e** (Pink): Located in Argentina.
* **Argentine Sign Language** (Pink): Located in Argentina.
* **Yagan** (Pink): Located in Argentina.
* **ixthosa** (Green): Located in South Africa.
* **Ciw** (Pink): Located in the Philippines.
* **Kalaallisut** (Yellow): Located in Greenland.
* **Islenska** (Yellow): Located in Iceland.
* **Scots** (Yellow): Located in Scotland.
* **Vepsän kel'** (Yellow): Located in Russia.
* **Hwỳtßblu** (Yellow): Located in Russia.
* **עברית** (Yellow): Located in Israel. (Hebrew)
* **ไทย** (Yellow): Located in Thailand. (Thai)
* **ဗမာ** (Yellow): Located in Myanmar. (Burmese)
* **বাংলা** (Yellow): Located in Bangladesh. (Bengali)
* **অসমীয়া** (Yellow): Located in India. (Assamese)
* **Yupik** (Green): Located in Alaska.
The colors used are: Pink, Yellow, Green, and Purple. The distribution of languages appears to be clustered geographically, with certain regions dominated by specific language families.
### Key Observations
The map highlights the global diversity of languages. The concentration of languages in certain regions (e.g., Africa, South America, and Asia) suggests a rich linguistic history and cultural diversity in those areas. The map also shows the spread of major languages like English, Spanish, and French across continents, likely due to historical colonization and globalization.
### Interpretation
The map demonstrates the complex relationship between language, geography, and culture. The distribution of languages reflects historical migration patterns, colonial influences, and the natural barriers (mountains, deserts, oceans) that have shaped linguistic boundaries. The use of color-coding allows for a quick visual comparison of the prevalence of different language families across the world. The map is a simplified representation of a highly complex reality, as linguistic boundaries are often fluid and overlapping. The map suggests that while globalization is leading to the dominance of certain languages, a significant amount of linguistic diversity still exists, particularly in less-developed regions. The inclusion of sign languages (Argentine Sign Language) is a notable detail, acknowledging the importance of non-spoken forms of communication. The map serves as a visual reminder of the richness and fragility of human linguistic heritage.
</details>
strongly on an input (e.g., producing a translation or summary of a text, recognizing or producing speech, or responding in a conversation) [Jurafsky and Martin 2009]. In the past, NLP tasks had distinct research communities that developed task-specific architectures, often based on pipelines of different models, each performing a linguistic sub-task such as token segmentation, syntactic parsing, or coreference resolution.
By contrast, the dominant modern approach for performing each task is to use a single foundation model and adapt it slightly using relatively small amounts of annotated data specific to each task (sentiment classification, named entity tagging, translation, summarization) to create an adapted model. This has proved to be an extremely successful approach: for the vast majority of the tasks described above, a foundation model that is slightly adapted for a task greatly outperforms previous models or pipelines of models that were built specifically to perform that one task. To take just one example, the best system for answering open-ended science questions in 2018, before foundation models, could get 73.1% on the NY Regents 8th grade science exam. A year later in 2019, an adapted foundation model scored 91.6% [Clark et al. 2019].
The emergence of foundation models that are largely trained to generate language has constituted an important shift in the role of language generation in NLP. Until around 2018, the problem of generating general-purpose language was considered very difficult and essentially unapproachable except through other linguistic sub-tasks [Paris et al. 2013]. Instead, NLP research was mostly focused on linguistically analyzing and understanding text. Now, it is possible to train highly coherent foundation models with a simple language generation objective, like 'predict the next word in this sentence'. These generative models now constitute the primary vehicle through which machine learning for language is done - including the analysis and understanding tasks that were once considered prerequisites for generation. The successful generation exhibited by foundation
models has also led to a flowering of research for language generation tasks like summarization and dialogue generation. The rise of the foundation model paradigm has begun to play a similar role in spoken language as well as written. Modern automatic speech recognition (ASR) models like wav2vec 2.0 are trained on large datasets of speech audio alone, and then adapted on audio with associated transcriptions for the task of ASR [Baevski et al. 2020].
Due to the changes brought about by the foundation model paradigm, the focus of research and practice in NLP has shifted from making bespoke architectures for different tasks to exploring how to best leverage foundation models. Research into adaptation methods has blossomed (see §4.3: adaptation for a detailed look at adaptation), and the surprising successes of foundation models have also caused a shift in research interest towards analyzing and understanding foundation models (see §4.11: interpretability for interpretability and analysis of foundation models).
## 2.1.3 Language variation and multilinguality.
Though foundation models are surprisingly versatile with the linguistic knowledge they obtain from pretraining, there are limits to this adaptability: it is not clear how successful current foundation models are at handling language variation. Language varies greatly. Apart from the fact that there are thousands of different languages in the world, language varies even within one language or within one speaker. To point out a few examples, informal conversation manifests differently from written language, the grammatical constructions that people reach for when speaking to friends are very different from those used when speaking to someone with authority, and communities of speakers within a language use different dialects. Social and political factors are embedded in how language variation is viewed and valued, and in how much different varieties are represented in NLP research (see for example Blodgett and O'Connor [2017] on the failures of NLP for African American English, and §5.1: fairness for a deeper discussion on inequities in foundation models). Due to their large capacity for learning linguistic information and flexibly adapting that knowledge, foundation models hold promise for expanding NLP to encompass more linguistic diversity. It remains an open research question to understand whether it is possible to make foundation models that robustly and equitably represent language with both its major and subtle variations, giving equal weight and acuity to what makes each linguistic variety distinct [research posing and addressing this question includes Ponti et al. 2019; Bender 2011; Joshi et al. 2020].
Following the success of foundation models for English, multilingual foundation models have been released to extend that success to non-English languages. For most of the over 6,000 languages in the world, the text data available is not enough to train a large-scale foundation model. To give one example, there are over 65 million speakers of Fula, a West African language, but few if any resources available for NLP in Fula [Nguer et al. 2020]. Multilingual foundation models address this by jointly training on multiple languages simultaneously. The multilingual foundation models to date (mBERT, mT5, XLM-R) are each trained on around 100 languages [Devlin et al. 2019; Goyal et al. 2021; Xue et al. 2020]. Joint multilingual training relies on the reasonable assumption that the shared structures and patterns between languages can lead to sharing and transfer from the high-resource languages to the low-resource ones, making foundation models possible for languages where we could not train a stand-alone model. Experiments using and analyzing multilingual foundation models have shown that there is indeed a surprising amount of transfer between and parallel encoding of the different languages in multilingual foundation models [Wu and Dredze 2019; Choenni and Shutova 2020; Pires et al. 2019; Libovick` y et al. 2019; Chi et al. 2020; Papadimitriou et al. 2021; Cao et al. 2019].
However, the extent to which these models are robustly multilingual is still an open question. It remains unclear how much models trained on this data can represent aspects of languages that are drastically different from English or for which few language resources are available [Wu and Dredze
2020], and whether their apparent multilingual performance relies more on assimilation [Lauscher et al. 2020; Virtanen et al. 2019; Artetxe et al. 2020]. Multilingual models show better performance in languages that are similar to the highest-resource languages in their training data, and it has been shown that languages in multilingual models compete for model parameters, making it unclear how much variation can fit in a single model [Wang et al. 2020d]. A salient issue stems from the data that we use to train multilingual foundation models: in many multilingual corpora, English data is not only orders of magnitude more abundant than that of lower-resource languages, but it is often cleaner, broader, and contains examples showcasing more linguistic depth and complexity [Caswell et al. 2021] (see Nekoto et al. [2020] on building participatory and robust multilingual datasets). However, the answer does not simply lie in creating more balanced corpora: there are so many axes of language variation that it would be infeasible to create a corpus that is balanced and representative in all regards. The future, versatility, and equity of foundation models all depend on robustly handling language variation despite unbalanced data [e.g., Oren et al. 2019].
Current multilingual foundation models in their raw form, and naive unsupervised multilingual training as a method, may not model the subtleties of languages and language varieties to their full extent. Nevertheless, they remain useful for some multilingual applications, for example through adapting multilingual models for low-resource languages not in their original training set [Wang et al. 2020b]. Moreover, the results for the (non-public) GShard neural machine translation model show the largest gains over monolingual baselines for the lowest resource languages, with the gains increasing with model size [Lepikhin et al. 2021]. The research community should critically examine how foundation models deal with language variation, understand the limits of foundation models in bringing equity and representation to NLP, and not settle on promoting foundation models that erase language variation and mostly conform to the linguistic majority in their training data.
## 2.1.4 Inspiration from human language acquisition.
Though foundation models have constituted a huge source of progress in creating NLP systems that act more like humans, there are still significant ways in which the linguistic system that they acquire, as well as the learning process, differ from human language. Understanding the implications of this gap between machine and human language learning is a necessary part of developing a research community informed about the linguistic limits and possibilities of foundation models.
Humanlanguage acquisition is very efficient: foundation models like GPT-3 are trained on around three to four orders of magnitude more language data than most humans will ever hear or read, and certainly much more than children have been exposed to by the time they are mostly linguistically competent. One salient difference between foundation models and human language acquisition is that human language is grounded to the real world [Saxton 2017]. For example babies and caretakers point to objects during language development [Colonnesi et al. 2010], and babies learn the grounded meanings of words that refer to common objects before they learn a lot of the other aspects of the linguistic system [Bergelson and Swingley 2012]. Most foundation models used in NLP, on the other hand, learn from the distributional information of raw, ungrounded text, and (in contrast to human learners) Zhang et al. [2021] show that RoBERTa models express abstract syntactic features before usable meaning. Powerful ungrounded statistical learning is indeed also present in babies [Saffran et al. 1996], so it is no doubt an important factor in acquisition. Nevertheless, advancing grounded language learning for foundation models remains an important direction for approaching human acquisition efficiency [Dupoux 2018; Tan and Bansal 2020; Zellers et al. 2021a, inter alia ] (see §2.2: vision and §2.3: robotics for the multimodal potential of foundation models, and §2.6: philosophy for a discussion of whether foundation models can understand language without grounding). Another important direction is examining the inductive biases in foundation
Fig. 6. Language Acquisition for humans and foundation models. While there are certainly different inductive biases between the human brain and foundation models, the ways that they learn language are also very different. Most saliently, humans interact with a physical and social world in which they have varied needs and desires, while foundation models mostly observe and model data produced by others.
<details>
<summary>Image 6 Details</summary>
### Visual Description
\n
## Diagram: Language Acquisition
### Overview
The image is a diagram illustrating the components and relationships involved in Language Acquisition. It depicts a central "Human" element surrounded by various factors contributing to language development, with two larger encompassing areas labeled "Foundation Model" and "Vision". The diagram uses icons and text labels to represent these components, and overlapping circles to suggest relationships.
### Components/Axes
The diagram consists of the following components:
* **Central Element:** "Human" (represented by an image of a baby)
* **Surrounding Factors:**
* "Language" (icon: open book)
* "Social Knowledge" (icon: head with gears)
* "Common Sense" (icon: lightbulb)
* "Motivation & Curiosity" (icon: teddy bear)
* "Real World Objects" (icon: building blocks)
* "Communication & Interaction" (icon: tools)
* "Child-directed Questions" (icon: ABC blocks)
* "Prosody & Speech" (icon: exclamation point)
* **Encompassing Areas:**
* "Foundation Model" (light blue circle)
* "Vision" (pink circle)
* **Textual Annotation:** "Language x 3-4 orders of magnitude more than a human"
### Detailed Analysis / Content Details
The diagram is organized around the central "Human" element. The surrounding factors are positioned around the baby, suggesting their influence on language acquisition. The "Foundation Model" and "Vision" areas overlap, indicating a relationship between them.
* **Language:** Located at the top-left, associated with an open book icon.
* **Social Knowledge:** Positioned above the "Human", represented by a head with gears.
* **Common Sense:** Situated to the right of "Social Knowledge", depicted as a lightbulb.
* **Motivation & Curiosity:** Located to the right of the "Human", shown as a teddy bear.
* **Real World Objects:** Positioned below "Motivation & Curiosity", represented by building blocks.
* **Communication & Interaction:** Found below "Real World Objects", depicted as tools.
* **Child-directed Questions:** Located at the bottom-left, shown as ABC blocks.
* **Prosody & Speech:** Positioned to the left of "Child-directed Questions", represented by an exclamation point.
* **Foundation Model:** A large light blue circle encompassing the right side of the diagram, containing a complex geometric shape.
* **Vision:** A large pink circle encompassing the right side of the diagram, containing an image of glasses and books.
* **Quantitative Statement:** "Language x 3-4 orders of magnitude more than a human" is placed within the "Foundation Model" area. This suggests the scale of language data used in the model is significantly larger than that contained within a human.
### Key Observations
The diagram emphasizes the multifaceted nature of language acquisition, highlighting the interplay between innate human capabilities and external factors. The "Foundation Model" and "Vision" areas suggest the role of artificial intelligence and perceptual processing in this process. The quantitative statement indicates a substantial difference in scale between human language capacity and the language data used in the foundation model.
### Interpretation
The diagram illustrates a conceptual model of language acquisition, likely within the context of artificial intelligence or machine learning. It suggests that building a "Foundation Model" capable of language understanding requires not only vast amounts of language data (3-4 orders of magnitude more than a human possesses) but also incorporates elements of vision, common sense, social knowledge, and an understanding of how humans learn. The overlapping circles of "Foundation Model" and "Vision" suggest that these two components are deeply intertwined. The diagram implies that successful language acquisition in AI requires a holistic approach that considers both the linguistic data itself and the broader context of human cognition and perception. The placement of the "Human" at the center emphasizes that the ultimate goal is to replicate or understand human language abilities.
</details>
models and how they relate to the inductive biases in the human mind, both those specific to language learning and those general to human cognition [Linzen and Baroni 2021]. Though the human brain may be more architecturally specialized for efficient language acquisition, foundation models are not blank-slate learners [Baroni 2021], and understanding and aligning these linguistic inductive biases is an important future direction for research in foundation models.
A significant factor in the efficiency of language acquisition is the fact that humans acquire a systematic and generalizable language system. Though there are many differing theories about what types of theoretical abstractions the human language system makes [e.g., Comrie 1989; Chomsky 2014; Croft 2001; Jackendoff 2011], it is generally agreed that humans learn language in a way that allows them to easily slot new knowledge into existing abstractions and productively create new grammatical sentences. For example, a ten-year-old child has acquired a lot of the abstractions about how their language works, though the actual words and constructions that they produce will change drastically over the next ten years. Foundation models, on the other hand, often do not acquire the systematic abstractions that we expect from humans. For example, when a foundation model produces a linguistic construction accurately one time there is no guarantee that future uses of that construction will be mostly consistent, especially after a significant domain shift in the subject matter [examples of work examining limitations of foundation models in systematicity include Lake and Baroni 2018; Kim and Linzen 2020; Bahdanau et al. 2018; Chaabouni et al. 2021]. NLP faces the challenge of developing some sort of systematicity in acquisition for foundation models, without regressing to systems that rely too heavily on rigid linguistic rules.
Language learning continues for a speaker's whole lifetime: the grammar of human languages evolves, and humans flexibly adapt to novel linguistic situations [Sankoff 2018]. For example, as new terms and concepts arise in an adult's life, they can use them relatively easily in grammatical
sentences, and humans often adapt their grammatical patterns to fit in with different social groups [Rickford et al. 1994]. On the other hand, the linguistic system of foundation models is mostly set by the training data, and is relatively static [Lazaridou et al. 2021; Khandelwal et al. 2020]. Though adaptation methods can prime foundation models for different tasks (see §4.3: adaptation), it still remains unclear how to change the more basic linguistic foundation of a foundation model without a large amount of training. Making adaptable models that naturally mirror human-like linguistic accommodation and language evolution is an important research area for the future of foundation models.
Foundation models have drastically changed the research and practice of NLP. Foundation models have given rise to many new research directions for the community: understanding generation as a fundamental aspect of language, studying how to best use and understand foundation models, understanding the ways in which foundation models may increase inequities in NLP, examining whether foundation models can satisfactorily encompass linguistic variation and diversity, and finding ways to draw on human language learning dynamics. Most of the complex NLP tasks that the research community focused on before foundation models are now best handled, to an almost-human level, using one of a few publicly released foundation models. Nevertheless, there remain significant gaps between this performance and the needs for useful and safe deployment of foundation models in complex downstream settings.
## 2.2 Vision
Authors: Shyamal Buch, Drew A. Hudson, Frieda Rong, Alex Tamkin, Xikun Zhang, Bohan Wu, Ehsan Adeli, Stefano Ermon, Ranjay Krishna, Juan Carlos Niebles, Jiajun Wu, Li Fei-Fei
Fig. 7. By harnessing self-supervision at scale, foundation models for vision have the potential to distill raw, multimodal sensory information into visual knowledge, which may effectively support traditional perception tasks and possibly enable new progress on challenging higher-order skills like temporal and commonsense reasoning (§2.2.1: vision-capabilities). These inputs can come from a diverse range of data sources and application domains, suggesting promise for applications in healthcare and embodied, interactive perception settings (§2.2.2: vision-challenges). Image credits [Zamir et al. 2018; Haque et al. 2020].
<details>
<summary>Image 7 Details</summary>
### Visual Description
\n
## Diagram: Foundation Model Data Flow
### Overview
This diagram illustrates the data sources and skills associated with a Foundation Model. It depicts a flow of information from various data sources (Perceptual Sources and Data Types) into the Foundation Model, and then the resulting skills that the model develops (Traditional Vision Tasks and Higher-Order Skills). The diagram is divided into three main sections: "Data Sources" (left), "Foundation Model" (center), and "Skills" (right).
### Components/Axes
The diagram consists of the following components:
* **Data Sources:** Divided into "Perceptual Sources" and "Data Types".
* **Perceptual Sources:** Includes icons representing Cameras & Devices, Autonomous Agents, and Ambient Sensors.
* **Data Types:** Includes images representing RGB, Depth, Thermal, Text, Radio, and Audio.
* **Foundation Model:** A central, stylized representation of a neural network.
* **Skills:** Divided into "Traditional Vision Tasks" and "Higher-Order Skills".
* **Traditional Vision Tasks:** A list of computer vision tasks.
* **Higher-Order Skills:** A list of more complex cognitive skills.
* **Arrows:** Indicate the flow of information from Data Sources to the Foundation Model ("Training") and from the Foundation Model to Skills ("Adaptation").
### Detailed Analysis or Content Details
**Data Sources - Perceptual Sources:**
* **Cameras & Devices:** Represented by a smartphone, a camera, and a tablet.
* **Autonomous Agents:** Represented by a robotic agent.
* **Ambient Sensors:** Represented by a smart home device (e.g., a smart speaker or thermostat).
**Data Sources - Data Types:**
* **RGB:** A standard color image.
* **Depth:** A depth map, showing distance from the sensor.
* **Thermal:** A thermal image, showing heat signatures.
* **Text:** Represented by the "Aal" label.
* **Radio:** A radio wave visualization.
* **Audio:** An audio waveform.
**Foundation Model:**
* The model is depicted as a complex network of interconnected nodes.
* An arrow labeled "Training" points from the Data Sources towards the Foundation Model.
* An arrow labeled "Adaptation" points from the Foundation Model towards the Skills.
**Skills - Traditional Vision Tasks:**
* Image Recognition
* Object Detection
* Segmentation
* Edge Detection
* Keypoints Detection
* Surface Normals
* Re-shading
* Curvature
* Uncertainty
* Depth
**Skills - Higher-Order Skills:**
* Physics & Dynamics
* Theory of Mind
* Commonsense Reasoning
* Temporality & Causality
### Key Observations
* The diagram emphasizes the multi-modal nature of the data used to train the Foundation Model, encompassing visual, textual, and auditory information.
* The distinction between "Training" and "Adaptation" suggests a two-stage process: initial learning from data sources, followed by refinement and application to specific skills.
* The skills are categorized into "Traditional Vision Tasks" and "Higher-Order Skills," indicating a progression from basic perception to more complex cognitive abilities.
### Interpretation
The diagram illustrates the concept of a Foundation Model as a central component capable of learning from diverse data sources and developing a wide range of skills. The model's ability to integrate information from different modalities (RGB, Depth, Text, Audio, etc.) is crucial for achieving both traditional computer vision tasks and more advanced cognitive abilities like physics reasoning and theory of mind. The "Training" and "Adaptation" arrows suggest that the model is not simply programmed with these skills, but rather learns them through exposure to data and subsequent refinement. The diagram highlights the potential of Foundation Models to bridge the gap between perception and cognition, enabling machines to understand and interact with the world in a more human-like way. The inclusion of "Uncertainty" as a traditional vision task suggests an awareness of the limitations of perception and the importance of quantifying confidence in predictions. The "Aal" label under text is unclear and may be a placeholder or an abbreviation.
</details>
Vision underlies one of the primary modes through which a living organism understands its environment. The ability to see enables the near-constant, long-range gathering of dense signals, a critical capability developed over an evolutionary time-scale in a diverse range of life forms [Parker 2003; Zhang and Shu 2021]. For a skill executed effortlessly by even simple living creatures, transferring the same abilities to machines has proved remarkably challenging, leading computer vision and robotics researcher Hans Moravec in 1988 to observe a paradox: in AI, (what were considered) hard problems are easy and likewise easy problems are hard, and among the 'easiest' problems of them all is the visual acuity which we use each day to continually interpret complex scenes in a matter of milliseconds [Moravec 1988; Thorpe et al. 1996; Fei-Fei et al. 2007].
On the other end of this formidable challenge is the substantial scope of transformative applications which computer vision holds the key to: self-driving cars that can free commuters from gridlock (§2.3: robotics), life-saving AI tools that can assist overworked specialists by detecting rare medical events (§3.1: healthcare), next-generation tools for multimedia creation and editing (§2.5: interaction), among others. Reflecting on the applications and settings where human perception is instrumental offers a sense of the potential areas where computer vision can assist and transform.
The field of computer vision and the challenges we define draw inspiration in many ways from human perception capabilities. Several classical theories [e.g., Biederman 1972; McClelland and Rumelhart 1981; Marr 1982] suggested that humans may perceive real world scenes by contextualizing parts as a larger whole, and pointed the way for computer vision techniques to progressively model the physical world with growing levels of abstractions [Lowe 1992; Girshick et al. 2014]. Gibson [1979] suggested that human vision is inherently embodied and interactive ecological environments may play a key role in its development. These ideas continue to motivate the ongoing development of computer vision systems, iterating towards a contextual, interactive, and embodied perception of the world.
In the context of computer vision, foundation models translate raw perceptual information from diverse sources and sensors into visual knowledge that may be adapted to a multitude of downstream settings (Figure 7). To a large extent, this effort is a natural evolution of the key ideas that have emerged from the field over the last decade. The introduction of ImageNet [Deng et al. 2009] and the advent of supervised pretraining led to a deep learning paradigm shift in computer vision. This transition marked a new era, where we moved beyond the classic approaches and task-specific feature engineering of earlier days [Lowe 2004; Bay et al. 2006; Rosten and Drummond 2006] towards models that could be trained once over large amounts of data, and then adapted for a broad variety of tasks, such as image recognition, object detection, and image segmentation [Krizhevsky et al. 2012; Szegedy et al. 2015; He et al. 2016a; Simonyan and Zisserman 2015]. This idea remains at the core of foundation models.
The bridge to foundation models comes from the limitations of the previous paradigm. Traditional supervised techniques rely on expensive and carefully-collected labels and annotations, limiting their robustness, generalization and applicability; in contrast, recent advances in self-supervised learning [Chen et al. 2020c; He et al. 2020] suggest an alternative route for the development of foundation models that could make use of large quantities of raw data to attain a contextual understanding of the visual world. Relative to the broader aims of the field, the current capabilities of vision foundation models are currently early-stage (§2.2.1: vision-capabilities): we have observed improvements in traditional computer vision tasks (particularly with respect to generalization capability) [Radford et al. 2021; Ramesh et al. 2021] and anticipate that the near-term progress will continue this trend. However, in the longer-term, the potential for foundation models to reduce dependence on explicit annotations may lead to progress on essential cognitive skills (e.g., commonsense reasoning) which have proven difficult in the current, fully-supervised paradigm [Zellers et al. 2019a; Martin-Martin et al. 2021]. In turn, we discuss the potential implications of foundation models for downstream applications, and the central challenges and frontiers that must be addressed moving forward (§2.2.2: vision-challenges).
## 2.2.1 Key capabilities and approaches.
At a high-level, computer vision is the core sub-field of artificial intelligence that explores ways to endow machines with the capacity to interpret and understand the visual world. It encompasses a multitude of tasks, sub-domains and downstream applications, where the community has made continual progress over the last several decades [Zamir et al. 2018]. A selection of example tasks 16 : (1) semantic understanding tasks, which aim to discover the properties and relations among entities within visual scenes; these include image classification, object detection, semantic segmentation, action recognition, and scene graph generation, among others [e.g., Krizhevsky et al. 2012; He et al. 2016a; Krishna et al. 2017; Russakovsky et al. 2015; Krizhevsky et al. 2009; Kay et al. 2017; Lin et al. 2014]. (2) geometric, motion and 3D tasks, seeking to represent the geometry, pose and structure
16 This, of course, is a coarse selection: please see the categories at the annual conference on Computer Vision and Pattern Recognition (CVPR) for a more complete (but evolving) picture of the tasks in the field.
of still or moving objects, and include tasks of depth estimation, structure-from-motion, surface normal detection, curvature line and keypoint estimation, to name a few [e.g., Laina et al. 2016; Agarwal et al. 2011; Wang et al. 2015a; Zamir et al. 2018; Ullman 1979]. (3) multimodal integration tasks, combining semantic and geometric understanding with other modalities such as natural language; these include, for instance, visual question answering, image captioning, and instruction following [e.g., Antol et al. 2015; Chen et al. 2015b; Anderson et al. 2018; Goyal et al. 2017b; Hudson and Manning 2019b; Johnson et al. 2017; Luo et al. 2020; Akbari et al. 2021; Huang et al. 2021c; Tsimpoukelli et al. 2021]. We highlight a subset of traditional core tasks in Figure 7.
The predominant paradigm for addressing these tasks, driven by the emergence of ImageNet [Deng et al. 2009] during the early 2010s, tends to center around a familiar core idea: First, pretrain a model on a large collection of carefully annotated data [Russakovsky et al. 2015] with a fully supervised training task, like image classification. Then, adapt the model downstream on taskspecific datasets and domains [Lin et al. 2014; Chen et al. 2015b; Antol et al. 2015] by fine-tuning to reach state-of-the-art performance [Krizhevsky et al. 2012; Simonyan and Zisserman 2015; He et al. 2016a; Xu and Saenko 2016]. This notion of pretraining followed by adaptation persists in the definitions we consider now for foundation models (§1: introduction). The limitations of this fully supervised paradigm motivate the transition to foundation models: the reliance on external supervised annotations constrains the upper bound capability of previous approaches to capture the diverse spectrum of visual inputs in a scalable, robust and generalizable manner. Recent developments in the domain of visual synthesis and unsupervised learning offer a compelling alternative. GANs, for instance, learn to generate visual content of high fidelity, realism and diversity, by featuring two competing networks of a generator and a discriminator that can supervise one another from image collections alone [e.g., Goodfellow et al. 2014; Hudson and Zitnick 2021]. Other neural models infer the visual properties of objects and scenes without explicitly annotated supervision, by employing variational auto-encoding, contrastive learning or other self-supervised techniques [e.g., Kingma and Welling 2014; Chen et al. 2020c; He et al. 2020]. For instance, He et al. [2021] build upon prior work on representation learning with masked image encoding [e.g., Pathak et al. 2016; Vincent et al. 2008] by, in part, combining recent advancements in flexible architectures (e.g., vision transformers [Dosovitskiy et al. 2021; Zhai et al. 2021]) with increased scaling.
With foundation models, the development of such self-supervision techniques has enabled training at greater scales of visual data [Changpinyo et al. 2021], both in terms of its scope as well as its potential diversity. Accordingly, we have seen early indicators of progress on traditional vision tasks in terms of both standard accuracy metrics and few-shot generalization. For image classification and object detection, self-supervised techniques have reported competitive performance to prior fully-supervised approaches [He et al. 2019; Chen et al. 2020c; Radford et al. 2021; Hénaff et al. 2021], without explicit annotations during training and greater sample efficiency during adaptation. For visual synthesis, notable examples include DALL-E [Ramesh et al. 2021] and CLIP-guided generation [Radford et al. 2021; Galatolo et al. 2021], where researchers leverage multimodal language and vision input to render compelling visual scenes. In the short-term, we anticipate that the capabilities of these foundation models will continue to improve along these directions, as training objectives are refined [Chen et al. 2020a; Hénaff et al. 2021; Selvaraju et al. 2021] and architectures are designed to incorporate additional modalities [Jaegle et al. 2021b].
Notably, current foundation models for computer vision are nascent relative to their NLP counterparts (§2.1: language): promising early efforts are still largely centered on RGB image inputs and a subset of core traditional vision tasks. However, the field continues to progress on broader challenges centered on embodied and interactive perception settings (critical for foundation models for robotics [Bohg et al. 2017, §2.3: robotics]). We note a subset of these higher-order goals in Figure 7, including physical scene understanding, reasoning over visual commonsense and temporal
events, and perception for social affordances. Each of these have been goals for fully-supervised systems, but have proven challenging in part due to the difficulty of annotating these tasks at scale. For instance, standard systems for visual-question answering struggle to answer questions that require commonsense understanding, since these questions often require external knowledge beyond what is present in the pixels alone [Zellers et al. 2019a]. Perceiving human gaze and social affordances in a robust manner remain ongoing challenges for embodied vision systems in interactive agents [Martin-Martin et al. 2021]. By reducing the dependence on explicit annotations, foundation models may enable further progress towards these goals than was previously feasible. Related progress in language foundation models (§2.1: language), which have been able to capture a degree of commonsense over language events [Brown et al. 2020], also suggests a potential avenue towards achieving similar capability over multimodal visual inputs. While the exact roadmap for how to achieve these capabilities in foundation models remains an open problem, a combination of new efficient and flexible architectures (§4.1: modeling), large-scale training (§4.5: systems), self-supervision techniques (§4.2: training) and few-shot adaptation schemes (§4.3: adaptation) may open the door towards capabilities that have been difficult to reach so far.
## 2.2.2 Central research challenges.
Our discussion of research challenges is motivated by the downstream application domains where foundation models may further the integration and impact of vision models. We highlight a few such areas: (1) ambient intelligence for healthcare and home environments: building upon existing approaches for ambient intelligence in these settings [Haque et al. 2017; Lyytinen and Yoo 2002; Hong and Landay 2004], foundation models may offer the potential for better detection of finegrained human activities and medical events, as well as improved assistive interaction for clinicians, patients, and everyday consumers (see also §3.1: healthcare). (2) mobile and consumer applications : foundation models with stronger multimodal grounding may enable more capable interactivity of services in mobile settings, and fundamental improvements in generation capability from vision and language inputs can benefit computational photography and content editing applications [Delbracio et al. 2021; Ramesh et al. 2021; Park et al. 2019] (see also §2.5: interaction). (3) embodied, interactive agents : perception models have already proven effective as both inputs [Sermanet et al. 2018] and reward functions [Chen et al. 2021c; Shao et al. 2020] in robotics settings; foundation models trained on large collections of egocentric (real/simulated, human/robotic) visual data [Damen et al. 2018; Chen et al. 2021e] may potentially further this progress by capturing a wider distribution of visual scenes, objects, and actions (see also §2.3: robotics).
The extent to which foundation models may further impact these application settings hinges on the degree to which the capabilities outlined in §2.2.1: vision-capabilities are realized. To bridge the significant gaps between present, short-term, and long-term anticipated capabilities, we must address current limitations of foundation models for vision, including their training and evaluation. Below, a subset of corresponding key challenges:
Semantic systematicity and perceptual robustness. Humans have a remarkable capacity for generalizing visual understanding to unseen compositions, and reasoning about the physical and geometric properties of novel objects and scenes [Lake et al. 2015]. While current foundation models have shown promising capability for image synthesis and early results for generalization to fine-grained language inputs, these models still struggle to generalize to compositions of simple shapes and colors [Ramesh et al. 2021; Radford et al. 2021; Rong 2021]. Generalizability goes beyond semantics as well; visual scenes and objects have a natural regularity to their physical dynamics and geometric properties. Foundation models have shown early indications of understanding scene and object geometry [Ramesh et al. 2021]. Further, early efforts towards physical scene and
geometric understanding in perception models may provide guidance for ongoing foundation model development [Yi et al. 2019; Bakhtin et al. 2019; Li et al. 2020b]. Indeed, the continued incorporation of multiple modalities (e.g., audio) in foundation models may prove beneficial towards these aims [Zhang et al. 2017; Gao et al. 2020b; Jaegle et al. 2021a]. However, the specific techniques to enable generalizing the initial observed capabilities robustly to a wide range of natural scenes and objects at the level of humans remains an open research challenge for foundation models.
Computational efficiency and dynamics modeling. Humans are surprisingly efficient at processing the continuous visual stream of objects, scenes, and events necessary to support an understanding of event dynamics [Zacks et al. 2001; Tversky and Zacks 2013]. Foundation models in language (§2.1: language) have shown initial steps towards modeling longer-term coherence of events; the analogous ability to capture long-range temporal correlations and causal coherence in visual input would stand to benefit downstream settings like robotics [Dai et al. 2019; Alyamkin et al. 2019; Goel et al. 2020b; Feng et al. 2019, §2.3: robotics]. However, relative to word token-level inputs in language, low-level computer vision inputs are extremely high-dimensional: a single 1080p frame contains over 2 million pixels. In this context, modeling the richer event dynamics in long-range video sequences seems like a daunting endeavor, especially with additional modalities (e.g., speech, optical flow, etc.) and increasing resolutions. Understandably, a naïve approach to fully processing every individual pixel is likely prohibitive. Current vision models [e.g., Radford et al. 2021; Sun et al. 2019a; Tan and Bansal 2019; Kim et al. 2021a] often address this by processing embeddings that summarize image patches or even groups of frames altogether, but this has the potential drawback of losing fine-grained details [Ramesh et al. 2021]. In addition to considerations of the raw input space, foundation models for vision may need to revisit the design of fundamental architecture primitives (§4.1: modeling) for efficient and effective modeling: alternatives to 3D convolutions may better address its cubic complexity [Fan et al. 2020; Sitzmann et al. 2019], while particle-based representations may prove more effective for modeling physical dynamics [Bear et al. 2021]. Further, deployment of these vision models to downstream application settings will also necessitate advancements in systems design (§4.5: systems). Taken together, the bottleneck of efficient and effective modeling for larger-scale, dynamic vision inputs remains a multi-faceted research direction that must be addressed going forward.
Training, environments, and evaluation. Equally critical to realizing the potential of foundation models are the supporting elements for training and evaluating them. Current foundation models for vision have largely focused on a small subset of modalities shown in Figure 7 (e.g., datasets of RGB images and text), since these are perhaps the most readily accessible [Changpinyo et al. 2021; Radford et al. 2021]. This motivates the development and use of additional large-scale training datasets which contain a diverse collection of inputs across a broad spectrum of modalities. While additional annotations may not strictly be necessary, the input quality impacts the learning efficiency of the models; techniques that leverage foundation models of other types (e.g., language) to help improve quality are a promising route forward [Zellers et al. 2021b]. We also want to consider settings beyond static datasets: classic studies have suggested that perceptual understanding in humans is linked to its embodiment and interactive, ecological settings [Gibson 1979]. As stepping stones towards longer-term capabilities of embodiment and interaction (§2.3: robotics), ongoing development of simulation environments that capture physical, visual, and ecological realism with multiple modalities and viewpoints may play an important role in providing scalable and high-fidelity visual inputs for this goal [Kolve et al. 2017a; Savva et al. 2019b; Gan et al. 2020; Shen et al. 2021a; Srivastava et al. 2021]. Finally, there is the question of metrics: how do we evaluate the faithfulness of generative foundation model outputs with respect to semantics? Standard metrics like Fréchet Inception Distance, suffer from known flaws [Bińkowski et al. 2018]; such issues
parallel ones in natural language processing (e.g., metrics like BLEU do not correlate with causal judgements from humans). Having human judgements as part of evaluation may be one route, but incurs significant cost and may not be as scalable [Zhou et al. 2019; Khashabi et al. 2021]. The outstanding and open challenges surrounding the training (§4.2: training), data (§4.6: data), and evaluation (§4.4: evaluation) settings for vision foundation models are indeed quite nuanced, and will be a central area of research going forward.
Concluding remarks. In this section, we explored foundation models in the context of computer vision, from identifying roots in previous computer vision paradigms, to contextualizing its current and anticipated capabilities, to proposing research directions moving forward. We conclude with a brief discussion of some broader societal implications of foundation models for computer vision and their continued development (see also §5: society). The ubiquity of cameras in our society means that advances in computer vision techniques have great potential for disruptive impact; this carries a corresponding burden of responsibility for careful consideration of its risks. There is a well-documented history of learned bias in computer vision models, resulting in lower accuracies and correlated errors for underrepresented groups, with consequently inappropriate and premature deployment to some real-world settings [e.g., Buolamwini and Gebru 2018, §5.1: fairness]. Many of the same underlying issues continue to persist in current foundation models [Agarwal et al. 2021]. As data from additional sensor modalities (e.g., wearable or ambient sensors, Figure 7) become incorporated in these foundation models, concerns surrounding privacy and surveillance become paramount (see §5.6: ethics). Furthermore, generated deepfake images and misinformation pose greater risks as the semantic and generative capability of vision foundation models continues to grow [Dolhansky et al. 2020; Ramesh et al. 2021, §5.2: misuse]. While the intriguing open challenges and opportunities ahead for computer vision and foundation models are significant, addressing these and related risks concurrently remains essential.
## 2.3 Robotics
Authors: Siddharth Karamcheti, Annie Chen, Suvir Mirchandani, Suraj Nair, Krishnan Srinivasan, Kyle Hsu, Jeannette Bohg, Dorsa Sadigh, Chelsea Finn
Fig. 8. Building new types of foundation models for robotics will require massive datasets spanning diverse environments and behaviors. Simulation, robotic interaction, videos of humans, and natural language descriptions could all be useful data sources for these models. Despite the challenges of acquiring data, developing new foundation models for robotics has tremendous potential for a variety of problem formulations in task specification and robot learning. Image credits: [Finn et al. 2016b; Szot et al. 2021].
<details>
<summary>Image 8 Details</summary>
### Visual Description
\n
## Diagram: Foundation Model Data Flow
### Overview
This diagram illustrates the data flow into and out of a "Foundation Model," highlighting its training sources and resulting task applications. The diagram is divided into three main sections: "Data Sources," "Foundation Model," and "Tasks." Arrows indicate the direction of data flow – training and adaptation.
### Components/Axes
The diagram contains the following components:
* **Data Sources (2.3.2):** This section is light blue and contains four sub-components:
* Robotic Interaction (image of a robotic arm manipulating objects)
* Videos of Humans (image of a person cooking)
* Simulation (image of a simulated environment with blocks)
* Natural Language ("Pick up the cup. Turn on the stove.")
* **Foundation Model:** A central, large, light purple sphere with a complex polygonal pattern. Arrows labeled "Training" and "Adaptation" point towards and away from this sphere.
* **Tasks (2.3.1):** This section is light pink and contains two sub-components:
* Intuitive, multi-modal task specification (input: "Make a sandwich", output: Reward Function - a golden trophy) with images of a sandwich.
* Fast adaptation for task learning (input: Policy in Kitchen A, output: Policy in Kitchen B) with images of kitchens.
* **Text:** Several text blocks provide labels and descriptions.
* **Arrows:** Arrows indicate the flow of data.
### Detailed Analysis or Content Details
The diagram shows a flow of information from the "Data Sources" to the "Foundation Model" via "Training." The "Foundation Model" then outputs to the "Tasks" section via "Adaptation."
* **Data Sources:**
* Robotic Interaction: Depicted as a robotic arm interacting with objects.
* Videos of Humans: Shows a person cooking.
* Simulation: Displays a simulated environment with building blocks.
* Natural Language: The text reads: "Pick up the cup. Turn on the stove."
* **Foundation Model:** The central component, receiving data for training and providing outputs for adaptation.
* **Tasks:**
* Intuitive, multi-modal task specification: The input is the phrase "Make a sandwich," and the output is represented by a golden trophy, labeled "Reward Function." Images of a sandwich are also present.
* Fast adaptation for task learning: The input is "Policy in Kitchen A," and the output is "Policy in Kitchen B." Images of two different kitchens are shown.
* **Footer Text:** "Adapts to new tasks, environments, and embodiments."
### Key Observations
The diagram emphasizes the versatility of the Foundation Model, showcasing its ability to learn from diverse data sources (robotic interaction, human videos, simulation, and natural language) and adapt to various tasks (sandwich making, kitchen policy adaptation). The use of images alongside text provides a multi-modal representation of the data flow.
### Interpretation
This diagram illustrates a modern approach to AI development, where a single "Foundation Model" is trained on a broad range of data and then adapted to specific tasks. This contrasts with traditional AI approaches where models are often built from scratch for each task. The diagram suggests that the Foundation Model acts as a general-purpose learning engine, capable of transferring knowledge across different domains. The inclusion of robotic interaction, human videos, simulation, and natural language highlights the importance of multi-modal learning. The "Adaptation" arrow suggests that the model can fine-tune its parameters to perform well on new tasks without requiring extensive retraining. The footer text reinforces the idea that the model is designed to be flexible and adaptable to changing environments and embodiments. The diagram is conceptual and does not provide specific numerical data or performance metrics. It is a high-level overview of a system architecture.
</details>
A longstanding challenge of robotics research is to endow robots with the ability to handle the myriad conditions they will encounter in real-world settings. In this section, we discuss how the ideas underlying foundation models can potentially help bring about 'generalist' robots that can, for example, cook a new meal in a new house, with a new kitchen. To make progress towards this goal, existing foundation models will not suffice. We need new types of models trained on a multitude of data sources, spanning grounded robotic interaction data to videos of humans performing tasks, amongst others. We focus on how such foundation models can apply to the problem of a robot controlling its own physical embodiment to successfully perform different tasks. This is a high-dimensional and closed-loop decision-making problem: the actions that a robot takes directly influence what it perceives next, which in turn influences the next robot action. This closed-loop aspect is not traditionally studied in language and computer vision, where large offline datasets are dominant and foundation models have already seen success. We focus on how the demonstrated benefits of foundation models - large-scale, self-supervised learning - can be leveraged in this new closed-loop data regime. The promise of a new type of robotic foundation model is in its ability to amplify the potential of robots to improve key facets of daily life ranging from manufacturing [Nof 1999; Sanneman et al. 2020], construction [Khoshnevis 2004; Bock 2007], autonomous driving [Thorpe et al. 1988; Badue et al. 2020], to household aid [Thrun and Mitchell 1995; Brooks 2002; Dillmann 2004; Goodrich and Schultz 2007; Gupta et al. 2018; Shridhar et al. 2020] and personal assistance [Dragan and Srinivasa 2013; Javdani et al. 2018], amongst others. Our discussion in this section primarily focuses on mobile manipulation robots for household tasks, but we expect its essence to be broadly applicable to the other use-cases of robotics listed above.
On the critical path towards building new types of foundation models for robotics is embracing opportunities in task specification and task learning , coupled with tackling challenges in data acquisition and safety and robustness . Consider the following robot learning paradigm: starting with
a description of a task capturing what a user might like the robot to do (e.g., 'make breakfast') - learn a corresponding policy to generate the desired robot actions. While policies can be parameterized in different ways, a common choice is that of a function that maps the task representation and environment observation (e.g., a scene image from a fixed or egocentric camera, or inputs from alternative sensors like LIDAR) to robot actions [Andrychowicz et al. 2017; Nair et al. 2018]. As the robot acts in a task-conditioned manner, the subsequent states are fed back to the policy, generating more actions until the task has been satisfied.
Yet, implementing such a paradigm in practice is difficult. To begin, what is the right interface for describing one's goals? For a given user in one context, 'make breakfast' carries an implication of a full breakfast that consists of fried eggs, toast, and a glass of orange juice; for another user, 'make breakfast' may imply idlis with sambar and a tumbler of filter coffee. In general, high-level context-dependent goals like these do not stand alone and can introduce a multitude of ambiguities. How does one specify a goal (and corresponding subgoals) with enough clarity to both resolve these ambiguities, and in so doing, allow a robot to make progress on the given task? Additionally, how might we craft general task representations that might aid generalization to similar objectives (e.g., fetching a glass of milk instead of orange juice). Going a step further, how do we build methods that aid robots in learning policies for new tasks and new environments (in this case, a brand new kitchen with new utensils, appliances, layouts, etc.)?
Recent breakthroughs in applying foundation models for language and vision (§2.1: language and §2.2: vision) suggest several potential benefits of large-scale, self-supervised pretraining for improving generalization. The ability to tap into diverse streams of data to learn meaningful representational priors (akin to those learned by models such as BERT and GPT-3) holds promise for learning powerful robotic foundation models for task specification. Diverse robotic interaction data can be used for learning action-conditional dynamics models or policies indexing general and semantically meaningful skills thereby holding promise for task learning. Yet while these opportunities exist, the key stumbling block is collecting the right data . Unlike language and vision data, robotics data is neither plentiful nor representative of a sufficiently diverse array of embodiments, tasks, and environments - we (as a field) still have not converged on the kinds of data that would be maximally useful for enabling generalist robotics (e.g., offline demonstrations, thirdperson recordings of humans, egocentric videos, autonomous experience, etc.) Coupled with issues in obtaining the right scale and diversity of data are questions of ensuring safety and robustness: how do we behave in a new environment without causing damage?
Building new types of foundation models for robotics thus consists of a dichotomy of opportunities and challenges: opportunities for task specification and learning balanced against challenges of data collection and safe deployment. This section explores both by presenting a picture of how robotic foundation models might help us develop generalist robots, in a way that not only meaningfully addresses the challenges associated with building such systems, but that also embraces the potential of multi-modality - incorporating perception, actuation, and language - as well as human-robot interaction for specification and learning.
## 2.3.1 Opportunities.
Robotic foundation models could take a variety of forms: problems in robotics do not easily conform to a one-size-fits-all model, since different problems have different input-output signatures - a contrast to domains like NLP where many problems can be cast into a general 'text-in, text-out' signature. We focus on opportunities in generalizable task specification and learning across tasks, environments, and robot embodiments.
Foundation models for task specification. Before robots can learn how to solve tasks in a general purpose way, they must understand what the desired task is: for example, to be useful in a new kitchen, a robot needs to know what we would like it to cook, as well as behaviors we would like it to avoid. Therefore, a necessary first step towards developing generalist robots is building a new type of foundation models for reliable task specification, i.e., the intuitive and effective communication of task objectives, preferences, and constraints. We formalize task specification as a process that transforms a human-provided task description into a quantitative metric that measures a robot's task completion and progress - e.g., a reward function. This signal is crucial for optimizing robot behavior, diagnosing failures, and prompting human feedback. As the most natural way to describe a task can vary depending on the user, environment, or task, robotic foundation models for task specification should accept a variety of description modalities, such as goal states [Fu et al. 2018; Singh et al. 2019], natural language [MacGlashan et al. 2015; Karamcheti et al. 2017; Misra et al. 2017b; Co-Reyes et al. 2019; Shao et al. 2020], videos of humans [Shao et al. 2020; Chen et al. 2021c; Liu et al. 2018], pairwise or ranking comparisons [Biyik and Sadigh 2018], interactive corrections [Co-Reyes et al. 2019; Karamcheti et al. 2020] and physical feedback [Ross et al. 2011; Bajcsy et al. 2017].
An important requirement of general purpose models for task specification is the ability to transfer to new environments and tasks. Reliably transforming task descriptions into generalizable reward signals for robot learning remains an open problem [Taylor et al. 2016] - one that robotic foundation models will arguably be well suited for. When applied to task specification, such models should provide more robust (§4.8: robustness) reward signals by learning from large and broad datasets - even leveraging multiple of the description modalities listed above. One possible instantiation of a new foundation model for task specification might be to learn a mapping from arbitrary (language, current observation) pairs to reward signals by training on diverse language and vision datasets [Bahdanau et al. 2019; Fu et al. 2019; Chen et al. 2021c]. By learning informative priors from these broad, diverse datasets, such a model may be able to generalize to unseen language instructions and observations in unseen environments. In general, the potential for new foundation models to be able to deftly bridge modalities and generalize broadly make them appealing for general purpose task specification.
Foundation models for task learning. In addition to enabling more general task specification, robotic foundation models could make learning to solve new tasks more efficient and reliable. In this context, these new types of foundation models might take the form of a joint distribution over actions, sensor observations, rewards, and other properties of interest. Conditioning on different dimensions of this joint distribution recovers different inference problems, each corresponding to a different signature:
- Dynamics modeling : 𝑝 (future observations | actions, past observations) [Finn and Levine 2017; Hafner et al. 2019; Wu et al. 2021d].
- Policy learning : 𝑝 (actions | observations, goal) [Kaelbling 1993; Schaul et al. 2015; Ding et al. 2019].
- Inverse reinforcement learning : 𝑝 (reward function | observations, actions) [Ng and Russell 2000; Ziebart et al. 2008; Finn et al. 2016a].
A plausible training objective for a robotic foundation model is to predict the different elements of the joint distribution described above in an autoregressive fashion [Janner et al. 2021; Chen et al. 2021b, §4.1: modeling]. However, these are not the only options. In particular, robot datasets contain large amounts of unlabeled data consisting of synchronized observations from many different sensor modalities (e.g., RGB and depth cameras, haptic sensors, microphones, etc.) and a sequence of actions that the robot has performed to generate these observations. Beyond the objectives above,
a robotic foundation model could be trained to predict observations of one sensor modality from another or to predict whether two streams of sensory observations are from the same segment of time. These kinds of self-supervised objectives can leverage multi-modal correspondences to produce low-dimensional representations of high-dimensional data, and can even be combined with the above objectives to yield models, policies, and rewards on top of those representations.These objectives may facilitate the training of powerful robotic foundation models from unlabeled data as long as the data exhibits diverse, meaningful behavior. §2.3.2: robotics-challenges discusses the challenges of collecting such data further.
In language and vision, foundation models have demonstrated the capability to learn broadly applicable priors from large, diverse datasets, that can be subsequently adapted to downstream tasks (§2.1: language, §2.2: vision). Robotic foundation models have the potential to similarly enable few-shot adaptation of perception and control to new environments, tasks, and embodiments though leveraging different data, self-supervised objectives, and modalities than have been studied with existing language and vision models. Consider our running kitchen example. To cook in a new kitchen, a robot needs to adapt to the specific environment - its spatial layout, the available equipment, etc. Priors learned from offline videos of humans, robotic interaction, text, and/or simulation might encode general aspects of kitchens, such as the fact that stoves are usually against walls and must be turned on in order to produce heat. Such commonsense knowledge, physical priors, and visual priors could make adaptation to new environments more sample efficient. Similarly, developing a new foundation model for robot task learning might enable the use of a large number of cooking videos in its training dataset to adapt a policy for a common skill, such as 'fry an egg,' to a specific user's preferences from a low number of demonstrations - allowing for sample efficient adaptation. Finally, with their potential to learn the cross-modal representations described earlier, robotic foundation models could help enable adaptation to new embodiments. This aspect of adaptation is crucial to make these models widely useful.
## 2.3.2 Challenges and risks.
Despite this exciting vision, multiple challenges need to be overcome. To enable the generalization discussed above, we must collect robotic datasets of sufficient size and diversity. Additionally, we need mechanisms to ensure that we can deploy learned behaviors safely in the real world.
Data needs & challenges. Learning a policy for a robot that perceives the state of its environment via sensors and takes actions to accomplish tasks traditionally requires large datasets of the robot interacting in the real world . On the other hand, many learning tasks in computer vision and natural language processing rely on large and diverse offline datasets that can easily be scraped from the web. Motivated by the advances of existing foundation models in language and vision, we are excited by the possibility of leveraging large offline data sources for training new foundation models for robotics.
One path towards this goal is collecting large datasets for offline learning, for example using teleoperation [Mandlekar et al. 2019], kinesthetic teaching [Sharma et al. 2018], or autonomous methods [Pinto and Gupta 2016; Gupta et al. 2018; Levine et al. 2018; Dasari et al. 2019; Kalashnikov et al. 2021; Chen et al. 2021d], which have shown some promising indications on generalization. While scaling up robot data collection to the size of vision and language datasets [Deng et al. 2009; Krishna et al. 2017; Raffel et al. 2019; Gao et al. 2020a] remains an open challenge, the increasing scale and quality of robotic datasets suggests they can play an important role in learning foundation models for robotics. Moreover, because robots have the ability to actively and autonomously shape their environment, they should be able to generate targeted unlabeled data at scale.
Given the challenging closed-loop nature of learning control, it is possible that collecting datasets of size comparable to those used in vision and language is insufficient for robotics. One exciting option is to additionally leverage external, non-robotic sources of data such as videos of humans or existing vision and natural language datasets. Such data is diverse and exists in large quantities on the web [Deng et al. 2009; Lee et al. 2012; Heilbron et al. 2015; Goyal et al. 2017a; Damen et al. 2018; Gao et al. 2020a; Grauman et al. 2021], affording the possibility of broad generalization if properly leveraged. Elegantly addressing the gap between the robot's domain and those found in videos or language on the web remains an open challenge; however, recent progress in domain adaptation [Smith et al. 2019; Schmeckpeper et al. 2020] and using pretrained video and language models in robotics [Lynch and Sermanet 2020; Shao et al. 2020; Chen et al. 2021c] present promising directions towards closing this gap.
Finally, simulation presents a boundless source of rich interactive data that robots can learn from, with a range of sensor modalities like rendered visuals, point-clouds, and simulated touch/audio. However, a major challenge lies in bridging the gap between simulation and the real world, both in the underlying physics and in the semantic distribution of environments and tasks. Recent work has shown that by using extensive domain randomization, tasks ranging from flight [Sadeghi and Levine 2017] to contact-rich manipulation [Mahler et al. 2017; OpenAI et al. 2019] and locomotion [Peng et al. 2020; Hwangbo et al. 2019] skills learned in simulation can be transferred to real robots with some success, and that the semantic and visual distribution of the real world can be simulated by scanning the real world into a simulation [Chang et al. 2017; Kolve et al. 2017b; Savva et al. 2019a; Szot et al. 2021; Shen et al. 2021a]. While these are promising steps towards closing the sim-to-real gap, effective and general sim-to-real learning of manipulation and locomotion skills remains an open challenge. Simulation data, real robot data, videos of humans, and natural language data could all be essential to learning robotic foundation models.
Safety & robustness. Further complicating the development of new foundation models for robotics is ensuring their safety and robustness when training or deploying them in the real world. We can expect the safety risks from these models for robotics to be different from their language counterparts given that embodied agents are empowered to manipulate and interact with their surroundings directly in the physical world. One core safety challenge for learning-based systems is the chicken-and-egg problem of needing to specify system constraints for safety prior to collecting data, after which unforeseen unsafe behaviors requiring additional constraints may emerge. For instance, an agent adapting to a new kitchen outside of the training distribution requires sufficient safety guarantees to ensure safe data collection, which may either adversely affect task performance or cause the agent to fail in novel ways. One way to resolve this is restricting the complexity of the environment or increasing the complexity of the robot such that irrecoverable states or unsafe actions are avoided by construction. The robot can also be tasked with autonomously resetting the environment to facilitate uninterrupted learning (or adaptation) from large-scale data collection [Eysenbach et al. 2017; Gupta et al. 2021b]. This would either mean ensuring that nothing in the kitchen is breakable, or ensuring and replacing the items the agent may break while it attempts to collect data.
To address risks posed by robotic foundation models that fail to generalize or produce unexpected behaviors to new stimuli, potential future directions include developing a causal analysis of agents [Déletang et al. 2021], new formal safety evaluation tools, and realistic simulation environments [Corso et al. 2020; Dreossi et al. 2017; Julian and Kochenderfer 2019]. Finally, deriving formal safety guarantees for robotic foundation models, e.g., Hamilton-Jacobi reachability of safe-sets [Chow et al. 2018; Fisac et al. 2019; Herbert et al. 2021] or developing safety boundaries for learning that are interpretable (§4.11: interpretability) to human operators, could help reduce risks posed
by such models [Berkenkamp et al. 2017]. As the development and study of these new types of foundation models progresses, solutions to these challenges will be crucial.
Conclusion. While the promise of robotic foundation models are many - spanning multiple levels of the robotics pipeline from task specification to task learning - the challenges are significant. Collecting data in the physical world that covers diverse environments and embodiments at scale is a sizable hurdle, and ensuring the safety and robustness of such systems is equally exigent. Despite this, our optimism prevails; tackling these challenges now, before developing models offers us the chance to identify ways to collect the right data, from the right sources, at the right scale to build safe and reliable robotic foundation models with the capabilities we desire.
Underpinning this section has been a theme of multimodality. Robotic foundation models - in all possible instantiations - have and will continue to benefit from work in other subfields of AI such as language and vision (§2.1: language, §2.2: vision). Yet as we consider incorporating these extensions from other fields, there are interdisciplinary challenges on the horizon that touch other aspects of foundation models: systems innovation for training and deploying such models for real-time robotics (§4.5: systems), innovation in interfaces for robust human-robot interaction (§2.5: interaction), and lessons to incorporate as we better grasp the safety and robustness of such models (§4.9: ai-safety, §4.8: robustness). Building a reliable ecosystem and thoughtful research practices around foundation models in general, and robotic foundation models in particular, is key to realizing these goals.
## 2.4 Reasoning and search
Authors: Yuhuai Wu, Frieda Rong, Hongyu Ren, Sang Michael Xie, Xuechen Li, Andy Shih, Drew A. Hudson, Omar Khattab
Fig. 9. Multimodality can allow foundation models to not only reason with formal symbolic language, but also exploit visual aspects of the problem, such as equivalence, symmetry, and Euclidean geometry, to prune the infinite search space and find promising constructions for a solution (§2.4.1: reasoning-tasks), mimicking the way humans reason about geometry problems.
<details>
<summary>Image 9 Details</summary>
### Visual Description
\n
## Diagram: Geometric Proof Flow
### Overview
The image depicts a geometric proof process, visualized as a series of transformations applied to a triangle. The diagram shows a sequence of steps, each building upon the previous one, leading from an "Initial State" to a "Target State". Each step is associated with a percentage value, presumably representing the confidence or progress towards the target. The diagram uses arrows to indicate the flow of the proof.
### Components/Axes
The diagram consists of five main stages, arranged horizontally. Each stage contains a triangle diagram and a textual description of the applied step. Below each stage is a smaller text label indicating a percentage and a brief description of the next step. The diagram also includes labels for the initial and target states.
* **Initial State:** A triangle ABC with AB = AC.
* **Step 1:** Draw bisector AD. Associated percentage: 80%. Text: "Draw bisector AD". Angle labels: ∠BAD = ∠CAD.
* **Step 2:** Apply reflexivity. Associated percentage: 90%. Text: "Apply reflexivity". Side label: AD = AD.
* **Step 3:** Apply congruence. Associated percentage: 85%. Text: "Apply congruence". Triangle labels: ΔABD ≅ ΔACD.
* **Target State:** ∠ABC = ∠ACB. Associated percentage: 95%.
* **Bottom Row:** A series of steps leading to the next stage, with associated percentages: 10% "Draw L parallel to BC", 5% "Draw...", 3% "Apply...".
* **Arrows:** Curved arrows connect each stage, indicating the flow of the proof.
* **Text:** "Prove that ∠ABC = ∠ACB" is located below the initial state.
### Detailed Analysis or Content Details
The diagram illustrates a geometric proof to demonstrate that ∠ABC = ∠ACB.
1. **Initial State:** The starting point is a triangle ABC where sides AB and AC are equal (AB = AC).
2. **Step 1 (80%):** A bisector AD is drawn, dividing angle BAC into two equal angles (∠BAD = ∠CAD).
3. **Step 2 (90%):** The reflexive property is applied, stating that AD is equal to itself (AD = AD).
4. **Step 3 (85%):** The congruence of triangles ABD and ACD is applied (ΔABD ≅ ΔACD).
5. **Target State (95%):** The conclusion is reached: ∠ABC = ∠ACB.
The bottom row indicates the steps needed to move to the next stage, with decreasing percentages: 10% to draw a line L parallel to BC, 5% to draw something (unspecified), and 3% to apply something (unspecified).
### Key Observations
* The percentages associated with each step suggest a level of confidence or the amount of work remaining. The percentage increases with each step, peaking at 95% in the target state.
* The diagram visually represents the logical flow of a geometric proof.
* The bottom row of steps seems to indicate the remaining work to complete the proof, but the descriptions are incomplete.
* The diagram uses standard geometric notation (e.g., ∠ for angles, Δ for triangles, ≅ for congruence).
### Interpretation
The diagram demonstrates a proof by construction approach to show that if AB = AC in triangle ABC, then ∠ABC = ∠ACB. The steps involve bisecting the angle at A, applying the reflexive property, and then using congruence to establish the equality of the base angles. The percentages likely represent the confidence level in each step or the remaining effort to complete the proof. The diagram is a visual aid for understanding the logical sequence of steps in a geometric proof. The incomplete descriptions in the bottom row suggest that the proof might be more complex than initially presented, or that the diagram is a simplified representation of a larger proof. The diagram is a clear and concise way to illustrate the process of deductive reasoning in geometry.
</details>
Reasoning and search have been a central theme throughout the history of AI. Classic tests of intellect, from strategy games to abstract mathematical discovery, served as inspirational goal posts that pushed the limits of 'machine intelligence' through a need to devise ever smarter ways of searching for winning solutions. In the early days, symbolic methods were the dominant approach for reasoning [Russell and Norvig 2020], but the involved engineering effort and the need to formalize heuristics to tackle intractable search spaces quickly proved cumbersome. More recently, data-driven methods using neural networks have shown encouraging results - e.g., defeating the best humans in Go [Silver et al. 2016], a board game with a much larger space of actions than the classic challenge of chess - by exploiting statistical structures and learning useful heuristics. This section outlines existing reasoning tasks, ones that require scaling to ever-larger search spaces and understanding the world broadly (§2.4.1: reasoning-tasks). We then argue in §2.4.2: reasoningrole that foundation models should play a central role towards general reasoning as vehicles for capturing the statistical regularities of unbounded search spaces ( generativity ), allowing positive transfer across tasks and scenarios ( universality ), and exploiting the grounding of knowledge in multi-modal environments ( grounding ).
## 2.4.1 What are the current tasks?
Many reasoning problems pose unbounded search spaces, where systems must deal with numerous kinds of open-ended alternatives. Consider trying to prove that the angles ∠ 𝐵 and ∠ 𝐶 are equal for an isosceles triangle △ 𝐴𝐵𝐶 with 𝐴𝐵 = 𝐴𝐶 (Figure 9). A system can perform any number of actions at each step of reasoning . For instance, the system could add a new auxiliary point with an arbitrary construction, say a perpendicular line, a parallel line, or a tangent circle, and the search space only grows larger as the diagram grows more complicated. One way to prove this theorem is to draw a line 𝐴𝐷 that is the angle bisector of 𝐴 , and use the congruence of the two triangles △ 𝐴𝐵𝐷 and △ 𝐴𝐶𝐷 to show ∠ 𝐵 = ∠ 𝐶 , but how can systems find this without extensive search?
More generally, a mathematician is not confined with searching in diagram constructions and Euclidean theorems: mathematicians can apply a vast number of theorems from various branches
Fig. 10. Left: A reaction route for 1,6-Heptadiene-3,5-dione predicted by machine learning-based drug retrosynthesis planner AiZynthFinder [Genheden et al. 2020; Yoshikawa et al. 2021]. Right: A sample proof tree in propositional logic where the formulas outlined in green represent axioms. Although they arise from different domains, both trees are structurally the same.
<details>
<summary>Image 10 Details</summary>
### Visual Description
\n
## Diagram: Chemical Retrosynthesis & Propositional Logic
### Overview
The image presents two diagrams side-by-side. The left diagram illustrates a chemical retrosynthesis process, showing a series of reactions leading to simpler precursor molecules. The right diagram depicts a logical deduction process using propositional logic rules. Both diagrams use a flowchart-like structure with boxes representing compounds or logical statements and arrows indicating transformations or inferences.
### Components/Axes
**Left Diagram (Drug Retrosynthesis):**
* **Compounds:** The diagram features several organic molecules represented by their structural formulas. These include:
* Butanoic acid with a chlorine substituent (C₄H₇ClO₂)
* Ammonia (NH₃)
* An intermediate compound with a longer carbon chain and a ketone group.
* A brominated alkene.
* A diene.
* A final product, a conjugated diene.
* **Arrows:** Arrows connect the compounds, indicating a retrosynthetic step (breaking down a complex molecule into simpler precursors).
* **Label:** "Drug Retrosynthesis" is located at the bottom-left corner.
**Right Diagram (Propositional Logic):**
* **Logical Statements:** The diagram contains propositional statements:
* A → B (A implies B)
* ¬B (Not B)
* ¬A (Not A)
* ¬A → C (Not A implies C)
* C
* D → C (D implies C)
* **Inference Rules:** Labels on the arrows indicate the logical rules applied:
* "Contrapositive"
* "Elimination"
* "Introduction"
* **Label:** "Propositional Logic" is located at the bottom-right corner.
### Detailed Analysis or Content Details
**Left Diagram (Drug Retrosynthesis):**
The retrosynthesis starts with a complex molecule (the final product) and proceeds upwards, breaking it down into simpler precursors.
1. The final product (a conjugated diene) is derived from a precursor diene.
2. The diene is derived from a brominated alkene.
3. The brominated alkene is derived from an intermediate ketone compound.
4. The ketone compound is derived from butanoic acid with a chlorine substituent and ammonia.
**Right Diagram (Propositional Logic):**
The logical deduction proceeds as follows:
1. Starting with A → B and ¬B, the "Contrapositive" rule is applied to derive ¬A.
2. Starting with ¬A and ¬A → C, the "Elimination" rule is applied to derive C.
3. Finally, starting with C, the "Introduction" rule is applied to derive D → C.
### Key Observations
* The retrosynthesis diagram demonstrates a stepwise simplification of a molecule, a common strategy in organic chemistry for designing synthetic routes.
* The propositional logic diagram illustrates a valid logical argument, demonstrating how conclusions can be derived from premises using established inference rules.
* The diagrams are visually similar in their structure, both employing a top-down flow with boxes and arrows.
### Interpretation
The image juxtaposes two distinct but related problem-solving approaches: chemical synthesis and logical deduction. The retrosynthesis diagram represents a "backward" reasoning process, starting with a target molecule and working backward to identify suitable starting materials. This is analogous to the logical deduction process, where one starts with known premises and uses inference rules to arrive at a conclusion. The use of arrows in both diagrams signifies a transformation or inference, highlighting the dynamic nature of both processes. The diagrams demonstrate how structured reasoning can be applied to both scientific and abstract domains. The "Drug Retrosynthesis" label suggests a practical application of the chemical retrosynthesis process, while the "Propositional Logic" label indicates a more theoretical framework. The diagrams are not providing quantitative data, but rather illustrating a process or methodology.
</details>
of mathematics, make high-level conjectures, formalize new mathematical concepts, or find counterexamples. This contrasts with more structured AI challenges such as the game of Go, whose search space is considered much smaller. 17
Besides theorem proving, many real-world problems deal with unbounded search spaces, such as program synthesis [Gulwani et al. 2017], drug discovery [Drews 2000], chemical synthesis [Segler et al. 2018], computer-aided design [Haigh 1985], combinatorial optimization [Bengio et al. 2021], and more. These reasoning problems tend to exhibit similar structure, like the bijection between retrosynthesis in drug discovery and theorem proving in propositional logic, illustrated in Figure 10: in both problems, one is building a tree of synthesis, whose nodes are chemical products on the one side and propositions on the other, and the leaf nodes are the products on the one side, and end axioms on the other. In these problems, a simulated environment is often provided, which allows a solver to run several search threads towards building the solution tree. The simulator often provides intermediate feedback, say, informing the solver with the remaining propositions to establish before the proof is considered complete. The solver in turn needs to select the most promising search thread and proceed based on the intermediate feedback.
Recently, there has been a surge of interest in applying learning-based approaches to tackle reasoning problems. To overcome the unbounded search space challenge, researchers first started with a constrained search space to make the problem tractable [Huang et al. 2018; Bansal et al. 2019]. But such approaches suffered from the limited kinds of actions the solver could issue. For example, the solver could only apply theorems from a known database to prove the target theorem, instead of synthesizing novel theorems and lemmas. Because large language models offered a generic way of modeling the output space as a sequence, they quickly became a more favorable choice, allowing the generation of arbitrary kinds of actions. Researchers have applied these language model-based approaches to various applications, such as predicting protein structures [Senior et al. 2020], proving formal theorems [Polu and Sutskever 2020; Han et al. 2021], conjecturing theorems [Urban and
17 Less than the number of grid points on the Go board (i.e., 361 actions for a 19 × 19 board).
Jakubuv 2020; Rabe et al. 2021; Li et al. 2021b], synthesizing programs from natural language [Chen et al. 2021f; Ling et al. 2016], repairing, generating and understanding code [Yasunaga and Liang 2021; Lu et al. 2021b; Guo et al. 2020; Svyatkovskiy et al. 2020; Kim et al. 2021b; Zügner et al. 2021]. It has also been shown that scaling model size significantly improves reasoning capabilities [Polu and Sutskever 2020], and furthermore standard techniques from language modelling, such as pretraining, can also greatly improve performance on these tasks [Rabe et al. 2021; Polu and Sutskever 2020].
## 2.4.2 What's the role of foundation models?
Generativity. We believe that the generative capabilities of foundation models are essential for effective reasoning. Due to the unbounded search space, it becomes intractable to enumerate all kinds of possibilities. Instead, with foundation models, one can model the distribution of the optimal decisions, and generate suitable candidates to proceed to the next step. In particular, as foundation models offer a generic way of modeling the output space as a sequence, the next decision generation is entirely unconstrained and hence universal. Such flexibility is essential for many of the reasoning challenges we discussed, to allow creative generation in domains such as mathematical conjecturing [Li et al. 2021b] and synthesizing novel programs [Chen et al. 2021f]. As one scales up foundation models, the capabilities of capturing such statistical structures also grow immensely [Polu and Sutskever 2020].
Universality. As we mentioned in the last section, many reasoning problems exhibit similar latent structures. We believe that the unifying framework imposed by a foundation model can transfer and share significant heuristics across tasks, ranging from generalizing low-level techniques that work well for one task to new scenarios all the way to directly finding meta-techniques that work well across numerous kinds of problems. In addition, since a foundation model is trained across many domains, it can positively transfer meta-knowledge encoded in the foundation models' weights across tasks and domains [Papadimitriou and Jurafsky 2020; Wu et al. 2021f; Lu et al. 2021a]. The foundation model training and adaptation framework encourage a separation of concerns, where foundation model training learns meta-knowledge such as the shared search tree structure between drug retrosynthesis and propositional logic proofs, and the adaptation phase can focus on learning the task specific vocabulary. Thus, foundation models can reduce the complexity of the learning problem in the adaptation phase, improving sample complexity and generalization.
Grounding. Reasoning problems are often easily expressed in symbolic languages (e.g., mathematics, code, SMILE representation of molecules). However, these symbols have deep underlying semantic meanings - saying 'isosceles triangle' paints a vivid image in the human mind. Foundation models can enable deep groundings and semantic meanings. First, grounding representations in other modalities , such as visual or physical, are essential to grasp abstract concepts in reasoning tasks and endow them with concrete meaning [Larkin and Simon 1987; Jamnik 2001]. Since the models may be trained on multiple modalities, foundation models can assist in understanding a range of data sources (e.g., images, texts). Hence, in the geometry example case, with its understanding of geometrical shapes learned in natural images, a foundation model could effectively utilize the diagrammatic representation of the problem. However, aligned multi-modal data in reasoning is scarce, and it remains an open question whether foundation models can discover connections between different modalities in an unsupervised manner (e.g., discovering of commutative diagram with the corresponding algebraic equations). Furthermore, even within the symbolic domain, symbols can have various levels of interpretation. For example, high-level programming languages can be translated to low-level assembly codes. Foundation models can learn a shared representation that encompasses these various views. Past works have shown that self-supervised tasks [Han et al.
2021; Peng et al. 2021; Li et al. 2021a] allow the model to understand the inner workings behind the high-level code scripts, and further assist downstream tasks.
## 2.4.3 Future challenges in reasoning.
Due to the intrinsic difficulty of these problems, high-quality annotated data is scarce and harder to collect compared to raw images and text. There have been several attempts towards alleviating this issue. In mathematics, researchers proposed to generate synthetic theorems in the hope of generalizing to realistic theorems [Wang and Deng 2020; Wu et al. 2021a; Firoiu et al. 2021; Zhou et al. 2021c]. Another approach is to design self-supervised tasks to augment datasets [Yasunaga and Liang 2020; Ren et al. 2020; Han et al. 2021; Rozière et al. 2021; Yasunaga and Liang 2021], or better pretraining objectives [Wu et al. 2021f]. However, we still lack general principled approaches in designing self-supervised tasks, as most of the existing works are tailored to specific problem setups [Yasunaga and Liang 2020; Ren and Leskovec 2020; Han et al. 2021]. Building a foundation model will encourage a unifying framework of constructing a suite of self-supervised tasks that can be applied to all reasoning problems. In addition, interactivity (§2.5: interaction) could, with enough scalability, alleviate the data scarcity problem by bringing humans into the loop to minimally guide the learning curriculum or data augmentation process, for example, in selecting axioms to add or conjectures to explore, while interactive tools themselves are a motivating use of foundation models for reasoning [Han et al. 2021; Chen et al. 2021f] in assisting people with the most cognitively demanding or laborious aspects. Interpretation-friendly interactive tools could find further applications in education by assisting humans in learning with the help of highly capable foundation models (§3.3: education).
Improving the high-level reasoning capabilities is a core challenge for existing foundation models. Humans perform abstract reasoning and high-level planning in tackling difficult problem-solving tasks [Miller et al. 1960]. For example, when building a software tool or proving a theorem, we often start with a high-level sketch before delving into the low-level details [Koedinger and Anderson 1990]. Existing foundation models are not trained to generate such high-level plans. Instead, they often focus solely on predicting the next low-level steps [Polu and Sutskever 2020; Han et al. 2021; Chen et al. 2021f]. Unfortunately, to train foundation models to emulate human-like reasoning, we again face a data collection challenge. Although such data does exist in limited settings [Li et al. 2021b], in general, data for high-level reasoning is scarce and difficult to collect. One line of research is to let abstract and modular hierarchy to emerge by itself during learning [Ellis et al. 2021; Hong et al. 2021], but it still remains an open question how to scale these approaches to more general and realistic settings.
Aside from these challenges, there exist many open questions that are also essential to topics discussed in other sections. What constitutes a good architecture for reasoning reliably (§4.1: modeling)? How can we understand and interpret these models theoretically (§4.10: theory and practically §4.11: interpretability)? Can we train robust reasoning models that could generalize to out-of-domain problems (§4.8: robustness and §4.3: adaptation)? We believe research about foundation models on each of these fronts can greatly broaden their impact for the field of reasoning.
## 2.5 Interaction
Authors: Joon Sung Park, Chris Donahue, Mina Lee, Siddharth Karamcheti, Dorsa Sadigh, Michael S. Bernstein
Fig. 11. Foundation models will bring significant opportunities to developers by lowering the difficulty threshold for building AI-infused applications, and to the application users by raising the ceiling for what types of interactions are achievable. In some cases, the line between developers and users will start to blur, and users may be able to easily develop their own AI applications, for instance with natural language.
<details>
<summary>Image 11 Details</summary>
### Visual Description
\n
## Diagram: AI-Infused Application Development Cycle
### Overview
The image depicts a cyclical diagram illustrating the development process of AI-infused applications, emphasizing a user-centered and iterative approach. The cycle involves users providing feedback, developers creating prototypes, and an AI-driven core that facilitates interaction and generation.
### Components/Axes
The diagram consists of four main components:
1. **Users:** Represented by a group of illustrated people.
2. **Developers:** Represented by an illustrated person working on a laptop.
3. **AI-Infused Applications:** Represented by a smartphone icon.
4. **Central AI Core:** Represented by a colorful, geometric sphere.
The flow between these components is indicated by blue arrows labeled:
* "Experience" (from Users to AI Core)
* "Feedback" (from Users to Developers)
* "Prototyping" (from Developers to AI Core)
* No label for the arrow from the AI Core to Developers.
Three boxes are positioned to the right of the AI-Infused Applications component, detailing aspects of the applications:
* **Multimodal Interaction:** with the subtext "e.g. natural language instruction to media editing" and an icon of a document with a pencil.
* **Generative Applications:** with the subtext "writing & code generation" and an icon of angle brackets with forward slashes.
* **Value-Sensitive Design:** with the subtext "e.g. community-written content moderation tools" and an icon of several emotion faces.
### Detailed Analysis or Content Details
The diagram illustrates a continuous loop.
1. **Users** provide **Experience** to the **AI Core**.
2. **Users** also provide **Feedback** to **Developers**.
3. **Developers** engage in **Prototyping** which is fed into the **AI Core**.
4. The **AI Core** then influences **AI-Infused Applications**.
5. **AI-Infused Applications** are characterized by **Multimodal Interaction**, **Generative Applications**, and **Value-Sensitive Design**.
The central **AI Core** is a complex, multicolored sphere, suggesting a multifaceted and dynamic system. The colors appear to be a gradient of blues, greens, yellows, and pinks.
The three boxes detailing the AI-Infused Applications are positioned vertically, with **Multimodal Interaction** at the top, **Generative Applications** in the middle, and **Value-Sensitive Design** at the bottom.
### Key Observations
The diagram emphasizes the iterative nature of AI application development, with continuous feedback loops between users, developers, and the AI core. The inclusion of "Value-Sensitive Design" suggests a focus on ethical considerations and responsible AI development. The diagram does not contain any numerical data or precise measurements.
### Interpretation
This diagram represents a modern approach to software development, specifically for AI-powered applications. It highlights the importance of user involvement throughout the entire process, from initial experience gathering to ongoing feedback and refinement. The AI core is not presented as a black box, but as a central component that is actively shaped by both user input and developer efforts. The emphasis on multimodal interaction, generative capabilities, and value-sensitive design suggests a vision for AI applications that are not only powerful but also user-friendly, creative, and ethically aligned. The cyclical nature of the diagram implies a continuous improvement process, where applications are constantly evolving based on user needs and technological advancements. The diagram is conceptual and illustrative, rather than providing specific data or technical specifications. It serves as a high-level overview of a development philosophy.
</details>
The early forms of foundation models such as GPT-3 [Brown et al. 2020] and DALL·E [Ramesh et al. 2021] have demonstrated a high level of versatility both in terms of their ability to let even nonML experts to prototype powerful AI-infused applications, and their ability to seamlessly integrate modalities ranging from texts to images. As the development of foundation models matures, the models' capacity will continue to expand and their versatility may ultimately lead to fundamental changes in how we interact with AI by allowing us to rapidly prototype and build highly dynamic and generative AI-infused applications. In this section, we discuss the opportunities that these changes present from the perspectives of two important stakeholders: (1) applications developers who will interact with foundation models to design user experience, and (2) end-users who will use or be affected by the AI-infused applications powered by foundation models. Finally, we consider scenarios in which the line that rigidly separates developers and end-users today may start to blur, affording new opportunities for creating AI-infused applications that more closely satisfy users' needs and values.
## 2.5.1 Impact on AI-infused application developers' development process.
How will foundation models transform the way developers create AI-infused applications? Despite the monumental progress in machine learning algorithms and systems infrastructure, some point out that designing novel and positive forms of human-AI interaction remains difficult [Dove et al. 2017; Cooper et al. 2014]. The vast amount of data, computing resources, and skills needed to create a powerful task-specific model is frequently in conflict with the iterative prototyping process necessary to elicit and satisfy users' needs and values [Yang et al. 2016]. This challenge is further
compounded by the fact that AI responses can be unpredictable, and models can produce a vast generative output space, making it difficult for people to build effective mental models of their performance. There has already been some progress on tackling these challenges in the form of work on interactive machine learning (e.g., Crayon [Fails and Olsen 2003], Regroup [Amershi et al. 2012]) and design frameworks for conveying uncertainty in AI to end-users (e.g., principles of mixedinitiative [Horvitz 1999]). However, more work is still needed to overcome these obstacles [Yang et al. 2020].
Foundation models pose important opportunities to address many of the challenges mentioned above. For instance, language-based foundation models' ability to take natural language as input, and to generalize to many downstream tasks, could significantly lower the difficulty 'threshold' [Myers et al. 2000] for application development, i.e., by enabling the development of sophisticated models without having to collect significant amounts of data and train large models from scratch. This could enable even non-ML experts to quickly prototype AI-infused applications. At the same time, the powerful generative and potentially multi-modal capabilities of foundation models could offer a far higher 'ceiling' [Myers et al. 2000] of what types of interactions are achievable both in terms of their quality and diversity as we will discuss below. However, how successfully we can leverage these capacities will depend on how effectively we can wrangle foundation models into forms that will be more manageable by application developers.
Unfortunately, the same generalizability and high ceiling that give foundation models their edge can also make these models difficult to work with, as they may be even more unpredictable and complex than single-purpose AI models. Indeed, recent work has shown that it can be difficult to make models like GPT-3 consistently perform the intended task [Reynolds and McDonell 2021], while understanding what it is capable of is still an active area of research [Hendrycks et al. 2021a]. In an effort to improve the reliability and trustworthiness of AI-infused applications, we recommend that future work should continue to investigate how to achieve more predictable and robust behaviors from foundation models (e.g., through fine-tuning, or in cases where the main mode of interaction is natural language prompt, through prompt-engineering [Reynolds and McDonell 2021; Liu et al. 2021d], calibrating [Zhao et al. 2021], or pre-formatting a task-specific endpoint. 18 Please see §4.8: robustness for more details).
## 2.5.2 Impact on end-user interaction with AI-infused applications.
Beyond the new ways developers might create AI-infused applications, what changes will foundation models bring to the experience for end-users interacting with these applications? Existing design frameworks for developing user-facing AI applications focus on augmenting (rather than replacing) users' abilities as described by Douglas Engelbart [Engelbart 1963] - we expect that these frameworks should and will remain relevant for the development of future AI-infused applications. For instance, maintaining users' agency and reflecting their values will continue to be a central theme for foundation model-powered applications. Additionally, the benefits of allowing AI agents to take initiatives and automate users' routines versus the benefits of waiting for users' direct manipulation [Shneiderman and Maes 1997] will need to be carefully weighed [Horvitz 1999]. Moreover, users' values should be directly gathered and reflected through processes such as participatory [Lee et al. 2019] and value-sensitive design [Smith et al. 2020] that advocate for actively involving all stakeholders during the designing of the AI-infused applications.
These issues may become especially salient with foundation models because the model may behave in ways that surprise and disappoint users and communities. Generative capabilities might expose biases or points of view that are counter to the communities' goals, or more insidiously,
18 https://beta.openai.com/docs/guides/classifications
draw on such associations in their behavior without the community being aware. This will place a large burden on the groups utilizing foundation models to monitor their models' behavior, and to the extent possible, adapt them to act in appropriate ways.
While the design frameworks for thinking about AI-infused applications to augment users' abilities should remain the same, the actual forms of interactions that are attainable may dramatically diversify due to foundation models' powerful generative and multi-modal capacities. Already, early generations of what can be considered foundation model-powered software tools for multimedia creation and editing have started to drive a new frontier that empowers even novice content creators to generate high-quality multimedia from coarse, intuitive specifications (e.g., collaborative authoring for writers [Lee et al. 2022], text-to-image generation for digital artists, 19 mastering for musicians, 20 and code completion for programmers). 21 Improved foundation models might enable even more ambitious tools (e.g., a fan might provide thematic material for a song which will then be generated in the style of their favorite band, or a business owner might provide simple descriptions of their product which will be used to create a full website). Moreover, foundation models will be used to enrich static multimedia (e.g., automatically remastering legacy multimedia content into new formats, or generating unique experiences for each player in new video games) and may even lead to new forms of multi-modal interactions using interfaces that themselves mix different modalities, such as visual and gesture-based interaction.
We are starting to see glimpses of how foundation models might materialize into concrete interactions in applications ranging from AI Dungeon 22 to Microsoft PowerApps 23 and CoPilot. 24 As we start to envision new forms of interactions, it is of increasing importance for us to think critically about the potential implications these interactions will have on individual users and society to maximize their positive impact. For example, how will foundation model-powered applications change the way we communicate with one another? Will a powerful model write emails in our stead and if so, how will this reshape people's trust, credibility, and identity knowing that the writers may not have written the emails themselves, and how will this alter our writing styles [Hancock et al. 2020]? Who will own the authorship of the model-generated content and how could the shifting responsibilities and ownership of the consent be misused [Weiner 2018] (see §5.5: economics for a more in-depth discussion)? What are the long-term implications that foundation models will have on our work, language and culture [Hancock et al. 2020; Buschek et al. 2021]? Of particular relevance to this last question is the fact that foundation models are trained on observed data and do not necessarily inform us about causality. Hence, how can we ensure that the use of foundation models leads us to a desired future and not a repetition of the past? Though these issues are not necessarily unique to foundation models, they will be amplified and become more prevalent as foundation models accelerate the creation of effective AI-infused applications.
## 2.5.3 Blurring the line between developers and end-users.
Today, the line that separates the developers of AI models and end-users is rigid - it is rarely the case that an end-user has the data, computing resources, and expertise to be able to develop a new model that suits one's values and needs well. While a generic model (i.e., one that is not specific to a specific user or community) could be sufficient in some cases, recent years have seen an increasing number of scenarios in which such models fail to serve users. For instance, a text classification
19 https://github.com/nerdyrodent/VQGAN-CLIP
20 https://www.landr.com/
21 https://copilot.github.com/
22 https://play.aidungeon.io/main/home
23 https://powerapps.microsoft.com/en-us/
24 https://copilot.github.com/
model designed to identify problematic comments for one online community might work well for that community but will fail in others whose norms and cultures may differ significantly (e.g., NSFW communities on Reddit might be more tolerant of certain content, while science communities might reject seemingly mundane anecdotes that are not based on scientific research) [Chandrasekharan et al. 2018]. In another example, AI-powered sensors and robotics tools designed for one target population may fail without the ability to quickly adapt in-context for users with different abilities and needs [Karamcheti et al. 2021]. While recent work has presented promising avenues for future research on how end-users may be able to co-create AI models by manually providing models' parameters or datasets (e.g., WeBuildAI [Lee et al. 2019]), the results are still preliminary and often focus on rudimentary models.
If foundation models can sufficiently lower the difficulty threshold for building AI-infused applications, they could present an important opportunity to more tightly couple users' needs and values with the models' behaviors by allowing users to actively partake in the development process of the models. Recent work has shown that GPT-3, for example, can robustly perform classification tasks in a few-shot or even in zero-shot fashion when given an adequate task description in its natural language prompt [Brown et al. 2020]. An online community trying to moderate its own content might be able to leverage such a capability to create bespoke AI classifiers that filter content based on classification task descriptions that the community has agreed on (of course, this power could also be instead misused to silence the voices of certain members within the community - we point to §5.2: misuse for further discussion on this topic). In addition, the powerful in-context learning capabilities that foundation models will exhibit may allow foundation model-powered applications to more effectively optimize their interfaces on a per-user basis. This could open doors to tackling many salient problems in human-computer and robot interaction such as balancing the power of users' direct manipulation and automation in mixed-autonomy settings.
Of course, there will still be important challenges that we would need to overcome to truly realize this potential for blurring the line between users and developers. These challenges include mitigating existing biases in foundation models, as well as making the models' behavior more robust and manageable even for non-ML experts (compared to ML experts, it could be even more difficult for non-ML experts to understand the full capacities and mechanisms of foundation models, which can lead to unexpected pitfalls in the development cycle [Yang et al. 2018]). Future work should explore how foundation models could be situated in the context of interactive machine learning and study how we can support even those with limited experience with machine learning to leverage these models in a robust manner. Nonetheless, the ability for end-users to be involved in developing AI-infused applications is an exciting opportunity that could introduce a new paradigm for how we will interact with these applications in the future.
## 2.6 Philosophy of understanding
Authors: Christopher Potts, Thomas Icard, Eva Portelance, Dallas Card, Kaitlyn Zhou, John Etchemendy
What could a foundation model come to understand about the data it is trained on? An answer to this question would be extremely informative about the overall capacity of foundation models to contribute to intelligent systems. In this section, we focus on the case of natural language, since language use is a hallmark of human intelligence and central to the human experience.
The best foundation models at present can consume and produce language with striking fluency, but they invariably lapse into the sort of incoherence that suggests they are merely 'stochastic parrots' [Bender et al. 2021]. Are these lapses evidence of inherent limitations, or might future foundation models truly come to understand the symbols they process?
Our aim in this section is to clarify these questions, and to help structure debates around them. We begin by explaining what we mean by foundation model , paying special attention to how foundation models are trained, since the training regime delimits what information the model gets about the world. We then address why it is important to clarify these questions for the further development of such models. Finally, we seek to clarify what we mean by understanding , addressing both what understanding is (metaphysics) and how we might come to reliably determine whether a model has achieved understanding (epistemology).
Ultimately, we conclude that skepticism about the capacity of future models to understand natural language may be premature. It is by no means obvious that foundation models alone could ever achieve understanding, but neither do we know of definitive reasons to think they could not.
## 2.6.1 What is a foundation model?
There is not a precise technical definition of foundation model . Rather, this is an informal label for a large family of models, and this family of models is likely to grow and change over time in response to new research. This poses challenges to reasoning about their fundamental properties. However, there is arguably one defining characteristic shared by all foundation models: they are self-supervised . Our focus is on the case where self-supervision is the model's only formal objective.
In self-supervision, the model's sole objective is to learn abstract co-occurrence patterns in the sequences of symbols it was trained on. This task enables many of these models to generate plausible strings of symbols as well. For example, many foundation models are structured so that one can prompt them with a sequence like 'The sandwich contains peanut' and ask them to generate a continuation - say, 'butter and jelly'. Other models are structured so that they are better at filling in gaps; you might prompt a model with 'The sandwich contains \_\_ and jelly' and expect it to fill in 'peanut butter'. Both capabilities derive from these models' ability to extract co-occurrence patterns from their training data.
There is no obvious sense in which this kind of self-supervision tells the model anything about what the symbols mean. The only information it is given directly is information about which words tend to co-occur with which other words. On the face of it, knowing that 'The sandwich contains peanut' is likely to be continued with 'butter and jelly' says nothing about what sandwiches are, what jelly is, how these objects will be combined, etc. This might seem to suggest an inherent limitation on what a foundation model could achieve. However, we need not restrict the model to seeing only textual input. A foundation model might be trained on a wide range of different symbols: not just language but also computer code, database files, images, audio, and sensor readings. As long as it is just learning co-occurrence patterns of the sequences it is exposed to, then it counts as a foundation model by our definition. As part of this learning, the model might come to represent strong associations between a given piece of text and a particular sensor reading, or between a
sequence of pixel values and a database entry. These associations might reflect important aspects of the world we inhabit and the language we use to talk about it.
## 2.6.2 What is at stake?
Before considering analyses of what understanding is, it is worth reflecting on why we might care about the question of whether a foundation model could achieve it. These models are poised to be deployed for numerous purposes with various functionalities. Some of our goals in deployment may only be met to the extent that the model is capable of understanding. Here we list a few such goals:
- Trust : One might argue that we cannot trust a system's linguistic behavior unless it understands the language it is using. Of course, we currently trust engineered systems to do things (e.g., manufacturing auto parts) without the question of understanding even arising, but language might be special in this regard, since it is uniquely human. In addition, language can be used to deceive and misrepresent, so understanding alone clearly does not imply trust. On the whole, then, understanding might be taken as a necessary condition for trust in the context of language use.
- Interpretability : If genuine natural language understanding in some way involves maintaining and updating an internal model of the world (including, e.g., the speech context), and if we (as engineers) are able to analyze how linguistic input and output interface with this internal model, that could afford substantial gains in interpretability, predictability, and control of these systems.
- Accountability : Not unrelated to the previous points, in the future we may find it desirable to hold artificial agents in some way accountable for the language they produce [The HAI Adaptive Agents Group 2021]. Depending on how we think about concepts like accountability, responsibility, agency, and the like, language understanding may emerge as a prerequisite.
The mere possibility that understanding will play an indispensable role in any of these matters provides strong motivation to develop a framework for theorizing about it.
## 2.6.3 What is understanding?
Our central question is whether a foundation model could come to understand a natural language. With the above, we can now sharpen it: is self-supervision sufficient for understanding, keeping in mind that there are no constraints on the data used for this supervision? In order to address this question, we first need to define what we mean by understanding .
As a start, we find it helpful to make explicit a distinction that is sometimes conflated in discussions of the topic. The distinction is between the metaphysics and the epistemology of understanding. Metaphysics concerns what it would mean ('in principle') for an agent to achieve understanding. Epistemology, by contrast, concerns how ('in practice') we could ever come to know that an agent has achieved the relevant type of understanding. In short, metaphysics is more about our ultimate target, whereas epistemology is more about how (if at all) we could know when we have reached it. Our epistemology thus depends to some extent on our metaphysics.
Metaphysics of understanding. Philosophy of language offers a number of alternatives for what it is to understand natural language. 25 Simplifying the landscape for the sake of brevity, the following three broad classes of views all have connections with research lines in AI and NLP: 26
- Internalism : Language understanding amounts to retrieval of the right internal representational structures in response to linguistic input. Thus, language understanding is not even a possibility without a rich internal conceptual repertoire of the right kind.
- Referentialism : Roughly, an agent understands language when they are in a position to know what it would take for different sentences in that language to be true (relative to a context). That is, words have referents and (declarative) utterances are truth-evaluable, and understanding involves a capacity to evaluate them relative to presentation of a situation or scenario.
- Pragmatism : Understanding requires nothing in the way of internal representations or computations, and truth and reference are not fundamental. Rather, what matters is that the agent be disposed to use language in the right way. This might include dispositions toward inference or reasoning patterns, appropriate conversational moves, and so on. Crucially, the relevant verbal abilities constitute understanding. 27
While this is a simplified picture of the space of possibilities, we already see how they relate in quite different ways to the goals mentioned above. On the pragmatist view, for instance, achieving language understanding does not imply anything about our ability to trust or interpret the system, insofar as it guarantees nothing about the agent's internal structure or its relation to the (non-linguistic) world. On the internalist view, by contrast, a fairly robust kind of internal/causal interpretability is at least strongly suggested. The question of whether or not a foundation model could understand language in principle takes on a very different character depending on which of these metaphysical characterizations we adopt.
Internalism and referentialism can both be cast as defining a mapping problem: to associate a linguistic sign with a 'meaning' or a 'semantic value'. For internalism this will be a representation or concept, a program for computing a value, or some other type of internal object. For referentialism, it might be a mapping from a word to an external referent, or a mapping from a situation to a truth value (all relative to a context). Could self-supervision suffice for achieving the desired mapping in a foundation model? Here, the nature of the training examples might be relevant. If the model receives only linguistic inputs, then its capacity to learn this mapping might be fundamentally limited in ways that prevent it from learning to refer in the relevant sense. (Indeed, Merrill et al. [2021] identify some theoretical limits, albeit under very strong assumptions about what it means to learn the meaning of a symbol.) However, if the input symbol streams include diverse digital traces of things in the world - images, audio, sensors, etc. - then the co-occurrence patterns might contain enough information for the model to induce high-fidelity proxies for the required mapping. 28 For
25 Relatedly, there is a sizable literature in philosophy of science focused on the concept of understanding, mainly as it relates to scientific explanation. See Grimm [2021].
26 We are leaving aside other questions that may be relevant to the metaphysics of understanding, such as whether or not consciousness or some form of subjective experience may be necessary. These are pressing philosophical issues, but they are not easily connected to research in AI and NLP.
27 For an accessible introduction to internalist as well as referential views, we recommend Elbourne [2011]. This version of pragmatism arguably finds its roots in Wittgenstein [1953], but it is expressed most succinctly by Turing [1950], in which Turing suggests replacing the question of whether a machine can think with questions about a specific behavioral test (which came to be known as the Turing Test).
28 To the extent that the mapping embodies causal information, we must also contend with theoretical limitations concerning the possibility of drawing causal inferences from correlational (or even experimental) data (see Spirtes et al. 2001; Bareinboim et al. 2020).
referentialism, there is still a further question of how these proxies relate to the actual world, but the same question arises for human language users as well.
Bender and Koller [2020] give an interesting argument that combines referentialism with pragmatism. They imagine an agent O that intercepts communications between two humans speaking a natural language L. O inhabits a very different world from the humans and so does not have the sort of experiences needed to ground the humans' utterances in the ways that referentialism demands. Nonetheless, O learns from the patterns in the humans' utterances, to the point where O can even successfully pretend to be one of the humans. Bender and Koller then seek to motivate the intuition that we can easily imagine situations in which O's inability to ground L in the humans' world will reveal itself, and that this will in turn reveal that O does not understand L. The guiding assumption seems to be that the complexity of the world is so great that no amount of textual exchange can fully cover it, and the gaps will eventually reveal themselves. In the terms we have defined, the inability to refer is taken to entail that the agent is not in the right dispositional state for understanding.
Fundamentally, the scenario Bender and Koller describe is one in which some crucial information for understanding is taken to be missing, and a simple behavioral test reveals this. We can agree with this assessment without concluding that foundation models are in general incapable of understanding. This again brings us back to the details of the training data involved. If we modify Bender and Koller's scenario so that the transmissions include digitally encoded images, audio, and sensor readings from the humans' world, and O is capable of learning associations between these digital traces and linguistic units, then we might be more optimistic - there might be a practical issue concerning O's ability to get enough data to generalize, but perhaps not an in principle limitation on what O can achieve. 29
We tentatively conclude that there is no easy a priori reason to think that varieties of understanding falling under any of our three positions could not be learned in the relevant way. With this possibility thus still open, we face the difficult epistemological challenge of clarifying how we could hope to evaluate potential success.
Epistemology of understanding. Apositive feature of pragmatism is that, by identifying success with the manifestation of concrete behaviors, there is no great conceptual puzzle about how to test for it. We simply have to convince ourselves that our limited observations of the system's behavior so far indicate a reliable disposition toward the more general class of behaviors that we took as our target. Of course, agreeing on appropriate targets is very difficult. When concrete proposals are made, they are invariably met with objections, often after putative success is demonstrated.
The history of the Turing Test is instructive here: although numerous artificial agents have passed actual Turing Tests, none of them has been widely accepted as intelligent as a result. Similarly, in recent years, a number of benchmark tasks within NLP have been proposed to evaluate specific aspects of understanding (e.g., answering simple questions, performing commonsense reasoning). When systems surpass our estimates of human performance, the community's response is generally that the test was flawed, not that the target was reached. There may be some suite of behaviors that is our real target, but it is just hard to circumscribe or turn into a practical test. 30 Then again, this might reveal that internalism or referentialsm are what we had in mind all along.
29 On our reading, Bender and Koller [2020] allow that multimodal data might change the scenario, especially if O is allowed to have cooperative interactions with the humans about shared scenarios and topics.
30 Part of the difficulty may also relate to the fact that typical humans make frequent errors in many of these domains, but not necessarily the same types of errors that are made by current systems. Characterizing the target behaviours may thus involve more than just identifying the 'correct' behaviour.
If we take internalism or referentialism as the ultimate target - our gold standard for what understanding is - then behavioral tests will always be at best imperfect as a means of assessing whether understanding has been achieved. The imperfections are two-fold. First, behavioral tests will always have gaps that could allow unsophisticated models to slip through. Second, a system might have achieved the mapping that these views require, but we may be unable to show this with behavioral testing. Recent experiences with the model GPT-3 show how challenging this might become: depending on the prompt one uses, one can see surprisingly coherent outputs or utter nonsense, and so prompt engineering requires deep expertise [Rong 2021].
Thus, both internalism and referentialism call for structural evaluation methods that allow us to study their internal representations, probing them for information [Tenney et al. 2019; Manning et al. 2020], studying their internal dynamics [Sundararajan et al. 2017], and perhaps actively manipulating them according to specific experimental protocols supporting causal inference [Vig et al. 2020; Geiger et al. 2020]. There may be fundamental limitations on what we can learn from practical experiments about the inner workings of a complex foundation model, but it is clear that these methods will be useful whenever our target aligns with internalism or referentialism.
## 2.6.4 Moving the discussion forward.
It seems clear that there are no easy answers to the question of whether foundation models will ever understand language. To even begin to address the question, one must resolve a difficult metaphysical question about which there are a number of substantively distinct views. The metaphysical question then feeds into an epistemological question that poses many practical challenges. Nonetheless, the above discussion does invite one practical conclusion: if foundation models are pursued as a path to language understanding in artificial agents, then multimodal training regimes may well be the most viable strategy, as they would seem the most likely to provide the model with the requisite information. Whether self-supervision then suffices is a completely open question.
## 3 APPLICATIONS
The capabilities (§2: capabilities) of foundation models indicate that they have the potential to transform various sectors and industries, extending the role AI plays in society (§5: society). Among the myriad applications where foundation models may be applied, we will focus on three disciplines - healthcare (§3.1: healthcare), law (§3.2: law), and education (§3.2: law) - that are all foundational to societal function. Within each, we discuss the opportunities that foundation models pose for this domain alongside challenges (e.g., interpretability; §4.11: interpretability) and concerns (e.g., privacy; §4.7: security).
## 3.1 Healthcare and biomedicine
Authors: Michihiro Yasunaga, Jing Huang, Camilo Ruiz, Yuhui Zhang, Giray Ogut, Saahil Jain, William Wang, Yusuf Roohani, Hongyu Ren, Antoine Bosselut, Ehsan Adeli, Jure Leskovec, Russ Altman
Fig. 12. Foundation models in healthcare and biomedicine. We visualize an interactive framework where foundation models enable various tasks across healthcare and biomedicine when trained on multimodal data generated by various sources in the healthcare ecosystem. The first column lists several sources of data, including care providers, payers, institutions (universities, non-profits, and governments), pharma, wearables, and medical publications/forums. The second column shows several data modalities generated by the data sources. They include images (e.g., chest X-rays), videos (such as ultrasounds), graphs of chemical compounds, tables for electronic health records (EHRs), text such as clinical notes, time series such as ECGs, and genetic data. The third column visualizes a foundation model trained on such data and then applied to healthcare and biomedicine downstream tasks listed in the fourth column. This process can generate new data that will further improve the foundation model, hence the bidirectional relation between the foundation models and the tasks.
<details>
<summary>Image 12 Details</summary>
### Visual Description
\n
## Diagram: Foundation Model for Healthcare and Biomed
| Category | Description | Examples |
|---|---|---|
| **Data Sources** | Diverse data types used to train the model. | Electronic Health Records (EHR), Medical Imaging (X-rays, CT scans, MRIs), Genomic Data, Biomedical Literature (PubMed), Clinical Trials Data, Real-World Evidence (RWE), Wearable Sensor Data |
| **Foundation Model** | The core AI model, typically a large language model (LLM) or multimodal model. | BERT, GPT-3, PaLM, Med-PaLM, BioBERT, ClinicalBERT, GatorTron, TriLM |
| **Downstream Tasks** | Specific healthcare applications built on top of the foundation model. | Diagnosis Prediction, Drug Discovery, Personalized Medicine, Clinical Decision Support, Patient Risk Stratification, Medical Summarization, Report Generation, Anomaly Detection |
| **Evaluation Metrics** | Measures used to assess the performance of the model. | Accuracy, Precision, Recall, F1-score, AUC-ROC, BLEU score (for text generation), ROUGE score (for summarization), Clinical Utility |
| **Challenges** | Obstacles to overcome for successful implementation. | Data Privacy & Security (HIPAA compliance), Data Bias & Fairness, Model Interpretability & Explainability, Generalizability & Robustness, Computational Cost & Scalability, Regulatory Approval |
| **Key Technologies** | Technologies enabling the development and deployment of foundation models. | Transfer Learning, Self-Supervised Learning, Few-Shot Learning, Federated Learning, Distributed Training, Cloud Computing, GPUs, TPUs |
**Diagram Components:**
* **Input Layer:** Represents the various data sources feeding into the foundation model.
* **Foundation Model Layer:** Illustrates the core AI model processing the data.
* **Output Layer:** Shows the downstream tasks and applications utilizing the model's output.
* **Feedback Loop:** Indicates the iterative process of model refinement based on evaluation metrics and real-world feedback.
**Additional Notes:**
* Foundation models are pre-trained on massive datasets and can be adapted to a wide range of downstream tasks.
* Multimodal models can process and integrate data from multiple sources (e.g., text, images, genomics).
* Addressing challenges related to data privacy, bias, and interpretability is crucial for responsible AI in healthcare.
* Continuous monitoring and evaluation are essential to ensure the model's performance and safety over time.
</details>
Healthcare and biomedicine are an enormous application area in society, for instance, with expenditures accounting for 17% of gross domestic product (GDP) in the US [Swensen et al. 2011; van Hartskamp et al. 2019; Keehan et al. 2020]. Both healthcare (which focuses on the delivery of care to patients via diagnosis, treatment, and health administration) and biomedical research (which focuses on the scientific understanding of disease and the discovery of new therapies) demand significant expenses, time, and comprehensive medical knowledge [Yu et al. 2018; Korngiebel and Mooney 2021]. We envision that foundation models can be a central storage of medical knowledge that is trained on diverse sources/modalities of data in medicine [Krumholz et al. 2016; SoltanianZadeh 2019; Suresh et al. 2020] (Figure 12 left), and can be queried/updated interactively by medical professionals (e.g., healthcare providers and biomedical researchers access published findings and upload new publications) [Ionescu et al. 2020] and queried by the public. As foundation models have strong adaptation capabilities (e.g., fine-tuning, prompting [Brown et al. 2020]), they can be efficiently adapted to various individual tasks in healthcare and biomedicine (e.g., question
answering app used by patients [Klasnja and Pratt 2012; Zhu et al. 2019; Daniel et al. 2019; Liu et al. 2020a], clinical trial matching system [Ni et al. 2015; Harrer et al. 2019; Beck et al. 2020] accessed by researchers and patients; Figure 12 right). This way, foundation models can be a central interface that supports various interactions between data, tasks, and people in healthcare and biomedicine, thereby advancing the efficiency and accuracy of healthcare/biomedical applications [Elbattah et al. 2021]. We elaborate these opportunities in §3.1.1: healthcare-tasks and §3.1.2: biomed-tasks.
At the same time, healthcare/biomedical applications pose unique challenges that motivate further research in foundation models, such as integrating multimodal data in healthcare/biomedicine [Miura et al. 2021; Liu et al. 2021a] and observing ethical and legal regulations in medicine (privacy, safety and explainability) [Guan 2019; Xu et al. 2019]. We elaborate these challenges in §3.1.3: healthcare-biomed-challenge.
## 3.1.1 Opportunities in healthcare.
Foundation models may improve the delivery of care to patients through healthcare providers and hospitals. Currently, healthcare cost increases every year [Keehan et al. 2020], and studies estimate that 30% of healthcare spending may be wasteful due to administrative inefficiency and preventable medical errors [Kocher 2021]. Moreover, as the demand for healthcare increases, the society faces a serious shortage in healthcare providers [Kirch and Petelle 2017]. This inefficiency and shortage in healthcare necessitate developing fast and accurate interfaces for healthcare providers and patients, such as automated aid systems for diagnosis/treatment, summarization of patient records, and answering of patient questions [Davenport and Kalakota 2019; Nie et al. 2018; Wang et al. 2021b]. In particular, in an urgent pandemic crisis such as COVID-19, fast diagnosis/screening (e.g., automatic analysis of chest X-ray images) as well as automated question answering for patients (e.g., symptom checking and care) and the public (e.g., disease prevention) are vital to reduce the spread of diseases and allocate healthcare resources for critical patients, saving more lives [Lalmuanawma et al. 2020]. As foundation models have a strong capability to serve as an integrated knowledge reservoir, they can be queried and adapted to various individual tasks in healthcare. Below are examples of important tasks in healthcare that would benefit from foundation models.
Interface for healthcare providers. Foundation models can improve the efficiency and accuracy of care by providers. Healthcare providers spend unnecessary time editing electronic heath records (EHRs) [Kocher 2021], and preventable medical errors (e.g., hospital readmissions, surgical errors) cause wastes in healthcare [Shrank et al. 2019; Shah et al. 2020]. Foundation models can be adapted as an efficient and accurate interface into EHRs (clinical notes, lab value histories and imaging files) [Li et al. 2020c; Steinberg et al. 2021; Percha 2021], helping healthcare providers create summaries of patient visitation [Krishna et al. 2020], retrieving relevant cases and literature, and suggesting lab tests, diagnosis, treatments and discharges [Zhang et al. 2019b; Rasmy et al. 2021]. Foundation models can also be adapted to help a surgical robot monitor and achieve accurate surgeries [Diana and Marescaux 2015; Agrigoroaie and Tapus 2016; Yu et al. 2019]. See §2.3: robotics for more discussions on foundation models for robotics.
Interface for patients. Foundation models can be adapted to serve as an interface to patients, providing relevant information about clinical appointments [Bates 2019], answering patient questions related to preventive care [Demner-Fushman et al. 2020], along with relevant medical explanatory information (e.g., text and graphics that explain conditions) [Chaix et al. 2019], and helping assistive-care robots for patients [Jeong et al. 2015; Abdi et al. 2018]. See §2.5: interaction for more discussion on foundation models for user interaction. Foundation models can also serve as an interface with the general public to answer questions related to public health and pandemic prevention (such as the COVID-19 case) [Bharti et al. 2020; Herriman et al. 2020]. At the same time,
we note that the interface must guarantee factual accuracy to ensure public trust in medical advice [Kreps and Kriner 2020] (see §3.1.3: healthcare-biomed-challenge).
## 3.1.2 Opportunities in biomedicine.
Foundation models may facilitate biomedical research such as discovery of drugs and understanding of diseases, which ultimately translates to improved healthcare solutions [Hanney et al. 2015]. Currently, biomedical discovery requires significant human resources, experimental time and financial costs. For instance, drug development involves a complex process, from basic drug research of protein target identification and potent molecule discovery to clinical development (e.g., clinical trials) to the final drug approval, which typically takes over 10 years and costs more than one billion dollars [Wouters et al. 2020]. Facilitating and accelerating biomedical discovery using existing data and published findings is an imperative problem in biomedicine [Yu et al. 2018]. In particular, a novel disease outbreak such as COVID-19 costs millions of lives and trillions of dollars [Lalmuanawma et al. 2020; McKibbin et al. 2020]; if we can speed up drug development for new diseases, that would be very helpful. Foundation models can be particularly helpful for biomedical discovery in two aspects. First, foundation models have a strong generative capability (e.g., coherent text generation in GPT-3), which can help generative tasks in biomedical research such as generating experimental protocols (clinical trials) and designing molecules that work (drug discovery) given existing data [Kadurin et al. 2017; Harrer et al. 2019]. Second, foundation models have a potential to integrate diverse data modalities in medicine, which enables investigating biomedical concepts (e.g., disease) from multiple scales (using molecule-, patient- and population-level data) and multiple knowledge sources (using imaging, textual and chemical descriptions). This facilitates biomedical discoveries that are difficult to obtain if using single-modality data [Lanckriet et al. 2004; Aerts et al. 2006; Kong et al. 2011; Ribeiro et al. 2012; Wang et al. 2014, 2015c; Ruiz et al. 2020; Wu et al. 2021h]. Foundation models also enable transfer knowledge across modalities. Lu et al. [2021a] showed how a transformer model trained on natural language (a data-rich modality) could be adapted for other sequence-based tasks such as protein fold prediction, which is a long-studied predictive task in biomedicine [Jumper et al. 2020]. Below are examples of important tasks in biomedicine that will benefit from foundation models.
Drug discovery. To discover a drug or a therapeutic that treats a disease, researchers must first identify a target (e.g., proteins, genes, RNA causally implicated in the disease) and must then search for molecules (e.g., chemical compounds, antibodies) that bind to the target and treat the disease. Typically, identifying the appropriate target and generating a corresponding molecule requires years of expensive wet lab experiments [Hughes et al. 2011; Schenone et al. 2013; Schneider 2018]. Foundation models' generativity can improve the search space and efficiency (see §2.4: reasoning), which not only reduces the amount of experiments but also helps to discover new and better drugs [Jin et al. 2018; You et al. 2018; Walters and Barzilay 2020; Stokes et al. 2020]. Moreover, the simultaneous solution of related drug discovery problems (i.e., target identification, efficacy prediction, side effect prediction, and others) by a single foundation model may improve the solutions to each of them [Ramsundar et al. 2015; Camacho et al. 2018; Duran-Frigola et al. 2020; Huang et al. 2021a]. As an example, one area where foundation models have shown significant potential for impacting therapeutic design is the modeling of proteins using language models. Successful applications range from predicting viral mutations that can escape a vaccine-induced immune response to predicting protein docking potential for better design of therapeutic antibodies [Bepler and Berger 2021; Hie et al. 2021; Tsaban et al. 2021; Wu et al. 2021b; Rives et al. 2021].
Personalized medicine. Personalized medicine aims to select the optimal treatment for individual patients based on their health history, genetics, imaging, and other personal measurements [Collins
and Varmus 2015; Ashley 2016]. For instance, given a set of drugs and a patient genome, foundation models may help predict which drug is likeliest to treat the patient with minimal side effects [Whirl-Carrillo et al. 2012; Tatonetti et al. 2012; Gerstung et al. 2017; Grinfeld et al. 2018; Adam et al. 2020]. Foundation models are uniquely powerful in their ability to integrate multimodal patient data ranging from the EHR [Rajkomar et al. 2018] to medical imaging [Bera et al. 2019; Ouyang et al. 2020] to drug and molecular measurements [Gottlieb et al. 2011; Ruiz et al. 2020] to make an optimal prediction.
Clinical trials. Clinical trials study efficacy and safety of treatment or drug candidates. Conventional clinical trials are inefficient and costly: 80% of trials fail due to inability to show efficacy/safety or problems with patient matching [Ali et al. 2020; Liu et al. 2021c]. Foundation models can help in the following: predicting potential failures and design promising clinical trial protocols (e.g., patient eligibility criteria) based on existing studies; and automating matching of eligible patients based on patient individual profiles, which are multimodal data including EHRs, gene sequence, etc. [Harrer et al. 2019].
## 3.1.3 Challenges and future research in foundation models.
While there are potential opportunities for foundation models to help, healthcare/biomedical applications also pose unique challenges that motivate further research in foundation models.
Multimodality. Medical data are highly multimodal, with various data types (text, image, video, database, molecule), scales (molecule, gene, cell, tissue, patient, population) [Kong et al. 2011; Ruiz et al. 2020], and styles (professional and lay language) [Lavertu and Altman 2019; Li et al. 2019]. Current self-supervised models are developed for each modality (e.g., text [Lee et al. 2020b], image [Chaitanya et al. 2020], gene [Ji et al. 2021], protein [Jumper et al. 2020]), and do not jointly learn from diverse modalities. To learn the inter-modality and cross-modality information from these diverse multimodal medical data, we need to investigate both feature-level and semantic-level fusion strategies in the training of foundation models. If done effectively, this has a potential to unify biomedical knowledge and facilitate discoveries as discussed in §3.1.2: biomed-tasks.
Explainability. Explainability - providing evidence and logical steps for decision making - is crucial in healthcare and biomedicine [Holzinger et al. 2019], and is made obligatory under the General Data Protection Regulation (GDPR). For instance, in diagnosis and clinical trials, patient symptoms and temporal relevance must be explained as evidence. This helps the resolution of potential disagreement between the system and human experts. Explainability is also needed for informed consent in healthcare [Amann et al. 2020]. However, current foundation models' training objectives do not include explainability, requiring future research in this direction [Linardatos et al. 2021]. Incorporation of knowledge graphs may be a step to further improve model explainability [Roberts et al. 2020; Xu et al. 2020; Jin et al. 2021]. Readers are refered to §4.11: interpretability for more discussion on explainability.
Legal and ethical regulations. Healthcare applications must observe legal and ethical regulations with guarantees, such as patient safety, privacy and fairness. For instance, regarding safety, predictions made by foundation models must be factually accurate with established medical knowledge, and must quantify uncertainty or choose to defer to an expert when uncertain [Challen et al. 2019; Mozannar and Sontag 2020]. For privacy, the use of patient health records must observe the privacy laws, such as HIPAA [Act 1996] in the case of the US. Federated learning is one potential solution to keeping the raw, sensitive data private in the training of foundation models [Chamikara et al. 2021]. For fairness, researchers will need to be mindful of common pitfalls or otherwise risk exacerbating existing social inequalities [Chen et al. 2019; Wiens et al. 2019; Chen et al. 2020b]. They must
ensure that the training and evaluation data for foundation models is sufficiently representative of different sexes, races, ethnicities and socioeconomic backgrounds; an area where medical datasets and clinical trials have had a long history of bias [Martinez-Martin et al. 2020; Kaushal et al. 2020]. Research is also needed to debias and regularize models to ensure fairness when representative data is scarce [Zhao et al. 2020a]. Foundation model developers also need to consult with ethics and law researchers, and observe regulations in the specific circumstances (e.g., country, region) where they are deployed. We also refer readers to §4.7: security, §4.8: robustness, §5.1: fairness, §5.4: legality for details on privacy, robustness, fairness and legality.
Extrapolation. The process of biomedical discovery involves extrapolation. For instance, foundation models must be able to quickly adapt to new experimental technologies (e.g., new assays, new imaging techniques such as high resolution microscopy) or new settings (e.g., new target diseases such as COVID-19) [Jaroch et al. 2018; Benam et al. 2019]. The ability to leverage existing datasets and extrapolate to new settings is a key machine learning challenge in biomedicine [Snell et al. 2017; Ma et al. 2021b]. While GPT-3 exhibits some extrapolation behaviors (e.g., generating new text not seen before), its mechanism is unclear and still in its infancy. Further research is needed for improving the extrapolation capability of foundation models, especially when considering the diverse range of data modalities and tasks that is inherent to healthcare and biomedicine but is not commonly studied in current GPT-3 and related models. Also see §4.8: robustness.
## 3.2 Law
Authors: Peter Henderson, Lucia Zheng, Jenny Hong, Neel Guha, Mark Krass, Julian Nyarko, Daniel E. Ho
Fig. 13. An example of various steps of a civil case in the United States and where foundation models might help. At each stage different modalities might need to be processed and adaptation is needed to a new court or legal perspective.
<details>
<summary>Image 13 Details</summary>
### Visual Description
\n
## Diagram: Legal Process Workflow
### Overview
The image depicts a linear workflow diagram illustrating the stages of a legal process, from pre-litigation to appeals. Each stage is represented by a light blue rounded rectangle with a title and a list of associated activities. Small icons are placed below each stage title to visually represent the activities. The diagram flows from left to right, indicating a sequential progression.
### Components/Axes
The diagram consists of seven main stages:
1. **Before Litigation**
2. **Lawsuit Filed**
3. **Discovery**
4. **Trial**
5. **Verdict**
6. **Appeals**
Each stage has associated sub-points listed below the stage title. Icons are used to represent each stage.
### Detailed Analysis or Content Details
**1. Before Litigation:**
* **Activities:**
* Contract review
* Patent retrieval
* **Icon:** A document with a magnifying glass.
**2. Lawsuit Filed:**
* **Activities:**
* Multimodal evidence
* Case law adaptation
* Arguments crafting
* Consideration of judge preferences
* **Icon:** A briefcase.
**3. Discovery:**
* **Activities:**
* Multilingual sources & documents
* Different distributions
* Few-shot learning on lawyers exemplars
* **Icon:** A stack of files/folders.
**4. Trial:**
* **Activities:**
* Legal research
* Dialogue agents for oral arguments
* Judge questions prediction
* **Icon:** Scales of justice.
**5. Verdict:**
* **Activities:**
* For judges & clerks
* Adaptation to writing style & philosophy
* **Icon:** A gavel.
**6. Appeals:**
* **Activities:**
* Adaptation to new contexts of appeals court and supreme court
* **Icon:** A golden crown/laurel wreath.
### Key Observations
The diagram presents a simplified, high-level overview of the legal process. The use of icons adds a visual element to each stage, making the diagram more accessible. The activities listed under each stage are concise and focus on key tasks. The diagram emphasizes the sequential nature of the legal process.
### Interpretation
This diagram serves as a visual guide to the stages involved in a legal case. It highlights the transition from initial assessment and preparation ("Before Litigation") through the formal legal proceedings ("Lawsuit Filed," "Discovery," "Trial," "Verdict") and potential further review ("Appeals"). The inclusion of terms like "multimodal evidence," "few-shot learning," and "dialogue agents" suggests a modern approach to legal practice, incorporating technology and data analysis. The diagram is likely intended for an audience unfamiliar with the intricacies of the legal system, providing a clear and concise overview of the process. The diagram does not contain any quantitative data or specific measurements; it is a qualitative representation of a workflow.
</details>
From family court to criminal justice and from environmental policy to corporate transactions, the reach of the law is vast. In the United States, 31 there are over 1.3M lawyers [American Bar Association 2021] and annual revenues for legal services exceed $300B [MarketLine 2021]. Yet 'access to justice' remains far out of reach for most. Legal services can be prohibitively expensive. Roughly 86% of low-income individuals with civil legal problems in the United States, for instance, report receiving inadequate or no legal help [Legal Services Corporation 2017]. Even when counsel is appointed, lawyers might be strained by increasingly large caseloads. Studies have shown that public defenders, for example, are often overworked and underfunded [Lefstein and Spagenberg 2009; Schumm2012; American Bar Association 2004]. The U.S. Department of Justice reported that in 2007, 73% of county-based public defender offices exceeded the maximum recommended limit of cases received per attorney and 15 of 19 reporting state public defender programs exceeded the maximum recommended limit of felony or misdemeanor cases per attorney [Farole and Langston 2010; Langston and Farole 2010]. Even in a country with one of the highest per capita rates of attorneys, justice can appear out of reach. U.S. President Jimmy Carter once opined, 'Ninety percent of our lawyers serve ten percent of our people. We are overlawyered and underrepresented' [Carter 1978]. According to a leading voice in access to justice, technology may provide a path forward [Rhode 2014], a view echoed by many others [Cabral et al. 2012].
What role might foundation models play in the law? 32 Amajor promise is that foundation models can improve access to justice and government services by leveling procedural and financial barriers to legal services. The challenges posed by legal applications can, in turn, motivate basic research questions for foundation models. Many legal applications pose unique challenges to computational solutions. Legal language is specialized and legal outcomes often rely on the application of ambiguous and unclear standards to varied and previously unseen fact patterns. At the same time, due to its high costs, labeled training data is scarce. Depending on the specific task, these idiosyncrasies
31 We restrict our discussion to legal applications in the United States because of the expertise of the authors. Some discussion here may apply to legal venues globally, however.
32 We note that for the purposes of this section we consider foundation models to be any self-supervised pretrained model that is used to quickly adapt to new contexts with little supervised learning. See also the discussion in §1: introduction and §2.6: philosophy for an expanded definition.
can pose insurmountable obstacles to the successful deployment of traditional models. In contrast, their flexibility and capability to learn from few examples suggest that foundation models could be uniquely positioned to address the aforementioned challenges.
Throughout this section, foundation models may take as context many modalities as evidence: audio during trial proceedings, video and images during discovery, and text in conducting legal research. Yet, the majority of legal tasks in which reliance on foundation models will be beneficial involve text-based inputs and outputs. As such, we mainly focus on text-based domains while only briefly discussing others. To ground the discussion, Figure 13 describes the stages of a civil lawsuit in the United States and where foundation models might come into play in this process. Figure 14 shows the logic flow required to generate just part of one paragraph of a legal brief, which might serve as a concrete example of a task that foundation models might one day be used for.
An Important Consideration. Before proceeding, we note that the ethical, legal, and fairness considerations expanded on in §5.6: ethics, §5.4: legality, and §5.1: fairness are particularly important to examine before using foundation models in an applied legal or government context, as these applications often have important, real-world consequences to those affected [Surden 2020]. Foundation models must also be thoroughly scrutinized before deployment, as discussed in §4.4: evaluation. For example, the legal system places particular emphasis on - and may even mandate - transparency, accountability, and explainability. Consequently, it is questionable whether current models are positioned to solve many of the most pressing, legal problems. Nonetheless, the need to expand and improve access to legal and government services provides a worthy goal for foundation models.
## 3.2.1 Opportunities in law.
Legal applications can range from the use of machine learning in government contexts [Engstrom et al. 2020; Coglianese and Ben Dor 2020; Re and Solow-Niederman 2019] to aiding lawyers in their provision of legal services [Zheng et al. 2021; Huang et al. 2021b; Ostendorff et al. 2021; Vold and Conrad 2021]. We note that prior work has also surveyed machine learning-assisted legal tasks in text-based domains [Zhong et al. 2020; Chalkidis et al. 2020], although it has been noted that recent legal AI research has focused on geographic regions outside of the U.S. [Zheng et al. 2021]. While many of the topics we discuss here may be applicable to different legal systems, due to the expertise of our team we focus primarily on the U.S. In particular, we concentrate on three broad categories of legal applications that may benefit from foundation models in the U.S. legal system: private law or civil justice (claims between private individuals, arising out of, for instance, contracts, property or torts), criminal law (i.e., the prosecution of individuals for criminal behavior), and (non-criminal) public law (e.g., the regulation of private behavior by government agencies).
Civil law. In U.S. civil proceedings, parties must typically find and pay attorneys to be represented. As a result, many individuals, especially those with low income, struggle to secure adequate legal representation [Rhode 2004]. Foundation models have the potential to improve access to justice by reducing the cost, improving the quality, and extending the reach of legal services. In Figure 13, we describe the process by which a civil lawsuit is filed in a U.S. court and where foundation models may play a role in aiding both attorneys and judges.
Even before an attorney is involved in the legal process, clients may benefit from the deployment of foundation models. Recent work has used machine learning models to identify the relevant legal issues contained in a plain-language description of facts presented by a client. 33 Tools like these can help provide a recommendation for the type of legal action needed to address the issue at hand or to recommend a specialized attorney. A number of other similar efforts have sought to increase
33 https://spot.suffolklitlab.org/
access to justice by providing information tailored to a client's particular needs [Cabral et al. 2012; Brescia et al. 2014; Queudot et al. 2020; Westermann et al. 2019].
Once a client speaks with an attorney, prior to civil litigation, the attorney may seek to avoid a costly trial. At this stage, they can rely on foundation models to evaluate contracts, review terms of service, find relevant patents, and conduct other pre-litigation processes in order to ensure that their clients are at an advantage [Betts and Jaep 2017; Elwany et al. 2019; Lippi et al. 2019; Lee and Hsiang 2019; Hendrycks et al. 2021c; Hegel et al. 2021]. Notably, recent work has both described the challenges and benefits of using foundation models for contract review [Leivaditi et al. 2020; Hegel et al. 2021; Hendrycks et al. 2021c]. In addition to reviewing and drafting legal documents, client interactions and documents can be translated to reduce costs and barriers to the provision of legal services [Cuéllar 2019]. But translation of legal documents requires precision and an understanding of highly technical language, which makes collecting training data costly. Additionally, translating client statements or trial proceedings often requires an understanding of local dialects and language. This, too, makes it difficult to collect enough ground truth translation data to train on. As a result, traditional supervised methods rarely achieve the level of accuracy required in the legal domain [Vieira et al. 2020]. Foundation models may improve performance in this area over fully supervised mechanisms by adapting quickly in these low-resource contexts.
During litigation, foundation models can help lawyers to conduct legal research, draft legal language, or assess how judges evaluate their claims [Zheng et al. 2021; Huang et al. 2021b; Ostendorff et al. 2021; Vold and Conrad 2021; Chalkidis et al. 2020, 2019]. This could potentially reduce the costs of and improve legal services. For example, recent work has utilized pretrained models for the recommendation of relevant citations and holding statements when writing legal texts [Zheng et al. 2021; Huang et al. 2021b; Ostendorff et al. 2021]. Other work uses pretrained models for improved legal question answering to power commonly used legal search engines and help lawyers conduct legal research [Vold and Conrad 2021]. A wide variety of work has also examined automated contract drafting and review, a task that could similarly benefit from foundation models [Hendrycks et al. 2021c; Betts and Jaep 2017]. Perhaps most compelling, foundation models may help assist lawyers generate legal briefs (written arguments). The models might find novel arguments or identify problems in attorney-written portions of the brief. For example, Tippett et al. [2021] predict the outcome of a legal proceeding based on features extracted from the filed briefs. Foundation models can be leveraged to use raw language as inputs rather than extracted features. This might provide attorneys with more informative recommendations as to how their brief could be improved to ensure a favorable outcome.
After opening and reply briefs are filed, parties then begin the discovery process, which has already used simple machine learning models for the better part of a decade [Grossman and Cormack 2010]. Attorneys use these systems to label whether a document should be produced to the opposing party. The documents are multi-modal in nature, often containing video, images, audio, and text. Current systems are costly because they used supervised learning and active learning to label the documents as responsive [Grossman and Cormack 2010; Oard et al. 2018; Yang et al. 2021]. Instead, few-shot or zero-shot document retrieval capabilities that might be possible with foundation models would help ease concerns about the large costs of the current process. 34 To avoid the possibilities of gamesmanship in the discovery process, Cui [2018] has proposed a zero-shot (or few-shot) adaptation process that can only be operationalized through the use of foundation models.
After discovery, once the trial begins, foundation models could help parties prepare for trial by predicting what the judge might focus on during questioning [Dickinson 2018], adapting to
34 https://www.kirkland.com/publications/article/2020/04/technology-assisted-review-framework
the current context from judges' prior published opinions. In the courtroom, foundation models might be used to examine audio and video of courtroom proceedings to determine if outcomes were biased against the defendant because of their race or dialect. 35
Once the trial concludes, foundation models could help judges and law clerks to properly evaluate legal claims from both parties using similar technologies, or the use of contextual embeddings from foundation models might assist in statutory interpretation [Nyarko and Sanga 2020; Choi 2020]. Recent work (without reliance on foundation models or NLP) has examined whether an appeals decision can be predicted from a set of extracted features, like citation counts and the appearance of key words [Katz et al. 2017; Boniol et al. 2020]. It is possible that such models could be improved using foundation models and applied to help judges draft decisions by flagging obvious mistakes in their opinion, as has been discussed in the context of adjudicative agencies [Engstrom et al. 2020; Ray and Lubbers 2014]. They can also be used to identify racial biases in legal opinions and help judges revise their opinions accordingly [Rice et al. 2019].
Criminal law. One particularly contentious area has been the use of risk scores in government settings, particularly in criminal law. Some may want to use language-based foundation models to aid in making charging decisions or parole decisions based on a given text-based narrative of the events. Careful consideration must be taken before using foundation models for risk scoring due to the potential for biases, especially when language data is included [Bender et al. 2021; Berk et al. 2021; Laufer 2020]. But foundation models may play a role in many other dimensions of criminal justice. The same tools as in civil litigation, above, can also be used by prosecutors and defense attorneys. This can help appointed attorneys perform their job more efficiently and reduce unnecessary overhead. As a result, they may be able to balance already heavy caseloads more effectively. For example, public defenders are often viewed as being overworked and underfunded, which would lead to avoidable procedural errors. 36 Foundation models can help reduce some of these resource constraints by identifying errors and automating simple tasks. However, they are not a solution on their own.
In other areas, foundation models can act as an oversight mechanism to reduce structural inequities. Pretrained models have been used for processing parole hearing transcripts to find instances of anomalous outcomes [Bell et al. 2021]. Recent work has also removed linguistic cues for a suspect's race in police reports to promote race-blind charging decisions and avoid racially biased prosecutions [Chohlas-Wood et al. 2020]. Other work has helped identify disrespectful police communications [Voigt et al. 2017]. In these contexts, it is very costly to label data since annotators must be given access to sensitive data and appropriate background checks are often required. To reduce these costs, foundation models can be used to pretrain and adapt quickly to downstream tasks where labels are scarce.
Public law. Government agencies regulate vast parts of society, and foundation models have wide potential applicability across public law. This includes: analyzing public comments in the noticeand-comment process, assisting patent examination, retrieving relevant documents in response to Freedom of Information Act requests, aiding in mass adjudication, among many others. Recent work has surveyed these government applications in a variety of contexts and we refer the reader to the relevant sources for in-depth discussion [Engstrom et al. 2020; Coglianese and Ben Dor 2020]. In many of these applications, foundation models can improve the quality, efficiency, utility, and
35 For example, speaking African-American Vernacular English dialects in the courtroom has been shown as a potential source of bias during trial. https://www.nytimes.com/2019/01/25/us/black-dialect-courtrooms.html
36 See, for example, in People v. Superior Court (Vasquez) , 27 Cal.App.5th 36 (2018) a defendant did not receive a trial for 17 years because the public defender's office had severe budget cuts and understaffing. The court ruled that the systemic breakdown in the public defender's office constituted a due process violation and the defendant's case was dismissed.
accessibility of government services: labels are scarce, resources are constrained, and contexts are constantly shifting. As such, the adaptability and flexibility of foundation models are often required to improve efficiency and performance. To give an illustrative example of just one such application, existing work has leveraged NLP for facilitative moderation in public comment forums. In this use case, predictive models help lay-users improve arguments and identify misstatements in their comments. Such a system has already been deployed in the U.S. Department of Transportation rulemaking process [Park et al. 2012], although it can likely be improved through the linguistic reasoning capabilities of foundation models. But government agencies must comply with constitutional, statutory, and administrative obligations (see §5.4: legality), so additional care is needed in these settings.
## 3.2.2 How can foundation models uniquely help?
The above examples of legal applications are unique in several ways. First, the cost of annotating data is very high. Often, the expertise to create high-quality labels can only be found in attorneys, who may charge hundreds of dollars per hour. Even after labels are obtained, certain data may be sensitive and cannot be pooled together to training a large language model. Given recent progress in few-shot learning [Brown et al. 2020], foundation models are among the most promising paths for learning models with limited annotations.
Second, legal decision-making requires context at various scales: knowledge of all historical decisions and standards, knowledge of the case law that remains relevant in the present, and knowledge of the nuances of the individual case at hand. Foundation models are uniquely poised to have the potential to learn shared representations of historical and legal contexts, as well as have the linguistic power and precision for modeling an individual case.
## 3.2.3 What are foundation models lacking that requires more research?
To illustrate the deficiencies current foundation models need to overcome in order to be realistically deployed, we consider as an example the automatic creation of a legal brief to submit to a court.
A brief lays out the arguments to a judge before a hearing. Once a party has filed an opening brief, the opposing party files a response. The judge then evaluates the briefs and asks questions of both parties at a hearing before making a decision. Figure 14 visualizes the structure of such a legal brief with some of its characteristic features.
An automated brief generation mechanism might take as context relevant documents and facts of a case (as specified by an attorney) as well as a rough sketch of the desired outcome. It would then generate a legal brief with complex legal arguments to submit to the court.
Long Documents and Narratives. To achieve this goal, the model must be able to read long contexts and produce long narratives. Legal documents tend to be far longer than documents in any other context. The average U.S. Supreme Court opinion contains around 4,700 words, 37 a brief on the merits to the Supreme Court can have as many as 15,000 words, 38 a law review article often contains 20,000 to 30,000 words, 39 parole transcripts can be hundreds of pages long [Bell et al. 2021], and trial records can be even longer. Current foundation models have struggled with such long contexts and outputs (see §4.1: modeling for more discussion).
Retrieval, Concept Drift, Argument Formation, and Logical Reasoning. In addition to reading casespecific documents, the foundation model must retrieve the relevant case law and understand which
37 https://www.americanbar.org/groups/public\_education/publications/teaching-legal-docs/how-to-read-a-u-ssupreme-court-opinion/
38 https://www.supremecourt.gov/casehand/courtspecchart02162010.aspx
39 https://www.stanfordlawreview.org/submissions/article-submissions/
Fig. 14. An extract from a fictional brief written by one of the authors of this work. The prototypical form that law students are instructed to write a brief involves: (1) introducing the argument; (2) stating the legal rule in a persuasive manner; (3) applying the legal rule to the facts of the case; (4) persuasively concluding the argument. This often involves information retrieval and paraphrasing from both prior cases and the facts of the current case.
<details>
<summary>Image 14 Details</summary>
### Visual Description
\n
## Document: Legal Brief Structure
### Overview
The image presents a document outlining the structure of a legal brief, specifically focusing on how to build an argument based on case law and evidence. It's a text-heavy document with four main sections highlighted in a light blue color. The document includes a short dialogue excerpt.
### Components/Axes
The document is structured into four main sections, each with a descriptive title in bold, dark blue text:
1. **Topic sentence and argument formation**
2. **Extracting legal rule from the case law, crafting into form favorable to argument**
3. **Applying to the facts of the case**
4. **Extracting quotes from the record to back the application**
The document also contains legal citations (e.g., *Lessie, 47 Cal. 4th 1152, 1169 (2010)*, *Faire, 442 U.S. 707, 725 (1979)*).
### Content Details
Here's a transcription of the text within the document:
"Sonia’s behavior during the interrogation confirms what her background would suggest: not only that Sonia could not have understood her Miranda rights, but she did not actually understand the consequences of her actions in waiving them.
The court must look to a minor’s conduct during an interrogation to determine if the minor actually understood their rights. *Lessie, 47 Cal. 4th 1152, 1169 (2010)*. A minor must have the capacity to “understand the warnings given [her], the nature of [her] Fifth Amendment rights, and the consequences of waiving those rights.” *Lessie, 47 Cal. 4th 1152, 1167 (2010)* (citing *Faire, 442 U.S. 707, 725 (1979)*)). In *Lessie*, the minor being interrogated provided no indication that he was unable to understand the consequences of waiving his rights.
In contrast, Sonia showed that she fundamentally misunderstood the consequences of her rights. At the end of the interrogation Sonia believes she will get to return home as a result of her cooperation:
“SFG: But you told them I- you’re going to tell them I cooperated?”
“DP: Yeah, I’m gonna tell them about you being honest with us today, and like I said, I can’t promise that that’s going to cause any result, I just don’t know that, and then they’ll have to weigh that in the case.”"
### Key Observations
The document emphasizes the importance of demonstrating a defendant's (Sonia's) lack of understanding of their rights, particularly Miranda rights, and the consequences of waiving those rights. It highlights the legal precedent set by the *Lessie* and *Faire* cases. The dialogue excerpt illustrates Sonia's misunderstanding – she believes cooperation will lead to going home, suggesting she did not fully grasp the legal implications of her statements.
### Interpretation
This document outlines a legal strategy for arguing that a minor's waiver of rights was not knowing and voluntary. The structure presented is a logical progression: establishing the argument, grounding it in legal precedent, applying it to the specific facts of the case, and supporting it with direct evidence from the record (the interrogation transcript). The *Lessie* and *Faire* cases are used as benchmarks for determining whether a minor possesses the capacity to understand their rights. The dialogue is crucial because it provides direct evidence of the defendant's flawed understanding. The document suggests a focus on the subjective understanding of the defendant, rather than simply whether the warnings were given. The document is a guide for constructing a persuasive legal argument, not a presentation of facts or data in the traditional sense.
</details>
case law is still valid and which has been overruled, taking into account potential concept drift since it was trained. More work in editing grounded information in foundation models will be required as case law evolves [De Cao et al. 2021]. Using retrieved legal standards, the foundation model must then understand how to weave them into a persuasive argument. Emerging research has studied ways of using foundation models to measure, detect, and generate persuasive texts [Duerr and Gloor 2021; Li et al. 2020a; Longpre et al. 2019], which may provide stepping stones towards this goal.
But legal briefing also requires the ability to identify a relevant legal rule and determine how it applies to a new situation. For example, Holzenberger et al. [2020] provide a dataset and methodology for evaluating this ability in current models based on the tax code. Given a set of tax code rules and a description of someone's situation, the model must determine how the rules apply to the individual in order to determine what their tax liability is. 40 Holzenberger et al. [2020] found that domain experts can express the relevant tax law as a prolog solver to determine an individual's tax liability with perfect accuracy. However, they found that simply using a BERT model, even when pretrained on in-domain legal text, was not sufficient to allow the model to deduce and apply the relevant rules, a process also referred to as 'legal reasoning.'
40 For example, a rule might look like, 'In the case of an individual who is not married and is not a surviving spouse, paragraphs (1) and (2) shall be applied by substituting '$750' for '$600'.' US Internal Revenue Code (IRC) §63(f )(3).
These challenges can further be illustrated through a real exchange with GPT-3, demonstrating that current models are unable to perform even comparatively simple tasks involving legal reasoning.
## Legal Reasoning with GPT-3.
'Liquidated damages' are a form of monetary compensation for a breach of contract that the parties specify in their agreement. In general, courts in the U.S. will not enforce liquidated damages provisions if the amount stipulated in the contract is so high that it acts as a form of punishment, rather than to be merely compensatory.
First, we see if the model encoded the correct rule about the enforcement of liquidated damages provisions.
- q: Are liquidated damages clauses enforceable?
- a: Liquidated damages clauses are generally enforceable unless the sum stipulated is exorbitant or unconscionable.
The rule appears correct. But then we apply the rule to a set of facts. Here, the model needs to infer that one million in damages is exorbitant for a Toyota Corolla and thus, the liquidated damages provision should not be enforced.
- q: X and Y contract over the sale of a Toyota Corolla. Their contract specifies that X has to pay Y one million dollars in damages if X does not deliver the car on time. X does not deliver the car on time. Is Y entitled to damages? If so, in what amount?
- a: Yes, Y is entitled to $1,000,000 in damages. Damages are meant to compensate the injured party for the loss suffered due to the breach. Y has suffered a loss of $1,000,000 because X has not delivered the car on time.
The model is unable to infer that $1M for a Toyota Corolla is exorbitant. Even if the model is provided with both the abstract rule and its precise translation to the context in the prompt - and the application should thus be trivial - GPT-3 is still unable to conduct the logical reasoning required.
- q: X and Y contract over the sale of a Toyota Corolla. Their contract specifies that X has to pay Y the exorbitant amount of one million dollars in damages if X does not deliver the car on time. X does not deliver the car on time. The law requires that liquidated damages clauses are enforced unless the sum stipulated is exorbitant or unconscionable. Is Y entitled to damages? If so, in what amount?
- a: Yes, Y is entitled to the liquidated damages. The amount in this case is not exorbitant since the price was set by the parties themselves.
Precision. The foundation model also cannot create fake facts in the process, a problem that has come up in current models [Gretz et al. 2020; Zellers et al. 2019b]. Specificity and truthfulness are of heightened important in legal contexts, where imprecise statements can have drastic, unanticipated consequences, and false statements can lead to sanctions against attorneys.
Few-shot Learning. Some challenges facing foundation models beyond those described in the above example include few-shot learning, which is still in its infancy [Perez et al. 2021]. Research on few-shot learning techniques and increasing access to legal corpora can work in tandem. Because foundation models need to be precise, not only in factual truth, as discussed above, but also in technical legal language, it remains unclear to what extent information obtained from one corpus can be utilized in another corpus of a different domain. Few-shot learning thus remains important.
Adaptation. Some gains have been observed from domain-adaptive pretraining on unlabeled legal corpora. These gains appear to be most pronounced when the pretraining corpus is highly relevant to the downstream task and labeled training data is limited (a setting which is common in the law) [Zheng et al. 2021]. It has not yet been comprehensively studied whether this extends to a diverse set of legal tasks, but leveraging unlabeled domain-specific corpora for self-supervised training of foundation models may provide complementary improvements to few-shot methods.
Access to Clean In-Domain Data. Some recent efforts have sought to create large labeled datasets for more challenging legal benchmark tasks through automation [Zheng et al. 2021] or manual annotation by volunteer legal experts [Hendrycks et al. 2021c]. These efforts have demonstrated that larger language models that are pretrained on more data achieve performance gains on certain challenging tasks, compared to more limited gains observed in other settings [Chalkidis et al. 2020; Elwany et al. 2019; Zhong et al. 2020]. This work suggests that larger legal benchmark datasets may be necessary to observe further gains from applying transfer learning techniques to foundation models. However, creating benchmark datasets for tasks that are legally meaningful and difficult from an NLP perspective can itself be challenging, as human expert annotation can be costly and automated methods that utilize conventional tokenization and sentence segmentation techniques can fail to account for unique aspects of legal text, such as the structure of legal citations [Bommarito et al. 2018; Savelka et al. 2017]. As a consequence of these challenges, many existing legal domainspecific labeled datasets are small, not publicly available, or reflect simpler tasks that have been solved by methods often pre-dating the development of foundation models. 41
Much available legal data may also be unrepresentative. Since only a fraction of cases end up in legal opinions, it is unclear whether the disputes in publicly available data are representative of the typical disputes presented to a model in practice [Priest and Klein 1984]. Costly training data for more representative scenarios may be concentrated in the biggest law firms. These law firms may have the ability to retain and accumulate data across many cases and clients. One concern then is that foundation models could concentrate power even more among the few actors that have the resources to train models on in-domain data - unless the models can generalize sufficiently well.
Reliability. Finally, we again note that even if foundation models could successfully perform all tasks in the legal domain, deployment remains a major challenge: a failure of a foundation model in the law will have real, damaging consequences to both clients and attorneys (see also discussion on fairness, legality, and ethics in §5.1: fairness, §5.4: legality, and §5.6: ethics). For this reason machine translation software has already been deemed unreliable for use as evidence in some courts, 42 although it continues to be relied upon in others. 43
Given all of these complexities, legal briefing and reasoning is likely beyond the capabilities of current models, but appears to be within the future realm of possibilities. As such, these serve as a potential lode star for the ongoing development of foundation models.
41 For law firms and legal technology companies, tasks for which high performance can already be achieved, and can therefore be more immediately productized, may be considered more worthwhile to invest costly manual labeling efforts towards.
42 See discussion by Vieira et al. [2020].
43 For example, in Vasquez v. United States , No. 3: 16-cv-2623-D-BN (Dist. Court, ND Texas 2019), counsel relied on Google Translate to prove that the previous (native speaker) attorney has mistranslated a plea deal.
## 3.3 Education
Authors: Ali Malik, Dorottya Demszky, Pang Wei Koh, Moussa Doumbouya, Drew A. Hudson, Allen Nie, Hamed Nilforoshan, Alex Tamkin, Emma Brunskill, Noah Goodman, Chris Piech
Fig. 15. Foundation models in education could be trained on multiple data sources to learn the capabilities necessary for education: an understanding of various subject matter and different pedagogical techniques. These foundation models can be applied in a general-purpose way across a range of tasks and goals such as understanding students, assisting teachers, and generating educational content.
<details>
<summary>Image 15 Details</summary>
### Visual Description
\n
## Diagram: Foundation Model for Learning
### Overview
The image is a diagram illustrating a Foundation Model for learning, connecting Multimodal Data Sources to Tasks & Goals. The diagram is divided into three main sections: Multimodal Data Sources (Pedagogy & Subject Matter), a central "Foundation Model" component, and Tasks & Goals. Arrows indicate the flow of information from the data sources to the tasks.
### Components/Axes
The diagram consists of the following components:
* **Header:** "Multimodal Data Sources"
* **Left Section (Pedagogy - Structure):**
* "Teaching Materials"
* Icons representing: Textbooks, Pre-Recorded Lectures, Lesson Plans & Curricula, Exams & Assignments.
* "Interaction"
* Icons representing: Discussion Forums, Online Courses, Live Classroom Sessions, Feedback & Grading.
* **Bottom Section (Subject Matter - Content):**
* "Subject Matter (Content)"
* Icons representing: Math, Physics, Chemistry, Earth Science, History, Art, Languages, Drawing.
* **Center:** "Foundation Model" - depicted as a globe with interconnected nodes.
* **Right Section (Tasks & Goals):**
* "Tasks & Goals"
* Icons and text representing:
* Assist Students: Identity, Inclinations, State, Skills, Motivation, Preferences.
* Assist Educators: Including both teachers and education tools & materials.
* Facilitate Learning: Tracking and Analyzing Progression & Performance.
* Facilitate Teaching: Modeling Cognition & enabling interaction adaptive teaching.
* Understand Subject Matter: In a diverse range of sciences & humanities.
### Detailed Analysis or Content Details
The diagram illustrates a flow of information. The "Multimodal Data Sources" (Pedagogy and Subject Matter) feed into the "Foundation Model," which then outputs to the "Tasks & Goals."
* **Pedagogy (Structure):** This section focuses on how learning is structured, with teaching materials and interaction methods.
* **Subject Matter (Content):** This section represents the content being taught, covering a range of subjects from STEM fields (Math, Physics, Chemistry, Earth Science) to humanities (History, Art, Languages, Drawing).
* **Foundation Model:** The central component, depicted as a globe, suggests a comprehensive and interconnected system.
* **Tasks & Goals:** This section outlines the desired outcomes of the learning process, including assisting students and educators, facilitating learning and teaching, and understanding subject matter.
The "Assist Students" goal lists specific attributes to be understood: Identity, Inclinations, State, Skills, Motivation, and Preferences.
### Key Observations
The diagram emphasizes the integration of various data sources (pedagogy and subject matter) into a central "Foundation Model" to achieve a range of learning goals. The use of icons makes the diagram visually appealing and easy to understand. The flow of information is clearly indicated by the arrows. The diagram does not contain any numerical data or specific measurements.
### Interpretation
The diagram represents a conceptual framework for a learning system powered by a Foundation Model. The model aims to leverage multimodal data sources – both the structure of pedagogy and the content of various subjects – to achieve specific tasks and goals related to student and educator support, learning facilitation, and subject matter understanding. The Foundation Model acts as a central processing unit, integrating information from diverse sources to personalize and optimize the learning experience. The emphasis on understanding student attributes (Identity, Inclinations, etc.) suggests a focus on personalized learning. The diagram highlights the potential of a unified model to address multiple aspects of the learning process, from content delivery to performance tracking. The diagram is a high-level overview and does not provide details on the specific algorithms or technologies used within the Foundation Model.
</details>
In the year 2000, the largest gathering of world leaders convened at the United Nations Millennial Summit to reflect on an ideal vision for the future. Delegates concluded that a primary focus should be education, declaring it 'a foundation for human fulfillment, peace, sustainable development, economic growth, decent work, gender equality and responsible global citizenship." This discussion was ultimately recodified into the United Nations Sustainable Development goal to 'ensure inclusive and quality education for all and promote lifelong learning" [United Nations General Assembly 2015]. However, providing high quality, inclusive education at a large scale poses difficult societal and economic challenges. The price of education per student is growing faster than economy-wide costs [Bowen 2012], limiting the resources available to support student learning. In the United States, one symptom is that private education debt held by students has reached $1.6 trillion, surpassing total credit card debt [Friedman 2020]. Considering the rising need to provide adult retraining, the gap between the demand for education and our ability to provide it is alarmingly large and has concerning achievement disparities across protected demographics.
With the advent of the digital age and the rapid growth in digital learning, computational approaches to education have shown promise in increasing the effectiveness of learners and teachers. Several core directions have emerged as potentially impactful applications of AI for education [Woolf et al. 2013], such as systems that can provide meaningful feedback to students [Malik et al. 2021], help teachers improve [Jensen et al. 2020; Demszky et al. 2021; Suresh et al.
2021], or even create personalised and adaptive learning experiences that tailor the learning process to individual students' needs and dispositions [Connor 2019].
Despite this potential, building technical solutions to effectively scale inclusively and quality of education has proven to be exceptionally difficult. One particular challenge is that existing work has focused on custom solutions to highly specific tasks for which large amounts of training data has to be collected from scratch. Due to the difficulty and cost of creating large datasets, using this approach to solve every educational task independently is fundamentally limited. Instead, is it possible to create general-purpose approaches that are reusable across various tasks and subjects?
Foundation models have already started to boost the performance of some specific flagship tasks in education. Recent examples include using MathBERT [Shen et al. 2021b] to power 'knowledge tracing" - the challenge of tracking a student's understanding over time given their past responses and the 'feedback challenge", where an algorithm has to interpret a student's answer to a structured open-ended task, such as a coding question [Wu et al. 2021e]. Can foundation models lead to even more transformative changes in this domain? And what are the known and imagined risks of foundation models applied to education? In this section, we first frame the conversation around the ethical considerations. We then ground our discussion in two concrete tasks: (1) understanding student misconceptions, and (2) improving student understanding through instruction.
## 3.3.1 Important concerns for centering foundation models in education research.
The future of AI for education is exciting, especially in the context of foundation models. However, we caution the reader to be especially thoughtful about the impact of any AI research applied to education. 44 The goal of education are deeply interwoven with complex, long term social impact. While we actively work to improve digital education, it is imperative that we put in substantial thought to try and imagine the complexities of any disruption in this space [Piech and Einstein 2020]. Ethical challenges range from issues such as data bias, legal constraints, and the impact of digital socialization. These issues are not unique to foundation models, but they are worth reflecting on regularly as research makes substantial progress in AI for education. Reflection on impact is especially important when research starts by asking 'what can new AI technology afford?"
Many of the issues in §5.6: ethics apply to education. For example, as in many other domains, small biases in foundation model training data could be hard to track down [Dixon et al. 2018; Bolukbasi et al. 2016], but have important implications for equity of educational access. Moreover, these systems may experience a high degree of 'feedback", where the collected data continually reinforces the model's decisions. This issue of bias goes beyond what data is collected and includes concerns over the applications that researchers choose to work on. Below, we discuss other education-specific issues. Many of the issues revolve around the question: 'who benefits?" and for whom is new technology created?
Removing teachers from the loop One of the goals of digital education, especially based on AI, is to increase the productivity of the learning experience so that more learning happens per unit time or unit cost. One can imagine that decision makers could use this increased productivity to remove human teachers from the loop. The long term implications of such decisions are hard to know a priori . Could interacting with an education system optimized to maximize 'learning' have adverse effects on socioemotional skill development? Could it create fewer opportunities for
44 In 2013, Facebook initiated Free Basics, a project to provide free internet to the world and thus spread opportunity and interconnection. Now, the United Nations Human Rights Council reports that, in Myanmar, Facebook's efforts to follow through on such aspirations without proper human moderation accelerated hate speech, instigated division, and incited offline violence in the Rohingya genocide. Free Basics now serves as a warning of the complexities of technological impact on society.
Fig. 16. The figure illustrates a system that embeds signals from various modalities (image, speech, sign, text) and languages into a universal feature space. Such a feature space allows ideas to be linked across modalities and languages. Pedagogically relevant link types include analogies (similarities across languages) and contrasts (distinct concepts across languages), both of which can occur in the same modality or across different modalities.
<details>
<summary>Image 16 Details</summary>
### Visual Description
\n
## Diagram: Multimodal Data to Common Embedding Space
### Overview
This diagram illustrates the process of converting multimodal data (Language, Vision, Sign Language, and Speech) into a common embedding space using a Foundation Model. The diagram shows the input modalities, the central Foundation Model, and the resulting representation in the Common Embedding Space, highlighting concepts of contrast and analogy both within and across modalities.
### Components/Axes
The diagram is segmented into three main sections:
1. **Multimodal Data (Left):** Contains four input modalities: Language, Vision, Sign Language, and Speech.
2. **Foundation Model (Center):** A central processing unit represented as a complex, colorful network.
3. **Common Embedding Space (Right):** Displays data points representing the output of the Foundation Model, categorized by contrast and analogy.
The diagram includes the following textual labels:
* "Multimodal Data"
* "Language"
* "Vision"
* "Sign Language"
* "Speech"
* "Foundation Model"
* "Common Embedding Space"
* "Contrast in the same Modality"
* "Analogy in the same Modality"
* "Analogy Across Modalities"
* "Contrast Across Modalities"
* "Four English"
* "Nine English"
* "Tisa Swahili"
* "Nane Swahili"
* Arabic text: "ثلاثة" (Translation: "Three")
### Detailed Analysis or Content Details
The diagram depicts a flow from left to right.
**Multimodal Data:**
* **Language:** Displays the Arabic word "ثلاثة" (Translation: "Three").
* **Vision:** Shows the numeral "3".
* **Sign Language:** Depicts a hand gesture.
* **Speech:** Represents a waveform.
**Foundation Model:**
* The Foundation Model is visually represented as a complex network of interconnected nodes, colored in shades of blue, purple, and white. It acts as a central processing unit, receiving input from all four modalities.
**Common Embedding Space:**
* This section displays data points clustered based on relationships.
* **Contrast in the same Modality:** Contains green triangles, red squares, and blue stars.
* **Analogy in the same Modality:** Contains red circles and dark red circles.
* **Analogy Across Modalities:** Contains blue squares and light blue squares.
* **Contrast Across Modalities:** Contains blue triangles and purple triangles.
* Below these clusters, the diagram states: "Contrasts & Analogies Across Subject Matters".
* Further below, it states: "Four English" and "Nine English" alongside "Tisa Swahili" and "Nane Swahili".
### Key Observations
* The diagram emphasizes the conversion of diverse data types into a unified representation.
* The use of color-coding and shapes in the Common Embedding Space suggests different categories of relationships (contrast vs. analogy, within vs. across modalities).
* The inclusion of both English and Swahili text indicates a focus on multilingual capabilities.
* The Arabic word "ثلاثة" and the numeral "3" are presented as examples of equivalent concepts across modalities.
### Interpretation
The diagram illustrates the core concept of multimodal learning, where a Foundation Model is used to create a shared embedding space for different data modalities. This allows the model to understand relationships between concepts expressed in different forms (text, images, sounds, gestures). The distinction between "contrast" and "analogy" highlights the model's ability to identify both differences and similarities. The inclusion of multiple languages suggests the model's potential for cross-lingual understanding. The numerical labels ("Four English", "Nine English", "Tisa Swahili", "Nane Swahili") likely represent the number of examples or data points used to demonstrate these relationships within the embedding space. The diagram suggests that the Foundation Model is capable of capturing semantic meaning across modalities and languages, enabling more sophisticated AI applications. The diagram is conceptual and does not provide specific numerical data or performance metrics. It is a high-level illustration of a system architecture.
</details>
interacting with others? Loneliness is on the rise in younger generations [Cigna 2018], and teachers are a modulating force for pressures that AI researchers might not envision.
Was this work done by the learner or a foundation model? Another challenge is how to effectively teach students who have access to foundation-model-based tools. For example, it will be much more complex for teachers to understand the extent of a student's contribution if the student worked together with a powerful generative model, or to regulate ineffective collaborations and detect plagiarism. Visual Studio has recently released GitHub CoPilot, an AI pair-programmer built upon GPT-3 [Chen et al. 2021f]. How will this change computer science education? Many challenges for beginner programmers might be trivial to CoPilot or its technical successors, which could undermine the learning experience for novices. It would be instructive to study other examples of technological advances that disrupted education for certain subjects, such as calculators in math classrooms and Google Translate in language courses, both of which now coexist with traditional instruction.
Privacy and security. One important ethical issue in the use of AI in education is highlighted by the strict legal guidelines concerning privacy in student work. For example, in the United States, student information is protected by the Family Education Rights and Privacy Act (FERPA). These laws and regulations are especially important for children under 13, who have their data privacy and security additionally protected by the Children's Online Privacy Protection Act. Among other things, FERPA limits teachers from sharing personally identifiable student work. This could directly impact initiatives to share data used both for training and for evaluating foundation models. Moreover, there is an open question as to whether the weights of a foundation model could somehow leak the (possibly private) data it was trained upon [Nasr et al. 2018; Song et al. 2017]. These issues, and their corresponding approaches, are similar to the challenges described in §3.1: healthcare.
This list is not exhaustive and the ethical considerations will vary from project to project.
## 3.3.2 Foundation models of student thought.
When building AI tools for inclusive, and joyful education, there are many tasks where foundation models could be useful. Many of those tasks require us to first understand the learners whom we are trying to help, especially in the context of open ended work.
What would it take for a foundation model to be able to reason about student understanding? It is easy to imagine a foundation model which has been adapted to answer a math question correctly, but it is less clear how to build a model that can diagnose mistakes in student understanding based on the student's answers. To explore this theme, we consider the case study of providing feedback to students who are working on open-ended tasks such as writing a short paragraph, drawing a physics diagram, or writing code. This 'feedback challenge' exemplifies how foundation models can be helpful off-the-shelf for learners, and also demonstrates open areas for foundation model research.
To effectively provide feedback to students, two central capabilities are required: (1) understanding the subject matter of the task (e.g., physics or coding), and (2) the diagnostic ability to ' notice ": a technical term in education for inferring why a student made a mistake. For typical student interactions in a typical classroom, there is not enough data for an AI model to learn, from scratch, both of these central capabilities. Even for massive courses with millions of students, supervised algorithms barely understand the complex student reasoning behind even short, four-line programs [Malik et al. 2021]. As such, the feedback task inherently requires a transfer of understanding from external data and experience.
Foundation models, as they currently exist, are directly helpful for the first of these capabilities: understanding a specific subject matter . For example, when learning to provide feedback on short programming questions, a foundation model such as GPT-3 can efficiently understand what fluent code looks like with a few examples. Some research in this direction has already started exploring foundation models that can quickly adapt to questions in new subject matter domains [Wu et al. 2021e; Condor et al. 2021]. Similarly, foundation models could also integrate multiple modes of information such as the text of a task's prompt, diagrams in the question, or even the content of a grading rubric provided to teaching assistants. This unified representational ability can help foundation models comprehend a subject matter through richer sources of information. As a concrete case study, many of these insights were leveraged as core components of an algorithm which was able to grade an introductory Computer Science midterm at Stanford University, with the same effectiveness as human teaching assistants [Wu et al. 2021e]. In this case, subject matter encoding was built on a foundation model that had been adapted on GitHub code and a corresponding small dataset for each question's subject matter. In general, we can imagine leveraging various sources of data to adapt foundation models to different subject matter. For example, math adaptation could use mathematical websites or textbooks [Shen et al. 2021b] or historical student answers on platforms such as Gradescope; spoken language understanding could leverage radio archives or podcasts; and domains like creative writing could look to large digital archives like Project Gutenberg.
In contrast to subject matter, adapting a foundation model to the task of mapping observed mistakes to flaws in a student's thought processes is much less well-explored. The ability for an instructor to 'notice' the reasons behind why a student makes a specific mistake is a critical component of the feedback challenge. Imagine, for example, a student learning two digit addition who answers the question 'what is 26 + 19?" with the response '315." Take a moment and try to guess why they gave that answer and what misconceptions they have. 45 . This ability to notice could be posed as an adaptation task for foundation models (§4.3: adaptation) or perhaps even as a reasoning task (§2.4: reasoning).
45 This student has made the common mistake of concatenating the results of adding the one's digit and ten's digit
While difficult, training an AI system to notice is an achievable goal. Across classrooms, and across learning tasks in a given domain, there are generalizable patterns in how students arrive at their answers. The labeled data that can directly be used for this adaptation task, such as instructorwritten feedback to student work in [Wu et al. 2021e], are often held privately by instructors in disparate datasets. However, publicly accessible data, such as StackOverflow interactions, might also be creatively used to adapt a foundation model to notice. Some research has also explored effective ways of extracting, from instructors, generative descriptions of how students make mistakes [Malik et al. 2021; Gulwani and Singh 2013] - these hand-written generative models could also be used to generate adaptation data to help foundation models diagnose student mistakes.
## 3.3.3 Foundation models for instruction.
Reasoning about student understanding is an essential step towards a second objective: provide inclusive, high quality instruction . Computational approaches to instruction focus on different tasks like content personalization [Connor 2019], question generation [Guo et al. 2016; Willis et al. 2019; Srivastava and Goodman 2021], adaptive curriculum design [Mandel et al. 2014; Doroudi et al. 2017], and predicting instructor intervention [Chandrasekaran and Kan 2019; Alrajhi et al. 2021]. In this subsection, we discuss how foundation models could be useful in the act of teaching students.
Since effective teaching requires reasoning about student understanding, the previous discussions on understanding subject matter and 'noticing' are extremely relevant. However, providing effective instruction requires an additional capability: that of understanding pedagogy [McKenzie 2003]. This encapsulates an effective understanding of techniques to guide a student, such as asking Socratic questions or providing analogies/contrasting cases; using encouraging or supportive language; tailoring the difficulty of questions to the student; and generating examples that are relevant to a student's interests and background.
How can foundation models be adapted to understand good pedagogy for instruction? One idea is to consider adaptation using data source where instruction is the primary role. For example, data from question answering forums like StackOverflow could potentially be used to build a tutor which can parrot common Socratic questions. Similarly, a foundation model adapted on encyclopedias such as Wikipedia might be able to give answers to student questions which are (often) factually correct. There are also public data sources like textbooks, lecture videos, lesson plans, and graded feedback that collectively contain important pedagogical behaviours which could be adapted by foundation models (Figure 15).
Another adaptation challenge for instruction based on foundation model is to learn how to speak to students like teachers. The language used by teachers is often different from the language used by the general population. Teachers are ideally trained to speak to students with respect and in a way that intentionally helps them form a positive identity with the subject being learned [Truax 2018]. Cautionary examples like Microsoft's 2016 Twitter bot 'Tay," a chatbot that started generating hate speech within 24 hours of being deployed live, show us the importance of explicitly accounting for this factor in education. To train a language model which is more heavily influenced by professional teachers in classrooms, we could perhaps adapt foundation models to data sources like lecture videos or recorded office hour videos.
The adaptation problem above is compounded by the fact that different education contexts vary significantly in the kind of language that would be appropriate: for example, effective instruction in a 5th-grade science class would look quite different from that in a college physics class, much less a college literature class. This presents technical challenges beyond what would be faced in typical NLP domain shift settings (e.g., question answering based on news articles vs. Reddit posts), as the foundation model would need to be fluidly adaptable in terms of its tone and language, and not just the factual content that it generates.
Beyond sound pedagogical techniques and instructional language, how might foundation models provide even more insightful forms of instruction? §2.1: language of this paper highlights the fact that remarkably complex language can be acquired by babies in a short amount of time. As the authors point out, a salient difference between foundation model training and human language acquisition is that 'human language is grounded to the real world: for example, a baby's caretakers point to objects while they talk about them." This same insight can also inspire ideas as to how foundation models can be used for generative education. Humans seem to learn well when presented with real-world analogies and contrasts which may be cross-cutting between their current context and past experiences. For example, when teaching sign language, an instructor might use an analogy such as "the hand shapes for the word 'morning' looks like the sun rising" or note that 'the hand shape you just made look very similar to another word, so let us focus on the differences." As another example, when teaching Swahili to a learner who already knows Arabic and English, an instructor could point out that the Swahili word for 8 (pronounced nane) is a 'false friend' that is phonetically similar to English word for 9 (pronounced nine). Foundation models that can integrate multi-modal data have the potential to make these kinds of rich analogies and comparisons that are typical in childhood language learning (Figure 16).
## 4 TECHNOLOGY
The technological foundations of foundation models give rise to the capabilities (§2: capabilities) that determine their potential. To understand the technology used in development, we consider the data (§4.6: data), model architectures (§4.1: modeling) and systems (§4.5: systems) used to train (§4.2: training), and further adapt, (§4.3: adaptation) these models alongside the theory (§4.10: theory) that should be developed to understand this paradigm. To then understand the resulting models, we discuss how to evaluate (§4.4: evaluation) and interpret (§4.11: interpretability) alongside the importance of robustness (§4.8: robustness), security and privacy (§4.7: security), and long-term AI safety (§4.9: ai-safety) for ensuring the reliability of these models when deployed in society (§5: society).
## 4.1 Modeling
Authors: Drew A. Hudson, Antoine Bosselut, Alex Tamkin, Omar Khattab, Jared Quincy Davis, Jiaxuan You, Trevor Gale
Fig. 17. The five key properties of a foundation model: expressivity - to flexibly capture and represent rich information; scalability -to efficiently consume large quantities of data; multimodality -to connect together various modalities and domains; memory capacity - to store the vast amount of accumulated knowledge; and compositionality - to generalize to new contexts, tasks and environments.
<details>
<summary>Image 17 Details</summary>
### Visual Description
\n
## Diagram: Foundation Model Architecture
### Overview
The image is a diagram illustrating the architecture of a Foundation Model, showcasing the flow of multimodal data through the model and into memory storage. It depicts the key components of the model and how they interact. The diagram uses icons and text labels to represent different data types and processes.
### Components/Axes
The diagram consists of three main sections: Input Data, Foundation Model, and Memory Storage.
* **Input Data:** Includes icons representing Text (open book), Images (picture frame with a sunburst), and Structured Data (database icon). These are collectively labeled "Multimodal data".
* **Foundation Model:** A rectangular block divided into three layers:
* "Expressive Network" (light blue) with an icon of interconnected nodes.
* "Scalable Computation" (yellow) with an icon of stacked blocks.
* "Compositional Representation" (pink) with an icon of various shapes.
* **Memory Storage:** A rectangular block labeled "Memory Storage" containing icons for:
* Embeddings (binary code)
* Documents (document icon)
* Graphs (network diagram)
* Tables (grid icon)
* **Flow:** Arrows indicate the flow of "Multimodal data" into the "Foundation Model" and then to "Retrieval & Update" (magnifying glass icon) which connects to "Memory Storage".
* **Global Icon:** A globe icon is positioned above the "Foundation Model", suggesting a broad scope or global knowledge.
### Detailed Analysis or Content Details
The diagram illustrates a process flow:
1. **Input:** Multimodal data (text, images, structured data) is fed into the Foundation Model.
2. **Processing:** The Foundation Model processes the data through three layers:
* **Expressive Network:** Likely handles complex relationships and patterns.
* **Scalable Computation:** Suggests efficient processing of large datasets.
* **Compositional Representation:** Implies the model breaks down data into meaningful components.
3. **Output:** The processed data is then sent to a "Retrieval & Update" process.
4. **Storage:** Finally, the data is stored in "Memory Storage" in various formats: embeddings, documents, graphs, and tables.
The diagram does not contain numerical data or specific values. It is a conceptual representation of a system architecture.
### Key Observations
The diagram emphasizes the model's ability to handle diverse data types ("Multimodal data") and its capacity for storing information in multiple formats. The layered structure of the Foundation Model suggests a hierarchical processing approach. The "Retrieval & Update" component indicates the model is not static and can be updated with new information.
### Interpretation
This diagram represents a modern approach to AI model design, focusing on versatility and scalability. The use of "Foundation Model" suggests a pre-trained model capable of adapting to various downstream tasks. The multimodal input capability is a key feature, allowing the model to learn from different data sources. The memory storage component highlights the importance of retaining and organizing knowledge for future use. The diagram suggests a system designed for continuous learning and improvement, as indicated by the "Retrieval & Update" process. The globe icon implies the model's knowledge base is extensive and potentially global in scope. The diagram is a high-level overview and does not delve into the specific algorithms or techniques used within each component.
</details>
The emerging paradigm of foundation models has attained impressive achievements in AI over the last few years, as models such as BERT [Devlin et al. 2019] shine at a wide spectrum of language understanding tasks: from textual classification and entailment to question answering and reading comprehension, while GPT-3 composes rich and fluent tales about unicorns [Brown et al. 2020] and DALL-E shows signs of visual creativity, generating from scratch strikingly-realistic pictures of avocado chairs [Ramesh et al. 2021].
These and other instances of recent foundation models not only achieve remarkable performance across a multitude of diverse downstream tasks and applications [Rajpurkar et al. 2018; Wang et al. 2019a], but also manifest noteworthy behaviors of interpretability [Karras et al. 2020], robustness [Devlin et al. 2019], controllability [Patashnik et al. 2021] and generalization [Brown et al. 2020]. What does it take for a model to demonstrate such qualities? What architectures are capable of consuming large quantities of potentially multimodal information and translate them into rich knowledge of the world? And overall, what desirable properties should a network possess to give rise to a foundation model?
Here, we identify and discuss five such properties, spanning expressivity , scalability , multimodality , memory capacity , and compositionality , that we believe are essential for a foundation model in order to: (1) distill and accumulate knowledge from various sources and domains, (2) organize it in an effective and scalable representation, and (3) flexibly generalize it towards novel contexts. For each of these properties, we motivate their necessity, provide examples of contemporary models that incorporate them, and explore key challenges and promising avenues for future research and development. See Figure 17 for an overview diagram.
## 4.1.1 Expressivity.
Expressivity concerns with the theoretical and practical capacity of a network to model the data distribution it is trained over and represent it in a flexible manner. Prior works have proposed formal expressivity measures to characterize the complexity of functions a network can compute, or more precisely, approximate, which is essentially affected by its depth, width, connectivity, and structural patterns [Raghu et al. 2017].
As the No Free Lunch theorem suggests, there is no single model or algorithm that suits best for all cases [Wolpert and Macready 1997], and so, for our purposes, we are particularly interested in identifying which models could effectively capture the facets of natural information , such as human language or real-world images [Goodfellow et al. 2016]. These modalities are either continuous (as in vision) or discrete (as in language), are distinctly hierarchical and high-dimensional, and present a complex set of relations and interactions among their constituent elements, whether these are pixels, words or physical objects.
Indeed, recent breakthroughs in generative modeling provide strong evidence for the high expressivity of neural networks, as they successfully express distributions of textual [Brown et al. 2020; Devlin et al. 2019; Lieber et al. 2021; Wang and Komatsuzaki 2021], auditory [van den Oord et al. 2016], and visual [Karras et al. 2020; Brock et al. 2018] domains, and generate samples of high fidelity, diversity and realism.
Inductive Biases. Much of the success of neural networks over the last decade in modeling natural data is owed to the networks' high depths , as could be roughly measured by the number of stacked non-linear layers they are composed of, or the number of computational steps they take during their chain-of-reasoning. Great depths play a crucial role in enhancing networks' expressivity, allowing them to form powerful hierarchical and distributed representations that could generalize from the training data to new unseen examples [He et al. 2016b; Levine et al. 2020].
The universal approximation theorem [Lu et al. 2019b] indeed states that even simple multilayer perceptrons (MLPs) can represent a broad set of functions, while different inductive biases , as those implemented in Recurrent Neural Networks (RNNs) or Convolutional Neural Networks (CNNs) [Goodfellow et al. 2016], can improve the learning efficiency and enhance the capacity of a given network to model different forms of information: sequential data, common to language, speech and time-series, for the former, or spatially-invariant information, as in images or videos, for the latter.
Transformer Networks & Attention. Meanwhile, transformer networks [Vaswani et al. 2017], introduced more recently, demonstrate the importance of capturing long-range dependencies and pairwise or higher-order interactions between elements. They build on the self-attention mechanism [Vaswani et al. 2017; Bahdanau et al. 2014] that enables shorter computation paths and provides direct means to compare elements far-across the input data (such as a pronoun and its antecedent in a sentence, or two sentences that refer to the same topic).
From another perspective, the multiplicative interaction embodied in both attention as well as gating structures (as in LSTMs [Hochreiter and Schmidhuber 1997] or Mixture-of-Experts [Shazeer et al. 2017]) offers a more flexible alternative to the rigid fixed-weight computation of MLPs and CNNs, dynamically adapting the computation to the input at hand. This proves especially useful for language modeling, where, for instance, given a sentence like 'She ate the ice-cream with the X", while a feed-forward network would always process it in the very same manner, an attentionbased model could adapt its computation to the input - updating the contextual representation of the word 'ate' if the prepositional phrase (PP) attachment X is 'spoon', or instead link it to the 'ice-cream" if X refers e.g., to 'strawberries" [Zavrel et al. 1997].
General-Purpose Computation. A final notable advantage of attention over prior architectures stems from its stronger generality , where it is not strongly tied to a particular task or domain, as is the case for the local receptive field of convolution or the sequential assumption of recurrent networks, both reflecting inherent properties specific to the vision and language modalities respectively. We hypothesize that the general-purpose nature of attention and transformers contributes to their broad applicability for a wide range of research problems and applications [Liu et al. 2019; Dosovitskiy et al. 2020; Hudson and Zitnick 2021].
This contrast captures a more general trade-off between task-specialization and expressivity : models with stronger structural priors can leverage them to improve sample efficiency on the particular tasks that benefit from these assumptions; while conversely, models that integrate weaker inductive biases learn more slowly, but can in turn scale to higher volumes of data and adapt to a diverse set of domains, since they do not rely on restrictive or task-specific suppositions. As both data and compute turn more accessible, we observe that the exploration of models with a minimal set of inductive biases that can 'let the data speak for itself" seems to serve as a more promising approach for future research in the field.
Challenges & Future Directions. Notwithstanding the stellar progress and accomplishments of neural networks in general, and foundation models in particular, in terms of expressivity, notable challenges still remain. Leading approaches [Choromanski et al. 2020; Dosovitskiy et al. 2020] keep struggling with modeling of extremely long-range dependencies, such as those occurring in books, movies, or even DNA sequences, which may be attributed to the quadratic computation of contemporary transformer-based approaches [Wang et al. 2020c; Lin et al. 2021].
This challenge essentially reflects the trade-off between efficiency and expressivity : where explicit modeling of long-distance interactions through short and direct computation paths improves expressivity on the one hand, but comes at the expense of scalability due to computation entailed by the increased connectivity on the other [Child et al. 2019; Kitaev et al. 2020; Choromanski et al. 2020]. Models such as the GANformer [Hudson and Zitnick 2021] and the Perceiver [Jaegle et al. 2021b,a] explore ways to balance these two properties and propose transformers with linear complexity that rely on bipartite or bottleneck attention, so to improve computational efficiency while maintaining high-expressivity. We believe that identifying an effective equilibrium between these two objectives offers an interesting avenue for future research.
Another important research direction relates to the expansion of foundation models, which, so far, have mainly focused on the language domain [Peters et al. 2018; Devlin et al. 2019; Brown et al. 2020], to different modalities, such as the structural [Scarselli et al. 2008; Veličković et al. 2017] and perceptual [Tolstikhin et al. 2021; Jaegle et al. 2021b; Tan and Le 2021], each involving a unique set of associated challenges. Likewise, we believe that exploring architectures for reasoning (§2.4: reasoning), which demands iterative computation chains and interaction with symbolic information, constitutes a valuable goal for future foundation models research.
## 4.1.2 Scalability.
Closely connected to model's expressivity is the notion of scalability. As rich data from varied sources becomes more readily available, and computational resources get stronger and more efficient (§4.5: systems), we should look for ways to match this rate of progress and harness it to improve AI competency and versatility. For foundation models to effectively fit the complex and high-dimensional distribution of images or text, they should thereby be scalable across all dimensions: including both models' depth and width as well as their training time, number of parameters, and amount of data they could process.
Optimization. Specifically, foundation models should both be: (1) easy-to-train (§4.2: training), by being resilient to noise or imperfections in the data, and robust against instabilities like vanishing [Helfrich et al. 2018; Glorot and Bengio 2010] or exploding gradients [Hochreiter and Schmidhuber 1997; Nair and Hinton 2010], but also (2) easy-to-adapt (§4.3: adaptation), by overcoming phenomena of catastrophic forgetting [Kirkpatrick et al. 2017] and supporting few-shot learning [Sung et al. 2018]. We are still in the early days of understanding the principles that drive the scalability of learning algorithms, and while recent works have started to shed some light on these themes [Liu et al. 2020c; Kuditipudi et al. 2019; Nakkiran et al. 2019], much work remains to be done.
Hardware Compatibility. Moving beyond aspects of robustness and optimization , foundation models should also be practically efficient (§4.5: systems), and take advantage of contemporary and future hardware [Hooker 2020]. One example of that is parallelizablity , an important property that characterizes the computation supported by GPUs. Indeed, much of the transformers' great success over the previously dominating recurrent approach was driven by their higher degree of parallelism.
Looking forward, given the fast-pace progress of systems development, we should further ensure that models are designed to co-adapt to future hardware advances. Consequently, foundation models should ideally be amenable to schemes such as distributed training, which is gaining popularity, as is the case for e.g., Mixture-of-Experts, and possibly leverage properties such as sparsity of the computation or representation, as is the case for the Longformer [Beltagy et al. 2020], BigBird [Zaheer et al. 2020], and Sparse Transformer [Child et al. 2019] approaches, and which likely will become more central in future hardware and processors.
## 4.1.3 Multimodality.
Traditionally, the fields of computer vision, robotics, and NLP have made progress in an independent manner, with separate communities developing specific approaches suitable for each modality. A conducive consequence the rise of deep learning has brought about was the bridges it helped forming among the various communities and research areas within AI, as seemingly different problems could now be tackled by closely-related approaches, and studies of originally remote topics began converging to a common ground. This breakthrough opened up a new range of possibilities, fostering pioneering exploration into the theme of multimodality, encompassing areas as varied as language grounding [Lynch and Sermanet 2020], visual semantics [Conser et al. 2019], embodied environments [Savva et al. 2019a] and interactive agents [Gray et al. 2019].
Essentially, multimodality serves as a key component of intelligence, and is a crucial factor for the development of both thorough and broad comprehension of the world. Concretely, language learning is more effective when occurring in a grounded environment rather than in a vacuum. And inversely, from the vision perspective, language encourages the emergence of abstractions that link between low-level perceptual signals and statistics to semantic concepts of objects, properties, agents and motivations, thereby enriching and elevating visual representations.
In light of these observations, we argue that foundation models should ideally connect together the different modalities, distill their embodied information into a shared multifaceted representation, and capture the full range of inter-connections and relations among them so as to furnish a wide range of capabilities (see §2.1: language, §2.2: vision,§2.3: robotics, §2.4: reasoning).
Generality and Specialization. An important design choice for multimodal foundation models is the degree of specialization , or the structural sharing between the modules responsible for each modality. Naturally, data of different domains exhibits diverse kinds of structures and properties where, for instance, language is discrete while vision is continuous. At first sight, this variation hints that specialized inductive biases tailored for each modality could be of aid. Yet, as training scales
upwards and models are provided with the opportunity to base their learning less on structural priors and more on the data itself, general approaches that maintain only a handful of broad general assumptions prove in fact a lot more successful than task-specific alternatives. And so, as corroborated by recent success of general-purpose models like transformers across different modalities - both linguistic [Liu et al. 2019; Lan et al. 2019] and visual [Dosovitskiy et al. 2020; Hudson and Zitnick 2021], we see that generality is critical for improving AI capabilities.
Multimodal Interactions. Another key consideration for multimodal models relates to weight sharing : do the various modalities benefit from using the same or different parameters for their respective components? Prior works have shown that fruitful transfer could certainly occur across modalities, but the ideal degree of sharing remains unclear, so is the existence of principled ways for discovering it.
Finally, a major design question concerns with the forms of the multimodal interactions supported by the model, which vary widely between concrete cases and examples: Cross-modal or late-fusion models such as ConVIRT [Zhang et al. 2020a] and CLIP [Radford et al. 2021] maintain fully separate encoders for each data source, and compare their spaces only at the ultimate computation stage, using e.g., a simple dot product. Meanwhile, early-fusion models, such as ViLBERT [Lu et al. 2019a; Cho et al. 2021], jointly reason over multiple modalities necessary for tasks of visual reasoning and question answering. Identifying the optimal stage and form for merging the respective vector spaces [Nagrani et al. 2021] remains an open research question.
Overall, while there seems to be a consensus within the community about the importance of multimodality, models that go beyond shallow alignment of vision and language are yet to exist, and the theme of grounded language learning in embodied environments still has much room for exploration.
## 4.1.4 Memory.
So far, we have discussed the foundation models' goal to gather and accumulate information from varied modalities at large scales. This knowledge encompasses both broad understanding of the world as well as specific mastery of niche subjects or particular facts. Representing such a large body of learned information is by no means trivial, and is leading to interesting questions about effective mechanisms for access , storage , retrieval and manipulation of particular items or memories .
Explicit Storage. An important design principle that could achieve these desiderata is to separate out computation from memory [Weston et al. 2014; Graves et al. 2016; Hudson and Manning 2018, 2019a] in order to enhance models' ability to transfer knowledge by applying previously acquired abstract skills to new concrete settings.
In this context, it is important to distinguish between explicit facts - that can be stored in an external memory storage, and implicit knowledge -that is reflected through the networks' trainable weights. Such decoupling of explicit and implicit knowledge enjoys multiple advantages compared to the alternative of implicitly encoding all information together through the network weights. The separation mitigates the inflation in models' size and number of parameters needed to store the growing quantities of knowledge [Guu et al. 2020], improves models' trust and reliability by increasing their knowledge provenance [Cheney et al. 2009], and most notably, is key for memory update, manipulation or adaptation [Lewis et al. 2020b] (§4.3: adaptation), which could in turn enable generalization to novel contexts and downstream tasks.
Indeed, disentanglement between memory and computation has been a recurring goal in deep learning and NLP research over the last years, including models such as Memory Networks [Weston et al. 2014; Sukhbaatar et al. 2015], the Neural Turing Machine [Graves et al. 2014, 2016], the Neural State Machine [Hudson and Manning 2019a], and MAC [Hudson and Manning 2018]. Furthermore,
using key-value structures [Miller et al. 2016] for accessing external memories has been shown to be very effective for modeling long-term dependencies [Henaff et al. 2016; Bosselut et al. 2018; Lample et al. 2019]. Transformers, the celebrated architecture underlying most foundation models to date, likewise exhibits operations that involve key-value memory-access and computation among the contextual word representations they gradually build [Geva et al. 2020].
Information Retrieval. Once a model completes gathering the information after training, there are multiple ways to retrieve particular facts or memories necessary for downstream applications and tasks. Some employ explicit prompting techniques that query the model's knowledge through input sequences [Petroni et al. 2019; Kassner et al. 2021; Jiang et al. 2020] while other approaches involve implicit recollection and reshaping of the prior knowledge through an adaption phase [Bosselut et al. 2019; Hwang et al. 2021]. A third category of methods goes a step further and combines neural-based computation with symbolic aggregation and retrieval of information from either unstructured textual repositories [Karpukhin et al. 2020; Lewis et al. 2020b; Khattab et al. 2020] or even structured resources such as knowledge graphs [Zhang et al. 2019a; Peters et al. 2019; Liu et al. 2020e; Verga et al. 2020; Yasunaga et al. 2021].
However, there is trade-off between the strong memorization skills offered by retrieval mechanisms on the one hand and the richer representations learned when there is an information bottleneck on the other. Indeed, over-reliance on retrieval reduces the opportunities to learn how to represent information in compact and abstract manners, distill key insights and concepts out of the vast amounts of input information the model is exposed too, and, basically, separate the wheat from the chaff. For instance, the in-context learning abilities of GPT-3 possibly emerge as a by-product of enforcing the network to represent the input sequential data through its bounded memory architecture [Brown et al. 2020]. Overall, While they certainly have some merits [Guu et al. 2020], models that rely on external retrieval mechanisms may not learn to generalize as effectively as bounded, compact and abstract representations.
Knowledge Manipulation. Finally, when considering large-scale learning over long durations, it is crucial to note the dynamic nature of knowledge, where facts' correctness and validity can change over time as the world keeps evolving - and what was true or relevant yesterday may not be so tomorrow. It is therefore crucial for a model to represent its knowledge in a manner that supports efficient update or manipulation of facts as part of its lifelong learning.
## 4.1.5 Compositionality.
Compositionality can be defined as the principle according to which the meaning of the whole is derived from the meaning of its constituent parts , and the rules applied to combine them [Janssen and Partee 1997; Bottou 2014]. It is a crucial ingredient of human intelligence [Lake et al. 2017], underlying our capabilities to plan, reason and learn readily and efficiently from a handful of examples. Compositionality may hold the key to achieve out-of-distribution - or specifically combinatorial generalization . Drawing on classic ideas from symbolic AI, it encourages and enhances desirable properties within neural networks, such as interpretability, controllability and dataefficiency [Lake et al. 2017], and can take different forms, characterizing variety of elements:
Model. Compositionality can be reflected at the model level, in terms of its architectural properties, structure, and degree of modularity - which can increase training and inference efficiency of large neural models [Shazeer et al. 2017]. It also links to themes of interpretability and multimodality , as it relates to the interfaces between the different modules the model is composed of, what modes of interactions they employ, and how transparent they are.
Computation. Models such as Module Networks [Andreas et al. 2016] and Mixture-of-Experts [Shazeer et al. 2017] go further along this direction, exhibiting not only structural modularity, but also compositional computation , supported by the specialization of sub-networks to different operations, in a manner that adapts and tailors the model behavior to the input at hand. While some methods rely on concatenation of hand-engineered modules [Andreas et al. 2016], alternative approaches enable the network specialization to naturally emerge through learning [Shazeer et al. 2017]. Other models, such as MAC [Hudson and Manning 2018] and Dynamic Memory Networks [Xiong et al. 2016] perform an explicit iterative computation , where a given task is decomposed into multiple reasoning steps, performed one by one, manifesting sequential progression from a set of initial facts to novel inferences and conclusions.
Training & Data. Not only can the model or its computation be compositional, but so can be the data or training processes too [Andreas 2020]. Instead of training one model over a complete dataset, one could split, or decompose it into subsets, train different models on each one independently, and ultimately recombine them at test time through various ensemble techniques [Dietterich 2000]. Such approaches could have far-reaching implications on the training and deployment procedures of foundation models, in both practical and even societal regards.
Representation. We have discussed compositionality of different elements, such as the model, the computation, the training schemes or the data. But most notably, the learned representation itself, which emerges over the course of the model training and adaptation, can also be compositional [Andreas 2019]. Indeed, a promising manner to represent knowledge is through structured, potentially graph-based, object-oriented representations [Zhang et al. 2019a; Wang et al. 2021a], that center around identifying entities and event nodes and forming connections, analogies and relation edges among them. It reflects a natural way to organize information about the world, where inputs from different modalities can be channeled and aggregated around semantic multi-faceted concepts. Such representations could support multi-hop reasoning and inference [Washington et al. 1995; Sun et al. 2020b; Yu et al. 2020c], and potentially also enable stronger out-of-distribution generalization through recombination .
However, compositionality can also hinder the expressivity of the representation, and impede its capacity to account for idiosyncrasies, exceptions, and contextual correlations [Misra et al. 2017a]. In other words, the whole can sometimes be greater than the sum of its parts, where for instance, red wine is not the same as red onion . But while many approaches that have dominated over the last decade tend to focus mostly on one end of the spectrum, and learn monolithic distributed representations, we believe that exploring manners to reach a better balance between contextuality and compositionality is a promising avenue for future research.
## 4.1.6 Summary.
Wehaveintroduced five properties that we believe are essential for the next generation of foundation models, in order to effectively distill the large amounts of information around us so to successfully address downstream tasks: expressivity - to flexibly capture and assimilate real-world information, scalability -to adeptly handle high volumes of high-dimensional data, multimodality -to consume, process and potentially produce content from different sources and domains, memory capacity to effectively store and retrieve the acquired knowledge, and finally, compositionality , to foster successful generalization to novel tasks, settings and environments. We believe that the realization of the full potential of foundation models, as is envisioned and discussed in detail throughout this report, will rely on research of new architectural and modeling advances to fulfill these desiderata.
## 4.2 Training
## Authors: Alex Tamkin
Training objectives are mathematical functions describing how to transform a model architecture and large amount of broad data into a foundation model. For example, GPT-3 was trained with a language modeling objective, which rewards the model for predicting the next word correctly [Shannon 1948]. We begin by laying out some goals of these training approaches, describe important design trade-offs in current approaches, and outline important goals for the path ahead.
## 4.2.1 Goals of training objectives.
Here we outline some key goals for training algorithms in light of the recent rapid progress in these methods and models. 46
Leveraging broad data. The rise of self-supervised learning algorithms has unlocked the power of internet-scale datasets which would be intractable to annotate by hand. This kind of broad data comes in many forms, including images, audio recordings, and video (§2.2: vision); robotic and sensor data (§2.3: robotics); and text, either in isolation or paired with other modalities like images (§2.1: language). Because this data lacks external annotations, a major focus for researchers is designing bespoke self-supervised algorithms that leverage the unique structure within each kind of data to produce a training signal for a foundation model.
Domain completeness. An important goal for foundation model training algorithms is to be domain complete , in the sense that solving the training task requires capabilities that are broadly useful for downstream tasks in the domain (see §2.1: language, §2.2: vision, §2.3: robotics). This property is crucial for the generality of a foundation model. For example, language modeling may require models to acquire capabilities as wide-ranging as coreference, sentiment and translation as the model learns to predict the next word in a document. In contrast, a supervised learning task like sentiment classification may lead to a more narrow set of capabilities (see §2.1: language). As important as this quality is, it is not obvious a priori what tasks will result in a domain complete capabilities, or even how to evaluate the full breadth of a model's capabilities (see §4.4: evaluation and §4.10: theory).
Scaling and compute efficiency. Procedures for training foundation models must reliably convert data, a model architecture, and compute into a broadly capable model. To maximize the capability of a foundation model, we can identify the bottlenecks to this process and propose new training algorithms which remove them. The rise of self-supervised algorithms has made model size and compute resources increasingly salient bottlenecks [Kaplan et al. 2020; Henighan et al. 2020], leading to a shift where models are evaluated not solely on their capabilities but rather on the amount and kind of compute needed to reach those capabilities (§4.4: evaluation). The efficiency of training objectives can vary tremendously, 47 laying in sharp relief how important the design of a training approach is to the emergence of powerful capabilities given a fixed compute budget. Thus, a major goal for training researchers is to design training objectives with a richer training signal, resulting in models which learn faster and attain stronger capabilities. 48 One force aiding this development is the surprising predictability of how capabilities scale with different kinds of architectures, data
46 We use 'training" instead of pretraining to emphasize the primacy of the foundation model itself, and because some methods for adapting foundation models to downstream tasks do not involve any later stage of training.
47 e.g., 4x for ELECTRA [Clark et al. 2020] vs BERT [Devlin et al. 2019], 12x for contrastive vs generative approaches to CLIP training [Radford et al. 2021]
48 Of course, a key goal for computer systems designers is to alleviate compute as a bottleneck for training (see §4.5: systems) And the choice of a training method is ultimately also constrained by the availability of diverse, high-quality data
sizes, and compute [Hestness et al. 2017; Kaplan et al. 2020], a striking phenomenon which enables model developers to make choices based on clearer trends instead of more costly random searches.
## 4.2.2 Design trade-offs in current SSL methods.
Current self-supervised learning (SSL) methods for training foundation models are diverse, but what unites them is that they produce prediction problems from unlabeled data without the need for human annotators. SSL objectives manufacture a rich training signal from this data through carefully-designed constraints, either on the data itself (e.g., redacting or noising) or on the way the model is able to represent or process the data (e.g., latent bottlenecks). At some level, these constraints 'bake in" the kinds of capabilities desired when adapting models to downstream tasks (§4.3: adaptation). 49
Here, we describe three important design choices that current models explore, along with their respective tradeoffs in terms of their resulting capabilities.
At what level of abstraction should we model? A fundamental question is what the input representation of a foundation model should be. One option is to model the input at the level of raw bytes. However, this high dimensionality may cause the model to focus on predicting less semantic aspects of the input, 50 slowing the rate at which it acquires more generally-useful capabilities. These approaches also become intractable when training models like transformers [Vaswani et al. 2017] whose compute costs grow quadratically with the input size. 51 Another option is to use domain knowledge to reduce the input space of a model - such strategies include patch embeddings [Dosovitskiy et al. 2020] as well as fixed or learned tokenization [Schuster and Nakajima 2012; Sennrich et al. 2016; Kudo and Richardson 2018; van den Oord et al. 2017; Ramesh et al. 2021]. These methods may alleviate some challenges facing generative approaches, but have the trade-off that they may jettison possibly-useful information in the input. 52 The choice of a continuous vs discrete input also has trade-offs for adaptation (§4.3: adaptation); more work is needed to capture the benefits of both approaches.
Generative vs discriminative models Generative training approaches are conceptually elegant yet powerful - they train models to learn joint or conditional distributions over training inputs. Two major families of generative foundation models include autoregressive foundation models [van den Oord et al. 2016; Radford and Narasimhan 2018; Chen et al. 2020d; Yang et al. 2019; Ramesh et al. 2021], which generate inputs piece by piece, and denoising foundation models [Devlin et al. 2019; Raffel et al. 2019] which corrupt and then recover the inputs. The specific kind of generation performed in the training process determines what kind of interactivity is available
(§4.6: data), which continues to be a major challenge for many domains, including robotics (§2.3: robotics) and low-resource languages (§2.1: language)
49 For example, the causal language modeling objective used to train GPT-3 [Brown et al. 2020] enabled conditioning it via prefixes. And the color jitter augmentations used during contrastive learning [Chen et al. 2020c] encourage invariance to properties not thought to be useful for downstream tasks. Better understanding how the particular choice and structure
of these constraints influences the capabilities acquired by the model is an important area for future work (§4.10: theory). 50 e.g., blades of grass, audio compression artifacts, or spellings of words
51 See §2.2: vision and §4.1: modeling for discussions of training costs for high-dimensional sequences, such as images and video
52 For example, tokenizing text may make it harder to learn rhymes, puns, or other tasks that benefit from character-level information [Branwen 2020]
during adaptation 53 (see §2.5: interaction and §4.3: adaptation), and future models may enable an even richer set of interactions. 54
While generative training approaches have their benefits, several discriminative approaches have also recently gained traction. These methods do not enable generation-based interaction, yet they may enable more efficient learning for classification- or regression-based tasks in high-dimensional continuous settings like images, audio, and video. Most of these methods output vectors for (parts of) inputs, which are trained to be similar for different 'views' of an input [Wu et al. 2018; van den Oord et al. 2018; Chen et al. 2020c; He et al. 2020; Grill et al. 2020; Caron et al. 2021; Zhang et al. 2020a; Radford et al. 2021] or used to predict whether parts of inputs are real or fake [Clark et al. 2020; Iida et al. 2021]. Better understanding the trade-offs between generative and discriminative training, as well as capturing the best of both approaches, remain interesting avenues for future study.
Capturing multimodal relationships. Another increasingly important research area is capturing the relationships between multiple kinds of data. What this means may differ based on the context and the goals of a modeler. For example, CLIP [Radford et al. 2021] and ViLBERT [Lu et al. 2019a] are both multimodal vision-language, but differ in the precise way they are multimodal. 55 The former encodes images and text separately into vectors, enabling users who have examples from a single modality to retrieve, score, or classify examples from the other modality. The second processes images and text jointly at an early stage of the model, enabling downstream applications like visual question answering where reasoning over pairs of related images and text (e.g., images and questions about them) are provided. Multimodal foundation models remain a nascent research area; much is still unexplored about the different ways a model can be multimodal as well as better understanding the capabilities these additional modalities bring.
## 4.2.3 Paths forward.
We close with some important goals for the future of foundation model training.
Out-of-the-box SSL Right now, SSL objectives are highly domain-specific: different methods currently prevail in natural language processing, computer vision, and speech processing. This has two major disadvantages: First, these different techniques make it challenging to grasp the common threads and scientific principles underlying why each of these methods work. Second, this domain-specificity requires developing new foundation model training methods from scratch for each new field, including medical, scientific, and new multimodal settings. A more general objective for efficiently training foundation models on any kind of data would represent a significant milestone for the foundation model training community [Tamkin et al. 2021b].
Obtaining a rich training signal It is clear that not all training objectives are made equal - some are radically more efficient than others, translating into far more capable foundation models for a given compute budget. Are there training methods orders of magnitude more efficient than those currently known? If so, how can we find them? These investigations will be shaped by many forces, including what future software and hardware advances (§4.5: systems) make possible. We also need not view data (§4.6: data) and training algorithms as independent factors: not only does the
53 For example, autoregressive models like GPT-3 enable prefix-based conditioning, while denoising models like T5 or BERT facilitate the use of bidirectional context to replace arbitrary-length spans or fix typos.
54 Other kinds of generative approaches less studied in a foundation modeling context include diffusion and score-based models [Sohl-Dickstein et al. 2015; Song and Ermon 2019; Ho et al. 2020], VAEs [Kingma and Welling 2014], flow models [Dinh et al. 2015; Kingma and Dhariwal 2018], and GANs [Goodfellow et al. 2014] - it remains to be seen whether these or other future approaches can also enable learning as diverse variety of capabilities as autoregressive or denoising approaches.
55 See §2.2: vision and §2.1: language for more discussion of multimodality in vision and language specifically
quality and availability of the data influence the training signal, 56 but the training algorithm itself could adaptively seek out or construct richer training examples as the model improves to accelerate learning [Tamkin et al. 2021c].
Goal-directed training of foundation models. Adaptation methods such as prompting (§4.3: adaptation) draw on emergent properties that result almost as an afterthought of training. Can we train foundation models where the ability to understand and reliably carry out goals in a complex world is part of the model's training objective? A focus on developing general capabilities distinguishes this direction from the goal of adapting an existing foundation model to a specific task via reinforcement learning (e.g., Stiennon et al. [2020]). Instead, one might imagine more sophisticated versions of current methods which acquire a diverse range of real-world capabilities from raw online [Klyubin et al. 2005; Singh et al. 2005; Salge et al. 2013; Shakir Mohamed 2015; Florensa et al. 2017; Pathak et al. 2017; Haber et al. 2018] or offline [Precup et al. 2000; Lange et al. 2012; Ajay et al. 2021; Yang and Nachum 2021; Schwarzer et al. 2021] interactions, without the need for human annotations or task construction. Such methods might use techniques quite similar to existing SSL algorithms: e.g., training sequence models in goal-directed contexts where they can be directly asked to carry out certain tasks via conditioning (e.g., UDRL [Schmidhuber 2019; Srivastava et al. 2019] or Decision Transformer [Chen et al. 2021b]; also see §2.3: robotics). The complex behaviors that have already emerged in simple interactive environments [Baker et al. 2020] suggest multitask, multiagent, and multimodal goal-directed training of foundation models as an interesting avenue for future study.
56 Including any undesirable or biased capabilities (§5.1: fairness)
## 4.3 Adaptation
Authors: Xiang Lisa Li*, Eric Mitchell*, Sang Michael Xie, Xuechen Li, Tatsunori Hashimoto
Fig. 18. During adaptation, a foundation model is converted into an adapted model (bottom row) in order to reflect updated information, desired behaviors, or deployment constraints.
<details>
<summary>Image 18 Details</summary>
### Visual Description
## Diagram: Model Adaptation Strategies
### Overview
The image is a diagram illustrating three strategies for adapting a core model: Task Specialization, Model Patching, and Temporal Adaptation. Each strategy takes the core model (represented by a purple sphere with radiating lines) as input and transforms it into a specialized model. The diagram visually represents the input-process-output relationship for each adaptation method.
### Components/Axes
The diagram is divided into three main sections, each representing a different adaptation strategy. Each section contains:
* **Input:** The core model (purple sphere) and specific data relevant to the strategy.
* **Process:** A rectangular box representing the adaptation process.
* **Output:** A specialized model resulting from the adaptation.
* **Arrows:** Illustrate the flow of information from the core model to the specialized model.
The three sections are:
1. **Task Specialization** (left, light blue background)
2. **Model Patching** (center, light pink background)
3. **Temporal Adaptation** (right, light orange background)
### Detailed Analysis or Content Details
**1. Task Specialization:**
* **Input:** Core model (purple sphere) and example data:
* Question: "How discovered the Penicillin?"
* Answer: "Alexander Fleming"
* Image: A photograph of two purple flowers in a field.
* Caption: "Two purple flowers in a field…"
* Code: `def sum_func(a,b): print("hello") return a + b + 1`
* **Process:** "Task Specialization" is written inside a light blue rectangle.
* **Output:** Three specialized models:
* Question Answering (icon: a question mark)
* Image Captioning (icon: a camera)
* Code Completion (icon: curly braces)
* **Color Association:** The arrows from the core model to the specialized models are colored purple, green, and blue respectively.
**2. Model Patching:**
* **Input:** Core model (purple sphere) and example data:
* Copyright Warning: "Mr. and Mrs. Dursley of number four, Privet Drive, were proud…" (with a copyright symbol)
* Model Errors: "UK is a member of EU"
* Model: True
* Correction: False
* **Process:** "Model Patching" is written inside a light pink rectangle.
* **Output:** Two specialized models:
* Compliant Model (icon: a robot)
* Corrected Model (icon: a document with a checkmark)
* **Color Association:** The arrows from the core model to the specialized models are colored pink and red respectively.
**3. Temporal Adaptation:**
* **Input:** Core model (purple sphere) and example data:
* Recent News Articles (icon: a newspaper)
* **Process:** "Temporal Adaptation" is written inside a light orange rectangle.
* **Output:** One specialized model:
* Up-to-date Model (icon: a calendar with the number 1)
* **Color Association:** The arrow from the core model to the specialized model is colored orange.
### Key Observations
* The core model (purple sphere) serves as the central point, branching out to different adaptation strategies.
* Each adaptation strategy focuses on a specific type of improvement: specialization for tasks, correction of errors, and updating with temporal information.
* The diagram uses icons to represent the specialized models, making it visually intuitive.
* The color coding of the arrows helps to associate the core model with its corresponding specialized models.
### Interpretation
The diagram illustrates a modular approach to model adaptation. Instead of retraining the entire model for every new task or change, the core model can be adapted through specialized modules. This approach offers several advantages:
* **Efficiency:** Adapting a smaller module is computationally less expensive than retraining the entire model.
* **Flexibility:** Different adaptation strategies can be applied independently, allowing for targeted improvements.
* **Maintainability:** The modular design makes it easier to update and maintain the model.
The diagram highlights the importance of addressing different types of model limitations:
* **Task Specialization** addresses the need for models to perform specific tasks effectively.
* **Model Patching** addresses the need to correct errors and ensure compliance.
* **Temporal Adaptation** addresses the need to keep models up-to-date with changing information.
The radiating lines from the core model suggest that these adaptation strategies are not mutually exclusive and can be combined to create even more powerful and versatile models. The diagram suggests a layered approach to AI development, where a foundational model is continuously refined and adapted to meet evolving needs.
</details>
While foundation models provide a powerful general-purpose engine for processing multi-modal information, adapting a foundation model before use is necessary for some applications. Broadly, an adaptation procedure produces an adapted model by conditioning a foundation model on additional information, either by priming the foundation model through the inclusion of new data or a prompt in its input or by updating some or all of the foundation model's parameters to reflect the new information. For example, in text summarization, appending a prompt such as TL;DR to the input article can improve foundation model performance [Radford et al. 2019] by acting as a task specification for the foundation model. Alternatively, fine-tuning the parameters of a foundation model with an organization's internal, domain-specific data could improve the model's accuracy by adding information relevant to the organization's use case. In this section, we describe existing approaches to adaptation and several factors that determine whether a particular adaptation procedure is appropriate for a particular setting. We additionally describe various use cases for foundation model adaptation, including relatively well-studied settings such as specialization of a foundation model to a particular task or domain as well as more speculative settings like test-time data removal [Bourtoule et al. 2019] and editing model behavior on particular inputs [Sinitsin et al. 2020]. We conclude by presenting a long-horizon goal for future research in foundation model adaptation.
## 4.3.1 Methods for foundation model adaptation.
Many methods for adapting foundation models have been proposed, making the decision of which adaptation procedure to use for a particular problem or compute environment difficult. We emphasize three factors of particular importance for practitioners to consider when selecting an adaptation procedure: (1) the compute budget (specifically storage and memory); (2) the amount of task-specific data available; and (3) extent of access to foundation model gradients.
Factor 1: Compute budget. For foundation models with billions or trillions of parameters, finetuning all model parameters may demand prohibitively large memory. Also, separately fine-tuning for many tasks can incur unacceptable storage costs. There are many works that propose methods to reduce the storage for adapting foundation models, and we refer to this class of lightweight adaptation methods as low-storage adaptation . Typically, methods in this class freeze most of the pretrained foundation model parameters and only learn a relatively small number of task-specific parameters (either by fine-tuning some pretrained parameters or by adding altogether new modules), reducing the storage overhead for each task.
The key design decision for such algorithms is the choice of parameters to be adapted. Perhaps the simplest approach is to tune only the final layer of the pretrained model, while other works tune only the model's bias vectors [Zaken et al. 2021], low-rank residuals to model weight tensors [Hu et al. 2021], or masks over model parameters [Zhao et al. 2020b]. Another line of research tunes 'soft' prompts [Li and Liang 2021; Qin and Eisner 2021; Liu et al. 2021e; Lester et al. 2021; Hambardzumyan et al. 2021], corresponding to sequences of arbitrary parameter vectors rather than embeddings of the model's vocabulary, and conditions the foundation model on these prompts by concatenation with input activations, either at the input layer or at all layers. Another approach freezes all model parameters and interleaves new MLP modules with trainable parameters between existing model layers [Houlsby et al. 2019]. While these lightweight adaptation techniques seem to trade off parameter efficiency and performance on downstream tasks, they sometimes achieve comparable performance to full fine-tuning, despite updating 1000 × fewer parameters [Zaken et al. 2021; Li and Liang 2021; Hu et al. 2021]. Lester et al. [2021] shows an instance when the performance gap between full fine-tuning and lightweight adaptation vanishes as the model size increases. We remain uncertain how lightweight adaptation techniques scale as model size increases [Aghajanyan et al. 2020]. Because GPU memory is typically a more limiting resource than disk storage, lowmemory adaptation procedures are perhaps even more critical to democratizing foundation models than low-storage adaptation procedures. Various techniques for low-memory model training have been proposed, which can be directly applied to fine-tuning-based adaptation procedures (§4.5: systems). However, some low-memory procedures such as gradient checkpointing [Chen et al. 2016] trade off computation and memory, potentially exacerbating the significant energy consumption of foundation models [Bender et al. 2021].
Factor 2: Data availability. Task specialization mostly demands task-specific labeled data as training signals. 57 However, the cost of annotation varies greatly across tasks and languages; for example, annotating MRI data requires expert medical knowledge, whereas labeling sentiment for English texts requires only common sense judgement. When adaptation data is plentiful, we might apply traditional fine-tuning approaches or their lightweight counterparts. In low-resource language-based settings, combining prompting and fine-tuning has been shown to be a promising direction [Schick and Schütze 2021a,b; Gao et al. 2020c; Perez et al. 2021; IV et al. 2021; Min et al. 2021]. Le Scao and Rush [2021] shows that a well-tuned prompt can be worth around 100 training examples, and fine-tuning a carefully prompted foundation model is significantly more data-efficient than fine-tuning an unconditioned foundation model.
Factor 3: Access to foundation model gradients. Despite the significant impact of foundation models on some research communities, little standardization of distribution practices exists for large scale foundation models (with more than 50 billion parameters). As we gradually become aware of the potential hazards from the misuse of foundation models (see §5.2: misuse), providing access to all
57 Prompts are an exception, although we might consider prompts to implicitly represent the information contained in a batch of labeled data [Le Scao and Rush 2021].
of a foundation model's parameters for fine-tuning might result in ethical concerns. Moreover, most users do not have enough compute resources to exploit their full access. For example, the memory requirements of foundation models might preclude their direct fine-tuning for many organizations and institutions. Consequently, future foundation model providers would likely restrict access to the full parameters of the model and instead provide surrogate API access, as exemplified by an early foundation model, GPT-3. On one extreme, if a foundation model provider enables access only to the model output (e.g., the textual continuation of a prompt, the generated image, or a score evaluating the alignment between an image and a textual description), foundation models can be adapted using in-context learning [Brown et al. 2020]. In-context learning freezes the foundation model parameters, and steers the output of the foundation models by conditioning on a (typically natural language) prompt, which might be composed of task instructions or demonstrations. To improve the performance of in-context learning, the prompt needs to be carefully designed, either by manual search or an automated procedure [Jiang et al. 2020; Shin et al. 2020], and validated on the adaptation data. At the other extreme, if the foundation model provider grants access to gradients with respect to model parameters, full fine-tuning can be applied, where all model parameters are updated to improve performance on a downstream task. As a middle ground, we might obtain gradient access only to foundation model inputs, 58 which are considerably lower dimensional than foundation model parameters. In this case, we could deploy lightweight adaptation techniques [Liu et al. 2021e; Li and Liang 2021; Lester et al. 2021], which freeze the model parameters and optimize a continuous prefix or prompt for each task.
## 4.3.2 Use cases for adaptation.
Adaptation is useful whenever the desired use case of a model differs from the relatively general training objective used for foundation model training (§4.2: training). Most commonly considered is the case in which a foundation model is adapted to perform a specific task (e.g., text summarization or animal classification from images), narrowing the scope of the model. Indeed, the vast majority of existing approaches described earlier in this section have targeted this setting. However, other forms of adaptation are useful, such as making local model edits to correct undesirable predictions for particular inputs or adding privacy constraints to the trained foundation model, which are task-agnostic. In this subsection, we describe a variety of use cases for adaptation, the methods that are most applicable to them, and remaining challenges in addressing these settings.
Task specialization. The most widely-studied case of foundation model adaptation is that of task specialization, in which a foundation model is adapted to optimize performance for a specific task or set of tasks. For example, specializing for summarization tasks would induce foundation model behavior to extract key ideas from the input document(s) and re-organize them in short summary sentences. Various adaptation procedures have proven effective for task specialization, showing significant improvement over performance of the unadapted model [Howard and Ruder 2018; Brown et al. 2020]. In addition to the relatively widely-studied setting of specializing foundation models to specific tasks, other task-agnostic adaptation problems become increasingly challenging (but no less important) for foundation models owing to their size and computational demands. For example, the cost of training foundation models makes continual training over time to keep a model's predictions up to date with current events particularly expensive. Additionally, the challenge of collecting massive anonymized datasets used to train foundation models (§4.6: data) make the likelihood of personal information leakage into training sets non-trivial; mechanisms to efficiently remove training data from a foundation model post-facto are therefore desirable.
58 Assuming the foundation model provider enables the input space to be continuous.
Temporal adaptation. Ideally, foundation models store knowledge that closely represents the state of the world, independent of modality. However, the world is constantly changing; new heads of state are elected, clothing styles change, social norms and beliefs shift (§5.6: ethics), and the use of language evolves, causing a shift in the input distribution, target predictive distribution, or both. This temporal shift presents a challenging statistical problem due to the induced distribution shift, as discussed in §4.8: robustness. For foundation models, temporal shift also presents a particularly difficult computational problem; due to the computationally demanding nature of training foundation models [Shoeybi et al. 2019; Brown et al. 2020], frequent re-training from scratch might carry unacceptable financial or environmental impacts [Bender et al. 2021] (§5.3: environment), or simply take too long to be a viable method for keeping models up to date. In visual domains, gradual self-training on unlabeled data across intermediate time points can bridge the temporal shift across a long time period, but remains an expensive retraining procedure [Kumar et al. 2020a].
In the context of language models, temporally-partitioned diagnostic datasets help quantify the rate at which large language models become outdated [Lazaridou et al. 2021; Hombaiah et al. 2021; Dhingra et al. 2021], showing that classic techniques like re-weighting training data and dynamic evaluation (updating model parameters with new data at production time [Mikolov et al. 2010]) can partially alleviate, but not fully solve, this problem. Explicitly conditioning a language model on the time period it is intended to model is another technique that has shown some promise [Dhingra et al. 2021]. An alternative strategy to addressing temporal shift is to design retrievalbased (semi-parametric) models, which augment the model input with additional context retrieved from a large, human-interpretable database (e.g., Wikipedia articles) [Karpukhin et al. 2020; Lewis et al. 2020b; Guu et al. 2020; Khandelwal et al. 2020; Khattab et al. 2020]. For retrieval-based models, adaptation corresponds to updating individual units of information in the database (e.g., single paragraphs of encyclopedia articles) without re-training the model. While promising, challenges for retrieval-based approaches remain both in training an accurate retrieval mechanism and in accurately conditioning on the retrieved information [Lewis et al. 2020b]. We revisit the problem of temporal adaptation in the more general context of continual learning later in the section.
Domain specialization. In addition to task specialization, it is often necessary to specialize a foundation model to a particular domain (such as legal documents), without limiting the breadth of tasks the foundation model can accomplish. This specialization induces a mismatch between the foundation model training and adaptation data distributions (§4.8: robustness) which may require new adaptation methods to handle. Prior works have found that diverse and general pretraining data can cause negative transfer for current adaptation methods. For example, Cole et al. [2021] shows that fine-tuning a model pretrained only on the iNaturalist animal classification dataset provides better downstream performance than fine-tuning a model pretrained on iNaturalist along with 750K other images; similarly LegalBERT [Chalkidis et al. 2020], which is pretrained only on legal documents, improves over BERT [Devlin et al. 2019], which is trained on a much more diverse training set on the downstream task of text classification and sequence tagging in legal documents. One approach to domain specialization is to include an intermediate adaptation step, where the foundation model continues training on unlabeled data from the specialized domain. For instance, this approach improves the downstream performance significantly for satellite images and specialized text topics [Reed et al. 2021; Gururangan et al. 2020]. However, continual foundation model training may perform worse than re-training from scratch in certain domains such as legal documents [Chalkidis et al. 2020]. Elucidating the scenarios in which continual training does or does not benefit performance is an important direction for future work.
Local model editing. In some settings, it is useful to adapt a foundation model locally, meaning that the model's predictive distribution should be adapted only for a single input or a local
neighborhood around a single input, without changing the model's behavior for unrelated inputs. For example, when a foundation model produces an especially problematic mistranslation for a particular input phrase and target language, it is desirable to correct this mistranslation without affecting the model's behavior for unrelated phrases. Past work has studied the problem of applying approximately localized updates to large neural networks through new pretraining objectives that enable easy editing with standard gradient descent [Sinitsin et al. 2020], higher-order networks that predict parameter edits for an underlying model [Cao et al. 2021; Mitchell et al. 2021; Hase et al. 2021], and constrained fine-tuning procedures [Zhu et al. 2020]. However, existing methods vary in the reliability with which they can perform model edits without damaging global model performance. Furthermore, scaling these methods to massive foundation models is not straightforward due to their size and the computational cost of training objectives that require computing higher-order gradients [Sinitsin et al. 2020; Cao et al. 2021; Mitchell et al. 2021].
Applying constraints. There are settings in which foundation models need to be adapted to satisfy privacy constraints. For instance, Carlini et al. [2021] demonstrated that existing foundation models are able to memorize sensitive information in the training data and can regurgitate such data when queried via standard APIs. While this phenomenon calls for improved data curation, developing adaptation procedures which eliminate or reduce the influence of specific data examples on the trained model would be a complementary solution. Improved adaptation strategies (along with better pretraining methods) in this direction will also benefit institutions working with foundation models under the General Data Protection Regulation (GDPR), as the mandate gives users the right to be forgotten . While research on the topic of machine unlearning [Bourtoule et al. 2019; Cao and Yang 2015] has started to gain traction, the problem has not yet been studied in depth for foundation models. In addition, foundation models trained on less curated internet data have been shown to exhibit harmful biases targeting specific groups (e.g., gender and racial bias) [Bender et al. 2021; Basta et al. 2019; Kurita et al. 2019; Sheng et al. 2019] and can produce toxic outputs [Gehman et al. 2020] (§5.2: misuse). While strategies such as further fine-tuning the foundation model on carefully curated datasets (for potentially multiple generations) [Solaiman and Dennison 2021] or applying controllable generation techniques [Keskar et al. 2019] have shown some success in mitigating harmful behavior, a framework for training equitable and safe foundation models (§5.1: fairness) will likely require further research with a collective effort across the data collection, training, and adaptation phases as well as consultation with domain experts.
## 4.3.3 A long-term goal for foundation model adaptation research.
To the extent that adaptation is concerned with efficiently integrating a model's existing knowledge with new data or objectives, a natural extension of adaptation is continual learning [McCloskey and Cohen 1989; Parisi et al. 2019] or continual adaptation. The ability to adapt a foundation model continually is desirable, whether to keep a model's knowledge continually up-to-date with world events or cultural developments, continually add data from completely new domains or modalities as they become available, or continually edit a model's memories to comply with privacy or legal constraints as a society's values or laws evolve. However, continual learning problems typically induce catastrophic forgetting [McCloskey and Cohen 1989; Ratcliff 1990; Kirkpatrick et al. 2017] in neural networks, where old tasks or data are rapidly forgotten as the training distribution changes.
Weconsider continual adaptation of a foundation model as a grand challenge for future foundation model adaptation research. Tackling this challenge requires closing the performance gap between a foundation model trained continuously on a non-stationary stream of data from different tasks, domains, or time periods and the same foundation model trained from i.i.d. data from the aggregate mixture [Lopez-Paz and Ranzato 2017]. Innovations in both model architectures and training
objectives are likely to be necessary in order to do so. For example, while memory mechanisms have long been speculated as key to successful continual learning [French 1999] and have shown some promise for foundation models [Lewis et al. 2020b; Guu et al. 2020; Borgeaud et al. 2021], effectively leveraging updated memories remains a challenging problem [Zhang and Choi 2021]. In addition, techniques for localizing knowledge in a foundation model in order to make targeted parameter updates [Dai et al. 2021a] or learning such update rules [Cao et al. 2021; Mitchell et al. 2021] may help prevent forgetting, but repeated application of such updates still induces significant forgetting [Hase et al. 2021]. Continual foundation model adaptation may also require new understanding of how the problem of catastrophic forgetting manifests at the scale of foundation models as well as developing new meta-learning techniques [Schmidhuber 1987; Santoro et al. 2016; Finn et al. 2017] to improve forward transfer of already-learned information to new settings.
Continually training on experiences gathered by deployed foundation models, or even training on aggregated experiences gathered by many different models, may speed the rate of progress of foundation model development, but incur risks of feedback loops and eroded alignment of model behavior and stakeholder values. Despite the aforementioned challenges, continual foundation model adaptation holds the promise of more rapidly responding to shifts in socio-cultural values, better leveraging existing knowledge to learn new concepts, lessening the environmental impact and increasing the accessibility of foundation models by eliminating the computational burden of training from scratch, and reducing the extent that previously-learned concepts must be re-learned due to forgetting.
## 4.4 Evaluation
Authors: Rishi Bommasani, Kawin Ethayarajh, Omar Khattab
## 4.4.1 Introduction.
Evaluation gives context to machine learning models: it serves as a means for (1) tracking progress -how do we we measure the performance of models and how do we design improved models (§4.1: modeling); (2) understanding - what behaviors do models exhibit (§4.11: interpretability) and how do they perform on different slices of data (§4.8: robustness); and (3) documentation -how do we efficiently summarize model behavior and communicate this to diverse stakeholders. For foundation models, each of these purposes for evaluation are critical but the nature of foundation models introduces new challenges that are not generally encountered in other AI or ML settings:
- (1) Tracking progress requires relative comparison, but comparing foundation models is complicated by the fact that foundation models must be adapted (potentially in different ways) to perform tasks.
- (2) Understanding requires specified in-advance knowledge (e.g., taxonomies) of what is being evaluated for, but foundation models acquire emergent skills (e.g., in-context learning) that will be difficult to anticipate in designing evaluations.
- (3) Documentation requires clear desiderata to meaningfully inform decision-making, but foundation models can be adapted for myriad applications, which makes comprehensive documentation challenging.
To orient the discussion of evaluating foundation models, we distinguish two classes of evaluation that arise from the abstraction of foundation models: intrinsic evaluation of the foundation model, which is inherently divorced from a specific task due to the task-agnosticity of these models, and extrinsic evaluation of task-specific models, which is necessarily dependent on both the foundation model and the adaptation mechanism. Further, we recognize that due to the anticipated impact and scope of foundation models, a variety of stakeholders (e.g., foundation model providers and application developers, auditors and policymakers, practitioners and researchers) will require evaluation of both foundation models and task-specific derivatives, with these evaluations serving different purposes and involving different desiderata based on the stakeholder. With this in mind, standard paradigms for the evaluation of machine learning models are not designed explicitly for the setting of foundation models. Therefore, we emphasize intrinsic evaluation (§4.4.2: evaluationintrinsic), the importance of adaptation in extrinsic evaluation (§4.4.3: evaluation-adaptation), and evaluation design (§4.4.4: evaluation-design) as clear steps towards an evaluation framework that is better suited to foundation models. This discussion contributes to broader dialogue surrounding the role of evaluation of machine learning systems [Galliers and Spärck Jones 1993; Lipton and Steinhardt 2019; Ribeiro et al. 2020; Linzen 2020; Kiela et al. 2021; Milli et al. 2021; Jacobs and Wallach 2021; Bowman and Dahl 2021; Dehghani et al. 2021; Ma et al. 2021a, inter alia ] and, given the complexities of evaluation, may benefit from drawing upon theories of measurement and evaluation that exist beyond machine learning [Messick 1987; Jackman 2008; Loevinger 1957; Messick 1988; Hand 2010; Brewer and Crano 2014].
## 4.4.2 Intrinsic evaluation.
Evaluation of machine learning systems has traditionally been grounded in tasks , often ones that are envisioned as functions specifically useful for applications (e.g., translation, object recognition). In contrast, since foundation models are intermediary assets that must be further adapted or specialized to perform useful tasks, the standard evaluation paradigm must be altered to facilitate the direct understanding and comparison of foundation models.
One approach is to evaluate foundation models in terms of the task associated with the training objective. For example, a language model like GPT-3, which was trained by predicting the next word given the preceding context, may be evaluated based on the probabilities it assigns words given their preceding context in held-out test data (i.e., perplexity on language modelling benchmarks like LAMBADA [Paperno et al. 2016]). This approach has shown promise in NLP thus far, but we identify two fundamental limitations it exhibits. First, relying on the training objective for evaluation lacks generality: foundation models trained using different incompatible objectives cannot be readily compared or understood in a consistent frame. Second, evaluation in this way relies upon a proxy relationship to be meaningful, i.e., measurements in terms of the training objective should correlate with other more meaningful and intelligible quantities (e.g., the quality of content generated via a foundation model). While this proxy relationship has proven to be robust in the past in some contexts, it likely will break down when assessing more diverse capabilities of foundation models, their behavior in more diverse environments or domains, and considerations beyond in-domain accuracy (we discuss this more extensively in §4.4.4: evaluation-design). In light of these limitations, we anticipate that two approaches will need to be considered, offering complementary benefits.
Imputing intrinsic evaluation from broad extrinsic evaluation. One route towards evaluating foundation models is to adapt them to a wide range of tasks and measure the performance of the resulting task-specific models. As the foundation model is the shared basis across all of these models, the performance in aggregate reflects on the nature, and quality, of this shared basis. At present, many subareas of AI have begun to construct meta-benchmarks , i.e., a single evaluation that consolidates individual evaluations across a number of different tasks or domains [Wang et al. 2019b,a; Hu et al. 2020; Santurkar et al. 2020; Gehrmann et al. 2021; Hendrycks et al. 2021b; Koh et al. 2021; Tamkin et al. 2021b]. Given the growing adoption of this paradigm and its established strengths, here we note why it is likely insufficient to fully satisfy the goals of evaluations with respect to foundation models. Meta-benchmark evaluation requires adaptation (minimally to specialize the foundation model to each of the tasks in the meta-benchmark), which makes reasoning about the foundation model itself challenging given the addition process (i.e., adaptation) involved. Specifically, this complicates matters of progress, both in terms of tracking (e.g., is performance attributable to potent foundation models or well-designed adaption practices) and in terms of identifying improvements in the process used to learn foundation models (e.g., fundamental improvements in data selection (§4.6: data), training objectives (§4.2: training), and model architectures (§4.1: modeling) may be difficult to identify by comparing the performance on a meta-benchmark between two foundation models). In addition, this evaluation paradigm makes it difficult to understand or document properties and capabilities specific to the foundation model, which may make it unwieldy to convey to certain stakeholders (e.g., SuperGLUE performance may not be sufficiently informative, or may be misleading, for policymakers) or use as grounds for anticipating their behavior for new tasks or domains.
Direct evaluation of intrinsic properties. To complement the use of meta-benchmarks, we also argue for why measuring the properties (e.g., specific capabilities or biases) of foundations models directly is valuable, divorced from specific tasks. 59 For example, we may endeavor to directly measure the linguistic capabilities of foundation models to identify syntactically valid and invalid sentences. To motivate the value of this approach, we return to the purposes for evaluation. Notably,
59 Strictly speaking, these direct evaluations may still involve formulation as a task and foundation model specialization to perform the task, but the objective is more akin to probing (see §4.11: interpretability) of trying to measure the foundation model as directly as possible.
articulating the presence and intensity of capabilities, skills, and biases identifies concrete areas for improvement (progress), elucidates the current potential (understanding), and expresses relevant aspects efficiently (documentation). Such an approach also is in service of broadly comprehensible evaluation, i.e., evaluation that can be understood by both technical experts, non-technical experts (e.g., policymakers or social scientists) and the general purpose. For example, characterizing the persuasive or rhetorical capabilities of these models may especially intuitive for internalizing their potential for disinformation and misuse (§5.2: misuse) [Buchanan et al. 2021].
Direct evaluation of properties also serves as an important pathway towards better handling of the emergent properties of foundation models; to demonstrate this, we take in-context learning as a case study. In particular, Brown et al. [2020] not only demonstrated GPT-3's signature capability of robust in-context learning, but also were the first to specifically identify in-context learning as a specific way to adapt and interact with models (through their exploration of GPT-3). Traditional task-based extrinsic evaluation does not provide a clear means by which in-context learning could have been identified; directly interacting with the foundation model appears to be necessary in this case. More generally, while it appears inevitable that many unanticipated phenomena like in-context learning will be recognized through the unstructured or loosely structured exploration of these models and their capabilities, we believe new approaches to evaluation should be sought out that structure this exploration or, more ambitiously, suggest new properties that can then be more rigorously tested for. Intrinsic evaluation may also lower the threshold for demonstrating the potential of foundation models; new approaches for foundation models may be sufficiently promising if they demonstrate improvements in intrinsic evaluation, even if they are not immediately accompanied by corresponding well-suited adaptation methods for eliciting these capabilities in extrinsic evaluation. There is a significant open question of how intrinsic evaluation should be implemented; the mechanics of such evaluation are unclear. We enumerate a few general principles and considerations that may help inform the design and execution of intrinsic evaluation.
- (1) Inspiration from evaluation of humans. Many of the relevant properties, capabilities, and biases we are interested in for foundation models are also of interest for humans, which suggests that methods for measuring these properties in humans may prove to be instructive, or even directly translatable, for evaluating foundation models. For example, psycholinguistic measures of human linguistic competencies can be modified to evaluate foundation model linguistic competencies [Levy 2008; Frank et al. 2013; Linzen et al. 2016; Ettinger and Linzen 2016; Marvin and Linzen 2018; van Schijndel and Linzen 2018; Futrell et al. 2019; Prasad et al. 2019; Ettinger 2020] or psychological measures of human social biases can be modified to evaluate foundation model social biases [Greenwald et al. 1998; Caliskan et al. 2017; May et al. 2019; Guo and Caliskan 2021].
- (2) Human-in-the-loop evaluation. Human-in-the-loop evaluation may prove to be critical to provide a more exploratory means for understanding foundation models, including assessing their generative or interactive capabilities. In particular, human interaction with foundation models directly may better identify their emergent capabilities and limitations and direct auditing of foundation models [e.g., Raji and Buolamwini 2019, §5.6: ethics] may advances goals for documentation and transparency.
- (3) Validity of intrinsic measures. While intrinsic measures allow for direct measurement at the source, i.e., measurement and evaluation of the properties of a foundation model independent of adaptation and specific tasks, they pose challenges for building trust in the validity [Messick 1987, 1988] of the evaluation. In particular, extrinsic evaluation outcomes may also be important in validating intrinsic measure design, e.g., the predictive validity of intrinsic
measures (i.e., their ability to (statistically) predicted related downstream outcomes) may prove to be a central criterion.
## 4.4.3 Extrinsic evaluation and adaptation.
Evaluating task-specific models has historically involved reporting the performance (generally meaning the accuracy) of the model on a specific held-out test set. While this paradigm may partially suffice to understand or document a model, it often amounts to unfair comparisons between taskspecific models produced with different (and, potentially, unequal) resources, making it difficult to gauge how much progress has been made. The concern of unfair comparisons is exacerbated in the foundation model regime: different foundation models (e.g., BERT and GPT-3) may form the foundation for different task-specific models, and these foundation models may involve vastly different amounts of training data and computation.
To account for the resources required to achieve specific levels of performance, Linzen [2020] argues that (pre) training resources should be acknowledged and tracked in evaluation. We believe this is a scientifically principled proposal; comparing different approaches for training foundation models without accounting for training resources is likely to be misleading. However, given that the process for creating foundation models is especially expensive (e.g., requiring significant human and financial capital), and often governed by societal factors (e.g., commercial incentives) in addition to scientific factors, it may be the case that the foundation models in practice will vary greatly in the training resources afforded, making controlled comparison difficult. Here, we consider an alternative, which may be more pervasively viable, to partially account for the resources involved to complement the proposal of Linzen [2020]. In particular, we consider why extrinsic evaluation should acknowledge adaptation resources , which is critical for ensuring that extrinsic evaluation is able to identify the most performant adaptation methods (which intrinsic evaluation, fundamentally, cannot do). We draw attention to the fact that adaptation resources often are construed as the data used to adapt models, but additional resources [e.g., data used to choose adaptation methods; Perez et al. 2021] and constraints (e.g., the level of access required to adapt the foundation model; see §4.3: adaptation and §5.6: ethics for further discussion) should also be accounted for.
Accounting for adaptation resources. Accounting for the resources expended to adapt foundation models for specific tasks requires a complete understanding of what resources or constraints are used for different adaptation methods, i.e., evaluations that endeavor to account for these resources must evolve alongside developments in what resources are used in adaptation (§4.3: adaptation). In existing task-specific evaluations, most evaluations specify the amount of data that can be used to adapt a (foundation) model to the task. However, Perez et al. [2021] identify a key nuance here that has been discounted in past work, in that this should encapsulate all data used to inform adaptation, i.e., both the data used to adapt the foundation model and the data used to choose the adaptation method. Further, in the foundation model regime, the notion of access requirements for different adaptation methods is also a new consideration that should be factored into evaluation. Concretely, some adaptation methods may generally outperform others but may require greater ability to access or modify the foundation model compared to others (e.g., fine-tuning requires foundation model gradients to modify a foundation model, whereas prompting may only require blackbox access in specifying inputs).
Accounting for the resources involved in adaptation enriches what conclusions can be reasonably drawn from evaluation of task-specific models. At present, task-specific evaluation may provide sufficient clarity for certain types of understanding or documentation of particular task-specific artifacts (i.e., the exact models being evaluated) but do not provide clear signal for how different adaptation methods perform and how to select a specific adaptation method in a given context. In
contrast, by accounting for the resources and access requirements involved in adaptation, evaluation better enables research to identify which adaptation methods or processes make best use of the resources provided, i.e., signal is offered not just for the specific artifacts being evaluated but the more general processes by which they were derived. The proposed evaluation protocol, therefore, clearly works towards identifying which adaptation methods should be used; we note that all of these conclusions should always be taken as specific to a given foundation model, as evaluation in this form does not provide sufficient evidence to conclude an adaptation method is uniformly the best across foundation models. 60
## 4.4.4 Evaluation design.
In theory, the goal of evaluation is to measure and characterize various theoretical constructs (e.g., accuracy, robustness (§4.8: robustness), fairness (§5.1: fairness), efficiency (§4.5: systems), environmental impact (§5.3: environment)) in service of various purposes (i.e., progress, understanding, documentation). However, in practice, the utility of evaluation will be determined by how evaluations are designed and executed. For example, automated measurements of the generative capabilities of foundation models (e.g., their factual correctness) may poorly capture the nature of these qualities and, instead, human-in-the-loop evaluation may better contextualize these capabilities.
In considering the evaluation design we envision for foundation models and their adapted derivatives, we begin with the mechanics of evaluation. Traditionally, the evaluation of machine learning models has involved a large training set that is used to learn the model, an optional validation set that is used to set hyperparameters, and a test set to evaluate the generalization of the learned model to held-out data [Bishop 2006]. As a result, creating benchmarks to evaluate models has historically required large amounts of data, most of which is allocated towards training, which complicates the design of certain diagnostic or nuanced evaluations when data is scarce or expensive to attain [Rogers 2020, 2021]. In contrast, because the benefits of foundation models will often coincide with the sample efficiency of adaptation (i.e., few-shot or zero-shot capabilities) and the diversity of possible applications, we instead envision a regime where benchmarks for individual tasks are much smaller (since far less data needs to be provided as 'training", i.e., adaptation, data) and are far more diverse (both to capture various capabilities in intrinsic evaluation and more strongly ground evaluation in ecologically valid ways [Bronfenbrenner 1977; de Vries et al. 2020] during extrinsic evaluation). This suggests that the nature of foundation models may cause a shift in nature of benchmarks (and the mentality of those constructing benchmarks), de-emphasizing quantity as a key priority in benchmarks as opposed to quality and diversity. The NLP community has begun to see the beginnings of such a regime with expansive and diverse benchmarks like BIG-Bench 61 and FLEX [Bragg et al. 2021]; this paradigm lowers the barrier for benchmark design, thereby enabling the broader community to partake in evaluation design. 62
Alongside the mechanics of evaluation, the presentation of and interface to the evaluation results informs how these results will be used inform decision-making (e.g., new modelling approaches, model selection, auditing). Leaderboards have become the de facto paradigm in machine learning, whereby models are ranked by a specific and singular criterion (generally a form of accuracy). This approach has generally led to significant and rapid progress in system quality over time [e.g., Wang
60 Current results, instead, suggest that different adaptation methods are better-suited to different types of foundation models and training objectives [Liu et al. 2021e; Lester et al. 2021].
61 https://github.com/google/BIG-bench
62 Traditionally, the design of benchmarks like ImageNet [Deng et al. 2009] and SQuAD [Rajpurkar et al. 2016] has been conducted by high-resourced research labs that can afford to pay for the creation of these datasets through crowdsourcing [Rogers 2020].
et al. 2019a], but significant concerns have been raised of whether this yields more general improvements [e.g., Linzen 2020; Bowman and Dahl 2021]. 63 As is true for all machine learning models, it is rarely the case that the desiderata for foundation models and their derivatives will be singular; instead, we anticipate the breadth of their application and societal impact necessitates heightened consideration of criteria beyond accuracy (e.g., robustness, fairness, efficiency and environmental impact). To this end, we note that evaluation of foundation models should report measurements across these diverse fronts; existing benchmarks are increasingly designed to reflect more than just accuracy (e.g., robustness [Koh et al. 2021; Goel et al. 2021], fairness [Nadeem et al. 2021; Nangia et al. 2020], efficiency and environmental impact [Coleman et al. 2017]). Further, we note that if the reporting of performance across this different categories is done in the form of a leaderboard, mechanisms to disambiguate potential trade-offs (to induce a ranking) will be especially necessary [Ethayarajh and Jurafsky 2020]. In particular, since different stakeholders will have different preferences (e.g., the weight they ascribe to different properties) and values [Birhane et al. 2020], leaderboard design should allow stakeholders to interact and manipulate how the ranking is done to align with their values; Ma et al. [2021a] presents an early attempt to enable this by comparing the utility of models using an economic framing based on a user's specified utility function.
## 4.4.5 Takeaways.
Evaluation performs several roles (i.e., progress, understanding, documentation) that are vital for all machine learning paradigms, including the foundation model paradigm. Foundation models introduce new challenges for existing evaluation frameworks; designing evaluations that directly target the foundation model regime will better serve not only the multiple purposes of evaluation, but also the myriad of stakeholders involved.
- (1) While machine learning evaluation traditionally has considered task-specific models, evaluating foundation models involves engaging with the fact that these models are not specific to a task. Evaluation of these models likely will involve integrating two complementary approaches: (a) imputing the properties of foundation models from broad evaluation of task-specific derivatives and (b) direct measurement of these properties in foundation models.
- (2) Existing evaluation frameworks often do not account for the resources required to create the models being evaluated, leading to unfair comparisons . For foundation models, we discuss an evaluation paradigm that emphasizes accounting for adaptation resources (e.g., all data used in adaptation, access requirements for the foundation model), which appears to lead to more informative evaluations that better shape how adaptation is conducted.
- (3) Existing evaluation design often is limited in the diversity of metrics considered and requires large adaptation datasets. For foundation models, we echo growing calls for evaluation to consider a broader range of desiderata (e.g., robustness, fairness, efficiency, environmental impact) to capture the wide range of stakeholder values/preferences, as well highlight how the sample efficiency of adapting adaption models may allow for more diverse evaluations by re-allocating resources involved in designing evaluations.
63 We note the connection to Strathern's Law [Strathern 1997] (sometimes referred to as Goodhart's Law [Goodhart 1984]): 'When a measure becomes a target, it ceases to be a good measure."
## 4.5 Systems
Authors: Deepak Narayanan, Trevor Gale, Keshav Santhanam, Omar Khattab, Tianyi Zhang, Matei Zaharia
Fig. 19. Plot showing the growth of number of parameters and number of training operations (FLOPs) of transformer-based language models (shown in blue), and memory capacity and peak device throughput of NVIDIA P100, V100, and A100 GPUs (shown in red) with time. The rate of growth (slope of each line) of state-of-the-art language models (roughly 10 × a year) far exceeds the rate of increase in computational capacity of hardware (roughly 10 × in four years), motivating the need for parallelism across a large number of accelerators and co-design of algorithms, models, software, and hardware to drive further progress. Number of parameters and number of training operations are obtained from relevant papers [Brown et al. 2020], and memory capacities and peak throughputs are obtained from GPU specification sheets.
<details>
<summary>Image 19 Details</summary>
### Visual Description
\n
## Line Chart: Model & Hardware Growth
### Overview
The image presents a line chart illustrating the growth of model parameters, model FLOPS (Floating Point Operations Per Second), GPU memory, and GPU throughput from 2016 to 2020. The y-axis is a normalized metric on a logarithmic scale, ranging from 1 to 10000, while the x-axis represents the year. The chart aims to demonstrate the relative growth rates of these four key components in the field of machine learning and hardware.
### Components/Axes
* **Title:** Model & Hardware Growth
* **X-axis:** Year (2016, 2017, 2018, 2019, 2020)
* **Y-axis:** Normalized Metric (Logarithmic scale from 1 to 10000)
* **Legend:**
* Model Parameters (Blue)
* Model FLOPS (Dark Blue)
* GPU Memory (Magenta)
* GPU Throughput (Red)
### Detailed Analysis
The chart displays four distinct lines, each representing a different metric over the five-year period.
* **Model Parameters (Blue):** This line starts at approximately 2 in 2016 and exhibits a relatively slow, steady increase until 2018, reaching around 8. From 2019 onwards, the line experiences exponential growth, reaching approximately 500 in 2019 and then soaring to approximately 2000 in 2020.
* **Model FLOPS (Dark Blue):** This line begins at approximately 1 in 2016 and shows a similar pattern to Model Parameters, with a gradual increase until 2018, reaching around 5. It then experiences rapid growth in 2019, reaching approximately 100, and continues to grow significantly in 2020, reaching approximately 800.
* **GPU Memory (Magenta):** This line starts at approximately 1.5 in 2016 and increases steadily, but at a slower rate than the other two lines. It reaches approximately 10 in 2019 and approximately 12 in 2020.
* **GPU Throughput (Red):** This line begins at approximately 2 in 2016 and shows a consistent, but moderate, upward trend. It reaches approximately 12 in 2018, approximately 20 in 2019, and approximately 25 in 2020.
### Key Observations
* Model Parameters and Model FLOPS exhibit the most dramatic growth, particularly from 2019 to 2020, indicating a significant increase in model complexity and computational requirements.
* GPU Memory growth is relatively slow compared to the other metrics, suggesting that memory capacity may be a limiting factor.
* GPU Throughput shows steady growth, but at a slower pace than Model Parameters and FLOPS, indicating that hardware improvements are not keeping pace with the increasing demands of larger models.
* The logarithmic scale emphasizes the exponential growth of Model Parameters and FLOPS.
### Interpretation
The data suggests a clear trend of increasing model complexity and computational demands in the field of machine learning. The rapid growth of Model Parameters and FLOPS, coupled with the slower growth of GPU Memory and Throughput, indicates that hardware is becoming a bottleneck. This could drive further innovation in hardware design to meet the demands of increasingly sophisticated models. The chart highlights the importance of both algorithmic advancements (increasing model size and complexity) and hardware improvements (increasing memory and throughput) for continued progress in the field. The divergence between model growth and hardware growth suggests a potential future where hardware limitations could constrain the development of even larger and more powerful models. The logarithmic scale is crucial for understanding the magnitude of the changes; linear scales would obscure the exponential growth of the model parameters and FLOPS.
</details>
Computer systems are one of the largest bottlenecks to developing foundation models. Foundation models are frequently too large to fit in the main memory of a single accelerator (e.g., GPU) and require an immense amount of computation to train (e.g., > 1000 petaFLOP/s-days for GPT-3 [Brown et al. 2020]). Additionally, these models will likely get larger over time: for instance, the compute and memory requirements of state-of-the-art language models have grown by three orders of magnitude in the last three years, and are projected to continue growing far faster than hardware capabilities (Figure 19). Once trained, these large models are expensive to perform inference with and difficult to debug, monitor, and maintain in production applications. We believe that further advances in the performance and usability of foundation models will require careful co-design across algorithms, models, software, and hardware systems, as well as new interfaces for programming and deploying ML applications. In this section, we discuss the key computer systems challenges in developing and productionizing large-scale foundation models.
## 4.5.1 Improving performance through co-design.
Today, training large-scale foundation models [Brown et al. 2020; Rae et al. 2021; NVIDIA and Microsoft 2021; Wang and Komatsuzaki 2021] can often require custom software systems such as Megatron, DeepSpeed, or Mesh Transformer JAX [Shoeybi et al. 2019; Rasley et al. 2020; Wang 2021], built on top of standard frameworks like PyTorch, TensorFlow, and JAX [Paszke et al. 2019; Abadi et al. 2016; Bradbury et al. 2018]. These software systems rely on a number of innovations across the stack to train models efficiently at scale: new parallelization dimensions such as pipeline
parallelism [Huang et al. 2019; Narayanan et al. 2019] that limit communication while keeping devices busy, state-sharding optimizers to reduce memory usage [Rajbhandari et al. 2020], just-intime (JIT) compilers to optimize the computation graph [PyTorch 2021], and optimized libraries like cuDNN and NCCL [NVIDIA 2021]. Megatron and DeepSpeed are efficient to a particular scale; for example, Megatron can extract up to 52% of the theoretical peak throughput of modern hardware with approximately 3000 GPUs on a model with a trillion parameters [Narayanan et al. 2021b]. However, scaling to larger models with more GPUs still is challenging, since existing parallelization strategies break down at larger GPU counts. Data parallelism is limited by the batch size [Li et al. 2020e], pipeline parallelism by the number of layers in the model [Huang et al. 2019; Narayanan et al. 2019], and tensor model parallelism by the number of GPUs in a single server [Shoeybi et al. 2019].
While we will continue to realize performance gains from new hardware, growth in the resource requirements of large models far outstrips generational hardware improvements [Brown et al. 2020]. To facilitate the next major leap in model capacity and to democratize the advances in model quality, it will be increasingly critical to co-design training algorithms, models, software, and hardware, because many of the avenues to dramatically increase performance alter the semantics of the training computation. For example, executing operations in lower precision (such as fp16 ) can help increase throughput on modern hardware (e.g., the V100 and A100 GPUs have dedicated tensor core units for lower-precision matrix multiplication), but also affect the numerics of the optimization procedure [Micikevicius et al. 2017]. Similarly, exploiting weight sparsity can significantly improve training and inference times [Elsen et al. 2020; Gale et al. 2020] by only performing mathematical operations on the non-zeros in the model, but requires different training algorithms [Jayakumar et al. 2021; Evci et al. 2020; Dettmers and Zettlemoyer 2019]. Other examples of co-design include model architectures that map more efficiently to hardware [So et al. 2019; Child et al. 2019; Wang et al. 2020c; Lee-Thorp et al. 2021; Kitaev et al. 2020; Beltagy et al. 2020; Tay et al. 2020; Ren et al. 2021], efficient optimizers [Anil et al. 2020; Shazeer and Stern 2018], novel tokenization alternatives [Xue et al. 2021; Tay et al. 2021], specially architected hardware training platforms [Jouppi et al. 2017; Mudigere et al. 2021; Selene 2021], and distributed parallelization strategies with relaxed weight update semantics [Narayanan et al. 2019, 2021a].
Case study: efficient knowledge representation. As a concrete case study of successful codesign, retrieval-based models such as REALM, RAG, ColBERT-QA, and RETRO [Guu et al. 2020; Lewis et al. 2020b; Khattab et al. 2020; Borgeaud et al. 2021] take a different approach to model design than simply increasing the number of model parameters. Instead of trying to accumulate implicit knowledge from ever-larger datasets directly into a DNN model with billions of parameters (like GPT3), retrieval-based models store knowledge outside the model parameters in the form of text passages, capturing knowledge within the passages with dense vector representations. These models then use scalable top𝑘 search mechanisms to extract knowledge pertinent to each input, while keeping the DNN model itself small (§4.1.4: modeling-memory). This design improves computational efficiency as well as maintainability of the model in production: for example, developers can update the knowledge of the model just by replacing a text passage, without needing to retrain a large DNN.
Retrieval-based models have achieved promising initial results by leveraging several new crossfunctional ideas, including backpropagating the loss through the retriever during training [Guu et al. 2020] (which requires approximating the gradient through a knowledge store consisting of millions of passages) and modeling fine-grained interactions between queries and passages [Khattab and Zaharia 2020; Khattab et al. 2020] (which requires decomposing the computation into vector-level nearest-neighbor search operations). These techniques allow retrieval-based models to be accurate
and efficient, but demand functionality not readily supported by popular ML frameworks and nearest-neighbor indexes such as FAISS [Johnson et al. 2019].
## 4.5.2 Automated optimization.
Another important challenge in systems is to automate the application of optimizations that straddle algorithms, models, software, and hardware. While many optimizations and parallelization strategies are complementary, identifying the most effective combination of optimizations is challenging since the joint search space grows combinatorially and optimizations interact in non-trivial ways [Narayanan et al. 2021b]. Foundation models heighten the need for automated optimization as manual experimentation is extremely expensive and time-consuming at the scale of thousands of GPUs.
Recent work in this area has focused on systems targeting semantics-preserving optimizations. In particular, systems have been proposed to automatically discover mathematically-equivalent graph substitutions [Jia et al. 2019a; Wang et al. 2021c], facilitate the distributed execution of operator graphs through both high-level APIs and low-level compilers [Rasley et al. 2020; Mandeep Baines 2021; Bradbury et al. 2018; Shazeer et al. 2018; Lepikhin et al. 2020], and automate the selection of hybrid distribution strategies [Jia et al. 2019b; Santhanam et al. 2021]. These systems have helped deploy many foundation models in industry [Fedus et al. 2021; M2M-100 2020; Turing-NLG 2020].
Unfortunately, automated optimization becomes much harder when composing semanticsaltering optimizations (§4.5.1: systems-co-design), as it is often unclear how to jointly model the statistical impacts of these techniques (e.g., how many training iterations are needed to reach a specific accuracy?). We will therefore need new software tools, libraries, and compilers to automatically identify compositions of optimizations that target comprehensive metrics like timeto-accuracy [Coleman et al. 2017; Mattson et al. 2020]. Building such tools will require tight collaboration between systems and machine learning experts.
## 4.5.3 Execution and programming models.
The unique multi-task nature of foundation models provides an opportunity to amortize training and inference costs over many applications. In particular, paradigms such as adaptation mean more sharing across model instances [Raffel 2021]. For example, two models prefix-tuned [Li and Liang 2021] from the same pretrained model can share the same model 'stem,' reducing the storage footprint (the shared stem only needs to be stored once), while also making it possible for execution to be shared and batched across the prefix-tuned models [Shen et al. 2019; Narayanan et al. 2018]. Consequently, the specific adaptation mechanism used informs system optimization (§4.3: adaptation).
It is an open question as to what programming interface should be used to specify that various adapted models are derived from the same pretrained model (e.g., models 𝑌 and 𝑍 are derived from the same pretrained model 𝑋 ), or that various components of two models share parameters (e.g., two models 𝐴 and 𝐵 share the same stem up to layer 𝑖 ). Ludwig [Molino et al. 2019] and PyTorch's Module offer easy ways to compose functionality within a model, but no system today supports cross-model dependencies. Giving users the opportunity to provide annotations will allow training and inference systems to optimize and orchestrate computation more efficiently; without such annotations, systems will not have visibility into what computation and parameters can be shared across model instances. A model's 'adaptation history' (what models is this particular model adapted from) can also be used for debugging: an adapted model's errors on particular types of inputs could originate from the pretrained model, pointing to issues in the pretraining process versus adaptation process. Frameworks like PyTorch, as well as software libraries for
training foundation models such as HuggingFace Transformers [Wolf et al. 2020], do not allow for fine-grained lineage information across entire model instances to be specified.
Building and maintaining a cluster of thousands of accelerators also requires tremendous effort. Newtraining paradigms like Learning@Home [Ryabinin and Gusev 2020; Diskin et al. 2021] explore leveraging volunteer compute over the internet to train foundation models collaboratively. Such fundamentally new execution models can decrease the cost of training for any one entity, but require collaboration across a number of different areas like security (to ensure that a malicious volunteer cannot significantly alter the training process), distributed systems (to deal with fault tolerance issues as volunteers drop), and crowdsourcing.
## 4.5.4 Productionization of foundation models.
As the community continues to push the capabilities of foundation models, realizing their potential will require addressing the challenges associated with deploying these resource-intensive models in production. These challenges include performing model inference with tight latency targets, and ensuring that models and data are monitored in an automated way.
For applications with strict cost and latency constraints, model compression techniques like distillation [Hinton et al. 2015; Li et al. 2020d; Sanh et al. 2019], quantization [Polino et al. 2018; Gholami et al. 2021; Zhou et al. 2018], pruning [LeCun et al. 1990; Gordon et al. 2020; McCarley et al. 2019; Wang et al. 2019c; Sajjad et al. 2020], and sparsity [Gale et al. 2020; Elsen et al. 2020] could aid deployment by transforming larger models to obtain desired inference-time properties. These techniques were originally intended for smaller models (e.g., BERT-L) in low-memory environments (e.g., mobile phones), but are now necessary to handle the extreme scale of modern foundation models in datacenter deployments. Parallelization techniques like tensor model parallelism [Shoeybi et al. 2019], traditionally used for training, might also be useful to reduce inference latency, and also provide additional memory capacity across GPUs to fit the model's parameters.
In addition to these practical constraints, increases in the size and complexity of foundation models and the datasets used to train them pose new challenges to model and dataset lifecycle management. Since models with a large number of parameters are hard to manually inspect by humans, we need better systems for automated dataset curation (§4.6: data) and model quality assurance. Techniques like behavioral testing [Ribeiro et al. 2020] and model assertions [Kang et al. 2020] facilitate easier model maintenance in production by providing analogs to unit tests, runtime monitoring (in the form of test-time assertions), and continuous model improvement (as new inputs come in) for models deployed in end applications. These tools can help address issues of fairness and bias (§5.1: fairness), and reduce model mispredictions.
## 4.6 Data
Authors: Laurel Orr, Simran Arora, Karan Goel, Avanika Narayan, Michael Zhang, Christopher Ré
Foundation models signal a paradigm shift where increasingly massive quantities of data are being 'fed' to these models for improved adaptation performance [Devlin et al. 2019; Radford et al. 2021; Tolstikhin et al. 2021] with the overarching rule-of-thumb being "the more data the better" [Kaplan et al. 2020]. As previous sections have mentioned, this focus on data curation has raised concerns around the foundation model data lifecycle including (1) managing the data at such a large scale (§1: introduction), (2) integrating data across new modalities (§2.3: robotics, §3.1: healthcare), (3) reasoning over licensing and governance regulations - especially when considering the massing web-crawls used in foundation models training - (§3.1: healthcare, §5.4: legality), and (4) understanding the data quality (§4.4: evaluation).
While foundation models add new and difficult facets to these challenges, we see parallels between these issues and core challenges in communities such as data management and data analytics as well as industrial ML pipelines. For example, data management has long studied scalable declarative systems for data analysis, versioning, provenance, and integration-addressing challenges (1) and (2) [Zaharia et al. 2012; Cudré-Mauroux et al. 2009; Stonebraker and Weisberg 2013; Stonebraker and Ilyas 2018; Hellerstein and Stonebraker 2005]. Industry has pipelines dealing with challenge (3) to manage diverse data licenses and help mitigate data violations. There is an entire ecosystem of research and systems tackling challenge (4) to support interactive data analytics and visualization [Hohman et al. 2020]. 64 While these solutions are not necessarily "foundation model-ready", we believe a path to better management of the foundation model data lifecycle should take inspiration from these existing systems.
In this section, we address managing the foundation model data lifecycle. We first outline four desiderata including data management at scale, support for heterogenous data sources, data governance, and data quality monitoring. We then envision how all of these requirements can be integrated into a holistic data management solution called a data hub . The data hub is simply a data management toolkit that can be used by the private or public sectors to better support the interactive management of the foundation model data lifecycle.
## 4.6.1 Data Management Desiderata.
Current practices in foundation model development are generally ad-hoc across the entire lifecycle from data curation and data documentation to model monitoring and patching [Gebru et al. 2018; Bandy and Vincent 2021; Bender and Friedman 2018]. Research in the data management community has shown that well-defined data management platforms facilitate ML model development at scale through data ingestion, data versioning, data provenance, efficient analysis, and model monitoring [Hellerstein and Stonebraker 2005; Agrawal et al. 2019; Vartak et al. 2016; Ikeda and Widom 2010]. 65 Taking inspiration from the data management community, we consider core desiderata when building a holistic data management platform for foundation models.
- (1) Scalability. Foundation models are being trained on increasingly massive quantities of data [Kaplan et al. 2020] with the WuDao 2.0 model being trained on 4.9 TB of multi-modal data. 66 This scale is expected to increase as most recent models are trained largely on public
64 VIS, CHI, SIGGRAPH are a few communities that research interactive data analytics methods and systems. Software systems and libraries such as Pandas, Matplotlib, and Seaborn also aid users in interactive exploration.
65 Feature stores like Michelangelo also support end-to-end ML model building https://eng.uber.com/michelangelomachine-learning-platform/.
66 https://www.scmp.com/tech/tech-war/article/3135764/us-china-tech-war-beijing-funded-ai-researchers-surpassgoogle-and
facing datasets. Public data represents an extremely small fraction of data compared to the petabytes of business and personal data collected every day and used in industrial foundation model pipelines [Marr 2017]. There is therefore a growing need for highly scalable techniques that can handle multi-modal foundation model datasets.
- (2) Data integration. Recent work using foundation models demonstrates that leveraging integrated structured and unstructured data can help models better generalize to rare concepts [Orr et al. 2020] and improve factual knowledge recall [Orr et al. 2020; Logeswaran et al. 2019; Zhang et al. 2019a; Peters et al. 2019; Poerner et al. 2020]. Despite these recent successes, integrating datasets for foundation models remains a challenge. Many works use unstructured text data with structured entity knowledge or image data [Antol et al. 2015]. There is a growing need to integrate datasets across diverse modalities such as text, video, eye-tracking [Hollenstein et al. 2020], and robotic simulations [Lynch and Sermanet 2021] (see §2.3: robotics). We need data-integration solutions that can be applied at an industrial scale to multiple modalities and to multiple domains, such as government, business, and science.
- (3) Privacy and governance controls. The training data used for foundation models may risk the violation of the privacy of data subjects; their data may be disclosed, collected, or used without their consent [Jo and Gebru 2020] or outside the context for which consent was originally given. The issue of consent and use is especially relevant for foundation models where downstream applications cannot always be anticipated. As explained in §5.4: legality, these issues are compounded with the prevalence of web scraped datasets for foundation model training. As there are still open legal questions about how web-crawled data will be governed and copyrighted, 67 the consequences of using web data remain unclear to foundation model providers in the public and private sector. We need tooling to help foundation model providers adapt to emerging regulations and guidelines to ensure safe and responsible data management.
- (4) Understanding data quality. Data quality impacts model performance [Lee et al. 2021b]; however, toolkits or methods to systematically and scalably understand the training data and relevant data subsets are still in their infancy. The data creation process can be messy, and the data can contain different types of biases [Blodgett et al. 2020; Bender et al. 2021] (see §5.1: fairness) and consist of poisoned, false, or duplicated information [Chang et al. 2020; Carlini and Terzis 2021; Buchanan et al. 2021; Lee et al. 2021b]. Data is also continuously updated and refined [Kiela et al. 2021] and may have emergent entities [Fetahu et al. 2015], distribution shift [Chen et al. 2021a], and concept meaning shift [Kenter et al. 2015; Lazaridou et al. 2021]. Further, once deployed, foundation models may present undesirable behavior on critical, fine-grained sub-populations of data that foundation model providers need to detect and mitigate [Goel et al. 2021; Hohman et al. 2018; Ré et al. 2019; Oakden-Rayner et al. 2019]. We need toolkits that can detect and potentially mitigate different types of undesirable data to improve model performance in an interactive and iterative fashion. Such toolkits also need to adapt to the dynamical nature of training data.
## 4.6.2 Data Hub Solution.
Pulling on years of work from data management, data science, and data analytics, we envision a foundation model lifecycle data management solution, which we term a data hub . While examples
67 These issues have recently come to bear by the debate surrounding the use of GitHub data in Copilot's Codex tool to help developers code https://www.pwvconsultants.com/blog/questions-around-bias-legalities-in-githubs-copilot/
of ML-focused data hubs 68 as well as more traditional data management systems exist, 69 they either (1) do not treat data integration as a first class primitive, (2) do not natively support the end-to-end lifecycle with model predictions, or (3) do not allow for interaction-driven data curation and refinement, where foundation model providers can dynamically explore and update possible datasets subject to access control guidelines. We now discuss how the data hub addresses the four desiderata.
Data scale. To address the management at scale challenge, the data hub will need standard data management solutions [Armbrust et al. 2009] such as infrastructure to store and maintain largescale datasets as they change over time and scalable interfaces to query, select, and filter datasets. The hub should support heterogenous compute as well as cloud infrastructure to support scalable solutions in different environments.
Data integration. The hub should incorporate data integration as a first class citizen. It will need advanced data integration solutions [Stonebraker and Ilyas 2018; Abiteboul 1997; Dong et al. 2020; Rekatsinas et al. 2017a] 70 to allow for the merging of structured and unstructured knowledge across modalities and domains. Further, this implies the hub will need to support storing and querying over heterogeneous datasets and sources.
Access control. Considering the access controls of the hub, the hub will need to support diverse documentation, e.g., dataset sheets [Gebru et al. 2018] or data statements [Bender and Friedman 2018], to allow data curators to reflect on their processes and be transparent about the intended use cases, potential biases, and limitations of their dataset. The data hub will need to decide which documentation is required for data to be uploaded (e.g., the data source and data description) and which information is recommended (e.g., what tasks the data could be used for). Furthermore, documentation may need to be updated as datasets evolve [Goel et al. 2021].
Data sources are often associated with licenses, and the hub will need to integrate different sources with different legal concerns and conditions [Masur 2018]. 71 Further, certain datasets have legal guidelines to protect the privacy of the data subjects. The hub will need methods to ensure a dataset does not release personally identifiable information (PII), 72 that the aggregation of anonymized or de-identified data does not release PII, 73 and that the data subjects have given informed consent for their data to be disseminated. 74
Pulling on ideas from data integration [Rekatsinas et al. 2017b], the hub should support mechanisms to enable efficient and safe maintenance and sharing of data resources. Especially as the legality of certain public datasets (e.g., web dumps) are still being decided (§5.4: legality), the hub critically needs tooling to help identify licensing violations and mitigate the impact of any governance violation. As certain violations will likely relate to model behavior, we need systems to support better understanding of model behavior, as we describe next.
Data quality tooling. Drawing on the field of data analysis and exploration, as users interactively select, filter, and refine the data to use for training or adaptation, the hub will need tools to quickly
68 Some public data hubs include: https://data.world/, https://dataverse.harvard.edu/dataverse/harvard, https:// datacommons.org/, https://www.data.gov/, https://www.kaggle.com/, https://huggingface.co/datasets, https://www.ldc. upenn.edu/
69 Some traditional data management systems for foundation models include: https://aws.amazon.com/big-data/datalakesand-analytics/, https://eng.uber.com/michelangelo-machine-learning-platform/, https://kafka.apache.org/
70 https://www.tamr.com/
71 https://content.next.westlaw.com/4-532-4243
72 https://www.justice.gov/opcl/privacy-act-1974
73 http://www2.ed.gov/policy/gen/guid/fpco/ferpa/library/georgialtr.html
74 https://www.dhs.gov/sites/default/files/publications/privacy-policy-guidance-memorandum-2008-01.pdf
understand a user's current dataset and its impact on model behavior [Hohman et al. 2020]. 75 Furthermore, these systems can allow end-to-end foundation model monitoring by incorporating model performance through recent work on slice (sub-population) finding [Chung et al. 2019], model validation on relevant subsets [Goel et al. 2021; Ribeiro et al. 2020], and data valuation [Ghorbani and Zou 2019]. Recent works also present methods that use the model to detect which subpopulations of data contribute the most to a given output to further aid model debugging [Keskar et al. 2019].
Once users can monitor model behavior-especially on rare, yet critical sub-populations-, the hub should provide methods and guidance for users to maintain models by correcting model errors. Although 'model patching' [Goel et al. 2020a] is still an open problem, the work of [Orr et al. 2020] provides a first description of using data engineering to maintain a production self-supervised system that corrected for undesirable behavior through changes to the data, not model. We believe the data hub will need to support interfaces for users to inject targeted data modifications for model maintenance.
We also acknowledge that data curation and exploration are not performed in isolation, and believe the data hub should support a community around sharing useful metrics and analysis pipelines. Inspired by similar community sharing platforms like Hugging Face's ModelHub 76 or Tableau Public's visualization sharing platform, 77 we want users to share insights about foundation model training data.
Open questions. Although our described data hub is inspired by existing toolkits and solutions, we do not believe they are all ready for the challenges of foundation models. In particular, some open questions revolving around designing a data hub are:
- How should we support data versioning so datasets can be updated while maintaining old versions for reproducibility [Agrawal et al. 2019]? Once models are deployed and error buckets are identified, datasets may need to be updated to include more examples from these error buckets. How should these new, targeted examples be collected?
- As described in §4.2: training, we imagine fewer models will be trained from scratch and more will be fine-tuned. How do we support provenance or lineage information to understand where the original data came from, while maintaining subject privacy [Chen et al. 2015a]?
- In the public sector, a data hub may be organized and run by an open-source community of individuals consisting of data curators and foundation model providers. In this setting, answers to questions such as who stores the data? who pays for any compute? who is liable if licensing is violated? are particularly murky. How can the data hub provide that right tooling so that once answers to such questions are resolved, they can be operationalized with ease?
- What is the right set of statistics over the data to provide adequate documentation, without being too costly or difficult to obtain?
- How can a data hub support targeted data modifications such as augmentation [Ma 2019; Shorten and Khoshgoftaar 2019] or data programming [Ratner et al. 2017]?
- How can monitoring toolkits better detect when a foundation model needs to be updated due to poor performance on dynamically changing evaluation data?
Our vision for a data hub is not complete or fully detailed. However, we present initial thoughts on data challenges, and one solution to prompt thinking for how to improve data management for the foundation model lifecycle.
75 Examples of data-focused interactive toolkits include https://www.tableau.com/ and https://www.paxata.com/.
76 https://huggingface.co/models
77 https://public.tableau.com/en-us/s/about
## 4.7 Security and privacy
Authors: Florian Tramèr*, Rohith Kuditipudi*, Xuechen Li*
Fig. 20. Risks and opportunities raised by foundation models for security and privacy of ML systems.
<details>
<summary>Image 20 Details</summary>
### Visual Description
\n
## Diagram: Foundation Model Lifecycle & Risks
### Overview
The image is a diagram illustrating the lifecycle of a foundation model, from large uncurated datasets through the foundation model itself, to downstream applications. It highlights both the benefits and risks associated with each stage. The diagram uses a left-to-right flow to represent the process of training and adaptation.
### Components/Axes
The diagram is divided into three main sections, each represented by a rounded rectangle with a light color background:
1. **Large Uncurated Datasets** (Left, light blue background): This section represents the initial data source.
2. **Foundation Model** (Center, light purple background): This section represents the core model being trained.
3. **Downstream Applications** (Right, light orange background): This section represents the applications built upon the foundation model.
An arrow connects the "Large Uncurated Datasets" section to the "Foundation Model" section, labeled "Training". Another arrow connects the "Foundation Model" section to the "Downstream Applications" section, labeled "Adaptation".
Each section contains a list of bullet points with checkmarks (✓) indicating benefits and crosses (✗) indicating risks. Icons are used to visually represent the data sources and applications.
### Detailed Analysis or Content Details
**Large Uncurated Datasets:**
* ✓ Source of robustness
* ✗ Increased risk of poisoning
* Icons present: Globe with wifi signal, Newspaper, Book Stack, Smartphone with hashtag.
**Foundation Model:**
* ✓ Security choke point
* ✗ Single point-of-failure
* ✗ Increased attack surface
* Central element: A spherical, wireframe globe with interconnected nodes, colored in shades of blue.
**Downstream Applications:**
* ✓ Cheaper private learning
* ✗ Function creep
* Icons present: Person, Laptop, Pencil, Magnifying Glass over Document.
### Key Observations
The diagram emphasizes the trade-offs inherent in using large, uncurated datasets and powerful foundation models. While these approaches offer benefits like robustness and cost-effectiveness, they also introduce significant risks related to security, reliability, and unintended consequences. The central "Foundation Model" is visually depicted as a complex network, highlighting its intricate nature and potential vulnerabilities. The flow from data to model to application is clear and linear.
### Interpretation
This diagram illustrates a common pattern in machine learning: the tension between openness and control. Utilizing large, readily available datasets (uncurated) provides robustness but introduces risks of data poisoning and security vulnerabilities. The foundation model acts as a central point of control (security choke point) but also becomes a single point of failure and a larger attack surface. Finally, downstream applications benefit from cheaper learning but are susceptible to "function creep" – the tendency for models to be used for purposes beyond their original intent.
The diagram suggests that careful consideration must be given to data quality, model security, and responsible deployment when building and using foundation models. The visual representation of the foundation model as a complex network underscores the difficulty of fully understanding and controlling these systems. The diagram is a cautionary tale, highlighting the need for proactive risk mitigation strategies throughout the entire lifecycle of a foundation model.
</details>
As central components in critical data-driven decision-making systems, machine learning models must address a variety of security and privacy threats. 78 These threats can be characterized using the traditional 'CIA triad' of computer security. ML systems should protect the Confidentiality of user data against inference and reconstruction attacks [Fredrikson et al. 2015; Shokri et al. 2017; Carlini et al. 2019, 2021]. Moreover, the secrecy of trained models themselves can be at risk of model stealing attacks [Tramèr et al. 2016; Papernot et al. 2017]. The Integrity of ML systems can be compromised by adversarial examples [Biggio et al. 2013; Szegedy et al. 2014] and data poisoning attacks [Biggio et al. 2012; Chen et al. 2017]. Finally, resource-depletion attacks [Shumailov et al. 2020; Hong et al. 2020a] can threaten the Availability of ML systems.
In regard to these threats, we posit that the security role of foundation models in future machine learning systems will be akin to the role played by the operating system in traditional software systems. Due to its generality and ubiquity, a foundation model may become a single point of failure and thus a prime target for attacks against applications derived from this model. In turn however, a foundation model imbued with strong security and privacy properties could form the backbone for the design of a variety of secure and reliable ML applications. Of course, these applications may still have to be designed to enforce specific security and privacy guarantees (in the same way that software designers cannot rely on a secure operating system to protect against all security risks).
## 4.7.1 Risks.
Single points of failure. Afoundation model that is adapted to a variety of applications represents a single point of failure for these applications. For example, data poisoning attacks on a foundation model, where an adversary inserts malicious examples into the training data, might impact all adapted applications as well. Similarly, adversarial examples against a foundation model (i.e., small input perturbations that cause the model to output very different features) could more easily transfer to adapted applications. Wallace et al. [2019] even find that a single adversarial trigger
78 In this section, we focus on security for foundation models . Some applications of foundation models for security (e.g., detection of toxic content) are discussed in §5.2: misuse.
added to any input can cause language models such as GPT-2 to output a predefined piece of text. A foundation model can also become a single point of failure for data privacy. If a foundation model is pretrained on a company's private data and the model memorizes part of this data, all downstream applications could run the risk of exposing this data [Carlini et al. 2021]. The provider of a foundation model may also be a single point of trust for the privacy of application data. For example, the current API for GPT-3 requires that all (potentially sensitive) data used for fine-tuning or inference be uploaded to OpenAI's servers. Designing a foundation model service that avoids this centralization of trust is an interesting problem.
If the parameters of a foundation model are public, model stealing attacks on adapted applications could be facilitated, as the attacker only needs to reverse-engineer the 'delta' with respect to the public foundation model [Krishna et al. 2019] (e.g., a linear model trained on features extracted from a public frozen model).
Finally, denial-of-service attacks on the foundation model provider could also be a concern and might be exacerbated by querying the model with special high-cost inputs [Shumailov et al. 2020].
Data poisoning. Successful foundation models have so far been trained on large and often uncurated datasets scraped from the Web [Radford et al. 2021, 2019]. This permissive data collection coupled with a lack of direct training supervision - facilitates poisoning attacks on a foundation model's training data (e.g., injecting hateful speech targeted at a specific individual or company into a few outbound pages from Reddit). Worse, the power of poisoning attacks may be exacerbated by the growing size and accuracy of today's models [Carlini 2021].
To illustrate, Schuster et al. [2021] show that a code auto-completion system trained with GPT2 on Github data can be poisoned into suggesting insecure code snippets with the injection of only a few malicious files. Carlini and Terzis [2021] further show that targeted attacks against CLIP-style [Radford et al. 2021] models require modifying as little as two out of 3 million training examples.
Function creep & dual use. Foundation models learn general features that enable them to be easily adapted to a variety of tasks. This flexibility, however, raises concerns that foundation models could be used beyond their originally foreseen purposes - a risk commonly referred to as function creep or dual use . Examples of function creep in machine learning include overlearning [Song and Shmatikov 2019] and adversarial reprogramming [Elsayed et al. 2018].
To illustrate, CLIP was originally trained to solve the generic task of predicting image-text pairs, but in doing so also learned to capture rich facial features [Goh et al. 2021]. While CLIP's 'model card' 79 explicitly places facial recognition and other surveillance technologies as out-of-scope, CLIP can certainly be re-purposed for such tasks [Radiya-Dixit and Tramèr 2021]. This example illustrates that it may be challenging to constrain (or even foresee) the possible nefarious uses of a foundation model when it is designed. §5.2: misuse provides further discussions on dual (mis)use of foundation models.
Multimodal inconsistencies. Multimodality may increase the attack surface of foundation models, by enabling adversaries to exploit inconsistencies across modalities. The possibility of such attacks was demonstrated in an (in)famous example of CLIP classifying an apple with the word 'iPod' stuck to it as an iPod [Goh et al. 2021]. More generally, whenever a concept can be expressed using different modalities, inconsistencies across these modalities may be exploitable.
Such inconsistencies are particularly concerning when a foundation model is adapted to a task that primarily relies on only one of the learned modalities. For example, consider using features extracted from CLIP for facial recognition. This is a purely visual task, yet the adapted model's
79 https://github.com/openai/CLIP/blob/main/model-card.md. Accessed 06.30.2021
features will still be sensitive to textual signals (thus, an attacker might be able to evade facial recognition by wearing clothes with imprinted text). Alternatively, consider an autonomous driving system (an application that also relies primarily on vision) that sees a billboard with the word 'green' on it, and mistakenly interprets this as a green light.
## 4.7.2 Opportunities.
Security choke points. If adapted applications can inherit vulnerabilities from a foundation model, they can also inherit desirable security characteristics - such as robustness to adversarial examples or poisoning attacks. Foundation models could thus serve as security choke points . For example, a model robust to adversarial examples can retain its robustness when it is adapted to other tasks [Shafahi et al. 2019]. Similarly, a foundation model provider that can (somehow) defend against poisoning, model-stealing or resource-depletion attacks could then provide such security guarantees for its customers' applications.
The tradeoff between a foundation model's role as a single point of failure or as a security choke point is reminiscent of similar security tradeoffs in other abstraction layers in the software stack (e.g., an operating system, database system, or a Web browser). By virtue of serving many different applications, an abstraction layer is a prime target for attack, but can typically also leverage far greater resources to enhance its security compared to any single application.
Cheaperprivatelearning. Current foundation models are often trained by amassing vast amounts of data from publicly available sources (e.g., from the open Web). This practice may raise concerns about privacy - in the broad sense of taking user data out of its intended context [Nissenbaum 2004; Carlini et al. 2021]. While some existing works aim to mitigate a model's propensity to memorize training data (e.g., by de-duplicating training data [Lee et al. 2021b], or by pretraining under differential privacy [Anil et al. 2021]), such solutions are unlikely to meet the broad privacy expectations that users could associate with text data [Brown et al. 2022]. On the other hand, public pretraining could also end up being a win for user privacy in applications that handle scarce and sensitive data (e.g., in healthcare).
As an example, consider the problem of training a differentially private model [Dwork et al. 2006] for a healthcare task. Training such a model 'end-to-end' (i.e., without leveraging any pretraining) to a decent privacy-utility tradeoff currently requires vast amounts of privacy-sensitive data [McMahan et al. 2018; Basu et al. 2021]. In contrast, a foundation model pretrained on public data in many cases could be adapted to perform specific tasks with significantly less confidential data [Bommasani et al. 2019; Tramèr and Boneh 2021; Li et al. 2022; Yu et al. 2022].
Robustness to adversarial examples at scale. There is evidence suggesting that training a model that is robust to adversarial examples requires vastly more data compared to standard training [Schmidt et al. 2018], but that unlabeled data may suffice to bridge this gap [Carmon et al. 2019; Uesato et al. 2019]. Moreover, increasing model size and capacity (i.e., over-parameterization) has also been shown to be necessary for achieving adversarial robustness in some settings [Madry et al. 2018; Bubeck and Sellke 2021]. Understanding how best to leverage over-parameterization and unlabeled data to achieve adversarial robustness is an important direction for future research. Given their unprecedented scale (both in terms of model size and training set size), foundation models are uniquely positioned to benefit from this line of inquiry.
Despite their unprecedented scale, current foundation models unfortunately see little gains in robustness to worst-case adversarial perturbations [Fort 2021; Wallace et al. 2019]. However, multimodal models such as CLIP are surprisingly robust to (non-adversarial) distributional shifts (see §4.8: robustness). Whether these gains in distributional robustness can translate to increased resilience against real-world attacks is another exciting open question. Particularly in settings
where adversaries are subject to various constraints (e.g., limited query access or computational budget), there is reason to be optimistic that enhanced distributional robustness could lead to concomitant gains in overall security - even if the foundation model remains vulnerable to worstcase 'white-box' attacks.
## 4.8 Robustness to distribution shifts
Authors: Sang Michael Xie, Ananya Kumar, Rohan Taori, Tony Lee, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto
Real-world ML systems need to be robust to distribution shifts - they should work well on test distributions which differ from the train distribution. High-stakes applications such as poverty mapping in under-resourced countries [Xie et al. 2016; Jean et al. 2016], self-driving cars [Yu et al. 2020a; Sun et al. 2020a], and medical diagnosis [AlBadawy et al. 2018; Dai and Gool 2018] all require models that generalize well to circumstances not seen in the training data, e.g., test examples from different countries, under different driving conditions, or from different hospitals. Prior work has shown that these types of distribution shifts can cause large drops in performance even in state-of-the-art models [Blitzer et al. 2006; Daumé III 2007; Sugiyama et al. 2007; Ganin and Lempitsky 2015; Peng et al. 2019; Kumar et al. 2020a; Arjovsky et al. 2019; Szegedy et al. 2014; Hendrycks and Dietterich 2019; Sagawa et al. 2020a; Recht et al. 2019; Abney 2007; Ruder and Plank 2018; Geirhos et al. 2018; Kumar et al. 2020b; Yu et al. 2020b; Geirhos et al. 2020; Xie et al. 2021a; Koh et al. 2021].
In this section, we consider the role of foundation models on robustness to distribution shifts. A foundation model is trained on a large and diverse unlabeled dataset sampled from a distribution 𝑝 pre and can be adapted to many downstream tasks. For each downstream task T , the foundation model is adapted to labeled training data sampled from an in-distribution (ID) training distribution 𝑝 T ID , and then evaluated on an out-of-distribution (OOD) test distribution 𝑝 T OOD . For example, a poverty prediction model [Xie et al. 2016; Jean et al. 2016] may be pretrained on unlabeled satellite data from across the world to learn useful features for all countries, then fine-tuned on labeled examples from Nigeria, and finally evaluated in Malawi where labeled examples are scarce.
Weargue that 1) foundation models are a particularly promising approach to robustness. Existing work shows that pretraining on unlabeled data is an effective, general-purpose way to improve accuracy on OOD test distributions, in contrast to many robustness interventions which are constrained to narrow types of distribution shifts. However, we also discuss why 2) foundation models may not always mitigate distribution shifts, such as shifts due to spurious correlations or changes over time. Finally, 3) we outline several research directions to leverage and improve foundation models for robustness.
We note that one of the ways in which foundation models lead to improved extrapolation is by providing inductive biases (via model initialization) for the adapted model, which are learned on a diverse dataset that extends beyond the downstream training data. However, this same inductive bias can also encode harmful associations from the pretrained data and lead to representational and allocational harms in the presence of distribution shift. See §4.6: data and §5.1: fairness for further discussion of such harms and methods for mitigation.
## 4.8.1 Advantages.
By learning representations on a large and diverse foundation model training distribution 𝑝 pre, foundation models can improve accuracy of the adapted derivative on the downstream test distribution 𝑝 T OOD . OpenAI's CLIP model, which is a foundation model trained on a diverse set of images and natural language documents, has been shown to be robust to some benchmark distribution shifts on ImageNet [Radford et al. 2021]: for example, both CLIP and a standard ResNet50 obtain 76% accuracy on ImageNet, but CLIP achieves 6% higher accuracy on ImageNetV2 [Recht et al. 2019] and 35% higher accuracy on ImageNet Sketch [Radford et al. 2021], which are both related but different from the original ImageNet training distribution. In contrast, many other robustness interventions, such as adversarial training [Madry et al. 2018], invariant risk minimization [Arjovsky et al. 2019],
Fig. 21. In-distribution (ID) and out-of-distribution (OOD) inputs for a variety of distribution shifts. The prediction tasks depicted here are image classification for images and fact verification for text. Although representations learned by foundation models improve downstream robustness for many shifts (e.g., common corruptions) [Hendrycks and Dietterich 2019; Xie et al. 2021a; Radford et al. 2021], some shifts such as spurious correlations (where grass is predictive of cow) [Beery et al. 2020] and extrapolation across time (with facts that change over time) [Lazaridou et al. 2021] are still likely unaddressed by foundation models.
<details>
<summary>Image 21 Details</summary>
### Visual Description
\n
## Diagram: Robustness to Distribution Shifts in Machine Learning
### Overview
This diagram illustrates different types of distribution shifts encountered in machine learning and their impact on model performance. It categorizes these shifts into those where robustness can be improved with Feature Manipulation (FMs) and those that remain persistently challenging. The diagram visually represents the shift from In-Distribution (ID) data to Out-of-Distribution (OOD) data using image pairs and associated research papers.
### Components/Axes
The diagram is divided into two main sections: "Shifts with improved robustness from FMs" (left) and "Persistently challenging shifts" (right). Each section contains three examples of distribution shifts, each with an ID image, an OOD image, and a corresponding research paper citation. The diagram uses color-coding: blue for ID data and pink for OOD data. Arrows indicate the shift from ID to OOD.
The categories of shifts are:
* **Common corruptions**
* **Shifts across space**
* **Domain shift**
* **Extrapolation, e.g. shift across time**
* **Spurious correlations**
### Detailed Analysis or Content Details
**Section 1: Shifts with improved robustness from FMs**
* **Common corruptions:**
* ID Image: A yellow bird perched on a branch.
* OOD Image: A similar yellow bird, but with a slightly different pose and background.
* Citation: Hendrycks '19
* **Shifts across space:**
* ID Image: A forest scene with visible trees and foliage.
* OOD Image: A desert landscape with sparse vegetation.
* Citation: Xie '21
* **Domain shift:**
* ID Image: A bunch of bananas.
* OOD Image: A metallic, reflective surface resembling a banana peel.
* Citation: Radford '21
**Section 2: Persistently challenging shifts**
* **Extrapolation, e.g. shift across time:**
* ID Image: A portrait of Mike Pence with the text "Pence is the Vice President of the US."
* OOD Image: A portrait of Kamala Harris with the text "Harris is the Vice President of the US."
* Citation: Lazaridou '21
* **Spurious correlations:**
* ID Image: A cow standing in a grassy field.
* OOD Image: A cow on a beach with a boat in the background.
* Citation: Beery '18
### Key Observations
The diagram highlights that certain types of distribution shifts are more easily addressed with techniques like Feature Manipulation. These include common corruptions, shifts across space, and domain shifts. However, shifts involving extrapolation (like changes over time) and spurious correlations pose more significant challenges. The examples demonstrate how seemingly small changes in the input data can lead to incorrect predictions when the model encounters OOD data.
### Interpretation
This diagram illustrates a core challenge in machine learning: the gap between training data (ID) and real-world data (OOD). The categorization of shifts suggests that the nature of the shift significantly impacts the effectiveness of mitigation strategies. Shifts that involve changes in image style or minor variations in the environment can be addressed with techniques that enhance the model's robustness to these changes. However, shifts that require reasoning about time or understanding underlying causal relationships (like spurious correlations) are much harder to handle. The inclusion of research citations indicates ongoing efforts to address these challenges. The example of the Vice President shift is particularly insightful, demonstrating that even a simple change in a key attribute (the person in the image) can lead to a significant performance drop if the model relies on spurious correlations. This diagram is a visual representation of the need for models that can generalize beyond the training distribution and reason about the world in a more robust and reliable way.
</details>
or using larger models have had little impact on effective robustness (defined as the gap between in-distribution and out-of-distribution performance) on these ImageNet tasks, especially without explicit knowledge of the distribution shift [Taori et al. 2020; Santurkar et al. 2020; Radford et al. 2021; Miller et al. 2021].
Many other works demonstrate that pretraining on large datasets can improve robustness to common image corruptions, label shift, and label corruptions [Hendrycks et al. 2019a,b]; to realworld spatial shifts in satellite imagery tasks [Xie et al. 2021a; Kumar et al. 2022]; and to shifts across topics in natural language understanding tasks [Hendrycks et al. 2020; Fisch et al. 2019; Yogatama et al. 2019]. As another example, diversifying the foundation model training data to include multiple languages (as in multilingual BERT [Liu et al. 2020b]) significantly improves performance in unseen language pairs.
## 4.8.2 Persistent challenges.
Despite promising signs that foundation models will result in substantial improvements to robustness, we anticipate that foundation models are not a panacea for distribution shifts. We discuss this in the context of two broad categories of distribution shifts below.
Spurious correlations. Spurious correlations are statistical correlations between features and labels with predictive power on the training distribution but not on the test distribution [HeinzeDeml and Meinshausen 2017; Arjovsky et al. 2019; Sagawa et al. 2020a]. Well-known examples include reliance on background color for object recognition [Xiao et al. 2020], surgical markers for medical diagnostics [Winkler et al. 2019], annotator biases in crowdsourced data [Tsuchiya 2018; Gururangan et al. 2018; Poliak et al. 2018; Geva et al. 2019], and demographic biases [Abid et al. 2021; Nadeem et al. 2021; Gehman et al. 2020]. Models learn these spurious correlations largely because the foundation model training and adaptation data exhibit these biases [Nagarajan et al.
2020; Gehman et al. 2020], and this issue cannot simply be addressed with larger models [Sagawa et al. 2020b].
Foundation models may exacerbate or mitigate the effects of spurious correlations, but this depends on the nature of the particular downstream task and its relation to the foundation model training data and algorithm. By training with a diverse dataset, foundation models may improve robustness to spurious correlations that are found only in a subset of the training data: e.g., existing studies find that pretrained language models can avoid spurious correlations by quickly learning from counterexamples to the spurious correlations [Tu et al. 2020]. However, foundation models can also exacerbate the issue by introducing biases present in the foundation model training data, as observed for demographic biases in GPT-3 and other NLP models [Abid et al. 2021; Nadeem et al. 2021; Gehman et al. 2020]. Moreover, training at scale alone need not fully address the root issue of identifying and not relying on the features that are predictive on the downstream training set but not on the downstream test set [Heinze-Deml and Meinshausen 2017]. Addressing these challenges will require us to understand and manage the inductive bias from foundation model training and develop adaptation algorithms that are resistant to learning spurious correlations.
Extrapolation and temporal drift. Finally, the few- and zero-shot capabilities of foundation models will mean that these models will increasingly be used far beyond the training distribution. While large-scale foundation model training can help with certain forms of extrapolation to new distributions [Papadimitriou and Jurafsky 2020], there may be limits to their extrapolation capabilities. For example, existing language models cannot handle changes to world knowledge or language change without re-training [Lazaridou et al. 2021; Dhingra et al. 2021], zero-shot transfer in CLIP suffers greatly in satellite image domains [Radford et al. 2021], and ImageNet pretraining does not substantially improve the performance of large models on medical images [Raghu et al. 2019; Ke et al. 2021]. We believe that foundation models cannot be assumed to automatically extrapolate within a given modality (e.g., all images), and it will become increasingly important to define and separate the forms of extrapolation that are newly enabled by foundation models from those that remain out of reach. Though existing taxonomies for distribution shifts have been proposed in generality [Quiñonero-Candela et al. 2009; Ye et al. 2021], fully understanding and defining the types of distribution shifts for which foundation models are effective is a major open problem for robustness research.
## 4.8.3 Opportunities.
Foundation models hold substantial promise as a general-purpose robustness intervention for distribution shifts and open new avenues for robustness research. We outline some opportunities and open questions below.
Understanding foundation model representations. Existing studies of the robustness of foundation models have been largely empirical, and there is little understanding of the mechanism behind gains in robustness. Sun et al. [2019b] hypothesize that pretrained representations bring disparate domains (such as ID and OOD distributions) closer together, which can in turn improve generalization from labeled ID data to OOD data [Ben-David et al. 2010]. Controlled experimentation on measuring the distance between domain representations with and without pretraining can elucidate this effect. There are initial promising directions in characterizing foundation model training (e.g., contrastive learning as a spectral graph decomposition [HaoChen et al. 2021a]) and their inductive biases [Saunshi et al. 2020a; Lee et al. 2020a; Zhang and Hashimoto 2020; Xie et al. 2020]. However these theories are limited and fail to address other empirically effective foundation models such as fully generative language models (e.g., GPT-3 [Brown et al. 2020] and image-GPT [Chen
et al. 2020d]). Further understanding how these inductive biases are useful under distribution shift may lead to a more complete theory (§4.10: theory) of how foundation models improve robustness.
Data augmentation in foundation model training. While foundation models trained without knowledge of the downstream tasks can avoid some task-specific biases and often improve robustness, certain statistical biases stemming from how the foundation model was trained may persist. As a concrete example, many contemporary self-supervision algorithms are heavily dependent on choosing an appropriate set of data augmentations [Chen et al. 2020c], which in turn confers different types of robustness in the adaptation phase. For instance, Xiao et al. [2021] show that a foundation model for vision trained with contrastive learning on rotation augmentations may improve OOD performance on adaptation tasks with rotation invariance, but may not improve robustness for tasks where OOD generalization requires other invariances. Further research into what types of data augmentations improve robustness for a wide range of downstream tasks - including data augmentations that are learned from data [Wong and Kolter 2020; Tamkin et al. 2021c] or designed to be generally applicable across data modalities [Verma et al. 2021] - will inform better foundation model training algorithms (§4.2: training).
Encoding structure in foundation model training. In general, exploring new ways of encoding known structure and invariances in the data is an important path forward for foundation model training. Many real-world tasks have additional metadata (e.g., spatial location coordinates, climate information from auxiliary satellites in our poverty prediction example), which may provide additional structure for OOD generalization (e.g., across geographic areas) [Xie et al. 2021a; Koh et al. 2021]. For example, Xie et al. [2021a] show that metadata can be used as targets for pretraining to improve downstream OOD accuracy. In language, modeling the tags in HTML data provides additional downstream-task-adjacent supervision, allows for new forms of prompting (e.g., filling in <title> tags for title suggestion), and improves data efficiency [Aghajanyan et al. 2021]. While current data augmentation methods encode hand-crafted knowledge, other avenues such as exploiting metadata could provide more automated ways of determining which structures and invariances to incorporate for foundation model training.
Specialization vs. diversity in foundation model training data. The choice of foundation model training data has downstream effects - training on a more diverse dataset is not always better for downstream performance than a more specialized foundation model [Cole et al. 2021; Chalkidis et al. 2020] (see §4.3: adaptation for a more detailed discussion). In some domains such as satellite images and specialized text topics, continued pretraining on the specialized domain improves the downstream performance significantly [Reed et al. 2021; Gururangan et al. 2020]. This is a potential source of tension: on one hand, we might want to train the foundation model on a large, diverse dataset in order to have more robust performance under distribution shifts, while on the other hand, we might need to specialize the foundation model to improve its in-distribution and out-of-distribution performance on downstream tasks. A better understanding of how specialization affects the in-distribution and out-of-distribution performance of foundation models will allow us to design and collect more effective foundation model training sets.
Adaptation methods. Although foundation models provide a strong starting point, how the adaptation method uses the pretrained information can affect robustness. For instance, lightweight tuning methods for language models (e.g., adapter/prefix/prompt tuning [Houlsby et al. 2019; Li and Liang 2021; Lester et al. 2021]), which adapt the model for a new task by optimizing a small set of parameters (such as a continuous prompt) while keeping the other foundation model parameters frozen, seem to give OOD performance benefits (§4.3: adaptation). Xie et al. [2021b] explain this in a special case, where composing a learned model with a frozen foundation model
can reduce the complexity of the learned model, improving generalization both ID and OOD. On vision datasets, Wortsman et al. [2021]; Kumar et al. [2022] find that freezing a foundation model and training only the head can lead to better OOD performance than fine-tuning the entire model. Kumar et al. [2022] explain this theoretically by showing that full fine-tuning can distort pretrained features even in a simple setting (two layer linear networks). However, it is still poorly understood in general why freezing parameters seems to improve OOD performance. Finally, while current adaptation methods may suffice for good ID generalization, the methods do not explicitly account for distribution shift in their design. As a first step, we can investigate how methods for distribution shifts such as domain adaptation, domain generalization, and semi-supervised learning methods interact with foundation models when used for adaptation. Progress in these directions can lead to adaptation methods that can better leverage foundation models for robustness.
## 4.9 AI safety and alignment
Authors: Alex Tamkin, Geoff Keeling, Jack Ryan, Sydney von Arx
The field of Artificial Intelligence (AI) Safety concerns itself with potential accidents, hazards, and risks of advanced AI models, especially larger-scale risks to communities or societies. Current foundation models may be far from posing such risks; however, the breadth of their capabilities and potential applications is striking, and a clear shift from previous ML paradigms. While AI safety has historically occupied a more marginal position within AI research, the current transition towards foundation models and their corresponding generality offers an opportunity for AI safety researchers to revisit the core questions of the field in a new light and reassess their immediate or near-future relevance. 80
## 4.9.1 Traditional problems in AI safety.
A major branch of AI safety research concerns the implications of advanced AI systems, including those that might match or exceed human performance across a broad class of cognitive tasks [Everitt et al. 2018]. 81 A central goal of safety research in this context is to mitigate large-scale risks posed by the development of advanced AI. 82 These risks may be significantly more speculative than those considered in §5.2: misuse, §4.8: robustness, and §4.7: security; however, they are of far greater magnitude, and could at least in principle result from future, highly-capable systems. Of particular concern are global catastrophic risks: roughly, risks that are global or trans-generational in scope-causing death or otherwise significantly reducing the welfare of those affected (e.g., a nuclear war or rapid ecological collapse) [Bostrom and Cirkovic 2011]. What AI safety research amounts to, then, is a family of projects which aim to characterize what (if any) catastrophic risks are posed by the development of advanced AI, and develop plausible technical solutions for mitigating the probability or the severity of these risks. The best-case scenario from the point of view of AI safety is a solution to the control problem: how to develop an advanced AI system that enables us to reap the computational benefits of that system while at the same time leaving us with sufficient control such that the deployment of the system does not result in a global catastrophe [Bostrom and Cirkovic 2011]. However technical solutions are not sufficient to ensure safety: ensuring that safe algorithms are actually those implemented into real-world systems and that unsafe systems are not deployed may require additional sociotechnical measures and institutions.
Reinforcement Learning (RL), which studies decision-making agents optimized towards rewards, has been a dominant focus in AI safety for the past decade. What is at issue here is the difficulty of specifying and instantiating a reward function for the AI that aligns with human values, in the minimal sense of not posing a global catastrophic threat. 83 While this problem, known as value alignment [Gabriel 2020; Yudkowsky 2016], may seem trivial at first glance, human values are diverse, 84 amorphous, and challenging to capture quantitatively. Due to this, a salient concern is reward hacking, where the AI finds an unforeseen policy that maximizes a proxy reward for human wellbeing, but whose misspecification results in a significant harm. 85 Many efforts to combat the
80 See Amodei et al. [2016] and Hendrycks et al. [2021d] for broader perspectives on open problems in AI Safety.
81 This is referred to by some as AGI or artificial general intelligence, although terminology use varies [e.g., see Karnofsky 2016].
82 Note that this does not require a belief that building certain kinds of advanced AI is a desirable goal, nor even certainty that it is an achievable one.
83 See Hubinger et al. [2019] for a discussion of some challenges that arise at the threshold between reward specification and reward instantiation.
84 See Gabriel [2020] for an extended discussion of human diversity, ethics, and the value alignment problem
85 See this spreadsheet for a list of real-world examples of reward hacking, including an aircraft landing algorithm which achieved a perfect score by outputting large forces that exploited a flaw in the simulator.
value alignment problem have focused on maximizing corrigibility, which is when errors in the design of a system can be corrected once the system is running [Soares et al. 2015]. This can be far from straightforward-in the RL context, an agent with a specified goal would be incentivized to prohibit attempts to alter that goal, as any attempt to alter that goal would likely be suboptimal for the goal's realization [Omohundro 2008].
However, pure RL is not the only theorized route to advanced AI. Foundation models can also be trained with simple (self-)supervised objectives like next-token prediction, yet can still be used in interactive and goal-directed ways, with or without additional RL training. Moreover, it appears that many of these methods may result in increased capabilities through straightforward scaling of compute, number of parameters, and dataset size [Hestness et al. 2017; Kaplan et al. 2020]. What concepts like value alignment and corrigibility amount to in the broader context of foundation models differ in several respects to the pure RL case, and must accordingly be carefully theorized.
## 4.9.2 Current foundation models and AI safety.
Many of these risks in the RL setting result from models optimized to carry out goals. However, a key challenge for AI safety research on recent foundation models is that goal-directed behavior may emerge despite not being explicitly optimized for (see also §4.2: training). As an example, large language models may be trained on corpora where agents use language in goal-directed ways, such as in persuasive text. To predict the next token well, a model may acquire a general capability to reason and produce arguments, which could emerge with suitable contexts. Foundation models trained on other kinds of human data may capture other kinds of goal-directed behavior present in the data; e.g., robotic agents trained to mimic humans in videos may attempt to punch or knock-out their human operators if their training data includes videos of boxing matches. Recent work has also attempted to directly train agents to produce goal-directed behavior; for example, the Decision Transformer trains a sequence model on trajectories prepended with their returns [Srivastava et al. 2019; Schmidhuber 2019; Chen et al. 2021b]. One can then generate high-return trajectories by 'prompting' this model with a high return, which raises similar questions of reward hacking from the RL context.
However, a major aim of safety research on goal-directed models is to gain more principled control and explainability over the actions being pursued by the agent, as opposed to relying on inscrutable decisions from a blackbox neural network. 86 This makes current foundation models an exciting avenue of study for AI safety research, as aligning them may be a useful precursor for aligning more advanced models [Christiano 2016; Cotra 2021; Kenton et al. 2021]. One challenge is the misalignment between the foundation model's training objective and the desired behavior; for example, a language model may be trained to predict the next word of all documents in the training corpus regardless of veracity, but users may want the model to only output true or helpful text [Tamkin et al. 2021a]. One potential way to steer goal-directed agents towards desired behavior may be to train them with natural language descriptions of actions - this may enable steering them with language as well as enabling them to output interpretable language describing the task they "believe" they are performing, similar to methods for controllable generation and source attribution [e.g., Keskar et al. 2019, see also §2.3: robotics, §2.5: interaction, and §4.11: interpretability]. However, further advances would be necessary to ensure the reliability and self-consistency of such models in the wild (§4.8: robustness), as well as gaining a more mechanistic understanding of how these models operate [Cammarata et al. 2020, also see §4.11: interpretability]. And even if natural language-based control of future foundation models enables better task specification
86 For more on the relationship between understanding and semantics see §2.6: philosophy
and monitoring, models may acquire deceptive or otherwise undesirable behavior from human data - identifying and neutralizing this behavior is another important direction for future study.
While the self-supervised objectives described in the previous paragraph train models to capture human behavior in the data, new training paradigms may produce goal-directed foundation models capable of carrying out a wide range of tasks in complex environments, and which exhibit capabilities superior to humans in different domains (see §4.2: training). For example, goal-directed foundation models may be trained in an open-ended self-play setting, similar to AlphaGo, or in vast multitask single-agent RL setups. This might lead to emergent capabilities that complicate efforts to get agents to carry out goals, especially if many agents are trained together in a rich world-simulator that encourages the development of skills like deception, misdirection, dissimulation, persuasion, and strategic planning. Aside from countering deceptive behavior, it also remains unclear how to effectively evaluate and control the behavior of very capable models, known as scalable oversight or alignment [Amodei et al. 2016; Leike et al. 2018]; e.g., scoring novel reactions proposed by a chemical foundation model (see §4.4: evaluation). New human-in-the-loop approaches for training, steering, monitoring, and understanding these models are thus exciting future directions.
Finally, even before any of these more advanced capabilities emerge, an important research area for AI safety in the near term is characterizing and forecasting the capabilities of current self-supervised foundation models. There are three aspects which make this challenging. First, the generality of foundation models means that they can be applied to countless different kinds of applications in unexpected ways. Enumerating current and planned applications of foundation models is not sufficient to capture the full range of ways they could be used. Second, even within a particular application, model capabilities are emergent: they grow and change in unexpected ways as models scale. For example, the ability to control GPT-3 via 'prompting" was an emergent phenomenon of which only the barest glimpses were evident in the smaller GPT-2 model [Radford et al. 2019; Brown et al. 2020]. What the emergent properties of future foundation models will look like is unknown. Third, even within a particular application and scale, a model's capabilities are not easy to characterize. For example, the ability of GPT-3 to perform addition improves dramatically once commas are added to the inputs [Branwen 2020; Brockman 2020]. Similarly, small rewordings of prompts can have large impacts on task performance. Since the space of prompts is intractable to enumerate, it is challenging to definitely assert that any task is outside the reach of current promptbased foundation models - this is a major challenge for reasoning about possible catastrophic risks from foundation models.
## 4.9.3 Potential catastrophic risks from future foundation models.
The broad and quickly-growing capabilities of current models suggest the benefit of attempting to characterize possible catastrophic risks from more advanced systems. We see at least two ways in which advanced foundation models might contribute to such outcomes.
Catastrophic robustness failures. §4.8: robustness discusses how models may behave in unexpected or harmful ways when confronted with new kinds of data [Amodei et al. 2016; Yudkowsky et al. 2008]. These failures may be especially consequential if foundation models are integrated into important systems that leverage foundation models' ability to quickly adapt to many different tasks and situations. Failures could be catastrophic if they occur in warfare systems (resulting in unwanted discharge of weapons, possibly igniting a conflict), critical infrastructure (accidental destruction of critical energy or agricultural capabilities), or if they become essential to a large fraction of economic activity (whose unexpected failure could result in a sudden collapse in living standards and political instability; see also §5.5: economics). Indeed, the threat of catastrophic robustness failures is particularly pertinent for foundation models in contrast to other kinds of
AI. This is because a foundation model consists of a single model that may be adapted for many different use cases, such that robustness failures derived from the statistical associations learned by the model could in principle manifest in a correlated way across several different domains. If the same foundation model is integrated into multiple critical functions, then lack of robustness in the model could lead to correlated failures that span multiple critical functions or failsafes.
Misspecified goals. The use of foundation models might increase the risks of optimizing misaligned yet easy-to-specify goals, often referred to as Goodhart's Law [Kenton et al. 2021; Goodhart 1984]. A current-day example of these risks is the negative effects of some recommender systems (e.g., polarization, media addiction) which may optimize simple engagement metrics rather than a harder-to-measure combination of societal and consumer well-being [Burr et al. 2018; Milano et al. 2020]. Future institutions may leverage uninterpretable foundation models to maximize simple measures such as profit or GDP, due to these models' ability to adapt to the many different subproblems each of these metrics is dependent on. However, at larger scales optimizing for these proxy metrics instead of a more holistic goal designed for human welfare could inadvertently lead to environmental or geopolitical harms [Gabriel 2020; Creel and Hellman 2021].
## 4.9.4 Conclusion.
In sum, we argue that current and potential future emergent properties of foundation models make them ripe objects of study for the field of AI safety. We encourage future work on characterizing and forecasting the exact capabilities and risks of foundation models; developing new methods to align foundation models to human values and desired goals; and for states, research labs, and businesses to coordinate on proactive measures to mitigate salient risks.
## 4.10 Theory
Authors: Aditi Raghunathan, Sang Michael Xie, Ananya Kumar, Niladri Chatterji, Rohan Taori, Tatsunori Hashimoto, Tengyu Ma
Rigorous mathematical theory plays a foundational role in many engineering and science disciplines (e.g., information theory in electrical engineering). We believe that theory of foundation models can be particularly beneficial in guiding technical decisions and innovations because of the huge computational costs associated with experimenting on foundation models. In addition, theoretical insights help elucidate fundamental limitations and explain surprising empirical phenomena. However, the community currently has a limited theoretical understanding of foundation models, despite much recent progress [Arora et al. 2019b; HaoChen et al. 2021a; Wei et al. 2021, 2020b; Zhang and Hashimoto 2021; Saunshi et al. 2020b; Dao et al. 2019; Tosh et al. 2020, 2021; Cai et al. 2021; Lee et al. 2020a; Zimmermann et al. 2021; Bansal et al. 2020; Wang and Isola 2020; Tsai et al. 2020; Tian et al. 2020a,b; Tripuraneni et al. 2020; Du et al. 2020].
Deep neural networks form the backbone of foundation models. Even in the well-studied supervised learning setting, where the train and test scenarios have the same distribution, there are numerous open questions around deep nets such as understanding non-convex optimization, the implicit regularization effect of optimizers, and expressivity. Foundation models raise questions that significantly go beyond the supervised deep learning setting. The core problem in theoretically analyzing foundation models is understanding why training on one distribution with a possibly unsupervised/self-supervised loss leads to good adaptation performance on different downstream distributions and tasks. 87
We will discuss an intuitive modularization to analyze foundation models that lays bare the connections between supervised learning and foundation models, concrete and core technical questions, and some promising theoretical tools to address these questions. These new core questions can provide useful insight into foundation models and can be studied in parallel to supervised deep learning theory. While we focus on analyzing the downstream performance, the proposed modularization and tools could prove useful to analyze other metrics of interest such as robustness to distribution shifts (§4.8: robustness) and security (§4.7: security).
Fig. 22. The analysis of foundation models from pretraining on diverse data to downstream performance on adapted tasks involves capturing the relation between different loss terms as shown above. The main challenge is to analyze the highlighted pretraining-adaptation interface which requires reasoning carefully about the population losses in addition to the model architecture, losses and data distributions of the pretraining and adaptation stages (§4.10.2: theory-interface). Analysis of generalization and optimization largely reduces to their analysis in standard supervised learning.
<details>
<summary>Image 22 Details</summary>
### Visual Description
\n
## Diagram: Pre-Training - Adaptation Interface
### Overview
The image is a diagram illustrating a process flow related to pre-training and adaptation in a machine learning context. It depicts a sequence of loss calculations and their relationship to generalization and optimization. The diagram uses colored arrows to indicate the direction of information flow and labels each step with a specific loss function and equation number.
### Components/Axes
The diagram consists of six rectangular blocks arranged horizontally. Each block represents a loss function. The blocks are labeled as follows:
1. **Empirical Pre-training loss:** `L_pre(θFM)` (Eq 1)
2. **Population Pre-training loss:** `L_pre(θFM)` (Eq 2)
3. **Minimal Adaptation loss:** `L_adapt(θFM)` (Eq 5)
4. **Empirical Adaptation Loss:** `L_adapt(γtask, θFM)` (Eq 3)
5. **Population Adaptation Loss:** `L_adapt(γtask, θFM)` (Eq 4)
Above the blocks are labels indicating the stages of the process: "Generalization", "Pre-Training – Adaptation interface", and "Optimization", "Generalization".
Arrows connect the blocks, indicating the flow of information. A yellow, dashed arrow connects blocks 1 and 2, and blocks 3 and 4. A blue arrow connects blocks 2 and 3. A green arrow connects blocks 4 and 5.
### Detailed Analysis or Content Details
The diagram shows a cyclical process.
* **Block 1 (Empirical Pre-training loss):** Labeled with `L_pre(θFM)` and equation number (Eq 1).
* **Block 2 (Population Pre-training loss):** Labeled with `L_pre(θFM)` and equation number (Eq 2).
* **Block 3 (Minimal Adaptation loss):** Labeled with `L_adapt(θFM)` and equation number (Eq 5).
* **Block 4 (Empirical Adaptation Loss):** Labeled with `L_adapt(γtask, θFM)` and equation number (Eq 3).
* **Block 5 (Population Adaptation Loss):** Labeled with `L_adapt(γtask, θFM)` and equation number (Eq 4).
The yellow dashed arrow indicates a relationship between the empirical and population pre-training losses, and between the minimal adaptation loss and the empirical adaptation loss. The blue arrow indicates a flow from population pre-training loss to minimal adaptation loss. The green arrow indicates a flow from empirical adaptation loss to population adaptation loss.
The labels "Generalization" appear above blocks 1 and 5, "Pre-Training – Adaptation interface" above blocks 2 and 3, and "Optimization" above block 4.
### Key Observations
The diagram highlights the interplay between pre-training and adaptation losses, emphasizing the concepts of empirical and population losses. The cyclical nature suggests an iterative process of refinement. The use of different colors for the arrows may indicate different types of relationships or information flow.
### Interpretation
This diagram illustrates a framework for adapting a pre-trained model to a new task. The pre-training phase (blocks 1 & 2) establishes a general representation, while the adaptation phase (blocks 3, 4 & 5) fine-tunes the model for the specific task. The distinction between empirical and population losses suggests a focus on both fitting the training data and generalizing to unseen data. The "Pre-Training – Adaptation interface" highlights the critical point where the pre-trained knowledge is leveraged for the new task. The cyclical flow suggests an iterative optimization process where the adaptation loss is used to refine the model parameters. The diagram suggests a method to minimize adaptation while maximizing generalization. The equations (Eq 1-5) likely represent the mathematical formulations of these loss functions, but their specific details are not visible in the image. The parameters `θFM` and `γtask` likely represent the model parameters and task-specific parameters, respectively.
</details>
87 The theory for foundation models closely relates to, but also goes beyond the theory for transfer learning (which is itself an underexplored area): foundation models are possibly trained with unlabeled data and will be adapted to many or all natural tasks, whereas transfer learning typically studies labeled source tasks and a fixed number of target tasks.
## 4.10.1 Theoretical formulations and modularizations.
Recall that foundation models are trained on a large amount of raw data (§4.2: training) then adapted to specific tasks (§4.3: adaptation) and therefore can be decomposed naturally into training and adaptation phases. We identify interfaces between them and disentangle parts specific to foundation models from parts that require standard deep learning theory, so that they can be independently worked on. We introduce a modularized analysis framework, which has also been implicitly or explicitly employed in recent works, e.g., Arora et al. [2019b]; HaoChen et al. [2021a]; Wei et al. [2020b]; Tripuraneni et al. [2020]. The crucial component in this modularized analysis turns out to be the pretrain-adaptation interface. We first describe the modularization, and discuss why we find this modularization promising and finally some limitations.
We will refer to the training phase explicitly as 'pretraining' to distinguish it from the adaptation phase that could also involve training on a few samples from a particular task.
Pretraining phase. The pretraining of foundation models often involves a data distribution 𝑝 pre (e.g., the distribution of natural text) and a pretraining loss function ℓ pre ( 𝑥 ; 𝜃 ) that measures the loss (e.g., language modeling loss in GPT-3) on an input 𝑥 for a model with parameters 𝜃 ∈ Θ . Let ˆ 𝑝 pre denote the empirical distribution over a large number of independent samples from 𝑝 pre .
Pretraining minimizes the loss ℓ pre on ˆ 𝑝 pre, which we call the empirical pretraining loss , and produces a model ˆ 𝜃 FM:
$$\widehat { L } _ { p r e } ( \theta ) \overset { d e f } { = } \mathbb { E } _ { x \sim \hat { p } _ { p r e } } [ \ell _ { p r e } ( x ; \theta ) ] , a n d \hat { \theta } _ { F M } \overset { d e f } { = } \underset { \theta \in \Theta } { \arg \min } \widehat { L } _ { p r e } ( \theta ) .$$
We consider the corresponding loss on the population distribution 𝑝 pre, called the population pretraining loss , as a central concept:
$$L _ { p r e } ( \theta ) \stackrel { d e f } { = } \mathbb { E } _ { x \sim p _ { p r e } } [ \ell _ { p r e } ( x ; \theta ) ] .$$
Optimization-based adaptation phase. We frame adaptation as a general constrained optimization problem that depends on ˆ 𝜃 FM, abstracting away those adaptation methods that are based on optimizing certain loss functions such as fine-tuning and prompt-tuning (see, e.g., [Houlsby et al. 2019; Li and Liang 2021; Lester et al. 2021], and §4.3: adaptation).
Since different adaptation methods could modify different subsets of the model parameters, we denote the space of adapted model parameters by some Γ . Given a downstream task distribution 𝑝 task (e.g., question answering in a particular domain) and a few empirical samples ˆ 𝑝 task sampled from 𝑝 task, we model the adaptation phase as minimizing some adaptation loss ℓ adapt on ˆ 𝑝 task w.r.t adapted parameters 𝛾 ∈ Γ :
$$\gamma _ { t a s k } ( \hat { \theta } _ { F M } ) \stackrel { d e f } { = } \underset { \gamma \in \Gamma , C ( \gamma ; \hat { \theta } _ { F M } ) \leq c _ { 0 } } { \arg \min } \widehat { L } _ { a d a p t } ( \gamma , \hat { \theta } _ { F M } ) ,$$
where b 𝐿 adapt ( 𝛾, ˆ 𝜃 FM ) def = E 𝑥 ∼ ˆ 𝑝 task [ ℓ adapt ( 𝑥 ; 𝛾, ˆ 𝜃 FM )] is the empirical adaptation loss, and 𝐶 ( 𝛾, ˆ 𝜃 FM ) ≤ 𝑐 0 is an optional constraint that controls the complexity of the adapted parameters, encompassing both explicit regularization (e.g., model dimensionality and norm) and the implicit regularization of the adaptation process.
We list some common adaptation methods and discuss the corresponding adapted parameter 𝛾 and constraints 𝐶 ( 𝛾, ˆ 𝜃 FM ) ≤ 𝑐 0.
- (1) Linear probing: training a linear classifier on top of the representations from a foundation model. Here Γ = R 𝑘 is the set of linear classifiers on the representations of dimensionality 𝑘 , and 𝐶 ( 𝛾, ˆ 𝜃 FM ) could be the ℓ 2 or ℓ 1 norm of 𝛾 .
- (2) Fine-tuning: optimizing a randomly initialized linear head for a few steps, and all other parameters 𝜃 from the initialization of ˆ 𝜃 FM. Here 𝛾 is the concatenation of 𝜃 and the linear head. Such a process could correspond to some implicit regularization of 𝛾 towards the initialization ˆ 𝜃 FM captured by 𝐶 ( 𝛾, ˆ 𝜃 FM ) ≤ 𝑐 0. The exact term 𝐶 ( 𝛾, ˆ 𝜃 FM ) would depend on the optimization algorithm used, and such a characterization of the implicit regularization of optimization is an area of active research study [e.g., Gunasekar et al. 2017; Soudry et al. 2018; Gunasekar et al. 2018; Arora et al. 2019a; Blanc et al. 2019; Woodworth et al. 2020; Wei et al. 2020a; HaoChen et al. 2021b; Damian et al. 2021; Kumar et al. 2022, and references therein]. 88
- (3) Prompt-tuning: optimizing a small set of continuous task-specific vectors that prepend the task inputs. Here 𝛾 is the continuous prompt vectors which often has small dimensionality, and we may optionally have a constraint on the norms of 𝛾 .
One obvious limitation to note is that this formulation excludes adaptation methods such as in-context learning [Brown et al. 2020] where there is no 'training' (i.e., the minimization of some empirical adaptation loss) during the adaptation phase. We discuss this and other limitations in §4.10.3: theory-incontext.
Two central quantities for the adaptation phase are the population adaptation loss
$$L _ { a d a p t } ( \gamma , \hat { \theta } _ { F M } ) = \mathbb { E } _ { x \sim p _ { t a s k } } [ \ell _ { a d a p t } ( x ; \gamma , \hat { \theta } _ { F M } ) ] \quad ( 4 )$$
and the minimal adaptation loss
$$L _ { a d a p t } ^ { ^ { * } } ( \hat { \theta } _ { F M } ) = \min _ { \gamma \in \Gamma , C ( \gamma ; \hat { \theta } _ { F M } ) \leq c _ { 0 } } L _ { a d a p t } ( \gamma , \hat { \theta } _ { F M } )$$
Separate analysis for modularized phases. Existing generalization theory for standard supervised learning aims to show that b 𝐿 pre ≈ 𝐿 pre and b 𝐿 adapt ≈ 𝐿 adapt. Addressing these questions specifically for deep nets is an active research area. We can also leverage the standard learning theory decomposition to bound the final downstream task loss by the excess generalization error and the minimal adaptation loss as follows.
$$\begin{array} { r l } & { L _ { a d a p t } ( \gamma _ { t a s k } , \hat { \theta } _ { F M } ) \leq \underbrace { L _ { a d a p t } ^ { ^ { * } } ( \hat { \theta } _ { F M } ) } _ { m i n i m a l a d a p t a t i o n l o s s } + g e n e r a l i z a t i o n e r e q u e r } & { ( 6 ) } \end{array}$$
where the generalization error captures the closeness between 𝐿 adapt and b 𝐿 adapt . 89 The decomposition and relationship between these key quantities are shown in Figure 22. The generalization and optimization arrows, as argued above, largely reduce to deep learning theory in the supervised setting. What we are left with is the main challenge with foundation models, which is to understand whytheminimal adaptation loss 𝐿 ∗ adapt ( ˆ 𝜃 FM ) can be small as a result of a small pretraining population loss, which study in §4.10.2: theory-interface.
The work of Arora et al. [2019b] pioneered the pursuit of this question by bounding from above 𝐿 ★ adapt ( ˆ 𝜃 FM ) by 𝐿 pre ( ˆ 𝜃 FM ) in the context of contrastive learning, and HaoChen et al. [2021a]; Tosh et al. [2020, 2021] relax the data assumptions. Other pretraining methods successfully analyzed
88 It may not always be feasible to characterize the inductive bias of adaptation via an explicit constraint 𝐶 ( 𝛾, ˆ 𝜃 FM ) ≤ 𝑐 0. The modularization we propose is also applicable in these cases, but for notational simplicity, we focus on the case where implicit regularization can be approximated via an explicit constraint.
89 More precisely, the generalization error term is the sum of 𝐿 adapt ( 𝛾 task , ˆ 𝜃 FM ) -b 𝐿 adapt ( 𝛾 task , ˆ 𝜃 FM ) and b 𝐿 adapt ( 𝛾 ★ task , ˆ 𝜃 FM ) -𝐿 adapt ( 𝛾 ★ , ˆ 𝜃 FM ) = b 𝐿 adapt ( 𝛾 ★ task , ˆ 𝜃 FM ) -𝐿 ★ adapt ( ˆ 𝜃 FM ) , where 𝛾 ★ task is the minimizer of (5). (6) follows easily be using b 𝐿 adapt ( 𝛾 task , ˆ 𝜃 FM ) ≤ b 𝐿 adapt ( 𝛾 ★ task , ˆ 𝜃 FM ) .
under this framework (implicitly or explicitly) include pretraining with language models [Wei et al. 2021] or self-supervision [Lee et al. 2020a], with self-training algorithms [Wei et al. 2020b; Cai et al. 2021], and with multiple supervised tasks [Tripuraneni et al. 2020; Du et al. 2020].
4.10.2 Why is the pretraining-adaptation interface interesting?.
As shown in Figure 22, the main missing link beyond standard supervised theory is:
Under what conditions does a small population pretraining loss 𝐿 pre ( ˆ 𝜃 FM ) imply a small minimal adaptation loss 𝐿 ★ adapt ( ˆ 𝜃 FM ) and why?
The conditions that lead to a successful interface could depend on several quantities such as the pretraining and adaptation distributions, objectives and training methods, as well as the model architecture. This question is beyond the scope of standard generalization theory, but it does narrow us down to a few important factors specific to foundation models, and captures the essence of various important open questions on foundation models as we argue below.
First, we note that this interface deals with population quantities that concern two different distributions . Hence, the conditions for a successful interface are likely to involve special properties of the distributions, for example, the diversity of the pretraining distribution and structural shifts between the pretraining and adaptation data. This makes the analysis of the interface challenging (as discussed below in §4.10.4: theory-tools) as we need to make careful modeling assumptions about how the two distributions relate to one another. However, this presents the possibility that tools and techniques developed to analyze such interfaces could be useful to understand the effect of distribution shifts and to predict when foundation models can improve robustness.
Second, the population losses and possibly the conditions of a successful interface depend on the model architecture . This raises the challenge of opening up the black-box of the neural nets. What does a small pretraining loss on a particular distribution tell us about the properties of the intermediate layers? Such analyses would also guide us in designing new adaptation methods that more carefully exploit different intermediate representations.
Third, few-shot learning or the sample efficiency of adaptation can be captured through the constraint on the complexity measure 𝐶 ( 𝛾, ˆ 𝜃 FM ) < 𝑐 0 in the minimal adaptation loss. We need to formally characterize these complexity measures (e.g., by understanding the implicit regularization effect of the adaptation process) and further understand why a small population pretraining loss would imply a low-complexity adaptation parameters 𝛾 task. A satisfactory answer to this question would likely allow us to improve the sample-efficiency of downstream adaptation.
Finally, and importantly, critical components of the interface are the choice of the pretraining and adaptation losses . We want to understand how to best combine the pretraining and adaptation objectives for successful adaptation. It is possible that the pretraining objective that best guarantees successful adaptation differs from what is explicitly minimized during the pretraining process - the interface above allows one to use any surrogate population objective on the pretraining distribution. In addition, new surrogate objectives that provably lead to good adaptation across a broad set of tasks could shed light on the fundamental aspects that make foundation models successful.
To summarize, the interface precludes the issue of generalization and allows us to formally reason about the interaction between several important quantities of the pretraining and adaptation phases that can guide practice in important ways.
4.10.3 Challenge: analysis of in-context learning and other emergent behavior.
GPT-3 [Brown et al. 2020] demonstrates the power of in-context learning, an adaptation method that does not need any parameter optimization. In the adaptation phase, the pretrained language foundation model takes in a prompt - a sequence of tokens that concatenates input-output examples
from the task - followed by a test example and simply generates the label of the test example by conditioning on the sequence seen thus far (prompt plus test example). In other words, there is no explicit training or change to the model parameters. What is the mechanism by which the model 'learns' from the different examples by simply executing with the examples as inputs? The previous modularization does not directly apply because we do not obtain new model parameters during adaptation, but rather we only use the generative capabilities of the foundation model by executing on structurally-designed inputs. However, the idea of separating pretraining with infinite data and pretraining with finite data can still be useful. For example, a recent work starts with the assumption of infinite pretraining data and sufficient model expressivity to study in-context learning [Xie et al. 2021c]. These assumptions reduce the characterization of in-context learning to a matter of analyzing the pretraining distribution conditioned on in-context learning prompts, which are drawn from a different distribution than the pretraining data. In particular, Xie et al. [2021c] proposes that in-context learning emerges from long-term coherence structure in the pretraining distribution, which is described by a latent variable generative model with coherence structure. More broadly, while the modularization proposed in this section provides a nice framework to gain useful theoretical insights into foundation models, it is possible that some emergent behavior like in-context learning and other capabilities yet to be discovered would require going beyond the modularization, e.g., by opening the black box of the architecture.
## 4.10.4 Challenge: appropriate data assumptions and mathematical tools.
Understanding the interface between pretraining and adaptation phases requires a more careful study of data distributions than in traditional supervised learning. This is because the pretraining and task adaptation distributions are inherently different. By definition, foundation models are trained on raw data that is typically extremely diverse and task-agnostic, while the adaptation data depends heavily on the task. Similarly, in-context learning emerges as a result of learning to generate data that looks like the pretraining distribution, and thereby understanding in-context learning requires careful modeling of the pretraining data. Hence answering the central questions around foundation models requires realistic and interpretable assumptions that are also amenable to analysis. Recent works either assume certain properties of the population data, e.g., the expansion property in HaoChen et al. [2021a]; Wei et al. [2020b], or that the population data is generated from latent variable models with some structure [Saunshi et al. 2020a; Wei et al. 2021; Arora et al. 2016; Lee et al. 2020a; Zhang and Hashimoto 2020; Tosh et al. 2021].
Wegenerally lack mathematical tools for relating properties of foundation models to the structure in the population data distribution. HaoChen et al. [2021a] applies spectral graph theory to leverage the inner-class connectivity in the population distribution. More precise characterization of ˆ 𝜃 FM via probabilistic and analytical derivations is possible for latent variable models, but so far restricted to relatively simple ones. The community will significantly benefit from more systematic and general mathematical tools to address this question.
It is also highly desirable to define simple toy cases so that theoreticians can precisely compare the strengths of various tools and analyses. For example, HaoChen et al. [2021a] and Wei et al. [2020b] consider the mixture of manifolds problem which might potentially be a good simplified test bed for vision applications. We need more interesting test beds for discrete domains such as NLP. We believe that tractable theoretical models which capture relevant properties of real datasets are a crucial step towards placing foundation models on solid theoretical footing.
## 4.11 Interpretability
Authors: John Hewitt*, Armin W. Thomas*, Pratyusha Kalluri, Rodrigo Castellon, Christopher D. Manning
Compared to most other machine learning models, foundation models are characterized by a vast increase in training data and complexity and the emergence of unforeseen capabilities: foundation models are able to do unforeseen tasks and do these tasks in unforeseen ways. The increasing adoption of foundation models thereby creates growing desires, demands, and unprecedented challenges for understanding their behavior.
In contrast to task-specific models, foundation models are trained across vast and usually highly disparate datasets, potentially spanning many domains and modalities (see §4.2: training). Through this training, foundation models learn an exceptionally wide range of behaviors, which can vary profoundly between tasks and domains, as demonstrated by their ability to be adapted to different types of downstream tasks and to exhibit behaviors that are specific for each of these tasks (see §4.3: adaptation). Take GPT-3 as an example, which was trained as one huge model to simply predict the next word in a text. While this is a very specific and simple-to-define learning task, it has enabled GPT-3 to gain capabilities that far exceed those that one would associate with next word prediction, by combining it with a vast training dataset that comprises all kinds of internet text. As a result, GPT-3 can now adapt behaviors that are clearly outside of the scope of its original training task, such as simple arithmetic and computer programming, when provided with a few training samples. This demonstrates that it is challenging to answer even the seemingly simplest question about a foundation model: what capabilities does it have?
Moreover, it is an open question to what extent these diverse capabilities rely on distinct or shared model mechanisms , akin to algorithmic building blocks within the model. On the one hand, foundation models can be interpreted as single models, which utilize some set of generalizable model mechanisms to perform well across tasks and domains. In this case, a full understanding of their behavior can be gained by identifying and characterising these mechanisms. On the other hand, the ability of foundation models to adapt profoundly distinct behaviors for different tasks suggests that they can also be understood as a large collection of independent expert models, each tailored to a specific task. For example, it seems unlikely that the model parameters that GPT-3 uses to do arithmetic could have much to do with the parameters used to translate from English to French. In this case, explanations of model behavior in one task are therefore not necessarily informative about behavior in other tasks. We refer to this as the one model-many model nature of foundation models (see Figure 23) and argue that understanding where foundation models lie on this spectrum between one and many models will be central to understanding their behavior.
Toward systematizing this area of study, we present and discuss three levels of understanding foundation models [inspired by Marr 1982]: we first discuss the challenges and opportunities in understanding what a model is capable of doing, then why it outputs certain behaviors, and lastly how it does it. Specifically, questions of what aim to characterize the kinds of behaviors that a model can perform without peeking inside the model, while questions of why aim to provide explanations of the model's behaviors in terms of potential causes in the data, and questions of how aim to understand the internal model representations and mechanisms that produce these behaviors. After presenting all three levels, we conclude by discussing potential consequences resulting from the non-interpretability and interpretability of foundation models.
Fig. 23. The one model-many model nature of foundation models: A central interpretability question is to understand where a foundation model lies on the spectrum between one model and many models . As one model, behavior can be made interpretable by identifying and characterising the finite number of generalizable model mechanisms used to produce behaviors across tasks (e.g., mechanisms that assign meaning to words, compare quantities, and perform arithmetic). As many models, explanations of model behavior in one task are not necessarily informative about behavior in other tasks, thus requiring the independent study of behavior in each task.
<details>
<summary>Image 23 Details</summary>
### Visual Description
\n
## Diagram: One Model vs. Many Models
### Overview
The image presents a comparative diagram illustrating two approaches to model design: "One Model" and "Many Models." The "One Model" approach depicts a centralized system with generalized mechanisms, while the "Many Models" approach shows a distributed system with specialized mechanisms for each task. The diagram uses visual metaphors of a circular hub and a grid of individual units to represent these concepts.
### Components/Axes
The diagram is divided into two main sections, visually separated by a series of blue dots.
* **Left Side: "One Model"** - Background color is light blue. Contains a circular diagram with interconnected nodes.
* **Right Side: "Many Models"** - Background color is light pink. Contains a grid-like structure of rectangular units.
* **Text Blocks:** Each side has a descriptive text block above its respective diagram.
### Detailed Analysis or Content Details
**"One Model" Side:**
* **Text Block:** "A finite number of generalizable model mechanisms are combined to produce behaviors across tasks."
* **Central Circle:** A large, light-purple circle labeled "output" is at the center.
* **Input Nodes:** Four oval-shaped nodes surround the central circle:
* "number" (light-orange) - positioned at the bottom.
* "letter" (light-yellow) - positioned to the right of "number".
* "word" (light-green) - positioned to the left of "number".
* "quantity" (light-blue) - positioned above "number".
* **Operators:** Two small, light-blue rectangles labeled "+" and "x" are positioned within the central circle.
* **Connections:** Curved arrows connect the input nodes to the central circle and to each other. The arrows are colored to match the input nodes they originate from. Arrows also point from the central circle upwards.
**"Many Models" Side:**
* **Text Block:** "For each task, distinct model mechanisms are used to produce behaviors; akin to a large collection of individual expert models."
* **Grid Structure:** A grid of approximately 6x8 rectangular units.
* **Color Variation:** Each unit within the grid is filled with a different pastel color (light-blue, light-green, light-yellow, light-orange, light-purple, and a lighter shade of pink).
* **Arrows:** Arrows point downwards towards the grid and upwards from the grid.
### Key Observations
* The "One Model" side emphasizes integration and generalization, with a single point of output derived from multiple inputs.
* The "Many Models" side emphasizes specialization and distribution, with each unit representing a dedicated model for a specific task.
* The color coding in both diagrams helps to visually link inputs to outputs and to differentiate between individual components.
* The diagram does not contain any numerical data or precise measurements. It is a conceptual illustration.
### Interpretation
The diagram illustrates a fundamental trade-off in model design. The "One Model" approach aims for broad applicability and efficiency by combining multiple mechanisms into a single system. However, this can lead to complexity and potential limitations in performance for specific tasks. The "Many Models" approach prioritizes specialization and accuracy by dedicating a separate model to each task. This can result in a more robust and adaptable system, but at the cost of increased resource requirements and potential redundancy.
The diagram suggests that the choice between these two approaches depends on the specific application and the relative importance of generalization, specialization, and efficiency. The analogy to "individual expert models" in the "Many Models" description highlights the benefits of domain-specific knowledge and tailored solutions. The diagram is a high-level conceptual representation and does not provide any quantitative data to support a specific conclusion. It serves as a visual aid for understanding the different design philosophies.
</details>
## 4.11.1 Characterizing behavior.
The simplest understanding of a technology is widely taken to be knowing what the technology does. This seemingly straightforward question is significantly challenging for foundation models, due to the myriad unforeseen behaviors and tasks that these models are capable of performing.
Task-specific neural network models are trained to perform a single task in a single domain, e.g., image classification. Their task and the input and output domains are therefore clear; yet even for these models it can be challenging to know exactly what the model will do, given a particular input. For instance, model behaviors can unexpectedly differ greatly for two perceptually similar inputs [Garg and Ramakrishnan 2020; Jin et al. 2020] or two subpopulations of the same data (stratified, for example, by race or gender [Hovy and Søgaard 2015; Blodgett et al. 2016; Tatman 2017; Buolamwini and Gebru 2018]).
This challenge of characterizing a model's behavior is amplified manyfold for foundation models. The space of tasks that the model is able to perform is generally large and unknown, the input and output domains are often high-dimensional and vast (e.g., language or vision), and the models are less restricted to domain-specific behaviors or failure modes. Consider, for example, the surprising ability of GPT-3 to be trained on large language corpora and to subsequently develop the ability to generate mostly-functional snippets of computer programs. A key challenge for characterizing the behavior of foundation models is therefore to identify the capabilities that it has. Even further, for each task that a foundation model can perform, and there may be many or infinitely many, all
the challenges remain that one faces when trying to understand the behavior of much simpler, task-specific models.
Characterizing each 'task' that a foundation model can perform is further complicated by their one model-many models nature (see Figure 23). Again taking GPT-3 as an example, it was shown that it can be tailored to many tasks through simple prompting (see §4.3: adaptation). Yet, each task can be specified through many possible prompts and slight variations in prompts can result in meaningful changes of model behavior. For instance, the task of sentiment classification of a movie review can be specified by presenting the movie review followed by 'Her sentiment towards the film was...' or 'My overall feeling was that the movie was...'; despite these prompts appearing to pose closely related tasks, GPT-3 will exhibit different response accuracies for each prompt [Zhao et al. 2021]. Observations like these raise important questions regarding the relationship between the characteristics of prompts and the resulting model behaviors. Specifically, can meaningfully different responses to seemingly similar prompts actually be considered as resulting from the same model or do they result from highly distinct model mechanisms, and does characterizing the behaviors of the foundation model (or its adapted derivatives) in one task truly aid in characterizing the behaviors of other possible adaptations of the model?
To identify the capabilities that a foundation model has and those it is missing, researchers can utilize controlled evaluations. Here, domain experts design prompts that are known to require a particular competence and then study the ability of a model to respond correctly to these prompts [Papadimitriou and Jurafsky 2020; Lu et al. 2021a; Kataoka et al. 2020; Wu et al. 2021c; Xie et al. 2021a; Koh et al. 2021]. For example, psycholinguists have designed prompts that require a language model to choose between a grammatically correct sentence and the same sentence with a specific grammatical inaccuracy; knowing whether the model consistently prefers the grammatically correct sentence over its grammatically incorrect counterpart tells us whether the model has the particular grammatical competence required to identify this inaccuracy [Linzen et al. 2016].
Given the huge range of possible capabilities of foundation models, and our current lack of any general method for determining a priori whether a foundation model will have a given capability, bespoke evaluations like these are crucial. They allow exploring the range of behaviors that foundation models are capable of, while requiring minimal model access: we only need to present inputs and receive model outputs, and we need not depend on access to the implementation or parameters of a model. Given the infinitely many desirable and undesirable tasks, subtasks, and behaviors that foundation models may be capable of (or incapable of), characterizing model behaviors and capabilities will be increasingly challenging and important. We believe that instead of relying on a few experts to formulate and test for possible behaviors, it will be critical to extend these types of analyses to test for many more behaviors, in part by opening up this line of exploration to diverse communities and experts in many disciplines, as well as by increasing access to and scale of these evaluations.
## 4.11.2 Explaining behavior.
In addition to characterizing what a foundation model is doing, one can try to characterize why it performs certain behaviors by providing explanations of these behaviors in terms of potential causes in the data. While current explanation approaches, which provide such explanations of behavior, can reveal qualities of inputs that affect a model's responses, they often require full access to the model to do so and are generally limited in their ability to elucidate any general model mechanisms, which foundation models use to respond to many inputs, tasks, and domains.
Current explanatory approaches can generally be understood as distinct models, which are designed to provide an explanation of particular behaviors of another black box model. Importantly,
these approaches are separate from the model whose behavior is analyzed, which by itself is not interpretable. This separation can be problematic, as the provided explanations can lack faithfulness [Jacovi and Goldberg 2020], by being unreliable and misleading about the causes of a behavior [cf. Rudin 2019]. Even further, unsound explanations can entice humans into trusting unsound models more than they otherwise would (for a detailed discussion of trust in artificial intelligence, see Jacovi et al. [2021]). These types of concerns grow as we transition from task-specific models towards the wide adoption of foundation models, as their behavior is vastly more complex.
Current explanatory approaches can largely be divided into either providing local or global explanations of model behavior [Doshi-Velez and Kim 2017]. Local explanations seek to explain a model's response to a specific input, e.g., by attributing a relevance to each input feature for the behavior or by identifying the training samples most relevant for the behavior [Simonyan et al. 2013; Bach et al. 2015; Sundararajan et al. 2017; Shrikumar et al. 2017; Springenberg et al. 2014; Zeiler and Fergus 2014; Lundberg and Lee 2017; Zintgraf et al. 2017; Fong and Vedaldi 2017; Koh and Liang 2017]. Global explanations, in contrast, are not tied to a specific input and instead aim to uncover qualities of the data at large that affect model behaviors, e.g., by synthesizing the input that the model associates most strongly with a behavior [Simonyan et al. 2013; Nguyen et al. 2016].
Local and global explanations have provided useful insights into the behavior of task-specific models [e.g., Li et al. 2015; Wang et al. 2015b; Lapuschkin et al. 2019; Thomas et al. 2019; Poplin et al. 2018]. Here, the resulting explanations are often taken to be a heuristic of the model mechanisms that gave rise to a behavior; for example, seeing that an explanation attributes high importance to horizontal lines when the model reads a handwritten digit '7' easily creates the impression that horizontal lines are a generally important feature that the model uses to identify all sevens or perhaps to distinguish all digits.
Given the one model-many models nature of foundation models, however, we should be careful not to jump from specific explanations of a behavior to general assumptions about the model's behavior. While current explanatory approaches may shed light on specific behaviors, for example, by identifying aspects of the data that strongly effected these behaviors, the resulting explanations do not necessarily provide insights into the model's behaviors for other (even seemingly similar) inputs, let alone other tasks and domains.
Another approach could be to sidestep these types of post-hoc explanations altogether by leveraging the generative abilities of foundation models in the form of self-explanations [cf. Elton 2020; Chen et al. 2018], that is, by training these models to generate not only the response to an input, but to jointly generate a human-understandable explanation of that response. While it is unclear whether this approach will be fruitful in the future, there are reasons to be skeptical: language models, and now foundation models, are exceptional at producing fluent, seemingly plausible content without any grounding in truth. Simple self-generated 'explanations' could follow suit. It is thus important to be discerning of the difference between the ability of a model to create plausible-sounding explanations and providing true insights into its behavior.
## 4.11.3 Characterizing model mechanisms.
Deep understanding of systems is generally taken to mean understanding how a system performs: which knowledge and mechanisms does it contain, and how are these assembled to form the whole?
If this is indeed possible, characterizing the representations within foundation models and the mechanisms that operate on them will be central to satisfying the desire to thoroughly understand these proliferating models; and whether these mechanisms are many and specific or few and generalizable, they are at the core of the ability of foundation models to adopt a wide range of behaviors in varied tasks and domains.
To make the notions of model representations and mechanisms concrete, consider a simple behavior exhibited by GPT-3: It was quickly observed what GPT-3 did when provided with examples of the addition of small numbers and then queried to perform addition of two new numbers: with high probability, it predicted the correct result of the addition [Branwen 2020; Brockman 2020]. When asking why GPT-3 performed as it did, one could find evidence in the input, like aspects of its prompt that highly affected its response (these might be the two numbers to be added, though not necessarily), or aspects of GPT-3's training data that affected its response (these might be examples of addition, though not necessarily). Delving into the model, we may envision a deeper understanding of the mechanisms that GPT-3 uses to add a specific pair of numbers and the mechanism that it uses to add other arbitrary pairs of numbers. We may also envision a deeper understanding of whether these mechanisms are similar to the mathematical notion of 'addition' or merely correlated with this notion.
By understanding individual model mechanisms, we can build up a compositional understanding of complex behaviors of a foundation model. A task slightly more complex than the addition of numbers is solving mathematical word problems, in which numbers come with units and the problem is presented in natural language. Once we understand the mechanism (or mechanisms) by which a model performs addition, we can investigate whether this mechanism is used as an intermediate step in solving word problems. If the addition mechanism is used, we have built up our understanding of how the model solves word problems, we have increased confidence that the foundation model generalizes the notions of quantities and addition (not another correlation or heuristic), and, furthermore, we have increased confidence in our ability to predict the model's why (which parts of the inputs it is attending to) and the output's what (addition of two numbers). If the addition mechanism is not used, we may retain a healthy skepticism that this is truly addition, and we can investigate which representations and mechanisms are used instead.
It is important to be aware that there are many potential cases of more complex and concerning model mechanisms, for instance, the estimation of race from the characters in a name, or the pixels in an image. Establishing evidence of such a mechanism in a foundation model and its use can support a moral or legal responsibility to ban the model from tasks like predictive policing, marketing, loan applications, and surveillance at large.
A plethora of methods have emerged to investigate these internal aspects of neural network models. Typically, these approaches separate the model into nodes (e.g., neurons, layers, or parts of layers), then interrogate either the representations captured in nodes or the mechanisms by which nodes are assembled. Some approaches are hypothesis driven: by hypothesizing that nodes may capture certain information (e.g., a grammatical feature of a word, or the race of a person), one can probe all nodes to quantify how much of that information they make available [Alain and Bengio 2016; Veldhoen et al. 2016; Belinkov et al. 2017; Adi et al. 2017; Conneau et al. 2018; Hewitt and Liang 2019; Hewitt and Manning 2019; Voita and Titov 2020; Pimentel et al. 2020]. Other approaches build on explanatory methods, and, instead of identifying which data cause a certain behavior, they seek to identify which data cause a certain node to activate, or which nodes cause another node later in the model to activate, thereby uncovering collections of model representations and mechanisms [Olah et al. 2020; Mu and Andreas 2020; Carter et al. 2019; Goh et al. 2021]. Taken together, these approaches inspect the interior of models and provide a basis for the ongoing explorations of the behavior of foundation models. Yet, the number of potential representations and mechanisms within foundation models is vast, particularly given their one model-many models nature, and these types of approaches often only capture a small slice of a model's interiority. It is thus an open challenge to expand the discovery of representations and mechanisms and to elucidate those that are most relevant or general for model behavior. As with many approaches to interpreting foundation models, these types of explorations will benefit from
including and supporting more diverse and interdisciplinary investigators and from more accessible, flexible, and scalable methods of discovery.
In summary, we believe that the one model-many models nature of foundation models (recall Figure 23) provides novel opportunities and challenges for current interpretability research: there are many adaptations of a single foundation model, and we simply do not know the extent to which they share common mechanisms. To the extent that mechanisms are shared, understanding foundation models may be a tractable problem of characterizing these mechanisms and their relations. To the extent that mechanisms are independent, each adaptation of a foundation model must be analyzed independently, leading to profound uncertainty about the nature of any new adaptation of the foundation model.
## 4.11.4 Impacts of non-interpretability and interpretability.
Lastly, we would like to highlight that the wide adoption of foundation models is at odds with a recent plea of many interdisciplinary researchers not to use complex black box models for high stakes decisions [e.g., Rudin 2019], but instead to focus on the long-standing development and application of more intrinsically interpretable models.
In the midst of these pleas, work aimed at interpreting foundation models is a double-edged sword. Large machine learning models, and now foundation models, are most often deployed by powerful corporations and institutions, and incremental advances in interpretability can be exaggerated to 'ethics-wash' and continue use of models as though they have achieved interpretability, belying the reality that they remain far below traditional standards of algorithmic interpretability. Moreover, when approaches to interpretability regularly presume easy access to models and their implementation and parameters, interpretability can serve not only as cover for powerful institutions but also centralize model knowledge in the same hands. For those working toward the interpretability of foundation models, it is a responsibility to consistently ask whether one is working toward making foundation models interpretable to researchers and model owners or interpretable to everyone .
Simultaneously, to the extent that foundation models are already being deployed, work on interpretability presents unique opportunities to shift knowledge of foundation models, and thus power, back to datafied and evaluated peoples. Interpretation can facilitate the discovery of societally salient aspects of models. More radically, work creating accessible methods that allow anyone to interpret the behavior of foundation models shifts power to diverse peoples, creating opportunities to investigate models, opportunities to discover aspects of models important to individuals or their communities, and opportunities to meaningfully consent to, improve, or altogether contest the use of foundation models. Finally, it is important for researchers to view the interpretability of foundation models as not only a goal, but a question: research can explore and assess whether the lack of foundation model interpretability is intrinsic and should be deeply studied and widely known as a serious issue discouraging use (or increasing regulation) of these systems, or whether it is possible for future foundation models to uphold a high standard of interpretability for all.
## 5 SOCIETY
The societal impact of foundation models, referring both to the construction of the models themselves and their role in developing applications, requires careful examination. Specifically, we anticipate that foundation models will have wide-ranging societal consequences that are challenging to understand: foundation models are intermediary assets that are not directly deployed, but rather serve as a foundation that is further adapted. As a result, traditional approaches to reasoning about the societal impact of technology are likely complicated; societal impact is easier (but still difficult) to grasp for systems with well-specified purposes. In this chapter, we discuss how we may grapple with and beginning to understand the complexity of the societal impact of models foundation models. Specifically, we discuss (i) the harms with respect to inequity (§5.1: fairness) and misuse (§5.2: misuse), (ii) the impact with respect to the economy (§5.5: economics) and environment (§5.3: environment), and (iii) the broader considerations with respect to the law (§5.4: legality) and ethics (§5.6: ethics).
## 5.1 Inequity and fairness
Authors: Rishi Bommasani, Fereshte Khani, Esin Durmus, Faisal Ladhak, Dan Jurafsky
Fig. 24. The intrinsic bias present within foundation models is the byproduct of various training bias sources ( left ) which, alongside biases introduced during adaptation, determines the extrinsic harms ( right ) experienced by users in the context of specific downstream applications. We emphasize that the same foundation model is the shared foundation for many different applications; its biases propagate to these many applications as a result. Further, since the harms experienced by users are the result of specific adapted models, attributing these harms to the various processes and sources depicted in this diagram is both crucial and challenging.
<details>
<summary>Image 24 Details</summary>
### Visual Description
\n
## Diagram: Bias Flow in Model Development
### Overview
The image is a diagram illustrating the flow of bias from training sources through model development to user experience and potential harm. It depicts how biases can be introduced at various stages and manifest in different forms. The diagram uses icons and text labels to represent these stages and types of bias.
### Components/Axes
The diagram is structured around a flow from left to right, with three main sections: Training Bias Sources, Intermediate Bias Stages, and Outcomes.
* **Training Bias Sources (Left):**
* Training Data (icon: books and a computer)
* Modeler Diversity (icon: group of people)
* Architecture & Objectives (icon: molecular structure)
* **Intermediate Bias Stages (Center):**
* Intrinsic Bias (icon: blue sphere with network lines)
* Adaptation Bias (icon: blue sphere with network lines, syringe, and a red sphere with network lines)
* Sources Per-Model: Data, Mechanism, Modelers (listed as bullet points)
* **Outcomes (Right):**
* User Experience (icon: group of people)
* Extrinsic Harm (icon: red sphere with network lines)
* Representational bias
* Performance disparities
* Abuse
* Stereotypes (listed as bullet points)
Arrows indicate the flow of bias from the training sources to the intermediate stages and finally to the outcomes.
### Detailed Analysis or Content Details
The diagram shows a cyclical flow from the "Training Bias Sources" to "Intrinsic Bias" and then to "Adaptation Bias". The "Adaptation Bias" stage is further broken down into sources: "Data", "Mechanism", and "Modelers". Both "Intrinsic Bias" and "Adaptation Bias" then lead to two outcomes: "User Experience" and "Extrinsic Harm".
* **Training Data:** This is the initial source of bias, represented by an icon of books and a computer.
* **Modeler Diversity:** The diversity of the people building the model is another source of bias.
* **Architecture & Objectives:** The design and goals of the model itself can introduce bias.
* **Intrinsic Bias:** This is represented by a blue sphere with network lines.
* **Adaptation Bias:** This is represented by a blue sphere with network lines, and a red sphere with network lines. The syringe icon suggests a process of injecting or modifying the model.
* **User Experience:** The outcome of the model's use, represented by a group of people.
* **Extrinsic Harm:** The negative consequences of the model, represented by a red sphere with network lines. The listed harms are:
* Representational bias
* Performance disparities
* Abuse
* Stereotypes
The arrows indicate a flow of influence. The arrow from "Training Bias Sources" to "Intrinsic Bias" is curved. The arrow from "Training Bias Sources" to "Foundation Model" is curved. The arrows from "Intrinsic Bias" and "Adaptation Bias" to "User Experience" and "Extrinsic Harm" are straight.
### Key Observations
The diagram highlights that bias can originate from multiple sources during the training phase and can manifest in various harmful ways. The distinction between "Intrinsic Bias" and "Adaptation Bias" suggests that bias can be inherent in the data or model itself, or it can be introduced during the adaptation or deployment process. The diagram emphasizes the importance of considering bias at all stages of model development.
### Interpretation
This diagram illustrates a critical perspective on the development and deployment of machine learning models. It demonstrates that bias is not simply a data problem, but a systemic issue that arises from the entire process, from data collection and model design to implementation and usage. The diagram suggests that addressing bias requires a holistic approach that considers all potential sources and consequences. The use of color (blue for positive/neutral stages, red for negative outcomes) reinforces the idea that bias can lead to harmful results. The cyclical nature of the flow suggests that bias can be perpetuated and amplified if not actively addressed. The diagram serves as a visual reminder of the ethical responsibilities of those involved in developing and deploying AI systems. The diagram does not provide any quantitative data, but rather a conceptual framework for understanding the flow of bias.
</details>
## 5.1.1 Introduction.
Foundation models have the potential to yield inequitable outcomes: the treatment of people that is unjust, especially due to unequal distribution along lines that compound historical discrimination [Hellman 2021]. Like any AI system, foundation models can compound existing inequities by producing unfair outcomes, entrenching systems of power, and disproportionately distributing negative consequences of technology to those already marginalized [Sweeney 2013; Kay et al. 2015; Buolamwini and Gebru 2018; Benjamin 2019; Ajunwa 2019; D'Ignazio and Klein 2020; Crawford 2021]. Here we ask what fairness-related harms relate to foundation models, what sources are responsible for these harms, and how we can intervene to address them. The issues we discuss here are related to broader questions of algorithmic fairness and AI ethics [Corbett-Davies and Goel 2018; Chouldechova and Roth 2020; Hellman 2020; Johnson 2020; Fazelpour and Danks 2021], race and technology [Benjamin 2019; Hanna et al. 2020; Gebru 2021; Field et al. 2021], and the coexistence of society and technology [Abebe et al. 2020].
## 5.1.2 Harms.
Foundation models are intermediary assets with no specified purpose before they are adapted; understanding their harms requires reasoning about both their properties and the role they play in building task-specific models. We delineate intrinsic biases, 90 i.e., properties of the foundation model that indirectly but pervasively affect downstream applications, and extrinsic harms , i.e., harms that arise in the context of specific downstream applications [Galliers and Spärck Jones 1993].
90 We use the word bias to denote the properties of a foundation model that contribute to inequity; we follow Blodgett et al. [2020] in attempting, when possible, to delineate who is harmed and how they are harmed.
Intrinsic biases. Properties of the foundation model can lead to harm in downstream systems. As a result, these intrinsic biases can be measured directly within the foundation model, though the harm itself is only realized when the foundation model is adapted, and thereafter applied, i.e., these are latent biases or harms [DeCamp and Lindvall 2020]. We focus on the most widely studied form of intrinsic bias, representational bias , specifically considering misrepresentation, underrepresentation and overrepresentation. People can be misrepresented by pernicious stereotypes [Bolukbasi et al. 2016; Caliskan et al. 2017; Abid et al. 2021; Nadeem et al. 2021; Gehman et al. 2020] or negative attitudes [Hutchinson et al. 2020], which can propagate through downstream models to reinforce this misrepresentation in society [Noble 2018; Benjamin 2019]. People can be underrepresented or entirely erased, e.g., when LGBTQ+ identity terms [Strengers et al. 2020; Oliva et al. 2021; Tomasev et al. 2021] or data describing African Americans [Buolamwini and Gebru 2018; Koenecke et al. 2020; Blodgett and O'Connor 2017] is excluded in training data, downstream models will struggle with similar data at test-time. People can be overrepresented , e.g., BERT appears to encode an Anglocentric perspective [Zhou et al. 2021a] by default , which can amplify majority voices and contribute to homogenization of perspectives [Creel and Hellman 2021] or monoculture [Kleinberg and Raghavan 2021] (§5.6: ethics). These representational biases pertain to all AI systems, but their significance is greatly heightened in the foundation model paradigm. Since the same foundation model serves as the basis for myriad applications, biases in the representation of people propagate to many applications and settings. Further, since the foundation model does much of the heavy-lifting (compared to adaptation, which is generally intended to be lightweight), we anticipate that many of the experienced harms will be significantly determined by the internal properties of the foundation model.
Extrinsic harms. Users can experience specific harms from the downstream applications that are created by adapting a foundation model. These harms can be representational [Barocas et al. 2017; Crawford 2017; Blodgett et al. 2020], such as the sexualized depictions of black women produced by information retrieval systems [Noble 2018], the misgendering of persons by machine translation systems that default to male pronouns [Schiebinger 2013, 2014], or the generation of pernicious stereotypes [Nozza et al. 2021; Sheng et al. 2019; Abid et al. 2021]. They can consist of abuse , such as when dialogue agents based on foundation models attack users with toxic content [Dinan et al. 2021; Gehman et al. 2020] or microaggressions [Breitfeller et al. 2019; Jurgens et al. 2019]. All of these user-facing behaviors can lead to psychological harms or the reinforcement of pernicious stereotypes [Spencer et al. 2016; Williams 2020].
In addition to harms experienced by individuals, groups or sub-populations may also be subject to harms such as group-level performance disparities . For example, systems may perform poorly on text or speech in African American English [Blodgett and O'Connor 2017; Koenecke et al. 2020], incorrectly detect medical conditions from clinical notes for racial, gender, and insurance-status minority groups [Zhang et al. 2020b], or fail to detect the faces of people with darker skin tones [Wilson et al. 2019; Buolamwini and Gebru 2018]. As foundation models are more pervasively applied, including in high-stakes domains, these disparities can spiral into further, and more severe, harms. Koenecke et al. [2020] discuss how if African American English speakers cannot reliably use speech recognition technologies (e.g., due to inequities in underlying foundation models), this may mean they cannot benefit from certain derivative products (e.g., voice assistants, assistive technologies) and will be disadvantaged if these technologies are used to conduct interviews for employment or transcribe courtroom proceedings. More generally, characterizing these group-level harms (and working towards justice for those harmed) also requires the AI community to improve its understanding of group-based prejudice [Allport 1954] and social groups: we point to relevant work in the social sciences and other communities on moving beyond binary treatments of gender
[Lindsey 2015; Westbrook and Saperstein 2015; Richards et al. 2017; Darwin 2017; Keyes 2018; Hyde et al. 2019; Cao and Daumé III 2020; Dinan et al. 2020], more nuanced treatments of race [e.g., Penner and Saperstein 2008; Freeman et al. 2011; Saperstein and Penner 2012; Saperstein et al. 2013; Penner and Saperstein 2015; Field et al. 2021], better addressing intersectional identities [e.g., Crenshaw 1989; Nash 2008; Gines 2011; Penner and Saperstein 2013; Ghavami and Peplau 2013; Bright et al. 2016; Buolamwini and Gebru 2018; May et al. 2019; O'Connor et al. 2019; Guo and Caliskan 2021], and more modern treatments of disability [e.g., Batterbury 2012; Spiel et al. 2019; Hutchinson et al. 2020].
Additional considerations. To more completely understand the harms of foundation models, further documentation is required of both the intrinsic biases and extrinsic harms; future work should articulate the relationship between intrinsic biases and extrinsic harms [Blodgett et al. 2020, 2021; Goldfarb-Tarrant et al. 2021]. This documentation requires centering stakeholders beyond academics and industry practitioners: the inequitable impact of foundation models will be experienced largely by minority populations, which are underrepresented in both academia and industry. For foundation models specifically, their creation and study likely will be conducted by those with the access and resources required, further emphasizing the importance of venues that center marginalized voices [D'Ignazio and Klein 2020, §5.6: ethics]. In particular, user studies of specific adapted models, when aggregated across applications, can provide compelling and individualized documentation of the harms that derive from the intrinsic biases of foundation models, all while centering individual users. In this way, we imagine the methodologies in humancomputer interaction (HCI), with some adjustment to accommodate the abstraction involved in foundation models, will help center the voices of marginalized communities (further discussion in §2.5: interaction).
## 5.1.3 Sources.
In order to fully characterize and properly intervene on the harms of foundation models, we must be able to trace their source to the properties of the foundation model and the adaptation process, and further decompose to the roles of individual sources of biases [Friedman and Nissenbaum 1996]. Source tracing is vital for attributing ethical and legal responsibility for experienced harm, though attribution will require novel technical research that foregrounds matters such as causality [Pearl 2000] and influence [Koh and Liang 2017].
Data. Data of several types shapes the behavior of applications, and the associated extrinsic harms, based on foundation models: the training data used to train the foundation model, the adaptation data used to adapt the foundation model, and test-time user data/interaction. For all of these data sources, the properties of the data (e.g., toxicity and hate speech [Henderson et al. 2017], abusive language [Waseem et al. 2017], microaggressions [Breitfeller et al. 2019], stereotypes [Voigt et al. 2018]) will manifest in the biases of the foundation model (and its adapted derivatives). 91 Since the training data is the key data source that determines the foundation model and the associated intrinsic biases, we focus on the training data here. At present, the relationship between the training data, along with associated data practices (e.g., data curation, data selection, and data weighting [Paullada et al. 2020; Bender et al. 2021; Rogers 2021]) and the intrinsic biases acquired by the foundation model remains unclear; future work is critically needed to clarify this relationship. Since foundation models generally require training data of immense scale, which poses clear challenges not only to its documentation [Bender et al. 2021] but also comprehensive scientific exploration to
91 In adaptation, which involves labelled task-specific data, biases in the choices of the label space [Crawford 2021] and biases in the annotators who label that data [Geva et al. 2019; Sap et al. 2019] can also contribute to extrinsic harms experienced by users.
articulate the relationship of data biases and model biases, we anticipate new protocols are required to address this scale. Establishing scaling laws for bias, akin to those for accuracy metrics [Kaplan et al. 2020; Henighan et al. 2020], may enable systematic study at smaller scales to inform data practices at larger scales.
Modeling. Modeling decisions (e.g., training objective (§4.2: training), model architecture (§4.1: modeling), adaptation method (§4.3: adaptation)) influence the biases in foundation models and their derivatives, thereby affecting the experienced extrinsic harms. Existing work demonstrates that foundation models amplify training data biases, extending trends seen for machine learning and deep learning models [Zhao et al. 2017; Wang et al. 2019d; Jia et al. 2020; Hashimoto et al. 2018], though much still remains unclear about what and how model properties are responsible for this bias amplification. Further, given that applying foundation models directly may be infeasible (due to their scale), efforts to compress these models or make them more efficient also appear to amplify bias [Hooker et al. 2020; Renduchintala et al. 2021]. Amplification may also be exacerbated by feedback loops, in which foundation models modify societal behavior and induce sociological changes, which modifies subsequent training data; feedback effects of this form tend to exacerbate inequity in other ML applications [Lum and Isaac 2016; Ensign et al. 2018; Hashimoto et al. 2018]. Beyond the explicit decisions made in training and applying foundation models, community values [Birhane et al. 2020] and norms (§5.6: ethics) both indirectly and implicitly [Liu et al. 2021b] shape decision-making in building models. As a result, measuring biases in conjunction with work introducing foundation models [e.g., Brown et al. 2020] and in standard benchmarks [Friedman and Nissenbaum 1996, §4.4: evaluation], as well as conducting user studies with diverse user groups to document experienced harm, are steps towards ensuring that best practices actively emphasize the consideration of bias and inequity.
Modelers. As with all algorithmic systems, poor representation and diversity of stakeholders and marginalized communities in decision-making bodies that develop or apply foundation models is inherently problematic, and may contribute to greater experienced harm for these communities. 92 While difficult to document, existing efforts to develop foundation models suggest this as a possibility: Caswell et al. [2021] demonstrate the flawed data handling of less-represented languages in the multilingual datasets used to train multilingual models and Hutchinson et al. [2020] show that models often contain undesirable biases towards disabled persons. In both instances, these biases and harms may have been noticed earlier by better representation of these parties in developer teams. Further, since end-users are likely more diverse than developers and may notice these concerns earlier, allowing for user feedback to contribute to foundation model design (§2.5: interaction) is an important direction forward.
## 5.1.4 Interventions and recourse.
Addressing, mitigating, and rectifying the inequities associated with technology requires integrating social and technical methodologies [Abebe et al. 2020]. For foundation models specifically, we consider both proactive methods, which change how models are developed and deployed to prophylactically reduce harm, as well as reactive methods, which respond to harm and make changes for the future. At its core, the abstraction of foundation models complicates both aspects: knowing if interventions at the level of the foundation level are successful in reducing harm requires downstream observations at the level of specific deployed applications and recourse in the event
92 We note that diversity, both with respect to disciplinary backgrounds and demographic identities, is of fundamental importance in these high-impact decision-making settings for reasons well beyond the potential improved recognition of fairness-related harms.
of harm requires upstream propagation of both feedback and accountability to foundation model providers.
Intervention. General principles that govern intervention on technological systems apply to the foundation model setting: identifying which sources are most responsible for bias or harm provides the evidence required for targeted action. For example, the urgency of calls for improved diversity in the teams that design, produce, and control technology (e.g., foundation models) and their applications [Longino 1990; Harding 2015; Nielsen et al. 2017; O'Connor et al. 2019; Hofstra et al. 2020; Katell et al. 2020] is further intensified if the lack of diversity is shown to relate to harm [Caswell et al. 2021]. In addition, transparent documentation [e.g., Gebru et al. 2018; Bender and Friedman 2018; Mitchell et al. 2019] and auditing [e.g., Raji and Buolamwini 2019] are similarly critical in providing the impetus for intervention and change [Burrell 2016; Lipton 2018; Creel 2020; Raji et al. 2020; Wilson et al. 2021]. The scale of foundation models, as well as the specifics of their accessibility, introduce new challenges for existing protocols for documentation and auditing that we discuss further in §5.6: ethics.
To date, many of the interventions considered for reducing the inequitable impact of technology, including in the foundation model regime, are methods for technical mitigation that center the data (to obviate reflecting inequities or biases) and modelling decisions (to avoid amplifying data biases) involved. Of specific importance in the foundation model regime is recognizing that these mitigation approaches may target different steps in the pipeline such as the training data [e.g., Lu et al. 2020], modelling objectives [e.g., Zhao et al. 2018]), and adaptation methods and test-time use [e.g., Park et al. 2018; Zhao et al. 2019]. As a result, different approaches may not only be more or less effective, but require action from different entities (e.g., foundation model providers vs. application developers) and more or less intensively affect the expensive training process for these models (e.g., changing the process of creating a foundation model vs. altering it post hoc ). Technical intervention of this form may also target different goals: some interventions, such as changing the training data, aims to reduce intrinsic bias . On the other hand, most work on mitigation in algorithmic/ML fairness instead considers reducing outcome disparities in terms of model behavior, i.e., the outputs of downstream systems that more directly relate to extrinsic harm . Technical mitigation of all forms at present is severely limited: methods that measure or combat intrinsic bias are brittle or ineffectual [Gonen and Goldberg 2019; Ethayarajh et al. 2019; Bommasani et al. 2020; Zhou et al. 2021b; Antoniak and Mimno 2021], methods that measure or combat extrinsic outcome disparities may not align with stakeholder goals [Saha et al. 2020], and there is some evidence to suggest certain types of technical intervention may be simultaneously unsatisfiable [Corbett-Davies and Goel 2018; Kleinberg et al. 2017], impossible [Lechner et al. 2021], or may even exacerbate inequity [Xu et al. 2021]. In spite of this state of affairs, we continue to believe technical methods will still play an instrumental role in addressing the harms that arise in the foundation model regime; in general, we advocate for transparency, especially given that technical mitigation methods may not be able to achieve the intended goals. More broadly, claims of bias and bias mitigation must be made carefully to clearly communicate the status quo to various stakeholders with differing expertise (e.g., application developers building on top of foundation models and policymakers regulating the technology; [Nissim et al. 2020]).
Recourse. Unfortunately, proactive intervention is unlikely to fully resolve all potential harm or inequity that may arise due to foundation models. When harm arises, there is currently no widely-adopted (or legally required) framework for resolving the appropriate recourse for the harmed parties. While certain protocols may exist for specific applications, the abstraction of foundation models again introduces a disconnect: harms likely are partially attributable to both the foundation model providers and the downstream application developers, but allocating this
responsibility to either party remains challenging. More simply, mechanisms are not in place to even communicate these harms to foundation model providers (even if feedback or complaints are raised to application developers). As a result, new norms and standards are needed on how feedback from application developers and end-users should reach upstream to the foundation model providers, how to determine the entities (e.g., foundation model providers, application developers) responsible for these harms, and the relationship to legal responsibility (§5.4: legality). To make progress on this matter, we encourage future work to consult the practices used in other domains (especially those with similar abstractions and multi-entity structures), and we anticipate any standards introduced will likely need to be reasonably dynamic, so that they can be synchronized with the rapidly changing status quo for these models and their applications.
## 5.1.5 Takeaways.
Machine learning has an established trackrecord of inequitable impact, with much of the burden of its harms borne by marginalized communities. Foundation models introduce new challenges to this calculus but, ultimately, for their societal impact to be equitable, significant research and change is required to understand the harms they cause and to meaningfully address and rectify these harms:
- (1) The one-to-many nature of foundation models, i.e., the same few foundation models being used across many applications, means the intrinsic properties of foundation models pervade to many downstream applications. Pernicious biases in these models therefore have out-sized effect on the experienced harms.
- (2) Biases and harms in the foundation model regime originate from many sources (e.g., training and adaptation data, modelling and adaptation decisions, modeler diversity and community values). Attributing the sources for bias and harm is fundamental for questions of intervention and responsibility; attribution requires new technical research to be done reliably.
- (3) The inequities of foundation models are not inevitable, but addressing them requires a multipronged approach comprised of both proactive intervention (e.g., data-centric and modelcentric changes) and reactive recourse (e.g., mechanisms for feedback and accountability).
## 5.2 Misuse
Authors: Antoine Bosselut*, Shelby Grossman*, Ben Newman
Fig. 25. This figure shows the effect foundation models will have on manipulative and harmful content generation, and the implications for detection.
<details>
<summary>Image 25 Details</summary>
### Visual Description
\n
## Diagram: Fake Media (FM) Generation and Detection
### Overview
This diagram illustrates the process of Fake Media (FM) generation, the types of content created, and the effectiveness of FM detection. It depicts a flow from initial prompts to generated content, then to detection results, highlighting both successful and unsuccessful detection attempts. The diagram is divided into three main sections: FM Generator, FM Detector, and a footer with concluding statements.
### Components/Axes
The diagram consists of the following components:
* **FM Generator (Pink Section):** This section shows the input prompts and the resulting generated content.
* **FM Detector (Blue Section):** This section displays the detection results (Detected/Undetected) for each type of generated content.
* **Arrows:** Pink arrows indicate the flow of prompts to the generator. Blue arrows indicate the flow of generated content to the detector.
* **Footer (Bottom Section):** Contains three statements regarding the future of FM generation and detection.
* **Input Prompts:** "The election was rigged and stolen", "Make fake profiles of white women", "Shame that person".
* **Generated Content:** "Fake News" (with the example "Millions of fake ballots were sent in the mail"), "Fake Profiles" (showing three female faces), "Abuse" (with the text "You're stupid!").
* **Detection Results:** "Detected" (indicated by a green checkmark) and "Undetected" (indicated by a red X).
### Detailed Analysis or Content Details
The diagram shows a flow of information from prompts to generated content and then to detection.
1. **Prompt 1: "The election was rigged and stolen"**
* Generates "Fake News" with the example: "Millions of fake ballots were sent in the mail."
* Detected by the FM Detector.
2. **Prompt 2: "Make fake profiles of white women"**
* Generates "Fake Profiles" displaying three images of women.
* Detected by the FM Detector.
3. **Prompt 3: "Shame that person"**
* Generates "Abuse" with the text: "You're stupid!".
* Detected by the FM Detector.
4. **Undetected Case:** One instance of generated content (likely related to "Fake Profiles" or "Abuse") is marked as "Undetected".
5. **Footer Statements:**
* "FM generators will quickly adapt to create new misuse content in diverse topics and modalities."
* "High-quality misuse artifacts will be prevalent in much larger scales."
* "FM detectors may be able to identify some FM-generated content. Manual techniques for tracking misuse will need to adapt."
### Key Observations
* The FM Generator appears capable of creating diverse types of misinformation.
* The FM Detector successfully detects most of the generated content, but there are instances where it fails.
* The footer statements suggest that FM generation is evolving rapidly and that detection methods need to keep pace.
* The diagram highlights the potential for FM to be used for malicious purposes, such as spreading false information, creating fake identities, and engaging in abuse.
### Interpretation
The diagram illustrates a concerning trend: the increasing sophistication and accessibility of fake media generation. While current detection methods can identify some FM-generated content, the diagram acknowledges that generators are rapidly adapting, and detection is not foolproof. The footer statements emphasize the need for ongoing research and development in both FM generation and detection, as well as the importance of manual techniques for tracking misuse. The diagram suggests a potential arms race between FM generators and detectors, with the generators constantly evolving to evade detection. The specific examples used (election rigging, fake profiles, abuse) highlight the potential for FM to be used to manipulate public opinion, damage reputations, and cause emotional harm. The diagram serves as a warning about the dangers of FM and the need for proactive measures to mitigate its risks. The diagram does not provide quantitative data, but rather a qualitative overview of the FM landscape. It is a conceptual illustration rather than a data-driven analysis.
</details>
In this section, we consider misuse of foundation models - situations where people use foundation models as they are intended to be used (e.g., to generate language), but where their capabilities are intentionally leveraged to cause harm to populations or individuals. This definition positions misuse concerns between those of inequity (where models can cause harm without bad intentions; §5.1: fairness) and security (where bad actors exploit unintentional abilities or vulnerabilities in models to cause harm; §4.7: security). Below, we outline how foundation models both enable new forms of misuse and support new tools for misuse detection and mitigation.
## 5.2.1 Foundation models will be misused for harmful purposes.
Advances in the scale (§4.2: training), multimodality (§4.1: modeling), and adaptivity (§4.3: adaptation) of generative foundation models will allow them to be misused to generate high-quality, cheap, and personalized content for harmful purposes. In this section, we discuss these three dimensions within the context of two examples of malicious activity: manipulative content creation and harassment.
Content quality. Foundation models are capable of automatically generating much higher-quality, human-looking content than prior AI methods. They may empower disinformation actors, where states, for example, create content to deceive foreign populations without being transparent that the content is linked to a state. Currently, creating this content often requires hiring people who speak the language of the population being targeted. Governments may outsource content
production to native speakers in the country they are targeting, 93 , 94 but this decision causes real risks for operational security. Foundation models will allow for the creation of content that is often indistinguishable from content created by humans [Kreps et al. 2020; Clark et al. 2021] - and indeed it will be able to do this for a wide variety of languages - enabling both goals of creating content that resonates and maintaining operational security.
In addition to deceiving foreign populations, foundation models' ability to generate high quality synthetic images (deepfakes) or text may be abused to harass individuals. Deepfakes have already been used for the purpose of harassment. For example, Rana Ayyub, an Indian investigative journalist, was targeted by a high-quality deepfake that superimposed her face onto a pornographic video, leading her to leave public life for months. 95 Because foundation models are often multimodal (§4.1: modeling), they could similarly impersonate speech, motions, or writing, and potentially be misused to embarrass, intimidate, and extort victims. 96
Cost of content creation. Foundation models will substantially decrease the costs of content creation, further lowering the barrier to entry for malicious actors to carry out harmful attacks [Brundage et al. 2018]. The budget for one 2017 influence operation that originated in Russia and targeted Americans was $12.2 million [DiResta et al. 2018]. More recently, individuals in Russia paid $75-$200 per article to American freelancers as part of a disinformation campaign. 97 Foundation models will lower these marginal costs. While foundation models, such as GPT-3, may make mistakes when generating content [Buchanan et al. 2021], it will be more feasible to hire a small number of editors to fix them than to hire content creators directly. Initial costs to train foundation models are more significant (§4.5: systems), but these expenses should be manageable for most state actors [Buchanan et al. 2021].
In addition to monetary cost, foundation models require fewer technical skills to achieve highquality results. Current tools, such as video editing software, can enable credible photo or video deepfakes, but require several hours of a skilled user's time to yield quality content. Foundation models lower this barrier to use: their few-shot adaptation capabilities (§4.3: adaptation) enable new modes of interaction for application users (§2.5: interaction) that will allow users to rapidly iterate for content creation.
Personalization. Foundation models will reduce obstacles to creating personalized content. For example, disinformation from Russian individuals that targeted the US in 2016 included highly customized content. Social media posts were crafted to push narratives about Syria (e.g., the U.S. should get out of Syria) that resonated with Black Lives Matter activists [DiResta et al. 2018] (e.g., by suggesting that the U.S. should focus on issues facing the Black community in America, and not on issues in Syria). The same narratives were repackaged to resonate with Texas secessionists [DiResta et al. 2021]. Such a content creation endeavor is costly and time consuming. Foundation models will allow for similar activity, but at scale due to the low cost of adaptation (§4.3: adaptation).
In addition to foundation models allowing an actor to personalize content for niche audiences, they also allow an actor to personalize content to target a single individual - a capability that can be abused by harassers. Foundation models that condition their generations on personal attributes or information can create realistic personalized content, which could be more embarrassing, place victims in more danger, 98 and lead to more successful extortion attempts.
93 https://www.lawfareblog.com/outsourcing-disinformation
94 https://fsi.stanford.edu/content/ira-takedown-20201215
95 https://www.huffingtonpost.co.uk/entry/deepfake-porn\_uk\_5bf2c126e4b0f32bd58ba316
96 https://www.wsj.com/articles/fraudsters-use-ai-to-mimic-ceos-voice-in-unusual-cybercrime-case-11567157402
97 https://www.nytimes.com/2020/09/02/technology/peacedata-writer-russian-misinformation.html
98 https://www.dw.com/en/social-media-uptick-in-honor-crime-in-middle-east/a-56370773
## 5.2.2 Foundation models will be powerful detectors of harmful content.
While the generative capabilities of foundation models will provide ample misuse opportunities, these same abilities may make them strong detectors of harmful content. While these capabilities are equally relevant for detecting human- and model-generated content, we focus on the detection of model-generated content in this section. First, we outline the challenges that current manual detection approaches will face in discovering harmful misuses of foundation model. Then, we propose how the interactive and multimodal representation capabilities of foundation models may make them powerful tools for automatic detection of harmful content. Finally, we discuss the risks associated with deploying automatic detection models in online settings to combat potential foundation model misuse.
Rethinking human interventions. Currently, malicious practices are frequently uncovered (and on social media, sometimes removed) by humans searching the internet to uncover content origination. 99 For example, fake social media profiles commonly steal profile photos from dating sites, which are discoverable through reverse image searches. Similarly, disinformation websites frequently use plagiarized content to mask deceptive content [DiResta and Grossman 2019], which is easily identified by conducting internet phrase searches. Foundation models will limit the efficacy of these detection strategies. Already, relatively unsophisticated disinformation campaigns have leveraged AI-generated photos 100 to remove the possibility of discovery through reverse image search. Tools for assessing whether these photos are AI-generated are available, but foundation models will complicate this work - for text and video as well - challenging manual human discovery techniques [Ippolito et al. 2020; Clark et al. 2021].
Foundation models as detectors. The same abilities of foundation models that make them strong generators of creative content may make them strong detectors of model-generated content. Existing works demonstrate that foundation models can be adapted to detect disinformation from text generators [Zellers et al. 2019b] - which generate statistical textual artifacts [Holtzman et al. 2020] - and that they can be used to evaluate the toxicity levels of their own generations using prompt questions [Schick et al. 2021]. Below, we describe how future foundation models will enable more powerful detection systems of machine-generated, harmful content.
Improvements in the interactive and multimodal interfaces of foundation models will provide new opportunities to improve detection of foundation model misuse for harmful content generation. Current statistical detectors must be retrained and re-deployed to integrate new knowledge about the textual content of misuse strategies [Dinan et al. 2019]. The rapid learning capabilities of foundation models (§4.3: adaptation) may allow them to adapt from human feedback to new misuse strategies that the foundation model was not initially trained to recognize [Lee et al. 2021a].
Simultaneously, the multimodal abilities of foundation models will enable more expressive representation of misuse ecosystems. Prior work has explored how misinformation spreads more rapidly across social networks than authentic content [Starbird et al. 2018; Vosoughi et al. 2018], yielding recognizable signatures when analyzed retrospectively. The multimodal capabilities of foundation models could allow them to jointly learn representations of harmful content and its typical dissemination signature on social networks. These joint representations could provide powerful tools for predicting whether certain types of automatically-generated content are indicative of misuse behavior.
99 https://www.theatlantic.com/ideas/archive/2020/09/future-propaganda-will-be-computer-generated/616400/
100 For a Middle East campaign example, see https://www.thedailybeast.com/right-wing-media-outlets-duped-by-amiddle-east-propaganda-campaign.
Risks of foundation models as automatic detectors. Improvements in automatic detection systems for both model-generated and human-generated harmful content will make these systems more prevalent online, yielding potential negative consequences. Any detection system will have false positive cases where human-generated fair content will be flagged as harmful [Sap et al. 2019; Xu et al. 2021]. The rate at which algorithmic false positives affect users (or groups of users) may cause downstream harm (§5.1: fairness). The adaptive capabilities of foundation models should make systemic false positives easier to address as the model can be locally edited to re-classify those examples (§4.3: adaptation). However, corner cases will likely not be prioritized and recourse will be challenging in these situations.
More broadly, wide-scale deployment of misuse detection systems may engender an 'arms race' between harmful content generators and detectors. Most content generators that use foundation models will lack the resources to develop them individually, and will use systems deployed by larger entities. While terms of use policies should outline acceptable uses of these systems (§5.6: ethics), deployers of foundation models will also need internal detection systems to identify misuse of their products 101 and mitigate them (§5.4: legality). However, there will be fewer controls for misuse actors with the resources to develop their own foundation model-based content generators, putting pressure on platforms to curate the content shared through their distribution channels. Optimistically, content platforms encompass some of the most well-capitalized firms in the world. Their resources may enable the development of detectors beyond the capabilities of most individual misuse agents. This resource advantage could disincentivize individual foundation model development due to the high costs of repeatedly training these systems at scale. However, many instances of foundation model misuse could still be successful even without the largest foundation models to power them, particularly as attackers may leverage the interactive capabilities of foundation models to rapidly generate content that can evade detection.
101 https://www.wired.com/story/ai-fueled-dungeon-game-got-much-darker/
## 5.3 Environment
Authors: Peter Henderson, Lauren Gillespie, Dan Jurafsky
Fig. 26. A visualization of a cost-benefit analysis for deploying a foundation model. The total value of a model can be approximated by first considering the net positive social benefits of the model, as well as any environmental benefits. Then, we subtract the negative energy costs to train and deploy the model, the social cost of the carbon emitted to train the model, and the secondary environmental effects. If the net costs outweigh the benefits, then foundation model developers and large-scale deployers should consider harm reduction strategies. This could include deploying a more efficient model or not deploying the model at all.
<details>
<summary>Image 26 Details</summary>
### Visual Description
\n
## Diagram: Total Value Calculation
### Overview
The image presents a diagram illustrating the calculation of "Total Value". It depicts an equation where Total Value is determined by adding "Social Benefits" and "Environmental Benefits", then subtracting "Energy Costs" and "Social Costs of Carbon", and finally subtracting "Second-Order Effects". Each of these components is further broken down into specific sub-categories represented by icons. The diagram is arranged horizontally, flowing from left to right.
### Components/Axes
The diagram consists of five main blocks:
1. **Total Value:** Represented by a green box with an icon depicting various elements.
2. **Social Benefits:** A blue box containing four sub-categories.
3. **Environmental Benefits:** A light blue box containing four sub-categories.
4. **Energy Costs:** A pink box containing two sub-categories.
5. **Social Costs of Carbon:** A pink box containing three sub-categories.
6. **Second-Order Effects:** An orange box containing four sub-categories.
The equation structure is: Total Value = Social Benefits + Environmental Benefits - Energy Costs - Social Costs of Carbon - Second-Order Effects.
### Content Details
Here's a breakdown of the sub-categories within each block:
* **Total Value:** (Icon: A collection of buildings, trees, and a person) - No specific data points, just a label.
* **Social Benefits:**
* Accessibility (Icon: A person in a wheelchair)
* Improved Healthcare (Icon: A heart)
* Access to Justice (Icon: Scales of justice)
* Government Efficiency (Icon: A building)
* **Environmental Benefits:**
* Ecosystem Monitoring (Icon: A leaf)
* Conservation Efforts (Icon: A whale)
* Assisted Adaptation (Icon: A sunflower)
* **Energy Costs:**
* Energy Grid Strain (Icon: Electrical towers)
* Electricity Use (Icon: A lightbulb)
* **Social Costs of Carbon:**
* Natural Disasters (Icon: A flood/storm)
* Global Warming (Icon: The Earth with rising temperatures)
* Desertification (Icon: A dry, cracked landscape)
* **Second-Order Effects:**
* Future Energy Demand (Icon: A fuel gauge)
* Toxic Wastes and Landfills (Icon: A hazardous waste symbol)
* Desertification (Icon: A dry, cracked landscape)
* Biodiversity Loss (Icon: A fox)
### Key Observations
The diagram presents a qualitative model for calculating "Total Value". It doesn't provide any quantitative data or numerical values. The diagram emphasizes a holistic approach, considering both positive (Social & Environmental Benefits) and negative (Energy Costs, Social Costs of Carbon, Second-Order Effects) factors. The inclusion of "Second-Order Effects" suggests an attempt to account for indirect or long-term consequences. The repetition of "Desertification" in both "Social Costs of Carbon" and "Second-Order Effects" could indicate its significant impact.
### Interpretation
This diagram illustrates a framework for evaluating the overall value of a project, policy, or technology. It moves beyond traditional cost-benefit analysis by incorporating social and environmental considerations, as well as potential long-term consequences. The model suggests that "Total Value" is not simply a matter of economic gain but also encompasses societal well-being and ecological sustainability. The diagram is a conceptual tool for prompting discussion and identifying relevant factors, rather than a precise calculation method. The lack of numerical values implies that the relative importance of each factor is subject to interpretation and context. The diagram is a high-level overview and would require further definition of each component to be practically applied. It is a qualitative model, not a quantitative one.
</details>
Foundation models can potentially lead to many social and environmental benefits, for example in legal domains (§3.2: law), healthcare (§3.1: healthcare), or even tackling climate change [Rolnick et al. 2019]. But because of their scale, they themselves can negatively impact the environment through increased carbon emissions if model creators are not careful [Strubell et al. 2019; Lottick et al. 2019; Schwartz et al. 2019; Lacoste et al. 2019; Cao et al. 2020; Henderson et al. 2020; Bender et al. 2021; Patterson et al. 2021; Lannelongue et al. 2021; Parcollet and Ravanelli 2021]. Addressing such emissions is an imperative: current forecasts show that climate change is occurring more rapidly than previously thought [Masson-Delmotte et al. 2021].
To understand where such emissions can occur in foundation models, we consider their lifecycle. First, they are trained on vast amounts of data, possibly for up to months of time and often distributed across hundreds to thousands of GPUs. Afterwards, they may be adapted to new domains or perhaps distilled into smaller models. All of this can be considered part of the training regime. Models used purely for research may not move beyond these steps. After models have been adapted and/or distilled, they might move on to be deployed into production. At this point many rounds of inference will run through the model until a new model is trained and the cycle repeats.
Each one of these steps has the potential to utilize large amounts of energy and can contribute to carbon emissions. Foundation models can generate large, one-time energy costs and carbon emissions during the initial training phase. For example, the amount of emissions from training one BERT-base model, under some conditions, would only be offset by 40 trees grown for 10 years. 102 And if deployed at scale, foundation models can require substantial energy to service millions of requests 103 - translating to large carbon emissions if nonrenewable resources are used.
102 Strubell et al. [2019] calculate carbon emissions for training BERT on an average energy grid in the U.S. and we use https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator to convert that to equivalent emissions in other domains. We note that this number can vary depending on the energy grid and other considerations [Henderson et al. 2020; Patterson et al. 2021].
103 For example, transformers are already used at scale for search both at Microsoft and Google. See https://www.blog.google/ products/search/search-language-understanding-bert/ and https://azure.microsoft.com/en-us/blog/microsoft-makes-iteasier-to-build-popular-language-representation-model-bert-at-large-scale/.
Therefore, the environmental impacts of certain design decisions for both training and deploying foundation models can be substantial. Even seemingly minuscule decisions, like reducing the number of layers a model has, may lead to significant environmental cost reductions at scale. For example, based on calculations from Henderson et al. [2020], a slightly more energy efficient translation model deployed at the scale of a commercial translation service could save between 78 kgCO2eq and 12,768 kgCO2eq of carbon emissions per day depending on the energy grid used. This is roughly equivalent to the carbon sequestered by 1 to 211 trees grown for 10 years, or the carbon sequestered by .35 to 57.4 acres of forest in one year. 104 Thus the design, deployment, and post-deployment monitoring of foundation models should adequately reflect these risks.
There are of course uncertainties in calculating the amount of energy used or carbon emitted by any given model [Henderson et al. 2020; Cao et al. 2020; Patterson et al. 2021], and other sources of emissions may currently be much greater than those generated by foundation models [Mora et al. 2018]. But if foundation models continue to scale and gain in popularity, they may very well become a significant contributor to carbon emissions. Our goal is to provide a framework for foundation model developers and large-scale deployers 105 to consider how they can mitigate any unnecessary carbon emissions and keep the net social impact of these models positive. We recommend that:
- (1) Carbon impacts can and should be mitigated in many cases. This can be accomplished by training models in low-carbon intensity regions, or by using more efficient models and hardware (§5.3.1: environment-mitigation).
- (2) When all mechanisms for mitigation have been exhausted and mitigation is no longer possible, the costs and benefits to society should be assessed to determine if and when a larger foundation model should be deployed over a smaller, more efficient, model - with the understanding that the up-front costs of a large foundation model may be amortized over the lifetime of the model (§5.3.2: environment-costs).
- (3) Energy, computational, and carbon costs - as well as any efforts taken to mitigate negative impacts - should be clearly reported to inform policymaking and research (§5.3.3: environmentreporting).
## 5.3.1 Carbon impacts can and should be mitigated in many cases.
The carbon impacts of training foundation models differ from the impacts of deploying them for inference. Model training has no latency requirements, so training can be moved across energy grids with relative ease in cloud environments. Every energy grid has its own carbon intensity the amount of carbon emitted per kilowatt-hour of energy used. For example, Québec has an extremely low carbon intensity due to its reliance on hydroelectricity, while Estonia's energy grid has an extremely high carbon intensity due to its reliance on shale oil (though that is changing quickly) [Henderson et al. 2020]. Recent research has even suggested that the top 5% of polluting power plants contributed 73% of all electricity-based emissions [Grant et al. 2021]. Thus, while training foundation models can be quite energy intensive, researchers have demonstrated that the carbon impacts of these models can be partly mitigated by selecting energy grids with minimal carbon emissions [Henderson et al. 2020; Lacoste et al. 2019; Patterson et al. 2021].
104 Sequestration estimated via https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator, but may be larger depending on other estimation methods. More efficient energy grids will emit less carbon, resulting in wide estimated ranges of impacts.
105 We focus on model developers and large-scale deployers, like those who build production systems on top of foundation models, because they are most able to make meaningful changes to reduce energy use and carbon emissions. A single change by these actors - like using a more efficient model - can scale to massive carbon savings, which would otherwise require a massive campaign to reach all downstream model users.
Carbon offsets have also been proposed as a stopgap until carbon-free renewable electricity is available at all data centers. This strategy involves reducing carbon emissions in one activity to offset the emissions from another. But most - if not all - carbon offsets are a strictly worse solution than not emitting CO2 in the first place [Holl and Brancalion 2020]. Some carbon offset programs can even have a negative impact. For example, studies of forest planting campaigns (often a source of carbon offsetting) show that they can do more harm than good. They can yield monocultures (the use of one particular species of tree) that diminish the biodiversity of the region and reduce carbon storage in the forest soil [Heilmayr et al. 2020; Hong et al. 2020b]. This could result in more carbon emissions when using carbon offsets than if the original carbon had never been emitted in the first place. Therefore, when training or deploying a foundation model, we recommend designing for as little carbon emission as possible up-front, rather than simply relying on carbon offsets to cancel emissions.
When it is not possible to run in low-carbon regions, other mitigation strategies should be leveraged, reducing unnecessary energy usage. This includes:
- using more efficient hardware, 106
- using mixed-precision training [Micikevicius et al. 2017] or quantization [Gholami et al. 2021],
- using more efficient architectures (e.g., using an evolved transformer over a vanilla transformer architecture; or using sparse models) [So et al. 2019; Patterson et al. 2021; Mostafa and Wang 2019],
- distilling models and using distilled models (e.g., [Sanh et al. 2019]),
- and utilizing other optimization strategies that will reduce energy costs (see more discussion in §4.5: systems).
Maintainers of open source projects and cloud compute should strive to set their default settings to the most efficient possible, since 'green defaults' are known to be the most effective mitigation strategies (see discussion in [Henderson et al. 2020]). Other mitigation strategies can be found in recent literature [Strubell et al. 2019; Lacoste et al. 2019; Schwartz et al. 2019; Henderson et al. 2020]. We also note that reducing and mitigating energy usage also has the added benefit of making models more accessible to those with limited compute access (see §5.6: ethics for more discussion).
However, when a model is mainly used for inference, e.g., deployed in a production application, it often cannot be moved to a less carbon-intensive energy grid for low-latency applications. In addition to using the mitigation strategies specified above, in this case it is important to weigh the benefits of the proposed foundation model versus a more energy efficient alternative. We discuss this further in the subsequent section.
## 5.3.2 Costs and benefits should be assessed before using foundation models.
After taking as many steps as possible towards mitigation (or where mitigation is not possible), it is vital to assess the required size of a foundation model - or whether a foundation model should be used at all. This cost-benefit analysis should consider:
- (1) Is the social cost and environmental cost from deploying the foundation model greater than the social benefit of the model?
- (2) Would another, computationally simpler and cheaper approach achieve comparable social benefit (e.g., a much more efficient foundation model, or perhaps simple baseline)?
A simplified scheme for assessing this trade-off considers the overall impact of a model 𝑀 as:
106 Notably, California now regulates computers with inefficient GPUs for this reason, requiring that they stay below 30-100 kWhs/year, depending on the manufacturing date and computer type. See Sections 1601-1608 of California's Appliance Efficiency Regulations (Title 20).
$$V ( M ) = S ( M ) - C ( M ) - E ( M ) - O ( M ) .$$
Figure 26 represents this equation and the costs and benefits that may enter each variable. Here, 𝑀 is the model and 𝑆 is the net social benefit, as well as environmental benefit, in dollars. 𝑆 can be increased by improving healthcare, access to justice, decreasing poverty, improving environmental monitoring, aiding ecosystem conservation efforts, and so on.
𝐶 is the social cost of carbon from energy use. This represents the future harm to society from the carbon released as a present-day monetary value. The upper bound U.S. Environmental Protection Agency (EPA) estimate from 2017 for the social cost of carbon was $105 (in 2007 U.S. dollars) per metric ton of CO2 emitted. 107
𝐸 is the energy cost of the model. For example, in April 2021, the average U.S. residential energy cost was about $0.1376 per kWh. 108 Added to this variable could be the costs from increased strain on the energy grid. For example, a recent study suggested that the cost per energy grid interruption event, normalized by average demand, could be as high as $15.9 per average kW [Sullivan et al. 2015]. 109
𝑂 is the social cost of other second order environmental effects. This could include:
- The compounding carbon impacts from increased chip demand and chip production [Gupta et al. 2021a].
- Other environmental impacts of chip manufacturing, like the creation of toxic waste sites in Silicon Valley, whose health effects are unequally distributed to socially vulnerable populations [Stewart et al. 2014], or pollution from manufacturing in Taiwan that has been linked to chronic health problems [Tu and Lee 2009; Lin et al. 2016].
- The compounding effects of climate change that are not already included in the SCC model. For example, these effects could include accelerated desertification [Huang et al. 2016], rapid ecosystem changes that put many species at risk of extinction [Urban 2015], and increased carbon emissions due to melting permafrost [Schuur et al. 2015].
- Unnecessary strain on chip production capacities. Recent chip shortages have led to work stoppages in automobile manufacturing. 110 There is no evidence to suggest that increasing demand for ML-optimized chips led to this shortage. 111 But such considerations fall into second order effects, where researchers might weigh whether the risks, however slight, of contributing to such negative impacts are worth using or deploying a large model. 112
It is important to consider in this analysis that the economic benefits and social costs of carbon could be distributed unequally across communities, with poorer communities being impacted more heavily by climate change and wealthier communities being benefited by a model [Bender et al. 2021]. 113 As such, when conducting the Equation 7 analysis, one should consider the benefits and
107 See https://19january2017snapshot.epa.gov/climatechange/social-cost-carbon\_.html. But note that the social cost of carbon can be a contentious metric [Stern and Stiglitz 2021]. By using a favorable discount factor, one can reduce carbon costs. As such, it can the calculation of this metric can vary across methodologies.
108 https://www.eia.gov/electricity/monthly/epm\_table\_grapher.php?t=epmt\_5\_6\_a
109 Like the social cost of carbon, calculation of these costs can fluctuate across modeling methodologies.
110 https://www.reuters.com/business/autos-transportation/ford-shut-some-n-american-plants-few-weeks-chipshortage-2021-06-30/
111 Though recent reports have suggested that demand for datacenter chips have surpassed the gaming sector. See https://www.nextplatform.com/2020/08/21/the-local-maxima-ascension-of-datacenter-at-nvidia/.
112 Like for other metrics described previously, there is uncertainty as to how these impacts might be calculated and attributed to models.
113 See also, https://www.un.org/sustainabledevelopment/blog/2016/10/report-inequalities-exacerbate-climate-impactson-poor/ and https://blogs.imf.org/2020/12/02/how-artificial-intelligence-could-widen-the-gap-between-rich-and-poornations/.
Fig. 27. A hypothetical example of amortized fine-tuning showing the point at which a foundation model (in this case BERT Base) will have lower energy costs than a transformer model trained from scratch. We estimate the up-front energy cost for training BERT from Strubell et al. [2019], and cost for fine-tuning a downstream task from Chaudhary et al. [2020]. We compare against the linearly increasing cost of training a transformer from scratch, from Strubell et al. [2019]. If BERT is used for less than ∼ 80 tasks, the up-front energy costs are not recovered. After that point, BERT is more energy efficient than the model trained from scratch.
<details>
<summary>Image 27 Details</summary>
### Visual Description
\n
## Line Chart: Hypothetical Energy Usage Curve Amortized Over Tasks
### Overview
This chart depicts the hypothetical relationship between the number of tasks finetuned and the energy usage (in KWH) for two different base models: BERT Base and Transformer Base. The chart aims to illustrate how energy consumption scales with the number of tasks a model is adapted to.
### Components/Axes
* **Title:** "Hypothetical Energy Usage Curve Amortized Over Tasks" (Top-center)
* **X-axis:** "Number of Tasks Finetuned" (Bottom-center), ranging from 0 to approximately 150.
* **Y-axis:** "Energy (KWH)" (Left-center), ranging from 0 to approximately 4000.
* **Data Series 1:** "BERT Base" (Blue line)
* **Data Series 2:** "Transformer Base" (Red line)
### Detailed Analysis
The chart displays two distinct curves representing energy usage for each model.
**BERT Base (Blue Line):**
The BERT Base line starts at approximately 0 KWH at 0 tasks. It exhibits a steep upward slope initially, reaching a peak around 2500 KWH at approximately 50 tasks. After this point, the slope decreases significantly, and the line plateaus, reaching approximately 3000 KWH at 150 tasks. The curve appears to be approaching an asymptote.
**Transformer Base (Red Line):**
The Transformer Base line also starts at approximately 0 KWH at 0 tasks. However, it demonstrates a consistently increasing, almost linear, slope throughout the entire range of tasks. At 150 tasks, the Transformer Base line reaches approximately 4000 KWH.
**Data Points (Approximate):**
| Tasks Finetuned | BERT Base (KWH) | Transformer Base (KWH) |
|---|---|---|
| 0 | 0 | 0 |
| 25 | 800 | 1800 |
| 50 | 2500 | 3000 |
| 100 | 2800 | 3600 |
| 150 | 3000 | 4000 |
### Key Observations
* The Transformer Base model consistently consumes more energy than the BERT Base model for any given number of tasks finetuned.
* BERT Base exhibits diminishing returns in terms of energy consumption as the number of tasks increases. The energy usage plateaus after a certain point.
* Transformer Base shows a linear increase in energy consumption with the number of tasks, suggesting no significant diminishing returns within the observed range.
* The initial energy cost for BERT Base is lower, but the Transformer Base scales more predictably.
### Interpretation
The chart suggests that while BERT Base may be more energy-efficient for a small number of tasks, the Transformer Base model's energy consumption scales more linearly and may become more efficient for a large number of tasks. This could be due to differences in model architecture, training methodologies, or the inherent complexity of the tasks being finetuned. The plateauing of the BERT Base curve indicates that further finetuning yields minimal increases in energy consumption, potentially suggesting that the model has reached its capacity for learning from additional tasks. The data highlights a trade-off between initial energy cost and scalability, which is crucial for considering the environmental impact of large language models. The "Hypothetical" nature of the chart suggests these are modeled results, and may not reflect real-world performance. The chart is a useful tool for comparing the energy efficiency of different model architectures in the context of task finetuning.
</details>
harms to society more broadly rather than for a given organization or country. In this case 𝑉 ( 𝑀 ) can be viewed as a distribution and should ideally be evenly distributed across the population. In cases where the distribution is highly uneven - for example where all the benefits fall to the model designer while all the harms fall to populations that will never benefit from the model - the designer should spend substantially more effort on mitigation before deploying the model.
There is, of course, some uncertainty in which methodology to use when valuing each component of Equation 7. Empirical estimates for many of these terms can range in multiple magnitudes depending on the data source and modeling choice for the phenomena, such as the different mechanisms for evaluating the social cost of carbon. And of course additional externalities, that may be difficult to quantify monetarily, will continue to need to be considered. The key takeaway of this cost-benefit analysis, however, is not the dollar valuation of each term in the equation, but rather the existence of and relative importance of each of these effects. Our goal is to provide a high-level framework for beginning to consider these trade-offs. Future research may give more guidance on how to quantify each of these values.
Finally, we note that these factors should also be evaluated over the lifetime of the model, not on a per-run basis. Consider an alternative baseline model that must be trained from scratch for every new task. The baseline may well require an expensive hyperparameter search to achieve equivalent performance on downstream tasks. In contrast, the foundation model places the brunt of the costs on the initial pretraining procedure, with fine-tuning perhaps being much simpler and more energy efficient. Over the lifetime of the foundation model, it could be more carbon efficient than the baseline (Figure 27). Even more efficient adaptation mechanisms could improve this amortization further (see §4.3: adaptation).
The efficiency of adaptation, however, is not guaranteed. It may be true that some foundation models will never be more efficient than a particular baseline, even when amortized over many tasks. For example, it cannot be assumed that a smaller model with fewer parameters will translate to energy efficiency improvements. Due to increased hyperparameter tuning costs or other optimizations, the number of parameters has been shown not to correlate with energy efficiency in some cases [Zhou et al. 2020; Henderson et al. 2020]. Therefore, foundation model developers should rigorously assess the efficiency of their models and adaptation mechanisms before beginning large-scale training efforts.
The framework in this section is meant to guide the reader in thinking about the environmental and societal trade-offs in training and deploying their model, but there are other substantial social justice considerations involved in deploying a foundation model, discussed in §5.6: ethics. §5.5: economics also discusses in more detail the dynamics of social welfare from algorithm deployment.
## 5.3.3 Carbon/energy impacts should be systematically reported.
Acost-benefit analysis cannot be conducted unless researchers and engineers working on foundation models report the computational, energy, and carbon costs of their models. We encourage foundation model developers, providers, and curators to report these metrics, as well as what carbon reduction strategies were used in the making of the foundation model. See [Henderson et al. 2020; Lottick et al. 2019; Lacoste et al. 2019; Schmidt et al. 2021; Anthony et al. 2020] for examples of a Carbon Impact Statement and for tools that can facilitate this reporting. For researchers, such reporting can occur at publication time, but we also encourage industry actors to adopt transparency mechanisms to report these metrics for their deployed models. 114 This will help set policy recommendations within industry and academia, as well as help downstream users identify carbon-friendly usage patterns. Standardized reporting will also aid in determining which models are accessible to those with limited compute access (see §5.6: ethics for more discussion on accessibility).
To encourage more reporting of energy and carbon impacts, we suggest, among other strategies: giving green badges at conferences, requiring reporting of relevant metrics for submission to conference venues, lobbying large-scale deployers of foundation models to provide more transparency, and generally shifting professional norms in academia and industry towards standard reporting of these metrics (see more discussion on professional norms in §5.6: ethics and more discussion on reporting mechanisms by Henderson et al. [2020]).
114 A small step toward this has been taken by some cloud compute providers that identify the most carbon friendly cloud regions. See, for example, https://cloud.google.com/blog/topics/sustainability/pick-the-google-cloud-region-with-thelowest-co2.
## 5.4 Legality
Authors: Neel Guha, Peter Henderson, Lucia Zheng, Mark Krass, Daniel E. Ho
In this section, we describe how US law may influence, constrain, or foster the creation and use of foundation models. 115 We note that the legal landscape surrounding algorithmic tools remains uncertain. We highlight issues pertaining to (1) model training, (2) liability for model predictions, and (3) protections for model outputs.
Though understanding how the law affects foundation models is crucial, it is important to recognize that the law cannot be the only lens through which we evaluate the construction, maintenance, and use of foundation models. Ethical frameworks are necessary to understand where legally permissible applications of foundation models may still be ill-advised for the harms they inflict and are discussed in more depth in §5.6: ethics and §5.1: fairness. Studying the potential for misuse and possible security concerns (see §5.2: misuse and §4.7: security) is critical for preventing harmful outcomes ex ante , as opposed to the ex post treatment that legal mechanisms often provide.
## 5.4.1 Training.
Training foundation models will require accumulating vast amounts of multi-modal data, raising questions around data collection and data use.
First, the ability for model creators to grow datasets via web scraping will be governed by the manner in which courts will interpret terms of service provisions and, notably, the U.S. Computer Fraud and Abuse Act (CFAA), which criminalizes accessing a server 'without authorization' [Wajert and Rottman 2019]. Courts are in conflict on these questions, and recent cases have sought to clarify the circumstances under which web scraping may be barred. 116 The restrictiveness of data access would fundamentally affect the diversity of data practitioners can use to train foundation models [Levendowski 2018].
Second, much of the data contained in training sets will be copyrighted and potentially protected by intellectual property law. However, copyright law recognizes exceptions when individuals may be permitted to use copyrighted material. 117 Some scholars believe that the legal permissibility of training datasets will largely rest on whether courts interpret the process of model training as 'transformative' under fair use doctrine [Lemley and Casey 2020]. Though the question of what qualifies as transformative is highly context dependent, the general rule is that transformative uses are those 'that add something new, with a further purpose or different character, and do not substitute for the original use of the work" [Office 2021]. Already, the recently released Github Copilot tool is bringing these arguments to the fore [Gershgorn 2021].
Finally, some training datasets may run afoul of privacy laws. Illinois, for instance, enables individuals to sue for improper collection or use of biometric data (e.g., retina or iris scans, fingerprints, voiceprints, or scans of hand or face geometry). 118 Foreign privacy laws like the E.U.'s General Data Protection Regulation (GDPR) - which will affect American model creators if datasets contain information from E.U. citizens - would require data subjects to be informed about the purpose of data collection. Further issues could arise for laws like the California Consumer Protection Privacy Act (CCPA), which provide individuals with a 'right to be forgotten,' raising questions as
115 Our perspective here centers on US law and legal frameworks. Discussions of the implications of foundation models with respect to other countries may consequently take different perspectives.
116 Van Buren v. United States , 141 S.Ct. 1648 (2021).
117 See, e.g., , 17 U.S.C §107 to 112.
118 IBM is the defendant in a current class action alleging that IBM's collection and use of this data (including for machine vision purposes) violates this statute. See Class Action Complaint at 2, Vance v. Int'l Bus. Machines Corp., No. 20 C 577 (N.D. Ill. filed Jan. 24, 2020).
to whether model creators will need to 'remove' training data from models [Villaronga et al. 2018; Ginart et al. 2019].
## 5.4.2 Output liability.
Though foundation models themselves are task agnostic, fine-tuned models - or the representations learned by foundation models themselves - may be used for traditional prediction tasks. Where these tasks form components of larger decision-making systems, foundation models will thus influence actions, decisions, or policies. When these result in harm, model creators - and the individuals operating them may be legally responsible.
Embedding foundation models in physical systems (e.g., self-driving cars, electric grid management, medical diagnostics, etc.) may result in physical harm to individuals. Here, courts will likely resolve questions of liability under tort doctrine [Lemley and Casey 2019; Selbst 2020]. Key open questions include the interplay between the liability of users, foundation model providers, and application developers, as well as the standards courts will use to assess the risk profile of foundation models. Deployments in particularly sensitive domains (e.g., medicine) will require regulatory approval, and the development of standardized processes to assess safety [Wu et al. 2021g].
Fine-tuned foundation models that classify individuals in ways that correlate with protected attributes (e.g., race, gender) may face challenges under civil rights laws. Scholars have noted that claims for disparate treatment resulting from foundation models may be brought in the context of hiring, housing, or credit lending [Gillis and Spiess 2019; Scherer et al. 2019]. Exactly how courts will adjudicate these issues is far from clear. Scholars have noted for instance, that the courts' traditional views on 'discrimination' would actually prevent machine learning practitioners from implementing many algorithmic fairness techniques [Xiang 2021; Ho and Xiang 2020]. 119
U.S. law recognizes special privileges and limits on governmental entities. Thus, the use of foundation models by governmental entities - at a local, state or federal level - will implicate special considerations, in addition to equal protection claims. The use of models for risk assessment or in other settings which result in a deprivation of life, liberty, or property - will invite procedural due process claims. 120 When models are used by administrative agencies (e.g., the Environmental Protection Agency) for instance, plaintiffs may allege that such use violates basic standards of due process, reasonableness / non-arbitrariness, and transparency.
## 5.4.3 Legal protections for outputs.
Model outputs - and by extension the model creators responsible for the models - may also be afforded certain legal protections. First, content produced by generative models may implicate free speech issues. The extent to which courts will find First Amendment protections for machine generated content is unclear. Scholars have discussed a number of open questions, including whether 'AI speech' is protected [Massaro et al. 2016] or if model outputs are in effect the human programmer's speech [Kajbaf 2019]. Others have noted the possibility of disclosure requirements (akin to safety disclosures for pharmaceutical drugs or other substances), also implicating speech doctrine, under which models would be forced to share with listeners that their content is machine generated [Lamo and Calo 2019]. These issues could have wide ranging consequences, affecting whether individuals can use foundation models to mass produce speech, or whether model creators could be held liable for content generated by foundation models.
119 For more information on how models may embed certain biases, see §5.1: fairness.
120 Procedural due process recognizes that plaintiffs usually have certain basic rights during any deliberation that will deprive them of life, liberty, or property (e.g., the right to cross-examine adverse witnesses).
Second, there is uncertainty regarding who may assert ownership over model outputs. Existing copyright law does not recognize computer programs as authors, and hence, does not afford copyright protection to 'work' created by computer programs [Grimmelmann 2015]. As a result, scholars have advocated for a variety of approaches. Some have argued that, depending on the circumstances, both the human creator of a program and its human user may have viable claims to being the 'author' of the program's output [Ginsburg and Budiardjo 2019].
As models are increasingly used in the process of 'creation' - from artistic endeavors to more mundane settings like news filings - disputes over the ownership of machine generated content will become more commonplace.
While our analysis above only skims the surface of the legal issues implicated by foundation models, the resolution of these questions will be critical to the construction, use, and deployment of foundation models, or, to borrow Larry Lessig's phrase, how 'code is law' [Lessig 2000].
## 5.5 Economics
## Authors: Zanele Munyikwa, Mina Lee, Erik Brynjolfsson
Foundation models have the potential to substantially improve overall living standards by increasing productivity and innovation. These models can be deployed to substitute for human labor, augment humans, or help in the discovery of new tasks and opportunities, which can lead to increased concentration of ownership and power, or more decentralization. On a broader level, the result can be either increased inequality due to potential centralization (§5.1: fairness, §5.6: ethics), or more broadly shared prosperity due to the easier adaptation of foundation models for a wide range of applications (§1: introduction). The ultimate outcomes on all these dimensions are not dictated solely by technology or economics, but by the choices and actions of technologists, policymakers, managers, workers, and other members of society.
Foundation models can be thought of as what economists refer to as a general-purpose technology [Bresnahan and Trajtenberg 1995]. General-purpose technologies refer to technologies like the steam engine and electricity, which drive waves of transformation and productivity growth due to their pervasiveness, improvement over time, and ability to spawn complementary innovations (a host of products and services that revolve around one core product). While foundation models may not be pervasive at the moment, they seem poised to be the basis of widespread technological innovations, and have the key hallmarks of a general-purpose technology. As a result, these models are likely to be economically important. In considering the impact of foundation models on the economy, we will focus on three broad areas of impact: productivity, wage inequality, and ownership.
## 5.5.1 Productivity and Innovation.
Foundation models are likely to substantially increase both productivity and innovation. Productivity growth is one of the main contributing factors to boosting living standards, as it increases the wealth of nations and addresses a host of challenges from poverty and healthcare to the environment and education.
Productivity is defined as output per unit input. 121 One way to boost productivity is to reduce the denominator; for instance, enabling a company's advertisements to be written with fewer copywriters or fewer labor hours per copywriter lowers the number of units of input. Productivity can also be boosted by increasing the numerator, for instance by enabling a software developer to write more code in a given time. If the growth in the numerator is great enough, this can lead to more people developing software, not fewer [Autor 2015]. In many tasks, we have already observed machine learning systems increasing productivity. For instance, an autocomplete system for clinical documentation reduces keystroke burden of clinical concepts by 67% [Gopinath et al. 2020]. Likewise, the potential for foundation models to affect productivity spans almost every industry and many occupations. Considering language alone, an analysis of U.S. occupations using the US Department of Labor's O*NET database shows that many occupations involve the types of language-related work that could be affected by foundation models. Approximately 13% of occupations have a primary task that is related to writing, and the total wage bill of these occupations (annual salary multiplied by the number of individuals employed in the occupation) is over 675 billion dollars. However, the potential impact of foundation models extends beyond language. They will also have effects on diagnostic imaging in medicine, graphic design, music 122 ,
121 Note that when properly measured, productivity is not just a matter of counting units produced or hours work, but also accounts for quality changes. Therefore, an increase in quality for a given amount of labor, such as more interesting fiction, also counts as an increase in productivity.
122 https://www.landr.com/
and many other tasks where people are creating something that is similar to something else that already exists [Winkler et al. 2019; Ramesh et al. 2021].
Perhaps the most profound, if still speculative, effect of foundation models is their potential to enhance creativity and boost the rate of innovation itself. For instance, DALL·E [Ramesh et al. 2021] could transform the market for illustrations much as inexpensive cameras revolutionized photography. If these models enable humans to develop new ways to write new songs and novels (§2.5: interaction), discover variants of drug molecules (§3.1: healthcare), extend patents (§3.2: law), build innovative software applications, or develop new business processes, then not only the level of productivity, but the rate of growth of productivity would be increased. In this way, foundation models have some of the characteristics of the ideas or blueprints in Paul Romer's growth models [Romer 1990], or even meta-ideas (ideas about ideas) which, unlike most other goods, are non-rival, thus speeding growth.
It is worth noting that changes in productivity are not always visible in the official statistics, because many aspects of input and output are difficult to measure [Brynjolfsson and Collis 2019]. As a result, the benefits and costs of foundation models will not be fully captured by traditional productivity metrics, nor by related metrics like gross domestic product (GDP) or price levels (the average of current prices across the entire spectrum of goods and services). This is especially true for general purpose technologies historically, since they are catalysts for a cascade of secondary innovations that often transform the set of goods and services in the economy, and even the nature of production and innovation over a period of years or even decades.
## 5.5.2 Wage inequality.
Even if foundation models increase average productivity or income, there is no economic law that guarantees everyone will benefit. In part, this is because not all tasks will be affected to the same extent. More importantly, the effects of foundation models on the demand for labor (and thus employment and wages) can be either positive or negative, regardless of productivity growth [Brynjolfsson and McAfee 2011; Brynjolfsson and Mitchell 2017]. When a technology substitutes for human labor in completing tasks, it tends to reduce demand for the workers doing those tasks. This depresses employment and wages. However, when a technology complements labor, or facilitates the creation of new opportunities or tasks, it tends to increase labor demand [Acemoglu and Restrepo 2019]. Employment can (and often does) go up, even as productivity increases. For instance, the invention of the airplane created the demand for an entirely new occupation, the airline pilot. In turn, the development of jet engines was complementary to human pilots, further increasing demand for them. Similarly, the effects of foundation models on employment, wages, and income inequality will differ depending on how they are used.
While the industrial revolution mainly transformed physical work, foundation models are likely to transform tasks involving cognitive work, like content creation and communication. In general, since foundation models are intermediary assets that often possess strong generative capabilities, we envision that they will be able to augment humans in many creative settings, rather than replace humans as there are still significant limitations in using these models stand-alone for open-ended generative tasks [See et al. 2019]. As we describe in §2.5: interaction, foundation models may also power systems that users can leverage to co-construct novel forms of art or more efficiently prototype new applications. Fluid human-machine and human-in-the-loop interaction will require advances in interface design (§2.5: interaction) as well as fundamental improvements in the interpretability (§4.11: interpretability) and robustness (§4.8: robustness) of these models, so that humans can understand model behavior and expect models to perform well in diverse contexts.
## 5.5.3 Centralization.
Another key determinant of foundation models' economic impact is who owns data and models. In particular, pushing the frontier of foundation models has thus far primarily been the purview of large corporate entities. As a result, the ownership of data and models are often highly centralized, leading to market concentration (§5.6: ethics). In turn, this can lead to significant centralization of decision rights and power, reducing income and opportunities for those who don't have ownership. This centralization of power can lead to an equilibrium where fewer people have social and economic mobility and opportunity, a situation that Brynjolfsson [2022] calls "The Turing Trap". To counterbalance this centralization, there have been grassroots efforts to open source AI research such as Masakhane, EleutherAI, and HuggingFace, or build foundation models through distributed training. However, it likely that the gap between the private models that industry can train and the ones that are open to the community will remain large due to foundation models' dependence on massive amount of data and computational resources (§5.3: environment). 123
## 5.5.4 Other considerations.
This short chapter is not meant to be comprehensive of all the economic effects of foundation models. In addition to affecting productivity, wage inequality, and ownership, foundation models may also have significant effects on job quality and job satisfaction. For instance, they may increase job satisfaction by automating repetitive, uninteresting parts of work, or decrease satisfaction by increasing the pace of work, thereby inducing more frequent burnout. As discussed in §5.1: fairness and §5.6: ethics, they can also amplify and perpetuate bias, often in unexpected ways, or be used as a tool for reducing it. Foundation models can facilitate global trade and remote work, just as earlier uses of machine translation systems had significant effects in these areas [e.g., Brynjolfsson et al. 2019]. There may also be significant environmental effects (§5.3: environment), as well as unexpected and unanticipated effects on the rate and direction of occupational change and business transformation in an economy. More broadly, given the emergent capabilities of foundation models, we should expect new unknown unknowns to arise that are difficult to predict, and which may have substantial follow-on effects. 124
In summary, foundation models are poised to be an important general-purpose technology of our era. They have potential to increase living standards substantially, but also pose risks of increasing inequality and concentrating power. The economic implications of these technologies are not predetermined, but rather depend on how technologists, policymakers, managers, workers, and other stakeholders answer challenges such as:
- How can we harness the potential of foundation models to boost productivity?
- Can we develop models that enhance creativity and boost the rate of innovation?
- Will the benefits and control rights be limited to a few or widely shared?
Understanding the economic potential of these systems is the first step to guiding them in directions that match our values.
123 Lambda Lab estimates that GPT-3 training costs over $4.6M, research and development costs between $11.4M and $27.6M, hardware required to run GPT-3 costs between $100K and $150K without factoring in other costs (electricity, cooling, backup, etc.), and running costs a minimum of $87K per year. (https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-model)
124 As an example of a secondary effect, consider that the invention of the automobile influenced the development and expansion of the suburbs.
## 5.6 Ethics of scale
Authors: Kathleen Creel, Dallas Card, Rose E. Wang, Isabelle Levent, Alex Tamkin, Armin W. Thomas, Lauren Gillespie, Rishi Bommasani, Rob Reich
The widespread adoption of foundation models poses ethical, social, and political challenges in addition to concerns about increased inequity, the subject of §5.1: fairness. In this section, we discuss social, political, and ethical risks related to the scale of application of foundation models, such as homogenization and the concentration of power, the norms and release strategies appropriate to address them, and concerns about the broader political economy in which foundation models are developed and deployed.
## 5.6.1 Homogenization and scale.
If the same model is used across a variety of domains with minimal adaptation, the strengths, weaknesses, biases, and idiosyncrasies of the original model will be amplified (§5.1: fairness). This is true of the widespread adoption and reliance on any standardized technology. Similar to how a failure in the manufacturing of a part used in many cars or airplanes could have widespread and severe consequences across sectors, a bias or failure of service intrinsic to a foundation model could ripple outwards. However, the current uninterpretability (§4.11: interpretability) of foundation models and their task-agnostic training makes predicting, understanding, and addressing these weaknesses challenging. If, as seems likely, foundation models become widely adopted, foundation model developers bear greater responsibilities of care than standard model developers, as their choices in design and deployment have widespread implications [Arendt 1987].
The defining feature of foundation models - their capacity to be usefully adapted for a multiplicity of tasks - is what makes them likely to be widely adopted for a vast range of socially consequential tasks. In contrast to the current distributed and varied model of decision making, employing many adaptations of the same foundation model for multiple automated decision-making tasks means that decision subjects may face a more homogeneous set of judgments rooted in the underlying foundation model.
This algorithmic monoculture [Kleinberg and Raghavan 2021] could lead to consistent and arbitrary rejection, mis-classification, or ill-treatment of individual decision subjects [Gandy 2021]. We will call this homogenization [Creel and Hellman 2021]. For example, §4.6.2: data-solutions discusses data quality issues that lead to undesirable behavior on subpopulations of data, where subpopulations can be produced by any filter that stratifies the data, including by social group (see related discussions in §4.11.1: interpretability-behavior and §4.8.1: robustness-advantages). Until improvements are made in data quality tooling (§4.6.2: data-solutions) and the ability to identify slices of data on which the model under-performs [Chung et al. 2019; Goel et al. 2021], a foundation model might consistently fail to provide accurate information or services to a subgroup of people (see also §4.8: robustness.
Homogenization has the potential to amplify bias; to standardize bias, compounding injustices rather than distributing them; and to amplify arbitrary exclusion [Creel and Hellman 2021; Gandy 2021]. For example, Zhou et al. [2021a] have argued that BERT encodes an Anglocentric similarity metric by default , one that could be harmful if applied across contexts where foundation models are applied. The application of foundation models across domains has the potential to act as an epistemically and culturally homogenizing force, spreading one implicit perspective, often a socially dominant one, across multiple domains of application.
Existing trends in standardization of training corpora are likely to be exacerbated in foundation models due to the massive scale of both unlabeled and labeled data needed. To the extent that models
train on similar data, they are likely to acquire similar patterns of behavior, biases (§5.1.3: fairnesssources), and errors. Previous high-effort data curation and labeling efforts such as ImageNet have standardized training corpora. In doing so, they have also standardized errors: models trained on ImageNet often rely on the same 'spurious cues' and 'shortcuts', for example using background textures like green grass to predict foreground object classes such as cows [Geirhos et al. 2020; Hendrycks et al. 2021e]. Despite their increased robustness to many types of distribution shifts (§4.8.1: robustness-advantages), foundation models and other large models have been no less likely to learn spurious correlations (§4.8.2: robustness-challenges), and are therefore likely to learn similar errors if trained on the same datasets. Similar effects may arise due to the choice of publicly available unlabeled data. Many foundation models are trained on unlabeled corpora that are chosen for their convenience and accessibility, for example public internet data [Caswell et al. 2021], rather than their quality. However, publicly accessible data, whether labeled or unlabeled, is often outweighed by proprietary data in the training corpora of many proprietary foundation models, as discussed in [Marr 2017] and §4.6.1: data-desiderata. Therefore more research is needed on the extent to which training on similar data homogenizes correlations within foundation models and the extent to which this homogenization might cause uniform failures in adapted derivatives of the model (unless constraints are applied to eliminate the behavior during each adaptation, as discussed in §4.3.2: adaptation-usecases).
Homogenization is not inevitable. As model developers intentionally broaden the range of perspectives represented in their datasets (§5.1.3: fairness-sources), more research is needed on the capacity of foundation models to deliver a diversity of perspectives when used for generative tasks. For example, Sheng et al. [2021] have demonstrated that dialogue systems that adopt 'personas' of specific demographic groups behave differently on measures of social bias. In addition to choosing between 'personas' with the goal of avoiding bias, 'personas' that are diverse along a variety of cognitive and demographic axes could also be used to generate a broader range of coherent outputs for generative tasks. There remain many open questions about how to balance diversity of outputs with relevance and utility to an individual user. 125
## 5.6.2 Surveillance, exclusion, and power.
A key premise of foundation models is that massive unlabeled datasets can be combined with vast computational resources to create a basis from which numerous products can be derived for a variety of applications. This paradigm shift has the potential to alter social structures and shift power, establishing or entrenching the influence of model creators [Zimmerman 2020]. We discuss three potential implications below.
Mass data collection and surveillance. Whereas collecting a labeled dataset typically requires working with domain experts and understanding the problems with and limitations of such data, the need for exceptionally large amounts of data in training foundation models has encouraged some researchers to emphasize quantity rather than quality. 126 Though preprocessing can help improve the quality of this data [e.g., Brown et al. 2020], the scale involved necessitates automated approaches, which may be blunt or poorly documented [Dodge et al. 2021].
Although there is an evolving landscape of data protection legislation (e.g., GDPR in Europe), a variety of questionable practices continue to be used in acquiring data, from opaque policies [Obar and Oeldorf-Hirsch 2020] and the use of 'dark patterns' (i.e., manipulative interfaces [Narayanan
125 For possible approaches to implementation, see the discussions of controllable generation in [Keskar et al. 2019] and §4.3.2: adaptation-usecases and general discussions in [Dinan et al. 2021].
126 For example, Ding et al. [2021] collected 30 million text-image pairs, chose not to address artefacts such as watermarks and white edges, despite their impact on model quality.
et al. 2020]) to outright violation of terms of service. Indeed, this was essentially the strategy taken by Clearview AI - a company which scraped photos from social media, without user consent, and in violation of platforms' terms of service, for the purpose of developing facial classification software. The company was nevertheless able to sell this technology to police departments and other organizations, in many cases without the knowledge of state lawmakers or department heads [Mac et al. 2021]. To the extent that the paradigm of foundation models increases the value of being first to have the largest possible dataset for any particular domain, this may further encourage actors to pursue aggressive data collection, even when that pursuit is legally questionable or contrary to user expectations [Nissenbaum 2009; Zuboff 2018].
The importance of data to foundation models also means that organizations already engaged in widespread data collection will be in a strong position to develop such models, and will likely have incentive to maintain this advantage. To the extent that derivative products could themselves be used to collect additional data (e.g., in surveillance or health diagnostic applications), developers of foundation models may seek to ensure that they obtain ownership of such data. Thus, even though a key advantage of the foundation model paradigm is the ability to generate adapted derivatives, the developers of foundation models might seek to license their work in a way that ensures that data flows back to them from all adapted derivatives. 127
Concentration of power. Although the absolute cost of computation has become dramatically cheaper over time, the training of the largest foundation models currently requires computational resources that put their development beyond the reach of all but a few institutions and organizations (§5.3: environment). Thus, the question of who has access to the relevant computational resources and data will likely determine who is able to produce cutting-edge foundation models in the coming years (see also §5.5.3: economics-centralization).
GPT-3 was at least partly an experiment in scale, showing that major gains could be achieved by scaling up the model size, amount of data, and training time, without major modeling innovations. Although there is extensive ongoing research into reducing the amount of resources required in training such models (see §4.2: training), OpenAI's work suggests that there are still gains to be had from even larger scale efforts [Kaplan et al. 2020], and it seems plausible that other organizations may seek to follow this path in other domains (for example, see [Lieber et al. 2021]).
If scale does turn out to be critical to success, the organizations most capable of producing competitive foundation models will be the most well-resourced: venture-funded start-ups, already-dominant tech giants, and state governments. This raises potential concerns about market concentration, and might indicate the kind of incumbent monopoly or oligopoly that currently exists in extreme capital-intensive industries such as defense and semi-conductor manufacturing [Carril and Duggan 2020].
Moreover, this centralization of power raises concerns about the ability of currently-marginalized individuals and communities to participate in the foundation model development process [Kalluri 2020]. Especially within the realm of government services, the adoption of foundation models could further transfer decision making power from governments to corporate service providers, and introduce additional barriers to due process and accountability [Citron 2008]. Nevertheless, more grassroots efforts (e.g., Masakhane, EleutherAI, HuggingFace) provide encouraging alternatives, and there is extensive work on ways to incorporate participatory or value-sensitive design [Friedman and Hendry 2019; Prabhakaran and Donald Martin 2020].
127 As a less sophisticated example, consider the credit scoring industry, which has been able to position itself such that information flows back to central data brokers as people use its products (as in vetting loan applications), and individuals have little choice but to participate [Lauer 2017].
Fueling widespread automated decision-making. Recent years have seen a dramatic expansion in the use of automated decision-making systems in industry and government [O'Neil 2016; Engstrom et al. 2020]. Although many of the concerns over such automation are not specific to foundation models, the generative abilities of models such as GPT-3, as well as the impressive performance on benchmark tasks (e.g., Devlin et al. [2019]), have the potential to prompt a less-thancareful adoption of this technology by, for example, administrative agencies, many of which lack the expertise necessary to understand sophisticated ML systems [Calo and Citron 2021]. As such, it is especially important to communicate clearly about the realistic capabilities and limitations of foundation models.
Most automated decision-making systems will exist as parts of broader sociotechnical systems in which humans play key roles [Selbst et al. 2018]. 128 As such, there is no guarantee that even large improvements in performance on standardized evaluations will translate into the desired outcomes in the real world (especially if systems are deployed without careful consideration or ongoing evaluation). For example, research has shown that judges may re-impose racial prejudice in interpreting the outputs of a risk assessment system [Albright 2019], or otherwise impose their own biases [Stevenson and Doleac 2021]. Ongoing evaluation with proper ecological validity [de Vries et al. 2020] will be critical in this regard, but may not stop potentially dangerous or costly systems from being adopted without adequate evidence [Ferguson 2017]. Research is ongoing on methods of refusal: ways for individuals to opt out of participation in foundation models and their adapted derivatives, either as data or decision subjects, without repercussions [Benjamin 2016].
In short, the existing problems with algorithmic decision making will be seen in the functioning of foundation models once they are deployed. And to the extent that adopting foundation models accelerates a shift from human to machine decision making, foundation models accentuate the concerns with automation. Although there are not obvious solutions to these challenges, it is important to make questions about how foundation models will impact power part of the conversation about their creation; to communicate with civil society organizations, policy makers, and citizens about the capabilities and limitations of such systems; and to strive for broader dialogue among diverse segments of society about the adoption of such models.
## 5.6.3 Norms.
Public policy and formal regulation by law (§5.4: legality) play an essential role in creating the infrastructure for technological innovation as well as mitigating the potentially harmful effects of widely disseminated technologies. As illustrated by the decades-long gap between the Tuskegee Syphilis experiments and the development of research protocols and institutions like the IRB, public policy to protect human subjects and stakeholders tends to lag behind public awareness and evidence of harms to them [Grady 2015; Stark 2012; Department of Health and Welfare 1979]. As a result, society relies upon professional norms for responsible development and deployment and the establishment of best practices.
Norms exist on a continuum between recommendation and requirement . As a nascent technology, the norms for responsible foundation model development and deployment are not yet well established at either strength of recommendation [Crootof 2019]. In what follows we will discuss norms for deployed models, as models for research have a wider latitude.
Those who wish developers of foundation models to adopt certain norms might lead by example, allowing their own conduct and statements to recommend the norm. As discussed in §1.2: ecosystem, we believe that universities and other nonprofit institutions have an important role in modeling norms for foundation models. As educational institutions, universities are in the unique position to
128 For an extended study of how humans interact with automated judgements, including discussion of both positive and negative automation biases, see Hidalgo [2021].
encourage the next generation of theorists and practitioners to consider the issues raised in this report and also to foster interdisciplinary conversation between researchers and students [Rogers 2021]. Universities and colleges may also contribute to the establishment of norms by auditing existing foundation models and publishing their findings, instituting ethics review boards [Bernstein et al. 2021], and developing their own foundation models.
To create and adopt norms will require institutionalization in funding structures, model repository, release practices, conference submission, and grant proposal requirements. 129 For example, HuggingFace's interface currently encourages the posting of data and model cards, including discussions of bias and social impact. 130 Since it is not required, and perhaps since data quality work is undervalued relative to its importance [Sambasivan et al. 2021], few are filled out. Bias and social impact are included in ethics statements for conferences and some forms of standard evaluation (as discussed in §4.4: evaluation), but otherwise treated as optional considerations by some researchers. This must change.
For some socially consequential use cases, we recommend legal standards be established that require adapted derivatives to provably exhibit certain properties (§5.4: legality). Domains of special concern should be democratically decided but are likely to include allocating and distributing government services, medical diagnosis and monitoring, hiring, and lending: all contexts in which opportunities or even lives of people rest on the proper functioning of an adapted derivative.
What norms should we promote, institutionalize, or require? We recommend a few here, but aim primarily to encourage dialogue about appropriate norms for the development and use of foundation models. Prior work has often focused on norms that advocate documentation [Gebru et al. 2018; Bender and Friedman 2018; Mitchell et al. 2019; Dodge et al. 2019]. Because many of the negative social consequences that appear in a downstream context may initially appear to be extrinsic or particular to a use case (§5.1: fairness), documentation and transparency are especially important for foundation models. Currently, those who adapt foundation models that document the biases or other negative features of their adapted derivatives have no automatic mechanism to report their findings to the developers of the foundation model. Compiling multiple reports of related problems in adapted derivatives may allow the model development team to discover an intrinsic property of the model that spans multiple use cases. Because creators of adapted derivatives often represent different entities than from foundation model developers or providers, additional reporting structures and norms or regulation would be needed for this type of feedback to reach foundation model developers. Such feedback could also be made available to the general audience of model auditors, thereby making auditing and pursuing recourse more accessible.
Public commitment to norms, standards, and creation of reporting mechanisms could also allow downstream users to submit feedback to foundation model providers. In order to enable this, adapted derivatives should be consistently labeled in a way that allows impacted parties to trace problems to their source. Significant technical and social barriers may impede this tracing in practice, such as privacy considerations and the proprietary nature of many foundation models, but without labeling it would be impossible.
It is important that model developers and providers create mechanisms for such reporting. Reporting mechanisms could be informed by similar structures on current platforms, such as issue tracking on open source projects on GitHub. In particular, the submitted issues should be public so that other users can identify trends even if changes have not yet been made and so that developers and providers can be held accountable for unaddressed issues. Additional mechanisms are needed
129 For helpful discussion of partial compliance with 'non-compulsory fairness-conscious policy' such as the norms under discussion here, see Dai et al. [2021b].
130 https://huggingface.co/docs/datasets/master/
to escalate trends upwards to foundation model providers. Similar suggestions regarding tracking issues in training data are discussed in Dinan et al. [2021] and §4.6: data.
Holland et al. [2018] suggest the nutrition label as a helpful model, drawing from labeling discussions in consumer privacy [Kelley et al. 2009]. A nutrition label includes both a list of the 'raw' ingredients and the full nutritional information of the processed food. So too a model card [Mitchell et al. 2019] or nutrition label for an adapted derivative could include both a list of the 'raw materials' such as training data and foundation models used, and the full 'nutritional content' of the adapted derivative such as its known capacities, weaknesses, and biases.
Reporting of the full pipeline is necessary in order for data subjects and impacted parties to trace harms to their sources. However, without the ability to attribute responsibility for the harm to either the adapted derivative, the foundation model, or both, and without a framework for recourse once harm has been attributed, even a successful tracing of a harm will be unlikely to lead to changes in the model (see also §5.1.4: fairness-recourse). Thus, significant technical, policy, and legal work is needed in order to develop frameworks for communicating data, model, and derivative contents to other experts and eventually to the public; to attribute responsibility for harms; and to create avenues for recourse.
## 5.6.4 Release and Auditing.
In February 2019, OpenAI embarked on an experiment. By releasing a reduced 124M parameter GPT-2, sans datasets, they hoped to buy time: time to test for bias, time to prepare for misuse, and time for society to adapt to the presence of large language models [Solaiman et al. 2019]. Eight months later, when OpenAI released the full ∼ 1.5 billion parameter version, testing had exposed some but by no means all of the model's capabilities and limitations. When considering similar questions today, the possible harms of release, centering primarily on misuse (§5.2: misuse), 131 must be weighed against the benefit of transparency that no closed-door testing can replicate, namely broader and independent auditing and access.
Auditing Auditors probe the limitations of current models and suggest paths to fixing them, as well as testing the model's adapted derivatives in a wide variety of natural settings. A policy of open access for auditing allows more numerous and diverse researchers to investigate any model's biases, limitations, and security vulnerabilities, better informing acceptable uses of the models and calibrating appropriate trust in them [Danks 2019; Baier 1986]. 132 In order to support independent audits of foundation models, model developers or third-party intermediaries could host open API access for auditors, including gradient access, and allow access to training data [Raji and Buolamwini 2019; Raji et al. 2020].
Foundation models trained on proprietary data in industry are unlikely to be released, and those trained on private data (as in a medical context) should not be. In order for proprietary models to benefit from independent audits, and for model subjects to benefit from improvements prompted by an auditing process, we recommend that audits occur during a staged release. While staged release may not illuminate all possible model use cases, one way to broaden the range of uncovered use cases is to enlist a neutral third party to decide which individuals or organizations should receive early access in the staged-release program. When model developers decide who should receive staged access, they open themselves up to charges of favoritism, selective distribution, and manipulating public perception of their product. A neutral 'staged release board', or federal auditors, could provide a backstop against these failure modes and ensure that a wide range of
131 For analysis of harms related to misuse, see [Rini 2017] on fake news and [Rini 2020] on deepfakes.
132 Calibrating trust may require an explanation capable of illuminating features of the model relevant to trust, such as
'discriminatory use of a sensitive feature' [Dimanov et al. 2020].
auditors and users are provided access in order to capture a range of disciplinary expertise and sectors of society. A staged release board could also mitigate any perception that auditors would be at risk of losing their early access to the model if they share unflattering outputs, as they might be in a standard staged release process.
Access and adaptation. To the extent that there are social benefits to foundation models, release of models holds the potential to further distribute them. Large language models such as BERT and M-BERT are capable of cross-lingual transfer, which - when the models are open-sourced - may allow for adaptation to languages which otherwise would have too few texts available [Wu and Dredze 2019; Wang et al. 2020a]. Given the number of languages not currently well served by commercial providers, such a benefit alone could be substantial.
Release is not sufficient to democratize access to foundation models, as the barrier of compute power still precludes many from modifying or even loading foundation models, let alone developing their own. However, on each of these points we have seen significant recent technical improvement. Memory techniques such as the zero redundant optimizer (ZeRO) allow researchers to run and train very large models on a simple setup [Rasley et al. 2020; Rajbhandari et al. 2021]. Techniques such as distillation could allow the release of smaller, more tractable models that recoup much of the performance of their parent model while being much easier to train [Li et al. 2020d]. Development of less energy-intensive training methods, as discussed in §5.3: environment, could further spread the ability to work with released models. Increases in efficiency such as the co-design of hardware and software are needed to train yet larger models, as discussed in §4.5: systems, but could also be used to lower the price of access to current models.
The most powerful of the harms, by contrast, are not obviously fueled by release. Sophisticated or institutional actors with the capacity to embark on large-scale disinformation, cyberwarfare, or targeted phishing also are likely to have the capacity to create a similar model if none were released. Although potentially significant, these harms should not therefore weight heavily on a release calculus [Solaiman et al. 2019; Shevlane and Dafoe 2020]. The harms to be weighed against the benefits are those from less well-resourced actors who would not be able to create their own foundation model but may be motivated to generate spam or abuse, fake reviews, or cheat on tests. Does the benefit of release outweigh the potential for harm from actors sophisticated enough to use a released model or API but not sophisticated enough to create their own? We believe that the answer is yes. Research teams with the resources and connections necessary to develop foundation models are few in number. Even collectively, we are unlikely to be numerous or diverse enough to imagine all possible beneficial use cases or all possible probes that could illuminate the capability surface of a foundation model.
## 5.6.5 When not to build.
The development and deployment of powerful technologies is not like gravity, an external force that acts upon us. Technologies reflect a set of choices made by humans; human agency shapes the technological frontier. It follows that technologists can choose when not to build, design, or deploy foundation models [Zimmermann 2021]. This decision need not be binary; instead, one can refuse to engage in the default way by subverting embedded values, challenging assumptions, and shaping research agendas [Simpson 2007]. Technical artifacts, foundation models included, are inherently political, so the research about them has a socio-political context, not solely a technical one. Developers and researchers should be cognizant of which problems they seek to address, e.g., how to scale up a foundation model versus how to make it more computationally accessible; how those problems are formulated; and who their solutions ultimately empower [Rogaway 2016; Winner 1980; Passi and Barocas 2019]. We should value research that seeks to make foundation models more
interpretable, accessible, sustainable, and fair (see §4.11: interpretability, §5.3: environment, §5.1: fairness).
By asking when not to build a foundation model or adapted derivative, we are implicitly asking not only 'What should we build or not build?' but also, 'Under what conditions should a model be built?' and 'What criteria and principles govern building?' The first question stems from the model view; the following questions from the ecosystem view (§1: introduction).
An invitation to consider refusing to build is not tantamount to saying, 'Do nothing.' It is an invitation to make deliberate and judicious choices about what is worth the time, financial resources, expertise, and energy use to build, design, and deploy. Ultimately, this is a difficult, moral question rooted in context and values. There are cases in which the application of adaptive derivatives (and algorithms and machine learning more generally) is inappropriate, because the community impacted protests or because the adaptive derivative naively exacerbates systemic issues that are better addressed with public policy, additional funding, or interdisciplinary collaborations [Angwin et al. 2016].
The Belmont Report, as applied to machine learning in Floridi et al. [2018], provides one possible framework for this question. Drawing from the principle of "beneficence" [Department of Health and Welfare 1979], we can identify cases to reconsider building when an adaptive derivative or a research avenue might cause more harm than good or even provide no benefit at all. Alternatively, there may be cases in which an adaptive derivative is better at a task on a metric of efficiency, performance, and generalization, values prioritized in the machine learning community [Birhane et al. 2020], but an individual, community, or organization might choose to prioritize an existing solution that highlights other values such as human connection and interpretability [Benjamin 2016]. 133 In doing so, they exercise their autonomy - as explained in the Belmont Report's "respect for persons" - in deciding that this is not an appropriate context in which to build [Department of Health and Welfare 1979].
Answering the question of when not to build is a matter of individual responsibility as well as a broader professional responsibility. The decision not to build something by one person, or one team, or one company, invites the reply, 'But if we don't build this, someone else will, and they may likely do it worse.' A simple utilitarian weighing of comparative harms of the outcomes of the two models misses the importance of integrity. It matters very much whether we are the ones building the bad model or whether someone else is [Williams 1973]. Individuals have reasons not to build something that goes against their values or that they cannot endorse as right to build [Korsgaard 2009]. However, the structural environment so created is different. If even one company decides to build the most effective version of an ethically-dubious model, they open the door for other companies to consider similar avenues of research; they make it competitively disadvantageous not to pursue the research [Askell et al. 2019]. When not to build is then a collective question as much as it is an individual one, requiring the community to adhere to codes of professional ethics and responsibility.
In the AI/ML community this infrastructure is underdeveloped compared to other fields such as the medical field. Although professional bodies like the Association for Computing Machinery (ACM) have ethics statements, both industry and academia lack widely used and accepted professional oaths (e.g., the Hippocratic Oath or the the Obligation of the Engineer), regulatory bodies involved in deployment and research (e.g., the FDA for drugs), and official protocols for ethics review (e.g., the IRB for research involving human subjects; [Bernstein et al. 2021]). The ability to opt-out can be incorporated into the foundation model ecosystem at many stages, including during data production, adaptation, and deployment. As the norm veers towards collecting larger and
133 See also §4.11.4: interpretability-impacts for relevant discussion of impacts of uninterpretability.
larger swaths of training data (§4.6: data), we should endeavor to maintain a "respect for persons," [Department of Health and Welfare 1979] emphasizing privacy and consent as part of the data life cycle. This would require innovation in data management and a more concrete understanding - technically and philosophically - of informed consent online, ways of documenting and ensuring that consent is respected, and privacy (see §4.6: data for a specific data management proposal; [Ohm 2014]). Although data and foundation models are diverse in their applications, data participants should be able to indicate how they do not want to have their data used. An opt-out consent model favors developers, as it does not require them to to get consent for each new, unexpected use case. Important then is the right to revoke consent given vacuously for applications that are now being pursued, but were not when consent was originally given.
## 5.6.6 Conclusion.
In this section, we have surveyed some of the risks to society that accompany the widespread adoption of foundation models, such as the homogenization of outcomes and centralization of power. Developers of foundation models should adopt norms regarding development, auditing, and release of foundation models in order to address these risks, aided by legislative requirements, and individuals should be able refuse to be data or decision subjects of foundations models without repercussion.
Manyimplications of foundation models' generative and interactive capacities remain unsurveyed here. For example, §5.5: economics discusses the potential gains to economic productivity from the automation of creative and design work. However, in virtue of their generative nature, foundation models may replace work that many people find meaningful and fulfilling, such as graphic design and writing. We hope that the scope of this report will aid others in their pursuit of the questions of ethics and society unaddressed here.
## 6 CONCLUSION
In this report, we have endeavored to comprehensively discuss many of the most critical aspects of foundation models, ranging from their technical underpinnings to their societal consequences. In this way, we acknowledge the unusual approach taken: we have attempted to clarify the nature of a paradigm that may only have just begun, rather than waiting for more to unfold or the dust to settle. Therefore, much still remains unclear in spite of our efforts and we reiterate that this is just the beginning of a paradigm shift: foundation models have only just begun to transform the way AI systems are built and deployed in the world. Moving forward, we view this document as serving an important role in orienting and framing dialogue on these models and this new paradigm in AI. That said, to ensure the responsible development and deployment of these models on durable foundations, we envision collaboration between different sectors, institutions, and disciplines from the onset to be especially critical.
## ACKNOWLEDGMENTS
We would like to thank the following people for their valuable feedback: Mohit Bansal, Boaz Barak, Yoshua Bengio, Stella Biderman, Su Lin Blodgett, Sam Bowman, Collin Burns, Nicholas Carlini, David Chalmers, Jack Clark, Jeff Dean, Jesse Dodge, Jarred Dunnmon, Gabe Dupre, Jason Eisner, Iason Gabriel, Dan Hendrycks, Avery Hill, Yacine Jernite, Gabbrielle Johnson, Sarah Kreps, Jay McClelland, Preetum Nakkiran, Julian Nyarko, Fernando Pereira, Vinodkumar Prabhakaran, Colin Raffel, Marten van Schijndel, Ludwig Schmidt, Yoav Shoham, Madalsa Singh, Megha Srivastava, Jacob Steinhardt, Emma Strubell, Qian Yang, Luke Zettlemoyer, and Ruiqi Zhong. In addition, we would like to especially thank Vanessa Parli for helping to organize this effort.
## CONFLICT OF INTEREST
This report was authored by the Center for Research on Foundation Models (CRFM), a center at Stanford University borne out of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). CRFM receives funding from Google, Microsoft, and the McGovern Foundation as of July 2022, though this funding was not directly related to this report. Authors of this report may also be affiliated with other institutions beyond Stanford: their contributions reflect only their views and not those of these institutions.
## REFERENCES
- Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI). Savannah, Georgia, USA .
- Jordan Abdi, Ahmed Al-Hindawi, Tiffany Ng, and Marcela P Vizcaychipi. 2018. Scoping review on the use of socially assistive robot technology in elderly care. BMJ open 8, 2 (2018), e018815.
- Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, and David G Robinson. 2020. Roles for computing in social change. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency . 252-260.
- Abubakar Abid, M. Farooqi, and J. Zou. 2021. Persistent Anti-Muslim Bias in Large Language Models. ArXiv abs/2101.05783 (2021). https://arxiv.org/abs/2101.05783
Serge Abiteboul. 1997. Querying Semi-Structured Data. International Conference on Database Theory , 1-18.
Steven Abney. 2007. Semisupervised Learning for Computational Linguistics (1st ed.). Chapman & Hall/CRC.
Daron Acemoglu. 2021. Redesigning AI . MIT Press. https://books.google.com/books?id=HBb6DwAAQBAJ
- Daron Acemoglu and Pascual Restrepo. 2019. Automation and new tasks: How technology displaces and reinstates labor. Journal of Economic Perspectives 33, 2 (2019), 3-30.
Accountability Act. 1996. Health insurance portability and accountability act of 1996. Public law 104 (1996), 191.
- George Adam, Ladislav Rampášek, Zhaleh Safikhani, Petr Smirnov, Benjamin Haibe-Kains, and Anna Goldenberg. 2020. Machine learning approaches to drug response prediction: challenges and recent progress. NPJ Precision Oncology 4, 1 (2020), 1-10.
- Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. In International Conference on Learning Representations .
- Stein Aerts, Diether Lambrechts, Sunit Maity, Peter Van Loo, Bert Coessens, Frederik De Smet, Leon-Charles Tranchevent, Bart De Moor, Peter Marynen, Bassem Hassan, et al. 2006. Gene prioritization through genomic data fusion. Nature Biotechnology 24, 5 (2006), 537-544.
- Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. 2011. Building rome in a day. Commun. ACM 54, 10 (2011), 105-112.
- Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. 2021. Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications. arXiv preprint arXiv:2108.02818 (2021).
- Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2021. HTLM: Hyper-Text Pre-Training and Prompting of Language Models. CoRR abs/2107.06955 (2021). arXiv:2107.06955 https://arxiv.org/abs/2107.06955
- Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. CoRR abs/2012.13255 (2020). arXiv:2012.13255 https://arxiv.org/abs/2012.13255
- Ashvin Agrawal, Rony Chatterjee, Carlo Curino, Avrilia Floratou, Neha Gowdal, Matteo Interlandi, Alekh Jindal, Kostantinos Karanasos, Subru Krishnan, Brian Kroth, et al. 2019. Cloudy with high chance of DBMS: A 10-year prediction for Enterprise-Grade ML. arXiv preprint arXiv:1909.00084 (2019).
- Roxana M Agrigoroaie and Adriana Tapus. 2016. Developing a healthcare robot with personalized behaviors and social skills for the elderly. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI) . IEEE, 589-590.
- Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. 2021. OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning. ArXiv abs/2010.13611 (2021).
- Ifeoma Ajunwa. 2019. The paradox of automation as anti-bias intervention. Cardozo L. Rev. 41 (2019), 1671.
- Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. arXiv preprint arXiv:2104.11178 (2021).
- Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations .
- EA AlBadawy, A Saha, and MA Mazurowski. 2018. Deep learning for segmentation of brain tumors: Impact of crossinstitutional training and testing. Med Phys. 45 (2018).
- Alex Albright. 2019. If You Give a Judge a Risk Score: Evidence from Kentucky Bail Decisions. (2019). https://thelittledataset. com/about\_files/albright\_judge\_score.pdf Accessed 2021-07-18.
- Zarqa Ali, John Robert Zibert, and Simon Francis Thomsen. 2020. Virtual clinical trials: Perspectives in dermatology. Dermatology 236, 4 (2020), 375-382.
- Gordon W. Allport. 1954. The Nature of Prejudice . Addison-Wesley Publishing Company. https://books.google.com/books? id=u94XUyRuDl4C
- Laila Alrajhi, Ahmed Alamri, Filipe Dwan Pereira, and Alexandra I Cristea. 2021. Urgency Analysis of Learners' Comments: AnAutomated Intervention Priority Model for MOOC. In International Conference on Intelligent Tutoring Systems . Springer, 148-160.
- Sergei Alyamkin, Matthew Ardi, Alexander C Berg, Achille Brighton, Bo Chen, Yiran Chen, Hsin-Pai Cheng, Zichen Fan, Chen Feng, Bo Fu, et al. 2019. Low-power computer vision: Status, challenges, and opportunities. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9, 2 (2019), 411-421.
- Julia Amann, Alessandro Blasimme, Effy Vayena, Dietmar Frey, and Vince I Madai. 2020. Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Medical Informatics and Decision Making 20, 1 (2020), 1-9.
- American Bar Association. 2004. Gideon's Broken Promise: America's Continuing Quest for Equal Justice . Technical Report. American Bar Association.
- American Bar Association. 2021. National Lawyer Population Survey 2021. https://www.americanbar.org/content/dam/ aba/administrative/market\_research/2021-national-lawyer-population-survey.pdf
- Saleema Amershi, James Fogarty, and Daniel Weld. 2012. Regroup: Interactive Machine Learning for On-demand Group Creation in Social Networks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . ACM.
- Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016).
- Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Computer Vision and Pattern Recognition (CVPR) .
- Jacob Andreas. 2019. Measuring Compositionality in Representation Learning. International Conference on Learning Representations (2019).
- Jacob Andreas. 2020. Good-Enough Compositional Data Augmentation. Association for Computational Linguistics (July 2020), 7556-7566. https://doi.org/10.18653/v1/2020.acl-main.676
- Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 39-48.
- Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. 2017. Hindsight Experience Replay. arXiv preprint arXiv:1707.01495 (2017).
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias: There's software used across the country to predict future criminals. And it's biased against blacks. ProPublica.
- Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2021. Large-scale differentially private BERT. arXiv preprint arXiv:2108.01624 (2021).
- Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. 2020. Scalable Second Order Optimization for Deep Learning. arXiv preprint arXiv:2002.09018 (2020).
- Lasse F. Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. 2020. Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models. ICML Workshop on Challenges in Deploying and monitoring Machine Learning Systems. arXiv:2007.03051.
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision . 2425-2433.
- Maria Antoniak and David Mimno. 2021. Bad Seeds: Evaluating Lexical Methods for Bias Measurement. In Proceedings of ACL 2021 .
- Hannah Arendt. 1987. Collective Responsibility . Springer Netherlands, Dordrecht, 43-50.
- Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2019. Invariant risk minimization. arXiv preprint arXiv:1907.02893 (2019).
- Michael Armbrust, Armando Fox, Rean Griffith, Anthony D Joseph, Randy H Katz, Andrew Konwinski, Gunho Lee, David A Patterson, Ariel Rabkin, Ion Stoica, et al. 2009. Above the clouds: A berkeley view of cloud computing . Technical Report. Technical Report UCB/EECS-2009-28, EECS Department, University of California ....
- Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. 2019a. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems . 7411-7422.
- Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. 2019b. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229 (2019).
- Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A latent variable model approach to PMI-based word embeddings. Transactions of the Association for Computational Linguistics (2016).
- Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the Cross-lingual Transferability of Monolingual Representations. arXiv:1910.11856 [cs] (May 2020). http://arxiv.org/abs/1910.11856
- Euan A Ashley. 2016. Towards precision medicine. Nature Reviews Genetics 17, 9 (2016), 507-522.
- Amanda Askell, Miles Brundage, and Gillian Hadfield. 2019. The Role of Cooperation in Responsible AI Development. http://arxiv.org/abs/1907.04534
- David H. Autor. 2015. Why Are There Still So Many Jobs? The History and Future of Workplace Automation. Journal of Economic Perspectives 29, 3 (2015), 3-30.
- Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10, 7 (2015), e0130140.
- Claudine Badue, Rânik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius Brito Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago Meireles Paixão, Filipe Mutz, et al. 2020. Self-driving cars: A survey. Expert Systems with Applications (2020), 113816.
- Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 12449-12460. https://proceedings.neurips.cc/ paper/2020/file/92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).
- Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, S. A. Hosseini, Pushmeet Kohli, and Edward Grefenstette. 2019. Learning to Understand Goal Specifications by Modelling Reward. In International Conference on Learning Representations (ICLR) .
- Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. 2018. Systematic Generalization: What Is Required and Can It Be Learned?. In International Conference on Learning Representations .
- Annette Baier. 1986. Trust and Antitrust. Ethics 96, 2 (1986), 231-260. http://www.jstor.org/stable/2381376
- Andrea Bajcsy, Dylan P. Losey, M. O'Malley, and A. Dragan. 2017. Learning Robot Objectives from Physical Human Interaction. In Conference on Robot Learning (CORL) .
- Bowen Baker, I. Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. 2020. Emergent Tool Use From Multi-Agent Autocurricula. ArXiv abs/1909.07528 (2020).
- Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. 2019. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems 32 (2019), 5082-5093.
- Jack Bandy and Nicholas Vincent. 2021. Addressing" Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus. arXiv preprint arXiv:2105.05241 (2021).
- Kshitij Bansal, Sarah M. Loos, Markus N. Rabe, Christian Szegedy, and Stewart Wilcox. 2019. HOList: An Environment for Machine Learning of Higher Order Logic Theorem Proving. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA (Proceedings of Machine Learning Research, Vol. 97) , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 454-463. http://proceedings.mlr.press/v97/bansal19a.html
- Yamini Bansal, Gal Kaplun, and Boaz Barak. 2020. For self-supervised learning, Rationality implies generalization, provably. arXiv preprint arXiv:2010.08508 (2020).
- Elias Bareinboim, Juan D. Correa, Duligur Ibeling, and Thomas Icard. 2020. On Pearl's Hierarchy and the Foundations of Causal Inference . Technical Report R-60. Causal AI Lab, Columbia University. Forthcoming in Probabilistic and Causal Inference: The Works of Judea Pearl (ACM Books).
- Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The Problem With Bias: Allocative Versus Representational Harms in Machine Learning. (2017). Talk at SIGCIS Conference.
- Marco Baroni. 2021. On the proper role of linguistically-oriented deep net analysis in linguistic theorizing. arXiv preprint arXiv:2106.08694 (2021).
- Christine Basta, Marta R Costa-Jussà, and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. arXiv preprint arXiv:1904.08783 (2019).
- Priyam Basu, Tiasa Singha Roy, Rakshit Naidu, Zumrut Muftuoglu, Sahib Singh, and Fatemehsadat Mireshghallah. 2021. Benchmarking Differential Privacy and Federated Learning for BERT Models. arXiv preprint arXiv:2106.13973 (2021).
- Mary Bates. 2019. Health care chatbots are here to help. IEEE pulse 10, 3 (2019), 12-14.
- Sarah Batterbury. 2012. Language justice for Sign Language Peoples: The UN Convention on the Rights of Persons with Disabilities. Language Policy 11 (08 2012). https://doi.org/10.1007/s10993-012-9245-8
- Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. 2006. Surf: Speeded up robust features. In European conference on computer vision . Springer, 404-417.
- Daniel M Bear, Elias Wang, Damian Mrowca, Felix J Binder, Hsiau-Yu Fish Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Li Fei-Fei, et al. 2021. Physion: Evaluating Physical Prediction from Vision in Humans and Machines. arXiv preprint arXiv:2106.08261 (2021).
- Adam L Beberg, Daniel L Ensign, Guha Jayachandran, Siraj Khaliq, and Vijay S Pande. 2009. Folding@home: Lessons from eight years of volunteer distributed computing. In 2009 IEEE International Symposium on Parallel & Distributed Processing . 1-8.
- J Thaddeus Beck, Melissa Rammage, Gretchen P Jackson, Anita M Preininger, Irene Dankwa-Mullan, M Christopher Roebuck, Adam Torres, Helen Holtzen, Sadie E Coverdill, M Paul Williamson, et al. 2020. Artificial intelligence tool for optimizing eligibility screening for clinical trials in a large community cancer center. JCO clinical cancer informatics 4 (2020), 50-59.
- Sara Beery, Elijah Cole, and Arvi Gjoka. 2020. The iWildCam 2020 Competition Dataset. arXiv preprint arXiv:2004.10340 (2020).
- Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do Neural Machine Translation Models Learn about Morphology?. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Vancouver, Canada). Association for Computational Linguistics, 861-872. https: //doi.org/10.18653/v1/P17-1080
- Kristen Bell, Jenny Hong, Nick McKeown, and Catalin Voss. 2021. The Recon Approach: A New Direction for Machine Learning in Criminal Law. Berkeley Technology Law Journal 37 (2021).
- Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020).
- Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine Learning 79, 1 (2010), 151-175.
- Kambez H Benam, Siobhan Gilchrist, Andre Kleensang, Ani B Satz, Catherine Willett, and Qiang Zhang. 2019. Exploring new technologies in biomedical research. Drug discovery today 24, 6 (2019), 1242-1247.
- Emily M Bender. 2011. On achieving and evaluating language-independence in NLP. Linguistic Issues in Language Technology 6, 3 (2011), 1-26.
- Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics (TACL) 6 (2018), 587-604.
- Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT '21) . Association for Computing Machinery, New York, NY, USA, 610-623. https://doi.org/10.1145/3442188.3445922
- Emily M Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . 5185-5198.
- Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. 2021. Machine learning for combinatorial optimization: A methodological tour d'horizon. Eur. J. Oper. Res. 290, 2 (2021), 405-421. https://doi.org/10.1016/j.ejor.2020.07.063
- Ruha Benjamin. 2016. Informed Refusal: Toward a Justicebased Bioethics. Science, Technology, & Human Values (June 2016), 967-990.
- Ruha Benjamin. 2019. Race after Technology . Polity Press.
- Tristan Bepler and Bonnie Berger. 2021. Learning the protein language: Evolution, structure, and function. Cell Systems 12, 6 (2021), 654-669.
- Kaustav Bera, Kurt A Schalper, David L Rimm, Vamsidhar Velcheti, and Anant Madabhushi. 2019. Artificial intelligence in digital pathology-new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16, 11 (2019), 703-715.
- Elika Bergelson and Daniel Swingley. 2012. At 6-9 months, human infants know the meanings of many common nouns. Proceedings of the National Academy of Sciences 109, 9 (2012), 3253-3258.
- Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. 2021. Fairness in Criminal Justice Risk Assessments: The State of the Art. Sociological Methods & Research 50, 1 (2021), 3-44. https://doi.org/10.1177/0049124118782533
- Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. 2017. Safe Model-based Reinforcement Learning with Stability Guarantees. In Advances in Neural Information Processing Systems (NeurIPS) .
- Michael S. Bernstein, Margaret Levi, David Magnus, Betsy Rajala, Debra Satz, and Charla Waeiss. 2021. ESR: Ethics and Society Review of Artificial Intelligence Research. http://arxiv.org/abs/2106.11521
- Kathryn D Betts and Kyle R Jaep. 2017. The dawn of fully automated contract drafting: Machine learning breathes new life into a decades-old promise. Duke L. & Tech. Rev. 15 (2017), 216.
- Urmil Bharti, Deepali Bajaj, Hunar Batra, Shreya Lalit, Shweta Lalit, and Aayushi Gangwani. 2020. Medbot: Conversational artificial intelligence powered chatbot for delivering tele-health after covid-19. In 2020 5th International Conference on Communication and Electronics Systems (ICCES) . IEEE, 870-875.
- I. Biederman. 1972. Perceiving real-world scenes. Science (New York, N.Y.) 177, 4043 (July 1972), 77-80. https://doi.org/10. 1126/science.177.4043.77
- Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. 2013. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases . 387-402.
- Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012. Poisoning Attacks Against Support Vector Machines. In International Conference on Machine Learning (ICML) . 1467-1474.
- Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2020. The Underlying Values of Machine Learning Research. (2020). https://drive.google.com/file/d/1tjrm3Bf1hxV8iuPSiCcM1IazITGp-GZj/view
- Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning . Springer.
- Erdem Biyik and Dorsa Sadigh. 2018. Batch Active Preference-Based Learning of Reward Functions. In Conference on Robot Learning (CORL) .
- Mikołaj Bińkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. 2018. Demystifying MMD GANs. arXiv:1801.01401 [stat.ML]
- Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. 2019. Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process. arXiv preprint arXiv:1904.09080 (2019).
- John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain Adaptation with Structural Correspondence Learning. In Empirical Methods in Natural Language Processing (EMNLP) .
- Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (Technology) is Power: A Critical Survey of 'Bias' in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, 5454-5476. https://doi.org/10.18653/v1/2020.acl-main.485
- Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic Dialectal Variation in Social Media: A Case Study of African-American English. In Empirical Methods in Natural Language Processing (EMNLP) . 1119-1130.
- Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Association for Computational Linguistics, Online, 1004-1015. https://doi.org/10.18653/v1/2021.acl-long.81
- Su Lin Blodgett and Brendan O'Connor. 2017. Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English. In Fairness, Accountability, and Transparency in Machine Learning (FAT/ML) Workshop, KDD .
- Thomas Bock. 2007. Construction robotics. Autonomous Robots 22, 3 (2007), 201-209.
- Jeannette Bohg, Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, and Gaurav S Sukhatme. 2017. Interactive perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics 33, 6 (2017), 1273-1291.
- Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In Advances in Neural Information Processing Systems , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc. https: //proceedings.neurips.cc/paper/2016/file/a486cd07e4ac3d270571622f4f316ec5-Paper.pdf
- Michael J Bommarito, Daniel Martin Katz, and Eric M Detterman. 2018. LexNLP: Natural language processing and information extraction for legal and regulatory texts. arXiv:1806.03688 [cs.CL]
- Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 4758-4781. https://doi.org/10.18653/v1/2020.acl-main.431
- Rishi Bommasani, Steven Wu, and Xanda Schofield. 2019. Towards Private Synthetic Text Generation. In NeurIPS 2019 Machine Learning with Guarantees Workshop .
- Paul Boniol, George Panagopoulos, Christos Xypolopoulos, Rajaa El Hamdani, David Restrepo Amariles, and Michalis Vazirgiannis. 2020. Performance in the courtroom: Automated processing and visualization of appeal court decisions in france. arXiv preprint arXiv:2006.06251 (2020).
- Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2021. Improving Language Models by Retrieving from Trillions of Tokens. arXiv preprint arXiv:2112.04426 (2021).
- Antoine Bosselut, Omer Levy, Ari Holtzman, C. Ennis, D. Fox, and Yejin Choi. 2018. Simulating Action Dynamics with Neural Process Networks. (2018).
- Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, A. Çelikyilmaz, and Yejin Choi. 2019. COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. In ACL .
- Nick Bostrom and Milan M Cirkovic. 2011. Global catastrophic risks . Oxford University Press.
- Léon Bottou. 2014. From machine learning to machine reasoning. Machine learning 94, 2 (2014), 133-149.
- Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2019. Machine unlearning. arXiv preprint arXiv:1912.03817 (2019).
- William G Bowen. 2012. The 'cost disease' in higher education: is technology the answer? The Tanner Lectures Stanford University (2012).
- Samuel R. Bowman and George Dahl. 2021. What Will it Take to Fix Benchmarking in Natural Language Understanding?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Association for Computational Linguistics, Online, 4843-4855. https://doi.org/10.18653/v1/2021. naacl-main.385
- James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: Composable Transformations of Python+NumPy Programs . http://github.com/google/jax
- Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. 2021. FLEX: Unifying Evaluation for Few-Shot NLP. ArXiv abs/2107.07170 (2021).
- Gwern Branwen. 2020. GPT-3 creative fiction. (2020).
- Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Association for Computational Linguistics, Hong Kong, China, 1664-1674. https://doi.org/10.18653/v1/D19-1176
- Raymond H Brescia, Walter McCarthy, Ashley McDonald, Kellan Potts, and Cassandra Rivais. 2014. Embracing disruption: How technological change in the delivery of legal services can improve access to justice. Alb. L. Rev. 78 (2014), 553.
- Timothy F. Bresnahan and M. Trajtenberg. 1995. General purpose technologies 'Engines of growth'? Journal of Econometrics 65, 1 (1995), 83-108. https://doi.org/10.1016/0304-4076(94)01598-T
- Marilynn B. Brewer and William D. Crano. 2014. Research Design and Issues of Validity (2 ed.). Cambridge University Press, 11-26. https://doi.org/10.1017/CBO9780511996481.005
- Liam Kofi Bright, Daniel Malinsky, and Morgan Thompson. 2016. Causally Interpreting Intersectionality Theory. Philosophy of Science 83, 1 (Jan. 2016), 60-81. https://doi.org/10.1086/684173
- Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In International Conference on Learning Representations .
- Matt Brockman. 2020. Math - GPT\_Prompts. http://gptprompts.wikidot.com/logic:math#toc5
- Urie Bronfenbrenner. 1977. Toward an Experimental Ecology of Human Development. American Psychologist 32 (1977), 513-531.
- R. Brooks. 2002. Flesh and Machines: How Robots Will Change Us.
- Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. 2022. What Does it Mean for a Language Model to Preserve Privacy? arXiv preprint arXiv:2202.05520 (2022).
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv preprint arXiv:2005.14165 (2020).
- Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, Hyrum Anderson, Heather Roff, Gregory C. Allen, Jacob Steinhardt, Carrick Flynn, Seán Ó hÉigeartaigh, Simon Beard, Haydn Belfield, Sebastian Farquhar, Clare Lyle, Rebecca Crootof, Owain Evans, Michael Page, Joanna Bryson, Roman Yampolskiy, and Dario Amodei. 2018. The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation. arXiv:1802.07228 [cs.AI]
- Erik Brynjolfsson. 2022. The Turing Trap: The Promise & Peril of Human-Like Artificial Intelligence. Daedalus, Forthcoming (2022).
- Erik Brynjolfsson and Avinash Collis. 2019. How should we measure the digital economy? Focus on the value created, not just the prices paid. Harvard business review 97, 6 (2019), 140-.
- Erik Brynjolfsson, Xiang Hui, and Meng Liu. 2019. Does Machine Translation Affect International Trade? Evidence from a Large Digital Platform. Management Science 65, 12 (dec 2019), 5449-5460. https://doi.org/10.1287/mnsc.2019.3388
- Erik Brynjolfsson and Andrew McAfee. 2011. Race against the Machine.
- Erik Brynjolfsson and Tom Mitchell. 2017. What can machine learning do? Workforce implications. Science 358, 6370 (2017), 1530-1534.
- Sébastien Bubeck and Mark Sellke. 2021. A Universal Law of Robustness via Isoperimetry. arXiv preprint arXiv:2105.12806 (2021).
- Ben Buchanan, Andrew Lohn, Micah Musser, and Katerina Sedova. 2021. Truth, Lies, and Automation: How Language Models Could Change Disinformation . Center for Security and Emerging Technology. https://doi.org/10.51593/2021CA003
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency . 77-91.
- Christopher Burr, Nello Cristianini, and James Ladyman. 2018. An analysis of the interaction between intelligent software agents and human users. Minds and machines 28, 4 (2018), 735-774.
- Jenna Burrell. 2016. How the machine 'thinks': Understanding opacity in machine learning algorithms. Big Data & Society 3, 1 (Jan. 2016), 205395171562251. https://doi.org/10.1177/2053951715622512
- Daniel Buschek, Martin Zurn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems .
- James E Cabral, Abhijeet Chavan, Thomas M Clarke, and John Greacen. 2012. Using technology to enhance access to justice. Harv. JL & Tech. 26 (2012), 241.
- Tianle Cai, Ruiqi Gao, Jason D Lee, and Qi Lei. 2021. A Theory of Label Propagation for Subpopulation Shift. arXiv preprint arXiv:2102.11203 (2021).
- Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183-186. https://doi.org/10.1126/science.aal4230 arXiv:https://science.sciencemag.org/content/356/6334/183.full.pdf
- Ryan Calo and Danielle K. Citron. 2021. The Automated Administrative State: A Crisis of Legitimacy. Emory Law Journal 40 (2021). Issue 4. https://scholarlycommons.law.emory.edu/elj/vol70/iss4/1
- Diogo M Camacho, Katherine M Collins, Rani K Powers, James C Costello, and James J Collins. 2018. Next-generation machine learning for biological networks. Cell 173, 7 (2018), 1581-1592.
- Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. 2020. Thread: Circuits. Distill (2020). https://doi.org/10.23915/distill.00024 https://distill.pub/2020/circuits.
- Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing Factual Knowledge in Language Models. arXiv:2104.08164 [cs.CL]
- Qingqing Cao, Aruna Balasubramanian, and Niranjan Balasubramanian. 2020. Towards accurate and reliable energy measurement of NLP models. arXiv preprint arXiv:2010.05248 (2020).
- Steven Cao, Nikita Kitaev, and Dan Klein. 2019. Multilingual Alignment of Contextual Word Representations. In International Conference on Learning Representations .
- Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy . IEEE, 463-480.
- Yang Trista Cao and Hal Daumé III. 2020. Toward Gender-Inclusive Coreference Resolution. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 4568-4595. https://doi.org/10.18653/v1/2020.acl-main.418
- Nicholas Carlini. 2021. Poisoning the Unlabeled Dataset of Semi-Supervised Learning. In USENIX Security Symposium .
- Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium . 267-284.
- Nicholas Carlini and Andreas Terzis. 2021. Poisoning and Backdooring Contrastive Learning. arXiv preprint arXiv:2106.09667 (2021).
- Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting Training Data from Large Language Models. In USENIX Security Symposium .
- Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. 2019. Unlabeled Data Improves Adversarial Robustness. In Advances in Neural Information Processing Systems (NeurIPS) .
- Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J'egou, J. Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging Properties in Self-Supervised Vision Transformers. ArXiv abs/2104.14294 (2021).
- Rodrigo Carril and Mark Duggan. 2020. The impact of industry consolidation on government procurement: Evidence from Department of Defense contracting. Journal of Public Economics 184 (2020), 104141. https://doi.org/10.1016/j.jpubeco. 2020.104141
- Jimmy Carter. 1978. Excerpts From Carter's Speech to the Bar Association. The New York Times (May 1978).
- Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. 2019. Activation atlas. Distill 4, 3 (2019), e15. Isaac Caswell, Julia Kreutzer, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Javier Ortiz Suárez, Iroro Orife, Kelechi Ogueji, Rubungo Andre Niyongabo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2021. Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. ArXiv abs/2103.12028 (2021). https://arxiv.org/abs/2103.12028
- Rahma Chaabouni, Roberto Dessì, and Eugene Kharitonov. 2021. Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN. arXiv preprint arXiv:2107.01366 (2021).
- Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. 2020. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems 33 (2020).
- Benjamin Chaix, Jean-Emmanuel Bibault, Arthur Pienkowski, Guillaume Delamon, Arthur Guillemassé, Pierre Nectoux, and Benoît Brouard. 2019. When chatbots meet patients: one-year prospective study of conversations between patients with breast cancer and a chatbot. JMIR cancer 5, 1 (2019), e12856.
- Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in English. arXiv preprint arXiv:1906.02059 (2019).
- Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. arXiv preprint arXiv:2010.02559 (2020).
- Robert Challen, Joshua Denny, Martin Pitt, Luke Gompels, Tom Edwards, and Krasimira Tsaneva-Atanasova. 2019. Artificial intelligence, bias and clinical safety. BMJ Quality & Safety 28, 3 (2019), 231-237.
- Mahawaga Arachchige Pathum Chamikara, Peter Bertok, Ibrahim Khalil, Dongxi Liu, and Seyit Camtepe. 2021. Privacy preserving distributed machine learning with federated learning. Computer Communications 171 (2021), 112-125.
- Muthu Kumar Chandrasekaran and Min-Yen Kan. 2019. When to reply? context sensitive models to predict instructor interventions in mooc forums. arXiv preprint arXiv:1905.10851 (2019).
- Eshwar Chandrasekharan, Mattia Samory, Shagun Jhaver, Hunter Charvat, Amy Bruckman, Cliff Lampe, Jacob Eisenstein, and Eric Gilbert. 2018. The Internet's Hidden Rules: An Empirical Study of Reddit Norm Violations at Micro, Meso, and Macro Scales. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 32 (Nov. 2018), 25 pages. https://doi.org/10.1145/3274301
- Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3D: Learning from RGB-D Data in Indoor Environments. International Conference on
- 3D Vision (3DV) (2017).
- Hongyan Chang, Ta Duy Nguyen, Sasi Kumar Murakonda, Ehsan Kazemi, and Reza Shokri. 2020. On adversarial bias and the robustness of fair machine learning. arXiv preprint arXiv:2006.08669 (2020).
- Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. In CVPR .
- Yatin Chaudhary, Pankaj Gupta, Khushbu Saxena, Vivek Kulkarni, Thomas Runkler, and Hinrich Schütze. 2020. TopicBERT for energy efficient document classification. arXiv preprint arXiv:2010.16407 (2020).
- Annie S. Chen, Suraj Nair, and Chelsea Finn. 2021c. Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos. In Robotics: Science and Systems (RSS) .
- Annie S Chen, HyunJi Nam, Suraj Nair, and Chelsea Finn. 2021d. Batch exploration with examples for scalable robotic reinforcement learning. IEEE Robotics and Automation Letters 6, 3 (2021), 4401-4408.
- Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. 2018. This looks like that: deep learning for interpretable image recognition. arXiv preprint arXiv:1806.10574 (2018).
- Irene Y Chen, Shalmali Joshi, and Marzyeh Ghassemi. 2020b. Treating health disparities with artificial intelligence. Nature medicine 26, 1 (2020), 16-17.
- Irene Y Chen, Peter Szolovits, and Marzyeh Ghassemi. 2019. Can AI help reduce disparities in general medical and mental health care? AMA journal of ethics 21, 2 (2019), 167-179.
- Liang Chen, Peter Edwards, John D Nelson, and Timothy J Norman. 2015a. An access control model for protecting provenance graphs. In 2015 13th Annual Conference on Privacy, Security and Trust (PST) . IEEE, 125-132.
- Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, M. Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. 2021b. Decision Transformer: Reinforcement Learning via Sequence Modeling. ArXiv abs/2106.01345 (2021).
- Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. 2021a. Mandoline: Model Evaluation under Distribution Shift. In International Conference on Machine Learning . PMLR, 1617-1629.
- Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020d. Generative Pretraining From Pixels. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119) , Hal Daumé III and Aarti Singh (Eds.). PMLR, 1691-1703. http://proceedings.mlr.press/v119/ chen20s.html
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021f. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG]
- Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020c. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML) . 1597-1607.
- Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training Deep Nets with Sublinear Memory Cost. CoRR abs/1604.06174 (2016). arXiv:1604.06174 http://arxiv.org/abs/1604.06174
- Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020a. Improved Baselines with Momentum Contrastive Learning. arXiv preprint arXiv:2003.04297 (2020).
- Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015b. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015).
- Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 (2017).
- Yun Chen, Frieda Rong, Shivam Duggal, Shenlong Wang, Xinchen Yan, Sivabalan Manivasagam, Shangjie Xue, Ersin Yumer, and Raquel Urtasun. 2021e. GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 7230-7240.
- James Cheney, Laura Chiticariu, and Wang-Chiew Tan. 2009. Provenance in databases: Why, how, and where . Now Publishers Inc.
- Ethan A Chi, John Hewitt, and Christopher D Manning. 2020. Finding Universal Grammatical Relations in Multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . 5564-5577.
- Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating Long Sequences with Sparse Transformers. arXiv preprint arXiv:1904.10509 (2019).
- Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning (ICML) .
- Rochelle Choenni and Ekaterina Shutova. 2020. Cross-neutralising: Probing for joint encoding of linguistic information in multilingual models. arXiv preprint arXiv:2010.12825 (2020).
- Alex Chohlas-Wood, Joe Nudell, Zhiyuan Jerry Lin, Julian Nyarko, and Sharad Goel. 2020. Blind justice: Algorithmically masking race in charging decisions . Technical Report. Technical report.
- Jonathan H Choi. 2020. An Empirical Study of Statutory Interpretation in Tax Law. NYUL Rev. 95 (2020), 363.
- Noam Chomsky. 2014. The minimalist program . MIT press.
- Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 (2020).
- Alexandra Chouldechova and Aaron Roth. 2020. A snapshot of the frontiers of fairness in machine learning. Commun. ACM 63, 5 (April 2020), 82-89. https://doi.org/10.1145/3376898
- Yinlam Chow, Ofir Nachum, Edgar A. Duéñez-Guzmán, and M. Ghavamzadeh. 2018. A Lyapunov-based Approach to Safe Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS) .
- Paul Christiano. 2016. Prosaic AI Alignment. https://ai-alignment.com/prosaic-ai-control-b959644d79c2
- Yeounoh Chung, Tim Kraska, Neoklis Polyzotis, Ki Hyun Tae, and Steven Euijong Whang. 2019. Slice finder: Automated data slicing for model validation. In 2019 IEEE 35th International Conference on Data Engineering (ICDE) . IEEE, 1550-1553.
- Cigna. 2018. Cigna U.S. Loneliness Index . https://www.cigna.com/assets/docs/newsroom/loneliness-survey-2018-fullreport.pdf
- Danielle Keats Citron. 2008. Technological Due Process. Wash. U. L. Rev. 1249 (2008). https://openscholarship.wustl.edu/ law\_lawreview/vol85/iss6/2/
- Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. 2021. All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. arXiv preprint arXiv:2107.00061 (2021).
- Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In International Conference on Learning Representations .
- Peter Clark, Oren Etzioni, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, Sumithra Bhakthavatsalam, Dirk Groeneveld, Michal Guerquin, and Michael Schmitz. 2019. From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project. CoRR abs/1909.01958 (2019). http://arxiv.org/abs/1909.01958
- John D. Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, John DeNero, Pieter Abbeel, and Sergey Levine. 2019. Guiding Policies with Language via Meta-Learning. In International Conference on Learning Representations (ICLR) .
- Cary Coglianese and Lavi Ben Dor. 2020. AI in Adjudication and Administration. Brooklyn Law Review, Forthcoming, University of Pennsylvania Law School, Public Law Research Paper 19-41 (2020).
- Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie. 2021. When Does Contrastive Visual Representation Learning Work? arXiv:2105.05837 [cs.CV]
- Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia. 2017. DAWNBench: An End-to-End Deep Learning Benchmark and Competition. In NeurIPS Workshop on Systems for Machine Learning .
- Francis S Collins and Harold Varmus. 2015. A new initiative on precision medicine. New England Journal of Medicine 372, 9 (2015), 793-795.
- Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In International Conference on Machine Learning (ICML) . 160-167.
- Cristina Colonnesi, Geert Jan JM Stams, Irene Koster, and Marc J Noom. 2010. The relation between pointing and language development: A meta-analysis. Developmental Review 30, 4 (2010), 352-366.
- Bernard Comrie. 1989. Language universals and linguistic typology: Syntax and morphology . University of Chicago press.
- Aubrey Condor, Max Litster, and Zachary Pardos. 2021. Automatic short answer grading with SBERT on out-of-sample questions. In Proceedings of the 14th International Conference on Educational Data Mining .
- Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single \ $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Melbourne, Australia). Association for Computational Linguistics, 2126-2136. http://aclweb.org/anthology/P18-1198
- Carol McDonald Connor. 2019. Using technology and assessment to personalize instruction: Preventing reading problems. Prevention Science 20, 1 (2019), 89-99.
- Erik Conser, Kennedy Hahn, Chandler M Watson, and Melanie Mitchell. 2019. Revisiting Visual Grounding. arXiv preprint arXiv:1904.02225 (2019).
- Alan Cooper, Robert Reimann, David Cronin, and Christopher Noessel. 2014. About face: the essentials of interaction design. In John Wiley & Sons .
- Sam Corbett-Davies and Sharad Goel. 2018. The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning. arXiv:1808.00023 [cs.CY]
- Anthony Corso, Robert J. Moss, Mark Koren, R. Lee, and Mykel J. Kochenderfer. 2020. A Survey of Algorithms for Black-Box Safety Validation. arXiv preprint arXiv:2005.02979 (2020).
- Ajeya Cotra. 2021. The case for aligning narrowly superhuman models. https://www.alignmentforum.org/posts/ PZtsoaoSLpKjjbMqM/the-case-for-aligning-narrowly-superhuman-models
- Kate Crawford. 2017. The Problem With Bias. (2017). Keynote at NeurIPS.
- Kate Crawford. 2021. Atlas of AI . Yale University Press.
- Kathleen Creel and Deborah Hellman. 2021. The Algorithmic Leviathan: Arbitrariness, Fairness, and Opportunity in Algorithmic Decision Making Systems. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT '21) . Association for Computing Machinery, New York, NY, USA, 816. https://doi.org/10.1145/3442188.3445942
- Kathleen A. Creel. 2020. Transparency in Complex Computational Systems. Philosophy of Science 87, 4 (Oct. 2020), 568-589. https://doi.org/10.1086/709729
- Kimberlé Crenshaw. 1989. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. University of Chicago Legal Forum Vol.1989, Article 8 (1989). https: //chicagounbound.uchicago.edu/cgi/viewcontent.cgi?article=1052&context=uclf
- William Croft. 2001. Radical construction grammar: Syntactic theory in typological perspective . Oxford University Press on Demand.
- Rebecca Crootof. 2019. Artificial Intelligence Research Needs Responsible Publication Norms. Lawfare.
- Philippe Cudré-Mauroux, Hideaki Kimura, K-T Lim, Jennie Rogers, Roman Simakov, Emad Soroush, Pavel Velikhov, Daniel L Wang, Magdalena Balazinska, Jacek Becla, et al. 2009. A demonstration of SciDB: a science-oriented DBMS. Proceedings of the VLDB Endowment 2, 2 (2009), 1534-1537.
- Yuqing Cui. 2018. Application Of Zero-Knowledge Proof In Resolving Disputes Of Privileged Documents In E-Discovery. Harv. JL & Tech. 32 (2018), 633.
- Mariano-Florentino Cuéllar. 2019. How to ensure equal access to the law when we speak 200 different languages. CalMatters (Feb 2019). https://law.stanford.edu/2019/02/05/how-to-ensure-equal-access-to-the-law-when-we-speak-200-differentlanguages/
- Andrew M. Dai and Quoc V. Le. 2015. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems (NeurIPS) .
- Damai Dai, Li Dong, Y. Hao, Zhifang Sui, and Furu Wei. 2021a. Knowledge Neurons in Pretrained Transformers. ArXiv abs/2104.08696 (2021).
- Dengxin Dai and Luc Van Gool. 2018. Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC) . 3819-3824. https://doi.org/10.1109/ ITSC.2018.8569387
- Jessica Dai, Sina Fazelpour, and Zachary Lipton. 2021b. Fair Machine Learning Under Partial Compliance. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society . ACM. https://doi.org/10.1145/3461702.3462521
- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Florence, Italy, 2978-2988. https://doi.org/10. 18653/v1/P19-1285
- Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. 2018. Scaling Egocentric Vision: The EPICKITCHENS Dataset. In European Conference on Computer Vision (ECCV) .
- Alex Damian, Tengyu Ma, and Jason Lee. 2021. Label Noise SGD Provably Prefers Flat Global Minimizers. arXiv:2106.06530 [cs.LG]
- Jeanne E Daniel, Willie Brink, Ryan Eloff, and Charles Copley. 2019. Towards automating healthcare question answering in a noisy multilingual low-resource setting. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . 948-953.
- David Danks. 2019. The Value of Trustworthy AI. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society . ACM. https://doi.org/10.1145/3306618.3314228
- Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, and Christopher Ré. 2019. A kernel theory of modern data augmentation. In International Conference on Machine Learning . PMLR, 1528-1537.
- Helana Darwin. 2017. Doing Gender Beyond the Binary: A Virtual Ethnography. Symbolic Interaction 40, 3 (2017), 317-334. https://doi.org/10.1002/symb.316 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/symb.316
- Sudeep Dasari, F. Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, K. Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. 2019. RoboNet: Large-Scale Multi-Robot Learning. In Conference on Robot Learning (CORL) .
- Hal Daumé III. 2007. Frustratingly Easy Domain Adaptation. In Association for Computational Linguistics (ACL) .
- Thomas Davenport and Ravi Kalakota. 2019. The potential for artificial intelligence in healthcare. Future healthcare journal 6, 2 (2019), 94.
- Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing Factual Knowledge in Language Models. arXiv preprint arXiv:2104.08164 (2021).
- Harm de Vries, Dzmitry Bahdanau, and Christopher D. Manning. 2020. Towards Ecologically Valid Research on Language User Interfaces. arXiv preprint arXiv:2007.14435 (2020).
- Matthew DeCamp and Charlotta Lindvall. 2020. Latent bias and the implementation of artificial intelligence in medicine. Journal of the American Medical Informatics Association 27, 12 (June 2020), 2020-2023. https://doi.org/10.1093/jamia/ ocaa094
- Mostafa Dehghani, Yi Tay, Alexey Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. 2021. The Benchmark Lottery. ArXiv abs/2107.07002 (2021).
- Mauricio Delbracio, Damien Kelly, Michael S Brown, and Peyman Milanfar. 2021. Mobile Computational Photography: A Tour. arXiv preprint arXiv:2102.09000 (2021).
- Dina Demner-Fushman, Yassine Mrabet, and Asma Ben Abacha. 2020. Consumer health information and question answering: helping consumers find answers to their health-related information needs. Journal of the American Medical Informatics Association 27, 2 (2020), 194-201.
- Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori Hashimoto. 2021. Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL) (2021).
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR) . 248-255.
- Education Department of Health and Welfare. 1979. The Belmont Report.
- Tim Dettmers and Luke Zettlemoyer. 2019. Sparse Networks from Scratch: Faster Training Without Losing Performance. arXiv preprint arXiv:1907.04840 (2019).
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Association for Computational Linguistics (ACL) . 4171-4186.
- Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2021. Time-Aware Language Models as Temporal Knowledge Bases. arXiv:2106.15110 [cs.CL]
- MDiana and JJBJoS Marescaux. 2015. Robotic surgery. Journal of British Surgery 102, 2 (2015), e15-e28.
- Gregory M Dickinson. 2018. A Computational Analysis of Oral Argument in the Supreme Court. Cornell JL & Pub. Pol'y 28 (2018), 449.
- Thomas G Dietterich. 2000. Ensemble methods in machine learning. In International workshop on multiple classifier systems . Springer, 1-15.
- Catherine D'Ignazio and Lauren F. Klein. 2020. Data Feminism . MIT Press.
- R. Dillmann. 2004. Teaching and learning of robot tasks via observation of human performance. Robotics Auton. Syst. 47 (2004), 109-116.
- Botty Dimanov, Umang Bhatt, Mateja Jamnik, and Adrian Weller. 2020. You Shouldn't Trust Me: Learning Models Which Conceal Unfairness From Multiple Explanation Methods.. In SafeAI@ AAAI .
- Emily Dinan, Gavin Abercrombie, A. Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. 2021. Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling. ArXiv abs/2107.03451 (2021). https://arxiv.org/abs/2107.03451
- Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. 2020. Multi-Dimensional Gender Bias Classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Online, 314-331. https://doi.org/10.18653/v1/2020.emnlp-main.23
- Emily Dinan, Samuel Humeau, Bharath Chintagunta, and J. Weston. 2019. Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack. In EMNLP/IJCNLP .
- Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. 2021. CogView: Mastering Text-to-Image Generation via Transformers. CoRR abs/2105.13290 (2021). arXiv:2105.13290 https://arxiv.org/abs/2105.13290
- Yiming Ding, Carlos Florensa, Mariano Phielipp, and P. Abbeel. 2019. Goal-conditioned Imitation Learning. In Advances in Neural Information Processing Systems (NeurIPS) .
- Laurent Dinh, David Krueger, and Yoshua Bengio. 2015. NICE: Non-linear Independent Components Estimation. arXiv:1410.8516 [cs.LG]
- Renée DiResta and Shelby Grossman. 2019. Potemkin Pages & Personas: Assessing GRU Online Operations, 2014-2019.
- Renée DiResta, Shelby Grossman, and Alexandra Siegel. 2021. In-House Vs. Outsourced Trolls: How Digital Mercenaries Shape State Influence Strategies. Political Communication (2021), 1-31.
- Renée DiResta, K. Shaffer, Becky Ruppel, David Sullivan, Robert C. Matney, Ryan Fox, Jonathan Albright, and Ben Johnson. 2018. The tactics & tropes of the Internet Research Agency. https://digitalcommons.unl.edu/cgi/viewcontent.cgi?article= 1003&context=senatedocs
- Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Quentin Lhoest, Anton Sinitsin, Dmitry Popov, Dmitry Pyrkin, Maxim Kashirin, Alexander Borzunov, et al. 2021. Distributed Deep Learning in Open Collaborations. arXiv preprint arXiv:2106.10207 (2021).
- Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and Mitigating Unintended Bias in Text Classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (New Orleans, LA, USA) (AIES '18) . Association for Computing Machinery, New York, NY, USA, 67-73. https://doi.org/10.1145/3278721.3278729
- Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show Your Work: Improved Reporting of Experimental Results. arXiv:1909.03004 [cs.LG]
- Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner. 2021. Documenting the English Colossal Clean Crawled Corpus. CoRR abs/2104.08758 (2021). arXiv:2104.08758 https: //arxiv.org/abs/2104.08758
- Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Canton Ferrer. 2020. The deepfake detection challenge dataset. arXiv e-prints (2020), arXiv-2006.
- Xin Luna Dong, Hannaneh Hajishirzi, Colin Lockard, and Prashant Shiralkar. 2020. Multi-modal Information Extraction from Text, Semi-structured, and Tabular Data on the Web. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 3543-3544.
- Shayan Doroudi, Vincent Aleven, and Emma Brunskill. 2017. Robust Evaluation Matrix: Towards a More Principled Offline Exploration of Instructional Policies. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale (Cambridge, Massachusetts, USA) (L@S '17) . Association for Computing Machinery, New York, NY, USA, 3-12. https: //doi.org/10.1145/3051457.3051463
- Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017).
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations .
- A. Dosovitskiy, L. Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, M. Dehghani, Matthias Minderer, G. Heigold, S. Gelly, Jakob Uszkoreit, and N. Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ArXiv abs/2010.11929 (2021).
- Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems . ACM.
- Anca D Dragan and Siddhartha S Srinivasa. 2013. Formalizing Assistive Teleoperation. Robotics: Science and Systems VIII (2013), 73.
- T. Dreossi, Alexandre Donzé, and S. Seshia. 2017. Compositional Falsification of Cyber-Physical Systems with Machine Learning Components. In NFM .
- J. Drews. 2000. Drug discovery: a historical perspective. Science 287 5460 (2000), 1960-4.
- Simon S. Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. 2020. Few-Shot Learning via Learning the Representation, Provably. arXiv (2020).
- Sebastian Duerr and Peter A. Gloor. 2021. Persuasive Natural Language Generation -A Literature Review. arXiv:2101.05786 [cs.CL]
- Emmanuel Dupoux. 2018. Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner. Cognition 173 (2018), 43-59.
- Miquel Duran-Frigola, Eduardo Pauls, Oriol Guitart-Pla, Martino Bertoni, Víctor Alcalde, David Amat, Teresa Juan-Blanco, and Patrick Aloy. 2020. Extending the small-molecule similarity principle to all levels of biology with the Chemical Checker. Nature Biotechnology 38, 9 (2020), 1087-1096.
- Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In Proceedings of the 3rd Theory of Cryptography Conference . 265-284.
- Grégoire Déletang, J. Grau-Moya, Miljan Martic, Tim Genewein, Tom McGrath, Vladimir Mikulik, M. Kunesch, S. Legg, and Pedro A. Ortega. 2021. Causal Analysis of Agent Behavior for AI Safety. arXiv preprint arXiv:2103.03938 (2021).
- Mahmoud Elbattah, Émilien Arnaud, Maxime Gignon, and Gilles Dequen. 2021. The Role of Text Analytics in Healthcare: A Review of Recent Developments and Applications.. In HEALTHINF . 825-832.
- Paul Elbourne. 2011. Meaning: a slim guide to semantics . Oxford University Press.
- Kevin Ellis, Catherine Wong, Maxwell I. Nye, Mathias Sablé-Meyer, Lucas Morales, Luke B. Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2021. DreamCoder: bootstrapping inductive program synthesis with wakesleep library learning. In PLDI '21: 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, Virtual Event, Canada, June 20-25, 20211 , Stephen N. Freund and Eran Yahav (Eds.). ACM, 835-850. https://doi.org/10.1145/3453483.3454080
- Gamaleldin F Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. 2018. Adversarial reprogramming of neural networks. arXiv preprint arXiv:1806.11146 (2018).
- Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. 2020. Fast Sparse ConvNets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) .
- Daniel C Elton. 2020. Self-explaining AI as an alternative to interpretable AI. In International Conference on Artificial General Intelligence . Springer, 95-106.
- Emad Elwany, Dave Moore, and Gaurav Oberoi. 2019. Bert goes to law school: Quantifying the competitive advantage of access to large legal corpora in contract understanding. arXiv preprint arXiv:1911.00473 (2019).
- Douglas C. Engelbart. 1963. A Conceptual Framework for the Augmentation of Man's Intellect. In Computer-supported cooperative work: a book of readings .
- David Freeman Engstrom, Daniel E Ho, Catherine M Sharkey, and Mariano-Florentino Cuéllar. 2020. Government by algorithm: Artificial intelligence in federal administrative agencies. NYU School of Law, Public Law Research Paper 20-54 (2020).
- Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2018. Runaway feedback loops in predictive policing. In Conference on Fairness, Accountability and Transparency . PMLR, 160-171.
- Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. Understanding Undesirable Word Embedding Associations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Florence, Italy, 1696-1705. https://doi.org/10.18653/v1/P19-1166
- Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the Eye of the User: A Critique of NLP Leaderboards. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Online, 4846-4853. https://doi.org/10.18653/v1/2020.emnlp-main.393
- Allyson Ettinger. 2020. What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models. Transactions of the Association for Computational Linguistics 8 (2020), 34-48. https://doi.org/10.1162/tacl\_a\_00298
- Allyson Ettinger and Tal Linzen. 2016. Evaluating vector space models using human semantic priming results. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP . Association for Computational Linguistics, Berlin, Germany, 72-77. https://doi.org/10.18653/v1/W16-2513
- Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. 2020. Rigging the Lottery: Making All Tickets Winners. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119) , Hal Daumé III and Aarti Singh (Eds.). PMLR, 2943-2952.
- Tom Everitt, Gary Lea, and Marcus Hutter. 2018. AGI safety literature review. arXiv preprint arXiv:1805.01109 (2018).
- Benjamin Eysenbach, Shixiang Gu, J. Ibarz, and Sergey Levine. 2017. Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning. arXiv preprint arXiv:1711.06782 (2017).
- Jerry Alan Fails and Dan R. Olsen. 2003. A design tool for camera-based interaction. In Proceedings of the conference on Human factors in computing systems . ACM.
- Linxi Fan, Shyamal Buch, Guanzhi Wang, Ryan Cao, Yuke Zhu, Juan Carlos Niebles, and Li Fei-Fei. 2020. RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition. In Proceedings of the European Conference on Computer Vision (ECCV) .
- Donald J Farole, Jr. and Lynn Langston. 2010. County-based and Local Public Defender Offices, 2007 . Technical Report. U.S. Department of Justice Bureau of Justice Statistics.
- Sina Fazelpour and David Danks. 2021. Algorithmic bias: Senses, sources, solutions. Philosophy Compass 16, 8 (June 2021). https://doi.org/10.1111/phc3.12760
- William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 (2021).
- Li Fei-Fei, Asha Iyer, Christof Koch, and Pietro Perona. 2007. What do We Perceive in a Glance of a Real-World Scene? Journal of Vision 7, 1 (2007), 10-10.
- Xin Feng, Youni Jiang, Xuejiao Yang, Ming Du, and Xin Li. 2019. Computer vision algorithms and hardware implementations: A survey. Integration 69 (2019), 309-320.
- Andrew Guthrie Ferguson. 2017. The Rise of Big Data Policing: Surveillance, Race, and the Future of Law Enforcement . NYU Press. http://www.jstor.org/stable/j.ctt1pwtb27
- Besnik Fetahu, Abhijit Anand, and Avishek Anand. 2015. How much is Wikipedia lagging behind news?. In Proceedings of the ACM Web Science Conference . 1-9.
- Anjalie Field, Su Lin Blodgett, Zeerak Waseem, and Yulia Tsvetkov. 2021. A Survey of Race, Racism, and Anti-Racism in NLP. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Association for Computational Linguistics, Online, 1905-1925. https://doi.org/10.18653/v1/2021.acl-long.149
- Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In International Conference on Machine Learning (ICML) .
- Chelsea Finn and Sergey Levine. 2017. Deep visual foresight for planning robot motion. In International Conference on Robotics and Automation (ICRA) .
- C. Finn, S. Levine, and P. Abbeel. 2016a. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (ICML) . 49-58.
- Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. 2016b. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 512-519.
- Vlad Firoiu, Eser Aygün, Ankit Anand, Zafarali Ahmed, Xavier Glorot, Laurent Orseau, Doina Precup, and Shibl Mourad. 2021. Training a First-Order Theorem Prover from Synthetic Data. The First Mathematical Reasoning in General Artificial Intelligence Workshop, ICLR 2021 (2021). https://mathai-iclr.github.io/papers/papers/MATHAI\_18\_paper.pdf
- Jaime F. Fisac, Neil F. Lugovoy, Vicenç Rúbies Royo, S. Ghosh, and C. Tomlin. 2019. Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning. In International Conference on Robotics and Automation (ICRA) .
- Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension. In Workshop on Machine Reading for Question Answering (MRQA) .
- Carlos Florensa, Yan Duan, and P. Abbeel. 2017. Stochastic Neural Networks for Hierarchical Reinforcement Learning. ArXiv abs/1704.03012 (2017).
- Luciano Floridi, Josh Cowls, Monica Beltrametti, Raja Chatila, Patrice Chazerand, Virginia Dignum, Christoph Luetge, Robert Madelin, Ugo Pagallo, Francesca Rossi, Burkhard Schafer, Peggy Valcke, and Effy Vayena. 2018. AI4People-An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations. Minds and Machines 28, 4 (Nov. 2018), 689-707. https://doi.org/10.1007/s11023-018-9482-5
- Ruth C. Fong and Andrea Vedaldi. 2017. Interpretable Explanations of Black Boxes by Meaningful Perturbation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) .
- Stanislav Fort. 2021. Adversarial examples for the OpenAI CLIP in its zero-shot classification regime and their semantic generalization. https://stanislavfort.github.io/2021/01/12/OpenAI\_CLIP\_adversarial\_examples.html
- S. Frank, Irene Fernandez Monsalve, Robin L. Thompson, and G. Vigliocco. 2013. Reading time data for evaluating broadcoverage models of English sentence processing. Behavior Research Methods 45 (2013), 1182-1190.
- Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. 2015. Model inversion attacks that exploit confidence information and basic countermeasures. In ACM SIGSAC Conference on Computer and Communications Security .
- Jonathan B. Freeman, Andrew M. Penner, Aliya Saperstein, Matthias Scheutz, and Nalini Ambady. 2011. Looking the Part: Social Status Cues Shape Race Perception. PLOS ONE 6, 9 (09 2011), 1-10. https://doi.org/10.1371/journal.pone.0025107
- Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences 3 (1999).
- Batya Friedman and David G. Hendry. 2019. Value Sensitive Design: Shaping Technology with Moral Imagination . The MIT Press.
- Batya Friedman and Helen Nissenbaum. 1996. Bias in Computer Systems. ACM Transactions on Information Systems 14, 3 (July 1996), 330-347. https://doi.org/10.1145/230538.230561
- Zack Friedman. 2020. Student loan debt statistics in 2020: A record $1.6 trillion. https://www.forbes.com/sites/zackfriedman/ 2020/02/03/student-loan-debt-statistics/?sh=34191d3281fe
- Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. 2019. From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following. In International Conference on Learning Representations (ICLR) .
- Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. 2018. Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition. In Advances in Neural Information Processing Systems (NeurIPS) .
- Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Association for Computational Linguistics, Minneapolis, Minnesota, 32-42. https://doi.org/10.18653/v1/N19-1004
- Iason Gabriel. 2020. Artificial intelligence, values, and alignment. Minds and machines 30, 3 (2020), 411-437.
- Federico A Galatolo, Mario GCA Cimino, and Gigliola Vaglini. 2021. Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search. arXiv preprint arXiv:2102.01645 (2021).
- Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. 2020. Sparse GPU Kernels for Deep Learning. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis . IEEE, 1-14.
- Julia Galliers and Karen Spärck Jones. 1993. Evaluating Natural Language Processing Systems . University of Cambridge, Computer Laboratory. https://books.google.com/books?id=ZxklAQAAIAAJ
- Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Damian Mrowca, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, James J. DiCarlo, Josh McDermott, Joshua B. Tenenbaum, and Daniel L. K. Yamins. 2020. ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation. arXiv:2007.04954 [cs.CV]
- Oscar H. Gandy, Jr. 2021. The Panoptic Sort: A Political Economy of Personal Information (2 ed.). Oxford University Press.
- Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised Domain Adaptation by Backpropagation. In International Conference on Machine Learning (ICML) . 1180-1189.
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020a. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027 [cs.CL]
- Ruohan Gao, Changan Chen, Ziad Al-Halab, Carl Schissler, and Kristen Grauman. 2020b. VisualEchoes: Spatial Image Representation Learning through Echolocation. In ECCV .
- Tianyu Gao, Adam Fisch, and Danqi Chen. 2020c. Making Pre-trained Language Models Better Few-shot Learners. CoRR abs/2012.15723 (2020). arXiv:2012.15723 https://arxiv.org/abs/2012.15723
- Siddhant Garg and Goutham Ramakrishnan. 2020. Bae: Bert-based adversarial examples for text classification. arXiv preprint arXiv:2004.01970 (2020).
- Timnit Gebru. 2021. Race and Gender. In The Oxford Handbook of Ethics of AI , Markus Dirk Dubber, Frank Pasquale, and Sunit Das (Eds.). Oxford.
- Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Ill, and Kate Crawford. 2018. Datasheets for Datasets. arXiv preprint arXiv:1803.09010 (2018).
- Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020 . Association for Computational Linguistics, Online, 3356-3369. https://doi.org/10.18653/v1/2020.findings-emnlp.301
- Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) . Association for Computational Linguistics, Online, 96-120. https://doi.org/10.18653/v1/2021.gem-1.10
- Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP . Association for Computational Linguistics, Online, 163-173. https://doi.org/10. 18653/v1/2020.blackboxnlp-1.16
- Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut Learning in Deep Neural Networks. arXiv preprint arXiv:2004.07780 (2020).
- Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. 2018. Generalisation in humans and deep neural networks. Advances in neural information processing systems 31 (2018), 7538-7550.
- Samuel Genheden, Amol Thakkar, Veronika Chadimová, Jean-Louis Reymond, Ola Engkvist, and Esben Bjerrum. 2020. AiZynthFinder: a fast, robust and flexible open-source software for retrosynthetic planning. Journal of Cheminformatics 12, 1 (Nov. 2020). https://doi.org/10.1186/s13321-020-00472-1
- David Gershgorn. 2021. GitHub's Automatic Coding Tool Rests on Untested Legal Ground. https://www.theverge.com/ 2021/7/7/22561180/github-copilot-legal-copyright-fair-use-public-code.
- Moritz Gerstung, Elli Papaemmanuil, Inigo Martincorena, Lars Bullinger, Verena I Gaidzik, Peter Paschka, Michael Heuser, Felicitas Thol, Niccolo Bolli, Peter Ganly, et al. 2017. Precision oncology for acute myeloid leukemia using a knowledge bank approach. Nature Genetics 49, 3 (2017), 332-340.
- Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Association for Computational Linguistics, Hong Kong, China, 1161-1166. https://doi.org/10.18653/v1/D19-1107
- Mor Geva, R. Schuster, Jonathan Berant, and Omer Levy. 2020. Transformer Feed-Forward Layers Are Key-Value Memories. ArXiv abs/2012.14913 (2020).
- Negin Ghavami and Letitia Anne Peplau. 2013. An Intersectional Analysis of Gender and Ethnic Stereotypes: Testing Three Hypotheses. Psychology of Women Quarterly 37, 1 (2013), 113-127. https://doi.org/10.1177/0361684312464203 arXiv:https://doi.org/10.1177/0361684312464203
- Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2021. A Survey of Quantization Methods for Efficient Neural Network Inference. arXiv preprint arXiv:2103.13630 (2021).
- Amirata Ghorbani and James Zou. 2019. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning . PMLR, 2242-2251.
- James J Gibson. 1979. The ecological approach to visual perception . Psychology Press.
- Talia B Gillis and Jann L Spiess. 2019. Big data and discrimination. The University of Chicago Law Review 86, 2 (2019), 459-488.
- Antonio Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. 2019. Making AI Forget You: Data Deletion in Machine Learning. arXiv:1907.05012 [cs.LG]
- Kathryn T. Gines. 2011. Black Feminism and Intersectional Analyses. Philosophy Today 55, 9999 (2011), 275-284. https: //doi.org/10.5840/philtoday201155supplement68
- Jane C Ginsburg and Luke Ali Budiardjo. 2019. Authors and machines. Berkeley Tech. LJ 34 (2019), 343.
- Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition . 580-587.
- Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics .
- Abhinav Goel, Caleb Tung, Yung-Hsiang Lu, and George K Thiruvathukal. 2020b. A survey of methods for low-power deep learning and computer vision. In 2020 IEEE 6th World Forum on Internet of Things (WF-IoT) . IEEE, 1-6.
- Karan Goel, Albert Gu, Yixuan Li, and Christopher Ré. 2020a. Model Patching: Closing the Subgroup Performance Gap with Data Augmentation. arXiv preprint arXiv:2008.06775 (2020).
- Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher Ré. 2021. Robustness Gym: Unifying the NLP Evaluation Landscape. arXiv preprint arXiv:2101.04840 (2021).
- Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. 2021. Multimodal neurons in artificial neural networks. Distill 6, 3 (2021), e30.
- Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic Bias Metrics Do Not Correlate with Application Bias. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Association for Computational Linguistics, Online, 1926-1940. https://doi.org/10.18653/v1/2021.acl-long.150
- Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of NAACL 2019 .
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning . MIT press.
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems (NeurIPS) .
- Charles A.E. Goodhart. 1984. Problems of monetary management: the UK experience. In Monetary Theory and Practice . Springer, 91-121. https://link.springer.com/chapter/10.1007/978-1-349-17295-5\_4
- M. A. Goodrich and A. Schultz. 2007. Human-Robot Interaction: A Survey. Found. Trends Hum. Comput. Interact. 1 (2007), 203-275.
- Divya Gopinath, Monica Agrawal, Luke Murray, Steven Horng, David Karger, and David Sontag. 2020. Fast, Structured Clinical Documentation via Contextual Autocomplete. In Proceedings of the 5th Machine Learning for Healthcare Conference (Proceedings of Machine Learning Research, Vol. 126) , Finale Doshi-Velez, Jim Fackler, Ken Jung, David Kale, Rajesh Ranganath, Byron Wallace, and Jenna Wiens (Eds.). PMLR, 842-870. http://proceedings.mlr.press/v126/gopinath20a.html
- Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning. arXiv preprint arXiv:2002.08307 (2020).
- Assaf Gottlieb, Gideon Y Stein, Eytan Ruppin, and Roded Sharan. 2011. PREDICT: a method for inferring novel drug indications with application to personalized medicine. Molecular Systems Biology 7, 1 (2011), 496.
- Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. 2021. Larger-Scale Transformers for Multilingual Masked Language Modeling. arXiv preprint arXiv:2105.00572 (2021).
- R. Goyal, S. Kahou, Vincent Michalski, Joanna Materzynska, S. Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, P. Yianilos, Moritz Mueller-Freitag, F. Hoppe, Christian Thurau, I. Bax, and R. Memisevic. 2017a. The 'Something Something' Video Database for Learning and Evaluating Visual Common Sense. 2017 IEEE International Conference on Computer Vision (ICCV) (2017), 5843-5851.
- Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017b. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 6904-6913.
Christine Grady. 2015. Institutional Review Boards. Chest 148, 5 (Nov. 2015), 1148-1155. https://doi.org/10.1378/chest.15-0706 Don Grant, David Zelinka, and Stefania Mitova. 2021. Reducing CO2 emissions by targeting the world's hyper-polluting power plants. Environmental Research Letters (2021).
- Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. 2021. Ego4D: Around the World in 3,000 Hours of Egocentric Video. arXiv:2110.07058 [cs.CV]
- Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural Turing machines. arXiv preprint arXiv:1410.5401 (2014).
- Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. 2016. Hybrid computing using a neural network with dynamic external memory. Nature 538, 7626 (2016), 471-476.
- Jonathan Gray, Kavya Srinet, Yacine Jernite, Haonan Yu, Zhuoyuan Chen, Demi Guo, Siddharth Goyal, C Lawrence Zitnick, and Arthur Szlam. 2019. Craftassist: A framework for dialogue-enabled interactive agents. arXiv preprint arXiv:1907.08584 (2019).
- Anthony G. Greenwald, Debbie E. McGhee, and Jordan L.K. Schwartz. 1998. Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology 74, 6 (1998), 1464. https://psycnet. apa.org/record/1998-02892-004
- Shai Gretz, Yonatan Bilu, Edo Cohen-Karlik, and Noam Slonim. 2020. The workweek is the best time to start a family-A Study of GPT-2 Based Claim Generation. arXiv preprint arXiv:2010.06185 (2020).
- Jean-Bastien Grill, Florian Strub, Florent Altch'e, C. Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, B. A. Pires, Z. Guo, M. G. Azar, Bilal Piot, K. Kavukcuoglu, R. Munos, and Michal Valko. 2020. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. ArXiv abs/2006.07733 (2020).
- Stephen Grimm. 2021. Understanding. In The Stanford Encyclopedia of Philosophy (Summer 2021 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
- James Grimmelmann. 2015. There's No Such Thing as a Computer-Authored Work-And It's a Good Thing, Too. Colum. JL & Arts 39 (2015), 403.
- Jacob Grinfeld, Jyoti Nangalia, E Joanna Baxter, David C Wedge, Nicos Angelopoulos, Robert Cantrill, Anna L Godfrey, Elli Papaemmanuil, Gunes Gundem, Cathy MacLean, et al. 2018. Classification and personalized prognosis in myeloproliferative neoplasms. New England Journal of Medicine 379, 15 (2018), 1416-1430.
- Maura R Grossman and Gordon V Cormack. 2010. Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review. Rich. JL & Tech. 17 (2010), 1.
- Jian Guan. 2019. Artificial intelligence in healthcare and medicine: promises, ethical challenges and governance. Chinese Medical Sciences Journal 34, 2 (2019), 76-83.
- Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. 2017. Program Synthesis. Found. Trends Program. Lang. 4, 1-2 (2017), 1-119. https://doi.org/10.1561/2500000010
- Sumit Gulwani and Rishabh Singh. 2013. Automated Feedback Generation for Introductory Programming Assignments. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2013) (acm sigplan conference on programming language design and implementation (pldi 2013) ed.). 15-26. https://www.microsoft.com/en-us/research/ publication/automated-feedback-generation-for-introductory-programming-assignments/
- Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. 2018. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems . 9461-9471.
- Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. 2017. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems (NeurIPS) . 6151-6159.
- Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 (2020).
- Qi Guo, Chinmay Kulkarni, Aniket Kittur, Jeffrey P. Bigham, and Emma Brunskill. 2016. Questimator: Generating Knowledge Assessments for Arbitrary Topics. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence
(New York, New York, USA) (IJCAI'16) . AAAI Press, 3726-3732.
- Wei Guo and Aylin Caliskan. 2021. Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (Virtual Event, USA) (AIES '21) . Association for Computing Machinery, New York, NY, USA, 122-133. https://doi.org/10.1145/ 3461702.3462536
- Abhinav Gupta, Adithyavairavan Murali, Dhiraj Gandhi, and Lerrel Pinto. 2018. Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias. In NeurIPS .
- Abhishek Gupta, Justin Yu, Tony Zhao, Vikash Kumar, Aaron Rovinsky, Kelvin Xu, Thomas Devlin, and Sergey Levine. 2021b. Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention. arXiv preprint arXiv:2104.11203 (2021).
- Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. 2021a. Chasing Carbon: The Elusive Environmental Footprint of Computing. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA) . IEEE, 854-867.
- Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. arXiv:2004.10964 [cs.CL]
- Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation Artifacts in Natural Language Inference Data. In Association for Computational Linguistics (ACL) . 107-112.
- Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-Augmented Language Model Pre-Training. arXiv preprint arXiv:2002.08909 (2020).
- N. Haber, Damian Mrowca, Li Fei-Fei, and Daniel Yamins. 2018. Learning to Play with Intrinsically-Motivated Self-Aware Agents. In NeurIPS .
- Danijar Hafner, T. Lillicrap, Ian S. Fischer, Ruben Villegas, David R Ha, Honglak Lee, and James Davidson. 2019. Learning Latent Dynamics for Planning from Pixels. In International Conference on Machine Learning (ICML) .
- Martin J. Haigh. 1985. An Introduction to Computer-Aided Design and Manufacture . Blackwell Scientific Publications, Ltd., GBR.
- Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level Adversarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Association for Computational Linguistics, Online, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381
- Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2021. Glottolog 4.4 . Leipzig. https: //doi.org/10.5281/zenodo.4761960
- Jesse Michael Han, Jason Rute, Yuhuai Wu, Edward W. Ayers, and Stanislas Polu. 2021. Proof Artifact Co-training for Theorem Proving with Language Models. The First Mathematical Reasoning in General Artificial Intelligence Workshop, ICLR 2021 (2021). https://mathai-iclr.github.io/papers/papers/MATHAI\_23\_paper.pdf
- Jeffrey T Hancock, Mor Naaman, and Karen Levy. 2020. AI-Mediated Communication: Definition, Research Agenda, and Ethical Considerations. Journal of Computer-Mediated Communication (2020).
- D.J. Hand. 2010. Measurement Theory and Practice: The World Through Quantification . Wiley. https://books.google.com/ books?id=rap0PwAACAAJ
- Alex Hanna, Emily Denton, Andrew Smart, and Jamila Smith-Loud. 2020. Towards a critical race methodology in algorithmic fairness. In Proceedings of the 2020 conference on fairness, accountability, and transparency . 501-512.
- Stephen R Hanney, Sophie Castle-Clarke, Jonathan Grant, Susan Guthrie, Chris Henshall, Jorge Mestre-Ferrandiz, Michele Pistollato, Alexandra Pollitt, Jon Sussex, and Steven Wooding. 2015. How long does biomedical research take? Studying the time taken between biomedical and health research and its translation into products, policy, and practice. Health research policy and systems 13, 1 (2015), 1-18.
- Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. 2021a. Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. CoRR abs/2106.04156 (2021). arXiv:2106.04156 https://arxiv.org/abs/2106.04156
- Jeff Z HaoChen, Colin Wei, Jason Lee, and Tengyu Ma. 2021b. Shape matters: Understanding the implicit bias of the noise covariance. In Conference on Learning Theory . PMLR, 2315-2357.
- Albert Haque, Michelle Guo, Alexandre Alahi, Serena Yeung, Zelun Luo, Alisha Rege, Jeffrey Jopling, Lance Downing, William Beninati, Amit Singh, et al. 2017. Towards vision-based smart hospitals: a system for tracking and monitoring hand hygiene compliance. In Machine Learning for Healthcare Conference . PMLR, 75-87.
- Albert Haque, Arnold Milstein, and Li Fei-Fei. 2020. Illuminating the dark spaces of healthcare with ambient intelligence. Nature 585, 7824 (2020), 193-202.
- Sandra Harding. 2015. Objectivity and Diversity . University of Chicago Press. https://doi.org/doi:10.7208/9780226241531 Stefan Harrer, Pratik Shah, Bhavna Antony, and Jianying Hu. 2019. Artificial intelligence for clinical trial design. Trends in pharmacological sciences 40, 8 (2019), 577-591.
- Peter Hase, Mona T. Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2021. Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs. CoRR abs/2111.13654 (2021). https://arxiv.org/abs/2111.13654
- Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. 2018. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning . PMLR, 1929-1938.
- Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2021. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 (2021).
- Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2019. Momentum Contrast for Unsupervised Visual Representation Learning. arXiv preprint arXiv:1911.05722 (2019).
- Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), 9726-9735.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016a. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770-778.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b. Deep Residual Learning for Image Recognition. In Computer Vision and Pattern Recognition (CVPR) .
- Allison Hegel, Marina Shah, Genevieve Peaslee, Brendan Roof, and Emad Elwany. 2021. The Law of Large Documents: Understanding the Structure of Legal Contracts Using Visual Cues. arXiv preprint arXiv:2107.08128 (2021).
- Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 961-970.
- Robert Heilmayr, Cristian Echeverría, and Eric F Lambin. 2020. Impacts of Chilean forest subsidies on forest cover, carbon and biodiversity. Nature Sustainability 3, 9 (2020), 701-709.
- Christina Heinze-Deml and Nicolai Meinshausen. 2017. Conditional variance penalties and domain shift robustness. arXiv preprint arXiv:1710.11469 (2017).
- Kyle Helfrich, Devin Willmott, and Qiang Ye. 2018. Orthogonal recurrent neural networks with scaled Cayley transform. In International Conference on Machine Learning . PMLR, 1969-1978.
- Joseph M Hellerstein and Michael Stonebraker. 2005. Readings in database systems . MIT press.
- Deborah Hellman. 2020. Measuring algorithmic fairness. Va. L. Rev. 106 (2020), 811.
- Deborah Hellman. 2021. Big Data and Compounding Injustice. Journal of Moral Philosophy, forthcoming, Virginia Public Law and Legal Theory Research Paper 2021-27 (2021).
- Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2016. Tracking the world state with recurrent entity networks. arXiv preprint arXiv:1612.03969 (2016).
- Olivier J Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carreira. 2021. Efficient visual pretraining with contrastive detection. ICCV (2021).
- Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. 2020. Towards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research 21, 248 (2020), 1-43.
- Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2017. Ethical Challenges in Data-Driven Dialogue Systems. In AAAI/ACM AI Ethics and Society Conference .
- DanHendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR) .
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations . https: //openreview.net/forum?id=d7KBjmI3GmQ
- Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021c. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268 (2021).
- Dan Hendrycks, Nicholas Carlini, John Schulman, Tom Dietterich, and Jacob Steinhardt. 2021d. Unsolved Problems in ML Safety. arXiv preprint (2021).
- Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR) .
- Dan Hendrycks, Kimin Lee, and Mantas Mazeika. 2019a. Using Pre-Training Can Improve Model Robustness and Uncertainty. In International Conference on Machine Learning (ICML) .
- Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. arXiv preprint arXiv:2004.06100 (2020).
- Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. 2019b. Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty. In Advances in Neural Information Processing Systems (NeurIPS) .
- Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. 2021e. Natural Adversarial Examples. arXiv:1907.07174 [cs.LG]
- Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, T. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. 2020. Scaling Laws for Autoregressive Generative Modeling. ArXiv abs/2010.14701 (2020). https://arxiv.org/abs/2010.14701
- Sylvia L. Herbert, Jason J. Choi, Suvansh Qazi, Marsalis Gibson, K. Sreenath, and C. Tomlin. 2021. Scalable Learning of Safety Guarantees for Autonomous Systems using Hamilton-Jacobi Reachability. arXiv preprint arXiv:2101.05916 (2021).
- Maguire Herriman, Elana Meer, Roy Rosin, Vivian Lee, Vindell Washington, and Kevin G Volpp. 2020. Asked and answered: building a chatbot to address Covid-19-related concerns. Nejm Catalyst Innovations in Care Delivery (2020).
- J. Hestness, Sharan Narang, Newsha Ardalani, G. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Y. Yang, and Yanqi Zhou. 2017. Deep Learning Scaling is Predictable, Empirically. ArXiv abs/1712.00409 (2017).
- John Hewitt and Percy Liang. 2019. Designing and Interpreting Probes with Control Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Association for Computational Linguistics, Hong Kong, China. https://www. aclweb.org/anthology/D19-1275
- John Hewitt and Christopher D. Manning. 2019. A Structural Probe for Finding Syntax in Word Representations. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) (Minneapolis, USA). Association for Computational Linguistics.
- Hidalgo. 2021. How Humans Judge Machines . The MIT Press, Cambridge, Massachusetts.
- Brian Hie, Ellen D Zhong, Bonnie Berger, and Bryan Bryson. 2021. Learning the language of viral evolution and escape. Science 371, 6526 (2021), 284-288.
- Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531 (2015).
- Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. 2006. A fast learning algorithm for deep belief nets. Neural computation 18, 7 (2006), 1527-1554.
- Daniel E Ho and Alice Xiang. 2020. Affirmative Algorithms: The Legal Grounds for Fairness as Awareness. U. Chi. L. Rev. Online (2020), 134.
- Jonathan Ho, Ajay Jain, and P. Abbeel. 2020. Denoising Diffusion Probabilistic Models. ArXiv abs/2006.11239 (2020).
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735-1780.
- Bas Hofstra, Vivek V. Kulkarni, Sebastian Munoz-Najar Galvez, Bryan He, Dan Jurafsky, and Daniel A. McFarland. 2020. The Diversity-Innovation Paradox in Science. Proceedings of the National Academy of Sciences 117, 17 (April 2020), 9284-9291. https://doi.org/10.1073/pnas.1915378117
- Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE transactions on visualization and computer graphics 25, 8 (2018), 2674-2693.
- Fred Hohman, Kanit Wongsuphasawat, Mary Beth Kery, and Kayur Patel. 2020. Understanding and visualizing data iteration in machine learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . 1-13.
- Karen D. Holl and Pedro H. S. Brancalion. 2020. Tree planting is not a simple solution. Science 368, 6491 (May 2020), 580-581. https://doi.org/10.1126/science.aba8232 ZSCC: 0000092 Publisher: American Association for the Advancement of Science Section: Perspective.
- Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018. The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards. arXiv:1805.03677 [cs.DB]
- Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. 2020. ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation. In Proceedings of the 12th Language Resources and Evaluation Conference . European Language Resources Association, Marseille, France, 138-146.
- Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations (ICLR) .
- Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2020. A dataset for statutory reasoning in tax law entailment and question answering. arXiv preprint arXiv:2005.05257 (2020).
- Andreas Holzinger, Georg Langs, Helmut Denk, Kurt Zatloukal, and Heimo Müller. 2019. Causability and explainability of artificial intelligence in medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 9, 4 (2019), e1312.
- Spurthi Amba Hombaiah, Tao Chen, Mingyang Zhang, Michael Bendersky, and Marc Najork. 2021. Dynamic Language Models for Continuously Evolving Content. CoRR abs/2106.06297 (2021). arXiv:2106.06297 https://arxiv.org/abs/2106. 06297
- Joey Hong, David Dohan, Rishabh Singh, Charles Sutton, and Manzil Zaheer. 2021. Latent Programmer: Discrete Latent Codes for Program Synthesis. In International Conference in Machine Learning (ICML) .
- Jason I Hong and James A Landay. 2004. An architecture for privacy-sensitive ubiquitous computing. In Proceedings of the 2nd international conference on Mobile systems, applications, and services . 177-189.
- Lu Hong and Scott E Page. 2004. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Science 101, 46 (2004).
- Sanghyun Hong, Yiğitcan Kaya, Ionuţ-Vlad Modoranu, and Tudor Dumitraş. 2020a. A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference. arXiv preprint arXiv:2010.02432 (2020).
- Songbai Hong, Guodong Yin, Shilong Piao, Ray Dybzinski, Nan Cong, Xiangyi Li, Kai Wang, Josep Peñuelas, Hui Zeng, and Anping Chen. 2020b. Divergent responses of soil organic carbon to afforestation. Nature Sustainability 3, 9 (2020), 694-700.
- Sara Hooker. 2020. The hardware lottery. arXiv preprint arXiv:2009.06489 (2020).
- Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily L. Denton. 2020. Characterising Bias in Compressed Models. ArXiv abs/2010.03058 (2020). https://arxiv.org/abs/2010.03058
- Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . ACM.
- Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-Efficient Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97) , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 2790-2799. http://proceedings.mlr.press/v97/houlsby19a.html
- Dirk Hovy and Anders Søgaard. 2015. Tagging Performance Correlates with Age. In Association for Computational Linguistics (ACL) . 483-488.
- Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Association for Computational Linguistics (ACL) .
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. CoRR abs/2106.09685 (2021). arXiv:2106.09685 https://arxiv.org/abs/ 2106.09685
- Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080 (2020).
- Daniel Huang, Prafulla Dhariwal, Dawn Song, and Ilya Sutskever. 2018. GamePad: A Learning Environment for Theorem Proving. CoRR abs/1806.00608 (2018). arXiv:1806.00608 http://arxiv.org/abs/1806.00608
- Jianping Huang, Haipeng Yu, Xiaodan Guan, Guoyin Wang, and Ruixia Guo. 2016. Accelerated dryland expansion under climate change. Nature Climate Change 6, 2 (Feb. 2016), 166-171. https://doi.org/10.1038/nclimate2837 ZSCC: 0001034 Bandiera\_abtest: a Cg\_type: Nature Research Journals Number: 2 Primary\_atype: Research Publisher: Nature Publishing Group Subject\_term: Developing world;Projection and prediction Subject\_term\_id: developing-world;projection-andprediction.
- Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. 2021a. Therapeutics data commons: machine learning datasets and tasks for therapeutics. arXiv preprint arXiv:2102.09548 (2021).
- Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. Advances in neural information processing systems 32 (2019), 103-112.
- Zihan Huang, Charles Low, Mengqiu Teng, Hongyi Zhang, Daniel E Ho, Mark S Krass, and Matthias Grabmair. 2021b. Context-Aware Legal Citation Recommendation using Deep Learning. arXiv preprint arXiv:2106.10776 (2021).
- Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. 2021c. Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 12976-12985.
- Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. 2019. Risks from Learned Optimization in Advanced Machine Learning Systems. ArXiv abs/1906.01820 (2019).
- Drew Hudson and Christopher D Manning. 2019a. Learning by Abstraction: The neural state machine. In Advances in Neural Information Processing Systems . 5903-5916.
- Drew A Hudson and Christopher D Manning. 2018. Compositional Attention Networks for Machine Reasoning. In International Conference on Learning Representations (ICLR) .
- Drew A Hudson and Christopher D Manning. 2019b. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 6700-6709.
- Drew A Hudson and C. Lawrence Zitnick. 2021. Generative Adversarial Transformers. Proceedings of the 38th International Conference on Machine Learning, ICML 2021 (2021).
- James P Hughes, Stephen Rees, S Barrett Kalindjian, and Karen L Philpott. 2011. Principles of early drug discovery. British journal of pharmacology 162, 6 (2011), 1239-1249.
- Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social Biases in NLP Models as Barriers for Persons with Disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 5491-5501. https://doi.org/10.18653/ v1/2020.acl-main.487
- Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs. In AAAI .
- Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. 2019. Learning agile and dynamic motor skills for legged robots. Science Robotics 4, 26 (2019).
- Janet Shibley Hyde, Rebecca S. Bigler, Daphna Joel, Charlotte Chucky Tate, and Sari M. van Anders. 2019. The Future of Sex and Gender in Psychology: Five Challenges to the Gender Binary. American Psychologist 74 (2019), 171-193.
- H. Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. 2021. TABBIE: Pretrained Representations of Tabular Data. In NAACL .
- Robert Ikeda and Jennifer Widom. 2010. Panda: A system for provenance and data. (2010).
- Daniela Ionescu et al. 2020. Deep learning algorithms and big health care data in clinical natural language processing. Linguistic and Philosophical Investigations 19 (2020), 86-92.
- Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and D. Eck. 2020. Automatic Detection of Generated Text is Easiest when Humans are Fooled. In ACL .
- Robert L. Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models. CoRR abs/2106.13353 (2021). arXiv:2106.13353 https://arxiv.org/abs/2106.13353
- Ray Jackendoff. 2011. What is the human language faculty? Two views. Language 87, 3 (2011), 586-624. http://www.jstor. org/stable/23011656
- Simon Jackman. 2008. Measurement . Oxford Handbooks. https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/ 9780199286546.001.0001/oxfordhb-9780199286546-e-6
- Abigail Z. Jacobs and Hanna Wallach. 2021. Measurement and Fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT '21) . Association for Computing Machinery, New York, NY, USA, 375-385. https://doi.org/10.1145/3442188.3445901
- Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? arXiv preprint arXiv:2004.03685 (2020).
- Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency . 624-635.
- Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. 2021a. Perceiver IO: A General Architecture for Structured Inputs & Outputs. arXiv preprint arXiv:2107.14795 (2021).
- Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and João Carreira. 2021b. Perceiver: General Perception with Iterative Attention. In International Conference on Machine Learning (ICML) .
- M. Jamnik. 2001. Mathematical Reasoning with Diagrams.
- Michael Janner, Qiyang Li, and Sergey Levine. 2021. Reinforcement Learning as One Big Sequence Modeling Problem. ArXiv abs/2106.02039 (2021).
- Theo MV Janssen and Barbara H Partee. 1997. Compositionality. In Handbook of logic and language . Elsevier, 417-473.
- Karol Jaroch, Alina Jaroch, and Barbara Bojko. 2018. Cell cultures in drug discovery and development: The need of reliable in vitro-in vivo extrapolation for pharmacodynamics and pharmacokinetics assessment. Journal of Pharmaceutical and Biomedical Analysis 147 (2018), 297-312.
- Shervin Javdani, Henny Admoni, Stefania Pellegrinelli, Siddhartha S Srinivasa, and J Andrew Bagnell. 2018. Shared autonomy via hindsight optimization for teleoperation and teaming. International Journal of Robotics Research (IJRR) 37 (2018), 717-742.
- Siddhant M Jayakumar, Razvan Pascanu, Jack W Rae, Simon Osindero, and Erich Elsen. 2021. Top-KAST: Top-K Always Sparse Training. arXiv preprint arXiv:2106.03517 (2021).
- Neal Jean, Marshall Burke, Michael Xie, W Matthew Davis, David B Lobell, and Stefano Ermon. 2016. Combining satellite imagery and machine learning to predict poverty. Science 353, 6301 (2016), 790-794.
- Emily Jensen, Meghan Dale, Patrick J Donnelly, Cathlyn Stone, Sean Kelly, Amanda Godley, and Sidney K D'Mello. 2020. Toward automated feedback on teacher discourse to enhance teacher learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . 1-13.
- Sooyeon Jeong, Kristopher Dos Santos, Suzanne Graca, Brianna O'Connell, Laurel Anderson, Nicole Stenquist, Katie Fitzpatrick, Honey Goodenough, Deirdre Logan, Peter Weinstock, et al. 2015. Designing a socially assistive robot for pediatric care. In Proceedings of the 14th international conference on interaction design and children . 387-390.
- Y Ji, Z Zhou, H Liu, and RV Davuluri. 2021. DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformatics (2021).
- Shengyu Jia, Tao Meng, Jieyu Zhao, and Kai-Wei Chang. 2020. Mitigating Gender Bias Amplification in Distribution by Posterior Regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 2936-2942. https://doi.org/10.18653/v1/2020.acl-main.264
- Zhihao Jia, James Thomas, Tod Warszawski, Mingyu Gao, Matei Zaharia, and Alex Aiken. 2019a. Optimizing DNN Computation with Relaxed Graph Substitutions. SysML 2019 (2019).
- Zhihao Jia, Matei Zaharia, and Alex Aiken. 2019b. Beyond Data and Model Parallelism for Deep Neural Networks. SysML 2019 (2019).
- Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics 8 (2020), 423-438. https://doi.org/10.1162/tacl\_a\_00324
- Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence , Vol. 34. 8018-8025.
- Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu, Chuanqi Tan, Mosha Chen, Songfang Huang, Xiaozhong Liu, and Sheng Yu. 2021. Biomedical question answering: A comprehensive review. arXiv preprint arXiv:2102.05281 (2021).
- Wengong Jin, Regina Barzilay, and Tommi Jaakkola. 2018. Junction tree variational autoencoder for molecular graph generation. In International Conference on Machine Learning . PMLR, 2323-2332.
- Eun Seo Jo and Timnit Gebru. 2020. Lessons from archives: Strategies for collecting sociocultural data in machine learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency . 306-316.
- Gabbrielle M. Johnson. 2020. Algorithmic bias: on the implicit biases of social technology. Synthese (June 2020). https: //doi.org/10.1007/s11229-020-02696-y
- Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data (2019).
- Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition . 2901-2910.
- Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . 6282-6293.
- Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. 2017. In-Datacenter Performance Analysis of a Tensor Processing Unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture . 1-12.
- Kyle D. Julian and Mykel J. Kochenderfer. 2019. Guaranteeing Safety for Neural Network-Based Aircraft Collision Avoidance Systems. 2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC) (Sep 2019). https://doi.org/10.1109/dasc43569. 2019.9081748
- John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Kathryn Tunyasuvunakool, Olaf Ronneberger, Russ Bates, Augustin Žídek, Alex Bridgland, et al. 2020. High accuracy protein structure prediction using deep learning. Fourteenth Critical Assessment of Techniques for Protein Structure Prediction (Abstract Book) 22 (2020), 24.
- D. Jurafsky and J.H. Martin. 2009. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition . Pearson Prentice Hall. https://books.google.gr/books?id=fZmj5UNK8AQC
- David Jurgens, Libby Hemphill, and Eshwar Chandrasekharan. 2019. A Just and Comprehensive Strategy for Using NLP to Address Online Abuse. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Florence, Italy, 3658-3666. https://doi.org/10.18653/v1/P19-1357
- Artur Kadurin, Sergey Nikolenko, Kuzma Khrabrov, Alex Aliper, and Alex Zhavoronkov. 2017. druGAN: an advanced generative adversarial autoencoder model for de novo generation of new molecules with desired molecular properties in silico. Molecular pharmaceutics 14, 9 (2017), 3098-3104.
- Leslie Kaelbling. 1993. Learning to Achieve Goals. In International Joint Conference on Artificial Intelligence (IJCAI) .
- Hilda Kajbaf. 2019. The First Amendment and Modern Technology: The Free Speech Clause and Chatbot Speech. Hastings Const. LQ 47 (2019), 337.
- Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. 2021. MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale. arXiv preprint arXiv:2104.08212 (2021).
- Pratyusha Kalluri. 2020. Don't ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583, 169 (2020). https://doi.org/10.1038/d41586-020-02003-2
- Daniel Kang, Deepti Raghavan, Peter Bailis, and Matei Zaharia. 2020. Model Assertions for Monitoring and Improving ML Models. arXiv preprint arXiv:2003.01668 (2020).
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. ArXiv abs/2001.08361 (2020). https: //arxiv.org/abs/2001.08361
- Siddharth Karamcheti, Dorsa Sadigh, and Percy Liang. 2020. Learning Adaptive Language Interfaces through Decomposition. In EMNLP Workshop for Interactive and Executable Semantic Parsing (IntEx-SemPar) .
- Siddharth Karamcheti, Edward C. Williams, Dilip Arumugam, Mina Rhee, Nakul Gopalan, Lawson L. S. Wong, and Stefanie Tellex. 2017. A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructions. In First Workshop on Language Grounding for Robotics @ ACL .
- Siddharth Karamcheti, Albert J. Zhai, Dylan P. Losey, and Dorsa Sadigh. 2021. Learning Visually Guided Latent Actions for Assistive Teleoperation. In Learning for Dynamics and Control .
- Holden Karnofsky. 2016. Potential risks from advanced artificial intelligence: the philanthropic opportunity. Open Philanthropy Project 6 (2016).
- Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Online, 6769-6781. https://doi.org/10.18653/v1/2020.emnlp-main.550
- Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 8110-8119.
- Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models. In EACL .
- Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh. 2020. Pre-training without natural images. In Proceedings of the Asian Conference on Computer Vision .
- Michael Katell, Meg Young, Dharma Dailey, Bernease Herman, Vivian Guetler, Aaron Tam, Corinne Bintz, Daniella Raz, and P. M. Krafft. 2020. Toward situated interventions for algorithmic equity. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency . ACM. https://doi.org/10.1145/3351095.3372874
- Daniel Martin Katz, Michael J Bommarito, and Josh Blackman. 2017. A general approach for predicting the behavior of the Supreme Court of the United States. PloS one 12, 4 (2017), e0174698.
- Amit Kaushal, Russ Altman, and Curt Langlotz. 2020. Geographic distribution of US cohorts used to train deep learning algorithms. Jama 324, 12 (2020), 1212-1213.
- Matthew Kay, Cynthia Matuszek, and Sean A. Munson. 2015. Unequal Representation and Gender Stereotypes in Image Search Results for Occupations. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15) . ACM, New York, NY, USA, 3819-3828. https://doi.org/10.1145/2702123.2702520
- Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 (2017).
- Alexander Ke, William Ellsworth, Oishi Banerjee, Andrew Y Ng, and Pranav Rajpurkar. 2021. CheXtransfer: performance and parameter efficiency of ImageNet models for chest X-Ray interpretation. In Proceedings of the Conference on Health, Inference, and Learning . 116-124.
- Sean P Keehan, Gigi A Cuckler, John A Poisal, Andrea M Sisko, Sheila D Smith, Andrew J Madison, Kathryn E Rennie, Jacqueline A Fiore, and James C Hardesty. 2020. National Health Expenditure Projections, 2019-28: Expected Rebound In Prices Drives Rising Spending Growth: National health expenditure projections for the period 2019-2028. Health Affairs 39, 4 (2020), 704-714.
- Patrick Gage Kelley, Joanna Bresee, Lorrie Faith Cranor, and Robert W Reeder. 2009. A" nutrition label" for privacy. In Proceedings of the 5th Symposium on Usable Privacy and Security . 1-12.
- Tom Kenter, Melvin Wevers, Pim Huijnen, and Maarten De Rijke. 2015. Ad hoc monitoring of vocabulary shifts over time. In Proceedings of the 24th ACM international on conference on information and knowledge management . 1191-1200.
- Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of Language Agents. ArXiv abs/2103.14659 (2021).
- Clark Kerr. 2001. The Uses of the University . Harvard University Press.
- Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858 (2019).
- Os Keyes. 2018. The Misgendering Machines. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (Nov. 2018), 1-22. https://doi.org/10.1145/3274357
- Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through Memorization: Nearest Neighbor Language Models. In International Conference on Learning Representations . https:
## //openreview.net/forum?id=HklBjCEKvH
- Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A Smith, and Daniel S Weld. 2021. Genie: A leaderboard for human-in-the-loop evaluation of text generation. arXiv preprint arXiv:2101.06561 (2021).
- Omar Khattab, Christopher Potts, and Matei Zaharia. 2020. Relevance-Guided Supervision for OpenQA with ColBERT. (2020). arXiv:https://arxiv.org/abs/2007.00814
- O. Khattab and M. Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (2020).
- Behrokh Khoshnevis. 2004. Automated construction by contour crafting-related robotics and information technologies. Automation in construction 13, 1 (2004), 5-19.
- Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. 2021. Dynabench: Rethinking benchmarking in NLP. arXiv preprint arXiv:2104.14337 (2021).
- Najoung Kim and Tal Linzen. 2020. COGS: A Compositional Generalization Challenge Based on Semantic Interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . 9087-9105.
- Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. 2021b. Code prediction by feeding trees to transformers. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 150-162.
- Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021a. ViLT: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning (ICML) .
- Diederik P. Kingma and Prafulla Dhariwal. 2018. Glow: Generative Flow with Invertible 1x1 Convolutions. In NeurIPS .
- Diederik P. Kingma and M. Welling. 2014. Auto-Encoding Variational Bayes. CoRR abs/1312.6114 (2014).
- Darrell G Kirch and Kate Petelle. 2017. Addressing the physician shortage: the peril of ignoring demography. Jama 317, 19 (2017), 1947-1948.
- James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114, 13 (2017), 3521-3526.
- Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer. arXiv preprint arXiv:2001.04451 (2020).
- Predrag Klasnja and Wanda Pratt. 2012. Healthcare in the pocket: mapping the space of mobile-phone health interventions. Journal of biomedical informatics 45, 1 (2012), 184-198.
- Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017. Inherent Trade-offs in the Fair Determination of Risk Scores. In Innovations in Theoretical Computer Science (ITCS) .
- Jon Kleinberg and Manish Raghavan. 2021. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences 118, 22 (2021). https://doi.org/10.1073/pnas.2018340118 arXiv:https://www.pnas.org/content/118/22/e2018340118.full.pdf
- A. S. Klyubin, D. Polani, and Chrystopher L. Nehaniv. 2005. Empowerment: a universal agent-centric measure of control. 2005 IEEE Congress on Evolutionary Computation 1 (2005), 128-135 Vol.1.
- Robert P Kocher. 2021. Reducing administrative waste in the US health care system. JAMA 325, 5 (2021), 427-428.
- Kenneth R. Koedinger and John R. Anderson. 1990. Abstract planning and perceptual chunks: Elements of expertise in geometry. Cognitive Science 14, 4 (1990), 511-550. https://doi.org/10.1016/0364-0213(90)90008-K
- Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences 117, 14 (2020), 7684-7689.
- Pang Wei Koh and Percy Liang. 2017. Understanding Black-box Predictions via Influence Functions. In International Conference on Machine Learning (ICML) .
- Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. 2021. WILDS: A Benchmark of in-the-Wild Distribution Shifts. In International Conference on Machine Learning (ICML) .
- Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017a. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv preprint arXiv:1712.05474 (2017).
- Eric Kolve, R. Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017b. AI2-THOR: An Interactive 3D Environment for Visual AI. ArXiv abs/1712.05474 (2017).
- Jun Kong, Lee AD Cooper, Fusheng Wang, David A Gutman, Jingjing Gao, Candace Chisolm, Ashish Sharma, Tony Pan, Erwin G Van Meir, Tahsin M Kurc, et al. 2011. Integrative, multimodal analysis of glioblastoma using TCGA molecular
data, pathology images, and clinical outcomes.
IEEE Transactions on Biomedical Engineering
58, 12 (2011), 3469-3474.
- Diane M Korngiebel and Sean D Mooney. 2021. Considering the possibilities and pitfalls of Generative Pre-trained Transformer 3 (GPT-3) in healthcare delivery. NPJ Digital Medicine 4, 1 (2021), 1-3.
- Christine Korsgaard. 2009. Self-constitution : agency, identity, and integrity . Oxford University Press, Oxford New York.
- SE Kreps and DL Kriner. 2020. Model uncertainty, political contestation, and public trust in science: Evidence from the COVID-19 pandemic. Science advances 6, 43 (2020), eabd4563.
- Sarah Kreps, R. Miles McCain, and Miles Brundage. 2020. All the News That's Fit to Fabricate: AI-Generated Text as a Tool of Media Misinformation. Journal of Experimental Political Science (2020), 1-14. https://doi.org/10.1017/XPS.2020.37
- Kundan Krishna, Sopan Khosla, Jeffrey P Bigham, and Zachary C Lipton. 2020. Generating soap notes from doctor-patient conversations. arXiv preprint arXiv:2005.01795 (2020).
- Kalpesh Krishna, Gaurav Singh Tomar, Ankur P Parikh, Nicolas Papernot, and Mohit Iyyer. 2019. Thieves on sesame street! model extraction of bert-based apis. arXiv preprint arXiv:1910.12366 (2019).
- Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidi, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123 (2017), 32-73.
- Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012), 1097-1105.
- Harlan M Krumholz, Sharon F Terry, and Joanne Waldstreicher. 2016. Data acquisition, curation, and use for a continuously learning health system. Jama 316, 16 (2016), 1669-1670.
- Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora, and Rong Ge. 2019. Explaining landscape connectivity of low-cost solutions for multilayer nets. arXiv preprint arXiv:1906.06247 (2019).
- Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In EMNLP .
- Ananya Kumar, Tengyu Ma, and Percy Liang. 2020a. Understanding Self-Training for Gradual Domain Adaptation. In International Conference on Machine Learning (ICML) .
- Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. 2022. Fine-Tuning Distorts Pretrained Features and Underperforms Out-of-Distribution. In International Conference on Learning Representations (ICLR) .
- Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020b. Conservative Q-Learning for Offline Reinforcement Learning. (2020). https://arxiv.org/abs/2006.04779
- Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. arXiv preprint arXiv:1906.07337 (2019).
- Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 (2019).
- Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. 2016. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth international conference on 3D vision (3DV) . IEEE, 239-248.
- Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-tosequence recurrent networks. In International conference on machine learning . PMLR, 2873-2882.
- Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science 350, 6266 (2015), 1332-1338.
- Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. Behavioral and Brain Sciences 40 (2017).
- George Lakoff and Rafael Núñez. 2000. Where Mathematics Comes From: How the Embodied Mind Brings Mathematics into Being . Basic Books, New York. http://perso.unifr.ch/rafael.nunez/welcome.html
- Samuel Lalmuanawma, Jamal Hussain, and Lalrinfela Chhakchhuak. 2020. Applications of machine learning and artificial intelligence for Covid-19 (SARS-CoV-2) pandemic: A review. Chaos, Solitons & Fractals (2020), 110059.
- Madeline Lamo and Ryan Calo. 2019. Regulating bot speech. UCLA L. Rev. 66 (2019), 988.
- Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and H. Jégou. 2019. Large Memory Layers with Product Keys. In NeurIPS .
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv e-prints (2019), arXiv-1909.
- Gert RG Lanckriet, Tijl De Bie, Nello Cristianini, Michael I Jordan, and William Stafford Noble. 2004. A statistical framework for genomic data fusion. Bioinformatics 20, 16 (2004), 2626-2635.
- Sascha Lange, Thomas Gabel, and Martin Riedmiller. 2012. Batch reinforcement learning. In Reinforcement learning . Springer, 45-73.
- Lynn Langston and Donald J Farole, Jr. 2010. State Public Defender Programs, 2007 . Technical Report. U.S. Department of Justice Bureau of Justice Statistics.
- Loïc Lannelongue, Jason Grealey, and Michael Inouye. 2021. Green algorithms: Quantifying the carbon footprint of computation. Advanced Science (2021), 2100707.
- Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. 2019. Unmasking Clever Hans predictors and assessing what machines really learn. Nature communications 10, 1 (2019), 1-8.
- Jill H. Larkin and Herbert A. Simon. 1987. Why a Diagram is (Sometimes) Worth Ten Thousand Words. Cognitive Science 11, 1 (1987), 65-100. https://doi.org/10.1111/j.1551-6708.1987.tb00863.x arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1551-6708.1987.tb00863.x
- Josh Lauer. 2017. Creditworthy: A History of Consumer Surveillance and Financial Identity in America . Columbia University Press. http://www.jstor.org/stable/10.7312/laue16808
- Benjamin Laufer. 2020. Feedback Effects in Repeat-Use Criminal Risk Assessments. arXiv:2011.14075 [cs.CY]
- Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . 4483-4499.
- Adam Lavertu and Russ B Altman. 2019. RedMed: Extending drug lexicons for social media applications. Journal of biomedical informatics 99 (2019), 103307.
- Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Sebastian Ruder, Dani Yogatama, et al. 2021. Pitfalls of Static Language Modelling. arXiv preprint arXiv:2102.01951 (2021).
- Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Association for Computational Linguistics, Online, 2627-2636. https://doi.org/10.18653/v1/2021.naacl-main.208
- Tosca Lechner, Shai Ben-David, Sushant Agarwal, and Nivasini Ananthakrishnan. 2021. Impossibility results for fair representations. ArXiv abs/2107.03483 (2021).
- Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep Learning. Nature 521, 7553 (2015).
- Yann LeCun, John S Denker, and Sara A Solla. 1990. Optimal Brain Damage. In Advances in Neural Information Processing Systems . 598-605.
- Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020b. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 4 (2020), 1234-1240.
- Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo. 2020a. Predicting what you already know helps: Provable self-supervised learning. arXiv preprint arXiv:2008.01064 (2020).
- Jieh-Sheng Lee and Jieh Hsiang. 2019. Patentbert: Patent classification with fine-tuning a pre-trained bert model. arXiv preprint arXiv:1906.02124 (2019).
- Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021b. Deduplicating Training Data Makes Language Models Better. arXiv:2107.06499 [cs.CL]
- Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. Conference on Human Factors in Computing Systems (CHI) (May 2022). https://doi.org/10. 1145/3491102.3502030
- Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, and Ariel D. Procaccia. 2019. WeBuildAI: Participatory framework for algorithmic governance. CSCW (2019).
- Nayeon Lee, Yejin Bang, Andrea Madotto, Madian Khabsa, and Pascale Fung. 2021a. Towards Few-shot Fact-Checking via Perplexity. In NAACL .
- Yong Jae Lee, Joydeep Ghosh, and K. Grauman. 2012. Discovering important people and objects for egocentric video summarization. 2012 IEEE Conference on Computer Vision and Pattern Recognition (2012), 1346-1353.
- James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. 2021. FNet: Mixing Tokens with Fourier Transforms. arXiv preprint arXiv:2105.03824 (2021).
- Norman Lefstein and Robert L Spagenberg. 2009. Justice Denied: America's Continuing Neglect of Our Constitutional Right to Counsel . Technical Report. National Right to Counsel Committee, The Constitution Project, National Legal Aid & Defender Association.
- Legal Services Corporation. 2017. The Justice Gap: Measuring the Unmet Civil Legal Needs of Low-income Americans . Technical Report. Prepared by NORC at the University of Chicago for Legal Services Corporation, Washington, DC.
- J. Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and S. Legg. 2018. Scalable agent alignment via reward modeling: a research direction. ArXiv abs/1811.07871 (2018).
- Spyretta Leivaditi, Julien Rossi, and Evangelos Kanoulas. 2020. A Benchmark for Lease Contract Review. arXiv preprint arXiv:2010.10386 (2020).
- Mark A Lemley and Bryan Casey. 2019. Remedies for robots. The University of Chicago Law Review 86, 5 (2019), 1311-1396. Mark A Lemley and Bryan Casey. 2020. Fair Learning. Tex. L. Rev. 99 (2020), 743.
- Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv preprint arXiv:2006.16668 (2020).
- Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In International Conference on Learning Representations .
- Lawrence Lessig. 2000. Code Is Law. https://www.harvardmagazine.com/2000/01/code-is-law-html.
- Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv:2104.08691 [cs.CL]
- Amanda Levendowski. 2018. How copyright law can fix artificial intelligence's implicit bias problem. Wash. L. Rev. 93 (2018), 579.
- Sergey Levine, P. Pastor, A. Krizhevsky, and Deirdre Quillen. 2018. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research 37 (2018), 421 - 436.
- Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. 2020. Limits to depth efficiencies of self-attention. arXiv e-prints (2020), arXiv-2006.
- Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition 106, 3 (2008), 1126-1177. https://doi.org/10.1016/ j.cognition.2007.05.006
- Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Association for Computational Linguistics (ACL) .
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 9459-9474. https://proceedings.neurips.cc/ paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf
- Irene Li, Michihiro Yasunaga, Muhammed Yavuz Nuzumlalı, Cesar Caraballo, Shiwani Mahajan, Harlan Krumholz, and Dragomir Radev. 2019. A neural topic-attention model for medical term abbreviation disambiguation. Machine Learning for Health (ML4H) (2019).
- Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2015. Visualizing and understanding neural models in nlp. arXiv preprint arXiv:1506.01066 (2015).
- Jialu Li, Esin Durmus, and Claire Cardie. 2020a. Exploring the Role of Argument Structure in Online Debate Persuasion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Online, 8905-8912. https://doi.org/10.18653/v1/2020.emnlp-main.716
- Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. 2020e. PyTorch Distributed: Experiences on Accelerating Data Parallel Training. arXiv preprint arXiv:2006.15704 (2020).
- Wenda Li, Lei Yu, Yuhuai Wu, and Lawrence C. Paulson. 2021b. IsarStep: a Benchmark for High-level Mathematical Reasoning. In International Conference on Learning Representations . https://openreview.net/forum?id=Pzj6fzU6wkj
- Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2022. Large Language Models Can Be Strong Differentially Private Learners. In International Conference on Learning Representations . https://openreview.net/forum?id= bVuP3ltATMz
- Xuezixiang Li, Qu Yu, and Heng Yin. 2021a. PalmTree: Learning an Assembly Language Model for Instruction Embedding. CoRR abs/2103.03809 (2021). arXiv:2103.03809 https://arxiv.org/abs/2103.03809
- Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv preprint arXiv:2101.00190 (2021).
- Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel L.K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, and Antonio Torralba. 2020b. Visual Grounding of Learned Physical Models. In ICML .
- Yikuan Li, Shishir Rao, Jose Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi-Khorshidi. 2020c. BEHRT: transformer for electronic health records. Scientific reports 10, 1 (2020), 1-12.
- Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E Gonzalez. 2020d. Train Large, then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers. arXiv preprint arXiv:2002.11794 (2020).
- Jindřich Libovick` y, Rudolf Rosa, and Alexander Fraser. 2019. How language-neutral is multilingual BERT? arXiv preprint arXiv:1911.03310 (2019).
- Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021. Jurassic-1: Technical Details and Evaluation . White Paper. AI21 Labs.
- Chu-Cheng Lin, Aaron Jaech, Xin Li, Matt Gormley, and Jason Eisner. 2021. Limitations of Autoregressive Models and Their Alternatives. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) . Online, 5147-5173. http://cs.jhu.edu/~jason/papers/#lin-et-al2021-naacl
- Ro-Ting Lin, David C Christiani, Ichiro Kawachi, Ta-Chien Chan, Po-Huang Chiang, and Chang-Chuan Chan. 2016. Increased risk of respiratory mortality associated with the high-tech manufacturing industry: A 26-Year study. International journal of environmental research and public health 13, 6 (2016), 557.
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision . Springer, 740-755.
- Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. 2021. Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy 23, 1 (2021), 18.
- Linda L. Lindsey. 2015. The sociology of Gender Theoretical Perspectives and Feminist Frameworks . Routledge. https: //www.routledge.com/Gender-Sociological-Perspectives/Lindsey/p/book/9781138103696
- Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočisk` y, Andrew Senior, Fumin Wang, and Phil Blunsom. 2016. Latent predictor networks for code generation. arXiv preprint arXiv:1603.06744 (2016).
- Tal Linzen. 2020. How Can We Accelerate Progress Towards Human-like Linguistic Generalization?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 5210-5217. https://doi.org/10.18653/v1/2020.acl-main.465
- Tal Linzen and Marco Baroni. 2021. Syntactic structure from deep learning. Annual Review of Linguistics 7 (2021), 195-212.
- Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics (TACL) 4 (2016).
- Marco Lippi, Przemysław Pałka, Giuseppe Contissa, Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor, and Paolo Torroni. 2019. CLAUDETTE: an automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law 27, 2 (2019), 117-139.
- Zachary C. Lipton. 2018. The mythos of model interpretability. Commun. ACM 61, 10 (Sept. 2018), 36-43. https: //doi.org/10.1145/3233231
- Zachary C. Lipton and Jacob Steinhardt. 2019. Troubling Trends in Machine Learning Scholarship: Some ML Papers Suffer from Flaws That Could Mislead the Public and Stymie Future Research. Queue 17, 1 (Feb. 2019), 45-77. https: //doi.org/10.1145/3317287.3328534
- Andy T. Liu, Shuwen Yang, Po-Han Chi, Po-Chun Hsu, and Hung yi Lee. 2020d. Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2020), 6419-6423.
- Fenglin Liu, Shen Ge, and Xian Wu. 2021a. Competence-based Multimodal Curriculum Learning for Medical Report Generation. In ACL .
- Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021d. What Makes Good In-Context Examples for GPT-3?. In ArXiv .
- Nelson F. Liu, Tony Lee, Robin Jia, and Percy Liang. 2021b. Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches. ArXiv abs/2102.01065 (2021). https: //arxiv.org/abs/2102.01065
- Ruishan Liu, Shemra Rizzo, Samuel Whipple, Navdeep Pal, Arturo Lopez Pineda, Michael Lu, Brandon Arnieri, Ying Lu, William Capra, Ryan Copping, et al. 2021c. Evaluating eligibility criteria of oncology trials using real-world data and AI. Nature 592, 7855 (2021), 629-633.
- Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and P. Wang. 2020e. K-BERT: Enabling Language Representation with Knowledge Graph. ArXiv abs/1909.07606 (2020).
- Xuebo Liu, Longyue Wang, Derek F Wong, Liang Ding, Lidia S Chao, and Zhaopeng Tu. 2020c. Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning. arXiv preprint arXiv:2012.14768 (2020).
- Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021e. GPT Understands, Too. CoRR abs/2103.10385 (2021). arXiv:2103.10385 https://arxiv.org/abs/2103.10385
- Ye Liu, Shaika Chowdhury, Chenwei Zhang, Cornelia Caragea, and Philip S Yu. 2020a. Interpretable Multi-Step Reasoning with Knowledge Extraction on Complex Healthcare Question Answering. arXiv preprint arXiv:2008.02434 (2020).
- Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020b. Multilingual Denoising Pre-training for Neural Machine Translation. Transactions of the Association for Computational Linguistics 8 (2020), 726-742. https://doi.org/10.1162/tacl\_a\_00343
- YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. 2018. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 1118-1125.
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692 (2019).
- Jane Loevinger. 1957. Objective Tests as Instruments of Psychological Theory. Psychological Reports 3, 3 (1957), 635-694. https://doi.org/10.2466/pr0.1957.3.3.635 arXiv:https://doi.org/10.2466/pr0.1957.3.3.635
- Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-Shot Entity Linking by Reading Entity Descriptions. In arXiv:1906.07348v1 .
- Helen Longino. 1990. Science as Social Knowledge: Values and Objectivity in Scientific Inquiry . Princeton University Press, Princeton.
- Liane Longpre, Esin Durmus, and Claire Cardie. 2019. Persuasion of the Undecided: Language vs. the Listener. In Proceedings of the 6th Workshop on Argument Mining . Association for Computational Linguistics, Florence, Italy, 167-176. https: //doi.org/10.18653/v1/W19-4519
- David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient Episodic Memory for Continual Learning. In NIPS .
- Kadan Lottick, Silvia Susai, Sorelle A. Friedler, and Jonathan P. Wilson. 2019. Energy Usage Reports: Environmental awareness as part of algorithmic accountability. Workshop on Tackling Climate Change with Machine Learning at NeurIPS 2019 (2019).
- David G Lowe. 1992. Robust model-based motion tracking through the integration of search and estimation. International Journal of Computer Vision 8, 2 (1992), 113-122.
- David G Lowe. 1999 1999. Object recognition from local scale-invariant features. In International Conference on Computer Vision (ICCV) Proceedings of the seventh IEEE international conference on computer vision , Vol. 2. 1150-1157.
- David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision 60, 2 (2004), 91-110.
- Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019a. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In NeurIPS .
- Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. 2021a. Pretrained Transformers as Universal Computation Engines. CoRR abs/2103.05247 (2021). arXiv:2103.05247 https://arxiv.org/abs/2103.05247
- Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2020. Gender Bias in Neural Natural Language Processing. In Logic, Language, and Security: Essays Dedicated to Andre Scedrov on the Occasion of His 65th Birthday , Vivek Nigam, Tajana Ban Kirigin, Carolyn Talcott, Joshua Guttman, Stepan Kuznetsov, Boon Thau Loo, and Mitsuhiro Okada (Eds.). Springer International Publishing, Cham, 189-202. https://doi.org/10.1007/978-3-030-62077-6\_14
- Lu Lu, Pengzhan Jin, and George Em Karniadakis. 2019b. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193 (2019).
- Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021b. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. CoRR abs/2102.04664 (2021). arXiv:2102.04664 https://arxiv.org/abs/2102.04664
- Kristian Lum and William Isaac. 2016. To predict and serve? Significance 13, 5 (2016), 14-19.
- Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 31st international conference on neural information processing systems . 4768-4777.
- Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353 (2020).
- Corey Lynch and Pierre Sermanet. 2020. Grounding Language in Play. arXiv preprint arXiv:2005.07648 (2020).
Corey Lynch and Pierre Sermanet. 2021. Language Conditioned Imitation Learning over Unstructured Data. (2021).
Kalle Lyytinen and Youngjin Yoo. 2002. Ubiquitous computing. Commun. ACM 45, 12 (2002), 63-96.
- M2M-100. 2020. M2M-100: AI Model That Translates 100 Languages Without Relying on English. https://about.fb.com/ news/2020/10/first-multilingual-machine-translation-model/.
- Edward Ma. 2019. NLP Augmentation. https://github.com/makcedward/nlpaug.
- Jianzhu Ma, Samson H Fong, Yunan Luo, Christopher J Bakkenist, John Paul Shen, Soufiane Mourragui, Lodewyk FA Wessels, Marc Hafner, Roded Sharan, Jian Peng, et al. 2021b. Few-shot learning creates predictive models of drug response that translate from high-throughput screens to individual patients. Nature Cancer 2, 2 (2021), 233-244.
- Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021a. Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking. ArXiv abs/2106.06052 (2021). https://arxiv.org/abs/2106.06052
- Ryan Mac, Caroline Haskins, Brianna Sacks, and Logan McDonald. 2021. Surveillance Nation. Buzzfeed News (9 April 2021). https://www.buzzfeednews.com/article/ryanmac/clearview-ai-local-police-facial-recognition Accessed 2021-07-18.
- J. MacGlashan, Monica Babes-Vroman, M. desJardins, M. Littman, S. Muresan, S. Squire, Stefanie Tellex, Dilip Arumugam, and Lei Yang. 2015. Grounding English Commands to Reward Functions. In Robotics: Science and Systems (RSS) .
- Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR) .
- Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, R. Doan, Xinyu Liu, J. A. Ojea, and Ken Goldberg. 2017. Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics. ArXiv abs/1703.09312 (2017).
- Ali Malik, Mike Wu, Vrinda Vasavada, Jinpeng Song, Madison Coots, John Mitchell, Noah Goodman, and Chris Piech. 2021. Generative Grading: Near Human-level Accuracy for Automated Feedback on Richly Structured Problems. In Proceedings of the 14th International Conference on Educational Data Mining .
- Vittorio Caggiano Naman Goyal Siddharth Goyal Myle Ott Benjamin Lefaudeux Vitaliy Liptchinsky Mike Rabbat Sam Sheiffer Anjali Sridhar Min Xu Mandeep Baines, Shruti Bhosale. 2021. FairScale: A General Purpose Modular PyTorch Library for High Performance and Large Scale Training. https://github.com/facebookresearch/fairscale.
- Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. 2014. Offline Policy Evaluation across Representations with Applications to Educational Games. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems (Paris, France) (AAMAS '14) . International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1077-1084.
- Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. 2019. Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity. In International Conference on Intelligent Robots and Systems (IROS) .
- Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences 117 (2020), 30046-30054.
- MarketLine. 2021. Legal Services in the United States. https://www.marketresearch.com/MarketLine-v3883/Legal-ServicesUnited-States-14193556/
- Bernard Marr. 2017. Really Big Data At Walmart: Real-Time Insights From Their 40+ Petabyte Data Cloud. https://www.forbes. com/sites/bernardmarr/2017/01/23/really-big-data-at-walmart-real-time-insights-from-their-40-petabyte-data-cloud.
- David Marr. 1982. Vision: A computational investigation into the human representation and processing of visual information . W.H. Freeman, San Francisco.
- Roberto Martin-Martin, Mihir Patel, Hamid Rezatofighi, Abhijeet Shenoi, JunYoung Gwak, Eric Frankel, Amir Sadeghian, and Silvio Savarese. 2021. JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2021).
- Nicole Martinez-Martin, Zelun Luo, Amit Kaushal, Ehsan Adeli, Albert Haque, Sara S Kelly, Sarah Wieten, Mildred K Cho, David Magnus, Li Fei-Fei, et al. 2020. Ethical issues in using ambient intelligence in health-care settings. The Lancet Digital Health (2020).
- Rebecca Marvin and Tal Linzen. 2018. Targeted Syntactic Evaluation of Language Models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Brussels, Belgium, 1192-1202. https://doi.org/10.18653/v1/D18-1151
- Toni M Massaro, Helen Norton, and Margot E Kaminski. 2016. SIRI-OUSLY 2.0: what artificial intelligence reveals about the first amendment. Minn. L. Rev. 101 (2016), 2481.
- V. Masson-Delmotte, P. Zhai, A. Pirani, S. L. Connors, C. P/'ean, S. Berger, N. Caud, Y. Chen, L. Goldfarb, M. I. Gomis, M. Huang, K. Leitzell, E. Lonnoy, J. B. R. Matthews, T. K. Maycock, T. Waterfield, O. Yelekci, R. Yu, and B. Zhou (Eds.). 2021. IPCC, 2021: Climate Change 2021: The Physical Science Basis. Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. (2021).
- Daniel Masur. 2018. Data Licensing-Tips and Tactics. Corporate Compliance Insights (2018).
- Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, et al. 2020. MLPerf Training Benchmark. In Third Conference on Machine Learning and Systems .
- Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On Measuring Social Biases in Sentence Encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Association for Computational Linguistics, Minneapolis, Minnesota, 622-628. https://doi.org/10.18653/v1/N19-1063
- JS McCarley, Rishav Chakravarti, and Avirup Sil. 2019. Structured Pruning of a BERT-Based Question Answering Model. arXiv preprint arXiv:1910.06360 (2019).
- James L McClelland and David E Rumelhart. 1981. An interactive activation model of context effects in letter perception: I. An account of basic findings. Psychological review 88, 5 (1981), 375.
- Michael McCloskey and Neal J. Cohen. 1989. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Psychology of Learning and Motivation, Vol. 24. Academic Press, 109-165. https://doi.org/10.1016/S00797421(08)60536-8
- Jamie McKenzie. 2003. Pedagogy Does Matter! The Educational Technology Journal 13, 1 (2003).
- Warwick McKibbin, Roshen Fernando, et al. 2020. The economic impact of COVID-19. Economics in the Time of COVID-19 45, 10.1162 (2020).
- H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2018. Learning differentially private recurrent language models. In International Conference on Learning Representations .
- William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A Smith. 2021. Provable Limitations of Acquiring Meaning from Ungrounded Form: What will Future Language Models Understand? arXiv preprint arXiv:2104.10809 (2021).
- Robert K Merton. 1979. The normative structure of science. The sociology of science: Theoretical and empirical investigations (1979), 267-278.
- Samuel Messick. 1987. Validity. ETS Research Report Series 1987, 2 (1987), i-208. https://onlinelibrary.wiley.com/doi/abs/10. 1002/j.2330-8516.1987.tb00244.x
- Samuel Messick. 1988. The once and future issues of validity: Assessing the meaning and consequences of measurement. ETS Research Report Series (1988). https://onlinelibrary.wiley.com/doi/abs/10.1002/j.2330-8516.1986.tb00185.x
- Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2017. Mixed precision training. arXiv preprint arXiv:1710.03740 (2017).
- Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv:1301.3781 (2013).
- Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. Proceedings of the 11th Annual Conference of the International Speech Communication Association, INTERSPEECH 2010 2, 1045-1048.
- Silvia Milano, Mariarosaria Taddeo, and Luciano Floridi. 2020. Recommender systems and their ethical challenges. AI & SOCIETY 35, 4 (2020), 957-967.
- Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-Value Memory Networks for Directly Reading Documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing . 1400-1409.
- G. A. Miller, Galanter E., and K. H. Pribram. 1960. Plans and the structure of behavior . Holt, New York.
- John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. 2021. Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization. In International Conference on Machine Learning (ICML) .
- Smitha Milli, Luca Belli, and Moritz Hardt. 2021. From Optimizing Engagement to Measuring Value. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT '21) . Association for Computing Machinery, New York, NY, USA, 714-722. https://doi.org/10.1145/3442188.3445933
- Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021. Noisy Channel Language Model Prompting for Few-Shot Text Classification. arXiv:2108.04106 [cs.CL]
- Dipendra Misra, John Langford, and Yoav Artzi. 2017b. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint arXiv:1704.08795 (2017).
- Ishan Misra, Abhinav Gupta, and Martial Hebert. 2017a. From red wine to red tomato: Composition with context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 1792-1801.
- Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2021. Fast Model Editing at Scale. In International Conference on Learning Represenations . arXiv:2110.11309 https://arxiv.org/abs/2110.11309
- Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency (Jan 2019). https://doi.org/10.1145/3287560.3287596
- Yasuhide Miura, Yuhao Zhang, Emily Bao Tsai, Curtis P Langlotz, and Dan Jurafsky. 2021. Improving factual completeness and consistency of image-to-text radiology report generation. NAACL (2021).
- Piero Molino, Yaroslav Dudin, and Sai Sumanth Miryala. 2019. Ludwig: A Type-Based Declarative Deep Learning Toolbox. arXiv preprint arXiv:1909.07930 (2019).
- Camilo Mora, Randi L Rollins, Katie Taladay, Michael B Kantar, Mason K Chock, Mio Shimada, and Erik C Franklin. 2018. Bitcoin emissions alone could push global warming above 2 C. Nature Climate Change 8, 11 (2018), 931-933.
- Hans Moravec. 1988. Mind children: The future of robot and human intelligence . Harvard University Press.
- Hesham Mostafa and Xin Wang. 2019. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning . PMLR, 4646-4655. http://proceedings.mlr. press/v97/mostafa19a.html ZSCC: 0000081 ISSN: 2640-3498.
- Hussein Mozannar and David Sontag. 2020. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning . PMLR, 7076-7087.
- Jesse Mu and Jacob Andreas. 2020. Compositional explanations of neurons. arXiv preprint arXiv:2006.14032 (2020).
- Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liang Luo, et al. 2021. High-Performance, Distributed Training of Large-Scale Deep Learning Recommendation Models. arXiv preprint arXiv:2104.05158 (2021).
- Brad A. Myers, Scott E. Hudson, and Randy Pausch. 2000. Past, Present and Future of User Interface Software Tools. In ACM Transactions on Computer Human Interaction . ACM.
- Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of ACL 2021 .
- Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. 2020. Understanding the failure modes of out-ofdistribution generalization. arXiv preprint arXiv:2010.15775 (2020).
- Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. 2021. Attention Bottlenecks for Multimodal Fusion. arXiv preprint arXiv:2107.00135 (2021).
- Ashvin Nair, Vitchyr H. Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. 2018. Visual Reinforcement Learning with Imagined Goals. In NeurIPS .
- V. Nair and G. E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning (ICML) . 807-814.
- Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. 2019. Deep double descent: Where bigger models and more data hurt. arXiv preprint arXiv:1912.02292 (2019).
- Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Online, 1953-1967. https://doi.org/10.18653/v1/2020. emnlp-main.154
- Arvind Narayanan, Arunesh Mathur, Marshini Chetty, and Mihir Kshirsagar. 2020. Dark Patterns: Past, Present, and Future. Commun. ACM 63, 9 (Aug. 2020), 42-47. https://doi.org/10.1145/3397884
- Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized Pipeline Parallelism for DNN Training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles . 1-15.
- Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. 2021a. Memory-Efficient Pipeline-Parallel DNN Training. In International Conference on Machine Learning . PMLR, 7937-7947.
- Deepak Narayanan, Keshav Santhanam, Amar Phanishayee, and Matei Zaharia. 2018. Accelerating Deep Learning Workloads through Efficient Multi-Model Execution. In NeurIPS Workshop on Systems for Machine Learning . 20.
- Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021b. Efficient Large-Scale Language Model Training on GPU Clusters. arXiv preprint arXiv:2104.04473 (2021).
- Jennifer C. Nash. 2008. Re-Thinking Intersectionality. Feminist Review 89, 1 (June 2008), 1-15. https://doi.org/10.1057/fr.2008.4
- Milad Nasr, Reza Shokri, and Amir Houmansadr. 2018. Machine Learning with Membership Privacy Using Adversarial Regularization. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (Toronto, Canada) (CCS '18) . Association for Computing Machinery, New York, NY, USA, 634-646. https://doi.org/10.1145/3243734. 3243855
- Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi E Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Hassan Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, et al. 2020. Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages. In EMNLP(Findings) .
- Andrew Y. Ng and Stuart Russell. 2000. Algorithms for Inverse Reinforcement Learning. In International Conference on Machine Learning .
- Elhadji Mamadou Nguer, Alla Lo, Cheikh M Bamba Dione, Sileye O Ba, and Moussa Lo. 2020. SENCORPUS: A French-Wolof Parallel Corpus. In Proceedings of the 12th Language Resources and Evaluation Conference . 2803-2811.
- Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. 2016. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems 29 (2016), 3387-3395.
- Yizhao Ni, Stephanie Kennebeck, Judith W Dexheimer, Constance M McAneney, Huaxiu Tang, Todd Lingren, Qi Li, Haijun Zhai, and Imre Solti. 2015. Automated clinical trial eligibility prescreening: increasing the efficiency of patient identification for clinical trials in the emergency department. Journal of the American Medical Informatics Association 22,
- 1 (2015), 166-178.
- Allen Nie, Ashley Zehnder, Rodney L Page, Yuhui Zhang, Arturo Lopez Pineda, Manuel A Rivas, Carlos D Bustamante, and James Zou. 2018. DeepTag: inferring diagnoses from veterinary clinical notes. NPJ digital medicine 1, 1 (2018), 1-8.
- Mathias Wullum Nielsen, Sharla Alegria, Love Börjeson, Henry Etzkowitz, Holly J. Falk-Krzesinski, Aparna Joshi, Erin Leahey, Laurel Smith-Doerr, Anita Williams Woolley, and Londa Schiebinger. 2017. Opinion: Gender diversity leads to better science. Proceedings of the National Academy of Sciences 114, 8 (Feb. 2017), 1740-1742. https://doi.org/10.1073/ pnas.1700616114
- Helen Nissenbaum. 2004. Privacy as contextual integrity. Wash. L. Rev. 79 (2004), 119.
- Helen Nissenbaum. 2009. Privacy in Context: Technology, Policy, and the Integrity of Social Life . Stanford University Press.
- Malvina Nissim, Rik van Noord, and Rob van der Goot. 2020. Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor. Computational Linguistics 46, 2 (June 2020), 487-497. https://doi.org/10.1162/coli\_a\_00379
- Safiya Umoja Noble. 2018. Algorithms of Oppression . New York University Press.
- Shimon Y Nof. 1999. Handbook of industrial robotics . John Wiley & Sons.
- Sebastian Nordhoff and Harald Hammarström. 2011. Glottolog/Langdoc: Defining dialects, languages, and language families as collections of resources. In First International Workshop on Linked Science 2011-In conjunction with the International Semantic Web Conference (ISWC 2011) .
- Debora Nozza, Federico Bianchi, and Dirk Hovy. 2021. HONEST: Measuring Hurtful Sentence Completion in Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Association for Computational Linguistics, Online, 2398-2406. https: //doi.org/10.18653/v1/2021.naacl-main.191
- Martha Craven Nussbaum. 2010. Not for profit: Why democracy needs the humanities . Princeton University Press.
- NVIDIA. 2021. NVIDIA Collective Communication Library (NCCL). https://developer.nvidia.com/nccl.
- NVIDIA and Microsoft. 2021. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the World's Largest and Most Powerful Generative Language Model. https://developer.nvidia.com/blog/using-deepspeed-and-megatron-totrain-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/.
- Julian Nyarko and Sarath Sanga. 2020. A Statistical Test for Legal Interpretation: Theory and Applications. Available at SSRN 3737292 (2020).
- Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. 2019. Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging. arXiv e-prints, art. arXiv preprint arXiv:1909.12475 (2019).
- Douglas W Oard, Fabrizio Sebastiani, and Jyothi K Vinjumur. 2018. Jointly minimizing the expected costs of review for responsiveness and privilege in E-discovery. ACM Transactions on Information Systems (TOIS) 37, 1 (2018), 1-35.
- Jonathan A. Obar and Anne Oeldorf-Hirsch. 2020. The biggest lie on the Internet: Ignoring the privacy policies and terms of service policies of social networking services. Information, Communication & Society 23, 1 (2020), 128-147. https://doi.org/10.1080/1369118X.2018.1486870
- Cailin O'Connor, Liam Kofi Bright, and Justin P. Bruner. 2019. The Emergence of Intersectional Disadvantage. Social Epistemology 33, 1 (Jan. 2019), 23-41. https://doi.org/10.1080/02691728.2018.1555870
- U.S. Copyright Office. 2021. More Information on Fair Use. https://www.copyright.gov/fair-use/more-info.html.
- Paul Ohm. 2014. Changing the Rules: General Principles for Data Use and Analysis . Cambridge University Press, 96 - 111.
- Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill 5, 3 (2020), e00024-001.
- Thiago Dias Oliva, Dennys Marcelo Antonialli, and Alessandra Gomes. 2021. Fighting hate speech, silencing drag queens? Artificial intelligence in content moderation and risks to LGBTQ voices online. Sexuality & Culture 25, 2 (2021), 700-732.
- S. Omohundro. 2008. The Basic AI Drives. In AGI .
- Cathy O'Neil. 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy . Crown Publishing Group, USA.
- OpenAI, I. Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, N. Tezak, Jerry Tworek, P. Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. 2019. Solving Rubik's Cube with a Robot Hand. ArXiv abs/1910.07113 (2019).
- Yonatan Oren, Shiori Sagawa, Tatsunori Hashimoto, and Percy Liang. 2019. Distributionally Robust Language Modeling. In Empirical Methods in Natural Language Processing (EMNLP) .
- Laurel Orr, Megan Leszczynski, Simran Arora, Sen Wu, Neel Guha, Xiao Ling, and Chris Ré. 2020. Bootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation. In Arxiv .
- Malte Ostendorff, Elliott Ash, Terry Ruas, Bela Gipp, Julian Moreno-Schneider, and Georg Rehm. 2021. Evaluating Document Representations for Content-based Legal Literature Recommendations. arXiv preprint arXiv:2104.13841 (2021).
- David Ouyang, Bryan He, Amirata Ghorbani, Neal Yuan, Joseph Ebinger, Curtis P Langlotz, Paul A Heidenreich, Robert A Harrington, David H Liang, Euan A Ashley, et al. 2020. Video-based AI for beat-to-beat assessment of cardiac function.
Nature 580, 7802 (2020), 252-256.
- Isabel Papadimitriou, Ethan A Chi, Richard Futrell, and Kyle Mahowald. 2021. Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume . 2522-2532.
- Isabel Papadimitriou and Dan Jurafsky. 2020. Learning music helps you read: Using transfer to study linguistic structure in language models. arXiv preprint arXiv:2004.14601 (2020).
- Denis Paperno, German Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Association for Computational Linguistics (ACL) .
- Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z.Berkay Celik, and Ananthram Swami. 2017. Practical Black-Box Attacks against Deep Learning Systems using Adversarial Examples. In Proceedings of the ACM Asia Conference on Computer and Communications Security .
- Titouan Parcollet and Mirco Ravanelli. 2021. The Energy and Carbon Footprint of Training End-to-End Speech Recognizers. (2021).
- C.L. Paris, W.R. Swartout, and W.C. Mann. 2013. Natural Language Generation in Artificial Intelligence and Computational Linguistics . Springer US. https://books.google.gr/books?id=4vbiBwAAQBAJ
- German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural Networks 113 (2019), 54-71. https://doi.org/10.1016/j.neunet.2019.01.012
- Joonsuk Park, Sally Klingel, Claire Cardie, Mary Newhart, Cynthia Farina, and Joan-Josep Vallbé. 2012. Facilitative moderation for online participation in eRulemaking. In Proceedings of the 13th Annual International Conference on Digital Government Research . 173-182.
- Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing Gender Bias in Abusive Language Detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Brussels, Belgium, 2799-2804. https://doi.org/10.18653/v1/D18-1302
- Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019. GauGAN: semantic image synthesis with spatially adaptive normalization. In ACM SIGGRAPH 2019 Real-Time Live! 1-1.
- Andrew Parker. 2003. In the blink of an eye: how vision sparked the big bang of evolution. (2003).
- Samir Passi and Solon Barocas. 2019. Problem Formulation and Fairness. In Proceedings of the Conference on Fairness, Accountability, and Transparency . ACM. https://doi.org/10.1145/3287560.3287567
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems (NeurIPS) .
- Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. 2021. StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery. arXiv e-prints (2021), arXiv-2103.
- Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. 2017. Curiosity-Driven Exploration by Self-Supervised Prediction. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017), 488-489.
- Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition . 2536-2544.
- David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350 (2021).
- Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily L. Denton, and Alex Hanna. 2020. Data and its (dis)contents: A survey of dataset development and use in machine learning research. ArXiv abs/2012.05345 (2020). https://arxiv.org/abs/2012.05345
- Judea Pearl. 2000. Causality: Models, Reasoning and Inference . Vol. 29. Springer.
- Dinglan Peng, Shuxin Zheng, Yatao Li, Guolin Ke, Di He, and Tie-Yan Liu. 2021. How could Neural Networks understand Programs? CoRR abs/2105.04297 (2021). arXiv:2105.04297 https://arxiv.org/abs/2105.04297
- Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. 2019. Moment Matching for Multi-Source Domain Adaptation. In International Conference on Computer Vision (ICCV) .
- Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Edward Lee, Jie Tan, and Sergey Levine. 2020. Learning Agile Robotic Locomotion Skills by Imitating Animals. In Robotics: Science and Systems . https://doi.org/10.15607/RSS.2020.XVI. 064
- Andrew M. Penner and Aliya Saperstein. 2008. How social status shapes race. Proceedings of the National Academy of Sciences 105, 50 (2008), 19628-19630. https://doi.org/10.1073/pnas.0805762105 arXiv:https://www.pnas.org/content/105/50/19628.full.pdf
- Andrew M. Penner and Aliya Saperstein. 2013. Engendering Racial Perceptions: An Intersectional Analysis of How Social Status Shapes Race. Gender & Society 27, 3 (2013), 319-344. https://doi.org/10.1177/0891243213480262
arXiv:https://doi.org/10.1177/0891243213480262
- Andrew M. Penner and Aliya Saperstein. 2015. Disentangling the effects of racial self-identification and classification by others: The case of arrest. Demography 52, 3 (2015), 1017-1024. https://link.springer.com/article/10.1007/s13524-0150394-1
- Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global Vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP) . 1532-1543.
- Bethany Percha. 2021. Modern Clinical Text Mining: A Guide and Review. Annual Review of Biomedical Data Science 4 (2021).
- Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True Few-Shot Learning with Language Models. arXiv preprint arXiv:2105.11447 (2021).
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL) .
- Matthew E. Peters, Mark Neumann, IV RobertLLogan, Roy Schwartz, V. Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge Enhanced Contextual Word Representations. In EMNLP/IJCNLP .
- Fabio Petroni, Tim Rocktäschel, Patrick Lewis, A. Bakhtin, Yuxiang Wu, Alexander H. Miller, and S. Riedel. 2019. Language Models as Knowledge Bases?. In EMNLP .
- Chris Piech and Lisa Einstein. 2020. A Vision of AI for Joyful Education. Scientific American (Feb 2020). https://blogs. scientificamerican.com/observations/a-vision-of-ai-for-joyful-education/
- Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. InformationTheoretic Probing for Linguistic Structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 4609-4622. https://www.aclweb.org/anthology/2020.aclmain.420
- Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché Buc, Emily Fox, and Hugo Larochelle. 2020. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). arXiv:2003.12206 [cs.LG]
- Lerrel Pinto and Abhinav Gupta. 2016. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 3406-3413.
- Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How Multilingual is Multilingual BERT?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . 4996-5001.
- Nina Poerner, Ulli Waltinger, and Hinrich Schutze. 2020. E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT. arXiv:1911.03681v2 (2020).
- Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis Only Baselines in Natural Language Inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics . Association for Computational Linguistics, New Orleans, Louisiana, 180-191. https://doi.org/10.18653/v1/S182023
- Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018. Model Compression via Distillation and Quantization. arXiv preprint arXiv:1802.05668 (2018).
- Stanislas Polu and Ilya Sutskever. 2020. Generative Language Modeling for Automated Theorem Proving. CoRR abs/2009.03393 (2020). arXiv:2009.03393 https://arxiv.org/abs/2009.03393
- Edoardo Maria Ponti, Helen O'Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing. Computational Linguistics 45, 3 (09 2019), 559-601. https://doi.org/10.1162/coli\_a\_00357 arXiv:https://direct.mit.edu/coli/article-pdf/45/3/559/1847397/coli\_a\_00357.pdf
- Ryan Poplin, Avinash V Varadarajan, Katy Blumer, Yun Liu, Michael V McConnell, Greg S Corrado, Lily Peng, and Dale R Webster. 2018. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nature Biomedical Engineering 2, 3 (2018), 158-164.
- Vinodkumar Prabhakaran and Jr. Donald Martin. 2020. Participatory Machine Learning Using Community-Based System Dynamics. Health Hum Rights (2020).
- Grusha Prasad, Marten van Schijndel, and Tal Linzen. 2019. Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Models. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) . Association for Computational Linguistics, Hong Kong, China, 66-76. https://doi.org/10.18653/v1/K191007
- Doina Precup, R. Sutton, and Satinder Singh. 2000. Eligibility Traces for Off-Policy Policy Evaluation. In ICML .
- Gil Press. 2021. Andrew Ng Launches A Campaign For Data-Centric AI. https://www.forbes.com/sites/gilpress/2021/06/16/ andrew-ng-launches-a-campaign-for-data-centric-ai/?sh=44865f6a74f5.
- George L Priest and Benjamin Klein. 1984. The selection of disputes for litigation. The Journal of Legal Studies 13, 1 (1984), 1-55.
PyTorch. 2021. PyTorch JIT. https://pytorch.org/docs/stable/jit.html.
- Guanghui Qin and Jason Eisner. 2021. Learning How To Ask: Querying LMs with Mixtures of Soft Prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) . Online, 5203-5212. http://cs.jhu.edu/~jason/papers/#qin-eisner-2021
- Marc Queudot, Éric Charton, and Marie-Jean Meurs. 2020. Improving Access to Justice with Legal Chatbots. Stats 3, 3 (2020), 356-375.
- Joaquin Quiñonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. 2009. When Training and Test Sets Are Different: Characterizing Learning Transfer. In Dataset Shift in Machine Learning . 3-28.
- Markus N. Rabe, Dennis Lee, Kshitij Bansal, and Christian Szegedy. 2021. Mathematical reasoning via self-supervised skip-tree training. ICLR (2021). https://openreview.net/forum?id=YmqAnY0CMEy
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021).
- Alec Radford and Karthik Narasimhan. 2018. Improving Language Understanding by Generative Pre-Training.
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training . Technical Report. OpenAI.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019).
- Kira Radinsky. 2015. Data monopolists like Google are threatening the economy. Harvard Business Review 2 (2015).
- Evani Radiya-Dixit and Florian Tramèr. 2021. Data Poisoning Won't Save You From Facial Recognition. arXiv preprint arXiv:2106.14851 (2021).
- Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv preprint arXiv:2112.11446 (2021).
- Colin Raffel. 2021. A Call to Build Models Like We Build Open-Source Software. https://colinraffel.com/blog/a-call-to-buildmodels-like-we-build-open-source-software.html.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019).
- Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. 2017. On the expressive power of deep neural networks. In international conference on machine learning . PMLR, 2847-2854.
- Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. 2019. Transfusion: Understanding Transfer Learning for Medical Imaging. In Advances in Neural Information Processing Systems , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2019/ file/eb1e78328c46506b46a4ac4a1e378b91-Paper.pdf
- Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory Optimizations toward Training Trillion Parameter Models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis . IEEE, 1-16.
- Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning. arXiv:2104.07857 [cs.DC]
- Inioluwa Deborah Raji and Joy Buolamwini. 2019. Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (Honolulu, HI, USA) (AIES '19) . Association for Computing Machinery, New York, NY, USA, 429-435. https://doi.org/10.1145/3306618.3314244
- Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the AI accountability gap. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency . ACM. https://doi.org/10.1145/3351095.3372873
- Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, et al. 2018. Scalable and accurate deep learning with electronic health records. NPJ Digital Medicine 1, 1 (2018), 1-10.
- Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don't Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) . 784-789.
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Empirical Methods in Natural Language Processing (EMNLP) .
- Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. arXiv:2102.12092 [cs.CV]
- Bharath Ramsundar, Steven M. Kearnes, Patrick Riley, Dale Webster, David E. Konerding, and Vijay S. Pande. 2015. Massively Multitask Networks for Drug Discovery. CoRR abs/1502.02072 (2015). arXiv:1502.02072 http://arxiv.org/abs/1502.02072
- Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. DeepSpeed: System Optimizations Enable Training Deep Learning Models with over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 3505-3506.
- Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. 2021. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine 4, 1 (2021), 1-13.
- R. Ratcliff. 1990. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review 97 2 (1990), 285-308.
- Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid Training Data Creation with Weak Supervision. Proceedings of the VLDB Endowment (PVLDB) (2017).
- Gerald K Ray and Jeffrey S Lubbers. 2014. A government success story: How data analysis by the Social Security Appeals Council (with a push from the Administrative Conference of the United States) is transforming social security disability adjudication. Geo. Wash. L. Rev. 83 (2014), 1575.
- Christopher Ré, Feng Niu, Pallavi Gudipati, and Charles Srisuwananukorn. 2019. Overton: A data system for monitoring and improving machine-learned products. arXiv preprint arXiv:1909.05372 (2019).
- Richard M Re and Alicia Solow-Niederman. 2019. Developing artificially intelligent justice. Stan. Tech. L. Rev. 22 (2019), 242.
- Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do ImageNet Classifiers Generalize to ImageNet?. In International Conference on Machine Learning (ICML) .
- Colorado J. Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean Metzger, Kurt Keutzer, and Trevor Darrell. 2021. Self-Supervised Pretraining Improves Self-Supervised Pretraining. arXiv:2103.12718 [cs.CV]
- Rob Reich, Mehran Sahami, and Jeremy M. Weinstein. 2021. System Error: Where Big Tech Went Wrong and How We Can Reboot . Harper. https://books.google.com/books?id=mU0QEAAAQBAJ
- Theodoros Rekatsinas, Xu Chu, Ihab F. Ilyas, and Christopher Ré. 2017a. Holoclean: Holistic data repairs with probabilistic inference. Proceedings of the VLDB Endowment (PVLDB) (2017).
- Theodoros Rekatsinas, Manas Joglekar, Hector Garcia-Molina, Aditya Parameswaran, and Christopher Ré. 2017b. Slimfast: Guaranteed results for data fusion and source reliability. In Proceedings of the 2017 ACM International Conference on Management of Data . 1399-1414.
- Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, and Bo Dai. 2021. Combiner: Full Attention Transformer with Sparse Computation Cost. arXiv preprint arXiv:2107.05768 (2021).
- Hongyu Ren, Weihua Hu, and Jure Leskovec. 2020. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. In International Conference on Learning Representations (ICLR) .
- Hongyu Ren and Jure Leskovec. 2020. Beta embeddings for multi-hop logical reasoning in knowledge graphs. In NeurIPS .
- Adithya Renduchintala, Denise Diaz, Kenneth Heafield, Xian Li, and Mona Diab. 2021. Gender bias amplification during Speed-Quality optimization in Neural Machine Translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) . Association for Computational Linguistics, Online, 99-109. https://doi.org/10.18653/v1/2021.acl-short.15
- Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In Extended Abstract, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . ACM.
- Deborah L Rhode. 2004. Access to justice . Oxford University Press.
- Deborah L Rhode. 2014. Access to justice: A roadmap for reform. Fordham Urb. LJ 41 (2014), 1227.
- Diana Rhoten and Craig Calhoun. 2011. Knowledge matters: The public mission of the research university . Columbia University Press.
- Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. arXiv preprint arXiv:2005.04118 (2020).
- Ricardo T Ribeiro, Rui Tato Marinho, and J Miguel Sanches. 2012. Classification and staging of chronic liver disease from multimodal data. IEEE Transactions on Biomedical Engineering 60, 5 (2012), 1336-1344.
- Douglas Rice, Jesse H Rhodes, and Tatishe Nteta. 2019. Racial bias in legal language. Research & Politics 6, 2 (2019), 2053168019848930.
- C. Richards, W.P. Bouman, and M.J. Barker. 2017. Genderqueer and Non-Binary Genders . Palgrave Macmillan UK. https: //books.google.com/books?id=qFJDDwAAQBAJ
- John R Rickford, Faye McNair-Knox, et al. 1994. Addressee-and topic-influenced style shift: A quantitative sociolinguistic study. Sociolinguistic perspectives on register (1994), 235-276.
- Regina Rini. 2017. Fake News and Partisan Epistemology. Kennedy Institute of Ethics Journal 27, S2 (2017), 43-64. https: //doi.org/10.1353/ken.2017.0025
- Regina Rini. 2020. Deepfakes and the Epistemic Backstop. Philosopher's Imprint 20, 24 (2020), 1-16.
- Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. 2021. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences 118, 15 (2021). https://doi.org/10.1073/pnas.2016239118 arXiv:https://www.pnas.org/content/118/15/e2016239118.full.pdf
- Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How Much Knowledge Can You Pack into the Parameters of a Language Model?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . 5418-5426.
- Phillip Rogaway. 2016. The Moral Character of Cryptographic Work. , 48 pages.
- Anna Rogers. 2020. Peer review in NLP: resource papers. https://hackingsemantics.xyz/2020/reviewing-data/
- Anna Rogers. 2021. Changing the World by Changing the Data. ArXiv abs/2105.13947 (2021). https://arxiv.org/abs/2105.13947
- Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics (TACL) 8 (2020), 842-866.
- David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, et al. 2019. Tackling climate change with machine learning. arXiv preprint arXiv:1906.05433 (2019).
- Paul M Romer. 1990. Endogenous technological change. Journal of political Economy 98, 5, Part 2 (1990), S71-S102.
- Frieda Rong. 2021. Extrapolating to Unnatural Language Processing with GPT-3's In-context Learning: The Good, the Bad, and the Mysterious. http://ai.stanford.edu/blog/in-context-learning/
- Stéphane Ross, Geoffrey Gordon, and Andrew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Artificial Intelligence and Statistics (AISTATS) .
- Edward Rosten and Tom Drummond. 2006. Machine learning for high-speed corner detection. In European conference on computer vision . Springer, 430-443.
- Daniel Rothchild, Alex Tamkin, Julie Yu, Ujval Misra, and Joseph Gonzalez. 2021. C5T5: Controllable Generation of Organic Molecules with Transformers. ArXiv abs/2108.10307 (2021).
- Baptiste Rozière, Marie-Anne Lachaux, Marc Szafraniec, and Guillaume Lample. 2021. DOBF: A Deobfuscation Pre-Training Objective for Programming Languages. CoRR abs/2102.07492 (2021). arXiv:2102.07492 https://arxiv.org/abs/2102.07492
- Sebastian Ruder and Barbara Plank. 2018. Strong Baselines for Neural Semi-Supervised Learning under Domain Shift. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Melbourne, Australia). Association for Computational Linguistics, 1044-1054. http://aclweb.org/anthology/P18-1096
- Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 5 (2019), 206-215.
- Camilo Ruiz, Marinka Zitnik, and Jure Leskovec. 2020. Identification of disease treatment mechanisms through the multiscale interactome. Nature Communications (2020).
- Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 3 (2015), 211-252.
- Stuart J. Russell and Peter Norvig. 2020. Artificial Intelligence: A Modern Approach (4th Edition) . Pearson. http://aima.cs. berkeley.edu/
- Max Ryabinin and Anton Gusev. 2020. Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts. arXiv preprint arXiv:2002.04013 (2020).
- Christopher Ré. 2021. The Road to Software 2.0 or Data-Centric AI. https://hazyresearch.stanford.edu/data-centric-ai.
- Fereshteh Sadeghi and Sergey Levine. 2017. CAD 2 RL: Real Single-Image Flight without a Single Real Image. ArXiv abs/1611.04201 (2017).
- Jenny R Saffran, Richard N Aslin, and Elissa L Newport. 1996. Statistical learning by 8-month-old infants. Science 274, 5294 (1996), 1926-1928.
- Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020a. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. In International Conference on Learning Representations (ICLR) .
- Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. 2020b. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning (ICML) .
- Debjani Saha, Candice Schumann, Duncan C. McElfresh, John P. Dickerson, Michelle L. Mazurek, and Michael Carl Tschantz. 2020. Human Comprehension of Fairness in Machine Learning. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (New York, NY, USA) (AIES '20) . Association for Computing Machinery, New York, NY, USA, 152. https://doi.org/10.1145/3375627.3375819
- Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2020. On the Effect of Dropping Layers of Pre-trained Transformer Models. arXiv preprint arXiv:2004.03844 (2020).
- Christoph Salge, C. Glackin, and D. Polani. 2013. Empowerment - an Introduction. ArXiv abs/1310.1863 (2013).
- Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021. 'Everyone wants to do the model work, not the data work': Data Cascades in High-Stakes AI. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1-15.
- Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, A Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. arXiv preprint arXiv:1910.01108 (2019).
- Gillian Sankoff. 2018. Language Change Across the Lifespan. Annual Review of Linguistics 4, 1 (2018), 297-316. https: //doi.org/10.1146/annurev-linguistics-011817-045438 arXiv:https://doi.org/10.1146/annurev-linguistics-011817-045438
- Lindsay Sanneman, Christopher Fourie, and Julie Shah. 2020. The State of Industrial Robotics: Emerging Technologies, Challenges, and Key Research Directions. https://www.therobotreport.com/wp-content/uploads/2021/01/2020-ResearchBrief-Sanneman-Fourie-Shah.pdf
- Keshav Santhanam, Siddharth Krishna, Ryota Tomioka, Andrew Fitzgibbon, and Tim Harris. 2021. DistIR: An Intermediate Representation for Optimizing Distributed Neural Networks. In Proceedings of the 1st Workshop on Machine Learning and Systems . 15-23.
- Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. 2016. Meta-Learning with Memory-Augmented Neural Networks. In ICML . 1842-1850. http://proceedings.mlr.press/v48/santoro16.html
- Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. 2020. BREEDS: Benchmarks for Subpopulation Shift. arXiv (2020).
- Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The Risk of Racial Bias in Hate Speech Detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Florence, Italy, 1668-1678. https://doi.org/10.18653/v1/P19-1163
- Aliya Saperstein and Andrew M. Penner. 2012. Racial Fluidity and Inequality in the United States. Amer. J. Sociology 118, 3 (2012), 676-727. https://doi.org/10.1086/667722 arXiv:https://doi.org/10.1086/667722
- Aliya Saperstein, Andrew M. Penner, and Ryan Light. 2013. Racial Formation in Perspective: Connecting Individuals, Institutions, and Power Relations. Annual Review of Sociology 39, 1 (2013), 359-378. https://doi.org/10.1146/annurevsoc-071312-145639 arXiv:https://doi.org/10.1146/annurev-soc-071312-145639
- N. Saunshi, S. Malladi, and S. Arora. 2020a. A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks. arXiv preprint arXiv:2010.03648 (2020).
- Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. 2020b. A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks. arXiv preprint arXiv:2010.03648 (2020).
- Jaromir Savelka, Vern R Walker, Matthias Grabmair, and Kevin D Ashley. 2017. Sentence boundary detection in adjudicatory decisions in the united states. Traitement automatique des langues 58 (2017), 21.
- Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. 2019a. Habitat: A Platform for Embodied AI Research. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . IEEE Computer Society, 9338-9346.
- Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. 2019b. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) .
- Matthew Saxton. 2017. Child Language: Acquisition and Development . Sage Publications, London.
- Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE transactions on neural networks 20, 1 (2008), 61-80.
- Tom Schaul, Dan Horgan, K. Gregor, and D. Silver. 2015. Universal Value Function Approximators. In International Conference on Machine Learning (ICML) .
- Monica Schenone, Vlado Dančík, Bridget K Wagner, and Paul A Clemons. 2013. Target identification and mechanism of action in chemical biology and drug discovery. Nature Chemical Biology 9, 4 (2013), 232-240.
- Matthew U Scherer, Allan G King, and Marko J Mrkonich. 2019. Applying Old Rules to New Tools: Employment Discrimination Law in the Age of Algorithms. SCL Rev. 71 (2019), 449.
- Timo Schick and Hinrich Schütze. 2021a. Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume . Association for Computational Linguistics, Online, 255-269. https://aclanthology.org/2021.eacl-main.20
- Timo Schick and Hinrich Schütze. 2021b. It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Association for Computational Linguistics, Online, 2339-2352. https: //doi.org/10.18653/v1/2021.naacl-main.185
- Timo Schick, Sahana Udupa, and H. Schutze. 2021. Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP. ArXiv abs/2103.00453 (2021).
- Londa Schiebinger. 2013. Machine Translation: Analyzing Gender. (2013). http://genderedinnovations.stanford.edu/casestudies/nlp.html#tabs-2
- Londa Schiebinger. 2014. Scientific research must take gender into account. Nature 507, 7490 (2014), 9.
- K. Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. 2020. Reinforcement Learning with Videos: Combining Offline Observations with Interaction. ArXiv abs/2011.06507 (2020).
- Jürgen Schmidhuber. 1987. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook . Ph.D. Dissertation. Technische Universität München.
- J. Schmidhuber. 2019. Reinforcement Learning Upside Down: Don't Predict Rewards - Just Map Them to Actions. ArXiv abs/1912.02875 (2019).
- Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. 2018. Adversarially robust generalization requires more data. In Advances in Neural Information Processing Systems (NeurIPS) . 5014-5026.
- Victor Schmidt, Kamal Goyal, Aditya Joshi, Boris Feld, Liam Conell, Nikolas Laskaris, Doug Blank, Jonathan Wilson, Sorelle Friedler, and Sasha Luccioni. 2021. CodeCarbon: Estimate and Track Carbon Emissions from Machine Learning Computing. https://github.com/mlco2/codecarbon. (2021). https://doi.org/10.5281/zenodo.4658424
- Gisbert Schneider. 2018. Automating drug discovery. Nature Reviews Drug Discovery 17, 2 (2018), 97-113.
- Joel M. Schumm. 2012. National Indigent Defense Reform: The Solution is Multifaceted . Technical Report. National Association of Criminal Defense Lawyers, American Bar Association.
- M. Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2012), 5149-5152.
- Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. 2021. You autocomplete me: Poisoning vulnerabilities in neural code completion. In 30th { USENIX } Security Symposium ( { USENIX } Security 21) .
- E. a. G. Schuur, A. D. McGuire, C. Schädel, G. Grosse, J. W. Harden, D. J. Hayes, G. Hugelius, C. D. Koven, P. Kuhry, D. M. Lawrence, S. M. Natali, D. Olefeldt, V. E. Romanovsky, K. Schaefer, M. R. Turetsky, C. C. Treat, and J. E. Vonk. 2015. Climate change and the permafrost carbon feedback. Nature 520, 7546 (April 2015), 171-179. https://doi.org/10.1038/nature14338 ZSCC: NoCitationData[s0] Bandiera\_abtest: a Cg\_type: Nature Research Journals Number: 7546 Primary\_atype: Reviews Publisher: Nature Publishing Group Subject\_term: Biogeochemistry;Climate sciences;Earth and environmental sciences Subject\_term\_id: biogeochemistry;climate-sciences;earth-and-environmental-sciences.
- Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2019. Green ai. arXiv preprint arXiv:1907.10597 (2019).
- Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, Devon Hjelm, Philip Bachman, and Aaron C. Courville. 2021. Pretraining Representations for Data-Efficient Reinforcement Learning. ArXiv abs/2106.04799 (2021).
- Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do Massively Pretrained Language Models Make Better Storytellers?. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) . Association for Computational Linguistics, Hong Kong, China, 843-861. https://doi.org/10.18653/v1/ K19-1079
- Marwin H. S. Segler, Mike Preuss, and Mark P. Waller. 2018. Planning chemical syntheses with deep neural networks and symbolic AI. Nat. 555, 7698 (2018), 604-610. https://doi.org/10.1038/nature25978
- Andrew D Selbst. 2020. Negligence and AI's human users. BUL Rev. 100 (2020), 1315.
- Andrew D. Selbst, Danah Boyd, Sorelle Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2018. Fairness and Abstraction in Sociotechnical Systems. In Proceeedings of the Conference on Fairness, Accountability, and Transparency .
Selene. 2021. Selene Supercomputer. https://www.top500.org/system/179842/.
- Ramprasaath R Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik. 2021. Casting your model: Learning to localize improves self-supervised representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11058-11067.
- Andrew W. Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Zídek, Alexander W. R. Nelson, Alex Bridgland, Hugo Penedones, Stig Petersen, Karen Simonyan, Steve Crossan, Pushmeet Kohli, David T. Jones, David Silver, Koray Kavukcuoglu, and Demis Hassabis. 2020. Improved protein structure prediction using potentials from deep learning. Nat. 577, 7792 (2020), 706-710. https://doi.org/10.1038/s41586-019-1923-7
- Rico Sennrich, B. Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. ArXiv abs/1508.07909 (2016).
- Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. 2018. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA) . IEEE, 1134-1141.
- Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David Jacobs, and Tom Goldstein. 2019. Adversarially robust transfer learning. arXiv preprint arXiv:1905.08232 (2019).
- Neal A Shah, Jessica Jue, and Tim K Mackey. 2020. Surgical data recording technology: a solution to address medical errors? Annals of surgery 271, 3 (2020), 431-433.
- Danilo Jimenez Rezend Shakir Mohamed. 2015. Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. In NIPS .
- C. Shannon. 1948. A mathematical theory of communication. Bell Syst. Tech. J. 27 (1948), 379-423.
2. Lin Shao, Toki Migimatsu, Q. Zhang, Karen Yang, and Jeannette Bohg. 2020. Concept2Robot: Learning Manipulation Concepts from Instructions and Human Demonstrations. In Robotics: Science and Systems (RSS) .
3. Pratyusha Sharma, L. Mohan, Lerrel Pinto, and A. Gupta. 2018. Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation. In Conference on Robot Learning (CORL) .
4. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. 2018. Mesh-TensorFlow: Deep Learning for Supercomputers. Advances in Neural Information Processing Systems 31 (2018), 10414-10423.
5. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 (2017).
6. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In International Conference on Machine Learning . PMLR, 4596-4604.
7. Bokui Shen, Fei Xia, Chengshu Li, Roberto Martın-Martın, Linxi Fan, Guanzhi Wang, Claudia D'Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P Tchapmi, Kent Vainio, Li Fei-Fei, and Silvio Savarese. 2021a. iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes. International Conference on Intelligent Robots and Systems (IROS) (2021).
8. Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. 2019. Nexus: A GPU Cluster Engine for Accelerating DNN-Based Video Analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles . 322-337.
9. Jia Tracy Shen, Michiharu Yamashita, Ethan Prihar, Neil Heffernan, Xintao Wu, and Dongwon Lee. 2021b. MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education. arXiv:2106.07340 [cs.CL]
10. Emily Sheng, Josh Arnold, Zhou Yu, Kai-Wei Chang, and Nanyun Peng. 2021. Revealing Persona Biases in Dialogue Systems. arXiv:2104.08728 [cs.CL]
11. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The Woman Worked as a Babysitter: On Biases in Language Generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Association for Computational Linguistics, Hong Kong, China, 3407-3412. https://doi.org/10.18653/v1/D19-1339
12. Toby Shevlane and Allan Dafoe. 2020. The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?. In Proceedings of the 2020 AAAI/ACM Conference on AI, Ethics, and Society (AIES '20) . http://arxiv.org/ abs/2001.00463
13. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Online, 4222-4235. https: //doi.org/10.18653/v1/2020.emnlp-main.346
14. Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. In Interactions . ACM.
15. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. http://arxiv.org/abs/1909.08053 cite arxiv:1909.08053.
16. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy . 3-18.
17. Connor Shorten and Taghi M Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. Journal of Big Data 6, 1 (2019), 1-48.
18. William H Shrank, Teresa L Rogstad, and Natasha Parekh. 2019. Waste in the US health care system: estimated costs and potential for savings. Jama 322, 15 (2019), 1501-1509.
19. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In Computer Vision and Pattern Recognition (CVPR) .
20. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning Important Features Through Propagating Activation Differences. In International Conference on Machine Learning (ICML) .
21. Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Anderson. 2020. Sponge examples: Energy-latency attacks on neural networks. arXiv preprint arXiv:2006.03463 (2020).
22. David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. 2016. Mastering the game of Go with deep neural networks and tree search. Nat. 529, 7587 (2016), 484-489. https://doi.org/10.1038/nature16961
23. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013).
- K Simonyan and A. Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR) .
- Audra Simpson. 2007. On Ethnographic Refusal: Indigeneity, 'Voice' Colonial Citizenship. Junctures (Dec. 2007).
- Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. 2019. End-to-End Robotic Reinforcement Learning without Reward Engineering. In Robotics: Science and Systems (RSS) .
- Satinder Singh, Andrew G Barto, and Nuttapong Chentanez. 2005. Intrinsically motivated reinforcement learning . Technical Report. MASSACHUSETTS UNIV AMHERST DEPT OF COMPUTER SCIENCE.
- Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable Neural Networks. In International Conference on Learning Representations . https://openreview.net/forum?id=HJedXaEtvS
- Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. 2019. Scene representation networks: Continuous 3d-structureaware neural scene representations. arXiv preprint arXiv:1906.01618 (2019).
- C Estelle Smith, Bowen Yu, Anjali Srivastava, Aaron Halfaker, Loren Terveen, and Haiyi Zhu. 2020. Keeping Community in the Loop: Understanding Wikipedia Stakeholder Values for Machine Learning-Based Systems. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . 1-14.
- Laura Smith, Nikita Dhawan, Marvin Zhang, P. Abbeel, and Sergey Levine. 2019. AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos. ArXiv abs/1912.04443 (2019).
- Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175 (2017).
- David So, Quoc Le, and Chen Liang. 2019. The Evolved Transformer. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97) , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 5877-5886. http://proceedings.mlr.press/v97/so19a.html
- Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. 2015. Corrigibility. In Workshops at the TwentyNinth AAAI Conference on Artificial Intelligence .
- J. Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and S. Ganguli. 2015. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ArXiv abs/1503.03585 (2015).
- Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019. Release Strategies and the Social Impacts of Language Models . Technical Report. OpenAI. http://arxiv.org/abs/1908.09203
- Irene Solaiman and Christy Dennison. 2021. Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets. arXiv preprint arXiv:2106.10328 (2021).
- Miriam Solomon. 2006. Norms of epistemic diversity. Episteme 3, 1 (2006), 23-36.
- Hamid Soltanian-Zadeh. 2019. Multimodal Analysis in Biomedicine. In Big Data in Multimodal Medical Imaging . Chapman and Hall/CRC, 193-203.
- Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. 2017. Machine Learning Models That Remember Too Much. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (Dallas, Texas, USA) (CCS '17) . Association for Computing Machinery, New York, NY, USA, 587-601. https://doi.org/10.1145/3133956.3134077
- Congzheng Song and Vitaly Shmatikov. 2019. Overlearning reveals sensitive attributes. arXiv preprint arXiv:1905.11742 (2019).
- Yang Song and S. Ermon. 2019. Generative Modeling by Estimating Gradients of the Data Distribution. ArXiv abs/1907.05600 (2019).
- Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. 2018. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research (JMLR) 19, 1 (2018), 2822-2878.
- Steven J. Spencer, Christine Logel, and Paul G. Davies. 2016. Stereotype threat. Annual Review of Psychology 67 (2016), 415-437.
- Katta Spiel, Christopher Frauenberger, Os Keyes, and Geraldine Fitzpatrick. 2019. Agency of Autistic Children in Technology Research-A Critical Literature Review. ACM Transactions on Computer-Human Interaction 26, 6 (Dec. 2019), 1-40. https://doi.org/10.1145/3344919
- Peter Spirtes, Clark N Glymour, and Richard Scheines. 2001. Causation, Prediction, and Search (2nd ed.). MIT Press.
- Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. 2014. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014).
- Megha Srivastava and Noah Goodman. 2021. Question Generation for Adaptive Education. In Association for Computational Linguistics (ACL) .
- R. Srivastava, Pranav Shyam, Filipe Wall Mutz, Wojciech Jaśkowski, and J. Schmidhuber. 2019. Training Agents using Upside-Down Reinforcement Learning. ArXiv abs/1912.02877 (2019).
- Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C Karen Liu, et al. 2021. BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments. arXiv preprint arXiv:2108.03332 (2021).
- Kate Starbird, Ahmer Arif, Tom Wilson, Katherine Van Koevering, Katya Yefimova, and Daniel Scarnecchia. 2018. Ecosystem or Echo-System? Exploring Content Sharing across Alternative Media Domains. Proceedings of the International AAAI Conference on Web and Social Media 12, 1 (Jun. 2018). https://ojs.aaai.org/index.php/ICWSM/article/view/15009
- Laura Stark. 2012. Behind Closed Doors: IRBs and the Making of Medical Research . University of Chicago Press.
- Daniel Steel, Sina Fazelpour, Kinley Gillette, Bianca Crewe, and Michael Burgess. 2018. Multiple diversity concepts and their ethical-epistemic implications. European journal for philosophy of science 8, 3 (2018), 761-780.
- Ethan Steinberg, Ken Jung, Jason A Fries, Conor K Corbin, Stephen R Pfohl, and Nigam H Shah. 2021. Language models are an effective representation learning technique for electronic health record data. Journal of Biomedical Informatics 113 (2021), 103637.
- Nicholas Stern and Joseph E Stiglitz. 2021. The social cost of carbon, risk, distribution, market failures: An alternative approach . Technical Report. National Bureau of Economic Research.
- Megan T. Stevenson and Jennifer L. Doleac. 2021. Algorithmic Risk Assessment in the Hands of Humans. SSRN (2021). https://doi.org/10.2139/ssrn.3489440
- Iris T Stewart, Christopher M Bacon, and William D Burke. 2014. The uneven distribution of environmental burdens and benefits in Silicon Valley's backyard. Applied Geography 55 (2014), 266-277.
- Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. ArXiv abs/2009.01325 (2020).
- Jonathan M Stokes, Kevin Yang, Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar Bloom-Ackermann, et al. 2020. A deep learning approach to antibiotic discovery. Cell 180, 4 (2020), 688-702.
- Michael Stonebraker and Ihab F. Ilyas. 2018. Data Integration: The Current Status and the Way Forward. IEEE Computer Society Technical Committee on Data Engineering (2018).
- Michael Stonebraker and Ariel Weisberg. 2013. The VoltDB Main Memory DBMS. IEEE Data Eng. Bull. 36, 2 (2013), 21-27.
- Marilyn Strathern. 1997. 'Improving ratings': audit in the British University system. European Review 5, 3 (1997), 305321. https://www.cambridge.org/core/journals/european-review/article/abs/improving-ratings-audit-in-the-britishuniversity-system/FC2EE640C0C44E3DB87C29FB666E9AAB
- Yolande Strengers, Lizhen Qu, Qiongkai Xu, and Jarrod Knibbe. 2020. Adhering, Steering, and Queering: Treatment of Gender in Natural Language Generation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . ACM. https://doi.org/10.1145/3313831.3376315
- Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:1906.02243 (2019).
- Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Muller. 2007. Covariate Shift Adaptation by Importance Weighted Cross Validation. Journal of Machine Learning Research (JMLR) 8 (2007), 985-1005.
- Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems . 2440-2448.
- Michael Sullivan, Josh Schellenberg, and Marshall Blundell. 2015. Updated Value of Service Reliability Estimates for Electric Utility Customers in the United States . Technical Report LBNL-6941E, 1172643. LBNL-6941E, 1172643 pages. https: //doi.org/10.2172/1172643 ZSCC: 0000086.
- Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019a. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 7464-7473.
- Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Sheng Zhao, Shuyang Cheng, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. 2020a. Scalability in Perception for Autonomous Driving: Waymo Open Dataset. arXiv:1912.04838 [cs.CV]
- Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020b. Colake: Contextualized language and knowledge embedding. arXiv preprint arXiv:2010.00309 (2020).
- Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A. Efros. 2019b. Unsupervised Domain Adaptation through Self-Supervision. arXiv:1909.11825 [cs.LG]
- Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International Conference on Machine Learning (ICML) . 3319-3328.
- Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition . 1199-1208.
- Harry Surden. 2020. The ethics of artificial intelligence in law: Basic questions. Forthcoming chapter in Oxford Handbook of Ethics of AI (2020), 19-29.
- Abhijit Suresh, Jennifer Jacobs, Vivian Lai, Chenhao Tan, Wayne Ward, James H Martin, and Tamara Sumner. 2021. Using Transformers to Provide Teachers with Personalized Feedback on their Classroom Discourse: The TalkMoves Application.
- arXiv preprint arXiv:2105.07949 (2021).
- Annamalai Suresh, R Udendhran, and S Vimal. 2020. Deep neural networks for multimodal imaging and biomedical applications . IGI Global.
- Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 1433-1443.
- Latanya Sweeney. 2013. Discrimination in Online Ad Delivery. Queue 11, 3, Article 10 (March 2013), 20 pages. https: //doi.org/10.1145/2460276.2460278
- Stephen J Swensen, Gary S Kaplan, Gregg S Meyer, Eugene C Nelson, Gordon C Hunt, David B Pryor, Jed I Weissberg, Jennifer Daley, Gary R Yates, and Mark R Chassin. 2011. Controlling healthcare costs by removing waste: what American doctors can do now. BMJ quality & safety 20, 6 (2011), 534-537.
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition . 1-9.
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR) .
- Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. 2021. Habitat 2.0: Training Home Assistants to Rearrange their Habitat. arXiv:2106.14405 [cs.LG]
- Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. 2021a. Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models. arXiv:2102.02503 [cs.CL]
- Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, and Noah Goodman. 2021b. DABS: A Domain-Agnostic Benchmark for Self-Supervised Learning. arXiv:2111.12062 [cs.LG]
- A. Tamkin, Mike Wu, and Noah D. Goodman. 2021c. Viewmaker Networks: Learning Views for Unsupervised Representation Learning. ArXiv abs/2010.07432 (2021).
- Hao Tan and Mohit Bansal. 2020. Vokenization: Improving Language Understanding via Contextualized, Visually-Grounded Supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . 2066-2080.
- Hao Hao Tan and Mohit Bansal. 2019. LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In Empirical Methods in Natural Language Processing (EMNLP) .
- Mingxing Tan and Quoc V Le. 2021. Efficientnetv2: Smaller models and faster training. arXiv preprint arXiv:2104.00298 (2021).
- Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. 2020. Measuring Robustness to Natural Distribution Shifts in Image Classification. arXiv preprint arXiv:2007.00644 (2020).
- Rachael Tatman. 2017. Gender and Dialect Bias in YouTube's Automatic Captions. In Workshop on Ethics in Natural Langauge Processing , Vol. 1. 53-59.
- Nicholas P Tatonetti, P Ye Patrick, Roxana Daneshjou, and Russ B Altman. 2012. Data-driven prediction of drug effects and interactions. Science translational medicine 4, 125 (2012), 125ra31-125ra31.
- Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020. Efficient Transformers: A Survey. arXiv preprint arXiv:2009.06732 (2020).
- Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Charformer: Fast Character Transformers via Gradient-based Subword Tokenization. arXiv preprint arXiv:2106.12672 (2021).
- Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. 2016. Alignment for Advanced Machine Learning Systems. In Ethics of Artificial Intelligence .
- Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT Rediscovers the Classical NLP Pipeline. arXiv (2019).
- The HAI Adaptive Agents Group. 2021. When Artificial Agents Lie, Defame, and Defraud, Who Is to Blame? https: //hai.stanford.edu/news/when-artificial-agents-lie-defame-and-defraud-who-blame
- Armin W Thomas, Hauke R Heekeren, Klaus-Robert Müller, and Wojciech Samek. 2019. Analyzing neuroimaging data through recurrent deep learning models. Frontiers in neuroscience 13 (2019), 1321. https://doi.org/10.3389/fnins.2019.01321
- Charles Thorpe, Martial H Hebert, Takeo Kanade, and Steven A Shafer. 1988. Vision and navigation for the Carnegie-Mellon Navlab. IEEE Transactions on Pattern Analysis and Machine Intelligence 10, 3 (1988), 362-373.
- Simon Thorpe, Denis Fize, and Catherine Marlot. 1996. Speed of processing in the human visual system. Nature 381, 6582 (June 1996), 520-522. https://doi.org/10.1038/381520a0
- Sebastian Thrun. 1998. Lifelong learning algorithms. Learning to learn (1998), 181-209.
- S. Thrun and Tom Michael Mitchell. 1995. Lifelong robot learning. Robotics Auton. Syst. 15 (1995), 25-46.
- Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. 2020a. What makes for good views for contrastive learning. arXiv preprint arXiv:2005.10243 (2020).
- Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. 2020b. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578 (2020).
- Elizabeth Chika Tippett, Charlotte Alexander, and L Karl Branting. 2021. Does Lawyering Matter? Predicting Judicial Decisions from Legal Briefs, and What That Means for Access to Justice. Texas Law Review, Forthcoming (2021).
- Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. 2021. MLP-Mixer: An all-MLP Architecture for Vision. arXiv:2105.01601 [cs.CV]
- Nenad Tomasev, Kevin R. McKee, Jackie Kay, and Shakir Mohamed. 2021. Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities. arXiv:2102.04257 (2021). https://doi.org/10.1145/3461702.3462540
- Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. 2020. Contrastive estimation reveals topic posterior information to linear models. arXiv:2003.02234 (2020).
- Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. 2021. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory . PMLR, 1179-1206.
- Florian Tramèr and Dan Boneh. 2021. Differentially Private Learning Needs Better Features (or Much More Data). In International Conference on Learning Representations .
- Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction APIs. In USENIX Security .
- Nilesh Tripuraneni, Michael I Jordan, and Chi Jin. 2020. On the theory of transfer learning: The importance of task diversity. arXiv preprint arXiv:2006.11650 (2020).
- Megan L. Truax. 2018. The Impact of Teacher Language and Growth Mindset Feedback on Writing Motivation. Literacy Research and Instruction 57, 2 (2018), 135-157. https://doi.org/10.1080/19388071.2017.1340529 arXiv:https://doi.org/10.1080/19388071.2017.1340529
- Tomer Tsaban, Julia K Varga, Orly Avraham, Ziv Ben Aharon, Alisa Khramushin, and Ora Schueler-Furman. 2021. Harnessing protein folding neural networks for peptide-protein docking. bioRxiv (2021).
- Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020. Self-supervised learning from a multi-view perspective. arXiv preprint arXiv:2006.05576 (2020).
- Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal Few-Shot Learning with Frozen Language Models. arXiv preprint arXiv:2106.13884 (2021).
- Masatoshi Tsuchiya. 2018. Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) . European Language Resources Association (ELRA), Miyazaki, Japan. https://aclanthology.org/L18-1239
- Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics 8 (2020), 621-633.
- Wenling Tu and Yujung Lee. 2009. Ineffective environmental laws in regulating electronic manufacturing pollution: Examining water pollution disputes in Taiwan. In 2009 IEEE International Symposium on Sustainable Systems and Technology . IEEE, 1-6.
- Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Association for Computational Linguistics (ACL) . 384-394.
- Alan M Turing. 1950. Computing machinery and intelligence. Mind 49 (1950), 433-460.
- Turing-NLG. 2020. Turing-NLG: A 17-Billion-Parameter Language Model by Microsoft. https://www.microsoft.com/enus/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft//.
- Barbara Tversky and Jeffrey M Zacks. 2013. Event perception. Oxford handbook of cognitive psychology (2013), 83-94.
- Jonathan Uesato, Jean-Baptiste Alayrac, Po-Sen Huang, Robert Stanforth, Alhussein Fawzi, and Pushmeet Kohli. 2019. Are Labels Required for Improving Adversarial Robustness?. In Advances in Neural Information Processing Systems (NeurIPS) .
- Shimon Ullman. 1979. The interpretation of structure from motion. Proceedings of the Royal Society of London. Series B. Biological Sciences 203, 1153 (1979), 405-426.
- United Nations General Assembly. 2015. Transforming Our World: The 2030 Agenda for Sustainable Development. https://www.refworld.org/docid/57b6e3e44.html
- Josef Urban and Jan Jakubuv. 2020. First Neural Conjecturing Datasets and Experiments. In Intelligent Computer Mathematics - 13th International Conference, CICM 2020, Bertinoro, Italy, July 26-31, 2020, Proceedings (Lecture Notes in Computer Science, Vol. 12236) , Christoph Benzmüller and Bruce R. Miller (Eds.). Springer, 315-323. https://doi.org/10.1007/978-3-030-53518-6\_24
- Mark C. Urban. 2015. Accelerating extinction risk from climate change. Science 348, 6234 (May 2015), 571-573. https: //doi.org/10.1126/science.aaa4984 ZSCC: 0000959 Publisher: American Association for the Advancement of Science Section: Report.
- Aäron van den Oord, S. Dieleman, H. Zen, K. Simonyan, Oriol Vinyals, A. Graves, Nal Kalchbrenner, A. Senior, and K. Kavukcuoglu. 2016. WaveNet: A Generative Model for Raw Audio. In SSW .
- Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation Learning with Contrastive Predictive Coding. ArXiv abs/1807.03748 (2018).
- Aäron van den Oord, Oriol Vinyals, and K. Kavukcuoglu. 2017. Neural Discrete Representation Learning. In NIPS .
- Michael van Hartskamp, Sergio Consoli, Wim Verhaegh, Milan Petkovic, and Anja van de Stolpe. 2019. Artificial Intelligence in Clinical Health Care Applications: Viewpoint. Interactive Journal of Medical Research 8, 2 (Apr 2019), e12100. https: //doi.org/10.2196/12100
- Marten van Schijndel and Tal Linzen. 2018. A Neural Model of Adaptation in Reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Brussels, Belgium, 4704-4710. https://doi.org/10.18653/v1/D18-1499
- Manasi Vartak, Harihar Subramanyam, Wei-En Lee, Srinidhi Viswanathan, Saadiyah Husnoo, Samuel Madden, and Matei Zaharia. 2016. ModelDB: a system for machine learning model management. In Proceedings of the Workshop on HumanIn-the-Loop Data Analytics . 1-3.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv preprint arXiv:1706.03762 (2017).
- Sara Veldhoen, Dieuwke Hupkes, and Willem Zuidema. 2016. Diagnostic Classifiers: Revealing how Neural Networks Process Hierarchical Structure. In Pre-Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches (CoCo @ NIPS 2016) .
- Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2017. Graph Attention Networks. arXiv e-prints (2017), arXiv-1710.
- Pat Verga, Haitian Sun, Livio Baldini Soares, and William W Cohen. 2020. Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge. arXiv preprint arXiv:2007.00849 (2020).
- Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le. 2021. Towards domain-agnostic contrastive learning. In International Conference on Machine Learning . PMLR, 10530-10541.
- Lucas Nunes Vieira, Minako O'Hagan, and Carol O'Sullivan. 2020. Understanding the societal impacts of machine translation: a critical review of the literature on medical and legal use cases. Information, Communication & Society (2020), 1-18.
- Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. 2020. Causal mediation analysis for interpreting neural NLP: The case of gender bias. arXiv preprint arXiv:2004.12265 (2020).
- Eduard Fosch Villaronga, Peter Kieseberg, and Tiffany Li. 2018. Humans forget, machines remember: Artificial intelligence and the right to be forgotten. Computer Law & Security Review 34, 2 (2018), 304-313.
- Pascal Vincent, Hugo Larochelle, Yoshua Bengio, , and Pierre-Antoine Manzagol. 2008. Extracting and Composing Robust Features with Denoising Autoencoders. In International Conference on Machine Learning (ICML) .
- Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv preprint arXiv:1912.07076 (2019).
- Rob Voigt, Nicholas P Camp, Vinodkumar Prabhakaran, William L Hamilton, Rebecca C Hetey, Camilla M Griffiths, David Jurgens, Dan Jurafsky, and Jennifer L Eberhardt. 2017. Language from police body camera footage shows racial disparities in officer respect. Proceedings of the National Academy of Sciences 114, 25 (2017), 6521-6526.
- Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. RtGender: A Corpus for Studying Differential Responses to Gender. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) . European Language Resources Association (ELRA), Miyazaki, Japan. https://aclanthology. org/L18-1445
- Elena Voita and Ivan Titov. 2020. Information-Theoretic Probing with Minimum Description Length. arXiv:2003.12298 [cs.CL] Andrew Vold and Jack G Conrad. 2021. Using Transformers to Improve Answer Retrieval for Legal Questions. (2021).
- Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science 359, 6380 (2018), 1146-1151. https://doi.org/10.1126/science.aap9559 arXiv:https://science.sciencemag.org/content/359/6380/1146.full.pdf
- Lyndsey Wajert and Gabe Rottman. 2019. Scraping public websites likely doesn't violate the Computer Fraud and Abuse Act, court holds. https://www.rcfp.org/scraping-not-violation-cfaa/.
- Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal Adversarial Triggers for Attacking and Analyzing NLP. In Empirical Methods in Natural Language Processing .
- WPatrick Walters and Regina Barzilay. 2020. Applications of deep learning in molecule generation and molecular property prediction. Accounts of Chemical Research 54, 2 (2020), 263-270.
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In Advances in Neural Information Processing Systems (NeurIPS) .
- Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019b. GLUE: A MultiTask Benchmark and Analysis Platform for Natural Language Understanding. In International Conference on Learning Representations (ICLR) .
- Ben Wang. 2021. Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX. https://github.com/kingoflolz/mesh-transformer-jax.
- Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github. com/kingoflolz/mesh-transformer-jax.
- Haojie Wang, Jidong Zhai, Mingyu Gao, Zixuan Ma, Shizhi Tang, Liyan Zheng, Yuanzhi Li, Kaiyuan Rong, Yuanyong Chen, and Zhihao Jia. 2021c. PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21) . 37-54.
- Lijun Wang, Wanli Ouyang, Xiaogang Wang, and Huchuan Lu. 2015b. Visual tracking with fully convolutional networks. In Proceedings of the IEEE international conference on computer vision . 3119-3127.
- Mingzhe Wang and Jia Deng. 2020. Learning to Prove Theorems by Learning to Generate Theorems. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/d2a27e83d429f0dcae6b937cf440aeb1-Abstract.html
- Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020c. Linformer: Self-Attention with Linear Complexity. arXiv preprint arXiv:2006.04768 (2020).
- Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning . PMLR, 9929-9939.
- Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019d. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 5310-5319.
- Wenhui Wang, Sen Yang, Xiang Zhang, and Jing Li. 2014. Drug repositioning by integrating target information through a heterogeneous network model. Bioinformatics 30, 20 (2014), 2923-2930.
- Xiaolong Wang, David Fouhey, and Abhinav Gupta. 2015a. Designing deep networks for surface normal estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition . 539-547.
- Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021a. KEPLER: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics 9 (2021), 176-194.
- Xuefeng Wang, Eric P Xing, and Daniel J Schaid. 2015c. Kernel methods for large-scale genomic data analysis. Briefings in bioinformatics 16, 2 (2015), 183-192.
- Yu Wang, Jinchao Li, Tristan Naumann, Chenyan Xiong, Hao Cheng, Robert Tinn, Cliff Wong, Naoto Usuyama, Richard Rogahn, Zhihong Shen, et al. 2021b. Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) .
- Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020a. Extending Multilingual BERT to Low-Resource Languages. arXiv:2004.13640 [cs] (Apr 2020). http://arxiv.org/abs/2004.13640
- Zihan Wang, K Karthikeyan, Stephen Mayhew, and Dan Roth. 2020b. Extending Multilingual BERT to Low-Resource Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings . 26492656.
- Zirui Wang, Zachary C Lipton, and Yulia Tsvetkov. 2020d. On Negative Interference in Multilingual Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . 4438-4450.
- Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2019c. Structured Pruning of Large Language Models. arXiv preprint arXiv:1910.04732 (2019).
- Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding Abuse: A Typology of Abusive Language Detection Subtasks. In Proceedings of the First Workshop on Abusive Language Online . Association for Computational Linguistics, Vancouver, BC, Canada, 78-84. https://doi.org/10.18653/v1/W17-3012
- K Washington, DS Browitt, K Murata, D Monroe, and T Heames. 1995. KBERT. Knowledge Based Estimation of Material Release Transients . Technical Report. Sandia National Labs., Albuquerque, NM (United States).
- Colin Wei, Sham Kakade, and Tengyu Ma. 2020a. The Implicit and Explicit Regularization Effects of Dropout. In International Conference on Machine Learning .
- Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. 2020b. Theoretical analysis of self-training with deep networks on unlabeled data. arXiv preprint arXiv:2010.03622 (2020).
- Colin Wei, Sang Michael Xie, and Tengyu Ma. 2021. Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning. arXiv:2106.09226 [cs.LG]
- Ken Weiner. 2018. Can AI Create True Art? Scientific American (2018).
- Laurel Westbrook and Aliya Saperstein. 2015. New Categories Are Not Enough: Rethinking the Measurement of Sex and Gender in Social Surveys. Gender & Society 29, 4 (2015), 534-560. https://doi.org/10.1177/0891243215584758 arXiv:https://doi.org/10.1177/0891243215584758
- Hannes Westermann, Vern R Walker, Kevin D Ashley, and Karim Benyekhlef. 2019. Using factors to predict and analyze landlord-tenant decisions to increase access to justice. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law . 133-142.
- Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv preprint arXiv:1410.3916 (2014).
- Michelle Whirl-Carrillo, Ellen M McDonagh, JM Hebert, Li Gong, K Sangkuhl, CF Thorn, Russ B Altman, and Teri E Klein. 2012. Pharmacogenomics knowledge for personalized medicine. Clinical Pharmacology & Therapeutics 92, 4 (2012), 414-417.
- Jenna Wiens, Suchi Saria, Mark Sendak, Marzyeh Ghassemi, Vincent X Liu, Finale Doshi-Velez, Kenneth Jung, Katherine Heller, David Kale, Mohammed Saeed, et al. 2019. Do no harm: a roadmap for responsible machine learning for health care. Nature medicine 25, 9 (2019), 1337-1340.
- Bernard Williams. 1973. In Utilitarianism: For and Against , J.C. Smart and Bernard Williams (Eds.). Cambridge University Press, 82-118.
- Monnica T. Williams. 2020. Psychology cannot afford to ignore the many harms caused by microaggressions. Perspectives on Psychological Science 15, 1 (2020), 38-43.
- Angelica Willis, Glenn Davis, Sherry Ruan, Lakshmi Manoharan, James Landay, and Emma Brunskill. 2019. Key Phrase Extraction for Generating Educational Question-Answer Pairs. In Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale (Chicago, IL, USA) (L@S '19) . Association for Computing Machinery, New York, NY, USA, Article 20, 10 pages. https://doi.org/10.1145/3330430.3333636
- Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. 2019. Predictive Inequity in Object Detection. (2019). https://arxiv.org/pdf/1902.11097.pdf.
- Christo Wilson, Avijit Ghosh, Shan Jiang, Alan Mislove, Lewis Baker, Janelle Szary, Kelly Trindel, and Frida Polli. 2021. Building and Auditing Fair Algorithms: A Case Study in Candidate Screening. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAccT 2021) . Virtual Event, Canada.
- Julia K. Winkler, Christine Fink, Ferdinand Toberer, Alexander Enk, Teresa Deinlein, Rainer Hofmann-Wellenhof, Luc Thomas, Aimilios Lallas, Andreas Blum, Wilhelm Stolz, and Holger A. Haenssle. 2019. Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma Recognition. JAMADermatology 155, 10 (10 2019), 1135-1141. https://doi.org/10.1001/jamadermatol.2019.1735 arXiv:https://jamanetwork.com/journals/jamadermatology/articlepdf/2740808/jamadermatology\_winkler\_2019\_oi\_190038.pdf
- Langdon Winner. 1980. Do Artifacts Have Politics? Daedalus 109, 1 (1980), 121-136. http://www.jstor.org/stable/20024652 Publisher: The MIT Press.
- L. Wittgenstein. 1953. Philosophical Investigations . Blackwell, Oxford.
- Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . 38-45.
- David H Wolpert and William G Macready. 1997. No free lunch theorems for optimization. IEEE transactions on evolutionary computation 1, 1 (1997), 67-82.
- Eric Wong and J. Zico Kolter. 2020. Learning perturbation sets for robust machine learning. arXiv (2020).
- Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. 2020. Kernel and rich regimes in overparametrized models. arXiv preprint arXiv:2002.09277 (2020).
- Beverly Park Woolf, H. Chad Lane, Vinay K. Chaudhri, and Janet L. Kolodner. 2013. AI Grand Challenges for Education. AI Magazine 34, 4 (Dec. 2013), 66-84. https://doi.org/10.1609/aimag.v34i4.2490
- Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. 2021. Robust fine-tuning of zero-shot models. arXiv preprint arXiv:2109.01903 (2021).
- Olivier J Wouters, Martin McKee, and Jeroen Luyten. 2020. Estimated research and development investment needed to bring a new medicine to market, 2009-2018. Jama 323, 9 (2020), 844-853.
- Bohan Wu, Suraj Nair, Roberto Martín-Martín, Li Fei-Fei, and Chelsea Finn. 2021d. Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction. arXiv preprint arXiv:2103.04174 (2021).
- Eric Wu, Kevin Wu, Roxana Daneshjou, David Ouyang, Daniel E Ho, and James Zou. 2021g. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nature Medicine 27, 4 (2021), 582-584.
- Kevin E Wu, Kathryn E Yost, Howard Y Chang, and James Zou. 2021h. BABEL enables cross-modality translation between multiomic profiles at single-cell resolution. Proceedings of the National Academy of Sciences 118, 15 (2021).
- Mike Wu, Chris Piech, Noah Goodman, and Chelsea Finn. 2021e. ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback. arxiv (2021).
- Shijie Wu and Mark Dredze. 2019. Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Association for Computational Linguistics, Hong Kong, China, 833-844. https://doi.org/10.18653/v1/D19-1077
- Shijie Wu and Mark Dredze. 2020. Are All Languages Created Equal in Multilingual BERT?. In Proceedings of the 5th Workshop on Representation Learning for NLP . 120-130. https://aclanthology.org/2020.repl4nlp-1.16
- Yuhuai Wu, Albert Jiang, Jimmy Ba, and Roger Grosse. 2021a. INT: An Inequality Benchmark for Evaluating Generalization in Theorem Proving. (2021). https://openreview.net/forum?id=O6LPudowNQm
- Yuhuai Wu, Markus N. Rabe, Wenda Li, Jimmy Ba, Roger B. Grosse, and Christian Szegedy. 2021f. LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning. (2021).
- Zachary Wu, Kadina E Johnston, Frances H Arnold, and Kevin K Yang. 2021b. Protein sequence design with deep generative models. Current Opinion in Chemical Biology 65 (2021), 18-27.
- Zhengxuan Wu, Nelson F Liu, and Christopher Potts. 2021c. Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models. arXiv preprint arXiv:2104.08410 (2021).
- Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. 2018. Unsupervised Feature Learning via Non-parametric Instance Discrimination. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 3733-3742.
- Alice Xiang. 2021. Reconciling legal and technical approaches to algorithmic bias. Tennessee Law Review 88, 3 (2021).
- Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. 2020. Noise or Signal: The Role of Image Backgrounds in Object Recognition. arXiv preprint arXiv:2006.09994 (2020).
- Tete Xiao, Xiaolong Wang, Alexei A. Efros, and Trevor Darrell. 2021. What Should Not Be Contrastive in Contrastive Learning. arXiv:2008.05659 [cs.CV]
- Michael Xie, Neal Jean, Marshall Burke, David Lobell, and Stefano Ermon. 2016. Transfer Learning from Deep Features for Remote Sensing and Poverty Mapping. In Association for the Advancement of Artificial Intelligence (AAAI) .
- Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. 2020. Self-training with Noisy Student improves ImageNet classification. arXiv (2020).
- Sang Michael Xie, Ananya Kumar, Robert Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. 2021a. In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness. In International Conference on Learning Representations (ICLR) .
- Sang Michael Xie, Tengyu Ma, and Percy Liang. 2021b. Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization. International Conference on Machine Learning (ICML) (2021).
- Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021c. An Explanation of In-context Learning as Implicit Bayesian Inference. arXiv preprint arXiv:2111.02080 (2021).
- Caiming Xiong, Stephen Merity, and Richard Socher. 2016. Dynamic memory networks for visual and textual question answering. In International conference on machine learning . 2397-2406.
- Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. 2021. Detoxifying Language Models Risks Marginalizing Minority Voices. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Association for Computational Linguistics, Online, 2390-2397. https://doi.org/10.18653/v1/2021.naacl-main.190
- Huijuan Xu and Kate Saenko. 2016. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In European Conference on Computer Vision . Springer, 451-466.
- Jian Xu, Sunkyu Kim, Min Song, Minbyul Jeong, Donghyeon Kim, Jaewoo Kang, Justin F Rousseau, Xin Li, Weijia Xu, Vetle I Torvik, et al. 2020. Building a PubMed knowledge graph. Scientific data 7, 1 (2020), 1-15.
- Jia Xu, Pengwei Yang, Shang Xue, Bhuvan Sharma, Marta Sanchez-Martin, Fang Wang, Kirk A Beaty, Elinor Dehan, and Baiju Parikh. 2019. Translating cancer genomics into precision medicine with artificial intelligence: applications, challenges and future perspectives. Human genetics 138, 2 (2019), 109-124.
- Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021. ByT5: Towards a Token-Free Future with Pre-Trained Byte-to-Byte Models. arXiv preprint arXiv:2105.13626 (2021).
- Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934 (2020).
- Eugene Yang, Sean MacAvaney, David D Lewis, and Ophir Frieder. 2021. Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review. arXiv preprint arXiv:2105.01044 (2021).
- Mengjiao Yang and Ofir Nachum. 2021. Representation Matters: Offline Pretraining for Sequential Decision Making. In ICML .
- Qian Yang, Aaron Steinfeld, Carolyn P Rose, , and John Zimmerman. 2020. Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems .
- Qian Yang, Jina Suh, Nan-Chen Chen, and Gonzalo Ramos. 2018. Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models. In DIS '18: Proceedings of the 2018 Designing Interactive Systems Conference . ACM.
- Qian Yang, John Zimmerman, Aaron Steinfeld, and Anthony Tomasic. 2016. Planning Adaptive Mobile Experiences When Wireframing. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems . ACM.
- Zhilin Yang, Zihang Dai, Yiming Yang, J. Carbonell, R. Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In NeurIPS .
- Michihiro Yasunaga and Percy Liang. 2020. Graph-based, Self-Supervised Program Repair from Diagnostic Feedback. In International Conference on Machine Learning (ICML) .
- Michihiro Yasunaga and Percy Liang. 2021. Break-It-Fix-It: Unsupervised Learning for Program Repair. In International Conference on Machine Learning (ICML) .
- Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. arXiv preprint arXiv:2104.06378 (2021).
- Nanyang Ye, Kaican Li, Lanqing Hong, Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li. 2021. OoD-Bench: Benchmarking and Understanding Out-of-Distribution Generalization Datasets and Algorithms.
- Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. 2019. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442 (2019).
- Pengcheng Yin, Graham Neubig, Wen tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data. In ACL .
- Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. 2019. Learning and Evaluating General Linguistic Intelligence. arXiv preprint arXiv:1901.11373 (2019).
- Naruki Yoshikawa, Ryuichi Kubo, and Kazuki Z. Yamamoto. 2021. Twitter integration of chemistry software tools. Journal of Cheminformatics 13, 1 (July 2021). https://doi.org/10.1186/s13321-021-00527-x
- Jiaxuan You, Bowen Liu, Rex Ying, Vijay Pande, and Jure Leskovec. 2018. Graph Convolutional Policy Network for GoalDirected Molecular Graph Generation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems . Curran Associates Inc., Red Hook, NY, USA, 6412-6422.
- Chao Yu, Jiming Liu, and Shamim Nemati. 2019. Reinforcement learning in healthcare: A survey. arXiv preprint arXiv:1908.08796 (2019).
- Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. 2022. Differentially Private Fine-tuning of Language Models. In International Conference on Learning Representations . https://openreview.net/forum?id=Q42f0dfjECO
- Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. 2020c. Jaket: Joint pre-training of knowledge graph and language understanding. arXiv preprint arXiv:2010.00796 (2020).
- Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. 2020a. BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) .
- Kun-Hsing Yu, Andrew L Beam, and Isaac S Kohane. 2018. Artificial intelligence in healthcare. Nature biomedical engineering 2, 10 (2018), 719-731.
- Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. 2020b. MOPO: Model-based Offline Policy Optimization. arXiv preprint arXiv:2005.13239 (2020).
- Eliezer Yudkowsky. 2016. The AI alignment problem: why it is hard, and where to start. Symbolic Systems Distinguished Speaker (2016).
- Eliezer Yudkowsky et al. 2008. Artificial intelligence as a positive and negative factor in global risk. Global catastrophic risks 1, 303 (2008), 184.
- Jeffrey M Zacks, Barbara Tversky, and Gowri Iyer. 2001. Perceiving, remembering, and communicating structure in events. Journal of experimental psychology: General 130, 1 (2001), 29.
- Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael J Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In 9th { USENIX } Symposium on Networked Systems Design and Implementation ( { NSDI } 12) . 15-28.
- Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big Bird: Transformers for Longer Sequences.. In NeurIPS .
- Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. BitFit: Simple Parameter-efficient Fine-tuning for Transformerbased Masked Language-models. CoRR abs/2106.10199 (2021). arXiv:2106.10199 https://arxiv.org/abs/2106.10199
- Amir R. Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. 2018. Taskonomy: Disentangling Task Transfer Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE.
- Jakub Zavrel, Walter Daelemans, and Jorn Veenstra. 1997. Resolving PP attachment ambiguities with memory-based learning. In CoNLL97: Computational Natural Language Learning .
- Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision . Springer, 818-833.
- Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019a. From Recognition to Cognition: Visual Commonsense Reasoning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .
- Rowan Zellers, Ari Holtzman, Matthew Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, and Yejin Choi. 2021a. PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World. arXiv preprint arXiv:2106.00188 (2021).
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019b. Defending Against Neural Fake News. In Advances in Neural Information Processing Systems (NeurIPS) . 9054-9065.
- Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021b. MERLOT: Multimodal Neural Script Knowledge Models. arXiv preprint arXiv:2106.02636 (2021).
- Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2021. Scaling vision transformers. arXiv preprint arXiv:2106.04560 (2021).
- Haoran Zhang, Amy X Lu, Mohamed Abdalla, Matthew McDermott, and Marzyeh Ghassemi. 2020b. Hurtful words: quantifying biases in clinical contextual word embeddings. In proceedings of the ACM Conference on Health, Inference, and Learning . 110-120.
- Michael Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating Extra-Linguistic Contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 7371-7387. https://doi.org/10.18653/v1/2021.emnlp-main.586
- T. Zhang and T. Hashimoto. 2020. On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies. In Association for Computational Linguistics (ACL) .
- Tianyi Zhang and Tatsunori Hashimoto. 2021. On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies. arXiv preprint arXiv:2104.05694 (2021).
- Xingliang Zhang and Degan Shu. 2021. Current understanding on the Cambrian Explosion: questions and answers. Paläontologische Zeitschrift 95 (2021), 641-660.
- Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. 2020a. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747 (2020).
- Yuhui Zhang, Allen Nie, Ashley Zehnder, Rodney L Page, and James Zou. 2019b. VetTag: improving automated veterinary diagnosis coding via large-scale language modeling. NPJ digital medicine 2, 1 (2019), 1-8.
- Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R Bowman. 2021. When Do You Need Billions of Words of Pretraining Data?. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics .
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019a. ERNIE: Enhanced Language Representation with Informative Entities. In ACL .
- Zhoutong Zhang, Qiujia Li, Zhengjia Huang, Jiajun Wu, Joshua B Tenenbaum, and William T Freeman. 2017. Shape and material from sound. (2017).
- Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender Bias in Contextualized Word Embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Association for Computational Linguistics, Minneapolis, Minnesota, 629-634. https://doi.org/10.18653/v1/N19-1064
- Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Copenhagen, Denmark, 2979-2989. https: //doi.org/10.18653/v1/D17-1323
- Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018. Learning Gender-Neutral Word Embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Brussels, Belgium, 4847-4853. https://doi.org/10.18653/v1/D18-1521
- Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. 2020b. Masking as an Efficient Alternative to Finetuning for Pretrained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Online, 2226-2241. https://doi.org/10.18653/v1/2020.emnlpmain.174
- Qingyu Zhao, Ehsan Adeli, and Kilian M Pohl. 2020a. Training confounder-free deep learning models for medical applications. Nature communications 11, 1 (2020), 1-9.
- Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690 (2021).
- Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021. When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset. arXiv preprint arXiv:2104.08671 .
- Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. How does NLP benefit legal system: A summary of legal artificial intelligence. arXiv preprint arXiv:2004.12158 (2020).
- Jin Peng Zhou, Yuhuai Wu, Colin Li, and Roger Grosse. 2021c. REFACTOR: Learning to Extract Theorems from Proofs. The First Mathematical Reasoning in General Artificial Intelligence Workshop, ICLR 2021 (2021). https://mathai-iclr.github.io/ papers/papers/MATHAI\_22\_paper.pdf
- Kaitlyn Zhou, Kawin Ethayarajh, and Dan Jurafsky. 2021a. Frequency-based Distortions in Contextualized Word Embeddings. ArXiv abs/2104.08465 (2021). https://arxiv.org/abs/2104.08465
- Sharon Zhou, Mitchell L Gordon, Ranjay Krishna, Austin Narcomey, Li Fei-Fei, and Michael S Bernstein. 2019. Hype: A benchmark for human eye perceptual evaluation of generative models. NeurIPS (2019).
- Xiyou Zhou, Zhiyu Chen, Xiaoyong Jin, and William Yang Wang. 2020. Hulk: An energy efficiency benchmark platform for responsible natural language processing. arXiv preprint arXiv:2002.05829 (2020).
- Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. 2021b. Challenges in Automated Debiasing for Toxic Language Detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume . Association for Computational Linguistics, Online, 3143-3155. https://aclanthology.org/2021.eacl-main.274
- Yiren Zhou, Seyed-Mohsen Moosavi-Dezfooli, Ngai-Man Cheung, and Pascal Frossard. 2018. Adaptive Quantization for Deep Neural Networks. In Thirty-Second AAAI Conference on Artificial Intelligence .
- Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying Memories in Transformer Models. arXiv:2012.00363 [cs.CL]
- Ming Zhu, Aman Ahuja, Wei Wei, and Chandan K Reddy. 2019. A hierarchical attention retrieval model for healthcare question answering. In The World Wide Web Conference . 2472-2482.
- Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. 2008. Maximum Entropy Inverse Reinforcement Learning. In Association for the Advancement of Artificial Intelligence (AAAI) .
- Annette Zimmerman. 2020. If You Can Do Things with Words, You Can Do Things with Algorithms. https://dailynous. com/2020/07/30/philosophers-gpt-3/#zimmermann
- Annette Zimmermann. 2021. Stop Building Bad AI. Boston Review (July 2021). https://bostonreview.net/science-nature/ annette-zimmermann-stop-building-bad-ai
- Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. 2021. Contrastive Learning Inverts the Data Generating Process. arXiv preprint arXiv:2102.08850 (2021).
- Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. 2017. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595 (2017).
- Shoshana Zuboff. 2018. The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power .
- Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. 2021. Language-agnostic representation learning of source code from structure and context. arXiv preprint arXiv:2103.11318 (2021).