## TOWARDS INTERVENTION-CENTRIC CAUSAL REASONING IN LEARNING AGENTS
Benjamin Lansdell Department of Bioengineering University of Pennsylvania Philadelphia, PA
## ABSTRACT
Interventions are central to causal learning and reasoning. Yet ultimately an intervention is an abstraction: an agent embedded in a physical environment (perhaps modeled as a Markov decision process) does not typically come equipped with the notion of an intervention - its action space is typically ego-centric, without actions of the form 'intervene on X'. Such a correspondence between ego-centric actions and interventions would be challenging to hard-code. It would instead be better if an agent learnt which sequence of actions allow it to make targeted manipulations of the environment, and learnt corresponding representations that permitted learning from observation. Here we show how a meta-learning approach can be used to perform causal learning in this challenging setting, where the actionspace is not a set of interventions and the observation space is a high-dimensional space with a latent causal structure. A meta-reinforcement learning algorithm is used to learn relationships that transfer on observational causal learning tasks. This work shows how advances in deep reinforcement learning and meta-learning can provide intervention-centric causal learning in high-dimensional environments with a latent causal structure.
## 1 INTRODUCTION
"...suppose that an individual ape ... for the first time observes the wind blowing a tree such that the fruit falls to the ground... we believe that most primatologists would be astounded to see the ape, just on the basis of having observed the wind make fruit fall ... create the same movement of the limb ... the problem is that the wind is completely independent of the observing individual and so causal analysis would have to proceed without references to the organism's own behavior." Tomasello and Call, 1997 (Tomasello and Call, 1997)
Learning causal relationships in an environment is necessary for flexible planning and problem solving. Humans are adept at learning the causal structure of an environment, not just from their own actions, but also through passive observation (Woodward, 2007). This includes imitating or taking cues from other animals (social learning), but also learning causal relationships in environments where no other agents are present. For instance, when a broom resting against a wall accidentally falls and hits a light switch, and a light subsequently turns on, we may infer that the switch causes the light to turn on. It has been argued that this ability is related to an intervention-centric view of causality, which makes us significantly more flexible observational learners and causal reasoners than other animals (Woodward, 2007; 2010).
According to the interventionist account of causality, a common view in statistics, philosophy and psychology (Gopnik and Schulz, 2007), a causal relationship exists between two events if intervening to make one occur results in the other occurring (Woodward, 2003). Importantly, an intervention is an abstract notion - it does not matter what is intervening, just as long as it is somehow external to the system being studied. In the light switch example, the accidentally-falling broom may be viewed as an intervention on the 'switch-light' system. Learning how to turn on the light by observing this scene can be achieved with the following things: first, by viewing the environment as one that has causal structure, that exists independently of the agent; second, understanding that some of the agent's
actions may be interventions that can exploit this structure; and, third, understanding that other objects/agents in the environment may also be able to intervene to exploit causal relationships. With these an agent could learn, just by observing the falling broom, that if it acts in a way to intervene on the light switch then the light will turn on. Since the notion of intervention is abstract - it could be the broom or the agent - this allows the agent to transfer what it observes to what will happen when it acts on the world in a very flexible manner (Figure 1a).
How else might causal relationships be learnt and exploited? Consider other ways an animal (or a more generic agent) might learn how to turn the light on. First, it could learn directly from its own experience, by turning the switch on itself. Any agent that learns through operant conditioning may exploit causal relationships in this manner (Gershman, 2017). And, second, it could learn by imitating or otherwise being directed to the switch by another animal (or agent). In fact these are the dominant forms of learning in the learning agent literature - reinforcement learning and imitation learning, respectively (Sutton and Barto, 2017; Edwards et al., 2018). While they may result in an agent learning to use the causal structure of the environment, they do not require the agent possess a sense of interventions. Thus such agents cannot learn from observation in the same way as the intervention-centric learner described above.
## 1.1 INTERVENTION-BASED CAUSAL LEARNING
What abilities are indicative of an intervention-centric notion of causal relationships? One telling ability is the ability to integrate information from observation and an agent's own actions into a single causal model of the environment, and to use this to execute novel actions in order to obtain some desired outcome. By being novel, the action could not have been learnt through operant conditioning, nor could it have been suggested by watching another agent. Instead, it must have come from predicting a desirable outcome based on an understanding that some observed relationships will also hold when the agent performs certain actions, that is, it must have come from predictions based on a causal model of the environment. In the animal kingdom, humans are superior tool users and observational causal learners. Corvids for instance, despite their cunning, seemingly are not able to create novel interventions from observation (Taylor et al., 2014; 2015). The notion of an intervention thus appears as a powerful and defining characteristic of human intelligence - it underlies our ability to understand and manipulate the world (Woodward, 2003). Yet tests of such abilities in the learning agent literature are largely lacking; the focus is instead typically on reinforcement learning or imitation learning. It remains relatively unexplored how this form of causal learning can be implemented in a learning agent setting.
Of course, one way is through an explicit causal model. For learning agents whose environment model is a causal Bayesian network (CBN) (Pearl, 2000), or structural causal model (SCM) (Peters et al., 2017), an intervention-centric sense of causality is hard-coded into the structure of the model and action space. That is, the environment factors into a network of cause-effect relationships, and the action space may straightforwardly be given as interventions on edges of that network. In these cases, observational learning is a very well explored problem with many different approaches (Pearl, 2000; Zhang and Bareinboim, 2016; Peters et al., 2017; Bareinboim et al., 2015; Bareinboim and Pearl, 2016; Zhang and Bareinboim, 2017). The problem is that SCMs are hard to scale to high-dimensional state spaces, perhaps with a latent causal structure, and thus they are hard to combine with modern deep-learning-based methods (though see Arjovsky et al. (2019) for interesting ideas). Thus we may want to look for other ways of constructing learning agents with intervention-centric learning abilities.
## 1.2 CONTRIBUTION
We focus on this problem in a reinforcement learning agent setting, i.e. with Markov decision processes (MDPs). We will break down our analysis into two Problems:
1. How can an agent learn a causal model from observation?
2. How can an agent learn which sequence of its actions constitute an intervention that can exploit the causal model it has learnt?
Here learning from observation means the agent observes only state transitions, o t , and not the actions that may have been taken by some other entity to generate them. In what sense does this get
Figure 1: a) A causal relationship exists between the state of the branch and the state of the apple: if the branch is shaken, the apple can fall. By definition, it does not matter what intervenes on the branch to make it fall. This permits learning the causal relationship from a diverse range of sources: self action, another's action, or some other perturbation. b) Causal inference assumes an observational distribution, and asks when and how can we learn about what happens when intervening on the environment. Here, we argue the analogous setting in learning agents is to assume access to an observational distribution, and to study when and how this can be transferred to learn policies over action-conditioned distributions.
<details>
<summary>Image 1 Details</summary>
### Visual Description
## Diagram: Causal Inference vs. Learning Agent
### Overview
The image presents two scenarios, labeled 'a' and 'b'. Scenario 'a' depicts a monkey observing an apple on a tree branch, with another monkey on the ground. Scenario 'b' contrasts causal inference and learning agent models using directed graphs to represent probabilities and interventions.
### Components/Axes
**Part a):**
* **Setting:** A tree with a branch.
* **Objects:** Two monkeys (one on the branch, one on the ground), an apple on the branch, and wavy lines suggesting wind.
**Part b):**
* **Titles:** "Causal inference" and "Learning agent".
* **Probabilities:** P(s), P(s | do(sᵢ = x)), P(sₜ₊₁ | sₜ), P(sₜ₊₁ | sₜ, aₜ).
* **Labels:** "Observation:" and "Intervention:".
* **Graphs:** Directed graphs with nodes (gray circles) and edges (arrows). One node is colored red in each of the "Intervention" graphs.
### Detailed Analysis
**Part a):**
* A monkey is sitting on a tree branch, looking at a red apple.
* Another monkey is on the ground, looking towards the tree.
* Wavy lines suggest wind or movement in the air.
**Part b):**
* **Causal Inference:**
* **Observation:** A directed graph with 5 gray nodes and multiple arrows indicating causal relationships.
* Probability: P(s)
* **Intervention:** A directed graph with 4 gray nodes and one red node. Arrows indicate causal relationships.
* Probability: P(s | do(sᵢ = x))
* **Learning Agent:**
* **Observation:** A directed graph with two gray nodes and a single arrow.
* Probability: P(sₜ₊₁ | sₜ)
* **Intervention:** A directed graph with three gray nodes and one red node. Arrows indicate relationships.
* Probability: P(sₜ₊₁ | sₜ, aₜ)
* **Flow:** An arrow labeled with a question mark connects "Observation:" to "Intervention:".
### Key Observations
* Part a) illustrates a real-world scenario of observation and potential action.
* Part b) uses directed graphs to model causal inference and learning agents, contrasting observational and interventional probabilities.
* The red node in each "Intervention" graph likely represents an intervened variable.
### Interpretation
The image contrasts how causal inference and learning agents handle observations and interventions. Part a) sets the stage with a simple scenario. Part b) then formalizes these concepts using probabilistic graphical models. The "Causal inference" section shows how interventions (represented by the red node) alter the causal structure. The "Learning agent" section shows how the agent's actions (aₜ) influence the transition between states (sₜ and sₜ₊₁). The question mark between "Observation" and "Intervention" suggests the need to infer causal relationships from observations before planning interventions.
</details>
at intervention-centric causal learning? By transferring from an observational setting to a setting where the agent is able to act on the environment, the agent exploits the fact that the observational and action phases share a common causal structure. This can be thought of as analogous to the typical problem in causal inference: how to transfer information from an observational distribution to candidate interventional ones (Figure 1b). While Problem 1 has been studied (Nair et al., 2019; Thomas et al., 2017; Sawada et al., 2018), to our knowledge Problem 2 has seen less attention. Yet arguably solutions to both are needed for a flexible learner that can fully exploit what it learns from observation and action.
We use a meta-reinforcement learning approach. The aim is that the agent learns for itself both what patterns of covariation indicate transferable relationships within the environment, and which of its actions may be able to exploit these relationships. In this way the agent performs causal learning in a flexible manner, without explicitly casting its environment into causal factors, and without having explicit labels that indicate when data is being drawn from different interventional distributions.
## 2 A MOTIVATING EXAMPLE
We start with a simple example. Consider a state space that factors into three variables: s t = ( s 1 t , s 2 t , s 3 t ) . Then suppose the environment has one of the two dynamic causal structures, where the differences between Model A and B is highlighted in bold :
- Model A (chain):
$$s _ { t } ^ { 1 } & = y _ { t } ^ { 1 } \\ s _ { t } ^ { 2 } & = y _ { t } ^ { 2 } + ( 1 - y _ { t } ^ { 2 } ) ( s _ { t - 1 } ^ { 1 } ) \\ s _ { t } ^ { 3 } & = y _ { t } ^ { 3 } + ( 1 - y _ { t } ^ { 3 } ) ( s _ { t - 1 } ^ { 2 } ) ,$$
<!-- formula-not-decoded -->
where y i t ∼ Bn ( p i ) denote Bernoulli random variables with probability of activation p i .
- And Model B (delayed fork):
<!-- formula-not-decoded -->
<!-- formula-not-decoded -->
Importantly, it is possible with certain parameter choices that, even though the models have a different causal structure, if all an agent observes is s t , it could not distinguish between Model A and Model B. It would have to interact with the system to tell the two apart. Consider then the task of learning which is the true causal structure. We will explore this problem using the meta-reinforcement learning approach of Wang et al. (2016), i.e. the same approach taken in Dasgupta et al. (2019) for causal inference through meta-RL, though they test their approach on static causal environments.
Figure 2: Average reward for meta-RL agent in simple causal inference environments. Curves show mean plus/minus standard error over n = 10 runs.
<details>
<summary>Image 2 Details</summary>
### Visual Description
## Line Charts: Learning Method Comparison
### Overview
The image presents four line charts comparing different learning methods: Confounded learning, Observational learning, Off-policy interventional learning, and On-policy interventional learning. Each chart displays the reward obtained over trials (x10^3). The y-axis represents the reward, ranging from 0.0 to 1.0. The x-axis represents the trial number, ranging from 0 to 50 (x10^3). The data is represented by a blue line with a lighter blue shaded area indicating the variance or confidence interval.
### Components/Axes
* **Titles (Top):**
* Confounded learning
* Observational learning
* Off-policy interventional learning
* On-policy interventional learning
* **Y-axis Label (Left):** reward
* **Y-axis Scale (Left):** 0.0, 0.2, 0.4, 0.6, 0.8, 1.0
* **X-axis Label (Bottom):** trial (x10^3)
* **X-axis Scale (Bottom):** 0, 25, 50
### Detailed Analysis
**1. Confounded Learning:**
* **Trend:** The reward remains relatively constant over the trials.
* **Values:** The reward fluctuates around 0.5, with a range of approximately 0.4 to 0.6.
**2. Observational Learning:**
* **Trend:** The reward initially increases, peaks around trial 15, and then decreases and stabilizes.
* **Values:** The reward starts around 0.5, rises to approximately 0.75, and then settles around 0.6.
**3. Off-policy Interventional Learning:**
* **Trend:** The reward increases sharply in the beginning and then stabilizes at a high level.
* **Values:** The reward starts near 0.5, rapidly increases to approximately 0.9, and then fluctuates around 0.9 with some variance.
**4. On-policy Interventional Learning:**
* **Trend:** The reward increases rapidly, overshoots, and then stabilizes at a high level.
* **Values:** The reward starts near 0.4, increases rapidly to approximately 0.95, dips slightly, and then fluctuates around 0.95 with some variance.
### Key Observations
* Confounded learning shows a stable but relatively low reward.
* Observational learning shows an initial improvement followed by a decline.
* Off-policy and On-policy interventional learning both achieve high rewards, but On-policy learning shows a more rapid initial increase.
### Interpretation
The charts suggest that interventional learning methods (both off-policy and on-policy) are more effective in achieving higher rewards compared to confounded and observational learning. On-policy interventional learning appears to have the fastest initial learning rate, but both off-policy and on-policy methods eventually converge to similar high reward levels. Confounded learning performs the worst, maintaining a low and stable reward. Observational learning shows some initial promise but ultimately plateaus at a lower reward than the interventional methods. The shaded areas indicate the variability in the rewards, which is generally higher in the interventional methods, especially during the initial learning phase.
</details>
## 2.1 META-LEARNING AGENT ARCHITECTURE
The meta-reinforcement learning approach works as follows. A task, or trial, is sampled from some distribution D . Here, this just means one of Model A or B is chosen with probability p A = 0 . 5 . Within the trial, states are generated according to the appropriate model, for N steps. An LSTM network (Hochreiter and Schmidhuber, 1997) (with 48 hidden units) was used. At each time step the LSTM receives the vector ( s t , a t -1 , r t -1 ) as input, where s t is the observation, a t -1 is the previous action (as a one-hot vector) and r t -1 the reward (as a single real-value). The outputs are a linear function of the LSTM's state. A set of logits are output (with dimensionality equal to the number of available actions), plus a scalar baseline. A softmax is applied to the logits, and then sampled to give a selected action. Learning was by the asynchronous advantage actor-critic (A3C) (Mnih et al., 2016) algorithm. In this framework, the loss function consists of three terms: the policy gradient, the baseline cost and an entropy cost. The baseline cost was weighted by 0.05 relative to the policy gradient cost. Optimization was done by ADAM optimization with learning rate 10 -3 , β 1 = 0 . 9 , β 2 = 0 . 999 , = 10 -7 . What follows was implemented in TensorFlow.
## 2.2 TASK DETAILS
The agent is presented with N = 10 observations from a sequence s t generated according to either Model A or Model B. The agent is rewarded at the end of the trial for correctly identifying the true model. That is, the action space is A = { A,B } . If a N is the correct model, then reward r N = 1 is given. Otherwise r N = 0 . At all other times in the trial, the reward is zero. We compare performance of this meta-RL approach in four different settings, each providing different amount of information to the agent, and allowing different levels of interaction with the environment. The specific values for these models are provided in the supplementary material.
1. Confounded - the agent only observes s t , thus has insufficient information to solve the problem.
2. Observational - there are now perturbations of the environment, changing the distribution over s t , but these perturbations are unobserved. More specifically, z i t ∼ Bn ( p int i ) are 'intervention indicator' variables sampled at each time step. If z i t = 1 then y i t = 1 , otherwise it follows the same dynamics as before. For some cases, this may allow the agent to identify the true causal structure.
3. Off-policy interventional - the agent now observes the perturbations z t , concatenated onto s t . Now the correct structure is identifiable, provided the agent can learn which of z 2 and z 3 are associated with each node s 2 and s 3 .
4. On-policy interventional - now the action space of the agent is the perturbation-space z . In this case, the action space for the agent can be considered as interventions on the environment. Here the agent additionally observes a go cue, at the end of the trial, to indicate it should provide its response as to which is the correct underlying model.
## 2.3 RESULTS AND DISCUSSION
Though this is a very simple example, it captures different forms of causal learning. In the purely observational setting, in this case, no causal learning is possible. In the observational with perturbations setting, causal learning may be possible, and is akin to something like emulative learning with 'ghost conditions' in cognitive science (e.g. the wind blowing the branch thought experiment above) (Hopper, 2010). In the off-policy interventional setting, causal learning may again be possible, and may be aided by learning to exploit the additional observed cues that a variable is being perturbed from its default dynamics, z . Finally, in the on-policy interventional setting, the agent can learn the causal model directly through its own interventions - agent-centric causal learning possible through any reinforcement learning algorithm. The results using the meta-RL algorithm show both the onand off-policy interventional settings are indeed solvable after a few thousand trials (Figure 2).
This setup allowed us to explore causal learning under two significant Assumptions:
1. The causal relationships are expressed more or less directly between the observed variables, the causal structure is not latent. 1 Further, the variables indicating whether and which variable was being perturbed were also directly observed, at least in the relevant setting.
2. The action space is given as direct manipulations of the underlying causal variables.
These assumptions relate to Problems 1 and 2 described in the introduction. Thus, to extend current approaches, we consider ways in which these assumptions can be dropped.
## 3 DEALING WITH LATENT CAUSAL STRUCTURE
We can first study settings that relax Assumption 1. In fact this is very close to the problem tackled by Nair et al. (2019), except in their case the focus is on planning to reach a goal state. And in their case the actions can be taken as direct manipulations of the underlying causal variables, an assumption we will drop in the next section.
## 3.1 TASK DETAILS
To extend our simple example to this case, we consider an 8x8x3 observation space, denoted o t . Three pixels in this space correspond to the state of s 1 t , s 2 t , s 3 t , obeying the same dynamics as in the previous section. When active, the corresponding pixel is colored white, otherwise it is black. Then, to mix the state of these variables in the observations, a Gaussian blur is applied to the image. In the relevant conditions, the additional indicator variables z t are added in the same locations as s i , but are added only to the red channel. Further, the observed image at each frame t is a 50-50 mix of the previous frame o t -1 and the current one (the Gaussian blurred pixels corresponding to the state of the system at frame t ). This introduces some temporal blurring in the observed dynamics also (Figure 3a). Now the relationship between the variables y i and the observed images is less obvious. As before, reward r N = 1 , is given for correctly identifying the underlying model, and r N = 0 otherwise. The observed image o t is fed to the learning agent, instead of the underlying state s t
## 3.2 RESULTS
The meta-RL agent's architecture is modified to add a fully connected layer of size 64, with inputs over the flattened image, before being fed to the LSTM. With this change, the learner is able to quickly learn the correct model. In the interventional conditions, it can learn how to figure out the correct model after only a few thousand trials. The observational setting takes closer to fifty thousand trials to reach the same performance (Figure 3b). Nonetheless, the agent is able to learn the latent causal structure underlying the observations in all but the confounded case.
1 Technically, the delay introduces some degree of unobservedness.
Figure 3: Environment with latent causal structure. a) Four frames from the confounded environment condition. Three, blurred, flashing pixels indicate the state of the variables y t , obeying either chain or delayed fork dynamics between them. For illustration, these are highlighted in red, though this information isn't provided to the agent. The activated pixels decay over time. The other settings are similar, with additional flags to represent the state of z t , where relevant. b) Curves show mean reward plus/minus standard error over n = 10 runs.
<details>
<summary>Image 3 Details</summary>
### Visual Description
## Learning Method Comparison: Confounded vs. Interventional
### Overview
The image presents a comparison of different learning methods, including confounded learning, observational learning, and interventional learning (both off-policy and on-policy). Part (a) shows a sequence of images representing confounded learning at different time steps (t=0 to t=3). Part (b) displays line graphs illustrating the performance of each learning method over trials.
### Components/Axes
**Part a)**
* **Title:** Confounded learning
* **Images:** Four grayscale images arranged in a 2x2 grid, labeled t=0, t=1, t=2, and t=3. Each image is a grid of pixels, with varying shades of gray. In the t=0 image, there are three red squares highlighting specific regions.
**Part b)**
* **Title:** Comparison of Learning Methods
* **X-axis:** trial (x10^3), ranging from 0 to 40.
* **Y-axis:** Performance (unlabeled, but implied), ranging from 0 to 1.0, with tick marks at 0.2, 0.4, 0.6, 0.8, and 1.0.
* **Graphs:** Four line graphs, each representing a different learning method:
* Confounded learning
* Observational learning
* Off-policy interventional learning
* On-policy interventional learning
### Detailed Analysis
**Part a) Confounded Learning Images:**
* **t=0:** A grid of grayscale pixels. Three red squares highlight specific regions. One is in the top-left corner, one is slightly below and to the right of the first, and the third is in the lower-center.
* **t=1:** Two bright spots appear in the image, one in the upper-right and one in the lower-left.
* **t=2:** The bright spots become more defined and pronounced.
* **t=3:** The bright spots are further refined and more clearly visible.
**Part b) Performance Graphs:**
* **Confounded learning:** The blue line rises sharply from approximately 0.2 to about 0.5 within the first 5000 trials, then plateaus and fluctuates around 0.5 for the remaining trials.
* **Observational learning:** The blue line starts near 0.2 and gradually increases to approximately 0.95 by the end of the 40,000 trials. The line shows a steady upward trend.
* **Off-policy interventional learning:** The blue line rises sharply from approximately 0.2 to about 0.9 within the first 5000 trials, then fluctuates around 0.9 for the remaining trials.
* **On-policy interventional learning:** The blue line rises sharply from approximately 0.2 to about 0.95 within the first 5000 trials, then fluctuates around 0.95 for the remaining trials.
### Key Observations
* Confounded learning plateaus at a relatively low performance level.
* Observational learning shows a gradual but consistent improvement over trials.
* Both off-policy and on-policy interventional learning methods achieve high performance levels quickly.
* Interventional learning methods appear to outperform confounded and observational learning in this scenario.
### Interpretation
The data suggests that interventional learning methods are more effective than confounded or observational learning in this particular context. Confounded learning seems to be limited in its ability to improve performance, while observational learning shows gradual improvement but does not reach the same high performance levels as interventional learning. The rapid increase in performance for both off-policy and on-policy interventional learning indicates that these methods are able to quickly learn and adapt to the task. The red squares in the confounded learning images at t=0 might be highlighting areas of interest or potential confounds that the learning algorithm is struggling with.
</details>
## 4 CAUSAL LEARNING IN AGENT-CENTRIC ACTION SPACES
We now focus on the setting that relaxes Assumption 2. That is, the action space is now ego-centric, and cannot be interpreted as a direct manipulation of the relevant causal variables underlying the model dynamics. To test causal learning in this setting, we use a simple grid-world environment.
## 4.1 TASK DETAILS
The task is an 'escape room' task. The environment consists of a 5x5 grid-world. On three edges of the grid are buttons. When anything overlaps with the pixel adjacent to the button, it is pushed. In a given trial of N steps, only one of the buttons (chosen uniformly at random) will activate/open the door on the fourth wall. This door will stay open for T time steps.
There are two phases to this environment. In the first observational phase, for N O steps, a white object bounces randomly between the three buttons, occasionally activating the door. The agent's actions are ineffective. In the second phase, the action phase, a green cue in the upper left corner indicates that the agent's actions now effect the environment - they move a separate (gray) object around the grid-world (Figure 4a). While the bouncing object can move one pixel in the cardinal directions, or one pixel diagonally, the agent can only move one pixel in the four cardinal directions. This phase lasts N A steps. The agent is rewarded if it moves to the pixel immediately adjacent to the open door. A reward of R d is administered.
We test the case that N A is too small for a valid policy to be the agent, in the action phase, tries each button sequentially to see which one opens the door - to solve the task it has to pay attention to the observation phase state transitions. Here N O = 20 , N A = 10 , the door is open for T = 5 time steps, and the reward is R d = 10 .
## 4.2 RESULTS
The same network architecture and algorithm as the previous section is used. With this, the meta-RL agent is able to learn to perform this causal learning task after about two hundred thousand trials (Figure 4b). The agent start location is randomly chosen, at the start of each action phase. It then successfully moves to the correct button, and then to the door for reward on a high proportion of trials. Thus this setup is able to successfully utilize information from the observation phase for use in the action phase. In doing so the agent is not mimicking the movement of the white box; it is
Figure 4: The escape-room environment. a) Task involves observation phase, in which the white box bounces randomly between the three buttons (all but the right). When the box is adjacent to the randomly chosen button that opens the door for that trial, it turns red. In the action phase, the agent moves the gray box around. It has to learn from the previous phase which button to go for to open the door and move to it in time to get reward. Some frames are skipped for simplicity in this illustration. b) Curves show mean reward plus/minus standard error over n = 5 runs.
<details>
<summary>Image 4 Details</summary>
### Visual Description
## Chart/Diagram Type: Multi-Panel Image
### Overview
The image consists of two panels, labeled a) and b). Panel a) shows a series of six images depicting an "Observation phase" and an "Action phase." These images appear to represent a visual task or game environment. Panel b) presents a line graph showing the "reward" over "trial (x10^4)".
### Components/Axes
**Panel a):**
* **Title:** Observation phase, Action phase
* **Content:** Six images arranged in two rows of three. Each image shows a black background with gray rectangular obstacles and a colored square (white, blue, green, red). The colored square changes position and color between the observation and action phases.
**Panel b):**
* **X-axis:** trial (x10^4). The axis ranges from 0 to 60. Axis markers are present at 0, 20, 40, and 60.
* **Y-axis:** reward. The axis ranges from approximately 8 to 16. Axis markers are present at 8, 12, and 16.
* **Data:** A blue line graph with a shaded blue area around the line, representing the standard deviation or confidence interval.
### Detailed Analysis or Content Details
**Panel a):**
* The images in the "Observation phase" show a white, blue, and red square respectively.
* The images in the "Action phase" show a green, black, and red square respectively.
**Panel b):**
* The blue line starts at approximately 8 reward units at trial 0.
* The line increases sharply until approximately trial 20 (x10^4), reaching a reward of approximately 14.
* From trial 20 (x10^4) to trial 60 (x10^4), the line fluctuates around a reward value of approximately 15, with some peaks and dips. The shaded area around the line also varies in width, indicating changes in the variability of the reward.
### Key Observations
* Panel a) shows a visual task with changing colored squares.
* Panel b) shows a learning curve, where the reward increases rapidly initially and then plateaus with fluctuations.
### Interpretation
The image likely represents a reinforcement learning experiment. Panel a) illustrates the environment in which an agent interacts, with observation and action phases. The agent observes the environment (colored squares) and then takes an action. Panel b) shows the learning progress of the agent, with the reward increasing as the agent learns to perform the task effectively. The initial rapid increase in reward suggests a quick learning phase, followed by a plateau indicating that the agent has reached a stable level of performance. The fluctuations in the reward and the shaded area indicate the variability in the agent's performance.
</details>
not performing imitation learning. This transfer from observation to action is a key indicator of intervention-centric causal learning.
## 5 DISCUSSION
In essence here we proposed to take a step back from a causal formalism such as Pearl's (Pearl, 2000) and ask how behaviors that constitute interventions and observational learning as studied in cognitive science can be recapitulated in a learning agent. Interventions in this sense are a more vague notion than in a statistical causal model, which takes the exact mechanism of the intervention as given. But the human and animal cognitive science literature has studied interventions in many settings and stages of development. So we can better understand what behaviors constitute sophisticated causal reasoning by turning to these studies.
## 5.1 INSPIRATION FROM COGNITIVE SCIENCE
What notion of causality do we and other animals possess? Philosopher James Woodward (Woodward, 2010) argues that a notion of interventions is unique to humans. Other animals have either an egocentric notion of causality - they are only capable of learning causal relationships as they are revealed by and apply to the agent; or an agent-centric notion of causality, in which learning the structure of the world can also be achieved through reproducing another agent's actions.
Operant conditioning is of course ubiquitous among animals, and thus so is agent-centric causal learning. This type of learning can be quite sophisticated: tool-using animals demonstrate a nuanced understanding of the consequences of their actions (Schloegl and Fischer, 2017), and model-based reinforcement learning involves learning a causal model of the environment (Gershman, 2017). Yet such learning does not require a notion of an environment with structure independent of the agent's actions, and thus no sense in which actions are interventions on that environment.
Social learning amongst animals too can be quite complicated. In imitative learning, the actions of another agent are copied to achieve some goal. In emulative learning, an observed action is not necessarily reproduced, but an action is taken to reproduce a desired, observed outcome (Hopper, 2010). However, the causal learning that occurs in these cases may not be perfect: animals may not show an understanding of the relevant parts of another agent's actions to copy, leading to overimitation - performing a sub-optimal action simply because the demonstrator did, while control animals that only try the task for themselves learn the optimal action more quickly (Hopper, 2010). For these reasons, social learning is also not indicative of a full causal understanding of the environment either.
These can be contrasted with an intervention-centric notion of causality. The hallmarks of which are the ability to appropriately imitate and emulate others' actions to produce a desired goal, to not over-imitate, to not misuse tools or select the wrong tool for the job, to produce novel interventions
that are suggested by relationships learned from observation, and to learn and integrate relationships from a diverse range of sources (not just from others). Though animals such as primates and corvids, and children, can perform some of these (Tennie et al., 2010; Meltzoff et al., 2012; Bonawitz et al., 2010; Hopper et al., 2008; Schloegl and Fischer, 2017; Völter et al., 2016; Bonawitz et al., 2010; Taylor et al., 2015; 2014; Jelbert et al., 2014; Taylor et al., 2007), adult humans can most robustly perform each of these things.
## 5.2 RELATED WORK AND OUTLOOK
Here we focused on a notion of causal learning relatively unaddressed in the learning agent literature. We provided a simple environment that gets at this issue, and one simple meta-RL algorithm that can solve it. However we do not propose any algorithmic innovations: there may be better approaches, and these may be closely related to those already in use. It's thus useful to highlight related work.
The environment presented here requires the agent to take advantage of data drawn from an observational setting, in which the actions of the agent are ineffective. This is related to learning problems where the action labels generating the observed state sequences are not provided. This setting has been explored recently in imitation learning settings, known as imitation learning from observation (ILO) (Torabi et al., 2019b;a; Ho and Ermon, 2016; Zołna et al., 2019; Zolna et al., 2019; Torabi et al., 2018; Li et al., 2018; Liu et al., 2018; Wu et al., 2018; Edwards et al., 2018). But these methods do not solve the problem by themselves - they just imitate. Imitating for the sake of it, over-imitation, is a sign of a lack of causal understanding. Recent work combining imitation learning and reinforcement learning, or learning from imperfect demonstrations (Gao et al., 2018), address this to some extent (Zołna et al., 2019). RILO does so in a setting where action labels are not provided (Zołna et al., 2019). However such approaches, when no reward signal is available, will default to imitation learning. This may be a practical learning approach in the presence of an expert, but it does not get at causal learning. Though some version of these works may prove promising in the task domain we have tested here.
Alternatively we can approach the problem by eschewing imitation learning, and viewing the setting presented here as doing a form of off-policy reinforcement learning where action labels are not provided (similar to Borsa et al. (2017)). Some of the approaches used in ILO may prove still useful. For instance, models based on inverse dynamics models could take the observational data (Christiano et al., 2016; Pathak et al., 2017), use the inverse model to infer what action the agent could have taken, and then run an RL algorithm using the inferred actions. A sort of vicarious learning, evocative of agency theories of causation (Woodward, 2010). A comparison between this approach and that taken here is future work.
Acaveat is that here we have just focused on the idea of transferring what is learnt from observation to an action phase. But, in addition to this transferability, a key notion in intervention-based learning is that interventions are specific - they only act on a particular object. An algorithm's ability to parse the environment into a discrete set of objects, and to learn how these could be manipulated individually, was not tested. This problem relates to recent work on learning environment affordances and controllable factors (Thomas et al., 2017; Sawada et al., 2018). Methods like Recurrent Independent Mechanisms (Goyal et al., 2019) have proposed learning separable components of an environment with their own autonomous dynamics, which may relate to causal factors in the environment, and thus may prove useful in this regard.
Closely related to ideas of transferring from observational to action settings is work in the causal inference literature that learns causal models from unknown or uncertain interventions. In invariant causal prediction (Peters et al., 2016; Arjovsky et al., 2019), for instance, robustness to changes in environment are used as a cue for causal relations. These ideas has been explored in a meta-learning setting too (Bengio et al., 2019). Emerging ideas of the importance of invariance needs to be integrated with this intervention-centric notion used here (Arjovsky et al., 2019; Schölkopf, 2019). Finally, and perhaps most related to the work here is work on meta-learning causal learning algorithms (Dasgupta et al., 2019). This work differs from that here in that the environment is low-dimensional, almost fully observed, not dynamic, and the actions are taken as manipulations of the state variables in the causal graph. Thus there is a lot of progress in all of these related areas. A more explicit testing of these learning agents' causal learning abilities in the settings such as those tested here may prove useful in providing AI with a more human-like notion of causality.
## ACKNOWLEDGEMENTS
For many useful discussions, the author thanks Anna Leshinskaya and Anirudh Goyal.
## REFERENCES
- Martin Arjovsky, Leon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant Risk Minimization. ArXiv e-prints , pages 1-30, 2019.
- Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences , 113(27):7345-7352, 2016. ISSN 0027-8424. doi: 10.1073/pnas.1510507113. URL http://www.pnas.org/lookup/doi/10.1073/pnas.1510507113 .
- Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with Unobserved Confounders: A Causal Approach. Advances in Neural Information Processing Systems , pages 1-9, 2015. ISSN 10495258.
- Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. ArXiv e-prints , pages 1-26, 2019.
- Elizabeth Baraff Bonawitz, Darlene Ferranti, Rebecca Saxe, Alison Gopnik, Andrew Meltzoff, James Woodward, and Laura E. Schulz. Just do it? Investigating the gap between prediction and action in toddlers' causal inferences. Cognition , 115(1):104-117, 2010. doi: 10.1016/j.cognition.2009.12.001.Just.
- Diana Borsa, Olivier Pietquin, Rémi Munos, and Bilal Piot. Observational Learning by Reinforcement Learning. ArXiv e-prints , 2017.
- Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojciech Zaremba. Learning Deep Inverse Dynamics Model. ArXiv e-prints , 2016.
- Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-nelson. Causal Reasoning from Meta-reinforcement learning. ArXiv e-prints , 2019. URL https://openreview.net/forum?id=H1ltQ3R9KQ .
- Ashley D. Edwards, Himanshu Sahni, Yannick Schroecker, and Charles L. Isbell. Imitating Latent Policies from Observation. ArXiv e-prints , 2018. URL http://arxiv.org/abs/1805.07914 .
- Yang Gao, Huazhe, Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement Learning from Imperfect Demonstrations. ArXiv e-prints , 2018. ISSN 0004-6361. doi: 10.1051/0004-6361/201527329. URL http://arxiv.org/abs/1802.05313 .
- Samuel J Gershman. Reinforcement learning and causal models. In Oxford Handbook of Causal Reasoning , pages 1-32. Oxford university press, 2017.
- Alison Gopnik and Laura Schulz. Causal learning: Psychology, philosophy, and computation . Oxford University Press, 2007.
- Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent Independent Mechanisms. ArXiv e-prints , pages 1-33, 2019. URL http://arxiv. org/abs/1909.10893 .
- Jonathan Ho and Stefano Ermon. Generative Adversarial Imitation Learning. ArXiv e-prints , 2016.
- Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation , 9(8):1-32, 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735.
- Lydia M. Hopper. 'Ghost' experiments and the dissection of social learning in humans and animals. Biological Reviews , 85(4):685-701, 2010. ISSN 14647931. doi: 10.1111/j.1469-185X.2010.00120.x.
- Lydia M Hopper, Susan P Lambeth, Steven J Schapiro, and Andrew Whiten. Observational learning in chimpanzees and children studied through ' ghost ' conditions. Proceedings of the Royal Society B , 275 (January):835-840, 2008. doi: 10.1098/rspb.2007.1542.
- Sarah A Jelbert, Alex H Taylor, Lucy G Cheke, Nicola S Clayton, and Russell D Gray. Using the Aesop' s Fable Paradigm to Investigate Causal Understanding of Water Displacement by New Caledonian Crows. PLoS ONE , 9(3), 2014. doi: 10.1371/journal.pone.0092895.
- Guohao Li, Matthias Mueller, Vincent Casser, Neil Smith, Dominik L. Michels, and Bernard Ghanem. Teaching UAVs to Race With Observational Imitation Learning. ArXiv e-prints , 2018. URL http://arxiv.org/ abs/1803.01129 .
- Yuxuan Liu, Abhishek Gupta, Pieter Abbeel, Sergey Levine, and Computer Science. Imitation from Observation : Learning to Imitate Behaviors from Raw Video via Context Translation. ArXiv e-prints , 2018.
- Andrew N Meltzoff, Anna Waismeyer, and Alison Gopnik. Learning about causes from people: observational causal learning in 24-month-old infants. Dev Psychol. , 48(5):1215-1228, 2012. doi: 10.1037/a0027440. Learning.
- Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, Timothy P. Lillicrap, and David Silver. Asynchronous methods for deep reinforcement learning. ArXiv e-prints , 48:1-28, 2016. ISSN 1938-7228. URL http://arxiv.org/ abs/1602.01783 .
- Suraj Nair, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Causal induction from visual observations for goal directed tasks. ArXiv e-prints , pages 1-13, 2019.
- Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven Exploration by Selfsupervised Prediction. ArXiv e-prints , 2017. ISSN 1938-7228. doi: 10.1109/CVPRW.2017.70. URL http://arxiv.org/abs/1705.05363 .
- Judea Pearl. Causality: models, reasoning and inference . Cambridge Univ Press, 2000.
- J Peters, D Janzing, and B Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms . MIT Press, Cambridge, MA, USA, 2017.
- Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , pages 1-42, 2016. URL http://arxiv.org/abs/1501.01332 .
- Yoshihide Sawada, Luca Rigazio, Koji Morikawa, Masahiro Iwasaki, and Yoshua Bengio. Disentangling Controllable and Uncontrollable Factors by Interacting with the World. Advances in Neural Information Processing Systems , 2018.
- Christian Schloegl and Julia Fischer. Causal Reasoning in Non-Human Animals. In Oxford Handbook of Causal Reasoning , pages 1-33. Oxford University Press, 2017. ISBN 9780199399550. doi: 10.1093/oxfordhb/ 9780199399550.013.36.
- Bernhard Schölkopf. Causality for Machine Learning. ArXiv e-prints , pages 1-20, 2019.
- Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction . The MIT Press, 2017. ISBN 0262193981. doi: 10.1016/S1364-6613(99)01331-5. URL http://linkinghub.elsevier.com/ retrieve/pii/S1364661399013315 .
- Alex H Taylor, Gavin R Hunt, C Holzhaider, and Russell D Gray. Spontaneous Metatool Use by New Caledonian Crows. Current Biology , 17:1504-1507, 2007. doi: 10.1016/j.cub.2007.07.057.
- Alex H Taylor, Lucy G Cheke, Anna Waismeyer, Andrew N Meltzoff, Rachael Miller, Alison Gopnik, Nicola S Clayton, Russell D Gray, and Alex H Taylor. Of babies and birds : complex tool behaviours are not sufficient for the evolution of the ability to create a novel causal intervention. Proceedings of the Royal Society B , 281, 2014.
- Alex H Taylor, Lucy G Cheke, Anna Waismeyer, Andrew Meltzoff, Rachael Miller, Alison Gopnik, Nicola S Clayton, Russell D Gray, Clayton Ns, Gray Rd, and Alex H Taylor. No conclusive evidence that corvids can create novel causal interventions. Proceedings of the Royal Society B , 282:6-8, 2015. doi: 10.1098/rspb.2014. 2504.
- Claudio Tennie, Kathrin Greve, and Heinz Gretscher. Two-year-old children copy more reliably and more often than nonhuman great apes in multiple observational learning tasks. Primates , 51:337-351, 2010. doi: 10.1007/s10329-010-0208-4.
- Valentin Thomas, Jules Pondard, Emmanuel Bengio, Marc Sarfati, Philippe Beaudoin, Marie-Jean Meurs, Joelle Pineau, Doina Precup, and Yoshua Bengio. Independently Controllable Factors. ArXiv e-prints , pages 1-13, 2017. URL http://arxiv.org/abs/1708.01289 .
- M Tomasello and J Call. Primate Cognition . Primate Cognition. Oxford University Press, 1997. ISBN 9780195106244. URL https://books.google.ca/books?id=bSYdl2ExJrEC .
- Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral Cloning from Observation. IJCAI , 2018. URL http://arxiv.org/abs/1805.01954 .
- Faraz Torabi, Garrett Warnell, and Peter Stone. Generative Adversarial Imitation from Observation. ArXiv e-prints , 2019a.
- Faraz Torabi, Garrett Warnell, and Peter Stone. Recent Advances in Imitation Learning from Observation. Joint Conference on Artificial Intelligence , (August), 2019b.
- Christoph J Völter, Inés Sentís, and Josep Call. Great apes and children infer causal relations from patterns of variation and covariation. Cognition , 155:30-43, 2016. doi: 10.1016/j.cognition.2016.06.009.
- Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. ArXiv e-prints , pages 1-17, 2016. URL http://arxiv.org/abs/1611.05763 .
- James Woodward. Making things happen: A theory of causal explanation . Oxford university press, 2003.
- James F. Woodward. Agency and Interventionist Theories. In The Oxford Handbook of Causation , number May 2017, pages 1-30. Oxford University Press, 2010. ISBN 9780191577246. doi: 10.1093/oxfordhb/ 9780199279739.003.0012.
- Jim Woodward. Causation with a Human Face, 2007.
- Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation Learning from Imperfect Demonstration. ArXiv e-prints , pages 1-25, 2018. URL https://www. basketball-reference.com/leagues/NBA{\_}stats.html .
- Junzhe Zhang and Elias Bareinboim. Markov Decision Processes with Unobserved Confounders: A Causal Approach. Technical Report R-23, Purdue AI Lab , 2016.
- Junzhe Zhang and Elias Bareinboim. Transfer learning in multi-armed bandits: A Causal approach. IJCAI International Joint Conference on Artificial Intelligence , pages 1340-1346, 2017. ISSN 10450823.
- Konrad Zolna, Scott Reed, Alexander Novikov, Sergio Gomez Colmenarej, David Budden, Serkan Cabi, Misha Denil, Nando de Freitas, and Ziyu Wang. Task-Relevant Adversarial Imitation Learning. Advances in neural information processing - Deep Reinforcement Learning Workshop , pages 1-16, 2019. URL http://arxiv.org/abs/1910.01077 .
- Konrad Zołna, Negar Rostamzadeh, Yoshua Bengio, Sungjin Ahn, and Pedro Pinheiro. Reinforced Imitation in Heterogeneous Action Space. ArXiv e-prints , pages 1-17, 2019.
## A PARAMETERS FOR ENVIRONMENTS
## A.1 SIMPLE ENVIRONMENT PARAMETERS
Trial lasts N = 20 steps. The probability of a given trial having the chain structure is 0.5. Spontaneous rates of activation for s i : p 1 = 0 . 1 , p 2 = 0 . 01 , p 3 = 0 . 01 . For the off-policy settings with the 'interventional' variables ( z i ), these are activated with probabilities p int 2 = 0 . 1 , p int 3 = 0 . 1 . Only the second and third variables are perturbed - a perturbation of the parent node s 1 t will not help discriminate between the two Models.
## A.2 VISUAL ENVIRONMENT PARAMETERS
The underlying dynamics have the same parameters as the basic environments.
## A.3 AGENT-CENTRIC ENVIRONMENT PARAMETERS
The parameters of the environment are as follows. Observation and action phase lengths: N O = 20 , N A = 10 . Reward administered if successful: R d = 10 . Door open for T = 5 time steps.