1802.07569v4

## Continual Lifelong Learning with Neural Networks: A Review German I. Parisi 1 , Ronald Kemker 2 , Jose L. Part 3 , Christopher Kanan 2 , Stefan Wermter 1 1 Knowledge Technology, Department of Informatics, Universit¨ at Hamburg, Germany 2 Chester F. Carlson Center for Imaging Science, Rochester Institute of Technology, NY, USA 3 Department of Computer Science, Heriot-Watt University, Edinburgh Centre for Robotics, Scotland, UK Abstract: Humans and animals have the ability to continually acquire, fine-tune, and transfer knowledge and skills throughout their lifespan. This ability, referred to as lifelong learning, is mediated by a rich set of neurocognitive mechanisms that together contribute to the development and specialization of our sensorimotor skills as well as to long-term memory consolidation and retrieval. Consequently, lifelong learning capabilities are crucial for computational systems and autonomous agents interacting in the real world and processing continuous streams of information. However, lifelong learning remains a long-standing challenge for machine learning and neural network models since the continual acquisition of incrementally available information from non-stationary data distributions generally leads to catastrophic forgetting or interference. This limitation represents a major drawback for state-of-the-art deep neural network models that typically learn representations from stationary batches of training data, thus without accounting for situations in which information becomes incrementally available over time. In this review, we critically summarize the main challenges linked to lifelong learning for artificial learning systems and compare existing neural network approaches that alleviate, to different extents, catastrophic forgetting. Although significant advances have been made in domain-specific learning with neural networks, extensive research efforts are required for the development of robust lifelong learning on autonomous agents and robots. We discuss well-established and emerging research motivated by lifelong learning factors in biological systems such as structural plasticity, memory replay, curriculum and transfer learning, intrinsic motivation, and multisensory integration. Keywords: Continual learning, lifelong learning, catastrophic forgetting, memory consolidation ## 1 Introduction Computational systems operating in the real world are exposed to continuous streams of information and thus are required to learn and remember multiple tasks from dynamic data distributions. For instance, an autonomous agent interacting with the environment is required to learn from its own experiences and must be capable of progressively acquiring, fine-tuning, and transferring knowledge over long time spans. The ability to continually learn over time by accommodating new knowledge while retaining previously learned experiences is referred to as continual or lifelong learning . Such a continuous learning task has represented a long-standing challenge for machine learning and neural networks and, consequently, for the development of artificial intelligence (AI) systems (Hassabis et al. 2017, Thrun & Mitchell 1995). The main issue of computational models regarding lifelong learning is that they are prone to catastrophic forgetting or catastrophic interference , i.e., training a model with new information interferes with previously learned knowledge (McClelland et al. 1995, McCloskey & Cohen 1989). This phenomenon typically leads to an abrupt performance decrease or, in the worst case, to the old knowledge being completely overwritten by the new one. Current deep neural network learning models excel at a number of classification tasks by relying on a large batch of (partially) annotated training samples (see Guo et al. (2016), LeCun et al. (2015) for reviews). However, such a learning scheme assumes that all samples are available during the training phase and, therefore, requires the retraining of the network parameters on the entire dataset in order to adapt to changes in the data distribution. When trained on sequential tasks, the performance of conventional neural network models significantly decreases on previously learned tasks as new tasks are learned (Kemker et al. 2018, Maltoni & Lomonaco 2018). Although retraining from scratch pragmatically addresses catastrophic forgetting, this methodology is very inefficient and hinders the learning of novel data in real time. For instance, in scenarios of developmental learning where autonomous agents learn by actively interacting with the environment, there may be no distinction between training and test phases, requiring the learning model to concurrently adapt and timely trigger behavioural responses (Cangelosi & Schlesinger 2015, Tani 2016). For overcoming catastrophic forgetting, learning systems must, on the one hand, show the ability to acquire new knowledge and refine existing knowledge on the basis of the continuous input and, on the other hand, prevent the novel input from significantly interfering with existing knowledge. The extent to which a system must be plastic in order to integrate novel information and stable in order not to catastrophically interfere with consolidated knowledge is known as the stability-plasticity dilemma and has been widely studied in both biological systems and computational models (Ditzler et al. 2015, Mermillod et al. 2013, Grossberg 1980, 2012). Due to the very challenging but highimpact aspects of lifelong learning, a large body of computational approaches have been proposed that take inspiration from the biological factors of learning from the mammalian brain. Humans and other animals excel at learning in a lifelong manner, making the appropriate decisions on the basis of sensorimotor contingencies learned throughout their lifespan (Tani 2016, Bremner et al. 2012). The ability to incrementally acquire, refine, and transfer knowledge over sustained periods of time is mediated by a rich set of neurophysiological processing principles that together contribute to the early development and experience-driven specialization of perceptual and motor skills (Zenke, Gerstner & Ganguli 2017, Power & Schlaggar 2016, Murray et al. 2016, Lewkowicz 2014). In Section 2, we introduce a set of widely studied biological aspects of lifelong learning and their implications for the modelling of biologically motivated neural network architectures. First, we focus on the mechanisms of neurosynaptic plasticity that regulate the stability-plasticity balance in multiple brain areas (Sec. 2.2 and 2.3). Plasticity is an essential feature of the brain for neural malleability at the level of cells and circuits (see Power & Schlaggar (2016) a survey). For a stable continuous lifelong process, two types of plasticity are required: (i) Hebbian plasticity (Hebb 1949) for positive feedback instability, and (ii) compensatory homeostatic plasticity which stabilizes neural activity. It has been observed experimentally that specialized mechanisms protect knowledge about previously learned tasks from interference encountered during the learning of novel tasks by decreasing rates of synaptic plasticity (Cichon & Gan 2015). Together, Hebbian learning and homeostatic plasticity stabilize neural circuits to shape optimal patterns of experience-driven connectivity, integration, and functionality (Zenke, Gerstner & Ganguli 2017, Abraham & Robins 2005). Importantly, the brain must carry out two complementary tasks: generalize across experiences and retain specific memories of episodic-like events. In Section 2.4, we summarize the complementary learning systems (CLS) theory (McClelland et al. 1995, Kumaran et al. 2016) which holds the means for effectively extracting the statistical structure of perceived events (generalization) while retaining episodic memories, i.e., the collection of experiences at a particular time and place. The CLS theory defines the complementary contribution of the hippocampus and the neocortex in learning and memory, suggesting that there are specialized mechanisms in the human cognitive system for protecting consolidated knowledge. The hippocampal system exhibits short-term adaptation and allows for the rapid learning of new information which will, in turn, be transferred and integrated into the neocortical system for its long-term storage. The neocortex is characterized by a slow learning rate and is responsible for learning generalities. However, additional studies in learning tasks with human subjects (Mareschal et al. 2007, Pallier et al. 2003) observed that, under certain circumstances, catastrophic forgetting may still occur (see Sec. 2.4). Studies on the neurophysiological aspects of lifelong learning have inspired a wide range of machine learning and neural network approaches. In Section 3, we introduce and compare computational approaches that address catastrophic forgetting. We focus on recent learning models that i) regulate intrinsic levels of synaptic plasticity to protect consolidated knowledge (Sec. 3.2); ii) allocate additional neural resources to learn new information (Sec. 3.3), and iii) use complementary learning systems for memory consolidation and experience replay (Sec. 3.4). The vast majority of these approaches are designed to address lifelong supervised learning on annotated datasets of finite size (e.g., Zenke, Poole & Ganguli (2017), Kirkpatrick et al. (2017) and do not naturally extend to more complex scenarios such as the processing of partially unlabelled sequences. Unsupervised lifelong learning, on the other hand, has been proposed mostly through the use of self-organizing neural networks (e.g., Parisi, Tani, Weber & Wermter (2018, 2017), Richardson & Thomas (2008)). Although significant advances have been made in the design of learning methods with structural regularization or dynamic architectural update, considerably less attention has been given to the rigorous evaluation of these algorithms in lifelong and incremental learning tasks. Therefore, in Sec. 3.5 we discuss the importance of using and designing quantitative metrics to measure catastrophic forgetting with large-scale datasets. Lifelong learning has recently received increasing attention due to its implications in autonomous learning agents and robots. Neural network approaches are typically designed to incrementally adapt to modality-specific, often synthetic, data samples collected in controlled environments, shown in isolation and random order. This differs significantly from the more ecological conditions humans and other animals are exposed to throughout their lifespan (Cangelosi & Schlesinger 2015, Krueger & Dayan 2009, Wermter et al. 2005, Skinner 1958). Agents operating in the real world must deal with sensory uncertainty, efficiently process continuous streams of multisensory information, and effectively learn multiple tasks without catastrophically interfering with previously learned knowledge. Intuitively, there is a huge gap between the above-mentioned neural network models and more sophisticated lifelong learning agents expected to incrementally learn from their continuous sensorimotor experiences. Humans can easily acquire new skills and transfer knowledge across domains and tasks (Barnett & Ceci 2002) while artificial systems are still in their infancy regarding what is referred to as transfer learning (Weiss et al. 2016). Furthermore, and in contrast with the predominant tendency to train neural network approaches with uni-sensory (e.g., visual or auditory) information, the brain benefits significantly from the integration of multisensory information, providing the means for an efficient interaction also in situations of sensory uncertainty (Stein et al. 2014, Bremner et al. 2012, Spence 2010). The multisensory aspects of early development and sensorimotor specialization in the brain have inspired a large body of research on autonomous embodied agents (Lewkowicz 2014, Cangelosi & Schlesinger 2015). In Section 4, we review computational approaches motivated by biological aspects of learning which include critical developmental stages and curriculum learning (Sec. 4.2), transfer learning for the reuse of knowledge during the learning of new tasks (Sec. 4.3), reinforcement learning for the autonomous exploration of the environment driven by intrinsic motivation and self-supervision (Sec. 4.4), and multisensory systems for crossmodal lifelong learning (Sec. 4.5). This review complements previous surveys on catastrophic forgetting in connectionist models (French 1999, Goodfellow et al. 2013, Soltoggio et al. 2017) that do not critically compare recent experimental work (e.g., deep learning) or define clear guidelines on how to train and evaluate lifelong approaches on the basis of experimentally observed developmental mechanisms. Together, our and previous reviews highlight lifelong learning as a highly interdisciplinary challenge. Although the individual disciplines may have more open questions than answers, the combination of these findings may provide a breakthrough with respect to current ad-hoc approaches, with neural networks being the stepping stone towards the increasingly sophisticated cognitive abilities exhibited by AI systems. In Section 5, we summarize the key ideas presented in this review and provide a set of ongoing and future research directions. ## 2 Biological Aspects of Lifelong Learning ## 2.1 The Stability-Plasticity Dilemma As humans, we have an astonishing ability to adapt by effectively acquiring knowledge and skills, refining them on the basis of novel experiences, and transferring them across multiple domains (Bremner et al. 2012, Calvert et al. 2004, Barnett & Ceci 2002). While it is true that we tend to gradually forget previously learned information throughout our lifespan, only rarely does the learning of novel information catastrophically interfere with consolidated knowledge (French 1999). For instance, the human somatosensory cortex can assimilate new information during motor learning tasks without disrupting the stability of previously acquired motor skills (Braun et al. 2001). Lifelong learning in the brain is mediated by a rich set of neurophysiological principles that regulate the stability-plasticity balance of the different brain areas and that contribute to the development and specialization of our cognitive system on the basis of our sensorimotor experiences (Zenke, Gerstner & Ganguli 2017, Power & Schlaggar 2016, Murray et al. 2016, Lewkowicz 2014). The stabilityplasticity dilemma regards the extent to which a system must be prone to integrate and adapt to new knowledge and, importantly, how this adaptation process should be compensated by internal mechanisms that stabilize and modulate neural activity to prevent catastrophic forgetting (Ditzler et al. 2015, Mermillod et al. 2013) Neurosynaptic plasticity is an essential feature of the brain yielding physical changes in the neural structure and allowing us to learn, remember, and adapt to dynamic environments (see Power & Schlaggar (2016) for a survey). The brain is particularly plastic during critical periods of early development in which neural networks acquire their overarching structure driven by sensorimotor experiences. Plasticity becomes less prominent as the biological system stabilizes through a well-specified set of developmental stages, preserving a certain degree of plasticity for its adaptation and reorganization at smaller scales (Hensch et al. 1998, Quadrato et al. 2014, Kiyota 2017). The specific profiles of plasticity during critical and post-developmental periods vary across biological systems (Uylings 2006), showing a consistent tendency to decreasing levels of plasticity with increasing age (Hensch 2004). Plasticity plays a crucial role in the emergence of sensorimotor behaviour by complementing genetic information which provides a specific evolutionary path (Grossberg 2012). Genes or molecular gradients drive the initial development for granting a rudimentary level of performance from the start whereas extrinsic factors such as sensory experience complete this process for achieving higher structural complexity and performance (Hirsch & Spinelli 1970, Shatz 1996, Sur & Leamey 2001). In this review, we focus on the developmental and learning aspects of brain organization while we refer the reader to Soltoggio et al. (2017) for a review of evolutionary imprinting. ## 2.2 Hebbian Plasticity and Stability The ability of the brain to adapt to changes in its environment provides vital insight into how connectivity and function of the cortex are shaped. It has been shown that while rudimentary patterns of connectivity in the visual system are established in early development, normal visual input is required for the correct development of the visual cortex. The seminal work of Hubel & Wiesel (1967) on the emergence of ocular dominance showed the importance of timing of experience on the development of normal patterns of cortical organization. The visual experience of newborn kittens was experimentally manipulated to study the effects of varied input on brain organization. The disruption of cortical organization was more severe when the deprivation of visual input began prior to ten weeks of age while no changes were observed in adult animals. Additional experiments showed that neural patterns of cortical organization can be driven by external environmental factors at least for a period early in development (Hubel & Wiesel 1962, 1970, Hubel et al. 1977). The most well-known theory describing the mechanisms of synaptic plasticity for the adaptation of neurons to external stimuli was first proposed by Hebb (1949), postulating that when one neuron drives the activity of another neuron, the connection between them is strengthened. More specifically, the Hebb's rule states that the repeated and persistent stimulation of the postsynaptic cell from the presynaptic cell leads to an increased synaptic efficacy. Throughout the process of development, neural systems stabilize to shape optimal functional patterns of neural connectivity. The simplest form of Hebbian plasticity considers a synaptic strength w which is updated by the product of a pre-synaptic activity x and the post-synaptic activity y : $$\Delta w = x \cdot y \cdot \eta , ( 1 )$$ where η is a given learning rate. However, Hebbian plasticity alone is unstable and leads to runaway neural activity, thus requiring compensatory mechanisms to stabilize the learning process (Abbott & Nelson 2000, Bienenstock et al. 1982). Stability in Hebbian systems is typically achieved by augmenting Hebbian plasticity with additional constraints such as upper limits on the individual synaptic weights or average neural activity (Miller & MacKay 1994, Song et al. 2000). Homeostatic mechanisms of plasticity include synaptic scaling and meta-plasticity which directly affect synaptic strengths (Davis 2006, Turrigiano 2011). Without loss of generality, homeostatic plasticity can be viewed as a modulatory effect or feedback control signal that regulates the unstable dynamics of Hebbian plasticity (see Fig. 1.a). The feedback controller directly affects synaptic strength on the basis of the observed neural activity and must be fast in relation to the timescale of the unstable system (Astr¨ om & Murray 2010). In its simplest form, modulated Hebbian plasticity can be modelled Figure 1: Schematic view of two aspects of neurosynaptic adaptation: a) Hebbian learning with homeostatic plasticity as a compensatory mechanism that uses observations to compute a feedback control signal (Adapted with permission from Zenke, Gerstner & Ganguli (2017)). b) The complementary learning systems (CLS) theory (McClelland et al. 1995) comprising the hippocampus for the fast learning of episodic information and the neocortex for the slow learning of structured knowledge. <details> <summary>Image 1 Details</summary>  ### Visual Description \n ## Diagram: Neural Learning Systems ### Overview The image presents two diagrams illustrating different theories of neural learning: a) Hebbian and Homeostatic Plasticity, and b) Complementary Learning Systems (CLS) theory. Both diagrams depict simplified models of neural processes and information flow. The diagrams are positioned side-by-side, labeled 'a' and 'b' respectively. ### Components/Axes **Diagram a: Hebbian and Homeostatic Plasticity** * **Labels:** "Hebbian and Homeostatic Plasticity", "Controller", "System", "Synaptic strength", "Plasticity", "Neural activity", "External stimuli", "Control signal", "Observations". * **Flow:** Arrows indicate the flow of information and influence between components. **Diagram b: Complementary Learning Systems (CLS) theory** * **Labels:** "Complementary Learning Systems (CLS) theory", "Hippocampus", "Neocortex", "Episodic Memory", "Generalization", "Fast learning of arbitrary information", "Slow learning of structured knowledge", "Storage, retrieval, replay". * **Flow:** A bidirectional arrow labeled "Storage, retrieval, replay" connects the Hippocampus and Neocortex. ### Detailed Analysis or Content Details **Diagram a: Hebbian and Homeostatic Plasticity** * A rectangular box labeled "System" dominates the diagram. * Within the "System" box, two components are stacked vertically: "Synaptic strength" (top) and "Neural activity" (bottom). * A bidirectional arrow labeled "Plasticity" connects "Synaptic strength" and "Neural activity", indicating a reciprocal relationship. * An arrow labeled "Control signal" originates from a box labeled "Controller" (top-left) and points to the "System" box. * An arrow labeled "Observations" originates from the "System" box and points to the "Controller" box. * An arrow labeled "External stimuli" points towards the bottom of the "System" box. **Diagram b: Complementary Learning Systems (CLS) theory** * Two rectangular boxes represent the "Hippocampus" (left) and "Neocortex" (right). * Within the "Hippocampus" box, "Episodic Memory" is labeled. Below it is the text "Fast learning of arbitrary information". * Within the "Neocortex" box, "Generalization" is labeled. Below it is the text "Slow learning of structured knowledge". * A bidirectional arrow labeled "Storage, retrieval, replay" connects the "Hippocampus" and "Neocortex" boxes. ### Key Observations * Diagram a illustrates a closed-loop system with feedback between a controller and the core neural system. * Diagram b highlights a division of labor between the hippocampus (fast, arbitrary learning) and the neocortex (slow, structured learning). * Both diagrams emphasize the dynamic nature of neural processes (plasticity, storage/retrieval). ### Interpretation Diagram a represents a model where a controller modulates a neural system based on observations, and the system's plasticity adjusts synaptic strength and neural activity in response to external stimuli. This suggests a self-regulating system that adapts to its environment. Diagram b proposes that the hippocampus and neocortex work in a complementary fashion. The hippocampus rapidly encodes new experiences (episodic memory), while the neocortex gradually extracts and generalizes patterns from these experiences. The "Storage, retrieval, replay" process suggests that memories are consolidated and transferred between these two brain regions. The two diagrams, taken together, offer different perspectives on neural learning. Diagram a focuses on the mechanisms of adaptation within a single system, while diagram b focuses on the division of labor between different brain regions. They are not mutually exclusive; the plasticity mechanisms described in diagram a could be operating within both the hippocampus and the neocortex, as described in diagram b. </details> by introducing an additional modulatory signal m to Eq. 1 such that the synaptic update is given by $$\Delta w = m \cdot x \cdot y \cdot \eta ,$$ Modulatory feedback in Hebbian neural networks has received increasing attention, with different approaches proposing biologically plausible learning through modulatory loops (Grant et al. 2017, Soltoggio et al. 2017). For a critical review of the temporal aspects of Hebbian and homeostatic plasticity, we refer the reader to Zenke, Gerstner & Ganguli (2017). Evidence on cortical function has shown that neural activity in multiple brain areas results from the combination of bottom-up sensory drive, top-down feedback, and prior knowledge and expectations (Heeger 2017). In this setting, complex neurodynamic behaviour can emerge from the dense interaction of hierarchically arranged neural circuits in a self-organized manner (Tani 2016). Input-driven self-organization plays a crucial role in the brain Nelson (2000), with topographic maps being a common feature of the cortex for processing sensory input (Willshaw & von der Malsburg 1976). Different models of neural self-organization have been proposed that resemble the dynamics of basic biological findings on Hebbian-like learning and plasticity (Kohonen 1982, Martinetz et al. 1993, Fritzke 1992, Marsland et al. 2002), demonstrating that neural map organization results from unsupervised, statistical learning with nonlinear approximations of the input distribution. To stabilize the unsupervised learning process, neural network self-organization can be complemented with top-down feedback such as task-relevant signals that modulate the intrinsic map plasticity (Parisi, Tani, Weber & Wermter 2018, Soltoggio et al. 2017). In a hierarchical processing regime, neural detectors have increasingly large spatio-temporal receptive fields to encode information over larger spatial and temporal scales (Taylor et al. 2015, Hasson et al. 2008). Thus, higher-level layers can provide the top-down context for modulating the bottom-up sensory drive in lower-level layers. For instance, bottom-up processing is responsible for encoding the co-occurrence statistics of the environment while error-driven signals modulate this feedforward process according to top-down, task-specific factors (Murray et al. 2016). Together, these models contribute to a better understanding of the underlying neural mechanisms for the development of hierarchical cortical organization. ## 2.3 The Complementary Learning Systems The brain learns and memorizes. The former task is characterized by the extraction of the statistical structure of the perceived events with the aim to generalize to novel situations. The latter, conversely, requires the collection of separated episodic-like events. Consequently, the brain must comprise a mechanism to concurrently generalize across experiences while retaining episodic memories. Sophisticated cognitive functions rely on canonical neural circuits replicated across multiple brain areas (Douglas et al. 1995). However, although there are shared structural properties, different brain areas operate at multiple timescales and learning rates, thus differing significantly from each other in a functional way (Benna & Fusi 2016, Fusi et al. 2005). A prominent example is the complementary contribution of the neocortex and the hippocampus in learning and memory consolidation (McClelland et al. 1995, O'Reilly 2002, 2004). The complementary learning systems (CLS) theory (McClelland et al. 1995) holds that the hippocampal system exhibits short-term adaptation and allows for the rapid learning of novel information which will, in turn, be played back over time to the neocortical system for its long-term retention (see Fig. 1.b). More specifically, the hippocampus employs a rapid learning rate and encodes sparse representations of events to minimize interference. Conversely, the neocortex is characterized by a slow learning rate and builds overlapping representations of the learned knowledge. Therefore, the interplay of hippocampal and neocortical functionality is crucial to concurrently learn regularities (statistics of the environment) and specifics (episodic memories). Both brain areas are known to learn via Hebbian and error-driven mechanisms (O'Reilly & Rudy 2000). In the neocortex, feedback signals will yield task-relevant representations while, in the case of the hippocampus, error-driven modulation can switch its functionally between pattern discrimination and completion for recalling information (O'Reilly 2004). Studies show that adult neurogenesis contributes to the formation of new memories (Altman 1963, Eriksson et al. 1998, Cameron et al. 1993, Gage 2000). It has been debated whether human adults grow significant amounts of new neurons. Recent research has suggested that hippocampal neurogenesis drops sharply in children to undetectable levels in adulthood (Sorrells et al. 2018). On the other hand, other studies suggest that hippocampal neurogenesis sustains human-specific cognitive function throughout life (Boldrini et al. 2018). During neurogenesis, the hippocampus' dentate gyrus uses new neural units to quickly assimilate and immediately recall new information (Altman 1963, Eriksson et al. 1998). During initial memory formation, the new neural progenitor cells exhibit high levels of plasticity; and as time progresses, the plasticity decreases to make the new memory more stable (Deng et al. 2010). In addition to neurogenesis, neurophysiological studies evidence the contribution of synaptic rewiring by structural plasticity on memory formation in adults (Knoblauch et al. 2014, Knoblauch 2017), with a major role of structural plasticity in increasing information storage efficiency in terms of space and energy demands. While the hippocampus is normally associated with the immediate recall of recent memories (i.e., short-term memories), the prefrontal cortex (PFC) is usually associated with the preservation and recall of remote memories (i.e., long-term memories; Bontempi et al. (1999)). Kitamura et al. (2017) showed that, when the brain learns something new, the hippocampus and PFC are both initially encoded with the corresponding memory; however, the hippocampus is primarily responsible for the recent recall of new information. Over time, they showed that the corresponding memory is consolidated over to PFC, which will then take over responsibility for recall of the (now) remote memory. It is believed that the consolidation of recent memories into long-term storage occurs during rapid eye movement (REM) sleep (Taupin & Gage 2002, Gais et al. 2007). Recently, the CLS theory was updated to incorporate additional findings from neuroscience (Kumaran et al. 2016). The first set of findings regards the role of the replaying of memories stored in the hippocampus as a mechanism that, in addition to the integration of new information, also supports the goal-oriented manipulation of experience statistics (O'Neill et al. 2010). The hippocampus rapidly encodes episodic-like events that can be reactivated during sleep or unconscious and conscious memory recall (Gelbard-Sagiv et al. 2008), thus consolidating information in the neocortex via the reactivation of encoded experiences in terms of multiple internally generated replays (Ratcliff 1990). Furthermore, evidence suggests that (i) the hippocampus supports additional forms of generalization through the recurrent interaction of episodic memories (Kumaran & McClelland 2012) and (ii) if the new information is consistent with existing knowledge, then its integration into the neocortex is faster than originally suggested (Tse et al. 2011). Overall, the CLS theory holds the means for effectively generalizing across experiences while retaining specific memories in a lifelong manner. However, the exact neural mechanisms remain poorly understood. ## 2.4 Learning without Forgetting The neuroscience findings described in Sec. 2.3 demonstrate the existence of specialized neurocognitive mechanisms for acquiring and protecting knowledge. Nevertheless, it has been observed that catastrophic forgetting may occur under specific circumstances. For instance, Mareschal et al. (2007) found an asymmetric interference effect in a sequential category learning task with 3- and 4-month-old infants. The infants had to learn two categories, dog and cat , from a series of pictures and would have to later distinguish a novel animal in a subsequent preferential looking task. Surprisingly, it was observed that infants were able to retain the category dog only if it was learned before cat . This asymmetric effect is thought to reflect the relative similarity of the two categories in terms of perceptual structure. Additional interference effects were observed for long-term knowledge. Pallier et al. (2003) studied the word recognition abilities of Korean-born adults whose language environment shifted completely from Korean to French after being adopted between the ages of 3 and 8 by French families. Behavioural tests showed that subjects had no residual knowledge of the previously learned Korean vocabulary. Functional brain imaging data showed that the response of these subjects while listening to Korean was no different from the response while listening to other foreign languages that they had been exposed to, suggesting that their previous knowledge of Korean was completely overwritten. Interestingly, brain activations showed that Korean-born subjects produced weaker responses to French with respect to native French speakers. It was hypothesized that, while the adopted subjects did not show strong responses to transient exposures to the Korean vocabulary, prior knowledge of Korean may have had an impact during the formulation of language skills to facilitate the reacquisition of the Korean language should the individuals be re-exposed to it in an immersive way. Humans do not typically exhibit strong events of catastrophic forgetting because the kind of experiences we are exposed to are very often interleaved (Seidenberg & Zevin 2006). Nevertheless, forgetting effects may be observed when new experiences are strongly immersive such as in the case of children drastically shifting from Korean to French. Together, these findings reveal a wellregulated balance in which, on the one hand, consolidated knowledge must be protected to ensure its long-term durability and avoid catastrophic interference during the learning of novel tasks and skills over long periods of time. On the other hand, under certain circumstances such as immersive long-term experiences, old knowledge can be overwritten in favour of the acquisition and refinement of new knowledge. Taken together, the biological aspects of lifelong learning summarized in this section provide insights into how artificial agents could prevent catastrophic forgetting and model graceful forgetting. In the next sections, we describe and compare an extensive set of neural network models and AI approaches that have taken inspiration from such principles. In the case of computational systems, however, additional challenges must be faced due to the limitations of learning in restricted scenarios that typically capture very few components of the processing richness of biological systems. ## 3 Lifelong Learning and Catastrophic Forgetting in Neural Networks ## 3.1 Lifelong Machine Learning Lifelong learning represents a long-standing challenge for machine learning and neural network systems (Hassabis et al. 2017, French 1999). This is due to the tendency of learning models to catastrophically forget existing knowledge when learning from novel observations (Thrun & Mitchell 1995). A lifelong learning system is defined as an adaptive algorithm capable of learning from a continuous stream of information, with such information becoming progressively available over time and where the number of tasks to be learned (e.g., membership classes in a classification task) are not predefined. Critically, the accommodation of new information should occur without catastrophic forgetting or interference. In connectionist models, catastrophic forgetting occurs when the new instances to be learned differ significantly from previously observed examples because this causes the new information to overwrite previously learned knowledge in the shared representational resources in the neural network (French 1999, McCloskey & Cohen 1989). When learning offline, this loss of knowledge can be recovered because the agent sees the same pseudo-randomly shuffled examples over and over, but this is not possible when the data cannot be shuffled and is observed as a continuous stream. The effects of catastrophic forgetting have been widely studied for over two decades, especially in networks learned using back-propagation (Ratcliff 1990, Lewandowsky & Li 1994) and in the Hopfield networks (Nadal et al. 1986, Burgess et al. 1991). Early attempts to mitigate catastrophic forgetting typically consisted of memory systems that store previous data and that regularly replay old samples interleaved with samples drawn from the new data (Robins 1993, 1995), and these methods are still used today (Gepperth & Karaoguz 2015, Rebuffi et al. 2016). However, a general drawback of memory-based systems is that they require explicit storage of old information, leading to large working memory requirements. Furthermore, in the case of a fixed amount of neural resources, specialized mechanisms should be designed that protect consolidated knowledge from being overwritten by the learning of novel information (e.g., Zenke, Poole & Ganguli (2017), Kirkpatrick et al. (2017)). Intuitively, catastrophic forgetting can be strongly alleviated by allocating additional neural resources whenever they are required (e.g., Parisi, Tani, Weber & Wermter (2018, 2017), Rusu et al. (2016), Hertz et al. (1991)). This approach, however, may lead to scalability issues with significantly increased computational efforts for neural architectures that become very large. Conversely, since in a lifelong learning scenario the number of tasks and samples per task cannot be known a priori, it is non-trivial to predefine a sufficient amount of neural resources that will prevent catastrophic forgetting without strong assumptions on the distribution of the input. In this setting, three key aspects have been identified for avoiding catastrophic forgetting in connectionist models (Richardson & Thomas 2008): (i) allocating additional neural resources for new knowledge; (ii) using non-overlapping representations if resources are fixed; and (iii) interleaving the old knowledge as the new information is represented. The brain has evolved mechanisms of neurosynaptic plasticity and complex neurocognitive functions that process continuous streams of information in response to both short- and long-term changes in the environment (Zenke, Gerstner & Ganguli 2017, Power & Schlaggar 2016, Murray et al. 2016, Lewkowicz 2014). Consequently, the differences between biological and artificial systems go beyond architectural differences, and also include the way in which these artificial systems are exposed to external stimuli. Since birth, humans are immersed in a highly dynamic world and, in response to this rich perceptual experience, our neurocognitive functions progressively develop to make sense of increasingly more complex events. Infants start with relatively limited capabilities for processing low-level features and incrementally develop towards the learning of higher-level perceptual, cognitive, and behavioural functions. Humans make massive use of the spatio-temporal relations and increasingly richer high-order associations of the sensory input to learn and trigger meaningful behavioural responses. Conversely, artificial systems are typically trained in batches, exposing the learning algorithm to multiple iterations of the same training samples in a (pseudo-)random order. After a fixed number of training epochs, it is expected that the learning algorithm has tuned its internal representations and can predict novel samples that follow a similar distribution with respect to the training dataset. Clearly, this approach can be effective (and this is supported by the state-of-the-art performance of deep learning architectures for visual classification tasks; see Guo et al. (2016), LeCun et al. (2015) for reviews), but it does not reflect the characteristics of lifelong learning tasks. In the next sections, we introduce and compare different neural network approaches for lifelong learning that mitigate, to different extents, catastrophic forgetting. Conceptually, these approaches can be divided into methods that retrain the whole network while regularizing to prevent catastrophic forgetting with previously learned tasks (Fig. 2.a; Sec. 3.2), methods that selectively train the network and expand it if necessary to represent new tasks (Fig. 2.b,c; Sec. 3.3), and methods that model complementary learning systems for memory consolidation, e.g. by using memory replay to consolidate internal representations (Sec. 3.4). Since considerably less attention has been given to the rigorous evaluation of these algorithms in lifelong learning tasks, in Sec. 3.5 we highlight the importance of using and designing new metrics to measure catastrophic forgetting with large-scale datasets. ## 3.2 Regularization Approaches Regularization approaches alleviate catastrophic forgetting by imposing constraints on the update of the neural weights. Such approaches are typically inspired by theoretical neuroscience models suggesting that consolidated knowledge can be protected from forgetting through synapses with a cascade of states yielding different levels of plasticity (Benna & Fusi 2016, Fusi et al. 2005). From a computational perspective, this is generally modelled via additional regularization terms that penalize changes in the mapping function of a neural network. Figure 2: Schematic view of neural network approaches for lifelong learning: a) retraining while regularizing to prevent catastrophic forgetting with previously learned tasks, b) unchanged parameters with network extension for representing new tasks, and c) selective retraining with possible expansion. <details> <summary>Image 2 Details</summary>  ### Visual Description \n ## Diagram: Neural Network Training Strategies ### Overview The image presents a comparative diagram illustrating three different strategies for training neural networks over time. Each strategy is represented by a separate neural network diagram, labeled (a), (b), and (c). All diagrams depict a recurrent neural network structure with input `x(t-1)` at time `t-1` and output `x(t)` at time `t`. The diagrams visually demonstrate how the network connections and structure change during retraining and expansion. ### Components/Axes Each diagram consists of the following components: * **Input Layer:** Represented by `x(t-1)` on the left. * **Hidden Layers:** Multiple layers of interconnected nodes. The number of nodes in each layer varies between the diagrams. * **Output Layer:** Represented by `x(t)` on the right. * **Connections:** Green lines connecting nodes between layers, representing the flow of information. * **Labels:** Each diagram is labeled with a descriptive title: * (a) Retraining with regularization * (b) Training with network expansion * (c) Selective network retraining and expansion ### Detailed Analysis or Content Details **Diagram (a): Retraining with Regularization** * The network has three layers: an input layer, a hidden layer with approximately 8 nodes, and an output layer with approximately 4 nodes. * All nodes in the hidden layer are connected to both the input and output layers. * The connections appear to remain constant throughout the retraining process, suggesting that the network's structure is not changing, but the weights are being adjusted (regularization). **Diagram (b): Training with Network Expansion** * The network has three layers: an input layer, two hidden layers (approximately 8 nodes each), and an output layer with approximately 4 nodes. * The first hidden layer is fully connected to the input layer, and the second hidden layer is fully connected to the first hidden layer. The output layer is connected to the second hidden layer. * The network expands by adding nodes to the hidden layers during training. **Diagram (c): Selective Network Retraining and Expansion** * The network has three layers: an input layer, two hidden layers (approximately 8 nodes each), and an output layer with approximately 4 nodes. * The connections are more complex than in diagrams (a) and (b). Some nodes in the hidden layers are connected to the input and output layers, while others are only connected to the adjacent layers. * The network selectively retrains and expands by adding or modifying connections between nodes. Some nodes are highlighted with a lighter shade of green, indicating they are being retrained or expanded. ### Key Observations * Diagram (a) shows a static network structure, implying weight adjustments without structural changes. * Diagram (b) demonstrates a network growing in size by adding layers and nodes. * Diagram (c) illustrates a more nuanced approach, where the network adapts by selectively retraining and expanding specific connections and nodes. * The input `x(t-1)` and output `x(t)` are consistent across all three diagrams, indicating that the input and output remain the same while the training strategy varies. ### Interpretation The diagram illustrates three different approaches to training recurrent neural networks for time-series data. * **Retraining with regularization (a)** is a common technique to prevent overfitting by penalizing complex models. It maintains the network's structure while adjusting its weights. * **Training with network expansion (b)** allows the network to increase its capacity as it learns, potentially capturing more complex patterns in the data. * **Selective network retraining and expansion (c)** offers a more flexible approach, where the network can adapt its structure and connections based on the specific needs of the task. This approach can be more efficient than expanding the entire network, as it focuses on the most relevant parts. The diagrams suggest a progression in complexity and adaptability. Regularization is the simplest approach, while selective retraining and expansion is the most sophisticated. The choice of strategy depends on the specific characteristics of the data and the desired level of performance. The use of `x(t-1)` and `x(t)` indicates that these are recurrent networks, designed to process sequential data where the output at time `t` depends on the input at time `t-1`. </details> Li & Hoiem (2016) proposed the learning without forgetting (LwF) approach composed of convolutional neural networks (CNN) in which the network with predictions of the previously learned tasks is enforced to be similar to the network with the current task by using knowledge distillation, i.e., the transferring of knowledge from a large, highly regularized model into a smaller model (Hinton et al. 2014). According to the LwF algorithm, given a set of shared parameters θ s across all tasks, it optimizes the parameters of the new task θ n together with θ s imposing the additional constraint that the predictions on the samples of the novel task using θ s and the parameters of old tasks θ o do not shift significantly in order to remember θ o . Given the training data on the new task ( X n , Y n ) , the output of old tasks for the new data Y o , and randomly initialized new parameters θ n , the updated parameters θ ∗ s , θ ∗ o , θ ∗ n are given by: $$\begin{array} { r l } & { \theta _ { s } ^ { * } , \theta _ { o } ^ { * } , \theta _ { n } ^ { * } \leftarrow \arg \min _ { \hat { \theta } _ { s } , \hat { \theta } _ { o } , \hat { \theta } _ { n } } \left ( \lambda _ { o } \mathcal { L } _ { o l d } ( Y _ { o } , \hat { Y } _ { o } ) + \mathcal { L } _ { n e w } ( Y _ { n } , \hat { Y } _ { n } ) + \mathcal { R } ( \hat { \theta } _ { s } , \hat { \theta } _ { o } , \hat { \theta } _ { n } ) \right ) , \quad ( 3 ) } \end{array}$$ where L old ( Y o , ˆ Y o ) and L new ( Y n , ˆ Y n ) minimize the difference between the predicted values ˆ Y and the ground-truth values Y of the new and old tasks respectively using ˆ θ s , ˆ θ o , ˆ θ n , λ o is used to balance new/old tasks, and R is a regularization term to prevent overfitting. However, this approach has the drawbacks of highly depending on the relevance of the tasks and that the training time for one task linearly increases with the number of learned tasks. Additionally, while distillation provides a potential solution to multi-task learning, it requires a reservoir of persistent data for each learned task. Jung et al. (2018) proposed to regularize the l 2 distance between the final hidden activations, preserving the previously learned input-output mappings by computing additional activations with the parameters of the old tasks. These approaches, however, are computationally expensive since they require to compute the old tasks' parameters for each novel data sample. Other approaches opt to either completely prevent the update of weights trained on old tasks (Razavian et al. 2014) or to reduce the learning rate in order to prevent significant changes in the network parameters while training with new data (Donahue et al. 2014). Kirkpatrick et al. (2017) proposed the elastic weight consolidation (EWC) model in supervised and reinforcement learning scenarios. The approach consists of a quadratic penalty on the difference between the parameters for the old and the new tasks that slows down the learning for task-relevant weights coding for previously learned knowledge. The relevance of the parameter θ with respect to a task's training data D is modelled as the posterior distribution p ( θ | D ) . Assuming a scenario with two independent tasks A with D A and B with D B , the log value of the posterior probability given by the Bayes' rule is: $$\log p ( \theta | \mathcal { D } ) = \log p ( \mathcal { D } _ { B } | \theta ) + \log p ( \theta | \mathcal { D } _ { A } ) - \log p ( \mathcal { D } _ { B } ) ,$$ where the posterior probability log p ( θ | D A ) embeds all the information about the previous task. However, since this term is intractable, EWC approximates it as a Gaussian distribution with mean given by the parameters θ ∗ A and a diagonal precision given by the diagonal of the fisher information matrix F . Therefore, the loss function of EWC is given by $$\mathcal { L } ( \theta ) = \mathcal { L } _ { B } ( \theta ) + \sum _ { i } \frac { \lambda } { 2 } F _ { i } ( \theta _ { i } - \theta _ { A , i } ^ { * } ) ^ { 2 } ,$$ where L B is the loss of B , λ sets the relevance of the old tasks with respect to the new one, and i denotes the indexes of the parameters. Therefore, this approach requires a diagonal weighting over the parameters of the learned tasks which is proportional to the diagonal of the Fisher information metric, with the synaptic importance being computed offline and limiting its computational application to low-dimensional output spaces. Furthermore, additional experiments by Kemker et al. (2018) have shown that, although EWC outperforms other methods for permutation tasks, it is not capable of learning new categories incrementally. Zenke, Poole & Ganguli (2017) proposed to alleviate catastrophic forgetting by allowing individual synapses to estimate their importance for solving a learned task. Similar to Kirkpatrick et al. (2017), this approach penalizes changes to the most relevant synapses so that new tasks can be learned with minimal forgetting. To reduce large changes in important parameters θ k when learning a new task, the authors use a modified cost function L ∗ n with a surrogate loss which approximates the summed loss functions of all previous tasks L ∗ o : $$\mathcal { L } _ { n } ^ { * } = \mathcal { L } _ { n } + c \sum _ { k } \Omega _ { k } ^ { n } ( \theta _ { k } ^ { * } - \theta _ { k } ) ^ { 2 } , & & ( 6 )$$ where c is a weighting parameter to balance new and old tasks, θ ∗ k are the parameters at the end of the previous task, and Ω n k is a per-parameter regulation strength. Similar to EWC by Kirkpatrick et al. (2017), this approach pulls back the more influential parameters towards a reference weight with good performance on previous tasks. In this case, however, synaptic relevance is computed in an online fashion over the entire learning trajectory in the parameter space. The two approaches have shown similar results on the Permuted MNIST benchmark (LeCun et al. 1998). Maltoni & Lomonaco (2018) proposed the AR1 model for single-incremental-task scenarios which combines architectural and regularization strategies. Regularization approaches tend to progressively reduce the magnitude of weight changes batch by batch, with most of the changes occurring in the top layers. Instead, in AR1 intermediate layers weights are adapted without negative impact in terms of forgetting. Reported results on CORe50 (Lomonaco & Maltoni 2017) and iCIFAR100 (Krizhevsky 2009) show that AR1 allows the training of deep convolutional models with less forgetting, outperforming LwF, EWC, and SI. Ensemble methods have been proposed to alleviate catastrophic forgetting by training multiple classifiers and combine them to generate a prediction. Early attempts showed a disadvantage linked to the intense use of storage memory which scales up with the number of sessions (Polikar et al. 2001, Dai et al. 2007), while more recent approaches restrict the size of the models through multiple strategies. For instance, Ren et al. (2017) proposed to adaptively adjust to the changing data distribution by combining sub-models after a new training phase, learning new tasks without referring to previous training data. Coop et al. (2013) introduced a multi-layer perceptron (MLP) augmented with a fixed expansion layer (FEL) which embeds a sparsely encoding hidden layer to mitigate the interference of previously learned representations. Ensembles of FEL networks were used to control levels of plasticity, yielding incremental learning capabilities while requiring minimal storage memory. Fernando et al. (2017) proposed an ensemble method in which a genetic algorithm is used to find the optimal path through a neural network of fixed size for replication and mutation. This approach, referred to as PathNet, uses agents embedded in a neural network to discover which parts of the network can be reused for the learning of new tasks while freezing task-relevant paths for avoiding catastrophic forgetting. PathNet's authors showed that incrementally learning new tasks sped up the training of subsequently learned supervised and reinforcement learning tasks; however, they did not measure performance on the original task to determine if catastrophic forgetting occurred. In addition, PathNet requires an independent output layer for each new task, which prevents it from learning new classes incrementally (Kemker et al. 2018). In summary, regularization approaches provide a way to alleviate catastrophic forgetting under certain conditions. However, they comprise additional loss terms for protecting consolidated knowledge which, with a limited amount of neural resources, may lead to a trade-off on the performance of old and novel tasks. ## 3.3 Dynamic Architectures The approaches introduced here change architectural properties in response to new information by dynamically accommodating novel neural resources, e.g., re-training with an increased number of neurons or network layers. For instance, Rusu et al. (2016) proposed to block any changes to the network trained on previous knowledge and expand the architecture by allocating novel sub-networks with fixed capacity to be trained with the new information. This approach, referred to as progressive networks , retains a pool of pre-trained models (one for each learned task T n ). Given N existing tasks, when a new task is T N +1 is given, a new neural network is created and the lateral connections with the existing tasks are learned. To avoid catastrophic forgetting, the learned parameters θ n for existing tasks T n are left unchanged while the new parameter set θ N +1 is learned for T N +1 . Experiments reported good results on a wide variety of reinforcement learning tasks, outperforming common baseline approaches that either pre-train or incrementally fine-tune the models by incorporating prior knowledge only at initialization. Intuitively, this approach prevents catastrophic forgetting but leads the complexity of the architecture to grow with the number of learned tasks. Zhou et al. (2012) proposed the incremental training of a denoising autoencoder that adds neurons for samples with high loss and subsequently merges these neurons with existing ones to prevent redundancy. More specifically, the algorithm is composed of two processes for (i) adding new features to minimize the residual of the objective function and (ii) merging similar features to obtain a compact feature representation and in this way prevent overfitting. This model was shown to outperform non-incremental denoising autoencoders in classification tasks with the MNIST (LeCun et al. 1998) and the CIFAR-10 (Krizhevsky 2009) datasets. Cortes et al. (2016) proposed to adapt both the structure of the network and its weights by balancing the model complexity and empirical risk minimization. In contrast to enforcing a pre-defined architecture, the algorithm learns the required model complexity in an adaptive fashion. The authors reported good results on several binary classification tasks extracted from the CIFAR-10 dataset. In contrast to previously introduced approaches that do not consider multi-task scenarios, Xiao et al. (2014) proposed a training algorithm with a network that incrementally grows in capacity and also in hierarchical fashion. Classes are grouped according to their similarity and self-organized into multiple levels, with models inheriting features from existing ones to speed up the learning. In this case, however, only the topmost layers can grow and the vanilla back-propagation training procedure is inefficient. Draelos et al. (2017) incrementally trained an autoencoder on new MNIST digits using the reconstruction error to show whether the older digits were retained. Their neurogenesis deep learning (NDL) model adds new neural units to the autoencoder to facilitate the addition of new MNIST digits, and it uses intrinsic replay (a generative model used for pseudo-rehearsal) to preserve the weights required to retain older information. Yoon et al. (2018) took this concept to the supervised learning paradigm and proposed a dynamically expanding network (DEN) that increases the number of trainable parameters to incrementally learn new tasks. DEN is trained in an online manner by performing selective retraining which expands the network capacity using group sparse regularization to decide how many neurons to add at each layer. Part & Lemon (2016, 2017) proposed the combination of a pre-trained CNN with a self-organizing incremental neural network (SOINN) in order to take advantage of the good representational power of CNNs and, at the same time, allow the classification network to grow according to the task requirements in a continuous object recognition scenario. An issue that arises from these types of approaches is scalability since the classification network grows with the number of classes that have been learned. Another problem that was identified through this approach is that by relying on fixed representations, e.g., pre-trained CNNs, the discrimination power will be conditioned by the dataset used to train the feature extractor. Rebuffi et al. (2016) deal with this problem by storing example data points that are used along with new data to dynamically adapt the weights of the feature extractor, a technique that is referred to as rehearsal. By combining new and old data, they prevent catastrophic forgetting but at the expense of a higher memory footprint. So far, we have considered approaches designed for (or at least strictly evaluated on) the classification of static images. However, in more natural learning scenarios, sequential input underlying spatio-temporal relations such as in the case of videos must be accounted for. Parisi, Tani, Weber & Wermter (2017) showed that lifelong learning of human action sequences can be achieved in terms of prediction-driven neural dynamics with internal representations emerging in a hierarchy of recurrent self-organizing networks. The self-organizing networks can dynamically allocate neural resources and update connectivity patterns according to competitive Hebbian learning. Each neuron of the neural map consists of a weight vector w j and a number K of context descriptors c k,j with w j , c k,j ∈ R n . As a result, recurrent neurons in the map will encode prototype sequence-selective snapshots of the input. Given a set of recurrent neurons, N , the best-matching unit (BMU) w b with respect to the input x ( t ) ∈ R n is computed as: $$b = \arg \min _ { j \in N } \left ( \alpha _ { 0 } \| x ( t ) - w _ { j } \| ^ { 2 } + \sum _ { k = 1 } ^ { K } \, \alpha _ { k } \| C _ { k } ( t ) - c _ { j , k } \| ^ { 2 } \right ) , \quad \ \ ( 7 )$$ where { α i } i =0 ...K are constant values that modulate the influence of the current input with respect to previous neural activity and C k ( t ) ∈ R n is the global context of the network. Each neuron is equipped with a habituation counter h i expressing how frequently it has fired based on a simplified model of how the efficacy of a habituating synapse reduces over time. The network is initialized with two neurons and, at each learning iteration, it inserts a new neuron whenever the activity of the network of a habituated neuron is smaller than a given threshold. The neural update rule is given by: $$\Delta { w } _ { i } = \epsilon _ { i } \cdot h _ { i } \cdot ( { x } ( t ) - { w } _ { i } ) ,$$ where i is a constant learning rate and h i acts as a modulatory factor (see Eq. 2) that decreases the magnitude of learning over time to protect consolidated knowledge. This approach has shown competitive results with batch learning methods on the Weizmann (Gorelick et al. 2005) and the KTH (Schuldt et al. 2004) action benchmark datasets. Furthermore, it learns robust action-label mappings also in the case of occasionally missing or corrupted class labels. Parisi, Ji & Wermter (2018) showed that self-organizing networks with additive neurogenesis show a better performance than a static network with the same number of neurons, thereby providing insights into the design of neural architectures in incremental learning scenarios when the total number of neurons is fixed. Similar GWR-based approaches have been proposed for the incremental learning of body motion patterns (Mici et al. 2017, Elfaramawy et al. 2017, Parisi et al. 2016) and human-object interaction (Mici et al. 2018). However, these unsupervised learning approaches do not take into account top-down task-relevant signals that can regulate the stability-plasticity balance, potentially leading to scalability issues for large-scale datasets. To address this issue, task-relevant modulatory signals were modelled by Parisi, Tani, Weber & Wermter (2018) which regulate the process of neurogenesis and neural update (see Sec. 3.4). This model shares a number of conceptual similarities with the adaptive resonance theory (ART; see Grossberg (2012) for a review) in which neurons are iteratively adapted to a non-stationary input distribution in an unsupervised fashion and new neurons can be created in correspondence of dissimilar input data. In the ART model, learning occurs through the interaction of top-down and bottom-up processes: top-down expectations act as memory templates (or prototypes) which are compared to bottom-up sensory observations. Similar to the GWR's activation threshold, the ART model uses a vigilance parameter to produce fine-grained or more general memories. Despite its inherent ability to mitigate catastrophic forgetting during incremental learning, an extensive evaluation with recent lifelong learning benchmarks has not been reported for continual learning tasks. However, it has been noted that the results of some variants of the ART model depend significantly upon the order in which the training data are processed. While the mechanisms for creating new neurons and connections in the GWR do not resemble biologically plausible mechanisms (e.g., Eriksson et al. (1998), Ming & Song (2011), Knoblauch (2017)), the GWR learning algorithm represents an efficient computational model that incrementally adapts to non-stationary input. Crucially, the GWR model creates new neurons whenever they are required and only after the training of existing ones. The neural update rate decreases as the neurons become more habituated, which has the effect of preventing that noisy input interferes with consolidated neural representations. Alternative theories suggest that an additional function of hippocampal neurogenesis is the encoding of time for the formation of temporal associations in memory (Aimone et al. 2006, 2009), e.g., in terms of temporal clusters of long-term episodic memories. Although the underlying mechanisms of neurogenesis and structural plasticity remain to be further investigated in biological systems, these results reinforce that growing neural models with plasticity constitute effective mitigation of catastrophic forgetting in non-stationary environments. ## 3.4 Complementary Learning Systems and Memory Replay The CLS theory (McClelland et al. 1995, Kumaran et al. 2016) provides the basis for a computational framework modelling memory consolidation and retrieval in which the complementary tasks of memorization and generalization are mediated by the interplay of the mammalian hippocampus and neocortex (see Sec. 2.3). Importantly, the interplay of an episodic memory (specific experience) and a semantic memory (general structured knowledge) provides important insights into the mechanisms of knowledge consolidation in the absence of sensory input. Dual-memory learning systems have taken inspiration, to different extents, from the CLS theory to address catastrophic forgetting. An early computational example of this concept was proposed by Hinton & Plaut (1987) in which each synaptic connection has two weights: a plastic weight with slow change rate which stores long-term knowledge and a fast-changing weight for temporary knowledge. This dual-weight method reflects the properties of complementary learning systems to mitigate catastrophic forgetting during sequential task learning. French (1997) developed a pseudorecurrent dual-memory framework, one for early processing and the other for long-term storage, that used pseudo-rehearsal (Robins 1995) to transfer memories between memory centers. In pseudorehearsal, training samples are not explicitly kept in memory but drawn from a probabilistic model. During the next two decades, numerous neural network approaches based on CLS principles were used to explain and predict results in different learning and memory domains (see O'Reilly (2002) for a review). However, there is no empirical evidence that shows that these approaches can scale up to a large number of tasks or current image and video benchmark datasets (see Sec. 3.5). More recently, Soltoggio (2015) proposed the use of short- and long-term plasticity for consolidating new information on the basis of a cause-effect hypothesis testing when learning with delayed rewards. In this case, the difference between the short- and long-term plasticity is not related to the duration of the memory but rather to the confidence of consistency of cause-effect relationships. This meta-plasticity rule, referred to as hypothesis testing plasticity (HTP), shows that such relationships can be extracted from ambiguous information flows, thus towards explaining the learning in more complex environments (see Sec. 4). Gepperth & Karaoguz (2015) proposed two approaches for incremental learning using (i) a modified self-organizing map (SOM) and (ii) a SOM extended with a short-term memory (STM). We refer to these two approaches as GeppNet and GeppNet+STM respectively. In the case of the GeppNet, taskrelevant feedback from a regression layer is used to select whether learning in the self-organizing hidden layer should occur. In the GeppNet+STM case, the STM is used to store novel knowledge which is occasionally played back to the GeppNet layer during sleep phases interleaved with training phases. This latter approach yielded better performance and faster convergence in incremental learning tasks with the MNIST dataset. However, the STM has a limited capacity, thus learning new knowledge can overwrite old one. In both cases, the learning process is divided into two phases: one for initialization and the other for actual incremental learning. Additional experiments showed that this approach performs significantly worse than EWC (Kirkpatrick et al. 2017) on different permutation tasks (see Sec. 3.5). Both GeppNet and GeppNet+STM require storing the entire training dataset during training. Inspired by the generative role of the hippocampus for the replay of previously encoded experiences, Shin et al. (2017) proposed a dual-model architecture consisting of a deep generative model and a task solver. In this way, training data from previously learned tasks can be sampled in terms of generated pseudo-data and interleaved with information from the new tasks. Thus, it is not necessary to explicitly revise old training samples for experience replay, reducing the requirements of working memory. This approach is conceptually similar to previous ones using a pseudo-rehearsal method, i.e., interleaving information of a new task with internally generated samples from previously learned tasks. Robins (1995) showed that interleaving information of new experiences with internally generated patterns of previous experiences help consolidate existing knowledge without explicitly storing training samples. Pseudo-rehearsal was also used by Draelos et al. (2017) for the incremental training of an autoencoder, using the output statistics of the encoder to generate input for the decoder during the replay. However, similar to most of the above-described approaches, the use of pseudo-rehearsal methods was strictly evaluated on two datasets of relatively low complexity, e.g. the MNIST and the Street View House Number (SVHN) (Netzer et al. 2011). Consequently, the question arises whether this generative approach can scale up to more complex domains. L¨ uders et al. (2016) proposed an evolvable Neural Turing Machine (ENTM) that enables agents to store long-term memories by progressively allocating additional external memory components. The optimal structure for a continually learning network is found from an initially minimal configuration by evolving networks topology and weights. The ENTM configurations can perform one-shot learning of new associations and mitigate the effects of catastrophic forgetting during incremental learning tasks. A set of reported experiments in reinforcement learning tasks showed that the dynamic nature of the ENTM approach will cause the agents to continually expand its memory over time. This can lead to an unnecessary memory expansion that would slow down the learning process significantly. A possible solution to address this issue can be the introduction of cost functions for a more efficient memory allocation and use. Lopez-Paz & Ranzato (2017) proposed the Gradient Episodic Memory (GEM) model that yields positive transfer of knowledge to previous tasks. The main feature of GEM to minimize catastrophic forgetting is an episodic memory used to store a subset of the observed examples from a given task. While minimizing the loss on the current task t , GEM treats the losses on the episodic memories of tasks k < t as inequality constraints, avoiding their increase but allowing their decrease. This method requires considerable more memory than other regularization approaches such as EWC (Kirkpatrick et al. 2017) at training time (with an episodic memory M k for each task k ) but can work much better in the single pass setting. Kemker & Kanan (2018) proposed the FearNet model for incremental class learning that is inspired by studies of recall and consolidation in the mammalian brain during fear conditioning (Kitamura et al. 2017). FearNet uses a hippocampal network capable of immediately recalling new examples, a PFC network for long-term memories, and a third neural network inspired by the basolateral amygdala for determining whether the system should use the PFC or hippocampal network for a particular example. FearNet consolidates information from its hippocampal network to its PFC network during sleep phases. FearNet's PFC model is a generative neural network that creates pseudo-samples that are then intermixed with recently observed examples stored in its hippocampal network. Kamra et al. (2018) presented a similar dual-memory framework that also uses a variational autoencoder as a generative model for pseudo-rehearsal. Their framework generates a short-term memory module for each new task; however, prior to consolidation, predictions are made using an oracle (i.e., they know which module contains the associated memory). Parisi, Tani, Weber & Wermter (2018) proposed a dual-memory self-organizing architecture for learning spatiotemporal representations from videos in a lifelong fashion. The complementary memories are modelled as recurrent self-organizing neural networks: the episodic memory quickly adapts to incoming novel sensory observations via competitive Hebbian Learning, whereas the semantic memory progressively learns compact representations by using task-relevant signals to regulate intrinsic levels of structural plasticity. For the consolidation of knowledge in the absence of sensory input, trajectories of neural reactivations from the episodic memory are periodically replayed to both memories. Reported experiments show that the described method significantly outperforms previously proposed lifelong learning methods in three different incremental learning tasks with the CORe50 benchmark dataset (Lomonaco & Maltoni (2017); see Sec. 3.5). Since the development of the neural maps is unsupervised, this approach can be used in scenarios where the annotations of training samples are sparse. ## 3.5 Benchmarks and Evaluation Metrics Despite the large number of proposed methods addressing lifelong learning, there is no established consensus on benchmark datasets and metrics for their proper evaluation. Typically, a direct comparison of different methods is hindered by the highly heterogeneous and often limited evaluation schemes to assess the overall performance, levels of catastrophic forgetting, and knowledge transfer. Lopez-Paz & Ranzato (2017) defined training and evaluation protocols to assess the quality of continual learning models in terms of their accuracy as well as their ability to transfer knowledge between tasks. The transfer of knowledge can be forwards or backwards . The former refers to the influence that learning a task T A has on the performance of a future task T B , whereas the latter refers to the influence of a current task T B on a previous task T A . The transfer is positive when learning about T A improves the performance of another task T B (forwards or backwards) and negative otherwise. (See Sec. 4.3 for an introduction to learning models addressing transfer learning.) Figure 3: Example images from benchmark datasets used for the evaluation of lifelong learning approaches: a) the MNIST dataset with 10 digit classes (LeCun et al. 1998), b) the Caltech-UCSD Birds-200 (CUB-200) dataset composed of 200 different bird species (Wah et al. 2011), and c) the CORe50 containing 50 objects with variations in background, illumination, blurring, occlusion, pose, and scale (adapted with permission from Lomonaco & Maltoni (2017)). <details> <summary>Image 3 Details</summary>  ### Visual Description \n ## Image Collection: Dataset Examples ### Overview The image presents a collection of examples from three different datasets used in machine learning, specifically for image recognition and classification tasks. The datasets are labeled as MNIST, CUB-200, and CORE50. Each dataset is displayed as a grid of images representing the variety of samples within it. ### Components/Axes The image is divided into three sections, labeled a), b), and c), each representing a different dataset. * **a) MNIST:** Displays a grid of handwritten digits (0-9). * **b) CUB-200:** Displays a grid of bird images. * **c) CORE50:** Displays a grid of images depicting human hands interacting with common objects. There are no explicit axes or legends in the traditional sense, but the labels (MNIST, CUB-200, CORE50) serve as identifiers for each dataset. ### Detailed Analysis or Content Details **a) MNIST:** The MNIST dataset consists of a 7x9 grid of handwritten digits. Each row represents a different instance of a digit from 0 to 9. The digits are grayscale. The grid appears to be a representative sample of the dataset, showing variations in handwriting style. **b) CUB-200:** The CUB-200 dataset displays a grid of approximately 6x8 images of birds. The images show a diverse range of bird species, poses, and backgrounds. The birds are of varying sizes and colors. The images appear to be photographs. **c) CORE50:** The CORE50 dataset displays a grid of approximately 6x6 images of human hands interacting with everyday objects. The objects include items like tennis balls, remote controls, water bottles, and sunglasses. The images show different hand poses and object orientations. The images appear to be photographs. ### Key Observations * **MNIST:** The dataset is relatively simple, consisting of clean, grayscale images of digits. * **CUB-200:** The dataset is more complex, with variations in bird species, lighting, and backgrounds. * **CORE50:** The dataset is the most complex, involving interactions between humans and objects, requiring the model to understand both object recognition and pose estimation. * The datasets vary significantly in their complexity and the types of visual features they require a machine learning model to learn. ### Interpretation The image illustrates the increasing complexity of datasets used in machine learning research. MNIST is a foundational dataset often used for introductory machine learning tasks. CUB-200 represents a more challenging dataset requiring fine-grained image recognition. CORE50 represents a highly complex dataset that requires understanding of object interactions and human pose. The image demonstrates the progression from simple, controlled datasets to more realistic and challenging datasets, reflecting the evolution of machine learning research towards more complex and real-world applications. The datasets are chosen to represent different levels of difficulty in image recognition tasks. MNIST is designed for basic digit recognition, CUB-200 for identifying specific bird species, and CORE50 for understanding human-object interactions. This progression allows researchers to test and improve the capabilities of machine learning models in increasingly complex scenarios. </details> Kemker et al. (2018) suggested a set of guidelines for evaluating lifelong learning approaches and performed complementary experiments that provide a direct quantitative comparison of a number of approaches. Such guidelines comprise the use of three benchmark experiments: (i) data permutation , (ii) incremental class learning , and (iii) multimodal learning . The data permutation experiment consists in training a model with a dataset along with a permuted version of the same dataset, which tests the model's ability to incrementally learn new information with similar feature representations. It is then expected that the model prevents catastrophic forgetting with the original data during the subsequent learning of randomly permuted data samples. In the incremental class learning experiment, the model performance reflects its ability to retain previously learned information while incrementally learning one class at a time. Finally, in the multimodal learning experiment, the same model is sequentially trained with datasets of different modalities, which tests the model's ability to incrementally learn new information with dramatically different feature representations (e.g., first learn an image classification dataset and then learn an audio classification dataset). In contrast to the datasets typically proposed in the literature to evaluate lifelong learning approaches (e.g., MNIST containing 10 digit classes with low-resolution images; Fig. 3.a), the above-mentioned experimental conditions were conducted using the Caltech-UCSD Birds-200 (CUB-200) dataset composed of 200 different bird species (Wah et al. (2011); Fig. 3.b) and the AudioSet dataset, which is built from YouTube videos with 10-second sound clips from 632 classes and over 2 million annotations (Gemmeke et al. 2017). The approaches considered were supervised: a standard MLP trained online as a baseline, the EWC (Kirkpatrick et al. 2017), the PathNet (Fernando et al. 2017), the GeppNet and GeppNet+STM (Gepperth & Karaoguz 2015), and the FEL (Coop et al. 2013). For the data permutation experiment, best results were obtained by PathNet followed by EWC, suggesting that models that use the ensembling and regularization mechanisms will work best at incrementally learning new tasks/datasets with similar feature distributions. In contrast, EWC performed better than PathNet on the multi-modal experiment because EWC does a better job on separating non-redundant (i.e., dissimilar) data. For the incremental learning task, best results were obtained with a combination of rehearsal and dual-memory systems (i.e. GeppNet+STM), yielding gradual adaptation and knowledge consolidation (see Fig.4). However, since rehearsal requires the storage of raw training examples, pseudo-rehearsal may be a better strategy for future work. Lomonaco & Maltoni (2017) proposed the CORe50, a novel dataset for continuous object recognition that includes 50 classes of objects observed from different perspectives and includes variations in background, illumination, blurring, occlusion, pose, and scale (Fig. 3.c). With respect to the above-discussed datasets, CORe50 provides samples collected in experimental conditions closer to what autonomous agents and robots are exposed to in the real world (see Sec. 4). Along with the dataset, the authors propose three incremental learning scenarios: (i) new instances (NI) where all classes are shown in the first batch while subsequent instances of known classes become available over time, new classes (NC) where, for each sequential batch, new object classes are available so that the model must deal with the learning of new classes without forgetting previously learned ones, and <details> <summary>Image 4 Details</summary>  ### Visual Description ## Line Charts: Accuracy vs. Number of Classes Trained ### Overview The image presents three line charts, each depicting the accuracy of different machine learning models as the number of classes trained increases. The charts compare the performance of EWC, FEL, MLP, GeppNet, GeppNet + STM, and Offline Model. Each chart focuses on a different dataset: (a) MNIST, (b) CUB-200, and (c) a third dataset (unspecified). The y-axis represents accuracy (in percentage) for classes trained so far, and the x-axis represents the number of classes trained. ### Components/Axes * **X-axis (all charts):** Number of Classes Trained. Scales vary per chart. * **Y-axis (all charts):** Accuracy (%) for Classes Trained so Far. Scale: 0 to 100%. * **Legends (all charts):** * EWC (Red) * FEL (Orange) * MLP (Yellow) * GeppNet (Green) * GeppNet + STM (Blue) * Offline Model (Magenta) * **Chart Titles:** * (a) MNIST * (b) CUB-200 * (c) [Dataset Name Missing] ### Detailed Analysis or Content Details **Chart (a) - MNIST:** * **EWC (Red):** Starts at approximately 98% accuracy, and rapidly declines to approximately 20% accuracy as the number of classes trained increases from 1 to 10. * **FEL (Orange):** Starts at approximately 98% accuracy, and declines more gradually to approximately 40% accuracy as the number of classes trained increases from 1 to 10. * **MLP (Yellow):** Starts at approximately 98% accuracy, and declines to approximately 60% accuracy as the number of classes trained increases from 1 to 10. * **GeppNet (Green):** Starts at approximately 98% accuracy, and declines to approximately 80% accuracy as the number of classes trained increases from 1 to 10. * **GeppNet + STM (Blue):** Starts at approximately 98% accuracy, and remains relatively stable at approximately 90% accuracy as the number of classes trained increases from 1 to 10. * **Offline Model (Magenta):** Starts at approximately 98% accuracy, and declines to approximately 70% accuracy as the number of classes trained increases from 1 to 10. **Chart (b) - CUB-200:** * **EWC (Red):** Starts at approximately 55% accuracy, and fluctuates between approximately 20% and 40% accuracy as the number of classes trained increases from 100 to 200. * **FEL (Orange):** Starts at approximately 55% accuracy, and declines to approximately 10% accuracy as the number of classes trained increases from 100 to 200. * **MLP (Yellow):** Starts at approximately 55% accuracy, and declines to approximately 30% accuracy as the number of classes trained increases from 100 to 200. * **GeppNet (Green):** Starts at approximately 55% accuracy, and declines to approximately 40% accuracy as the number of classes trained increases from 100 to 200. * **GeppNet + STM (Blue):** Starts at approximately 55% accuracy, and remains relatively stable at approximately 50% accuracy as the number of classes trained increases from 100 to 200. * **Offline Model (Magenta):** Starts at approximately 55% accuracy, and declines to approximately 30% accuracy as the number of classes trained increases from 100 to 200. **Chart (c) - Unspecified Dataset:** * **EWC (Red):** Starts at approximately 55% accuracy, and fluctuates between approximately 10% and 30% accuracy as the number of classes trained increases from 50 to 100. * **FEL (Orange):** Starts at approximately 55% accuracy, and declines to approximately 10% accuracy as the number of classes trained increases from 50 to 100. * **MLP (Yellow):** Starts at approximately 55% accuracy, and declines to approximately 20% accuracy as the number of classes trained increases from 50 to 100. * **GeppNet (Green):** Starts at approximately 55% accuracy, and declines to approximately 40% accuracy as the number of classes trained increases from 50 to 100. * **GeppNet + STM (Blue):** Starts at approximately 55% accuracy, and remains relatively stable at approximately 45% accuracy as the number of classes trained increases from 50 to 100. * **Offline Model (Magenta):** Starts at approximately 55% accuracy, and declines to approximately 30% accuracy as the number of classes trained increases from 50 to 100. ### Key Observations * **GeppNet + STM consistently outperforms other models** across all three datasets, maintaining higher accuracy as the number of classes trained increases. * **EWC and FEL generally exhibit the most significant accuracy decline** as more classes are added, indicating a greater susceptibility to catastrophic forgetting. * **The MNIST dataset shows the highest initial accuracy** for all models, suggesting it is the easiest dataset to learn. * **The CUB-200 and the third dataset show similar accuracy ranges**, indicating comparable difficulty. * **The Offline Model performs better than EWC and FEL**, but generally worse than GeppNet and GeppNet + STM. ### Interpretation These charts demonstrate the performance of different continual learning algorithms in mitigating catastrophic forgetting – the tendency of neural networks to forget previously learned tasks when learning new ones. The "Offline Model" likely represents a standard training approach without continual learning techniques. The superior performance of "GeppNet + STM" suggests that this method is particularly effective at retaining knowledge from previously learned classes while adapting to new ones. The rapid decline in accuracy for "EWC" and "FEL" indicates that these methods struggle to maintain performance as the number of classes increases. The differences in initial accuracy and rate of decline across the datasets suggest that the difficulty of the learning task influences the effectiveness of these algorithms. MNIST, being a simpler dataset, allows all models to achieve higher initial accuracy. The more complex datasets (CUB-200 and the third dataset) result in lower initial accuracy and a more pronounced decline as new classes are added. The fluctuations in the EWC line in the CUB-200 and the third dataset charts could indicate instability or sensitivity to the order in which classes are presented during training. Further investigation would be needed to determine the cause of these fluctuations. </details> (c) AudioSet Figure 4: Results of several lifelong learning approaches for incremental class learning. The meanclass test accuracy evaluated on the MNIST (a), CUB-200 (b), and AudioSet (c) is shown for the following approaches: FEL (red), MLP (yellow), GeppNet (green), GeppNet+STM (blue), EWC (pink), and offline model (dashed line). Adapted with permission from Kemker et al. (2018). (iii) new instances and classes (NIC) where both new classes and instances are presented in each training batch. According to the reported results, EWC (Kirkpatrick et al. 2017) and LwF (Li & Hoiem 2016) perform significantly worse in NC and NIC than in NI. Perhaps not surprisingly, overall performance generally drops when using datasets of higher complexity such as CUB-200 and CORe50 than when tested on the MNIST. Such results indicate that lifelong learning is a very challenging task and, importantly, that the performance of most approaches can differ significantly according to the specific learning strategy. This suggests that while there is a large number of approaches capable of alleviating catastrophic forgetting in highly controlled experimental conditions, lifelong learning has not been tackled for more complex scenarios. Therefore, additional research efforts are required to develop robust and flexible approaches subject to more exhaustive, benchmark evaluation schemes. ## 4 Developmental Approaches and Autonomous Agents ## 4.1 Towards Autonomous Agents Humans have the extraordinary ability to learn and progressively fine-tune their sensorimotor skills in a lifelong manner (Tani 2016, Bremner et al. 2012, Calvert et al. 2004). Since the moment of birth, humans are immersed in a highly dynamic crossmodal environment which provides a wealth of experiences for shaping perception, cognition, and behaviour (Murray et al. 2016, Lewkowicz 2014). A crucial component of lifelong learning in infants is their spontaneous capacity of autonomously generating goals and exploring their environment driven by intrinsic motivation (Cangelosi & Schlesinger 2015, Gopnik et al. 1999). Consequently, the ability to learn new tasks and skills autonomously through intrinsically motivated exploration is one of the main factors that differentiate biological lifelong learning from current continual neural networks models of classification. While there has been significant progress in the development of models addressing incremental learning tasks (see Sec. 3), such models are designed to alleviate catastrophic forgetting from a set of annotated data samples. Typically, the complexity of the datasets used for the evaluation of lifelong learning tasks is very limited and does not reflect the richness and level of uncertainty of the stimuli that artificial agents can be exposed to in the real world. Furthermore, neural models are often trained with data samples shown in isolation or presented in a random order. This significantly differs from the highly organized manner in which humans and animals efficiently learn from samples presented in a meaningful order for the shaping of increasingly complex concepts and skills (Krueger & Dayan 2009, Skinner 1958). Therefore, learning in a lifelong manner goes beyond the incremental accumulation of domain-specific knowledge, enabling to transfer generalized knowledge and skills across multiple tasks and domains (Barnett & Ceci 2002) and, importantly, benefiting from the interplay of multisensory information for the development and specialization of complex neurocognitive functions (Murray et al. 2016, Tani 2016, Lewkowicz 2014). Intuitively, it is unrealistic to provide an artificial agent with all the necessary prior knowledge to effectively operate in real-world conditions (Thrun & Mitchell 1995). Consequently, artificial agents must exhibit a richer set of learning capabilities enabling them to interact in complex environments with the aim to process and make sense of continuous streams of (often uncertain) information (Hassabis et al. 2017, Wermter et al. 2005). In the last decade, significant advances have been made to embed biological aspects of lifelong learning into neural network models. In this section, we summarize well-established and emerging neural network approaches driven by interdisciplinary research introducing findings from neuroscience, psychology, and cognitive sciences for the development of lifelong learning autonomous agents. We focus on discussing models of critical developmental stages and curriculum learning (Sec. 4.2), transfer learning for the reuse of consolidated knowledge during the acquisition of new tasks (Sec. 4.3), autonomous exploration and choice of goals driven by curiosity and intrinsic motivation (Sec. 4.4), and the crossmodal aspects of lifelong learning for multisensory systems and embodied agents (Sec. 4.5). In particular, we discuss on how these components (see Fig. 5) can be used (independently or combined) to improve current approaches addressing lifelong learning. ## 4.2 Developmental and Curriculum Learning Learning and development interact in a very intricate way (Elman 1993). Humans show an exceptional capacity to learn throughout their lifespan and, with respect to other species, exhibit the lengthiest developmental process for reaching maturity. There is a limited time window in development in which infants are particularly sensitive to the effects of their experiences. This period is commonly referred to as the sensitive or critical period of development (Lenneberg 1967) in which early experiences are particularly influential, sometimes with irreversible effects in behaviour (Senghas et al. 2004). During these critical periods, the brain is particularly plastic (Fig. 5.a) and neural networks acquire their overarching structure driven by sensorimotor experiences (see Power & Schlaggar (2016) for a survey). Afterwards, plasticity becomes less prominent and the system stabilizes, preserving a certain degree of plasticity for its subsequent adaptation and reorganisation at smaller scales (Hensch et al. 1998, Quadrato et al. 2014, Kiyota 2017). The basic mechanisms of critical learning periods have been studied in connectionist models (Thomas & Johnson 2006, Richardson & Thomas 2008), in particular with the use of selforganizing learning systems which reduce the levels of functional plasticity through a two-phase training of the topographic neural map (Kohonen 1982, 1995, Miikkulainen 1997). In a first organization phase, the neural map is trained with a high learning rate and large spatial neighbourhood size, allowing the network to reach an initial rough topological organization. In a second tuning phase, the learning rate and the neighbourhood size are iteratively reduced for fine-tuning. Implementations of this kind have been used to develop models of early visual development (Miller et al. 1989), Figure 5: Schematic view of the main components for the development of autonomous agents able to learn over long periods of time in complex environments: Developmental and curriculum learning (Sec. 4.2), transfer learning (Sec. 4.3), curiosity and intrinsic motivation (Sec. 4.4), and crossmodal learning (Sec. 4.5). <details> <summary>Image 5 Details</summary>  ### Visual Description \n ## Diagram: Learning Paradigms ### Overview The image presents four diagrams illustrating different learning paradigms: Developmental & Curriculum Learning, Multi-Task Transfer Learning, Curiosity and Intrinsic Motivation, and Crossmodal Learning. Each diagram uses simplified graphical representations to convey the core concepts of each paradigm. The diagrams are arranged in a 2x2 grid. ### Components/Axes Each sub-diagram has its own labels. * **a) Developmental & Curriculum Learning:** X-axis labeled "Time". Y-axis is not explicitly labeled, but represents a scale for both "Task complexity" and "Plasticity". * **b) Multi-Task Transfer Learning:** X-axis labeled "Time". * **c) Curiosity and Intrinsic Motivation:** No explicit axes. * **d) Crossmodal Learning:** No explicit axes. ### Detailed Analysis or Content Details **a) Developmental & Curriculum Learning:** * **Task complexity:** Represented by a blue line. The line starts at a low value at the left (approximately 0 at Time=0) and increases monotonically to a higher value at the right (approximately 1.2 at Time=1). The trend is upward. * **Plasticity:** Represented by a red line. The line starts at a high value at the left (approximately 1.0 at Time=0) and decreases monotonically to a lower value at the right (approximately 0.2 at Time=1). The trend is downward. **b) Multi-Task Transfer Learning:** * **Task A:** A gray rectangle labeled "Task A" positioned on the left side of the diagram. * **Task B:** A gray rectangle labeled "Task B" positioned on the right side of the diagram. * **Forward transfer:** A purple arrow originating from Task A and pointing towards Task B. * **Backward transfer:** A purple arrow originating from Task B and pointing towards Task A. **c) Curiosity and Intrinsic Motivation:** * **Environment:** A gray rectangle labeled "Environment" positioned at the top of the diagram. * **Agent:** A gray rectangle labeled "Agent" positioned below the "Environment". * **Strategy / action selection:** A blue rectangle positioned within the "Agent" rectangle. * **External reward:** A red arrow originating from the "Environment" and pointing towards the "Strategy / action selection". * **Intrinsic motivation:** A green rectangle positioned below the "Agent" rectangle. * **Internal reward:** A green arrow originating from the "Intrinsic motivation" and pointing towards the "Strategy / action selection". **d) Crossmodal Learning:** * **Modality A:** A stack of light orange rectangles labeled "Modality A" positioned on the left side of the diagram. * **Modality B:** A stack of light blue rectangles labeled "Modality B" positioned on the right side of the diagram. * **Enhancement:** Light orange and light blue rectangles stacked on top of "Modality A" and "Modality B" respectively. * **Integration:** A stack of light orange and light blue rectangles positioned above the "Enhancement" rectangles, labeled "Integration". ### Key Observations * Diagram (a) shows an inverse relationship between task complexity and plasticity over time. * Diagram (b) illustrates bidirectional transfer of knowledge between two tasks. * Diagram (c) depicts the interplay between external rewards from the environment and internal motivation within an agent. * Diagram (d) demonstrates how information from two modalities can be enhanced and integrated. ### Interpretation The image provides a high-level overview of four distinct learning paradigms. * **Developmental & Curriculum Learning (a)** suggests that learning progresses from simpler to more complex tasks, accompanied by a decrease in the brain's plasticity. * **Multi-Task Transfer Learning (b)** highlights the potential for leveraging knowledge gained from one task to improve performance on another, and vice versa. * **Curiosity and Intrinsic Motivation (c)** emphasizes the role of internal drives in learning, alongside external rewards. * **Crossmodal Learning (d)** illustrates how combining information from different sensory modalities can lead to richer and more robust representations. The diagrams are conceptual and do not provide specific numerical data. They serve as visual aids to understand the core principles of each learning paradigm. The use of arrows and color-coding effectively conveys the direction of information flow and the relationships between different components. The diagrams are interconnected in that they all represent different approaches to improving learning systems, whether artificial or biological. </details> language acquisition (Lambon Ralph & Ehsan 2006, Li et al. 2004), and recovery from brain injuries (Marchman 1993). Recent studies on critical periods in deep neural networks showed that the initial rapid learning phase plays a key role in defining the final performance of the networks (Achille et al. 2017). The first few epochs of training are critical for the allocation of resources across different layers dictated by the initial input distribution. After such a critical period, the initially allocated neural resources can be re-distributed through additional learning phases. Developmental learning strategies have been experimented on with embedded agents to regulate the embodied interaction with the environment in real time (Cangelosi & Schlesinger 2015, Tani 2016). In contrast to computational models that are fed with batches of information, developmental agents acquire an increasingly complex set of skills based on their sensorimotor experiences in an autonomous manner. Consequently, staged development becomes essential for bootstrapping cognitive skills with less amount of tutoring experience. However, the use of developmental strategies for artificial learning systems has shown to be a very complex practice. In particular, it is difficult to select a well-defined set of developmental stages that favours the overall learning performance in highly dynamic environments. For instance, in the predictive coding framework (Adams et al. 2015, Rao & Ballard 1999), the intention towards a goal can be generated through the prediction of the consequence of an action by means of the error regression with the prediction error. The use of generative models, which are implicit in predictive coding, is one component embedded in the framework of active inference (Friston et al. 2015). Active inference models aim to understand how to select the data that best discloses its causes in dynamic and uncertain environments through the bilateral use of action and perception. Nevertheless, it remains unclear how to systematically define developmental stages on the basis of the interaction between innate structure, embodiment, and (active) inference. Humans and animals exhibit better learning performance when examples are organized in a meaningful way, e.g., by making the learning tasks gradually more difficult (Krueger & Dayan 2009). Following this observation, referred to as curriculum learning , Elman (1993) showed that having a curriculum of progressively harder tasks (Fig. 5.a) leads to faster training performance in neural network systems. This has inspired similar approaches in robotics (Sanger 1994) and more recent machine learning methods studying the effects of curriculum learning in the performance of learning (Bengio et al. 2009, Reed & de Freitas 2015, Graves et al. 2016). Experiments on datasets of limited complexity (such as MNIST) showed that curriculum learning acts as unsupervised pretraining, leading to improved generalization and faster speed of convergence of the training process towards the global minimum. However, the effectiveness of curriculum learning is highly sensitive with respect to the modality of progression through the tasks. Furthermore, this approach assumes that tasks can be ordered by a single axis of difficulty. Graves et al. (2017) proposed to treat the task selection problem as a stochastic policy over the tasks that maximizes the learning progress, leading to an improved efficiency in curriculum learning. In this case, it is necessary to introduce additional factors such as intrinsic motivation (Oudeyer et al. 2007, Barto 2013), where indicators of learning progress are used as reward signals to encourage exploration (see Sec. 4.4). Curriculum strategies can be seen as a special case of transfer learning (Weiss et al. 2016), where the knowledge collected during the initial tasks is used to guide the learning process of more sophisticated ones. ## 4.3 Transfer Learning Transfer learning refers to applying previously acquired knowledge in one domain to solve a problem in a novel domain (Barnett & Ceci 2002, Pan & Yang 2010, Holyoak & Thagard 1997). In this context, forward transfer refers to the influence that learning a task T A has on the performance of a future task T B , whereas backward transfer refers to the influence of a current task T B on a previous task T A (Fig. 5.b). For this reason, transfer learning represents a significantly valuable feature of artificial systems for inferring general laws from (a limited amount of) particular samples, assuming the simultaneous availability of multiple learning tasks with the aim to improve the performance at one specific task. Transfer learning has remained an open challenge in machine learning and autonomous agents (see Weiss et al. (2016) for a survey). Specific neural mechanisms in the brain mediating the high-level transfer learning are poorly understood, although it has been argued that the transfer of abstract knowledge may be achieved through the use of conceptual representations that encode relational information invariant to individuals, objects, or scene elements (Doumas et al. 2008). Zero-shot learning (Lampert et al. 2009, Palatucci et al. 2009) and one-shot learning (Fei-Fei et al. 2003, Vinyals et al. 2016) aim at performing well on novel tasks but do not prevent catastrophic forgetting on previously learned tasks. An early attempt to realize lifelong learning through transfer learning was proposed by Ring (1997) through the use of a hierarchical neural network that solves increasingly complex reinforcement learning tasks by incrementally adding neural units and encode a wider temporal context in which actions take place. More recent deep learning approaches have attempted to tackle lifelong transfer learning in a variety of domains. Rusu et al. (2017) proposed the use of progressive neural networks (Rusu et al. 2016) to transfer learned low-level features and high-level policies from a simulated to a real environment. The task consists of learning pixel-to-action reinforcement learning policies with sparse rewards from raw visual input to a physical robot manipulator. Tessler et al. (2017) introduced a hierarchical deep reinforcement learning network that uses an array of skills and skill distillation to reuse and transfer knowledge between tasks. The approach was evaluated by teaching an agent to solve tasks in the Minecraft video game. However, skill networks need to be pre-trained and cannot be learned along with the overarching architecture in an end-to-end fashion. Lopez-Paz & Ranzato (2017) proposed the Gradient Episodic Memory (GEM) model that alleviates catastrophic forgetting and performs positive transfer to previously learned tasks. The model learns the subset of correlations common to a set of distributions or tasks, able to predict target values associated with previous or novel tasks without making use of task descriptors. However, similar to an issue shared with most of the approaches discussed in Sec. 3, the GEM model was evaluated on the MNIST and CIFAR100 datasets. Therefore, the question remains whether GEM scales up to more realistic scenarios. ## 4.4 Curiosity and Intrinsic Motivation Computational models of intrinsic motivation have taken inspiration from the way human infants and children choose their goals and progressively acquire skills to define developmental structures in lifelong learning frameworks (Baldassarre & Mirolli (2013); see Gottlieb et al. (2013) for a review). Infants seem to select experiences that maximize an intrinsic learning reward through an empirical process of exploration (Gopnik et al. 1999). From a modelling perspective, it has been proposed that the intrinsically motivated exploration of the environment, e.g., driven by the maximization of the learning progress (Oudeyer et al. (2007), Schmidhuber (1991), see Fig. 5.c for a schematic view), can lead to the self-organization of human-like developmental structures where the skills being acquired become progressively more complex. Computational models of intrinsic motivation can collect data and acquire skills incrementally through the online (self-)generation of a learning curriculum (Baranes & Oudeyer 2013, Forestier & Oudeyer 2016). This allows the efficient, stochastic selection of tasks to be learned with an active control of the growth of the complexity. Recent work in reinforcement learning has included mechanisms of curiosity and intrinsic motivation to address scenarios where the rewards are sparse or deceptive (Forestier et al. 2017, Pathak et al. 2017, Tanneberg et al. 2017, Bellemare et al. 2016, Kulkarni et al. 2016). In a scenario with very sparse extrinsic rewards, curiosity-driven exploration provides intrinsic reward signals that enable the agent to autonomously and progressively learn tasks of increasing complexity. Pathak et al. (2017) proposed an approach to curiosity-driven exploration where curiosity is modelled as the error in an agent's ability to predict the consequences of its own actions. This approach has shown to scale up to high-dimensional visual input, using the knowledge acquired from previous experiences for the faster exploration of unseen scenarios. However, the method relies on interaction episodes that convert unexpected interactions into intrinsic rewards, which does not extend to scenarios where interactions are rare. In this case, internally generated representations of the previous sparse interactions could be replayed and used to guide exploration (in a similar way to generative systems for memory replay; see Sec. 3.4). ## 4.5 Multisensory Learning The ability to integrate multisensory information is a crucial feature of the brain that yields a coherent, robust, and efficient interaction with the environment (Spence 2014, Ernst & B¨ ulthoff 2004, Stein & Meredith 1993). Information from different sensor modalities (e.g. vision, audio, proprioception) can be integrated into multisensory representations or be used to enhance unisensory ones (see Fig. 5.d). Multisensory processing functions are the result of the interplay of the physical properties of the crossmodal stimuli and prior knowledge and expectations (e.g., in terms of learned associations), scaffolding perception, cognition, and behaviour (see Murray et al. (2016), Stein et al. (2014) for reviews). The process of multisensory learning is dynamic across the lifespan and is subject to both short- and long-term changes. It consists of the dynamic reweighting of exogenous and endogenous factors that dictate to which extent multiple modalities interact with each other. Low-level stimulus characteristics (e.g., spatial proximity and temporal coincidence) are available before the formation of learned perceptual representations that bind increasingly complex higher-level characteristics (e.g., semantic congruency). Sophisticated perceptual mechanisms of multisensory integration emerge during development, starting from basic processing capabilities and progressively specializing towards more complex cognitive functions on the basis of sensorimotor experience (Lewkowicz 2014, Spence 2014). From a computational perspective, modelling multisensory learning can be useful for a number of reasons. First, multisensory functions aim at yielding robust responses in the case of uncertain and ambiguous sensory input. Models of causal inference have been applied to scenarios comprising the exposure to incongruent audio-visual information for solving multisensory conflicts (Parisi, Barros, Kerzel, Wu, Yang, Li, Liu & Wermter 2017, Parisi, Barros, Fu, Magg., Wu, Liu & Wermter 2018). Second, if trained with multisensory information, one modality can be reconstructed from available information in another modality. Moon et al. (2015) proposed multisensory processing for an audio-visual recognition task in which knowledge in a source modality can be transferred to a target modality. Abstract representations obtained from a network encoding the source modality can be used to fine-tune the network in the target modality, thereby relaxing the imbalance of the available data in the target modality. Barros et al. (2017) proposed a deep architecture modelling crossmodal expectation learning. After a training phase with audio-visual information, unisensory network channels can reconstruct the expected output from the other modality. Finally, mechanisms of attention are essential in lifelong learning scenarios for processing relevant information in complex environments and efficiently triggering goal-directed behaviour from continuous streams of multisensory information (Spence 2014). Such mechanisms may be modelled via the combination of the exogenous properties of crossmodal input, learned associations and crossmodal correspondences, and internally generated expectations (Chen & Spence 2017) with the aim of continually shaping perception, cognition, and behaviour in autonomous agents. ## 5 Conclusion Lifelong learning represents an utterly interesting but challenging component of artificial systems and autonomous agents operating on real-world data, which is typically non-stationary and temporally correlated. The mammalian brain remains the best model of lifelong learning, which makes biologically-inspired learning models a compelling approach. The general notion of structural plasticity (Sec. 2.2) is widely used across the machine learning literature and represents a promising solution to lifelong learning in its own right, even when disregarding biological desiderata. Proposed computational solutions for mitigating catastrophic forgetting and interference have focused on regulating intrinsic levels of plasticity to protect acquired knowledge (Sec. 3.2), dynamically allocating new neurons or network layers to accommodate novel knowledge (Sec. 3.3), and using complementary learning networks with experience replay for memory consolidation (Sec. 3.4). However, despite significant advances, current models of lifelong learning are still far from providing the flexibility, robustness, and scalability exhibited by biological systems. The most popular deep and shallow learning models of lifelong learning are restricted to the supervised domain, relying on large amounts of annotated data collected in controlled environments (see Sec. 3.5). Such a domain-specific training scheme cannot be applied directly to autonomous agents that operate in highly dynamic, unstructured environments. Additional research efforts are required to combine multiple methodologies that integrate a variety of factors observed in human learners. Basic mechanisms of critical periods of development (Sec. 4.2) can be modelled to empirically determine convenient (multilayered) neural network architectures and initial patterns of connectivity that improve the performance of the model for subsequent learning tasks. Methods comprising curriculum and transfer learning (Sec. 4.3) are a fundamental feature for reusing previously acquired knowledge and skills to solve a problem in a novel domain by sharing low- and high-level representations. For agents learning autonomously, approaches using intrinsic motivation (Sec. 4.4) are crucial for the self-generation of goals, leading to an empirical process of exploration and the progressive acquisition of increasingly complex skills. Finally, multisensory integration (Sec. 4.5) is a key feature of autonomous agents operating in highly dynamic and noisy environment, leading to robust learning and behaviour also in situations of uncertainty. ## Acknowledgment This research was partially supported by the German Research Foundation (DFG) under project Transregio Crossmodal Learning (TRR 169). The authors would like to thank Sascha Griffiths, Vincenzo Lomonaco, Sebastian Risi, and Jun Tani for valuable feedback and suggestions. ## References - Abbott, L. F. & Nelson, S. B. (2000), 'Synaptic plasticity: taming the beast', Nature Neuroscience 3 , 1178-1183. - Abraham, W. C. & Robins, A. (2005), 'Memory retention and weight plasticity in ANN simulations', Trends in Neurosciences 28 (2), 73-78. - Achille, A., Rovere, M. & Soatto, S. (2017), Critical learning periods in deep neural networks, arXiv:1711.08856. - Adams, R. A., Friston, K. J. & Bastos, A. M. (2015), 'Active inference, predictive coding and cortical architecture', In M. F. Casanova and I. Opris (Eds.), Recent Advances on the Modular Organization of the Cortex. Springer Netherlands . - Aimone, J. B., Wiles, J. & Gage, F. H. (2006), 'Potential role for adult neurogenesis in the encoding of time in new memories.', Nature Neuroscience 9 , 723-727. - Aimone, J. B., Wiles, J. & Gage, F. H. (2009), 'Computational influence of adult neurogenesis on memory encoding.', Neuron 61 , 187-202. - Altman, J. (1963), 'Autoradiographic investigation of cell proliferation in the brains of rats and cats', The Anatomical Record 145 (4), 573-591. - Astr¨ om, K. J. & Murray, R. M. (2010), Feedback Systems: An Introduction for Scientists and Engineers , Princeton University Press. - Baldassarre, G. & Mirolli, M. (2013), Intrinsically motivated learning in natural and artificial systems , Springer-Verlag, Berlin. - Baranes, A. & Oudeyer, P.-Y. (2013), 'Active learning of inverse models with intrinsically motivated goal exploration in robots', Robotics and Autonomous Systems 61 (1), 49-73. - Barnett, S. & Ceci, S. (2002), 'When and where do we apply what we learn? a taxonomy for far transfer', Psychological Bulletin 128 , 612-637. - Barros, P., Parisi, G. I., Fu, D., Liu, X. & Wermter, S. (2017), Expectation learning for adaptive crossmodal stimuli association, EUCog Meeting Proceedings, Zurich, Switzerland. - Barto, A. (2013), Intrinsic motivation and reinforcement learning , Baldassarre, G., Mirolli, M. (Eds.), Intrinsically Motivated Learning in Natural and Artificial Systems. Springer. - Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D. & Munos, R. (2016), Unifying count-based exploration and intrinsic motivation. - Bengio, Y., Louradour, J., Collobert, R. & Weston, J. (2009), Curriculum learning, pp. 41-48. - Benna, M. K. & Fusi, S. (2016), 'Computational principles of synaptic memory consolidation', Nature Neuroscience 19 (12), 1697-1708. - Bienenstock, E., Cooper, L. & Munro, P. (1982), 'Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex', The Journal of Neuroscience 2 , 32-48. - Boldrini, M., Fulmore, C., Tartt, A., Simeon, L. & Pavlova, I. e. a. (2018), 'Human hippocampal neurogenesis persists throughout aging', Cell Stem Cell 22 (4), 589-599. - Bontempi, B., Laurent-Demir, C., Destrade, C. & Jaffard, R. (1999), 'Time-dependent reorganization of brain circuitry underlying long-term memory storage', Nature 400 , 671-675. - Braun, C., Heinz, U., Schweizer, R., Weich, K., Birbaumer, N. & Topka, H. (2001), 'Dynamic organization of the somato-sensory cortex induced by motor activity', Brain 124 , 2259-2267. - Bremner, A., Lewkowicz, D. & Spence, C. (2012), 'Multisensory development', Oxford University Press, Oxford, UK . - Burgess, N., Shapiro, J. L. & Moore, M. A. (1991), 'Neural network models of list learning', Networks 2 , 399-422. - Calvert, G., Spence, C. & Stein, B. (2004), The Handbook of Multisensory Processes , Cambridge, MA: The MIT Press. - Cameron, H. A., Woolley, C. S., McEwen, B. S. & Gould, E. (1993), 'Differentiation of newly born neurons and glia in the dentate gyrus of the adult rat', Neuroscience 56 (2), 337-344. - Cangelosi, A. & Schlesinger, M. (2015), Developmental robotics: From babies to robots , MIT Press. - Chen, Y.-C. & Spence, C. (2017), 'Assessing the role of the unity assumption on multisensory integration: A review', Frontiers in Psychology 8 (445). - Cichon, J. & Gan, W. (2015), 'Branch-specific dendritic ca(2+) spikes cause persistent synaptic plasticity', Nature 520 , 180-185. - Coop, R., Mishtal, A. & Arel, I. (2013), 'Ensemble learning in fixed expansion layer networks for mitigating catastrophic forgetting', IEEE Transactions on Neural Networks and Learning Systems 24 (10), 1623-1634. - Cortes, C., Gonzalvo, X., Kuznetsov, V., Mohri, M. & Yang, S. (2016), AdaNet: Adaptive structural learning of artificial neural networks, arXiv:1607.01097. - Dai, W., Yang, Q., Xue, G.-R. & Yu, Y. (2007), Boosting for transfer learning, ICML'17, Sydney, Australia, pp. 193-200. - Davis, G. W. (2006), 'Homeostatic control of neural activity: from phenomenology to molecular design', Annual Review of Neuroscience 29 , 307-323. - Deng, W., Aimone, J. B. & Gage, F. H. (2010), 'New neurons and new memories: how does adult hippocampal neurogenesis affect learning and memory?', Nature Reviews Neuroscience 11 (5), 339-350. - Ditzler, G., Roveri, M., Alippi, C. & Polikar, R. (2015), 'Learning in nonstationary environments: A survey', IEEE Computational Intelligence Magazine 10 (4), 12-25. - Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E. & Darrell, T. (2014), Decaf: A deep convolutional activation feature for generic visual recognition, ICML'14, Beijing, China. - Douglas, R. J., Koch, C., Mahowald, M., Martin, K. A. & Suarez, H. H. (1995), 'Recurrent excitation in neocortical circuits', Science 269 , 981-985. - Doumas, L., Hummel, J. & Sandhofer, C. (2008), 'A theory of the discovery and predication of relational concepts', Psychological Review 115 , 1-43. - Draelos, T. J., Miner, N. E., Lamb, C. C., Vineyard, C. M., Carlson, K. D., James, C. D. & Aimone, J. B. (2017), Neurogenesis deep learning, IJCNN'17, Anchorage, Alaska, pp. 526-533. - Elfaramawy, N., Barros, P., Parisi, G. I. & Wermter, S. (2017), Emotion recognition from body expressions with a neural network architecture, Proceedings of the International Conference on Human Agent Interaction (HAI'17), Bielefeld, Germany, pp. 143-149. - Elman, J. L. (1993), 'Learning and development in neural networks: The importance of starting small', Cognition 48 (1), 71-99. - Eriksson, P. S., Perfilieva, E., Bjork-Eriksson, T., Alborn, A.-M., Nordborg, C., Peterson, D. A. & Gage, F. H. (1998), 'Neurogenesis in the adult human hippocampus', Nature medicine 4 (11), 1313-1317. - Ernst, M. O. & B¨ ulthoff, H. (2004), 'Merging the senses into a robust percept', Trends in Cognitive Sciences 8 (4), 162-169. - Fei-Fei, L., Fergus, R. & Perona, P. (2003), A bayesian approach to unsupervised one-shot learning of object categories, ICCV'03, Nice, France. - Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D., Rusu, A. A., Pritzel, A. & Wierstra, D. (2017), Pathnet: Evolution channels gradient descent in super neural networks, arXiv:1701.08734. - Forestier, S., Mollard, Y . & Oudeyer, P.-Y . (2017), Intrinsically motivated goal exploration processes with automatic curriculum learning, arXiv:1708.02190. - Forestier, S. & Oudeyer, P.-Y. (2016), Curiosity-driven development of tool use precursors: a computational model, Proceedings of the Annual Conference of the Cognitive Science Society. - French, R. M. (1997), 'Pseudo-recurrent connectionist networks: An approach to the sensitivitystability dilemma', Connection Science 9 (4), 353-380. - French, R. M. (1999), 'Catastrophic forgetting in connectionist networks', Trends in Cognitive Sciences 3 (4), 128-135. - Friston, K., Rigoli, F., Ognibene, D., Mathys, D., Fitzgerald, T. & Pezzulo, G. (2015), 'Active inference and epistemic value', Cognitive neuroscience 6 , 187-214. - Fritzke, B. (1992), A growing neural gas network learns topologies, Vol. 7, NIPS'95, Denver, CO, pp. 625-632. - Fusi, S., Drew, P. J. & Abbott, L. F. (2005), 'Cascade models of synaptically stored memories', Neuron 45 (4), 599-611. - Gage, F. H. (2000), 'Mammalian neural stem cells', Science 287 , 1433-1438. - Gais, S., Albouy, G., Boly, M., Dang-Vu, T. T., Darsaud, A., Desseilles, M., Rauchs, G., Schabus, M., Sterpenich, V. & Vandewalle, G. e. a. (2007), 'Sleep transforms the cerebral trace of declarative memories', Proceedings of the National Academy of Sciences 104 (47), 18778-18783. - Gelbard-Sagiv, H., Mukamel, R., Harel, M., Malach, R. & Fried, I. (2008), 'Internally generated reactivation of single neurons in human hippocampus during free recall', Science 322 , 96-101. - Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M. & Ritter, M. (2017), Audio set: An ontology and human-labeled dataset for audio events, ICASSP'17, New Orleans, LA. - Gepperth, A. & Karaoguz, C. (2015), 'A bio-inspired incremental learning architecture for applied perceptual problems', Cognitive Computation 8 (5), 924-934. - Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A. & Bengio, Y. (2013), An empirical investigation of catastrophic forgetting in gradient-based neural networks, arXiv:1312.6211. - Gopnik, A., Meltzoff, A. N. & Kuhl, P. K. (1999), The scientist in the crib: Minds, brains, and how children learn , William Morrow & Co. - Gorelick, L., Blank, M., Shechtman, E., Irani, M. & Basri, R. (2005), Actions as space-time shapes, ICCV'05, Beijing, China, pp. 1395-1402. - Gottlieb, J., Oudeyer, P.-Y., Lopes, M. & Baranes, A. (2013), 'Information seeking, curiosity and attention: Computational and neural mechanisms', Trends in Cognitive Science 17 (11), 585-596. - Grant, W. S., Tanner, J. & Itti, L. (2017), 'Biologically plausible learning in neural networks with modulatory feedback', Neural Networks 88 , 32-48. - Graves, A., Bellemare, M. G., Menick, J., Munos, R. & Kavukcuoglu, K. (2017), Automated curriculum learning for neural networks, arXiv:1704.03003. - Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwinska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T. & Agapiou, J. e. a. (2016), 'Hybrid computing using a neural network with dynamic external memory', Nature 538 , 471-476. - Grossberg, S. (1980), 'How does a brain build a cognitive code?', Psychol. Rev. 87 , 1-51. - Grossberg, S. (2012), 'Adaptive resonance theory: how a brain learns to consciously attend, learn, and recognize a changing world.', Neural Networks 37 , 1-41. - Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S. & Lew, M. (2016), 'Deep learning for visual understanding: A review', Neurocomputing 187 (8), 27-48. - Hassabis, D., Kumaran, D., Summerfield, C. & Botvinick, M. (2017), 'Neuroscience-inspired artificial intelligence', Neuron Review 95 (2), 245-258. - Hasson, U., Yang, E., Vallines, I., Heeger, D. J. & Rubin, N. (2008), 'A hierarchy of temporal receptive windows in human cortex', The Journal of Neuroscience 28 (10), 2539-2550. - Hebb, D. (1949), The Organization of Behavior , John Wiley & Sons. - Heeger, D. J. (2017), 'Theory of cortical function', Proceedings of the National Academy of Sciences of the United States of America 114 (8), 1773-1782. - Hensch, T. (2004), 'Critical period regulation', Annual Review of Neuroscience 27 , 549-579. - Hensch, T. K., Fagiolini, M., Mataga, N., Stryker, M. P., Baekkeskov, S. & Kash, S. F. (1998), 'Local gaba circuit control of experience-dependent plasticity in developing visual cortex', Science 282 , 1504-1508. - Hertz, J., Krogh, A. & Palmer, R. G. (1991), Introduction to the Theory of Neural Computation , Redwood City CA: Addison-Wesley. - Hinton, G. E. & Plaut, D. C. (1987), Using fast weights to deblur old memories, Proceedings of the Annual Conference of the Cognitive Science Society, pp. 177-186. - Hinton, G., Vinyals, O. & Dean, J. (2014), Distilling the knowledge in a neural network, NIPS'14, Workshop on Deep Learning and Representation, Montreal, Canada. - Hirsch, H. & Spinelli, D. (1970), 'Visual experience modifies distribution of horizontally and vertically oriented receptive fields in cats', Science 168 , 869-871. - Holyoak, K. & Thagard, P. (1997), 'The analogical mind', American Psychologist 52 , 35-44. - Hubel, D. H. & Wiesel, T. H. (1962), 'Receptive fields, binocular and functional architecture in the cats visual cortex', Journal of Physiology 160 , 106-154. - Hubel, D. H. & Wiesel, T. H. (1967), 'Cortical and callosal connections concerned with the vertical meridian of visual fields in the cat', Journal of Neurophysiology 30 , 1561-1573. - Hubel, D. H. & Wiesel, T. H. (1970), 'The period of susceptibility to the psychological effects of unilateral eye closure in kittens', Journal of Physiology 206 , 419-436. - Hubel, D. H., Wiesel, T. H. & LeVay, S. (1977), 'Plasticity of ocular dominance columns in monkey striate cortex', Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences 278 , 377-409. - Jung, H., Ju, J., Jung, M. & Kim, J. (2018), Less-forgetting learning in deep neural networks, AAAI'18, New Orleans, LA. - Kamra, N., Gupta, U. & Liu, Y. (2018), Deep generative dual memory network for continual learning, arXiv:1710.10368. - Kemker, R. & Kanan, C. (2018), Fearnet: Brain-inspired model for incremental learning, ICLR'18, Vancouver, Canada. - Kemker, R., McClure, M., Abitino, A., Hayes, T. & Kanan, C. (2018), Measuring catastrophic forgetting in neural networks, AAAI'18, New Orleans, LA. - Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D. & Hadsell, R. (2017), 'Overcoming catastrophic forgetting in neural networks', Proceedings of the National Academy of Sciences 114 (13), 3521-3526. - Kitamura, T., Ogawa, S. K., Roy, D. S., Okuyama, T., Morrissey, M. D., Smith, L. M., Redondo, R. L. & Tonegawa, S. (2017), 'Engrams and circuits crucial for systems consolidation of a memory', Science 356 , 73-78. - Kiyota, T. (2017), Neurogenesis and brain repair , Neuroimmune Pharmacology. Springer. - Knoblauch, A. . (2017), Impact of structural plasticity on memory formation and decline , Rewiring the Brain: A Computational Approach to Structural Plasticity in the Adult Brain, eds A. van Ooyen and M. Butz, Elsevier, Academic Press. - Knoblauch, A., K¨ orner, E., Krner, U. & Sommer, F. T. (2014), 'Structural plasticity has high memory capacity and can explain graded amnesia, catastrophic forgetting, and the spacing effect.', PLoS ONE 9:e96485 . - Kohonen, T. (1982), 'Self-organized formation of topologically correct feature maps', Biological Cybernetics 43 , 59-69. - Kohonen, T. (1995), Self-organizing maps , New York: Springer. - Krizhevsky, A. (2009), Learning multiple layers of features from tiny images , Master's thesis, University of Toronto. - Krueger, K. A. & Dayan, P. (2009), 'Flexible shaping: how learning in small steps helps', Cognition 110 , 380-394. - Kulkarni, T., Narasimhan, K., Saeedi, A. & Tenenbaum, J. (2016), Hierarchical deep reinforcement learning: integrating temporal abstraction and intrinsic motivation, arXiv:1604.06057. - Kumaran, D., Hassabis, D. & McClelland, J. L. (2016), 'What learning systems do intelligent agents need? complementary learning systems theory updated', Trends in Cognitive Sciences 20 (7), 512534. - Kumaran, D. & McClelland, J. L. (2012), 'Generalization through the recurrent interaction of episodic memories: A model of the hippocampal system', Psychological Review 119 , 573-616. - Lambon Ralph, M. & Ehsan, S. (2006), 'Age of acquisition effects depend on the mapping between representations and the frequency of occurrence: empirical and computational evidence', Visual Cognition 13 (7-8), 928-948. - Lampert, C., Nickisch, H. & Harmeling, S. (2009), Learning to detect unseen object classes by between-class attribute transfer, CVPR'09, Miami Beach, Florida. - LeCun, Y., Bengio, Y. & Hinton, G. (2015), 'Deep learning', Nature 521 , 436-444. - LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. (1998), Gradient-based learning applied to document recognition , In Proceedings of the IEEE. - Lenneberg, E. (1967), Biological foundations of language , New York: Wiley. - Lewandowsky, S. & Li, S. (1994), Catastrophic Interference in Neural Networks: Causes, Solutions, and Data , Dempster, F.N. & Brainerd,C. (Eds.) New Perspectives on Interference and Inhibition in Cognition. Academic Press, New York. - Lewkowicz, D. J. (2014), 'Early experience & multisensory perceptual narrowing', Developmental Psychobiology 56 (2), 292-315. - Li, P., Farkas, I. & McWhinney, B. (2004), 'Early lexical development in a self-organising neural network', Neural Networks 17 , 1345-1362. - Li, Z. & Hoiem, D. (2016), Learning without forgetting, ECCV'16, Amsterdam, The Netherlands, pp. 614-629. - Lomonaco, V. & Maltoni, D. (2017), CORe50: A new dataset and benchmark for continuous object recognition, CoRL, Mountain View, CA. - Lopez-Paz, D. & Ranzato, M. (2017), Gradient episodic memory for continual learning, NIPS'17, Long Beach, CA. - L¨ uders, B., Schl¨ ager, M. & Risi, S. (2016), Continual learning through evolvable neural turing machines, NIPS'16, Workshop on Continual Learning and Deep Networks, Barcelona, Spain. - Maltoni, D. & Lomonaco, V. (2018), Continuous learning in single-incremental-task scenarios., arXiv:1806.08568. - Marchman, V. A. (1993), 'Constraints on plasticity in a connectionist model of English past tense', Journal of Cognitive Neuroscience 5 (2), 215-234. - Mareschal, D., Johnson, M., Sirios, S., Spratling, M., Thomas, M. & Westermann, G. (2007), Neuroconstructivism: How the brain constructs cognition , Oxford: Oxford University Press. - Marsland, S., Shapiro, J. & Nehmzow, U. (2002), 'A self-organising network that grows when required', Neural Networks 15 (8-9), 1041-1058. - Martinetz, T., Berkovich, S. & Schulten, K. (1993), 'Neural-gas network for vector quantization and its application to time-series prediction', IEEE Transactions on Neural Networks 4 (4), 558-569. - McClelland, J. L., McNaughton, B. L. & O'Reilly, R. C. (1995), 'Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory', Psychological Review 102 , 419-457. - McCloskey, M. & Cohen, N. J. (1989), 'Catastrophic interference in connectionist networks: The sequential learning problem', The Psychology of Learning and Motivation 24 , 104-169. - Mermillod, M., Bugaiska, A. & Bonin, P. (2013), 'The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects', Frontiers in Psychology 4 (504). - Mici, L., Parisi, G. I. & Wermter, S. (2017), An incremental self-organizing architecture for sensorimotor learning and prediction, arXiv:1712.08521. - Mici, L., Parisi, G. I. & Wermter, S. (2018), 'A self-organizing neural network architecture for learning human-object interactions', Neurocomputing 307 , 14-24. - Miikkulainen, R. (1997), 'Dyslexic and category-specific aphasic impairments in a self-organizing feature map model of the lexicon', Brain and Language 59 , 334-366. - Miller, K., Keller, J. & Stryker, M. (1989), 'Occular dominance column development: Analysis and simulation', Science 245 , 605-615. - Miller, K. & MacKay, D. J. (1994), 'The role of constraints in Hebbian learning', Neural Computation 6 , 100-126. - Ming, G. L. & Song, H. (2011), 'Adult neurogenesis in the mammalian brain: Significant answers and significant questions.', Neuron 70 , 687-702. - Moon, S., Kim, S. & Wang, H. (2015), Multimodal transfer deep learning with applications in audio-visual recognition, NIPS'15, Montreal, Canada. - Murray, M. M., Lewkowicz, D. J., Amedi, A. & Wallace, M. T. (2016), 'Multisensory processes: A balancing act across the lifespan', Trends in Neurosciences 39 , 567-579. - Nadal, J. P., Toulouse, G., Changeux, J. P. & Dehaene, S. (1986), 'Networks of formal neurons and memory palimpsets', Europhysics Letters 1 , 535-543. - Nelson, C. A. (2000), 'Neural plasticity and human development: The role of early experience in sculpting memory systems', Developmental Science 3 (2), 115-136. - Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B. & Ng, A. Y. (2011), Reading digits in natural images with unsupervised feature learning, NIPS'11, Workshop on deep learning and unsupervised feature learning, Granada, Spain. - O'Neill, J., Pleydell-Bouverie, B., Dupret, D. & Csicsvari, J. (2010), 'Play it again: Reactivation of waking experience and memory', Trends in Neuroscience 33 , 220-229. - O'Reilly, R. C. (2004), The Division of Labor Between the Neocortex and Hippocampus. Connectionist Modeling in Cognitive (Neuro-)Science , George Houghton, Ed., Psychology Press. - O'Reilly, R. C. and. Norman, K. A. (2002), 'Hippocampal and neocortical contributions to memory: Advances in the complementary learning systems framework', Trends in Cognitive Sciences 6 (12), 505-510. - O'Reilly, R. C. & Rudy, J. W. (2000), 'Computational principles of learning in the neocortex and hippocampus', Hippocampus 10 , 389-397. - Oudeyer, P.-Y., Kaplan, F. & Hafner, V. (2007), 'Intrinsic motivation systems for autonomous mental development', IEEE Transactions on Evolutionary Computation 11 (2), 265-286. - Palatucci, M., Pomerleau, D. A., Hinton, G. E. & Mitchell, T. (2009), Zero-shot learning with semantic output codes, NIPS'09, Vancouver, Canada. - Pallier, C., Dehaene, S., Poline, J.-B., LeBihan, D., Argenti, A.-M., Dupoux, E. & Mehler, J. (2003), 'Brain imaging of language plasticity in adopted adults: can a second language replace a first?', Cerebral Cortex 13 , 155-161. - Pan, S. J. & Yang, Q. (2010), 'A survey on transfer learning', IEEE Transactions on Knowledge and Data Engineering 22 (10), 1345-1359. - Parisi, G., Barros, P., Fu, D., Magg., S., Wu, H., Liu, X. & Wermter, S. (2018), A neurorobotic experiment for crossmodal conflict resolution in complex environments, arXiv:1802.10408. - Parisi, G. I., Barros, P., Kerzel, M., Wu, H., Yang, G., Li, Z., Liu, X. & Wermter, S. (2017), A computational model of crossmodal processing for conflict resolution, ICDL-EPIROB'17, Lisbon, Portugal, pp. 33-38. - Parisi, G. I., Magg, S. & Wermter, S. (2016), Human motion assessment in real time using recurrent self-organization, Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication, New York, NY, pp. 71-79. - Parisi, G. I., Tani, J., Weber, C. & Wermter, S. (2017), 'Lifelong learning of humans actions with deep neural network self-organization', Neural Networks 96 , 137-149. - Parisi, G., Ji, X. & Wermter, S. (2018), On the role of neurogenesis in overcoming catastrophic forgetting., NIPS'18, Workshop on Continual Learning, Montreal, Canada. - Parisi, G., Tani, J., Weber, C. & Wermter, S. (2018), Lifelong learning of spatiotemporal representations with dual-memory recurrent self-organization, arXiv:1805.10966. - Part, J. L. & Lemon, O. (2016), Incremental on-line learning of object classes using a combination of self-organizing incremental neural networks and deep convolutional neural networks, IROS'16, Workshop on Bio-inspired Social Robot Learning in Home Scenarios, Daejeon, South Korea. - Part, J. L. & Lemon, O. (2017), Incremental online learning of objects for robots operating in real environments, ICDL-EPIROB'17, Lisbon, Portugal. - Pathak, D., Agrawal, P., Efros, A. A. & Darrell, F. (2017), Curiosity-driven exploration by selfsupervised prediction, ICML'17, Sydney, Australia. - Polikar, R., Upda, L., Upda, S. S. & Honavar, V. (2001), 'Learn++: An incremental learning algorithm for supervised neural networks', IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 31 (4), 497-508. - Power, J. D. & Schlaggar, B. L. (2016), 'Neural plasticity across the lifespan', Wiley Interdisciplinary Reviews: Developmental Biology 6 (216). - Quadrato, G., Elnaggar, M. Y. & Di Giovanni, S. (2014), 'Adult neurogenesis in brain repair: Cellular plasticity vs. cellular replacement', Frontiers in Neuroscience 8 (17). - Rao, R. P. N. & Ballard, D. H. (1999), 'Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects', Nature Neuroscience 2 (1), 79-87. - Ratcliff, R. (1990), 'Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions', Psychological Review 97 (2), 285-308. - Razavian, A. S., Azizpour, H., Sullivan, J. & Carlsson, S. (2014), CNN features off-the-shelf: An astounding baseline for recognition, CVPR'14, Columbus, OH, pp. 806-813. - Rebuffi, S.-A., Kolesnikov, A., Sperl, G. & Lampert, C. H. (2016), iCaRL: Incremental classifier and representation learning, arXiv:1611.07725. - Reed, S. & de Freitas, N. (2015), Neural programmer interpreters, arXiv:1511.06279. - Ren, B., Wang, H., Li, J. & Gao, H. (2017), 'Life-long learning based on dynamic combination model', Applied Soft Computing 56 , 398-404. - Richardson, F. M. & Thomas, M. S. C. (2008), 'Critical periods and catastrophic interference effects in the development of self-organising feature maps', Developmental Science 11 (3), 371-389. - Ring, M. B. (1997), 'CHILD: A first step towards continual learning', Machine Learning 28 , 77104. - Robins, A. V. (1993), Catastrophic forgetting in neural networks: The role of rehearsal mechanisms, Proceedings of the First New Zealand International Two-stream Conference on Artificial Neural Networks and Expert Systems. IEEE Computer Society Press. - Robins, A. V. (1995), 'Catastrophic forgetting, rehearsal and pseudorehearsal', Connection Science 7 (2), 123-146. - Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R. & Hadsell, R. (2016), Progressive neural networks, arXiv:1606.04671. - Rusu, A. A., Vecerik, M., Roth¨ orl, T., Heess, N., Pascanu, R. & Hadsell, R. (2017), Sim-to-real robot learning from pixels with progressive nets, CoRL'17, Mountain View, CA. - Sanger, T. D. (1994), 'Neural network learning control of robot manipulators using gradually increasing task difficulty', IEEE Transactions on Robotics and Automation 10 . - Schmidhuber, J. (1991), Curious model-building control systems. - Schuldt, C., Laptev, I. & Caputo, B. (2004), Recognizing human actions: A local SVM approach, ICPR'04, Cambridge, UK, pp. 32-36. - Seidenberg, M. & Zevin, J. (2006), Connectionist models in developmental cognitive neuroscience: Critical periods and the paradox of success , In Y. Munakata & M.H. Johnson (Eds.) Processes of change in brain and cognitive development: Attention and Performance XXI. Oxford: Oxford University Press. - Senghas, A., Kita, S. & ¨ Ozy¨ urek, A. (2004), 'Children creating core properties of language: Evidence from an emerging sign language in Nicaragua', Science 305 , 1779-1782. - Shatz, C. J. (1996), 'Emergence of order in visual system development', Proceedings of the National Academy of Sciences 93 , 602-608. - Shin, H., Lee, J. K., Kim, J. & Kim, J. (2017), Continual learning with deep generative replay, NIPS'17, Long Beach, CA. - Skinner, B. F. (1958), 'Reinforcement today', American Psychologist 13 , 94-99. - Soltoggio, A. (2015), 'Short-term plasticity as cause-effect hypothesis testing is distal reward learning', Biological Cybernetics 109 , 75-94. - Soltoggio, A., Stanley, K. O. & Risi, S. (2017), Born to learn: the inspiration, progress, and future of evolved plastic artificial neural networks, arXiv:1703.10371. - Song, S., Miller, K. D. & Abbott, L. F. (2000), 'Competitive Hebbian learning through spike-timingdependent synaptic plasticity', Nature Neuroscience 3 , 919-926. - Sorrells, S. F., Paredes, M. F., Arantxa, C.-S., Sandoval, K., Dashi, Q. & Kelley, K. W. e. a. (2018), 'Human hippocampal neurogenesis drops sharply in children to undetectable levels in adults', Nature 555 , 377-381. - Spence, C. (2010), 'Crossmodal spatial attention', Annals of the New York Academy of Sciences 1191 , 182-200. - Spence, C. (2014), 'Orienting attention: A crossmodal perspective', The Oxford Handbook of Attention. Oxford, UK: Oxford University Press pp. 446-471. - Stein, B. E. & Meredith, M. A. (1993), The merging of the senses , The MIT Press, Cambridge, MA, US. - Stein, B. E., Stanford, T. R. & Rowland, B. A. (2014), 'Development of multisensory integration from the perspective of the individual neuron', Nature Reviews Neuroscience 15 (8), 520-535. - Sur, M. & Leamey, C. A. (2001), 'Development and plasticity of cortical areas and networks', Nature Reviews Neuroscience 2 , 251-262. - Tani, J. (2016), Exploring Robotic Minds: Actions, Symbols, and Consciousness a Self-Organizing Dynamic Phenomena , Oxford University Press. - Tanneberg, D., Peters, J. & Rueckert, E. (2017), Online learning with stochastic recurrent neural networks using intrinsic motivation signals, CoRL'17, Mountain View, CA. - Taupin, P. & Gage, F. H. (2002), 'Adult neurogenesis and neural stem cells of the central nervous system in mammals', Journal of neuroscience research 69 (6), 745-749. - Taylor, P., Hobbs, J. N., Burroni, J. & Siegelmann, H. T. (2015), 'The global landscape of cognition: hierarchical aggregation as an organizational principle of human cortical networks and functions', Scientific Reports 5 (18112). - Tessler, C., Givony, S., Zahavy, T., Mankowitz, D. J. & Mannor, S. (2017), A deep hierarchical approach to lifelong learning in Minecraft, AAAI'17, San Francisco, CA. - Thomas, M. & Johnson, M. (2006), 'The computational modelling of sensitive periods', Developmental Psychobiology 48 (4), 337-344. - Thrun, S. & Mitchell, T. (1995), 'Lifelong robot learning', Robotics and Autonomous Systems 15 , 25-46. - Tse, D., Takeuchi, T., Kakeyama, M., Kajii, Y., Okuno, H., Tohyama, C., Bito, H. & Morris, R. G. (2011), 'Schema-dependent gene activation and memory encoding in neocortex', Science 333 , 891-895. - Turrigiano, G. (2011), 'Too many cooks? intrinsic and synaptic homeostatic mechanisms in cortical circuit refinement', Annual Review of Neuroscience 34 , 89-103. - Uylings, H. (2006), 'Development of the human cortex and the concept of critical or sensitive periods', Language Learning 56 , 59-90. - Vinyals, O., Blundell, C., Lillicrap, T. & Wierstra, D. (2016), Matching networks for one shot learning, NIPS'16, Barcelona, Spain. - Wah, C., Branson, S., Welinder, P., Perona, P. & Belongie, S. (2011), Caltech-UCSD Birds-2002011 dataset, Tech Report: CNS-TR-2011-001. - Weiss, K., Khoshgoftaar, T. M. & Wang, D.-D. (2016), 'A survey of transfer learning', Journal of Big Data 3 (9). - Wermter, S., Palm, G. & Elshaw, M. (2005), Biomimetic Neural Learning for Intelligent Robots , Intelligent Systems, Cognitive Robotics, and Neuroscience. Springer, Berlin, Heidelberg. - Willshaw, D. J. & von der Malsburg, C. (1976), 'How patterned neural connections can be set up by self-organization', Proceedings of the Royal Society of London B: Biological Sciences 194 (1117), 431-445. - Xiao, T., Zhang, J., Yang, K., Peng, Y. & Zhang, Z. (2014), Error-driven incremental learning in deep convolutional neural network for large-scale image classification, Proceedings of the ACM International Conference on Multimedia, Orlando, FL, pp. 177-186. - Yoon, J., Yang, E. & Lee, J. Hwang, S. J. (2018), Lifelong learning with dynamically expandable networks, ICLR'18, Vancouver, Canada. - Zenke, F., Gerstner, W. & Ganguli, S. (2017), 'The temporal paradox of hebbian learning and homeostatic plasticity', Neurobiology 43 , 166-176. - Zenke, F., Poole, B. & Ganguli, S. (2017), Continual learning through synaptic intelligence, ICML'17, Sydney, Australia. - Zhou, G., Sohn, K. & Lee, H. (2012), Online incremental feature learning with denoising autoencoders, International Conference on Artificial Intelligence and Statistics, pp. 1453-1461.