## NEURAL ARCHITECTURE SEARCH WITH REINFORCEMENT LEARNING
∗
Barret Zoph , Quoc V. Le Google Brain
{ barretzoph,qvl } @google.com
## ABSTRACT
Neural networks are powerful and flexible models that work well for many difficult learning tasks in image, speech and natural language understanding. Despite their success, neural networks are still hard to design. In this paper, we use a recurrent network to generate the model descriptions of neural networks and train this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. On the CIFAR-10 dataset, our method, starting from scratch, can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy. Our CIFAR-10 model achieves a test error rate of 3 . 65 , which is 0 . 09 percent better and 1.05x faster than the previous state-of-the-art model that used a similar architectural scheme. On the Penn Treebank dataset, our model can compose a novel recurrent cell that outperforms the widely-used LSTM cell, and other state-of-the-art baselines. Our cell achieves a test set perplexity of 62.4 on the Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art model. The cell can also be transferred to the character language modeling task on PTB and achieves a state-of-the-art perplexity of 1.214.
## 1 INTRODUCTION
The last few years have seen much success of deep neural networks in many challenging applications, such as speech recognition (Hinton et al., 2012), image recognition (LeCun et al., 1998; Krizhevsky et al., 2012) and machine translation (Sutskever et al., 2014; Bahdanau et al., 2015; Wu et al., 2016). Along with this success is a paradigm shift from feature designing to architecture designing, i.e., from SIFT (Lowe, 1999), and HOG (Dalal & Triggs, 2005), to AlexNet (Krizhevsky et al., 2012), VGGNet (Simonyan & Zisserman, 2014), GoogleNet (Szegedy et al., 2015), and ResNet (He et al., 2016a). Although it has become easier, designing architectures still requires a lot of expert knowledge and takes ample time.
Figure 1: An overview of Neural Architecture Search.
<details>
<summary>Image 1 Details</summary>
### Visual Description
## Diagram: Controller-Based Neural Architecture Search
### Overview
The image presents a diagram illustrating a controller-based neural architecture search process. It depicts a loop where a controller (RNN) samples architectures, trains child networks, and updates itself based on the performance of those networks.
### Components/Axes
* **The controller (RNN):** A pink rounded rectangle on the left side of the diagram.
* **Trains a child network with architecture A to get accuracy R:** A light blue rounded rectangle on the right side of the diagram.
* **Sample architecture A with probability p:** Text above the top arrow, indicating the controller samples architectures with a certain probability.
* **Compute gradient of p and scale it by R to update the controller:** Text below the bottom arrow, indicating the controller updates itself based on the gradient of the probability and the accuracy of the child network.
### Detailed Analysis
The diagram shows a cyclical process:
1. The controller (RNN) samples an architecture A with probability p. This is represented by an arrow going from the pink controller box to the light blue "Trains a child network" box.
2. The child network is trained with architecture A, resulting in accuracy R.
3. The gradient of p is computed and scaled by R to update the controller. This is represented by an arrow going from the light blue "Trains a child network" box back to the pink controller box.
### Key Observations
* The diagram illustrates a reinforcement learning approach to neural architecture search.
* The controller learns to sample architectures that lead to high accuracy in the child networks.
* The accuracy R is used as a reward signal to update the controller.
### Interpretation
The diagram illustrates a common method for automating neural architecture search. The controller, typically an RNN, learns to generate promising architectures. The performance of these architectures, evaluated by training child networks, is used to refine the controller's search strategy. This process allows for the discovery of novel and effective neural network architectures without extensive manual design. The use of the gradient of 'p' scaled by 'R' suggests a policy gradient reinforcement learning approach.
</details>
This paper presents Neural Architecture Search, a gradient-based method for finding good architectures (see Figure 1) . Our work is based on the observation that the structure and connectivity of a
∗ Work done as a member of the Google Brain Residency program ( g.co/brainresidency .)
neural network can be typically specified by a variable-length string. It is therefore possible to use a recurrent network - the controller - to generate such string. Training the network specified by the string - the 'child network' - on the real data will result in an accuracy on a validation set. Using this accuracy as the reward signal, we can compute the policy gradient to update the controller. As a result, in the next iteration, the controller will give higher probabilities to architectures that receive high accuracies. In other words, the controller will learn to improve its search over time.
Our experiments show that Neural Architecture Search can design good models from scratch, an achievement considered not possible with other methods. On image recognition with CIFAR-10, Neural Architecture Search can find a novel ConvNet model that is better than most human-invented architectures. Our CIFAR-10 model achieves a 3.65 test set error, while being 1.05x faster than the current best model. On language modeling with Penn Treebank, Neural Architecture Search can design a novel recurrent cell that is also better than previous RNN and LSTM architectures. The cell that our model found achieves a test set perplexity of 62.4 on the Penn Treebank dataset, which is 3.6 perplexity better than the previous state-of-the-art.
## 2 RELATED WORK
Hyperparameter optimization is an important research topic in machine learning, and is widely used in practice (Bergstra et al., 2011; Bergstra & Bengio, 2012; Snoek et al., 2012; 2015; Saxena & Verbeek, 2016). Despite their success, these methods are still limited in that they only search models from a fixed-length space. In other words, it is difficult to ask them to generate a variable-length configuration that specifies the structure and connectivity of a network. In practice, these methods often work better if they are supplied with a good initial model (Bergstra & Bengio, 2012; Snoek et al., 2012; 2015). There are Bayesian optimization methods that allow to search non fixed length architectures (Bergstra et al., 2013; Mendoza et al., 2016), but they are less general and less flexible than the method proposed in this paper.
Modern neuro-evolution algorithms, e.g., Wierstra et al. (2005); Floreano et al. (2008); Stanley et al. (2009), on the other hand, are much more flexible for composing novel models, yet they are usually less practical at a large scale. Their limitations lie in the fact that they are search-based methods, thus they are slow or require many heuristics to work well.
Neural Architecture Search has some parallels to program synthesis and inductive programming, the idea of searching a program from examples (Summers, 1977; Biermann, 1978). In machine learning, probabilistic program induction has been used successfully in many settings, such as learning to solve simple Q&A (Liang et al., 2010; Neelakantan et al., 2015; Andreas et al., 2016), sort a list of numbers (Reed & de Freitas, 2015), and learning with very few examples (Lake et al., 2015).
The controller in Neural Architecture Search is auto-regressive, which means it predicts hyperparameters one a time, conditioned on previous predictions. This idea is borrowed from the decoder in end-to-end sequence to sequence learning (Sutskever et al., 2014). Unlike sequence to sequence learning, our method optimizes a non-differentiable metric, which is the accuracy of the child network. It is therefore similar to the work on BLEU optimization in Neural Machine Translation (Ranzato et al., 2015; Shen et al., 2016). Unlike these approaches, our method learns directly from the reward signal without any supervised bootstrapping.
Also related to our work is the idea of learning to learn or meta-learning (Thrun & Pratt, 2012), a general framework of using information learned in one task to improve a future task. More closely related is the idea of using a neural network to learn the gradient descent updates for another network (Andrychowicz et al., 2016) and the idea of using reinforcement learning to find update policies for another network (Li & Malik, 2016).
## 3 METHODS
In the following section, we will first describe a simple method of using a recurrent network to generate convolutional architectures. We will show how the recurrent network can be trained with a policy gradient method to maximize the expected accuracy of the sampled architectures. We will present several improvements of our core approach such as forming skip connections to increase model complexity and using a parameter server approach to speed up training. In the last part of
the section, we will focus on generating recurrent architectures, which is another key contribution of our paper.
## 3.1 GENERATE MODEL DESCRIPTIONS WITH A CONTROLLER RECURRENT NEURAL NETWORK
In Neural Architecture Search, we use a controller to generate architectural hyperparameters of neural networks. To be flexible, the controller is implemented as a recurrent neural network. Let's suppose we would like to predict feedforward neural networks with only convolutional layers, we can use the controller to generate their hyperparameters as a sequence of tokens:
Figure 2: How our controller recurrent neural network samples a simple convolutional network. It predicts filter height, filter width, stride height, stride width, and number of filters for one layer and repeats. Every prediction is carried out by a softmax classifier and then fed into the next time step as input.
<details>
<summary>Image 2 Details</summary>
### Visual Description
## Diagram: Convolutional Neural Network Layer Structure
### Overview
The image depicts a diagram of a convolutional neural network (CNN) layer structure, illustrating the flow of data and the application of filters and strides across multiple layers. The diagram shows the connections between neurons in adjacent layers, as well as the feedback loops that allow information to propagate through the network.
### Components/Axes
* **Layers:** The diagram shows three layers labeled "Layer N-1", "Layer N", and "Layer N+1".
* **Neurons:** Each layer consists of multiple neurons represented by squares.
* **Connections:** Neurons in adjacent layers are connected by arrows, indicating the flow of data.
* **Filters:** The diagram highlights the application of filters with parameters such as "Number of Filters" (light blue), "Filter Height" (light red), and "Filter Width" (yellow).
* **Strides:** The diagram also shows the application of strides with parameters such as "Stride Height" (light green) and "Stride Width" (light purple).
* **Feedback Loops:** Dotted arrows indicate feedback loops, allowing information to propagate through the network.
### Detailed Analysis
The diagram illustrates the structure of a convolutional neural network (CNN) layer. Each layer consists of multiple neurons, represented by squares. The neurons in adjacent layers are connected by arrows, indicating the flow of data.
The diagram highlights the application of filters with parameters such as "Number of Filters", "Filter Height", and "Filter Width". These filters are used to extract features from the input data. The diagram also shows the application of strides with parameters such as "Stride Height" and "Stride Width". Strides determine how far the filter moves across the input data.
Dotted arrows indicate feedback loops, allowing information to propagate through the network. These feedback loops can be used to improve the accuracy of the network.
The parameters are arranged in the following order:
1. Number of Filters (light blue)
2. Filter Height (light red)
3. Filter Width (yellow)
4. Stride Height (light green)
5. Stride Width (light purple)
This sequence repeats across the layers.
### Key Observations
* The diagram shows the basic structure of a convolutional neural network (CNN) layer.
* The diagram highlights the application of filters and strides.
* The diagram shows the presence of feedback loops.
* The diagram illustrates the flow of data through the network.
### Interpretation
The diagram provides a visual representation of the structure and operation of a convolutional neural network (CNN) layer. It demonstrates how filters and strides are applied to extract features from the input data, and how feedback loops can be used to improve the accuracy of the network. The diagram is useful for understanding the basic principles of CNNs and how they are used in various applications such as image recognition and natural language processing.
</details>
In our experiments, the process of generating an architecture stops if the number of layers exceeds a certain value. This value follows a schedule where we increase it as training progresses. Once the controller RNN finishes generating an architecture, a neural network with this architecture is built and trained. At convergence, the accuracy of the network on a held-out validation set is recorded. The parameters of the controller RNN, θ c , are then optimized in order to maximize the expected validation accuracy of the proposed architectures. In the next section, we will describe a policy gradient method which we use to update parameters θ c so that the controller RNN generates better architectures over time.
## 3.2 TRAINING WITH REINFORCE
The list of tokens that the controller predicts can be viewed as a list of actions a 1: T to design an architecture for a child network. At convergence, this child network will achieve an accuracy R on a held-out dataset. We can use this accuracy R as the reward signal and use reinforcement learning to train the controller. More concretely, to find the optimal architecture, we ask our controller to maximize its expected reward, represented by J ( θ c ) :
$$J ( \theta _ { c } ) = E _ { P ( a _ { 1 \colon T } ; \theta _ { c } ) } [ R ]$$
Since the reward signal R is non-differentiable, we need to use a policy gradient method to iteratively update θ c . In this work, we use the REINFORCE rule from Williams (1992):
$$\bigtriangledown _ { \theta _ { c } } J ( \theta _ { c } ) = \sum _ { t = 1 } ^ { T } E _ { P ( a _ { 1 \colon T } ; \theta _ { c } ) } \left [ \bigtriangledown _ { \theta _ { c } } \log P ( a _ { t } | a _ { ( t - 1 ) \colon 1 } ; \theta _ { c } ) R \right ]$$
An empirical approximation of the above quantity is:
$$\frac { 1 } { m } \sum _ { k = 1 } ^ { m } \sum _ { t = 1 } ^ { T } \bigtriangledown _ { \theta _ { c } } \log P ( a _ { t } | a _ { ( t - 1 ) \colon 1 } ; \theta _ { c } ) R _ { k }$$
Where m is the number of different architectures that the controller samples in one batch and T is the number of hyperparameters our controller has to predict to design a neural network architecture.
The validation accuracy that the k -th neural network architecture achieves after being trained on a training dataset is R k .
The above update is an unbiased estimate for our gradient, but has a very high variance. In order to reduce the variance of this estimate we employ a baseline function:
$$\frac { 1 } { m } \sum _ { k = 1 } ^ { m } \sum _ { t = 1 } ^ { T } \bigtriangledown _ { \theta _ { c } } \log P ( a _ { t } | a _ { ( t - 1 ) \colon 1 } ; \theta _ { c } ) ( R _ { k } - b )$$
As long as the baseline function b does not depend on the on the current action, then this is still an unbiased gradient estimate. In this work, our baseline b is an exponential moving average of the previous architecture accuracies.
Accelerate Training with Parallelism and Asynchronous Updates: In Neural Architecture Search, each gradient update to the controller parameters θ c corresponds to training one child network to convergence. As training a child network can take hours, we use distributed training and asynchronous parameter updates in order to speed up the learning process of the controller (Dean et al., 2012). We use a parameter-server scheme where we have a parameter server of S shards, that store the shared parameters for K controller replicas. Each controller replica samples m different child architectures that are trained in parallel. The controller then collects gradients according to the results of that minibatch of m architectures at convergence and sends them to the parameter server in order to update the weights across all controller replicas. In our implementation, convergence of each child network is reached when its training exceeds a certain number of epochs. This scheme of parallelism is summarized in Figure 3.
Figure 3: Distributed training for Neural Architecture Search. We use a set of S parameter servers to store and send parameters to K controller replicas. Each controller replica then samples m architectures and run the multiple child models in parallel. The accuracy of each child model is recorded to compute the gradients with respect to θ c , which are then sent back to the parameter servers.
<details>
<summary>Image 3 Details</summary>
### Visual Description
## Diagram: Distributed System Architecture
### Overview
The image depicts a distributed system architecture with three tiers: Parameter Servers, Controller Replicas, and Child Replicas. The diagram illustrates the flow of parameters and accuracy between these tiers.
### Components/Axes
* **Top Tier:** Parameter Servers (pink boxes) labeled "Parameter Server 1", "Parameter Server 2", and "Parameter Server S".
* **Middle Tier:** Controller Replicas (green boxes) labeled "Controller Replica 1", "Controller Replica 2", and "Controller Replica K".
* **Bottom Tier:** Child Replicas (light blue boxes) labeled "Child Replica 1", "Child Replica 2", and "Child Replica m".
* **Parameters:** Labeled as "Parameters" and denoted by "Θc" (Theta c), flowing from the Parameter Servers to the Controller Replicas.
* **Accuracy:** Labeled as "Accuracy" and denoted by "R", flowing from the Child Replicas to the Controller Replicas.
* Ellipses ("...") are used to indicate that there are more Controller Replicas and Child Replicas than explicitly shown.
### Detailed Analysis
* **Parameter Servers:** There are 'S' number of parameter servers. Each parameter server sends parameters (Θc) to each of the controller replicas.
* **Controller Replicas:** There are 'K' number of controller replicas. Each controller replica receives parameters from all parameter servers and accuracy information from its associated child replicas.
* **Child Replicas:** Each controller replica has 'm' number of child replicas associated with it. Each child replica sends accuracy information (R) to its respective controller replica.
* **Connections:**
* Each Parameter Server is connected to every Controller Replica.
* Each Controller Replica is connected to its 'm' Child Replicas.
### Key Observations
* The architecture is hierarchical, with Parameter Servers at the top, Controller Replicas in the middle, and Child Replicas at the bottom.
* The system is distributed, with multiple replicas of both Controllers and Child processes.
* The diagram highlights the flow of parameters from the Parameter Servers to the Controller Replicas and the flow of accuracy information from the Child Replicas to the Controller Replicas.
### Interpretation
The diagram illustrates a distributed machine learning or optimization system. Parameter Servers likely hold model parameters, Controller Replicas manage and coordinate the learning process, and Child Replicas perform computations or data processing. The flow of parameters (Θc) from the Parameter Servers to the Controller Replicas suggests a model distribution or parameter update mechanism. The flow of accuracy information (R) from the Child Replicas to the Controller Replicas indicates a feedback loop for evaluating performance and guiding the learning process. The replication of Controllers and Child processes suggests a design for scalability and fault tolerance. The architecture allows for parallel processing and distributed learning, potentially enabling faster training and better performance on large datasets.
</details>
## 3.3 INCREASE ARCHITECTURE COMPLEXITY WITH SKIP CONNECTIONS AND OTHER LAYER TYPES
In Section 3.1, the search space does not have skip connections, or branching layers used in modern architectures such as GoogleNet (Szegedy et al., 2015), and Residual Net (He et al., 2016a). In this section we introduce a method that allows our controller to propose skip connections or branching layers, thereby widening the search space.
To enable the controller to predict such connections, we use a set-selection type attention (Neelakantan et al., 2015) which was built upon the attention mechanism (Bahdanau et al., 2015; Vinyals et al., 2015). At layer N , we add an anchor point which has N -1 content-based sigmoids to indicate the previous layers that need to be connected. Each sigmoid is a function of the current hiddenstate of the controller and the previous hiddenstates of the previous N -1 anchor points:
$$P ( L a y e r \, j \, i s \, a n \, i n p u t \, t o l a y e r \, i ) = s i g m o i d ( v ^ { T } \tanh ( W _ { p r e v } * h _ { j } + W _ { c u r r } * h _ { i } ) ) ,$$
where h j represents the hiddenstate of the controller at anchor point for the j -th layer, where j ranges from 0 to N -1 . We then sample from these sigmoids to decide what previous layers to be used as inputs to the current layer. The matrices W prev , W curr and v are trainable parameters. As
these connections are also defined by probability distributions, the REINFORCE method still applies without any significant modifications. Figure 4 shows how the controller uses skip connections to decide what layers it wants as inputs to the current layer.
Figure 4: The controller uses anchor points, and set-selection attention to form skip connections.
<details>
<summary>Image 4 Details</summary>
### Visual Description
## Diagram: Neural Network Architecture with Skip Connections
### Overview
The image depicts a neural network architecture, specifically illustrating the connections between layers and the implementation of skip connections. The diagram shows three layers (N-1, N, and N+1) with interconnected nodes and labeled components such as "Number of Filters," "Anchor Point," "Filter Height," "Filter Width," and "Stride Height/Width." Skip connections are highlighted, bypassing one or more layers.
### Components/Axes
* **Layers:** Layer N-1, Layer N, Layer N+1 (arranged horizontally from left to right)
* **Nodes:** Represented by small squares, arranged in two rows within each layer.
* **Connections:** Solid lines indicate connections between adjacent nodes within a layer and between corresponding nodes in adjacent layers. Dotted lines represent skip connections.
* **Labels (Top Row):**
* Number of Filters (light blue)
* Anchor Point (uncolored)
* Filter Height (light pink)
* Filter Width (yellow)
* Stride Height (light green)
* Stride Width (light purple)
* Anchor Point (uncolored)
* Number of Filters (light blue)
* Filter Height (light pink)
* **Skip Connections:** Labeled as "N-1 skip connections" with arrows indicating the direction of the skip.
### Detailed Analysis
* **Layer Structure:** Each layer (N-1, N, N+1) consists of two rows of nodes. Nodes in each layer are connected horizontally to their neighbors.
* **Inter-Layer Connections:** Nodes in the top row of one layer are connected to the corresponding nodes in the top row of the next layer. Similarly, nodes in the bottom row are connected to corresponding nodes in the bottom row of the next layer.
* **Skip Connections:** The skip connections originate from nodes in Layer N-1 and bypass Layer N to connect to nodes in Layer N+1. These connections are represented by dotted lines that curve over Layer N.
* **Component Labels:** The labels above the top row of nodes indicate the function or parameter associated with that particular node or group of nodes. The colors of the labels alternate.
### Key Observations
* The diagram emphasizes the connectivity pattern between layers, including both direct connections and skip connections.
* The skip connections bypass one layer (N) and connect directly to the subsequent layer (N+1).
* The labels indicate that the network architecture involves parameters related to filters (height, width, number) and stride, suggesting a convolutional neural network.
### Interpretation
The diagram illustrates a neural network architecture that incorporates skip connections, a technique commonly used to address the vanishing gradient problem and improve the flow of information in deep networks. The skip connections allow the network to learn both shallow and deep features, potentially leading to better performance. The presence of filter-related parameters suggests that this architecture is likely used for image processing or other tasks involving convolutional operations. The diagram highlights the modularity of the network, with repeating blocks of "Number of Filters," "Anchor Point," "Filter Height," "Filter Width," and "Stride Height/Width" parameters.
</details>
In our framework, if one layer has many input layers then all input layers are concatenated in the depth dimension. Skip connections can cause 'compilation failures' where one layer is not compatible with another layer, or one layer may not have any input or output. To circumvent these issues, we employ three simple techniques. First, if a layer is not connected to any input layer then the image is used as the input layer. Second, at the final layer we take all layer outputs that have not been connected and concatenate them before sending this final hiddenstate to the classifier. Lastly, if input layers to be concatenated have different sizes, we pad the small layers with zeros so that the concatenated layers have the same sizes.
Finally, in Section 3.1, we do not predict the learning rate and we also assume that the architectures consist of only convolutional layers, which is also quite restrictive. It is possible to add the learning rate as one of the predictions. Additionally, it is also possible to predict pooling, local contrast normalization (Jarrett et al., 2009; Krizhevsky et al., 2012), and batchnorm (Ioffe & Szegedy, 2015) in the architectures. To be able to add more types of layers, we need to add an additional step in the controller RNN to predict the layer type, then other hyperparameters associated with it.
## 3.4 GENERATE RECURRENT CELL ARCHITECTURES
In this section, we will modify the above method to generate recurrent cells. At every time step t , the controller needs to find a functional form for h t that takes x t and h t -1 as inputs. The simplest way is to have h t = tanh( W 1 ∗ x t + W 2 ∗ h t -1 ) , which is the formulation of a basic recurrent cell. A more complicated formulation is the widely-used LSTM recurrent cell (Hochreiter & Schmidhuber, 1997).
The computations for basic RNN and LSTM cells can be generalized as a tree of steps that take x t and h t -1 as inputs and produce h t as final output. The controller RNN needs to label each node in the tree with a combination method (addition, elementwise multiplication, etc.) and an activation function ( tanh , sigmoid , etc.) to merge two inputs and produce one output. Two outputs are then fed as inputs to the next node in the tree. To allow the controller RNN to select these methods and functions, we index the nodes in the tree in an order so that the controller RNN can visit each node one by one and label the needed hyperparameters.
Inspired by the construction of the LSTM cell (Hochreiter & Schmidhuber, 1997), we also need cell variables c t -1 and c t to represent the memory states. To incorporate these variables, we need the controller RNN to predict what nodes in the tree to connect these two variables to. These predictions can be done in the last two blocks of the controller RNN.
To make this process more clear, we show an example in Figure 5, for a tree structure that has two leaf nodes and one internal node. The leaf nodes are indexed by 0 and 1, and the internal node is indexed by 2. The controller RNN needs to first predict 3 blocks, each block specifying a combination method and an activation function for each tree index. After that it needs to predict the last 2 blocks that specify how to connect c t and c t -1 to temporary variables inside the tree. Specifically,
Figure 5: An example of a recurrent cell constructed from a tree that has two leaf nodes (base 2) and one internal node. Left: the tree that defines the computation steps to be predicted by controller. Center: an example set of predictions made by the controller for each computation step in the tree. Right: the computation graph of the recurrent cell constructed from example predictions of the controller.
<details>
<summary>Image 5 Details</summary>
### Visual Description
## Diagram: Neural Network Architectures
### Overview
The image presents three diagrams illustrating different neural network architectures or components. The first diagram shows a tree-like structure, the second depicts a recurrent neural network (RNN) cell with various operations, and the third shows a graph-like representation of operations within a cell.
### Components/Axes
**Diagram 1: Tree Structure**
* **Nodes:** Labeled "Tree Index 0", "Tree Index 1", and "Tree Index 2". These are rectangular boxes.
* **Inputs:** Labeled "h<sub>t-1</sub>" and "x<sub>t</sub>" at the bottom, feeding into "Tree Index 0" and "Tree Index 1".
* **Output:** Labeled "h<sub>t</sub>" at the top, originating from "Tree Index 2".
* **Connections:** Arrows indicate the flow of information from inputs to tree nodes and from lower-level tree nodes to higher-level ones.
**Diagram 2: Recurrent Neural Network (RNN) Cell**
* **Horizontal Flow:** A series of interconnected rectangular boxes, representing the flow of information through time steps.
* **Vertical Operations:** Above each rectangular box in the top row are colored squares representing operations:
* Orange: "Add"
* Dark Red: "Tanh"
* Blue: "Elem Mult" (Element-wise Multiplication)
* Pink: "ReLU" (Rectified Linear Unit)
* Green: "Sigmoid"
* White: "1" and "0"
* **Recurrent Connections:** Dashed lines with arrows indicate feedback loops, connecting the output of certain operations back to earlier stages.
* **Input Labels:** "x<sub>t</sub>" below the first set of boxes.
* **Sections:** Labeled "Tree Index 0", "Tree Index 1", "Tree Index 2", "Cell Inject", and "Cell Indices" to indicate different parts of the RNN cell.
**Diagram 3: Graph Representation**
* **Nodes:** Circles representing operations, colored according to the same scheme as in Diagram 2:
* Orange: "add"
* Dark Red: "tanh"
* Blue: "elem_mult"
* Pink: "relu"
* Green: "sigmoid"
* White: Input/Output nodes
* **Inputs:** Labeled "x<sub>t</sub>", "h<sub>t-1</sub>", and "c<sub>t-1</sub>".
* **Output:** Labeled "h<sub>t</sub>" and "c<sub>t</sub>".
* **Connections:** Arrows indicate the flow of information between operations.
### Detailed Analysis
**Diagram 1: Tree Structure**
* The diagram represents a hierarchical processing of inputs h<sub>t-1</sub> and x<sub>t</sub>.
* "Tree Index 0" processes h<sub>t-1</sub> and x<sub>t</sub>.
* "Tree Index 1" processes h<sub>t-1</sub> and x<sub>t</sub>.
* "Tree Index 2" combines the outputs of "Tree Index 0" and "Tree Index 1" to produce h<sub>t</sub>.
**Diagram 2: Recurrent Neural Network (RNN) Cell**
* The diagram illustrates the sequential processing of information in an RNN cell.
* The horizontal flow represents the progression through time steps.
* The vertical operations (Add, Tanh, Elem Mult, ReLU, Sigmoid) represent transformations applied to the data at each time step.
* The recurrent connections (dashed lines) allow the network to retain information from previous time steps.
* The "Cell Inject" and "Cell Indices" sections likely represent mechanisms for injecting external information or controlling the cell's behavior.
**Diagram 3: Graph Representation**
* The diagram shows the interconnections between different operations within a neural network cell.
* The colors of the nodes correspond to the types of operations (Add, Tanh, Elem Mult, ReLU, Sigmoid).
* The arrows indicate the flow of data between these operations.
* The graph structure highlights the complex dependencies and interactions between different components of the cell.
### Key Observations
* The diagrams represent different levels of abstraction in neural network architectures.
* Diagram 1 shows a tree-like structure, Diagram 2 shows an RNN cell with sequential processing and recurrent connections, and Diagram 3 shows a graph representation of operations within a cell.
* The use of color-coding in Diagrams 2 and 3 helps to visually distinguish different types of operations.
### Interpretation
The diagrams illustrate the complexity and diversity of neural network architectures. The tree structure in Diagram 1 may represent a hierarchical processing of data, while the RNN cell in Diagram 2 shows how information can be processed sequentially and retained over time. The graph representation in Diagram 3 highlights the intricate relationships between different operations within a cell. These diagrams are useful for understanding the underlying principles and design choices in neural networks. The consistent color-coding across diagrams helps to connect the different levels of abstraction and understand the relationships between different components.
</details>
according to the predictions of the controller RNN in this example, the following computation steps will occur:
- The controller predicts Add and Tanh for tree index 0, this means we need to compute a 0 = tanh( W 1 ∗ x t + W 2 ∗ h t -1 ) .
- The controller predicts ElemMult and ReLU for tree index 1, this means we need to compute a 1 = ReLU ( ( W 3 ∗ x t ) ( W 4 ∗ h t -1 ) ) .
- The controller predicts 0 for the second element of the 'Cell Index', Add and ReLU for elements in 'Cell Inject', which means we need to compute a new 0 = ReLU( a 0 + c t -1 ) . Notice that we don't have any learnable parameters for the internal nodes of the tree.
- The controller predicts ElemMult and Sigmoid for tree index 2, this means we need to compute a 2 = sigmoid( a new 0 a 1 ) . Since the maximum index in the tree is 2, h t is set to a 2 .
- The controller RNN predicts 1 for the first element of the 'Cell Index', this means that we should set c t to the output of the tree at index 1 before the activation, i.e., c t = ( W 3 ∗ x t ) ( W 4 ∗ h t -1 ) .
In the above example, the tree has two leaf nodes, thus it is called a 'base 2' architecture. In our experiments, we use a base number of 8 to make sure that the cell is expressive.
## 4 EXPERIMENTS AND RESULTS
We apply our method to an image classification task with CIFAR-10 and a language modeling task with Penn Treebank, two of the most benchmarked datasets in deep learning. On CIFAR-10, our goal is to find a good convolutional architecture whereas on Penn Treebank our goal is to find a good recurrent cell. On each dataset, we have a separate held-out validation dataset to compute the reward signal. The reported performance on the test set is computed only once for the network that achieves the best result on the held-out validation dataset. More details about our experimental procedures and results are as follows.
## 4.1 LEARNING CONVOLUTIONAL ARCHITECTURES FOR CIFAR-10
Dataset: In these experiments we use the CIFAR-10 dataset with data preprocessing and augmentation procedures that are in line with other previous results. We first preprocess the data by whitening all the images. Additionally, we upsample each image then choose a random 32x32 crop of this upsampled image. Finally, we use random horizontal flips on this 32x32 cropped image.
Search space: Our search space consists of convolutional architectures, with rectified linear units as non-linearities (Nair & Hinton, 2010), batch normalization (Ioffe & Szegedy, 2015) and skip connections between layers (Section 3.3). For every convolutional layer, the controller RNN has to select a filter height in [1, 3, 5, 7], a filter width in [1, 3, 5, 7], and a number of filters in [24, 36, 48,
64]. For strides, we perform two sets of experiments, one where we fix the strides to be 1, and one where we allow the controller to predict the strides in [1, 2, 3].
Training details: The controller RNN is a two-layer LSTM with 35 hidden units on each layer. It is trained with the ADAM optimizer (Kingma & Ba, 2015) with a learning rate of 0.0006. The weights of the controller are initialized uniformly between -0.08 and 0.08. For the distributed training, we set the number of parameter server shards S to 20, the number of controller replicas K to 100 and the number of child replicas m to 8, which means there are 800 networks being trained on 800 GPUs concurrently at any time.
Once the controller RNN samples an architecture, a child model is constructed and trained for 50 epochs. The reward used for updating the controller is the maximum validation accuracy of the last 5 epochs cubed. The validation set has 5,000 examples randomly sampled from the training set, the remaining 45,000 examples are used for training. The settings for training the CIFAR-10 child models are the same with those used in Huang et al. (2016a). We use the Momentum Optimizer with a learning rate of 0.1, weight decay of 1e-4, momentum of 0.9 and used Nesterov Momentum (Sutskever et al., 2013).
During the training of the controller, we use a schedule of increasing number of layers in the child networks as training progresses. On CIFAR-10, we ask the controller to increase the depth by 2 for the child models every 1,600 samples, starting at 6 layers.
Results: After the controller trains 12,800 architectures, we find the architecture that achieves the best validation accuracy. We then run a small grid search over learning rate, weight decay, batchnorm epsilon and what epoch to decay the learning rate. The best model from this grid search is then run until convergence and we then compute the test accuracy of such model and summarize the results in Table 1. As can be seen from the table, Neural Architecture Search can design several promising architectures that perform as well as some of the best models on this dataset.
| Model | Depth | Parameters | Error rate (%) |
|-----------------------------------------------------|----------|--------------|------------------|
| Network in Network (Lin et al., 2013) | - | - | 8.81 |
| All-CNN (Springenberg et al., 2014) | - | - | 7.25 |
| Deeply Supervised Net (Lee et al., 2015) | - | - | 7.97 |
| Highway Network (Srivastava et al., 2015) | - | - | 7.72 |
| Scalable Bayesian Optimization (Snoek et al., 2015) | - | - | 6.37 |
| FractalNet (Larsson et al., 2016) | 21 | 38.6M | 5.22 |
| with Dropout/Drop-path | 21 | 38.6M | 4.60 |
| ResNet (He et al., 2016a) | 110 | 1.7M | 6.61 |
| ResNet (reported by Huang et al. (2016c)) | 110 | 1.7M | 6.41 |
| ResNet with Stochastic Depth (Huang et al., 2016c) | 110 | 1.7M | 5.23 |
| Wide ResNet (Zagoruyko &Komodakis, 2016) | 16 | 11.0M | 4.81 |
| | 28 | 36.5M | 4.17 |
| ResNet (pre-activation) (He et al., 2016b) | 164 1001 | 1.7M 10.2M | 5.46 |
| DenseNet ( L = 40 ,k = 12) Huang et al. (2016a) | | 1.0M | 4.62 5.24 |
| | 40 | | |
| DenseNet ( L = 100 ,k = 12) Huang et al. (2016a) | 100 | 7.0M | 4.10 |
| DenseNet ( L = 100 ,k = 24) Huang et al. (2016a) | 100 | 27.2M | 3.74 |
| DenseNet-BC ( L = 100 ,k = 40) Huang et al. (2016b) | 190 | 25.6M | 3.46 |
| Neural Architecture Search v1 no stride or pooling | 15 | 4.2M | 5.50 |
| Neural Architecture Search v2 predicting strides | 20 | 2.5M | 6.01 |
| Neural Architecture Search v3 max pooling | 39 | 7.1M | 4.47 |
| Neural Architecture Search v3 max pooling + more | 39 | | 3.65 |
| filters | | 37.4M | |
Table 1: Performance of Neural Architecture Search and other state-of-the-art models on CIFAR-10.
First, if we ask the controller to not predict stride or pooling, it can design a 15-layer architecture that achieves 5.50% error rate on the test set. This architecture has a good balance between accuracy and depth. In fact, it is the shallowest and perhaps the most inexpensive architecture among the top performing networks in this table. This architecture is shown in Appendix A, Figure 7. A notable feature of this architecture is that it has many rectangular filters and it prefers larger filters at the top layers. Like residual networks (He et al., 2016a), the architecture also has many one-step skip connections. This architecture is a local optimum in the sense that if we perturb it, its performance becomes worse. For example, if we densely connect all layers with skip connections, its performance becomes slightly worse: 5.56%. If we remove all skip connections, its performance drops to 7.97%.
In the second set of experiments, we ask the controller to predict strides in addition to other hyperparameters. As stated earlier, this is more challenging because the search space is larger. In this case, it finds a 20-layer architecture that achieves 6.01% error rate on the test set, which is not much worse than the first set of experiments.
Finally, if we allow the controller to include 2 pooling layers at layer 13 and layer 24 of the architectures, the controller can design a 39-layer network that achieves 4.47% which is very close to the best human-invented architecture that achieves 3.74%. To limit the search space complexity we have our model predict 13 layers where each layer prediction is a fully connected block of 3 layers. Additionally, we change the number of filters our model can predict from [24, 36, 48, 64] to [6, 12, 24, 36]. Our result can be improved to 3.65% by adding 40 more filters to each layer of our architecture. Additionally this model with 40 filters added is 1.05x as fast as the DenseNet model that achieves 3.74%, while having better performance. The DenseNet model that achieves 3.46% error rate (Huang et al., 2016b) uses 1x1 convolutions to reduce its total number of parameters, which we did not do, so it is not an exact comparison.
## 4.2 LEARNING RECURRENT CELLS FOR PENN TREEBANK
Dataset: Weapply Neural Architecture Search to the Penn Treebank dataset, a well-known benchmark for language modeling. On this task, LSTM architectures tend to excel (Zaremba et al., 2014; Gal, 2015), and improving them is difficult (Jozefowicz et al., 2015). As PTB is a small dataset, regularization methods are needed to avoid overfitting. First, we make use of the embedding dropout and recurrent dropout techniques proposed in Zaremba et al. (2014) and (Gal, 2015). We also try to combine them with the method of sharing Input and Output embeddings, e.g., Bengio et al. (2003); Mnih & Hinton (2007), especially Inan et al. (2016) and Press & Wolf (2016). Results with this method are marked with 'shared embeddings.'
Search space: Following Section 3.4, our controller sequentially predicts a combination method then an activation function for each node in the tree. For each node in the tree, the controller RNN needs to select a combination method in [ add, elem mult ] and an activation method in [ identity, tanh, sigmoid, relu ] . The number of input pairs to the RNN cell is called the 'base number' and set to 8 in our experiments. When the base number is 8, the search space is has approximately 6 × 10 16 architectures, which is much larger than 15,000, the number of architectures that we allow our controller to evaluate.
Training details: The controller and its training are almost identical to the CIFAR-10 experiments except for a few modifications: 1) the learning rate for the controller RNN is 0.0005, slightly smaller than that of the controller RNN in CIFAR-10, 2) in the distributed training, we set S to 20, K to 400 and m to 1, which means there are 400 networks being trained on 400 CPUs concurrently at any time, 3) during asynchronous training we only do parameter updates to the parameter-server once 10 gradients from replicas have been accumulated.
In our experiments, every child model is constructed and trained for 35 epochs. Every child model has two layers, with the number of hidden units adjusted so that total number of learnable parameters approximately match the 'medium' baselines (Zaremba et al., 2014; Gal, 2015). In these experiments we only have the controller predict the RNN cell structure and fix all other hyperparameters. The reward function is c (validation perplexity) 2 where c is a constant, usually set at 80.
After the controller RNN is done training, we take the best RNN cell according to the lowest validation perplexity and then run a grid search over learning rate, weight initialization, dropout rates
and decay epoch. The best cell found was then run with three different configurations and sizes to increase its capacity.
Results: In Table 2, we provide a comprehensive list of architectures and their performance on the PTB dataset. As can be seen from the table, the models found by Neural Architecture Search outperform other state-of-the-art models on this dataset, and one of our best models achieves a gain of almost 3.6 perplexity. Not only is our cell is better, the model that achieves 64 perplexity is also more than two times faster because the previous best network requires running a cell 10 times per time step (Zilly et al., 2016).
Table 2: Single model perplexity on the test set of the Penn Treebank language modeling task. Parameter numbers with ‡ are estimates with reference to Merity et al. (2016).
| Model | Parameters | Test Perplexity |
|--------------------------------------------------------------|--------------|-------------------|
| Mikolov &Zweig (2012) - KN-5 | 2M ‡ | 141 . 2 |
| Mikolov &Zweig (2012) - KN5 + cache | 2M ‡ | 125 . 7 |
| Mikolov &Zweig (2012) - RNN | 6M ‡ | 124 . 7 |
| Mikolov &Zweig (2012) - RNN-LDA | 7M ‡ | 113 . 7 |
| Mikolov &Zweig (2012) - RNN-LDA + KN-5 + cache | 9M ‡ | 92 . 0 |
| Pascanu et al. (2013) - Deep RNN | 6M | 107 . 5 |
| Cheng et al. (2014) - Sum-Prod Net | 5M ‡ | 100 . 0 |
| Zaremba et al. (2014) - LSTM (medium) | 20M | 82 . 7 |
| Zaremba et al. (2014) - LSTM (large) | 66M | 78 . 4 |
| Gal (2015) - Variational LSTM (medium, untied) | 20M | 79 . 7 |
| Gal (2015) - Variational LSTM (medium, untied, MC) | 20M | 78 . 6 |
| Gal (2015) - Variational LSTM (large, untied) | 66M | 75 . 2 |
| Gal (2015) - Variational LSTM (large, untied, MC) | 66M | 73 . 4 |
| Kim et al. (2015) - CharCNN | 19M | 78 . 9 |
| Press &Wolf (2016) - Variational LSTM, shared embeddings | 51M | 73 . 2 |
| Merity et al. (2016) - Zoneout + Variational LSTM (medium) | 20M | 80 . 6 |
| Merity et al. (2016) - Pointer Sentinel-LSTM (medium) | 21M | 70 . 9 |
| Inan et al. (2016) - VD-LSTM + REAL (large) | 51M | 68 . 5 |
| Zilly et al. (2016) - Variational RHN, shared embeddings | 24M | 66 . 0 |
| Neural Architecture Search with base 8 | 32M | 67 . 9 |
| Neural Architecture Search with base 8 and shared embeddings | 25M | 64 . 0 |
| Neural Architecture Search with base 8 and shared embeddings | 54M | 62 . 4 |
The newly discovered cell is visualized in Figure 8 in Appendix A. The visualization reveals that the new cell has many similarities to the LSTM cell in the first few steps, such as it likes to compute W 1 ∗ h t -1 + W 2 ∗ x t several times and send them to different components in the cell.
Transfer Learning Results: To understand whether the cell can generalize to a different task, we apply it to the character language modeling task on the same dataset. We use an experimental setup that is similar to Ha et al. (2016), but use variational dropout by Gal (2015). We also train our own LSTM with our setup to get a fair LSTM baseline. Models are trained for 80K steps and the best test set perplexity is taken according to the step where validation set perplexity is the best. The results on the test set of our method and state-of-art methods are reported in Table 3. The results on small settings with 5-6M parameters confirm that the new cell does indeed generalize, and is better than the LSTM cell.
Additionally, we carry out a larger experiment where the model has 16.28M parameters. This model has a weight decay rate of 1 e -4 , was trained for 600K steps (longer than the above models) and the test perplexity is taken where the validation set perplexity is highest. We use dropout rates of 0.2 and 0.5 as described in Gal (2015), but do not use embedding dropout. We use the ADAM optimizer with a learning rate of 0.001 and an input embedding size of 128. Our model had two layers with 800 hidden units. We used a minibatch size of 32 and BPTT length of 100. With this setting, our model achieves 1.214 perplexity, which is the new state-of-the-art result on this task.
Finally, we also drop our cell into the GNMT framework (Wu et al., 2016), which was previously tuned for LSTM cells, and train an WMT14 English → German translation model. The GNMT
Table 3: Comparison between our cell and state-of-art methods on PTB character modeling. The new cell was found on word level language modeling.
| RNN Cell Type | Parameters | Test Bits Per Character |
|----------------------------------------------------------|--------------|---------------------------|
| Ha et al. (2016) - Layer Norm HyperLSTM | 4.92M | 1.25 |
| Ha et al. (2016) - Layer Norm HyperLSTM Large Embeddings | 5.06M | 1.233 |
| Ha et al. (2016) - 2-Layer Norm HyperLSTM | 14.41M | 1.219 |
| Two layer LSTM | 6.57M | 1.243 |
| Two Layer with New Cell | 6.57M | 1.228 |
| Two Layer with New Cell | 16.28M | 1.214 |
network has 8 layers in the encoder, 8 layers in the decoder. The first layer of the encoder has bidirectional connections. The attention module is a neural network with 1 hidden layer. When a LSTM cell is used, the number of hidden units in each layer is 1024. The model is trained in a distributed setting with a parameter sever and 12 workers. Additionally, each worker uses 8 GPUs and a minibatch of 128. We use Adam with a learning rate of 0.0002 in the first 60K training steps, and SGD with a learning rate of 0.5 until 400K steps. After that the learning rate is annealed by dividing by 2 after every 100K steps until it reaches 0.1. Training is stopped at 800K steps. More details can be found in Wu et al. (2016).
In our experiment with the new cell, we make no change to the above settings except for dropping in the new cell and adjusting the hyperparameters so that the new model should have the same computational complexity with the base model. The result shows that our cell, with the same computational complexity, achieves an improvement of 0.5 test set BLEU than the default LSTM cell. Though this improvement is not huge, the fact that the new cell can be used without any tuning on the existing GNMT framework is encouraging. We expect further tuning can help our cell perform better.
Control Experiment 1 - Adding more functions in the search space: To test the robustness of Neural Architecture Search, we add max to the list of combination functions and sin to the list of activation functions and rerun our experiments. The results show that even with a bigger search space, the model can achieve somewhat comparable performance. The best architecture with max and sin is shown in Figure 8 in Appendix A.
Control Experiment 2 - Comparison against Random Search: Instead of policy gradient, one can use random search to find the best network. Although this baseline seems simple, it is often very hard to surpass (Bergstra & Bengio, 2012). We report the perplexity improvements using policy gradient against random search as training progresses in Figure 6. The results show that not only the best model using policy gradient is better than the best model using random search, but also the average of top models is also much better.
Figure 6: Improvement of Neural Architecture Search over random search over time. We plot the difference between the average of the top k models our controller finds vs. random search every 400 models run.
<details>
<summary>Image 6 Details</summary>
### Visual Description
## Line Chart: Perplexity Improvement vs. Iteration
### Overview
The image is a line chart showing the perplexity improvement versus iteration for three different models: Top_1_unique_models, Top_5_unique_models, and Top_15_unique_models. The chart illustrates how the perplexity improvement changes as the number of iterations increases for each model.
### Components/Axes
* **X-axis:** Iteration, ranging from 0 to 25000.
* **Y-axis:** Perplexity Improvement, ranging from 0 to 40.
* **Legend (top-left):**
* Blue: Top\_1\_unique\_models
* Green: Top\_5\_unique\_models
* Red: Top\_15\_unique\_models
### Detailed Analysis
* **Top\_1\_unique\_models (Blue):**
* The line starts at approximately 0 at iteration 0.
* It increases to approximately 5 by iteration 2500.
* It fluctuates between 3 and 7 until around iteration 12500.
* It then gradually increases to approximately 9 by iteration 20000, where it plateaus.
* **Top\_5\_unique\_models (Green):**
* The line starts at approximately 0 at iteration 0.
* It rapidly increases to approximately 9 by iteration 1000.
* It increases to approximately 13 by iteration 10000.
* It gradually increases to approximately 18 by iteration 17500, where it plateaus.
* **Top\_15\_unique\_models (Red):**
* The line starts at approximately 0 at iteration 0.
* It rapidly increases to approximately 20 by iteration 1000.
* It increases to approximately 25 by iteration 2500.
* It gradually increases to approximately 32 by iteration 17500.
* It increases to approximately 36 by iteration 22500.
### Key Observations
* The Top\_15\_unique\_models consistently shows the highest perplexity improvement across all iterations.
* The Top\_5\_unique\_models shows a moderate perplexity improvement, consistently above Top\_1\_unique\_models.
* The Top\_1\_unique\_models shows the lowest perplexity improvement and exhibits more fluctuation compared to the other two models.
* All three models show a trend of increasing perplexity improvement with increasing iterations, but the rate of improvement decreases as the iterations increase.
### Interpretation
The chart suggests that increasing the number of unique models considered (from Top 1 to Top 15) leads to a higher perplexity improvement. This indicates that considering a larger set of unique models results in better performance. The diminishing rate of improvement with increasing iterations suggests that there is a point of diminishing returns, where further iterations do not significantly improve the perplexity. The fluctuations in the Top\_1\_unique\_models line might indicate instability or sensitivity to specific iterations. Overall, the data demonstrates the impact of model selection and iteration count on perplexity improvement.
</details>
## 5 CONCLUSION
In this paper we introduce Neural Architecture Search, an idea of using a recurrent neural network to compose neural network architectures. By using recurrent network as the controller, our method is flexible so that it can search variable-length architecture space. Our method has strong empirical performance on very challenging benchmarks and presents a new research direction for automatically finding good neural network architectures. The code for running the models found by the controller on CIFAR-10 and PTB will be released at https://github.com/tensorflow/models . Additionally, we have added the RNN cell found using our method under the name NASCell into TensorFlow, so others can easily use it.
## ACKNOWLEDGMENTS
We thank Greg Corrado, Jeff Dean, David Ha, Lukasz Kaiser and the Google Brain team for their help with the project.
## REFERENCES
- Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. In NAACL , 2016.
- Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. arXiv preprint arXiv:1606.04474 , 2016.
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR , 2015.
- Yoshua Bengio, R´ ejean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. JMLR , 2003.
- James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. JMLR , 2012.
- James Bergstra, R´ emi Bardenet, Yoshua Bengio, and Bal´ azs K´ egl. Algorithms for hyper-parameter optimization. In NIPS , 2011.
- James Bergstra, Daniel Yamins, and David D Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. ICML , 2013.
- Alan W. Biermann. The inference of regular LISP programs from examples. IEEE transactions on Systems, Man, and Cybernetics , 1978.
- Wei-Chen Cheng, Stanley Kok, Hoai Vu Pham, Hai Leong Chieu, and Kian Ming Adam Chai. Language modeling with sum-product networks. In INTERSPEECH , 2014.
- Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR , 2005.
- Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V. Le, et al. Large scale distributed deep networks. In NIPS , 2012.
- Dario Floreano, Peter D¨ urr, and Claudio Mattiussi. Neuroevolution: from architectures to learning. Evolutionary Intelligence , 2008.
- Yarin Gal. A theoretically grounded application of dropout in recurrent neural networks. arXiv preprint arXiv:1512.05287 , 2015.
- David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016a.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027 , 2016b.
- Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine , 2012.
- Sepp Hochreiter and Juergen Schmidhuber. Long short-term memory. Neural Computation , 1997.
- Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993 , 2016a.
- Gao Huang, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993 , 2016b.
- Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382 , 2016c.
- Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv preprint arXiv:1611.01462 , 2016.
- Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML , 2015.
- Kevin Jarrett, Koray Kavukcuoglu, Yann Lecun, et al. What is the best multi-stage architecture for object recognition? In ICCV , 2009.
- Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network architectures. In ICML , 2015.
- Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. Character-aware neural language models. arXiv preprint arXiv:1508.06615 , 2015.
- Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS , 2012.
- Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science , 2015.
- Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648 , 2016.
- Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 1998.
- Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeplysupervised nets. In AISTATS , 2015.
- Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv:1606.01885 , 2016.
- Percy Liang, Michael I. Jordan, and Dan Klein. Learning programs: A hierarchical Bayesian approach. In ICML , 2010.
- Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR , 2013.
- David G. Lowe. Object recognition from local scale-invariant features. In CVPR , 1999.
- Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Towards automatically-tuned neural networks. In Proceedings of the 2016 Workshop on Automatic Machine Learning , pp. 58-65, 2016.
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 , 2016.
- Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In SLT , pp. 234-239, 2012.
- Andriy Mnih and Geoffrey Hinton. Three new graphical models for statistical language modelling. In ICML , 2007.
- Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML , 2010.
- Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. Neural programmer: Inducing latent programs with gradient descent. In ICLR , 2015.
- Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6026 , 2013.
- Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859 , 2016.
- Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732 , 2015.
- Scott Reed and Nando de Freitas. Neural programmer-interpreters. In ICLR , 2015.
- Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics. In NIPS , 2016.
- Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In ACL , 2016.
- Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.
- Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical Bayesian optimization of machine learning algorithms. In NIPS , 2012.
- Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mostofa Ali, Ryan P. Adams, et al. Scalable bayesian optimization using deep neural networks. In ICML , 2015.
- Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 , 2014.
- Rupesh Kumar Srivastava, Klaus Greff, and J¨ urgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387 , 2015.
- Kenneth O. Stanley, David B. D'Ambrosio, and Jason Gauci. A hypercube-based encoding for evolving large-scale neural networks. Artificial Life , 2009.
- Phillip D. Summers. A methodology for LISP program construction from examples. Journal of the ACM , 1977.
- Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In ICML , 2013.
- Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS , 2014.
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR , 2015.
- Sebastian Thrun and Lorien Pratt. Learning to learn . Springer Science & Business Media, 2012.
- Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS , 2015.
- Daan Wierstra, Faustino J Gomez, and J¨ urgen Schmidhuber. Modeling systems with internal state using evolino. In GECCO , 2005.
- Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning , 1992.
- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 , 2016.
- Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC , 2016.
- Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 , 2014.
- Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutn´ ık, and J¨ urgen Schmidhuber. Recurrent highway networks. arXiv preprint arXiv:1607.03474 , 2016.
## A APPENDIX
Figure 7: Convolutional architecture discovered by our method, when the search space does not have strides or pooling layers. FH is filter height, FW is filter width and N is number of filters. Note that the skip connections are not residual connections. If one layer has many input layers then all input layers are concatenated in the depth dimension.
<details>
<summary>Image 7 Details</summary>
### Visual Description
## Diagram: Neural Network Architecture
### Overview
The image depicts a neural network architecture diagram. It shows the flow of data from an "Image" input through multiple layers with varying filter heights (FH), filter widths (FW), and number of nodes (N), culminating in a "Softmax" output. The diagram illustrates connections between layers, indicating a complex network structure.
### Components/Axes
* **Nodes:** Represented as rectangular boxes, each labeled with "FH: x FW: y N: z", where x, y, and z are numerical values.
* **Input:** Labeled "Image" (blue box) at the bottom of the diagram.
* **Output:** Labeled "Softmax" (red box) at the top of the diagram.
* **Connections:** Represented as black arrows indicating the flow of data between nodes.
### Detailed Analysis
The diagram consists of 16 layers, including the input and output layers. The layers are arranged vertically, with the input layer at the bottom and the output layer at the top. Each layer is connected to one or more layers above it, forming a complex network structure.
Here's a breakdown of the layers and their parameters, starting from the bottom:
1. **Image** (Input Layer, blue box)
2. **FH: 3 FW: 3 N: 36**
3. **FH: 3 FW: 3 N: 48**
4. **FH: 3 FW: 3 N: 36**
5. **FH: 5 FW: 5 N: 36**
6. **FH: 3 FW: 7 N: 48**
7. **FH: 7 FW: 7 N: 48**
8. **FH: 7 FW: 7 N: 48**
9. **FH: 7 FW: 3 N: 36**
10. **FH: 7 FW: 1 N: 36**
11. **FH: 7 FW: 7 N: 36**
12. **FH: 5 FW: 7 N: 36**
13. **FH: 7 FW: 7 N: 48**
14. **FH: 7 FW: 5 N: 48**
15. **FH: 7 FW: 5 N: 48**
16. **FH: 7 FW: 5 N: 48**
17. **Softmax** (Output Layer, red box)
The connections between layers are complex and not strictly sequential. Many layers have connections to multiple layers above them, suggesting skip connections or dense connections.
### Key Observations
* The filter height (FH), filter width (FW), and number of nodes (N) vary across different layers.
* The diagram shows a complex network architecture with multiple connections between layers.
* The input layer is labeled "Image" and the output layer is labeled "Softmax".
### Interpretation
The diagram represents a neural network architecture designed for image processing. The varying filter sizes and number of nodes in each layer suggest that the network is designed to extract features at different scales and levels of abstraction. The complex connections between layers indicate a sophisticated network structure that may be capable of learning complex patterns in the input data. The use of "Softmax" as the output layer suggests that the network is designed for classification tasks. The skip connections or dense connections may help to improve the network's performance by allowing it to learn more efficiently and avoid vanishing gradients.
</details>
Figure 8: A comparison of the original LSTM cell vs. two good cells our model found. Top left: LSTM cell. Top right: Cell found by our model when the search space does not include max and sin . Bottom: Cell found by our model when the search space includes max and sin (the controller did not choose to use the sin function).
<details>
<summary>Image 8 Details</summary>
### Visual Description
## Diagram: Recurrent Neural Network Cell Variations
### Overview
The image presents three variations of a recurrent neural network (RNN) cell, likely Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) cells, visualized as computational graphs. Each cell takes inputs *x<sub>t</sub>*, *h<sub>t-1</sub>*, and *c<sub>t-1</sub>* and produces outputs *h<sub>t</sub>* and *c<sub>t</sub>*. The diagrams illustrate the flow of data through different operations within each cell, such as sigmoid, tanh, element-wise multiplication (elem_mult), and addition (add).
### Components/Axes
Each cell diagram contains the following components:
* **Inputs:**
* *x<sub>t</sub>*: Input at time *t*.
* *h<sub>t-1</sub>*: Hidden state at time *t-1*.
* *c<sub>t-1</sub>*: Cell state at time *t-1*.
* **Outputs:**
* *h<sub>t</sub>*: Hidden state at time *t*.
* *c<sub>t</sub>*: Cell state at time *t*.
* **Operations:**
* `sigmoid`: Sigmoid activation function. Represented by green nodes.
* `tanh`: Hyperbolic tangent activation function. Represented by brown/dark red nodes.
* `elem_mult`: Element-wise multiplication. Represented by blue nodes.
* `add`: Addition. Represented by orange nodes.
* `identity`: Identity function. Represented by yellow nodes.
* `relu`: Rectified Linear Unit activation function. Represented by pink nodes.
* `max`: Maximum function. Represented by red nodes.
### Detailed Analysis
**Cell 1 (Top-Left)**
* Input *x<sub>t</sub>* is fed into three `add` operations.
* Input *h<sub>t-1</sub>* is fed into three `add` operations.
* The outputs of these `add` operations are then passed through `tanh`, `sigmoid`, and `sigmoid` activations (green, brown, green).
* The outputs of the `tanh` and `sigmoid` activations are combined using `elem_mult` (blue).
* The output of the `elem_mult` is then passed through an `identity` function (yellow) to produce *h<sub>t</sub>*.
* Input *c<sub>t-1</sub>* is passed through an `identity` function (yellow) and then combined with the output of another `elem_mult` (blue) via an `add` operation (orange).
* The output of this `add` operation is passed through an `identity` function (yellow) to produce *c<sub>t</sub>*.
**Cell 2 (Top-Right)**
* Input *x<sub>t</sub>* is fed into four `add` operations.
* Input *h<sub>t-1</sub>* is fed into four `add` operations.
* The outputs of these `add` operations are then passed through `tanh`, `sigmoid`, `sigmoid`, and `tanh` activations (brown, green, green, brown).
* Input *c<sub>t-1</sub>* is fed into four `add` operations.
* The outputs of these `add` operations are then passed through `relu`, `sigmoid`, `sigmoid`, and `relu` activations (pink, green, green, pink).
* The outputs of the `tanh` and `sigmoid` activations are combined using `elem_mult` (blue).
* The output of the `elem_mult` is then passed through a `tanh` activation (brown) to produce *h<sub>t</sub>*.
* Input *c<sub>t-1</sub>* is passed through an `identity` function (yellow) and then combined with the output of another `elem_mult` (blue) via an `add` operation (orange).
* The output of this `add` operation is passed through an `elem_mult` function (blue) to produce *c<sub>t</sub>*.
**Cell 3 (Bottom)**
* Input *x<sub>t</sub>* is fed into four `add` operations.
* Input *h<sub>t-1</sub>* is fed into three `max` operations (red).
* Input *c<sub>t-1</sub>* is fed into three `max` operations (red).
* The outputs of these `add` operations are then passed through `tanh`, `identity`, `relu`, and `tanh` activations (brown, yellow, pink, brown).
* The outputs of the `tanh` and `sigmoid` activations are combined using `elem_mult` (blue).
* The output of the `elem_mult` is then passed through a `tanh` activation (brown) to produce *h<sub>t</sub>*.
* Input *c<sub>t-1</sub>* is passed through an `identity` function (yellow) and then combined with the output of another `elem_mult` (blue) via an `add` operation (orange).
* The output of this `add` operation is passed through an `identity` function (yellow) to produce *c<sub>t</sub>*.
### Key Observations
* Each cell processes inputs *x<sub>t</sub>*, *h<sub>t-1</sub>*, and *c<sub>t-1</sub>* differently, using various combinations of activation functions and operations.
* The `elem_mult` operation is a key component in combining the outputs of different activation functions.
* The `add` operation is used to combine the inputs with the outputs of previous operations.
* The `identity` function is used to pass the cell state *c<sub>t-1</sub>* to the next time step.
* The variations in the cells likely represent different architectures or design choices for RNNs.
### Interpretation
The diagrams illustrate the computational flow within three different RNN cell architectures. These cells likely represent variations of LSTM or GRU cells, which are designed to address the vanishing gradient problem in traditional RNNs. The different combinations of activation functions, element-wise multiplications, and additions allow each cell to learn different patterns in the input data. The presence of cell state *c<sub>t</sub>* allows the network to store and propagate information across time steps, enabling it to capture long-range dependencies in sequential data. The variations in the cells likely reflect different trade-offs between computational complexity and performance on specific tasks.
</details>