Neural Architecture Search (NAS)

Search Space

The search space defines the type and range of neural network architectures that can be designed and optimized by NAS. The search space can be divided into two levels: the cell level and the network level²

Cell Level

The cell level search space consists of a set of basic operations, such as convolution, pooling, activation, or skip connection, that can be applied to the input tensors of a cell. A cell is a small sub-network that can be repeated multiple times in a larger network. The goal of cell level NAS is to find the best combination and order of operations within a cell²

Network Level

The network level search space consists of a set of hyperparameters, such as the number of cells, the number of layers, the width and depth of the network, the learning rate, the dropout rate, or the weight decay, that affect the overall performance of the network. The goal of network level NAS is to find the best values for these hyperparameters²

Search Strategy

The search strategy defines the approach used to explore the search space and find the best architecture. The search strategy can be categorized into three types: reinforcement learning (RL), evolutionary algorithms (EA), and gradient-based methods²

Reinforcement Learning

Reinforcement learning (RL) is a type of machine learning that learns from its own actions and rewards. RL can be used to train an agent, such as a recurrent neural network (RNN), to generate neural network architectures based on a reward function, such as validation accuracy or inference speed. The agent iteratively samples architectures from its policy distribution, evaluates them on a given task, and updates its policy based on the feedback. RL-based NAS methods can generate diverse and novel architectures, but they are often computationally expensive and require a large number of trials² ³

Evolutionary Algorithms

Evolutionary algorithms (EA) are a type of optimization technique that mimics natural evolution. EA can be used to maintain a population of candidate architectures that are randomly initialized or mutated from previous generations. The candidates are evaluated on a given task and assigned a fitness score based on their performance. The best candidates are selected to produce offspring for the next generation. EA-based NAS methods can explore a large and complex search space efficiently, but they may suffer from premature convergence or stagnation² ⁴

Gradient-Based Methods

Gradient-based methods are a type of optimization technique that uses the gradient information of the objective function to update the parameters. Gradient-based methods can be used to optimize both the architecture and the weights of a neural network simultaneously by using differentiable representations of the architecture, such as continuous relaxation, one-shot models, or super-networks. Gradient-based NAS methods can leverage existing optimization algorithms and frameworks, but they may introduce approximation errors or regularization issues² ⁵

Performance Estimation Strategy

The performance estimation strategy defines how to evaluate the quality of a candidate architecture without fully training and testing it on a given task. The performance estimation strategy can reduce the computational cost and time of NAS significantly. The performance estimation strategy can be categorized into three types: proxy tasks, early stopping, and prediction models²

Proxy Tasks

Proxy tasks are simplified versions of the original task that can be used to estimate the performance of an architecture quickly and accurately. Proxy tasks can include using smaller datasets, lower resolutions, fewer classes, or shorter sequences than the original task. Proxy tasks can also include using surrogate tasks that are related but easier than the original task, such as image segmentation for image classification or language modeling for machine translation. Proxy tasks can provide useful feedback for NAS, but they may not reflect the true performance on the original task²

Early Stopping

Early stopping is a technique that stops the training process of an architecture before it reaches convergence based on some criteria, such as validation loss or accuracy. Early stopping can save computational resources and prevent overfitting by avoiding unnecessary training epochs. Early stopping can also be combined with learning curve extrapolation or ranking preservation to improve its accuracy and stability. Early stopping can provide fast and reliable performance estimation for NAS, but it may depend on the choice of hyperparameters and initialization²

Prediction Models

Prediction models are machine learning models that learn to predict the performance of an architecture based on its features, such as the number of parameters, the depth, the width, or the operations. Prediction models can be trained on a large and diverse set of architectures that have been evaluated on a given task. Prediction models can provide instant and consistent performance estimation for NAS, but they may suffer from bias or variance due to the limited size and quality of the training data²

Conclusion

Neural architecture search (NAS) is a technique that uses a model, such as a neural network, to search for the best architecture or design for another model, such as another neural network. NAS can automate the process of finding the optimal neural network architecture for a given task, such as image classification, natural language processing, or reinforcement learning. NAS can potentially discover more complex and efficient architectures than human experts, and has already achieved state-of-the-art results on many benchmarks. NAS involves three main components: the search space, the search strategy, and the performance estimation strategy. Each component can be implemented in different ways, depending on the task, the resources, and the preferences of the user. NAS is a rapidly evolving field that offers many challenges and opportunities for future research and applications.

¹: Neural Architecture Search: Insights from 1000 Papers ²: Neural architecture search - Wikipedia ³: Neural Architecture Search with Reinforcement Learning ⁴: Large-Scale Evolution of Image Classifiers ⁵: DARTS: Differentiable Architecture Search : Progressive Neural Architecture Search : Efficient Neural Architecture Search via Parameter Sharing : Learning Transferable Architectures for Scalable Image Recognition

Neural Architecture Search (NAS)