Neural Architecture Search (NAS) for Computer Vision Models

Overview 

Deep learning based specifically in neural networks is an area that has grown and advanced significantly in the last few years specifically when it comes to the processing of signals from computer vision. However, the selection of a precise neural network for a particular task can be a tedious and time-consuming job. Well, this is where Neural Architecture Search, also commonly known as NAS, steps in operations. Therefore, NAS intelligently avoids the process of designing the structure of a neural network and can minimize the amount of human effort to develop top performing computer vision models.  

In this blog, we will learn how NAS works, why it is revolutionary for computer vision, and how it is utilized to design better models. 

What is Neural Architecture Search (NAS)? 

Neural Architecture Search (NAS) is an autonomous procedure for finding suitable neural network architecture. NAS stands for neural architecture search, which digitally creates architectures using known spaces and seeks out the most effective model for the task at hand. The idea is to find an approximate or near-best design that provides high accuracy and low computational cost. 

NAS has three main components: 

Search Space

Defines the possible neural architectures the algorithm explores, including aspects like the number of layers, layer types, filter sizes, and activation functions

Search Strategy

Specifies the method NAS uses to navigate the search space, typically involving reinforcement learning, evolutionary algorithms, or gradient-based techniques

Evaluation Strategy

Evaluates the selected architecture’s performance using indicators like error rate, often aiming for efficient estimation methods without fully training the model

NAS has been particularly impactful in the field of computer vision, where neural networks are applied to tasks like object detection, image classification, and semantic segmentation. 

Why is NAS Crucial for Computer Vision? 

This is because designing high-performance neural networks for computer vision systems is not trivial because of the involved task complexities and enhanced possibilities of architectural options available. Conventional neural network architecture design would require a lot of know-how, time, and guesswork. 

Some reasons why NAS is crucial for computer vision include: 

• Automation of Architecture Design: First, NAS eliminates the need for time-consuming and labor-intensive architecture searches through manual cross-experiments. This is especially helpful in computer vision in which the required architectures are usually the best and deepest.  

• Optimizing for Accuracy and Efficiency: NAS can search for architectures that achieve the highest accuracy and the highest accuracy per computational operation required. This is especially important for real-world applications where bringing the models to edge devices or smartphones is resource-constrained.  

• Transferability: NAS can prescribe architectures that can be generalized and can be used for different computer vision tasks. For instance, an architecture discovered for image classification can be transferred to object detection or segmentation. 

NAS Strategies for Computer Vision Models 

There are several strategies that researchers use to perform NAS. Each strategy varies in how it navigates the search space and evaluates performance. Below are some of the most common NAS methods used for computer vision: 

1. Reinforcement Learning-based NAS 

The first techniques used for addressing NAS include Reinforcement Learning(RL) to identify models of superior performance. In this regard, an agent (controller) creates potential architectures for a deep neural network, and then the quality of the underlying architecture is determined by its efficiency in solving a problem such as image classification. By maximizing a reward, more often the model accuracy, the agent is trained to create better architectures. 

• Example: In the NASNet framework developed by Google, RL was used to search for a neural network architecture that achieved state-of-the-art results on the ImageNet dataset. NASNet outperformed many hand-designed models like ResNet and Inception-v4. 

2. Evolutionary Algorithms 

Evolutionary algorithms are with reference to the evolutionary model and genetic algorithms. This approach begins from a pool of randomly initialized neural architectures that are appropriately initialized. These architectures are then “evolved” over successive generations through mechanisms such as mutation – in which some aspects of the architecture are changed – and crossover – in which parts of two architectures are taken. The only architectures that are retained are the ones that yield the best performance, while the other architectures that do not yield positive results are eliminated. 

• Example: Google's AmoebaNet uses evolutionary algorithms to find optimal architectures. It has produced highly efficient architectures for image classification, outperforming RL-based methods in some cases. 

3. Gradient-Based NAS 

Despite the richness of reinforcement learning and evolutionary approaches, they are computationally demanding mainly because they entail building and training a large number of candidate models. The most related idea to Gradient-based NAS is the differentiable architecture search in which the search space is simply a convex space. 

• Example: The DARTS (Differentiable Architecture Search) framework is a gradient-based method that significantly reduces the time and resources required for architecture search. DARTS uses a continuous relaxation of the architecture search space, allowing architectures to be optimized using gradient descent. 

This method is particularly useful for computer vision tasks, where architectures tend to be deep and complex, and training each candidate model fully can be prohibitively expensive.