Evolutionary Neural Architecture Search using Random Weight Distributions

We consider the problem of efficiently searching for high-performing neural architectures whilst simultaneously favouring networks of reduced complexity. It is theorised that a complementary set of proxies can be employed for multi-objective optimisation to balance model performance with the size of the network. We demonstrate that a low-cost proxy for the test accuracy of a candidate architecture can be derived from a series of inferences alone. The proxy is paired with a complexity metric based on the number of parameters in the model and the two properties are used in a multi-objective setting. A Pareto Archived Evolutionary Strategy is used to optimise the two objectives simultaneously and deliver a diverse collection of solutions as output. This method is shown to successfully discover low-complexity architectures with minor loss of accuracy as compared to the global optima and does so with statistical reliability. This work offers a proof-of-concept Neural Architecture Search algorithm that removes training from the process entirely. The proposed approach is examined in terms of search behaviour and the complexity reduction that can be achieved by comparing discovered solutions to the top-performing architectures in the search space.