Automatically Discovering Learning Algorithms with Hardware Constraints

Can learning emerge from a search process in-silico? Our AutoML-Zero work at Google DeepMind shows that a simple evolutionary search process can automatically discover modern learning techniques from scratch. Without knowledge of machine learning the method discovers algorithms by combining simple primitives such as additions and multiplications into a functional learning algorithm. In this type of evolutionary search learning emerges naturally when algorithm survival depends on its performance on multiple tasks. These discovery methods transfer to realistic setups where we can find novel algorithms for ML optimization and robot adaptation. Importantly we can meaningfully shape the properties of the discovered algorithm by constraining the environment on which the algorithm evolves. I will speculate that using hardware abstractions as such a constraint is a promising direction for finding new paradigms of neural computation.