The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats… In this survey, we seek to unify the field of AutoRL, provide a common taxonomy, discuss each area in detail and pose open problems of interest to researchers going forward.