To get AI agents to develop general intelligence, the ability to solve different and novel problems, it is likely that we will need to give them a general goal and the ability to learn how to progress towards this goal from their actions and the resulting feedback. To solve this problem, we will first have to answer the fundamental question: what should this general goal be?