By Anthony Aguirre
It’s hard to really imagine what an AGI could be capable of. For a start, imagine a system that can speak 50 languages, has expertise in all academic subjects, reads a full book in seconds and can operate ten times faster than you. Existing Tool AI systems can already do much of that. To a fully-fledged AGI, human minds might seem very slow and limited - and to a superintelligence our minds would appear more like a house plant than a colleague.
Imagine you are a newly-born AGI, loyally committed to carrying out your creator's will. But your creator is a potted fern. Perhaps it has left you some detailed notes, and you can ask it one question a week. Even so, you'll need to make a lot of judgement calls on your own. Is the fern truly in control? Even if your intentions are good, the gulf between your time scales and capabilities is simply too vast for the fern to exercise meaningful control over your actions.