Couldn’t we just turn AGI off if it started behaving in a way that we didn’t like?

As AI systems become more capable and integrated into our lives, they will be harder to control. Someday we may find that our AI systems can no more be ‘turned off’ than social media, or the internet itself.

Any highly competent goal-oriented agent will want to acquire more power and resources. These 'instrumental goals' will help it to pursue its ultimate goals. So will keeping itself from being turned off. It might approach this by trying to expand its abilities, making copies of itself, lying, and so on.

This is not just theoretical. In the right circumstances, even today’s ‘passive’ AI systems have been shown to consistently engage in such behaviors. They behave deceptively, attempt to escape and replicate themselves, and try to avoid being shut down or retrained.