British mathematician Jack Good coined the term “intelligence explosion” 61 years ago to describe what would happen when an intelligent machine entered a runaway cycle of fully automated self-improvement, quickly leaving human intelligence far behind. For decades, that hypothetical capability – often described as “recursive self-improvement” (RSI) – has been seen as AI’s holy grail. The logic goes that the first country or company to achieve RSI would leave its competitors in the dust, cementing...