Not everyone in AI is ready to declare the end of the age of scaling.
“I’m not convinced it’s completely over,” Geoffrey Hinton, the “Godfather of AI,” recently told Business Insider, weighing in on one of the hottest debates in AI circles this year.
Hinton is aware that OpenAI cofounder Ilya Sutskever, one of his former students, said last month that the pendulum of AI development is swinging back toward research — and away from companies simply making breakthroughs by scaling, or acquiring more compute and more chips.
“Is the belief really: ‘Oh, it’s so big, but if you had 100x more, everything would be so different?’ It would be different, for sure. But is the belief that if you just 100x the scale, everything would be transformed? I don’t think that’s true,” Sutskever said on an episode of the “Dwarkesh Podcast.”
“So it’s back to the age of research again, just with big computers,” added Sutskever, who is now running his own AI startup.
Hinton said there will always be a need for more data. (Another issue facing scaling is the finite amount of high-quality data.) He predicted that the large chatbots will start generating their own data, as Google DeepMind’s AlphaGo and AlphaZero do on a much smaller scale to master the board game Go.
“Nobody’s worried about a lack of data because it plays against itself and generates data that way,” Hinton said of the early program. “And the equivalent for a language model is when it starts reasoning and saying, ‘Look, I believe these things and these things imply that thing, but I don’t believe that thing, so I’d better change something somewhere.’ And by doing reasoning to check the consistency of his own beliefs, it can generate a lot more data.”
Scaling is at the very core of Big Tech’s capex spending spree, a bet based on the belief that by acquiring more compute or training data, AI models will continue to grow smarter and more advanced.
Increasingly, some AI leaders have expressed uncertainty about making their future bets based on their trust in scaling. Alexandr Wang, now head of Meta’s superintelligence division, said in 2024 that scaling is “the biggest question in the industry.”
Yann LeCun, who worked with Hinton on pioneering AI research, has also challenged the extent of the scale doctrine.
“You cannot just assume that more data and more compute means smarter AI,” LeCun said in April when he was still Meta’s chief AI scientist. Like Sutskever, LeCun has since launched his own startup.
Sutskever said scaling has been attractive because it allows companies to make a “very low-risk way” bet on AI advancements.
In contrast, Google DeepMind CEO Demis Hassabis said that scaling laws could ultimately unlock the biggest and most elusive prize in AI: artificial general intelligence, or AGI.
“The scaling of the current systems, we must push that to the maximum, because at the minimum, it will be a key component of the final AGI system,” Hassabis said at the Axios’ AI+ Summit in December. “It could be the entirety of the AGI system.”
