What do all intelligent beings have in common? Four things, according to Meta’s chief AI scientist, Yann LeCun.
At the AI Action Summit in Paris earlier this year, political leaders and AI experts gathered to discuss AI development. LeCun shared his baseline definition of intelligence with IBM’s AI leader, Anthony Annunziata.
“There’s four essential characteristics of intelligent behavior that every animal, or relatively smart animal, can do, and certainly humans,” he said. “Understanding the physical world, having persistent memory, being able to reason, and being able to plan, and planning complex actions, particularly planning hierarchically.”
LeCun said AI, especially large language models, have not hit this threshold, and incorporating these capabilities would require a shift in how they are trained. That’s why many of the biggest tech companies are cobbling capabilities onto existing models in their race to dominate the AI game, he said.
“For understanding the physical world, well, you train a separate vision system. And then you bolt it on the LLM. For memory, you know, you use RAG, or you bolt some associative memory on top of it, or you just make your model bigger,” he said. RAG, which stands for retrieval augmented generation, is a way to enhance the outputs of large language models using external knowledge sources. It was developed at Meta.
All those, however, are just “hacks,” LeCun said.
LeCun has spoken on several occasions about an alternative he calls world-based models. These are models trained on real-life scenarios and have higher levels of cognition than pattern-based AI. LeCun, in his chat with Annunziata, offered another definition.
“You have some idea of the state of the world at time T, you imagine an action it might take, the world model predicts what the state of the world is going to be from the action you took,” he said.
But, he said, the world evolves according to an infinite and unpredictable set of possibilities, and the only way to train for them is through abstraction.
Meta is already experimenting with this through V-JEPA, a model it released to the public in February. Meta describes it as a non-generative model that learns by predicting missing or masked parts of a video.
“The basic idea is that you don’t predict at the pixel level. You train a system to run an abstract representation of the video so that you can make predictions in that abstract representation, and hopefully this representation will eliminate all the details that cannot be predicted,” he said.
The concept is similar to how chemists established a fundamental hierarchy for the building blocks of matter.
“We created abstractions. Particles, on top of this, atoms, on top of this, molecules, on top of this, materials,” he said. “Every time we go up one layer, we eliminate a lot of information about the layers below that are irrelevant for the type of task we’re interested in doing.”
That, in essence, is another way of saying we’ve learned to make sense of the physical world by creating hierarchies.
