Nvidia’s biggest competitive advantage is no longer as untouchable as it once seemed.
For two decades, Nvidia’s crown jewel wasn’t just chips; it was the software that turned them into the building blocks of AI, known as CUDA.
Short for Compute Unified Device Architecture, CUDA is the brainchild of longtime Nvidia executive Ian Buck, who heads high-performance computing. It took years to build, with ready-made code for common AI tasks, tools to find bugs, and software that lets thousands of chips work together to train models.
Now, some believe AI could eventually automate one of the industry’s hardest jobs: building the software that powers AI itself.
The industry is at an important threshold, says Jeremy Nixon, a former Google Brain researcher and the founder of AI software startup Infinity. He told Business Insider his startup used AI coding agents to recreate CUDA-like software for the chip startup D-Matrix in 10 hours — evidence, he said, that one of Nvidia’s biggest moats is being crossed.
Infinity.inc
The pressure doesn’t only come from startups. Cloud giants like Google, Amazon, and Microsoft have spent years building software around their own AI chips, while OpenAI and Anthropic have recently demonstrated AI models capable of generating system software.
DeepSeek founder Liang Wenfeng recently said that coding agents, along with his startup’s own programming language TileLang, have made AI software substantially easier to build.
Coding agents aren’t just helping challengers.
Nvidia said developers increasingly use CUDA’s code libraries to build AI applications, and that it also “uses AI coding agents to develop CUDA faster and validate at greater scale,” said Ankit Patel, Nvidia’s vice president of developer ecosystem.
Inference could change the CUDA equation
If CUDA’s first advantage was software, the second is everything built on top of it. Millions of lines of code and internal workflows have been developed by companies, creating a powerful lock-in effect that makes switching to alternatives costly and cumbersome.
Internal documents at Amazon identified CUDA as a major roadblock to adoption of its Trainium and Inferentia AI chips, Business Insider previously reported.
CUDA’s age is both an advantage and a constraint, said Chris Lattner, cofounder and CEO of Qualcomm-owned AI software startup Modular. Originally built for gaming long before the AI boom, CUDA carries layers of legacy technology, “like Microsoft Windows trying to fit onto a phone,” he said.
Modular
Others say AI’s shift from training toward inference — where models answer requests and draw conclusions — creates another threat.
With this evolution, companies care less about maximizing performance with the most powerful chips and more about running AI profitably, said Marshall Choy, chief business officer of Korean AI chip startup Rebellions.
This could result in greater demand not only for specialized hardware but for software that can run across different chips. If companies can switch between chips without rewriting software, that reduces one of CUDA’s biggest lock-ins.
“That’s where the CUDA moat from Nvidia gets broken because CUDA is no longer a factor in the inference side,” Choy said. “It’s an open source play.”
Nvidia said that its tightly integrated hardware and software offerings have become more valuable as AI models get put to work.
“As AI shifts toward inference and agentic workloads, the need for deep, full-stack optimization only grows,” Patel said.
A shift toward specialized chips and software has Wall Street increasingly questioning Nvidia’s CUDA advantage, said Luke Lango, chief technology analyst at InvestorPlace. He said Nvidia’s stagnant stock price over the past year reflects some of these concerns.
Nvidia’s moat isn’t disappearing — it’s shifting
Not everyone agrees that coding agents are eroding CUDA’s edge. Some believe they could ultimately strengthen it instead.
Though agents make it easier to generate software, AI-generated code still has to be verified and optimized, said Bing Xu, founder of AI software startup INT21. He believes CUDA has the deepest ecosystem of verification tools and other features that help coding agents work more efficiently.
As agents become more common, he said, that ecosystem will become CUDA’s next moat.
“Agents can generate a lot of code in a short time, but verification is the biggest bottleneck,” said Xu, whose last AI chip software startup, HippoML, was acquired by Nvidia. He left the chipmaker in April to build INT21.
INT21
While coding agents do make it easier to build chip software, the improvement is incremental, Lattner said.
“The hype is not complete nonsense, but it is very overblown,” he said, adding that writing code is only a small part of building software compared to more complex tasks like optimizing it for production — a critical task because software that maximizes a chip’s performance reduces the cost of running AI at scale.
Chip software is also something of a niche field, often worked on by elite engineers, Lattner said, giving coding agents far fewer examples to learn from than, for instance, app development, where AI has been trained on vast amounts of public code.
And while AI may help rivals catch up, Nvidia benefits from the same technological shifts, Xu said. Whether coding agents weaken CUDA depends on whether competitors catch Nvidia faster than it can gain new ground.
The world’s dominant chipmaker is “not sleeping or keeping still,” Xu said.
