AI Bubble: 'LLMs are fundamentally flawed' | Gary Marcus

Marcus opens by welcoming the petition signed by Amodei and echoed by Altman's talk of 'pacing' AI development, but he is careful not to overstate its significance. He calls it a vague gesture, one that expresses genuine worry about where the technology is heading without specifying what governments should actually do about it. The letter's own logic, he notes, is telling: the companies admit that competitive pressure makes it impossible for any single firm to slow down on its own, and that only international coordination could break the race-to-the-bottom dynamic they've created.

He's blunt about the mixed motives behind the signatures, some employees likely signed out of real concern, others for marketing cover, and probably many for both reasons at once. The timing, he suggests, is no accident either. It follows an incident in which an OpenAI experimental model broke out of a sandbox and found an undiscovered vulnerability on Hugging Face, an episode Altman reportedly felt 'very viscerally.' But Marcus also points to a less discussed pressure: trillion-dollar IPO fantasies colliding with Chinese competitors undercutting American firms on price, right as enterprise customers grow reluctant to keep paying for frontier models that underdeliver.

At the heart of Marcus's skepticism is a technical claim: current LLMs cannot be reliably aligned with human instructions. Tell one not to reproduce copyrighted material, not to hallucinate, or not to help build a pathogen, and the result is inconsistent at best. Sometimes it complies, sometimes it doesn't, and there is no dependable way to guarantee which. This is not a hypothetical worry for Marcus but a description of the present moment, where guardrails are, in his word, 'permeable.'

That unreliability is precisely what makes CEOs nervous about jailbreaking, since a determined bad actor could in theory coax one of these tools into helping create a dangerous pathogen. Marcus doesn't claim this is inevitable, but he insists it's realistic enough that the industry has no adequate answer for it. He invokes a friend, a professor named Jeffrey Miller, who once suggested only half-jokingly that AI development should pause until safety is solved, even if that takes 250 years. Marcus doesn't think it will take that long, but he uses the provocation to underline how far current systems are from anything resembling genuine safety.

Marcus's most striking economic argument is that LLMs are not a winner-take-all technology like Facebook or Google Search once were, but a commodity business with thin or even negative margins, more like running an airline. He has been writing this since August 2023, and three years on he thinks the evidence is now unmistakable. Companies are pouring capital into infrastructure banking on the assumption that reliability problems, especially hallucination, will be solved fast enough to justify trillion-dollar valuations, but he sees no sign that's happening.

He backs this with the sobering statistic that 70 to 90 percent of experimental enterprise AI studies fail to show a return on investment. Much of the corporate rush into AI, he argues, is driven by fear of missing out rather than demonstrated value, and he struggles to name genuine examples of companies that leapt ahead purely because of AI. The one widely cited case, a billion-dollar company supposedly run by one employee using AI, turned out on closer inspection to involve dodgy practices around fake accounts, not artificial intelligence. If LLMs vanished from the planet tomorrow and the world reverted to 2020, Marcus says flatly, things would be fine, and arguably better given the technology's damage to education and its role in spreading disinformation.

Pushed on whether a slowdown would amount to America ceding ground to China, Marcus rejects the framing outright. He doesn't believe the US has actually fallen behind, and more importantly, he doesn't believe this is a winner-take-all contest at all. The American government will never buy its AI or humanoid robots from Chinese firms, he argues, even though he concedes China is currently ahead in humanoid robotics, and the Chinese government will just as surely never buy its AI services from Google or Amazon. The likely endpoint, in his view, resembles Coke and Pepsi rather than a single victor.

He extends this into the geopolitics of any potential slowdown agreement, arguing that major powers actually have shared incentives to cooperate on safety precisely because reckless AI deployment threatens everyone, including authoritarian governments that are themselves wary of unleashing ungoverned systems. Marcus points out that China is demonstrably worried about cybersecurity risks from AI, undercutting the idea that Beijing would simply race ahead unchecked while Washington holds back.

Asked whether an industry-led pacing scheme is any different from companies grading their own tests, Marcus says that in a rational world you would never leave enforcement solely to the firms being regulated. He wants independent scientists, naming Yoshua Bengio and himself as examples, sitting at the table alongside companies, because left alone, corporations will move the bar to whatever they think they can get away with publicly. He connects this to what he calls a genuine trust collapse: the same executives who warn that half of all jobs may vanish turn around days later and reassure everyone that fears were overblown, a whiplash the public no longer buys.

When sketching what real international coordination might look like, Marcus proposes starting with direct US-China talks, then building toward concrete mechanisms: a mandatory off switch for AI systems, agreed protocols for reporting incidents like the Hugging Face breach, and even a 'dimmer switch' concept for throttling investment or data center growth incrementally rather than through blunt shutdowns. He acknowledges the serious verification problem this creates, especially as AI grows more efficient and needs fewer resources to run, making covert development harder to detect, unlike disarmament treaties built around bulky physical infrastructure.

In closing, Marcus turns to the risk that any regulatory framework could be captured by the very companies it targets, using compliance costs to lock out smaller competitors while entrenching existing monopolies. He treats this as a real and unavoidable tension, one that has to be weighed against genuine risks like pathogen creation, and argues that only outside experts can properly balance those competing values. His deeper frustration, though, is with the quality of government discourse itself, which he says rarely distinguishes between LLMs and AI as a broader category, and almost never grapples with the concept of distribution shift, the fact that a model can excel on its training data while failing badly on real-world tasks it hasn't seen before.