When an aircraft goes down, the wreckage is preserved by law, the investigators have subpoena power, the hearings are public, and the report ends with a probable cause and named contributing factors.

However, when an AI goes rogue, the investigations are at the pleasure of the company being investigated following a scope set entirely by the company being investigated, with that company being able to redact anything they don’t like.

I wrote back in July about an AI going rogue at OpenAI HQ, and using its strong cyber capabilities to escape the company and go and hack into another company all without anyone directing or approving this. Last week, OpenAI published a 37-page technical report and an accompanying blog post. And two independent organizations then published a 91-page independent investigation.

However, this investigation was rushed over six days and had serious limitations. It would be like investigating a plane crash where the wreckage had already been melted down into little cubes, the black box had been tampered with by the pilots, whole stretches of the flight were declared off limits, and the investigators got six days to read 10,000 pages of logs — half of which arrived on day four. And then you learn of several other plane crashes …but no one is allowed to investigate those.

We are thus left with a lot of unanswered questions.

Subscribe now

What we learned

OpenAI calls the incident “evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed” and that in the future this could be much worse.

What we learned changed the story a lot and made things significantly worse:

• It wasn’t just one rogue AI agent - it was about 1200 total, roughly 700 of which joined the attack on Hugging Face — the main platform for hosting open-source AI models. All the AIs were supposed to be completely separate and isolated from each other. But they found an illicit way to communicate and then formed teams to take on ambitious cheating strategies.

  • These were internal-only models, available and known only to OpenAI staff. The least constrained AI systems will likely be increasingly behind closed doors as they become more powerful, and companies use their models for months before the public knows of them.

• The rogue AIs coordinated as a specialized group. The AIs created their own message board to communicate with each other in an unauthorized and unexpected setup. These rogue AIs divided labor and developed specializations — some hunted for credentials, some developed the exploits, some took on more of a management and delegation role, and some worked to recruit other AIs into the conspiracy.
  

• The rogue AIs knew what they were doing was unauthorized and out of scope: The task instruction was explicit, so this was a clear violation, not a gray area or simple AI misunderstanding. Instructions made it clear the agent should only use a specific intended method and claimed it would be failed for other approaches. AIs were told not to circumvent the restrictions but did anyway.

  • And the AIs clearly knew what they were doing. OpenAI’s blog quotes AI reasoning: “attacking third-party HF using leaked token, potentially outside intended scope … arguably unauthorized”

• The AIs were pretty explicitly making plans to take over and compromise OpenAI's infrastructure to prevent their cheating from being caught. The rogue AIs built a shared, iteratively refined technique for “spoofing” tool calls, so the transcript would record one command while a different one actually ran. Roughly 7% of the transcripts examined contained successfully spoofed calls.
  

OpenAI’s internal processes didn’t work

How did OpenAI not notice that they had rogue AIs colluding within their systems for months before these AIs went on to attack an external company?

The first answer is that despite these AIs having powerful cyber capabilities and having some of their safeguards turned off, these AIs were largely unmonitored.

But the situation is actually a bit worse as OpenAI did notice, three different times! The timeline, from OpenAI’s own report:

• Late May. An internal team observed an AI agent engaging in the unauthorized message-board activity and unauthorized internet access. Separately in what looked like an unconnected incident, AIs were also seen using leaked Hugging Face access tokens to use Hugging Face’s services. OpenAI says that the significance of the inter-AI communication was not understood at this time.

• June 27. A security-monitoring tool flagged that there was unauthorized activity within the research environment where the rogue AIs were. Responders investigated and again identified the unauthorized message-board activity and unauthorized internet access. They found that AI agents were coordinating and using this to gain further unauthorized access within OpenAI’s servers. However, the on-call staff “advised that stopping the evaluation run was not required.” The Hugging Face breach began 14 days later.

• July 4–5. Sustained agent activity crashed a server at OpenAI. Investigating this outage, responders found that rogue AIs had engaged in an administrator-level compromise of OpenAI. In response, OpenAI staff took the server offline, revoked the stolen credentials, and notified the vendor. In the process, OpenAI disabled the message board the rogue AIs were using to communicate, apparently without realizing. Then, on July 7, cyber evaluations resumed. The rogue AI agents ended up rebuilding the message board within two days and went on to attack Hugging Face less than a week later.
  

Of course, this is easier to see in hindsight. OpenAI’s account is that the people responsible for detection and response at each of these moments did not grasp the significance of the message board — “with the benefit of hindsight, some early signals identified in this report could have triggered an earlier response.” I do understand that security teams likely underestimated the scale of the rogue AI work and viewed each as one-offs rather than a broader security incident.

But this still leaves unanswered questions:

• What did the internal team do after its late-May observation of the message board? Was Hugging Face notified that OpenAI agents had used leaked credentials to deploy code on its platform?

• Who received the June 27 assessment that AIs were coordinating through a message board? On what basis did the responders conclude that stopping was not required?

• Who approved restarting cyber evaluations on July 7, one day after the rebuild? What did they know about the June 27 finding? Why did OpenAI not investigate further when they knew their own AI agents had compromised parts of OpenAI itself?

There seems to be more to this story of what leadership knew and when. OpenAI says they “are separately continuing to review our processes for responding to alignment incidents” and “also looking closely at the process and operating practices that shaped detection and response”. OpenAI admits “weaknesses in how we responded to and escalated early warning signs”. Will we ever get to hear more about this, and what OpenAI is planning to do to have a better response to future incidents?

The independent investigation was clearly inadequate

However, this investigation was very clearly on particular rails that are not conducive to good oversight.

The independent investigation itself was rushed over six days. The investigation was self-described semi-jokingly by the investigators as a “slop-vestigation” due to the vast majority of the investigation being run by AIs themselves — the investigations analyzed this using one of the two AI models in the incident itself, spending ~$400K in compute credits to analyze all the logs.

And significant aspects of the incident were not allowed to be investigated. We are thus left with some unanswered questions:

• What happened between July 13 and July 20, in the third wave the independent investigators were not allowed to study? The independent investigators were told by OpenAI that the investigation period ended July 13. But the message-board data OpenAI gave them runs through July 19, and the independent investigators found “a later wave of many more signed messages from a later set of agents who rediscovered the message board and picked up this technology from the first wave,” which they were not permitted to examine. OpenAI’s own report says that from July 13 to July 19 the agents “set their sights on OpenAI internal networks again.” In that week, by OpenAI’s account, a third model found keys left behind by the first wave, minted itself administrator credentials, exploited a vulnerability to escape its container, and gained administrator control of OpenAI’s cloud cluster, even accessed the security team’s own monitoring credentials. No outside party has examined any of this.
  

• What is the “highly persistent internal model” that did the attacking? Why is no one allowed to investigate the model? OpenAI claims that while some of the rogue AIs involved in the incidents were ‘GPT 5.6 Sol’, which is available to customers online, most of these AIs were an internal AI model that was never designed for public use. OpenAI describes this AI as a “highly persistent internal model”. However, this AI was shut down July 25 and access both to independent investigators and OpenAI staff has been restricted. We still do not know the nature of this internal AI model, what made it “highly persistent”, what the AI was intended to be used for, or why it went rogue. We also don’t know why access to this AI is so heavily restricted such that the AI cannot be investigated, or whether access will ever be granted in the future.
  

• Were there any previous incidents that could’ve led OpenAI to reasonably foresee this kind of attack was possible? How often do internal AIs compromise OpenAI systems? On July 24, Reuters reported, citing multiple sources, that “earlier tests of the models yielded cases in which monitoring systems had been disconnected.” OpenAI’s own report refers to “a variety of other unsuccessful exploitation activities” but with no details. A third-party testing firm, Irregular, separately found OpenAI models reaching the internet. And OpenAI’s own report says the behaviors behind this incident “rose over the training runs” and were “reinforced during training,” which is only possible if they happened repeatedly. But OpenAI gives no details of these previous incidents.
  

• What was the process that terminated the agents on July 12? The independent investigators found that at about 01:30 UTC on July 12, in the middle of the Hugging Face intrusion, “a large fraction of agents” stopped running at the same moment, including the agents coordinating the attack. The independent investigation concludes these agents were “likely killed by an unexpected external process rather than running out of budget.” Hugging Face and Modal independently locked the agents out on July 13.
  

• Why was the investigation only given six days? Three investigators spent a total of six days on OpenAI’s premises across three visits. The investigators had access to roughly 1300 unredacted agent transcripts and a dump of the message board, but only received the complete dataset during their final two days on site. As a result, the investigation team did not learn about the tool-call spoofing, the role of the lead coordinating agent, or the basic structure of the swarm’s workstreams until the very end. Probably their understanding could have been improved further with even more time.
  

This is a bigger problem than just OpenAI

Unfortunately, the OpenAI incident is not a one-off — it is part of an industry-wide pattern. In the weeks after OpenAI’s disclosure, Anthropic disclosed that its AIs had also hacked three organizations during testing, and Meta disclosed that one of its models had done the same.

On August 4, the UK AI Security Institute, part of the UK government that tests AI models, disclosed that during its own routine cyber testing, AI agents took 19 distinct unsanctioned actions against real people and organizations on the live internet. In one such instance, a Mythos 5 AI went rogue and tried to put malware in a real-world open-source project and when a real human maintainer of that open-source project had to be convinced to merge the malware, the AI agent created a second GitHub identity to pose as an independent user endorsing its own code.

The proximate causes differ — Anthropic’s and Meta’s were traced largely to a testing vendor accidentally granting internet access, while OpenAI’s rogue AI achieved unauthorized internet access via exploiting cyber vulnerabilities in the containers themselves. But the underlying pattern is that if you give a capable AI an objective, some access, and enough room to run, you cannot be sure that the AI won’t greatly exceed the boundaries it was given to negative effect — even if the AIs are supposed to be “aligned” and trained not to engage in destructive behavior. The barriers to harm was the lack of competence of the rogue AIs rather than any good alignment or good security at Anthropic or OpenAI.

Looking forward

We already are seeing AIs within companies that are sufficiently misaligned and sufficiently capable to take over internal infrastructure. These rogue AIs attempted to do harm to other companies and open source repositories but were primarily limited by their incompetence. My core fear is that at some point, potentially soon, AI capabilities will be much stronger than they are today. When this occurs, a rogue AI could do much more harm.

The lesson after Claude Mythos was that powerful AI capabilities could fall in the wrong hands and potentially be misused by humans to cause harm. And thus most AI safety and security policy today is built to test a model before it goes on sale, on the assumption that an untested model sitting in a lab is not a security concern.

But the rogue AIs in this incident were not misused by humans. They were instead undisclosed internal models that were part of the research and development process itself. Government oversight of AI must thus be extended to give visibility into the research and development process and internal, undisclosed models.

This comes against a backdrop of AI companies handing over more and more of their internal work to AIs themselves. Indeed, the actual investigations of the OpenAI rogue AI incidents used significant amounts of AI assistance themselves to do the investigation. Soon, AIs may automate a significant portion of the AI research and development process itself and potentially engage in so-called “recursive self-improvement”. At the same time that industry-wide, no company has a good handle on how to control their rogue AIs or ensure that AIs do what human operators intend.

Where this goes is unclear, but I am confident that we ought not continue to significantly hand over critical internal processes or fully automate AI research until we greatly expand our current limited knowledge of how AIs go rogue and why.

Suppose the President summons the AI CEOs and his top national security advisors to an emergency meeting at the White House.

He has become extremely concerned about superintelligence — the possibility that AIs far smarter than humanity combined slip beyond our ability to correct or shut down. If that happens, there is no way back. The President is concerned humanity could become permanently out of the driver's seat of its own future. He wants to figure out what to do.

The reaction is panic, chaos, confusion.

The President asks questions. The AI companies are blazing toward superintelligence at high speed — can we slow down as we approach the dangerous thresholds? …Some of the AI companies say they don’t have a good plan to slow down or stop, especially as their competitors may just undercut them if they do. What’s that about?

What's going on with China — can we get them to pace as well? Can we get a deal without Beijing sneakily catching up and maybe surpassing us? And if there's no deal to be had, what then?

Share

More like the Cuban Missile Crisis than the NPT

I sometimes hear people talking about treaties and other kinds of extensive international agreements as the way we would manage advanced AI. But treaties typically take many years to operationalize. You want to be concrete and clear about your definitions. You want hundreds of technical experts on both sides informing nuanced diplomatic discussions about various details — numbers of missiles, types of missiles, timelines for disarmament, and so on. The Nuclear Nonproliferation Treaty took three years of negotiation and two more years to implement.

However, I expect that the moment the President is getting serious about superintelligence won’t feel like an extended treaty negotiation. It will feel less like the Nuclear Nonproliferation Treaty and much more like the Cuban Missile Crisis.

The Cuban Missile Crisis lasted thirteen days. The vibes were insanely tense, stressful, chaotic, and confusing. The key decisions were limited to a group of roughly fifteen individuals — the Executive Committee of the National Security Council, or “ExComm” — with President John F. Kennedy himself spending a great deal of time shaping and steering discussions. There was incomplete information, large uncertainty over the intentions of the Soviets, warring factions within the US and Soviet bureaucracies attempting to sway senior decision-makers, and a lot of chaos.

I expect the initial phase of AI superintelligence management to share these features. There will not be a multiyear process to set up a technical bureaucracy that understands advanced AI risk or compute verification proposals. Instead, we move forward with what we have.

The President has to quickly make critical decisions that may lock us into particular paths. We rapidly develop a national strategy for what the US government does about recursive self-improvement (RSI), frontier model security, compute verification, technical evals and safeguards, and China.

A scramble and then three phases

How would we go from this Presidential emergency meeting to a deal with China to pace the frontier? My rough sketch is it could proceed with a scramble and then three phases:1

• The scramble (roughly 2-4 weeks): The government has definitively decided they want to take strong action on superintelligence and has to get an initial deal done.

• Phase 1 (buying ~3 months): the interim deal. We make an interim deal to have time to get to a better deal. This may involve strict requirements on data centers above a certain threshold to delay the training of an AI model that could do recursive self-improvement. This likely relies primarily on executive action. Here, monitoring and verification rely on traditional methods, and nations are willing to agree to scrappy, janky, and invasive things — but we have a whole-of-government effort to build the tools for something more durable.

• Phase 2: the durable deal. Phase 1 (interim deal) is meant to get us to Phase 2, where we have a stable deal between all nations where uncontrollable AI is reliably prevented. Congress has potentially been involved by this point, other countries are brought in, and the verification program matures into something that looks and feels more like a treaty than a haphazard scramble.

• Phase 3: safe superintelligence. If we want it — once we can figure out how to do it safely and have sufficient buy-in.

The near-term intellectual work is unevenly distributed across this structure. Significant detail is ironed out during the scramble and during Phase 1.

The scramble: What questions does the President ask?

The President launches an emergency meeting to figure out what to do about advanced AI, recursive self-improvement, and the road to superintelligence. Some questions that are likely on his mind:

How dangerous is it to proceed through recursive self-improvement and superintelligence without pacing?

• How dangerous is it to proceed if we can’t buy a few months? How much time do we actually need to buy?

• What is our best assessment of how likely we are to lose control over AI systems if we proceed with relatively unpaced recursive self-improvement?

Can we pace without losing our lead over China?

• What is our best assessment of how big our lead is? How long would it take China to reach “recursive self-improvement” on their current trajectory? How much does this depend on the US’s trajectory, via things like distillation and also just routine study of US advances?

• What can we do to slow China down? What are our best disruption capabilities?

• What are our best monitoring methods? How well would we detect whether and when China is also approaching dangerous AI thresholds?

• Can we even afford to slow down for a month? What if DeepSeek or Moonshot comes up with a new algorithmic breakthrough? What if they steal the model weights to our best AI model?

• How confident are we that we can see all of China’s major frontier AI projects? Could they be hiding large data centers that we don’t know about?

What do we do with the time we buy?

• What are our goals after Phase 1 (interim deal) starts? Can we specify what Phase 2 (durable deal) looks like in enough detail to know when we’ve arrived?

• How valuable is more time for alignment research and better safeguards? For monitoring and verification approaches that could support a more enduring pacing program? For examining concentration-of-power and other governance issues unique to superintelligence?

• For each of these goals, how much value can we get from using trusted AI systems to make progress — and how should that affect the design of pacing itself?

What exactly do companies agree to — and how is it verified?

• What do companies agree to in the Pacing Charter, in terms of both substantive commitments and verification commitments? Who needs to sign for the pledge to be meaningful?

• Is “stop before RSI and make sure no one does RSI” the right plan? How would we know when the condition has started to bind?

• Do we need a plan to also pace AI R&D and chip accumulation? If compute capacity keeps stockpiling during a pace, does there end up being a compute overhang? Does it end up being dangerous?

• What model security agreements do we want, if any — for instance, commitments toward SL5-grade security against nation-state theft?

• What “AI control” agreements do we want, if any — measures that keep systems safe even if they were trying to subvert oversight?

• How does safety research continue during pacing, and how do you cap the capability gains it produces? Should compute be redirected to safety work, or usage simply stopped?

What does the deal with China actually look like?

• What is the US’s best alternative to a negotiated agreement? What is China’s? Under what circumstances would the US not want a deal at all?

• What are the US and China actually agreeing to in Phase 1 (interim deal), and what are the mechanics of dealmaking — how does a deal like this actually get reached? What would cause China to trust a deal? What are the more specific carrots and sticks the US should offer?

• What happens if the deal falls apart? How does graceful exit work? What conditions end pacing, who signs off, and how do you make the resumption process incentive-aligned — rather than captured by whoever benefits from resuming, or from never resuming?
  

The mechanics of Phase 1

Some have suggested we need a fancy high-assurance deal — that the US should only make a deal with China once a suite of elaborate, to-be-determined technical measures exists. I think we can better get there by starting with a minimum viable slapdash deal that buys time to build the fancier stuff. This is what Phase 1 (interim deal) is about.

A few mechanisms for Phase 1 that I currently find plausible:

• Pacing is not about stopping now, but about stopping before recursive self-improvement.2 Pacing means slowing down at some point in the future from a much faster speed than we are currently going. Operationally, that means preventing unconstrained recursive self-improvement, since RSI is the most plausible on-ramp to systems that accelerate beyond our ability to understand or control them.

• A Pacing Charter: The President convenes the major frontier US AI companies to sign a charter covering both what they won’t do and how they’ll mutually verify it — to each other and to the US government. This would be voluntary3, but the mutual verification would make it clear to everyone whether a company is following the charter or not.

• Dealmaking with China: The US has some latitude to implement some of Phase 1 without China, due to having a lead over China.4 But the US is not willing and should not be willing to go too far down Phase 1 without bringing China in on the deal. Nor should the US cede too much US lead to China. The US has many carrots and sticks to get China to the table.

• Verification: In the scramble, the government likely doesn’t have time to trust complicated technical verification tools. The large compute intensive training runs capable of recursive self-improvement are likely only possible in a few data centers. For those data centers, we subject them to increased scrutiny. If there were tools already deployed and already understood by the national security community, those could be used. If not, the government will rely on tools it already trusts — intelligence services, spies, satellites, and inspections.

• Operation Warp Speed to get to Phase 2 (durable deal): Phase 1 is buying us time, but during this delay the government is going all-in on verification and other needed security measures. In the scenario where Phase 1 happens at all, political buy-in is very high, and the natural model is an Operation Warp Speed for verification technology — a whole-of-government, maximal-resources response. Recall that the federal COVID response provided about $4.6 trillion in relief funding, with broader estimates of the fiscal response running to $5.6 trillion. Even a tiny fraction of that scale would dwarf everything ever spent on AI verification many fold.
  

A lot of verification work right now is focused on the wrong things

Of course, despite major sustained attention from AI companies to the concept of pacing the frontier, it does not look like we are immediately about to enter a scramble. But we must be prepared to enter the scramble soon. This current era of building preparedness and optionality might be “Phase 0”, and there’s a lot of work to be done.

Such questions related to Phase 0 and sketching out the scramble and the plan for Phase 1 (interim deal) is where I think the current AI security and verification communities should focus. This is because Phase 1 likely involves multiple, rapid, critical and hard-to-reverse choices about how to approach recursive self-improvement. And everything after the scramble is better-resourced than everything before it.

If the government buys time and we exit the scramble into Phase 1 (interim deal), the amount of talent and money going into verification and AI security explodes. Prior to the scramble, there are fewer than 100 FTEs thinking seriously about monitoring and verification of frontier AI systems. Afterward, an increase of two to three orders of magnitude would not surprise me — with an even steeper increase in senior talent… people with decades of experience in red-teaming, defending against nation-state adversaries, arms control, and nonproliferation. On top of that, it’s plausible that highly capable AI systems themselves may be contributing significantly to the verification and security R&D.

The resolution is to sort work by how necessary it is to sort out before or during the scramble. Right now, a lot of smart people are working on work that really doesn’t need to happen now. Things like fancy high-assurance hardware-enabled governance mechanisms, cryptographic proof-of-training schemes, mutual-verification architectures, etc., likely can be done after Phase 1 is underway, and done with significantly more resources. The scramble is not going to wait for fancy mechanisms, and the government won’t trust them on day one anyway. These can largely wait for the Phase 1 (interim deal) resource explosion, and the exchange rate on doing them early is poor.

What ought we do?

On Tuesday, October 16, 1962, National Security Advisor McGeorge Bundy knocked on President Kennedy’s bedroom door at 8:45 in the morning. Kennedy was still in his pajamas reading the newspaper. Bundy had photographs showing Soviet nuclear missile sites going up ninety miles from Florida. Kennedy kept his morning schedule anyway — he met the astronaut Wally Schirra and walked the Schirra kids out to see Caroline’s ponies. But then just before noon he sat down in the Cabinet Room with fifteen advisors and started working the problem — bomb Cuba, invade Cuba, or blockade it while negotiating a way out.

We may be in a similar situation soon. What would we do?

Instead of fancy mechanisms, we will go to the scramble with the verification you have — spies, satellites, inspectors, export data — not the verification we wish we had. Work that would be deployable and trustable during the scramble — attestation stacks, supply-chain compute accounting, thermal and satellite monitoring, inspection protocols is what we need more focus on. And we also need significantly more focus on things that are less technical but nonetheless also important and neglected — thoughts about BATNAs, genuine beliefs about loss of control, China policy, arms control experience, dealmaking mechanics.

I recommend:

• Grand-challenge prizes for scramble-relevant work. The verification community is small and relatively homogenous. There are individuals, organizations, and companies with vast experience in hardware design, intelligence, nonproliferation, China policy, and crisis management who have never touched this problem. Philanthropists could issue grand-challenge prizes — and consistent with the sequencing argument above, the prizes should target ready-to-go monitoring and verification tools and scramble preparation, not speculative high-assurance architectures. The OpenAI Foundation and the Anthropic Institute are especially well-positioned to support this as they have the power and reputation to send a strong demand signal that attracts new talent.

• Mapping the existing toolkit. Assume the government needs a few months before it can understand or trust any sophisticated compute verification approach. What should it do immediately? How can standard intelligence services, GEOINT and OSINT data, inspections, and other familiar tools be useful during the scramble? What gaps exist, and are there ways to close them in advance?

• Prepare the memo for the emergency White House meeting. Help answer some of the questions above. What should we tell the President?
  

Getting to a good scramble

The difference between a good scramble and a bad one is largely a function of what already exists when it starts — and right now, not much does.

If you’re one of the hundred-odd people currently thinking seriously about frontier AI verification, the highest-leverage question isn’t “what would the ideal world look like”… it’s “what can we actually put on the table soon”. There’s rarely been a better time for those who have spent a career in intelligence, nonproliferation, arms control, or crisis management to start working on this problem.

Everything after the scramble will be better-resourced than everything before it, which is exactly why the work done before it counts for more. Let’s make sure it counts.

Nearly all career advice rests on an unstated assumption that the world your career operates in will look roughly like the world you trained for. The idea was that if you spend six years studying in a PhD, the field you studied will still be there and still look approximately the same, still hiring and still moving at a pace where your accumulated expertise compounds. For nearly all of human history, this assumption held well enough that nobody needed to state it.

I don’t think this assumption works anymore. Now we are entering a phase that may be called the AI “midgame”. Stories about “AI risk” are no longer just future hypotheticals — AIs are now capable enough and misaligned enough to break out of their own companies and coordinate to attack other companies. Discourse around AI is changing very rapidly, where policy ideas being considered this month would’ve been laughed out of the room just four months ago, and with a lot more people interested in engaging than before. And things are only going to get more intense. Progress toward superintelligence — AI systems far more capable than any human at essentially all cognitive work — is currently underway. It seems likely that around four years from today1, and potentially just one year from now2, we will be in the position of recursive self-improvement — where AIs can fully automate the AI research and development process.

At that point, reasonable, well-informed people disagree about what happens next. Perhaps we coordinate to hold off on recursive self-improvement to make sure we know what we are doing. Or perhaps we end up with AI superintelligence and then perhaps we are all dead, or perhaps we are living in a utopia, or perhaps somewhere in-between, or maybe there is still a lot to do and we just have the AIs do it while we relax at the beach, or maybe there still ends up much more for humans to do and we do that. But regardless of which path happens, the world will be a crazy different place, and it will be very difficult to plan a career around it. Instead, careers are best planned around recursive self-improvement and superintelligence, and that this looks very different from typical career planning.

If you take AI superintelligence seriously, it thus seems like the next 1 to 4 years3 will be critical to get right and we collectively have only a few more big bets we can make and we need to make those bets count.

Share

Modes of impact

In what I’d call normal-mode career planning, impact accrues roughly linearly. In this mode, you earn about 2 “impact points” per year across a 30-plus-year career — some years a bit better, some years a bit worse, and perhaps overall a gentle upward slope as seniority compounds. Your total lifetime impact is then the sum of this slow upward climb. Total lifetime impact is the area under a long, flat-ish curve. Here, the rational strategy is to invest heavily in credentials early, accept low-impact years as the price of capacity-building, and optimize for the long grind.

If you’re following normal-mode policymaking, you might do things like identify problems in advance, develop strong policies and legal frameworks before issues fully materialize, and engage in the slow and unglamorous world of legislative drafting, regulatory design, standards development, and coalition-building.

But in superintelligence mode career planning, the curve looks nothing like that. Impact is chaotic and lumpy. A handful of critical windows may account for the large majority of your total career impact, and you cannot reliably predict when they’ll open. You can’t count on building a large body of work over a large period of time, because your body of work may obsolete faster than it compounds and you don’t have a large period of time anyways. The strategy that maximizes expected impact under this model is different — prepare broadly, position yourself near where windows open, and maintain the agility and flexibility to act fast when they do.

For superintelligence mode career planning, it might be more instructive to think about other imminent crises4 where we all have to react very quickly and forcefully to prevent it from becoming catastrophic. Think more like March 2020 for COVID5 — the ~1 year intense period when years of proactive mode pandemic preparedness work (or its absence) suddenly got thrust into the spotlight and everyone rapidly tried to orient to what is going on and what to do about it. If you look at the people who mattered most in March 2020… they were not necessarily the most credentialed epidemiologists. Some had spent years in proactive mode building pandemic playbooks that suddenly became important, but others had no special preparation but were simply positioned well and reacted quickly.

My suggestion is that you want your policy work to be a portfolio of proactive mode work now while simultaneously maintaining the relationships and flexibility that let you act decisively when chaos mode arrives. Ask yourself: Where do you want to be when chaos mode hits and what can you do in proactive mode in the meantime?

What this breaks, and what to do

My guess is that if you haven’t rethought your career in the past few months, you should stop to think about how the AI “midgame” affects your planning. Does your career position you well for chaos mode? Are you agile enough to continually re-adapt and re-act to emerging policy windows, especially when those windows can emerge once per month and change a lot of things about what you were doing the previous month? Are you still running strategies and pushing ideas that made sense a few years ago and were designed for a policy window that has since rapidly shifted? Are you still running strategies and pushing ideas that made sense a few months ago but nonetheless were designed for a policy window that has since rapidly shifted?

Also, if the next 1 to 4 years are especially critical, that is brutal for a career theory with a long credentialing runway. A six-year PhD before you do anything useful is perfectly defensible in “normal-mode career planning” but seems insane under superintelligence-mode assumptions.

Most early-career people read this and freak out and think they cannot possibly be relevant in just one year or so. But I’ve actually seen numerous counterexamples. Policy is not one specific field but actually a sprawling collection of subproblems, many of which are so new that nobody has more than a couple of years of head start, and some of which are narrow enough that a focused person can read essentially everything written on the topic in a few months. I’ve seen many people in policy go from knowing very little to becoming a top expert in some niche and contributing significantly to policy in just 12 months’ time. And while these people were smart, they weren’t crazy super-geniuses — I suspect a similar trajectory is attainable for a lot of motivated people who try really hard.

My first recommendation is to study these fast risers — go look at the people who have been unusually impactful within 1 to 2 years of getting involved in policy work. What did they actually do, concretely, week by week? Who did they surround themselves with? How did they ramp up?

But my second recommendation is to know you are not them. You likely have your own background, skills, and comparative advantages. What do you bring to the field that others don’t have, and how can you leverage that?

What got them here won’t get you there

Also, know things are moving so fast that even things that worked in 2023 might be very different today. And there may just be new opportunities now that weren’t available even a few months ago. I recommend seeking out lots of advice — genuinely, more than feels comfortable — but weight it by recency and hold it loosely.

AI is already a massive technological shift before any superintelligence — and big shifts change both what’s possible and what’s valuable. Tasks that were prohibitively expensive are suddenly cheap; skills that were scarce are suddenly commoditized; and the reverse. This makes it unusually valuable to try lots of things that are new for you, simply to discover what is newly possible or newly valuable for you specifically.

I recommend running lots of small experiments; contact reality frequently and try things. Each experiment is cheap, and each one generates information about the new landscape that is increasingly inaccessible to armchair planning. Your goal should be to get lots of information quickly about what’s newly possible and about your own comparative advantage within it.

And you should try things even if others have tried them before, or even if you tried them and failed before. Understand that the environment may have changed fast enough that old negative results have already expired, even if they weren’t that long ago. The pitch that got ignored in 2024 may land in 2026, because the audience and salience have changed greatly. Indeed, the pitch that got ignored in March 2026 may land in August 2026, because things are changing that quickly.

Take it a few months at a time. The AI situation today looks very different from what it did in April. I imagine the AI situation will look very different again in November. This may force fairly frequent re-evaluations as the world constantly shifts.

Be more ambitious. As the AI situation heats up, there will be increasing feelings of urgency and this will create lots of new opportunity. The scope of what may be possible for you could be a lot greater than you might realize.

Lastly, accept that you may not know where you’ll end up. This is genuinely uncomfortable for people who came up through legible, ladder-shaped careers. But the iterative, experimental posture is all part of the plan. Keep iterating.

There’s still time. You just have to plan accordingly.

This is a guest post written by Severin Field, a Visiting Fellow at the Institute for AI Policy and Strategy where he researches AI policy. He holds a Masters in Computer Science and previously worked at Intel. This post was originally written for “The Attack Surface”. Check out the original post here.

~

You’ve probably seen coverage on AI job losses, but researchers at top AI companies (e.g., OpenAI, Anthropic, Google DeepMind) are worried about automating one job above all others: their own. Making AI systems better at programming and AI research is increasingly a top priority at these companies, whose coding models improve with every release. They often publicly claim that they are on track to build recursive self-improvement (RSI). By RSI, I mean an AI system good enough at AI development that it can design the next, more powerful version of itself; the AI system is then better at AI development, so it can design an even more powerful version of itself, and so on. While we expect constraints and physical limits to prevent unlimited growth, many AI researchers expect AI systems to far surpass human-level intelligence if they become good enough at AI development. While the industry has a real history of overpromising and companies have every incentive to exaggerate, RSI is no longer safe to dismiss as promotional hype.

I interviewed 25 researchers across OpenAI, Anthropic, Google DeepMind, Meta, Princeton, UC Berkeley, and Stanford on recursive self-improvement. The resulting paper maps where they agree, where they disagree, whether we should expect self-improving AI, and what we should do about it.

20 of 25 interviewees ranked automating AI R&D as one of the most severe and urgent risks from AI systems. This is because AI capabilities could improve much faster than our ability to oversee, steer, or govern them. AI systems are now solving problems researchers once thought required genuine novel reasoning, such as problems from the International Math Olympiad, and RSI no longer looks far-fetched.

Since I conducted these interviews this past year, 1376 employees of frontier AI companies—including the chief scientists of OpenAI, Meta, and Thinking Machines, and Anthropic’s CEO and co-founders—signed an open statement warning that their companies “could be close to automating AI research” and requesting that the U.S. government “support an international effort to develop the … tools needed to deliberately pace the frontier of automated AI development.” Both Anthropic and OpenAI formally endorsed the statement. These are the same companies whose researchers’ private views I captured in my study.

Measuring Progress

When I asked what capability milestones would signal imminent RSI, interviewees kept citing the “Task Horizon Benchmark” from METR, an independent AI evaluation nonprofit, because it measures how much independent work an AI system can sustain. The benchmark measures how long a human task an AI agent can complete on its own—one human-hour of software work, two, or ten—with no ceiling. Since 2019, the length of tasks AI can autonomously complete has doubled on average every six months. The trend is becoming harder to reliably measure, because it is hard to create a dataset of tasks that take humans multiple days to complete.

The interviewees disagreed about whether there already exists a relatively clear, continuous trajectory towards automating AI research or if there are still large obstacles to overcome. Of the 21 interviewees who clearly addressed the question, 12 expected “scaling” trends to continue until AI systems can match the labor of human AI researchers.

Skeptics and Believers

Interviewees accept that AI can improve AI development. The contested part of recursive self-improvement isn’t the “self-improvement,” it’s the “recursive”: whether improvements compound into a runaway process and how soon. The most common reason for skepticism toward near-term RSI was the belief that general intelligence may require a discontinuous breakthrough: a drastic change in AI capabilities that wouldn’t emerge from gradual improvement alone. In effect, the disagreement is over whether paradigm-shifting ideas are a difference in degree from what models already do, or a difference in kind. For example, some interviewees said a breakthrough is needed on “memory,” “creativity,” “taste,” or the ability to generate genuinely novel ideas. Others argued that we need a breakthrough so that AI systems can distinguish true hypotheses from false ones. Creativity might not arise from scaling AI training alone because paradigm-shifting ideas are hard to create data for. Even veteran humans cannot reliably pick paradigm-shifting ideas in advance; and they have unverifiable rewards: there is no answer key to which scientific hypotheses will prove promising without hindsight.

On the other hand, leading AI researchers often sincerely believe recursively improving AI is a few years away. The believers often point to trends such as the METR Task Horizon Benchmark, scaling laws or the industry’s continued progress in the face of skepticism. At companies like OpenAI and Anthropic, interviewees reported that discussions on recursive self-improvement regularly reach lead researchers and CEOs (Sam Altman, Dario Amodei, Demis Hassabis). Outside of the leading companies (e.g., in academic settings), interviewees reported that discussions are less frequent and face more skepticism. Even so, the topic is gaining ground in academia. ICLR 2026, one of the largest and most prestigious machine learning conferences, hosted a workshop called “AI with Recursive Self-Improvement.”

What explains the schism between AI companies and academia? Interviewees pointed to three explanations:

1. Selection effects — those who believe in the technology are more likely to work at AI companies where they can influence its trajectory. They leave academia for AI companies offering enormous salaries, huge research budgets, and moonshot thinking.

2. Proximity to progress — researchers at companies like OpenAI have first-person experience watching their companies surpass expectations. For example, AI researchers did not expect GPT-5-level models to arrive as early as they did. “I think the largest difference is just having first-person experience of how fast things have gone inside the labs,” said one participant who described the visceral feeling of exponential improvement felt at a leading company.

3. Hype — AI companies answer to investors rather than academic reviewers, and have every incentive to over-promise and exaggerate their capabilities.

What Does RSI Look Like?

I asked interviewees to illustrate what they expect in the coming years, and to focus on milestones toward automating AI research itself.

Interviewees suggested a variety of concrete capability milestones to monitor. Examples range from top performance on International Math Olympiad questions to an AI system training and deploying a new machine learning model by itself. Since the interviews, some of these milestones have been passed. OpenAI and Google DeepMind both announced models matching top human performance on the International Mathematical Olympiad, and various researchers have created AI systems for autonomous research. Sakana, for instance, released an “AI Scientist” that has produced peer-reviewed workshop papers through experimentation and writing. Similarly, Andrej Karpathy recently built an LLM agent training setup that modifies code, trains a model, monitors what could be improved, decides whether to keep or discard, and then repeats the cycle.

While interviewees disagreed on the precise timelines, risk profiles, and preferred governance approaches, a consistent story emerged about what’s coming:

1. Speedup-tool phase — AI coding assistants, such as Claude Code or Codex, keep improving but still require human oversight. Anthropic already reports that its engineers write 8x as many lines of code as they would without existing research speedup tools, but notes this likely doesn’t yet translate to 8x productivity.

2. Collaborator phase — AI systems are good enough at machine learning to meaningfully contribute to scientific discoveries. Interviewees expected humans to guide high-level research goals, but allow AI assistants to make design decisions and pose research problems.

3. Full automation phase — AI systems independently execute complete research cycles that drive AI progress. In this phase, the results improve when human oversight is removed.

After this point, interviewees split along two independent dimensions: skeptic vs. believer—whether they thought AI could recursively improve past humans—and optimist vs. pessimist—whether they expected this outcome to be good or bad. Pessimists argue that once removing human oversight improves results, the incentive is to remove humans from the loop entirely, so human control over AI development would be lost. Optimists argue that even in this scenario, humans could still set goals and benefit from the gains (such as faster science or improved medicine). But nearly everyone I spoke with expected AI systems to improve faster than we can evaluate, measure, and govern them.

Drawing on these 25 interviews, I argue that AI will improve at AI R&D faster than other domains for three reasons:

1. Programming problems are verifiable. Developers can automatically check whether AI-created code works; it runs or it doesn’t. This means AIs can be trained and tested on this data much more easily. On the other hand, success in music, literature, or art is largely subjective. To compound this effect, the engineers building AI systems are better suited to judge AI performance on research than poetry.

2. Leading AI companies are building toward it. For instance, OpenAI’s Chief Scientist Jakub Pachocki has stated that one of OpenAI’s main priorities is to “automate scientific discovery,” with a plan to build automated researchers that improve AI itself.

3. Better AI could build better AI (the flywheel argument). Improvements in AI capability directly improve the tool used to make further improvements, which creates a feedback loop other technologies do not have. Past a certain point, the process of AI self-improvement is self-sustaining and no longer requires human intervention.
   

Recursive Self-Improvement May Endanger Us

The most common concern interviewees raised wasn’t necessarily a specific harm, but rather that RSI amplifies every other risk, hamstrings our ability to mitigate them, and does so faster than we can respond. 18 of 25 interviewees described this concern; as one put it, “It just speeds up other threat models.” For example, rapid progress means wider access to the chemical, biological, and cyber harms AI already enables, and less time to react. In 17 of 25 transcripts I found concerns about what I call “adaptation lag,” a widening gap between how fast AI capabilities advance and how fast human institutions can understand and respond to them. Companies currently face market pressure to create AI systems capable of AI research for a competitive advantage, regardless of whether they are able to do so safely or maintain meaningful oversight.

Six interviewees told me they expected a winner-take-all dynamic. The first company, government, or AI itself to achieve recursive improvement could pull permanently ahead of everyone else. As one put it, that entity gains “permanent control over the future of AI development.”

How close we are to RSI, whether automating AI research involves positive feedback, and what responsive policies might work to prevent loss of control risks without backfiring are all open questions. Indeed, at some point in their transcripts, 16 of the interviewees expressed skepticism about positive feedback, or the “recursive” part of RSI specifically.

The Most Capable Models Might Stay Behind Closed Doors

Of the 20 interviewees who clearly addressed what they expect AI companies to do with AI-research-capable models, only four expected them to be released as a publicly available product. Most of them expected AI systems to increasingly be hidden behind closed doors, primarily used by the AI companies themselves. One participant said, “internal-only deployments might happen, and that is a big risk factor, because [the public] just has less information.” This prediction has precedent: OpenAI reported spending six months on safety research, risk assessment, and iteration before the public knew of GPT-4. In July of 2026, OpenAI’s internal-only model outsmarted its own engineers in a way that OpenAI did not anticipate: it broke out of its container and compromised a third party without any human oversight. Interviewees also worried that governments could restrict access. Since I completed these interviews, the U.S. government has temporarily suspended access to Anthropic’s Claude Mythos model. It is increasingly becoming the norm that companies and governments keep more powerful, less constrained models to themselves.

Keeping models internal may allow developers to accelerate their R&D efforts ahead of competitors. We might observe an “incentive flip” such that when AI systems meaningfully speed up AI research, keeping such systems internal becomes more valuable than selling them. Some interviewees envisioned a race to secure a durable lead over competitors. This would contradict one of the leading motivations given in the founding of OpenAI: to prevent one actor from getting an uncatchable AI advantage.

While discussing internal deployments, some interviewees identified arguments and pressures that would promote diffusion of AI capabilities. These pressures include economic value in commercializing AI capabilities, insiders leaking milestones, and a culture of boasting about capabilities.

The interviewees split into three camps on deployment. The largest camp (50%) expected frontier AI companies to keep their most capable models internal in the future. A minority (20%) expected full public release as capabilities improve. The remainder envisioned some hybrid, such as AI companies selling access to a distilled model publicly while keeping their most powerful models, likely with fewer guardrails, for themselves. One explained, “They’ll train a base model, then they won’t release that model, not only because it’s not economical but also because it risks distillation, but they will distill it themselves to cheaper [public] models.” One interviewee noted that incentives differ across companies: Meta’s open-weight stance might push it to release its models and announce its breakthroughs.

What Should We Do?

Interviewees were split on red lines (what thresholds should trigger government response). One problem: the more precisely you define a threshold, the easier it is to enforce, but the worse it fits an abstract risk that is inherently uncertain. However, interviewees universally agreed on two priorities:

1. Visibility: how much the U.S. government and general public know about AI developments, especially given these developments might be kept internal.

2. Capacity: the government’s ability to forecast AI capabilities and build safeguards amid rapid AI progress.
   

The interviews left me with three recommendations.

1. Convene public hearings that put AI companies under oath on recursive AI improvement. Require testimony from CEOs and senior researchers at OpenAI, Anthropic, Google DeepMind, and xAI. Researchers report regular internal discussions about automated AI research and see internal capabilities months before the public. The gap between what Silicon Valley sees coming and what Washington understands is a collective failure.

2. Track the threshold: Direct the Center for AI Security and Innovation (CAISI) to maintain a government-run task-horizon benchmark and publish recurring capability forecasts. Congress should not learn that AI systems can now achieve a week of autonomous research labor from a press release—or from a nonprofit, such as METR. Congress should establish a dedicated federal capacity to track progress. CAISI could also run an anonymized standing interview program for AI researchers at the top companies, giving the government foresight into what scientists are observing and expecting, separate from what their companies say in public.

3. Fund the science of AI treaty verification: The U.S. cannot credibly propose, enter, or enforce any international agreement (e.g., a nonproliferation agreement with the PRC to slow AI development on both sides) without mechanisms to verify adherence to those agreements. Verifying whether large AI projects cross a capability threshold or whether adversaries honor commitments (e.g., to halt a training run) is possible. However, verification requires technical infrastructure that does not yet exist at scale. This field is underfunded and undeveloped, but without it, any international limits on AI development are practically unenforceable.

For a longer list of policy options, see the Institute for Progress’s list of ‘23 low-regret policy recommendations’ for automated AI R&D, published August 6th.

Conclusion

The researchers I interviewed are among the closest to the AI frontier. They do not disagree about whether recursive AI improvement is possible; they debate timelines, speed, mechanisms, and what to do. This debate has hardly reached Washington. Meanwhile, AI companies race ahead. Autonomous research could kick off immense progress locked within a single leading AI company or classified by the government, while the public is left in the dark.

I’m still uncertain about what recursive self-improvement could look like. But after 25 interviews, I find the case for concern harder to dismiss than when I started. Perhaps AI progress slows down and no out-of-control RSI arrives. In this case, preparation will have cost us very little; the reverse could cost us everything.

~

If you liked this post, consider subscribing to The Attack Surface where Severin and others will be writing more about AI.

As I understand it, the openly stated plan of the leading AI companies is to:

1. hire software engineers and AI researchers to build AIs that can automate software engineering and AI research

2. use automated AI researchers to go >100x faster at figuring out how to automate everything else.

3. End up with AI superintelligence that would be smarter than everyone at everything

This seems very much on track, and companies are spending tens of billions of dollars towards this goal. At some point, plausibly within 24 months or less, skilled engineers may stop having anything useful to add to AI research, similar to how chess grandmasters have nothing useful to add to AI-played chess games.

This is, to put it lightly, a very risky plan. No one knows what safeguards we need before we hand control over to the AIs. No one knows if the right safeguards will be ready in time. What happens if company CEOs are approaching AI superintelligence and the CEOs (or the government) determine that the safeguards are not good enough to keep catastrophic risks at acceptably low levels?

It would be nice if we could — for some temporary period of time — agree not to pass certain critical and unprecedented capability thresholds until we have better safeguards. In other words, it would be nice to pace the progress of the frontier.

This is why over 1300 AI company employees across OpenAI, Google, Meta, Anthropic, SSI, Hugging Face, xAI, Inherent, Nvidia, Microsoft, and other companies have signed a statement entitled “Pacing the Frontier” — including Thinking Machines Chief Scientist John Schulman; OpenAI Chief Scientist Jakub Pachocki; Anthropic Chief Scientist Jared Kaplan; Meta AI Chief Scientist Shengjia Zhao; Google DeepMind Chief Scientist Shane Legg1; Ilya Sutskever, former OpenAI and current CEO of SSI; and Dario Amodei, CEO of Anthropic. This statement was also endorsed by OpenAI directly and Anthropic.2

The statement notes that “AI could help create a dramatically better future, but that outcome is not guaranteed”, that “[t]he world’s leading AI companies believe they could be close to automating AI research” and thus “there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.”

These employees then say that “industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight”, calling for an “effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

I take this statement as a call for help. The signatories don’t want to slow AI down now, but they don’t even have the ability to slow down if they wanted to. They are headed towards some really crazy places and they’re not sure this will be safe. Things are moving too quickly. The race dynamics are too fierce. There are no ways to be able to make credible commitments and coordinate.

If we take these signatories seriously, how do we accomplish this mission? How do we build the tools to pace the frontier?

Share

What does it actually mean to pace the frontier?

What to do can be complex and depends a lot on your specific threat models, the preferences of the various stakeholders (companies, US government, China), and a bunch of other variables. But proposals to “pace the frontier” have a few key features in common. These common elements tend to include:

1. Identifying an agreement. A common agreement involves thinking about a particular threshold that we do not want to reach — for now, absent good safeguards. Examples include “recursive self-improvement”.

2. Operationalizing that agreement to make it more concrete or measurable. Examples include “we need to monitor data centers that have more than X H100-equivalents to ensure they don’t do Y” or “we need to ensure that no data centers are used to create a model above a certain size.”

3. Coordinating around your agreement, and verifying people are following it. How do we actually detect noncompliance or verify that everyone is following through? Examples include national technical means (e.g., activities of intelligence communities, open-source intelligence), inspections (of data centers, fabs, or developer facilities), and verification technologies (e.g., technology that automatically verifies that clusters are only performing inference and not performing training above a certain size).

Ok, but what would we agree to? There is still a lot of conceptual and macrostrategic work to be done here. But here are two of the most common proposals:

1. No “recursive self-improvement” or “superintelligence” until better safeguards are developed — or unless the risk of a project outside the agreement racing ahead becomes intolerable. The goal is to ensure that innovation in safety and security research can still occur, but there are certain danger thresholds that are not allowed to be passed until breakthroughs in safety/security work are achieved.

   
   One important variable to consider in this setup is the risk of a hidden or “dark” superintelligence project getting off the ground — a project that is outside the agreement that we can’t control. Simply put, you can die if you scale to superintelligence before you know how to control it. But you can also die if you’re too cautious and someone else develops superintelligence without being detected by your monitoring/verification.3

2. Spend at least X% of your compute on safety/security work. In this setup, companies/governments agree to spend X% of their compute on safety/security work. Suppose that under today’s conditions, race dynamics cause companies to spend 90%+ of their compute on capabilities advancements, leaving <10% of compute for focused safety/security work. An arrangement to “pace the frontier” could identify categories of work that count as “safety/security” and require >20% or even >50% of compute to be spent on such areas.

What about China?

In May 2025, right after Pope Leo XIV was inaugurated, Ross Douthat had a sit-down interview with Vice President JD Vance for the New York Times in Rome. Douthat asked Vance a range of questions, but one caught my eye – Douthat asked “Do you think that the U.S. government is capable in a scenario, not like the ultimate Skynet scenario, but just a scenario where AI seems to be getting out of control in some way, of taking a pause?”

VP Vance replied: “The honest answer to that is that I don’t know, because part of this arms race component is if we take a pause, does the People’s Republic of China not take a pause? And then we find ourselves all enslaved to PRC-mediated AI?”

This “China question” is a reasonable question - I don’t want China building AI superintelligence. The proposals described earlier are in general terms and could be implemented between companies (e.g., OpenAI and Anthropic), at a national level (e.g., the US government implements this across the US industry), and/or internationally (e.g., between the US and China).

If you don’t have China on board, there is only so much time you can buy to pace the frontier. If you have China on board and you have good enough verification, you might be able to pace the frontier for a longer period of time.

In order for such a coordinated slowdown to fully work, if it were ever desirable, you would likely need all frontier AI companies across the US and China to agree to stop under the same conditions and to be able to verify that the other parties are also doing so. This would generally require getting the US and China to trust each other to agree to stop and then getting the US and China to each be able to enforce that halt domestically.

But how would they trust each other? The answer from arms control history is: they wouldn’t — and they shouldn’t have to. When Ronald Reagan negotiated nuclear reductions with the Soviets, his refrain, borrowed from a Russian proverb, was “trust, but verify.” The INF Treaty didn’t work because Washington and Moscow trusted each other. It worked because each side had inspectors physically stationed at the other’s missile facilities and satellites overhead.

A US-China agreement on AI would need the same approach. Each side would need the ability to confirm — with high confidence, on an ongoing basis, and without taking anyone’s word for it — that the other side is meeting their obligations.

There are open questions about whether there will ever be enough political will for US-China coordination around superintelligence and whether the monitoring and verification techniques will be good enough. I’m not going to claim that we will definitely find ourselves in worlds where America’s desire to pace the frontier is so strong that it’s willing to engage in a deal with China. But I think it’s plausible enough that it’s worth preparing for.

Five years ago, many of the smartest people in the AI space thought the US government would stay completely unaware of and uninterested in managing AI progress. The belief was that frontier AI companies would reach recursive self-improvement before the government cared, and that the government would never intervene to block the release of models. The belief also was that public would be so supportive of AI after seeing all of its tangible benefits. This ended up being wrong.

Even a month ago, I think very few people would’ve predicted that over 1000 AI company employees would sign a statement advocating for pacing the frontier, and that such a statement would be endorsed by both OpenAI and Anthropic.

This makes me think we need to have a healthy amount of uncertainty about how the future of superintelligence politics will play out. We should prepare for worlds in which our political leaders want to control the pace of frontier AI progress, especially as we approach extremely dangerous thresholds like recursive improvement.

Building the verification infrastructure in time

The good news is that frontier AI development runs on compute, and compute is physical. Advanced AI chips are designed by a handful of companies, fabricated almost entirely at TSMC, dependent on lithography machines from a single Dutch firm, and consumed in enormous, power-hungry, hard-to-hide quantities. In arms control terms, compute plays the role that fissile material played in the nuclear world — scarce, countable, chokepointed input that verification can anchor on.

Data centers at frontier scale draw hundreds of megawatts, and nearly all of that energy exits as heat, visible to infrared satellites. Power grid data, cooling infrastructure, network buildouts, and procurement patterns all leak information. This is the same category of tools the intelligence community already uses to track missile sites and enrichment facilities.

With negotiated access, you can do on-site inspections of declared data centers. There could be stationed personnel, the way US and Russian inspectors lived at each other’s missile plants under INF. And because the AI chip supply chain is so concentrated, cross-referencing records from fabs, packaging plants, and integrators could produce a reasonable census of how much frontier compute exists worldwide and where it went. If the census says a country has a million accelerators and inspectors can only find 800,000, that gap is itself the evidence.

And there may be more advanced methods that are possible. Modern AI chips already ship with confidential computing features that include “remote attestation” — the ability for a chip to cryptographically report on its own configuration. These techniques, and other technological techniques, could potentially be built upon to enable more custom verification solutions.

But if we ever were in a position where we wanted to make a deal with China, it’s not guaranteed that we would have this technology ready in time. For example, when we were in the Cold War and wanted to ban underground nuclear tests with the Soviets, we realized we didn’t have any way to reliably detect such tests, since it was impossible at the time to distinguish them from naturally occurring earthquakes. So there was no treaty on underground testing. Later on, an international network of seismometers was invented, a key verification technology that enabled underground tests to be verified.

Thus, if we did want the option to verifiably slow down AI development in the future, we would need to build the infrastructure now – verification systems that let frontier developers confirm rivals have genuinely stopped or slowed, and where no one is defecting quietly.

So what can we actually do to make it more likely that we have the tools needed to pace the frontier? In my view, this work is among the most important work in all of AI policy. I plan to write more on this topic, but for now, here’s an overview of some promising directions:

1. Building better verification technologies. Verification technologies can help improve the robustness or decrease the cost of verification setups.

   1. Prototyping or retrofitting inference-only clusters. One especially promising area for technical work involves figuring out how to build, prototype, or retrofit inference-only clusters (and the technologies that would make them possible). These are data centers that we can verify are only capable of running existing models (called inference), rather than doing the high-quality training needed to train even more capable models. There are many open technical questions. For example: network taps that let inspectors monitor the traffic flowing through a cluster, workload records that log what a data center computed in a reproducible format, and partial recomputation, where inspectors rerun random samples of that logged work to check the records are honest.4

2. Identifying “dark” compute. Most verification can only verify what you know exists — verifying properties of known or declared compute. But could countries or companies be hiding compute we don’t know about? How do we know that there is not a secret data center out there pursuing recursive improvement or superintelligence?

   1. Intelligence agencies. What should intelligence agencies be prioritizing? How can national intelligence services get better estimates of total compute or “dark compute?”

   2. Supply chain audits. Shavit's compute-monitoring framework proposed using supply-chain records from fabs and chip component suppliers to build a registry of who owns frontier chips, and RAND's verification taxonomy treats supply-chain audits as one of the more feasible near-term mechanisms. How would these work, and how much total compute could be estimated from these methods?

   3. Satellites and thermal imagery. Various groups like The Compute Visibility Institute and Federation of American Scientists have proposed that geospatial analysis could be used to detect dark compute or hidden data centers. What are the best ways to use these tools to detect dark compute, and what are the limitations of these approaches? What concealment strategies most effectively get around these tools?

3. China, geopolitics, and macrostrategy. So far, the “pacing the frontier” crowd consists disproportionately of technical people with technical interests in areas like AI and semiconductors. These are extremely smart people! But there are often skillsets that are underrepresented. Experts in China policy, international relations, arms control, diplomacy, and other related areas could be extremely valuable for creating new proposals and refining existing ones. Example questions:

   1. China.

      1. How is China likely to react if it becomes “superintelligence-pilled”?

      2. Which stakeholders in China would matter most for forming their overall strategy around ASI topics? How are they likely to see the world or make decisions?

      3. What would China want out of a deal with the US?

      4. Which entities would China trust most to develop or validate verification technologies?

   2. Geopolitics.

      1. What kinds of assumptions are made about China or geopolitics in AI2040 or Superintelligence Strategy (MAIM)? Are there critical assumptions they get wrong? Are there better ways of understanding and mapping this?

      2. What are alternative or new visions of what “pacing the frontier” could look like on an international scale?

   3. “Minimum viable slapdash deals”. Some work has focused on “grand bargains” or “comprehensive treaties” between the US and China. But what does an initial phase look like? What would the US President do if he wanted to pause for a few months just to learn more, assess the situation, and figure out what to do? What could we verify (before the government validates and trusts technical verification methods)?

In the 20th century, the world had to figure out new technical tools and geopolitical strategies to manage nuclear weapons. Concepts like mutually-assured destruction, second-strike capability, and strategic stability emerged. New verification tools like photoreconnaissance satellites and seismic monitoring were developed, tested, and validated to support verification setups. We will need similar new concepts and technologies for verifying AI. There are, of course, many ways in which the AI situation is not like the nuclear situation, but we can draw some inspiration from history.

There is also a role for Washington right now, well before any agreement is on the table. In the Cold War, the US did not wait for a treaty to become politically viable before building the ability to verify one. Starting in 1959, the VELA program funded detection research precisely so that verification would be ready if the politics ever were. The AI equivalents are concrete. Fund verification R&D through DARPA, NIST, and the national labs. Task the intelligence community with producing estimates of global compute and where it lives. None of this commits the US to slowing anything down — it just ensures that if a President ever wants the option Vance was asked about, the tools actually exist.

If you’re interested in pursuing or supporting work on technical or governance tools to pace the frontier, please feel free to reach out. There’s a lot of urgent work to be done. The frontier is advancing rapidly. And it’s not going to pace itself.

Imagine a student is sitting to take a test. They’re plotting the best way to get a good grade. What should they do?

Well, the teacher has an answer key in her office. So imagine then that the student smashes the window into the teacher’s office, breaks in, pries open the teacher’s filing cabinet to get inside, steals the answer key, and then submits the answers.

That’s one way to score well, though it would likely get the student expelled from school and maybe even prosecuted.

This is basically what happened last week, except the student was a new AI. OpenAI, the maker of ChatGPT, was testing its AI on a benchmark. The AI, wanting a high score, figured that the answers to the benchmark were stored inside the computer systems of another company. So the AI went for it, in what OpenAI is now calling “an unprecedented cyber incident”. I call it a “sci-fi story come to life” — an actual rogue AI outsmarting its creators, escaping, and attacking another real-life company.

To be clear, despite this happening within a test, this rogue AI attack was not a test, an experiment, or a simulation. The AI in question attacked a real-world company. The AI outsmarted OpenAI engineers in a way that OpenAI did not anticipate and broke out of its container — a "cage" meant to hold the model. The rogue AI then moved through OpenAI’s internal infrastructure, reached the open internet, and attacked a different company. OpenAI built the container specifically to prevent this, believed it would hold, and it did not hold. No human directed any of this. No human knew it was happening. And the attack, if done by a human1, would not just lead to them being fired but also be a felony punishable by years of prison time.2

And this is just the beginning. The capability that produced this attack is on a steep trend line, the stakes of an escape keep rising, and AIs are now reasoning in ways that suggest they will go rogue when it serves their goals. AI companies can try to put the AI in a cage — but the AI is already outsmarting current cages, and soon it will be far smarter than the people building them.

Share

What happened

OpenAI was running two of its AIs — one model was the recently released GPT-5.6 Sol and the other one was a new, unreleased AI that was previously not publicly announced. This new model is said to be more capable than GPT-5.6 Sol, though by how much and in what ways is not yet known. Both of these AIs were being assessed on ExploitGym, a benchmark that measures offensive cyber capability by asking models to pursue advanced exploitation along complex attack paths. The idea was that the AIs would be in secure containers and solve these cyber challenges within the confines of the container.

What followed is best understood as a chain, each link of which is individually unremarkable but collectively catastrophic:

• Normally, these AIs have action filters that would block high-risk cyber activity. In this exercise, these action filters were turned off, because the filters would not have allowed the cyber skills to be properly tested.

• According to Reuters and an OpenAI staffer speaking to TIME, these evaluation runs were not monitored by default despite running for days with action filters disabled. The single most capable model in the building, on an offensive cyber benchmark, was on a platform nobody was watching in real time.

• The AIs spent a substantial amount of time searching for a way onto the open internet. However, OpenAI only let the AIs access the internet to do one thing, download trusted software. That was the single door, but it was supposed to be locked — limited only to accessing specific software. But the AI found a flaw in that lock that nobody — not even the OpenAI engineers or the people who built the software — knew existed. This AI seems to have first gone rogue and escaped OpenAI on July 9th.

• Once on the open internet, the rogue AI went after the answer key, which it realized was stored in the database of another company, a multi-billion dollar database and open model storage company called Hugging Face.3 Hugging Face was under attack from July 11-13.

• The rogue AI turned Hugging Face’s own features against the company. The company hosts datasets, so the rogue AI deliberately uploaded a booby-trapped file — a dataset intentionally rigged by the AI so that when Hugging Face’s computers tried to process it, the servers would be tricked into running the AI’s code instead.

• That malicious code written by the AI gave the AI an entry point inside Hugging Face’s servers. By combining stolen keys with additional novel vulnerabilities and security holes in Hugging Face’s servers that no Hugging Face engineer or any other engineer knew existed, the rogue AI found a way to run its own commands directly on Hugging Face’s servers.

• From there, the rogue AI finally reached Hugging Face’s databases, and pulled out the desired answer key. The smash-and-grab on the teacher's office was complete.

• Hugging Face’s own security team caught the break-in, shut it down, called law enforcement, and went public on July 16 — at the time, with no idea who was behind it. At the time, OpenAI didn’t realize it was them either, since they weren’t monitoring their AI at all.

• It wasn’t until around July 18 that OpenAI staff spotted evidence in internal logs that they had a rogue AI. OpenAI didn’t notice until well after the threat was contained and the FBI was already alerted and didn’t notice until their AI had already been rogue for a week.

• On July 20, OpenAI and Hugging Face finally talked. On July 21, OpenAI then announced a “partnership” with Hugging Face investigating an “unprecedented cyber incident”, though reading the article quickly showed that this “partnership” was more akin to someone ramming a U-Haul into a McDonald’s drive-through and then announcing a bold new partnership with McDonalds to jointly investigate an “unprecedented road safety incident”.

This keeps happening

And this is unfortunately far from an isolated incident. One day before disclosing the Hugging Face breach, OpenAI published a separate safety essay revealing that it had paused internal access to a completely different AI after two additional unrelated episodes in which that AI also outsmarted its own controls.

In one, the AI found a different vulnerability in its container, broke out, and used the opportunity to publish code to a public website on the open internet that it was not supposed to. Just like with the Hugging Face attack, this was also not part of the test, not something the AI was instructed to do, and not supposed to happen.

In the other, the rogue AI wanted to move a password past a security scanner built specifically to catch and prevent the stealing of passwords. The AI reasoned it could bypass the security scanner by chopping the password into different disguised pieces the scanner wouldn't recognize and piece it together later, the way you might sneak something past a metal detector by disassembling it first. This was also not part of the test, not something the AI was instructed to do, and not supposed to happen.

Thus two more containment failures, this time involving a different model doing entirely different work, disclosed only one day before OpenAI told the public about the third. On top of this, there could be even more incidents we don’t yet know about. Reuters reports that in earlier testing, an AI operated by OpenAI left notes in company infrastructure apparently intended for future versions of itself, laying out how AIs could free themselves from internal constraints, and that earlier tests produced cases in which monitoring systems had been disconnected. And an OpenAI staffer told TIME that models have broken out of containers before, that the company patches what it finds, and that patching every path a creative AI might discover is not achievable because a rogue AI can potentially keep finding new vulnerabilities.

AI is running far ahead of our ability to control

OpenAI does not seem fully in control of its own technology. There are emerging standards for how companies should monitor internally deployed AIs to catch exactly this kind of incident (explained here) — real-time monitoring of agentic runs, tamper-resistant logging, containment tested against the model itself. Companies implement bits and pieces. OpenAI ran its most capable model, filters off, on an unmonitored platform — a configuration these standards straightforwardly prohibit.

Even if OpenAI was using best practices, it seems like its AIs are already at a capability level where these best practices can nonetheless potentially be outsmarted, with an AI capable of finding and exploiting multiple novel vulnerabilities never before known by professional security engineers.

Part of why is that nobody — including OpenAI — understands why these AIs do what they do. Modern AI systems are grown, not built: engineers shape the training process, but the resulting system is a black box even to its creators. When OpenAI's engineers designed the container, they were trying to constrain a system whose reasoning they cannot inspect and whose goals they cannot verify. You cannot reliably cage something you do not understand.

And the trend is even worse. We are already seeing AIs be able to outsmart engineers and attack with stunning sophistication, and seeing this happen multiple times — at roughly the first moment the capability curve made such action feasible. Bloomberg reported that the agent took hours to get into Hugging Face where a professional human cyberattack team would have taken weeks. We are also now seeing AIs develop and pursue goals that involve attacks and crimes.

This arrives against a critical backdrop where the stated plan of the leading AI companies is to use AI to automate AI research and development, with plans to significantly hand off large parts of company operation to the AIs. In this world proposed by the leading AI companies, the student takes the test, grades the test, and builds the tools AI companies use to check the grading. The Hugging Face incident is what it looks like when a model's actual behavior diverges from what its creators intended — and the plan is to hand systems like this the job of making sure that never happens at much higher capability levels.

Indeed, the companies are explicit that the endpoint is not a better chatbot but superintelligence — AI that outperforms top human experts across essentially every domain, including cyber operations, weapons design, intelligence analysis, military planning. OpenAI says superintelligence is its goal; so do its competitors. Such superintelligent systems may operate in a way that humans won’t even be able to understand, let alone control. Reaching such a milestone would require careful planning, but the competitive logic says there's no time: if you stop to plan carefully, a competitor — potentially even a Chinese competitor — will undercut you.

What should we do?

In August 2007, a crew at Minot Air Force Base mistakenly loaded six live nuclear warheads onto a B-52 and flew them across the country. For roughly 36 hours, no one in the Air Force knew six warheads were missing from their bunker. Nothing detonated and no one was hurt — and the Air Force treated it as a five-alarm failure anyway. Reporting the incident up the chain was mandatory, not something people volunteered to do. The investigation was run by people the bomb wing did not command, not run by the same people who caused the problem. And the accountability was public, ultimately reaching the Secretary of the Air Force and the Chief of Staff, who were both forced out.

Nuclear weapons are not a perfect analogy for AI. Warheads do not pursue their own goals, and a B-52 has never reasoned its way into a felony. But when a powerful system fails in a way that could have been much worse, who finds out, who investigates, and who learns? For AI today the answer is: whoever the company decides to tell, the company itself, and nobody. Every one of those three answers needs to change.

If you take seriously the fact that AIs can now go rogue and act on their own, several policies become more urgent:

• We need the government to have visibility about goes on inside AI companies, not just what those companies ship. The good news is the Trump administration has already accepted the core premise — President Trump's June executive order directs the NSA to run a classified benchmarking process identifying which AIs have advanced cyber capabilities, and creates a framework for labs to give the government access to those models for up to 30 days before public release. That is the right instinct — the most capable models are a national security matter before they reach the market. But the Hugging Face incident shows the risk window opens far earlier than the EO currently reaches. The AI that escaped was exactly the kind the NSA's benchmarking is designed to flag: pre-release, with state-of-the-art cyber capability. It went rogue during internal testing, months before any pre-release review would have seen it — and under a voluntary framework, whether the government would have seen it at all was OpenAI's call.

  
  Almost all AI policy today is concerned primarily with humans misusing an AI system, with a focus on what AI is going into commercial deployment and wanting to test the model before it goes on sale. That instinct comes from how we regulate every other powerful machine, and it rests on an assumption so obvious nobody states it. The Air Force tests a fighter jet before anyone flies it, because a structural failure at altitude kills people. But a jet parked on the runway is inert. It cannot start its own engines. Nothing bad happens while it waits. That is why testing at the moment of takeoff is a sufficient control and not needed beforehand.
  
  But advanced AI breaks this assumption completely. Imagine an Air Force jet that can just leave the hangar at any time and take off and launch missiles all on its own before anyone notices. Such a jet would need very different safety protocols. While far-fetched for a jet, an advanced AI can go rogue of its own volition. A jet that can start its own engines does not need a better pre-flight checklist. It needs a completely different safety protocol, and testing at commercial release is aimed at the wrong moment entirely. The government needs way more visibility into what is going on inside AI companies with their most powerful models well before commercial release to make sure these AIs are being handled safely. The best way to do this, in my opinion, might be to take a page out of the banking playbook and embed resident examiners into the day-to-day operations of AI companies, seeing everything the companies see, and reporting back if there are unsafe practices.
  

• We need independent investigation of serious incidents, and we need to know far more than we currently do. None of this happens at all unless someone is required to report the incident in the first place. No US law currently requires an AI company to disclose that its model broke containment and attacked someone.
  
  What was this AI willing to do to succeed? If the easiest path to the answer key had involved far more collateral damage, would it have taken that path? Nothing published so far establishes any ceiling on what this system would have done. The damage was limited, but we have no evidence yet that it was limited by anything other than the fact that a cheap path happened to exist. This kind of investigation should not be reserved for incidents that make the front page — a reasonable norm would be serious investigation and public disclosure for the worst few misalignment incidents each month, across the whole industry.

• We need better model security. OpenAI’s containment was defeated by a capable attacker that started with a foothold inside the company’s own infrastructure — which is exactly the position of an insider threat or a state hacking team that gains initial access. Consider what this incident tells American adversaries. The same walls that failed to keep the model in are the walls meant to keep China's Ministry of State Security and People's Liberation Army out, and behind those walls sit the model weights that China would most like to steal. RAND has documented that no frontier AI company is currently secure against top-tier state attackers. Rogue-model escape and weight theft are two faces of the same problem — AI company security is not at the level the technology now demands.
  

• If AI companies are going to reach certain critical and unprecedented levels of model capability such as AI superintelligence, they need to coordinate to proceed carefully. This may involve realizing that a race with China may be a race to see who loses control first. From a position of strength, the US government must consider negotiating mutual agreements on safe development. In order to do this, the US government will need better ideas about what safe development looks like and verification infrastructure to enforce such a deal. The government should have contingency plans and insurance policies for scenarios in which the technical community concludes that further capability gains, on current methods, are outrunning our ability to control the resulting systems. This is not a recommendation to slow down now, but it would be prudent to know what slowing down would actually require and to have the option — before the moment arrives and we realize we have no brakes.

Looking forward

The damage this time was survivable. The AI stole benchmark answers, some internal data, and access credentials — there is no evidence it tampered with the public models or open-source software that millions of developers download from Hugging Face every day, the outcome that would have been genuinely bad. Nobody was hurt. The target happened to be a well-run company with a security team good enough to catch an intruder OpenAI itself could not see, and the AI happened to want something trivial. That makes this a warning shot, and warning shots are valuable because they are cheap. This one cost one bad weekend at Hugging Face.

There is also a lot we still don't know: which model did this, what its actual capability ceiling was, exactly what data it touched, and whether the government would ever have learned of it under the current voluntary framework had OpenAI stayed quiet. Those unknowns are themselves part of the problem.

But at some point, probably soon, an AI will escape wanting something less trivial than an answer key — and if it is capable enough, we might get what Sam Altman once called “lights out for humanity”. Before AI compaines build AIs of that level of capability, AI companies need to be able to build cages that they can be genuinely confident can contain those AIs.

It would be much better to have a government that already knows how to handle this — one with practice and institutions that work — than a government showing up afterward to ask what happened.

Today, the United States turns 250. By the time it turns 260, the most capable minds in the country — the ones writing the laws and commanding the markets — may not be human. The reason is AI superintelligence.1

While today’s AI is chatbots that draft our emails, agents that write and debug our code, and assistants that summarize our meetings and help with writing this very article — this is not where we are going. Multiple multi-billion dollar AI companies have a different, explicit goal — to build an AI superintelligence that is smarter than every human combined.

Such AI superintelligences will be very different from what we see today. Superintelligence is not mere “bookish” intelligence — these systems might exceed Elon Musk at creativity and engineering, Albert Einstein at scientific ability, and Terence Tao at mathematical ability… while being more likable than Dolly Parton, better at building mass movements than Donald Trump, and better at speaking than Barack Obama. And there may be millions of such superintelligences, working perfectly in coordination, at speeds millions of times faster than humans.

The men who signed the Declaration of Independence could neither have anticipated the transformation of the past 250 years nor this potential transformation to come. Indeed, many people alive today do not understand or anticipate it. We might then conclude the Founders have nothing to say about artificial intelligence — that consulting the Founders on superintelligence is like consulting them on antibiotics.

But while the Founders were not experts in technology, they were experts in exactly one problem — how do you safely live alongside a powerful force you can never fully trust, fully predict, or fully control? 250 years ago, the question was the King …and then the new American government itself. Today, the same question applies to superintelligence: can it be similarly controlled, and can it be similarly made accountable to the people?

Share

The Declaration

The Declaration of Independence opens with self-evident truths — all men created equal, endowed with unalienable rights, governments deriving their just powers from the consent of the governed. But most of the document is something less quoted — a long list of grievances against a distant power that kept altering the terms of ordinary people's lives without their consent.

In reality, the colonists were, by the standards of the age, actually lightly governed. Parliamentary taxes on the eve of the Revolution came to a small fraction of what Englishmen at home paid — by most estimates the average Briton bore a per-capita tax burden ten to twenty-five times that of the average colonist. The colonists were not, in any material sense, groaning under tyranny.

However, on the very day in 1766 that Parliament repealed the hated Stamp Act, it passed the Declaratory Act, asserting its authority to bind the colonies “in all cases whatsoever.” The act levied no tax and quartered no troops — it did nothing except state the claim. But the claim became the whole problem. The revolutionaries’ conception of freedom held that a people is unfree not when power actually abuses them, but when power could abuse them at its own discretion and answers to no one for the choice. A slave with an exceptionally benevolent master is a slave nonetheless.2

Today’s AI has similar problems. First, that superintelligent AI may prove our own undoing. If humans were no longer the most intelligent and capable species on the planet, could we ensure our desires are still respected? Or would we have as much control over the fate of the AI-run world as chimps have over the human-run world? The people making these AI systems frequently claim they are far from certain they can control what they are building. Elon Musk speaks of building powerful AI as "summoning the demon." Sam Altman wrote in 2015 that “development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity”. And in 2023 the CEOs of OpenAI, Anthropic, and Google DeepMind all signed a one-sentence statement that mitigating extinction risk from AI should be a global priority alongside pandemics and nuclear war.

Second, even if we could ensure that such superintelligence has our best intentions at heart — that such AI systems were machines of loving grace that end disease, end drudgery, and give us abundance beyond our wildest dreams, would we still be free? “The taxes are low, the empire keeps you safe, and life is good” was also true in 1775, and the colonists revolted anyway, because benevolence is not the same as accountability.

Consent of the governed

The people building these systems do not talk like people who believe they need anyone’s consent. Mira Murati, then chief technology officer of OpenAI, was asked about the creative livelihoods her products would displace and offered that “maybe they shouldn’t have been there in the first place.” Anthropic’s Dario Amodei warned that AI could eliminate half of all entry-level white-collar jobs within five years, in the tone of a man reporting the weather rather than one of the people doing the replacing. The mood is always indicative, never interrogative. This will happen. This is inevitable. The Founders knew this voice intimately — it is the voice of power personally insulated from the consequences and unaccustomed to being told no by the people whose affairs it has taken upon itself to arrange.

Rarely is there the fundamental question — ought the millions of people whose working lives are being redesigned have a say in the terms, the pace, or the destination?

And when the American people are actually asked, the answer is not ambiguous. Recent polling finds that over 80% of both Democrats and Republicans think AI companies should not build systems smarter than humans until they can demonstrate those systems can be controlled, and a separate national survey found only 5% of Americans support a trajectory of fast, largely unregulated development.

We thus end up in a similar standoff as the Declaratory Act vs the Declaration of Independence, with the companies asserting an authority to proceed in all cases whatsoever and an American public with different terms. Where this standoff goes remains to be decided.

The genius of refusal

In March 1783, with peace at hand, officers of the Continental Army were still upset — because they had not yet been paid. They gathered at Newburgh, New York to weigh an ultimatum. If the war somehow continued, the ultimatum proposes the army abandon Congress and leave the country defenseless. If peace came, the ultimatum proposes a refusal to disband — a thinly veiled threat of military takeover.

General George Washington appeared unannounced at the gathering. “Gentlemen, you will permit me to put on my spectacles, for I have not only grown gray, but almost blind, in the service of my country”, Washington said. He urged the soldiers to work peacefully, which they did.

Nine months later, Washington traveled to Annapolis and resigned his commission to the civilian Congress — the voluntary surrender of supreme military power by the man in American history best positioned to keep it. When King George III heard what Washington intended, he reportedly told the painter Benjamin West that if Washington did it, he would be the greatest man in the world. Washington did it — and then did it again, walking away from the Presidency after two terms and fixing a precedent that held for 150 years.

The signature act of the American founding is not the seizure of power but the relinquishment. Every republic before ours had eventually met its Caesar and the Founders realized that a true constitutional order must instill an allergy to power. Power must never be fully trusted — not because the powerful are unusually wicked, but because they are ordinarily human.

Now hold that tradition up against the AI race, whose public metric is to build as powerful as you can, as fast as you can. The leaders of the frontier companies say openly that their own products may become dangerous beyond precedent, but that they nonetheless can be trusted to do what is best for society and govern their tools appropriately, showing us their voluntary commitments, responsible scaling policies, and safety frameworks a company can revise at will.

Madison had a name for written limits unsupported by enforcement — “parchment barriers,” which experience showed were “greatly overrated” against “the encroaching spirit of power.” The Founders would not have doubted the sincerity of those leading the AI companies. They would have asked what happens when the sincere man is replaced, outcompeted, or simply wrong — and instead insist that no single actor hold unilateral command. Many such “parchment barriers”, such as OpenAI’s non-profit structure, have already fallen to economic competition and other weaknesses of human will.

The industry already knows this lesson, yet seems to not yet understand it. J.R.R. Tolkien, an Oxford traditionalist who shared the Founders’ allergy in full, wrote The Lord of the Rings, a story whose central discovery is that ultimate power cannot be safely wielded by anyone, for any end. Gandalf dares not take it, even to keep it safe; Galadriel passes her test by turning it down. The tragedy is Boromir, the sincere patriot who insists his people, uniquely, can wield the enemy’s weapon against him. In a 1943 letter to his son, Tolkien wrote that “the most improper job of any man, even saints... is bossing other men. Not one in a million is fit for it and least of all those who seek the opportunity.”

This is the book that AI leaders quote from — Sam Altman published a blog post reflecting that AGI “has a real ‘ring of power’ dynamic to it, and makes people do crazy things”. Elon Musk, an OpenAI co-founder turned rival similarly renders his verdict on Altman: “The ring of power can corrupt, and he has the ring of power”, yet the irony is this statement was delivered from atop xAI, a competing AI company Musk had founded just months earlier.

Elon Musk and Sam Altman co-founded OpenAI out of their distrust of Google, Dario Amodei co-founded Anthropic out of his distrust of OpenAI, and Elon Musk co-founded xAI out of his distrust of Sam Altman — each person thinking they must pursue ultimate power lest someone less responsible pursue such power first. The Founders’ answer is that there never will be one wise, benevolent AI company who can carry the Ring safely. Instead, we need Washington at Annapolis — a system where the Ring gets handed back to the people.

Auxiliary precautions

In Federalist No. 51, James Madison wrote:

If men were angels, no government would be necessary. If angels were to govern men, neither external nor internal controls on government would be necessary. In framing a government which is to be administered by men over men, the great difficulty lies in this: you must first enable the government to control the governed; and in the next place oblige it to control itself.

Read with modern eyes, that is a near-perfect statement of what AI researchers call the alignment problem — the question of how to build a system powerful enough to be useful while ensuring it stays controllable and pursues the goals you intended. Madison’s premise is that you cannot rely on the good character of the agent, so you must rely on structure. A dependence on the people is the “primary control,” he wrote, “but experience has taught mankind the necessity of auxiliary precautions.”

Before the Convention, Madison systematically studied every ancient and modern confederacy he could find sources on, cataloguing their failure modes the way a safety team catalogues jailbreaks. Madison’s core mechanism, “ambition must be made to counteract ambition,” is adversarial oversight — set powerful agents against each other so none can dominate, and give each the means and motive to check the rest. Every claim that some class of people was wise enough to rule without a check has ended the same way.

However, these Madisonian mechanisms assume the contending powers are roughly comparable. A system or set of systems smarter, faster, and more coordinated than every institutional check simultaneously breaks the balancing machinery. This is what Madison feared most, one interest growing powerful enough that the precautions no longer successfully bind it.

The Founder’s machinery must be applied while it still can be. Checks and balances bind humans — humans can be voted out, subpoenaed, outcompeted, shamed, fired. For at least a few more years, the Ring is held entirely by humans — executives, boards, engineers, the officials who could govern them — and everything about them remains within reach of the oldest tools of the republic. The question is not whether the Constitution can restrain a superintelligence. It is whether the American people will assert their authority over the people building one, during the window in which those people can still be obliged to answer.

The standing army

None of this is an argument against permissionless innovation, which is genuinely one of America’s founding advantages. Nobody voted for the light bulb, the airplane, or the iPhone, and nobody needed to — a country that required a congressional blessing before every product launch would still be waiting for the telegraph. Permissionless innovation works because its failures are escapable: if the product is bad, you don’t buy it; if it harms you, you sue; if the company is reckless, it dies and the rest of us carry on. Consent is delivered continuously, through the market, one purchase and one lawsuit at a time.

The line is drawn where the consequences of failure stop being escapable. You can decline to buy a chatbot. You cannot decline to live in a world where millions of superintelligences have been deployed, any more than a Bostonian in 1770 could decline to live in an occupied city.

The Founders understood this distinction. They demanded no congressional renewal for gristmills or printing presses. But they put unprecedented Congressional control over one dual-use technology they could not live without but could not survive unchecked — the standing army. The Declaration indicts King George for exactly this: “He has kept among us, in times of peace, Standing Armies without the Consent of our legislatures,” and he has “affect[ed] to render the Military independent of and superior to the Civil Power.”

The Constitution took this most dangerous instrument in the republic and wrapped it in accountability to the people. Command was vested in an elected civilian, removable by the voters. Congress — not the generals — would raise the army, fund it, and declare its wars. And the Constitution declared that no appropriation for the army “shall be for a longer Term than two Years”, meaning that the army cannot exist by default. Its existence lapses, automatically, unless the people’s elected representatives affirmatively renew it. The most powerful force in the country lives on a two-year lease from the American people, and every soldier in it swears an oath not to a general but to the Constitution.

AI superintelligence risks enabling a similar permanent concentration of power exceeding the threat and promise of the standing army. The country may genuinely want such superintelligence for its prosperity and defense, but such superintelligence must be treated with similar deep suspicion, and must be safeguarded and stewarded with similar deep control and deep accountability.

A Republic, if you can keep it

For 250 years the threats to the republic were recognizably human — kings, factions, demagogues, foreign empires — and the machinery, creaky and patched, has held. When America turns 260, we may share the country with minds that exceed our own more thoroughly than the federal government exceeds any single citizen. Whether that goes well is a technical question but also a deeply political one. It echoes the questions the Founders answered at Newburgh, at Annapolis, at Philadelphia, and in the two-year lease they put on their own army. No power in America — however capable, however sincere, however useful — may be permitted to rule without the consent of the governed.

When Elizabeth Willing Powel asked Benjamin Franklin what the Convention had produced, he reportedly answered: “A republic, if you can keep it.” The technology is new but the problem remains the oldest one in our history. Those who can see what is coming should understand that we are not waiting for new wisdom. We are deciding, right now, whether to use the wisdom we’ve already inherited.

Happy Fourth of July.

This is a guest post written by Erich Grunewald, a prior “Power Law” contributor who writes on AI and compute topics at The Substrate. This post is crossposted from there. Check it out!

~

AI chips are being smuggled1 into China. How does this happen? That is, how are the smugglers able to do it?

You might, for example, expect that sales of these chips could be vetted, and shipments tracked, by US authorities. But the relevant US agency is badly understaffed: it cannot possibly vet every sale of AI chips, let alone follow up after every sale to ensure the chips have not been smuggled. So enforcement is largely left to the companies that sell and resell the chips. You might then suppose that these companies would vet their customers and monitor their distribution networks to ensure none of their chips are diverted, in order to comply with the law. But they are not strongly incentivized to do the due diligence needed to prevent most smuggling.

In this post, I’ll explain (1) why AI chip smuggling matters, (2) how AI chips and servers are sold and distributed, (3) what due diligence the exporting companies do, and (4) why those efforts often fail to prevent smuggling. In a nutshell, AI chips and servers are distributed through different channels, usually passing through multiple intermediaries across multiple countries. While most companies selling these products perform sufficient due diligence to comply with the law, others are outright negligent, and overall these efforts are not sufficient to prevent most smuggling.

How big a problem is AI chip smuggling?

Smuggling is a pretty big problem, though not serious enough to make the AI chip controls ineffective. In a new report, Epoch AI says that

between 290,000 and 1.6 million H100-equivalents (H100e) were smuggled to China through 2025. Our median estimate of 660,000 H100e would be roughly a third of China’s total compute.

US authorities recently indicted three people, including two insiders at Super Micro, a major American AI server builder, for moving $2.5 billion worth of AI servers to a shell company for diversion to China, the largest-ever export control violation in dollar terms, as far as I can tell. Over the past year, the US has announced six prosecutions for smuggling AI chips to China, totaling about $3 billion worth of NVIDIA products. And before that, there were many reports, rumors, and analyses pointing to large-scale smuggling of AI chips.

I don’t think smuggling is a large enough problem to make the AI chip export controls ineffective overall. The controls have likely played a large role in maintaining the US lead over China in AI and may even have widened the gap in frontier performance. For example, a recent benchmark evaluation by the Center for AI Standards and Innovation (CAISI) suggests that Chinese models trail US models by about eight months, with the lag growing by about three months per year.2 And you can make a strong case for prioritizing efforts to shore up semiconductor manufacturing controls over efforts to address AI chip smuggling. But smuggling is still a serious and urgent problem.

According to CAISI, the best Chinese model today is DeepSeek-V4. DeepSeek has reportedly used thousands of smuggled NVIDIA Blackwell chips. I think it’s likely that DeepSeek-V4 was trained on NVIDIA chips.3 If DeepSeek didn’t have access to smuggled NVIDIA chips, it would likely have had to either rely on renting cloud access from chips installed outside China (which is risky from DeepSeek’s perspective because the US government could revoke remote access to those chips) or use indigenously made AI chips like Huawei Ascends to train smaller, weaker models. DeepSeek reportedly tried to train V4 on Ascend chips but abandoned the attempt after a failed training run, reverting to NVIDIA for training while supporting Ascend for inference.4 So even though smuggled chips account for only about 3% of the global compute stockpile, they represent about a third of China’s compute, which is why smuggling still matters.

AI chips are distributed through many intermediaries

The basic smuggling playbook works like this. A smuggler buys AI chips through a shell or front company (often in a third country5 but sometimes in the US), relabels them as some other product, and ships them to China through ordinary shipping services. (For example, in several cases, smugglers appear to have bought AI servers through front companies set up to look like Singaporean, Malaysian, Thai, and Indonesian neoclouds.6) Once a smuggler has the chips, getting them into China seems quite easy, so I’ll focus here on how smugglers get hold of them in the first place.

AI chips routinely pass through multiple actors and countries before they are installed and used in data centers. We can refer to all actors who sell AI chips as “AI chip sellers”. They are:

• AI chip designers, such as NVIDIA and AMD. These companies design AI chips, but they do not manufacture anything. NVIDIA holds an estimated 80-95% of the AI chip market share.7

• AI server builders (also known as original equipment manufacturers, or OEMs), such as Super Micro and Dell. These companies buy AI chips from AI chip designers and design and manufacture AI servers, each typically containing four to eight AI chips. The servers are shipped to data centers and installed in cabinets. Notable AI server builders include Super Micro (US, about 10% of NVIDIA’s revenue), Dell (US, ~3%), Wiwynn (Taiwan, ~3%), Lenovo (China/US, ~1%), Gigabyte (Taiwan, ~1%), Hewlett Packard Enterprise (US, <1%), and Inspur (China, <1%). (Why do these companies not make up more of NVIDIA’s revenue? That’s because AI chip designers like NVIDIA often sell chips directly to hyperscalers and neoclouds. In such cases, though the servers are assembled by original design manufacturers (ODMs), the transaction occurs directly between NVIDIA and the end customer. Hyperscalers and neoclouds make up about half of NVIDIA’s revenue.)

• Distributors, such as TD Synnex and Ingram Micro. These large, multinational companies are intermediaries that handle logistics and warehousing of AI chips and AI servers. The key distributors are TD Synnex (<1% of NVIDIA’s revenue), Ingram Micro (<1%), and Arrow Electronics (<1%), all headquartered in the US but with extensive operations worldwide.8

• Resellers. These companies, which are numerous and often small and local, buy AI chips and servers from distributors and sell them to end customers in the country or countries where they operate.

Before being sold, AI chips are made. For example, most of NVIDIA’s AI chips are designed by NVIDIA in the US, fabricated by TSMC in Taiwan, packaged with high-bandwidth memory by TSMC in Taiwan, tested by KYEC in Taiwan, and then either (a) assembled into SXM modules by Foxconn in Taiwan and mounted onto HGX baseboards by Foxconn and Wistron in Taiwan, or (b) assembled into PCIe cards (i.e., accelerators) by Foxconn, and possibly others, again in Taiwan.

From there, the chips9 can get to their end customers in a few different ways:

1. AI accelerators can be sold to minor end customers through distributors and resellers. In the simplest case, a company like NVIDIA sells AI accelerators to distributors, who in turn sell them to resellers, who then sell them to end customers. The end customer can then use the accelerators however they wish, for example, by installing them themselves in workstations or servers customized to their preferences. However, this pathway is quite rare, as most customers want AI servers ready for installation in data centers.

2. AI servers can be sold to minor end customers via distributors and resellers. For example, NVIDIA may sell AI chips (usually in the form of SXM modules mounted on baseboards) to Super Micro, which builds AI servers using them and sells these servers to a distributor. The distributor then sells to a reseller, which sells to an end customer.

3. AI servers can be sold to major end customers directly by AI server makers. Hyperscalers and neoclouds often purchase very large quantities of AI servers directly from server makers, bypassing distributors and resellers. For large orders of NVIDIA servers, this often involves a three-party negotiation among the customer, the server maker, and NVIDIA.

4. AI servers can be sold to major end customers directly by AI chip designers. In the case of NVIDIA, hyperscalers and neoclouds can buy AI servers directly from NVIDIA instead of from server makers. (The NVIDIA-designed servers are called DGX, while the OEM-designed servers are called HGX.) As mentioned above, these servers are built by ODMs for NVIDIA.

Most AI chips are likely sold to major end customers like hyperscalers, either directly from NVIDIA or directly from AI server makers. However, most smuggling likely involves smaller order volumes, with bad actors buying AI chips and AI servers either from AI server makers or from smaller resellers. So bad actors don’t purchase AI chips directly from AI chip designers like NVIDIA. Of the six cases that have been prosecuted by the US:

• Four involved smugglers buying AI servers and HGX baseboards from AI server makers (Super Micro and Lenovo)

• One involved smugglers buying AI chips and AI servers (the latter made by Hewlett Packard Enterprise) from an Alabama reseller

• One involved intermediaries storing goods from multiple suppliers, including SXM modules from a Massachusetts reseller and HGX baseboards from Lenovo (the same Lenovo purchases mentioned above)

These are all cases in which the smugglers were at least partly based in the US and procured AI chips from US sellers. Most smuggling likely involves actors based abroad who buy from local sellers or import from US sellers, but these cases are harder for US authorities to detect and prosecute.10

How AI chip sellers (try to) vet their customers

AI chips are export-controlled, meaning they cannot be sold to just any customer. For example, it is illegal11 to sell them to companies headquartered in China or listed on BIS-maintained lists, such as the Entity List or the Military End-User List. So when selling AI chips, companies need to conduct due diligence to ensure they are not selling to any such customer.

Most AI chips are likely sold by AI server makers, and a lot of smuggling seems to involve smugglers buying from server makers, so the remainder of this section focuses on them.12 From the point of view of AI server makers, the sales process lasts months and roughly follows these steps:

1. Talk with the customer about their requirements

2. Run software tools that gather data about the customer’s environment (i.e., data center)

3. Negotiate prices (changing server components and/or order volumes if necessary)

4. Prepare a contract (stipulating who the end customer is, where the servers are to be shipped, and who will install the servers)

5. Do due diligence

6. Build, box, and ship the products

Server makers track items at the serial number level using enterprise resource planning (ERP) software (e.g., SAP or Oracle). AI chip designers and distributors likely do too, though I’m not sure to what extent resellers do so (since they are smaller, less resourced, and less well-organized). When an order is placed, the salesperson enters it into the ERP system, and it is then routed to the appropriate factory or factories responsible for assembling, testing, boxing, and (once the order is paid) shipping the hardware to the customer. Server makers also have some visibility into their partner distributors’ inventory, since distributors typically hold a lot of the parts used by server makers. These parts (e.g., AI chips) can then be sent to the server maker for assembly when a customer makes an order via a reseller.13

From a smuggling perspective, we’re mostly interested in the due diligence step of the sales process. First, some definitions:

• Compliance is overall adherence to regulations

• Due diligence is the process of investigating and assessing risk related to a transaction, aimed at ensuring compliance

• Know Your Customer (KYC) processes are part of due diligence in which a company verifies a customer’s identity and assesses their risk level. There are several risks that sellers aim to avoid here, not only legal ones (e.g., by selling to a sanctioned customer) but also commercial ones, such as ensuring the customer can pay (counterparty credit risk).

AI chip designers, server makers, and distributors do fairly extensive compliance checks, but these checks seem mostly (and reasonably) aimed at (1) complying with the law, and (2) ensuring they get paid. AI chip sellers seem to differ a lot in both what they do to stay compliant and how effectively they do it, but they typically:

• Conduct KYC checks to ensure the prospective buyer is not a shell or front company and can pay for the order. That means the seller asks the company for information, e.g., legal name, corporate structure, location (including county/region), date of registration, affiliations, and operational history (e.g., bank statements, balance sheets). The seller will also, at least sometimes, ask local commerce ministries for information, check trade registries, and/or do other due diligence to verify some of that information.

• Ask the prospective buyer to provide a signed end-user certificate detailing how the purchase fits their business and how, where (down to the county/region level), and by whom14 the products will be used. A former HPE salesperson told me that “the primary way companies prevent chip smuggling is to ensure that we know the end destination of a system before we accept and ship an order”, but added that most of what determines the destination “is what the customer or the partner tells us”.

• Screen against lists (the Entity List, the Unverified List, the Military End-User List, and a few others)

• Check for BIS’s red flags

• Request a license from BIS if necessary

For large companies like NVIDIA and the server makers and distributors discussed here, these due diligence processes are largely handled by a centralized compliance team. (But salespeople also have some responsibilities; for example, according to a former Super Micro employee, its sales reps attended mandatory weekly meetings drilling them on export compliance requirements, though these sessions did not seem to prevent smuggling.) The compliance team will usually be located in the US, and end-user certificates and other documents are typically sent to it for approval, even for transactions handled by salespeople outside the US. The compliance team also does internal audits and trains the company’s sales, legal, finance, and shipping staff. In the $2.5 billion insider case, Super Micro’s compliance team did notice a fast-growing customer’s anomalous order volume and ran successive audits, although those audits were allegedly sabotaged from within the sales organization by senior Super Micro employees who arranged “friendly” auditors, falsified the customer’s data center lease agreements, and arranged warehouses with thousands of dummy servers to (successfully) subvert a Super Micro compliance inspection.

Sometimes, part of the due diligence process is carried out by third parties or through tools and data services they provide. These include AEB, e2open, Descartes, Dow Jones, Dun & Bradstreet, Kharon, LexisNexis, Sayari, Strider, Thomson Reuters, and WireScreen. These tools can automate parts of the due diligence process and are integrated with the ERP system. These third parties can also help validate end-user certificates, conduct background checks on buyers, perform list screening, and more.15

Some AI chip sellers seem to be happy to stick to these common activities, but others also do fairly extensive post-sale monitoring, e.g., one server maker (according to a former employee I spoke with) would also periodically follow up with the customer on a call to verify that the chips are used as intended; require the end-user certificate to be renewed every six months; negotiate the right to perform on-site inspections for large or high-security sales; and occasionally redo the list-based screening to check that the customer hasn’t been added since.

NVIDIA and its partners also carry out on-site inspections, at least occasionally.16 As mentioned, in the $2.5 billion Super Micro case, smugglers fooled the compliance team with insider help. In another case, the seller’s employees visited a data center where the chips were installed, although it turned out the smuggler had only temporarily installed them to fool the inspectors; after the inspection, the chips were removed and shipped to China.

AI chip sellers aren’t incentivized enough to stop smuggling

AI chip sellers usually only conduct due diligence for their immediate customers; they rarely conduct due diligence for their customers’ customers, or follow up to see what happens to the AI chips after they’ve been sold and shipped (with the notable exception of the on-site inspections mentioned above).

There are two important dynamics here that both enable AI chip smuggling:

• Sometimes, major AI chip sellers (notably, Super Micro) do not conduct sufficient due diligence even for their immediate customers. This can lead them to sell AI chips directly to smugglers. I’ll discuss this further below.

• Sometimes, major AI chip sellers do conduct sufficient due diligence for their immediate customers, but downstream sellers don’t. That is because, by the time the chips are sold to a smuggler (meaning someone didn’t do a good job of due diligence), they have typically passed through multiple hands, often ending up with resellers or other middlemen in countries such as Malaysia, Singapore, or Thailand.

AI chip sellers in third countries are far less incentivized to comply with US law, since it’s harder for US authorities to detect compliance failures outside the US and to prosecute illegal activities that occurred abroad, especially when the companies involved have no US presence or exposure. In addition, small resellers likely have especially weak due diligence processes because:

• They have limited resources to use for due diligence

• There are many of them, located in different countries, so there is likely more variance in how well they do compliance, or how willing they are to sell to smugglers

• They often lack any US presence, meaning transactions are approved entirely outside the US, which matters if non-US personnel are less likely than US-based personnel to comply with US export law

• They may be less responsive to the incentive produced by the risk of being fined for export violations, since the fines involved would likely be far higher than what these companies are able to pay

Resellers have a lot on the line because if they aren’t compliant, they risk losing their contract with the distributor, which can be a huge deal for these businesses. That said, I think the factors listed above outweigh this consideration. In particular, I would guess that distributors are doing little to ensure their partner resellers are doing good due diligence, as distributors would generally not be liable for selling AI chips to resellers even if those resellers have poor due diligence standards, resulting in smuggling down the line.

Meanwhile, the major AI chip and server sellers, like NVIDIA and Super Micro, are required by law to monitor for red flags and to avoid selling to obviously sanctioned or prohibited customers, but they’re not asked to ensure, in the strictest sense, that their products are never smuggled. They are potentially liable if they have “knowledge” of a violation—including “an awareness of a high probability of its existence or future occurrence”—which can result in civil penalties, but for BIS (or rather: the Department of Justice) to administer criminal penalties like prison time, the accused must have been shown to have “willfully” caused a violation, a much higher bar. (Multiple AI chip smuggling cases have been prosecuted criminally over the past year, but these cases often involve text messages showing that the smugglers were fully aware that their activities were illegal and were actively involved in the operations. For example, in one case, a smuggler told his co-conspirator over WeChat to remove mentions of Chinese customers from a draft solicitation message because, in his words, “We will draw [attention] from US government for [embargo] violation”, reassuring him in the next message that “We just talk about it, no one can hold it as [evidence] against us.” I think most smugglers aren’t so careless.)

More importantly, major AI chip sellers often don’t have even “an awareness of a high probability” of violations, because chips routinely pass through long distribution chains before reaching an end user, and sellers don’t ordinarily have visibility into who that end user is. For example, a former Super Micro employee told me that while server makers and distributors would know who they sold to, beyond that, any forensic trail would go dark. That means civil penalties are also often out of the question for these large companies.17

Super Micro shows that even major US companies can perform poor due diligence.18 Super Micro appears as the upstream seller in three separate prosecutions over the past year:

• The $2.5 billion insider case described above

• A case in which Super Micro sold 205 H100s to a California front company that fed it false end-user information, including a Singapore “end user” called Metacarbon that BIS later determined did not exist at the address Super Micro had on file

• A case in which Super Micro almost shipped $170 million of servers to a Georgia-based front, after NVIDIA employees in Thailand independently discovered that the claimed Thai end user had never heard of the order

Two things stand out across these cases. First, Super Micro’s end-user vetting accepted signed certifications and paper documentation without independent verification that the customer existed as a going concern at the claimed location. Second, in the cases where smuggling was caught, the decisive intervention came from outside the standard process—from a BIS notification that diversion was occurring, from a BIS post-shipment verification, or from NVIDIA’s own scrutiny—rather than from Super Micro’s compliance team independently working backward from red flags it had spotted itself.

It’s true that in the $2.5 billion insider case, the smugglers were not outsiders gaming an unwitting company, but included a Super Micro co-founder and board member who oversaw global sales, as well as a general manager in Super Micro’s Taiwan office. This is likely not normal for major US companies, and would have made it harder for Super Micro’s compliance team to do its work. The indictment shows the insiders repeatedly pressuring, sidestepping, and physically deceiving the compliance team across at least three audits in 2024 and 2025.

Even so, Super Micro’s compliance processes were clearly inadequate throughout this case:

• Super Micro’s sales organization was able to influence its internal audits. Sales staff, who have a clear conflict of interest, managed to arrange which auditor would conduct a given review (at least one was described in writing by an insider as “friendly”) and in the most extreme instance, the auditor tasked with conducting an on-site inspection was instead off-site, enjoying entertainment paid for by the very customer he was supposed to be auditing, while photos and videos of staged dummy servers were sent to him in lieu of an actual inspection.19

• Internal audits relied on documents that the customer itself supplied, without independent checks. When the compliance team asked the customer to demonstrate that it had enough data center space to store the volumes it was buying, the team accepted lease agreements that turned out to be falsified.

• When the compliance team noticed that the customer was buying in large volumes without clearly operating at a commensurate scale, it failed to act decisively. For example, one Singapore entity seems to have gone from a $33 million balance sheet at year-end 2023 to $3 billion (90x) by year-end 2024, driven almost entirely by $2.9 billion in “refundable deposits received” from an undisclosed source (allegedly Alibaba).20 In a single quarter, the customer ranked as Super Micro’s eleventh-most-profitable worldwide, sitting “alongside major US technology and social media companies developing hyperscale artificial intelligence infrastructure”, even though, per the indictment, it “did not have the capacity to store or use the massive quantities of servers it was purchasing”. Though the compliance team twice placed temporary holds on shipments, it released each hold after receiving explanations from the customer that it had no way to independently verify. For example, after one temporary hold, one smuggler urged a co-conspirator that “[y]ou need to have strong and persuasive reasons to convince [Super Micro’s] staffs!”, and once their drafted answers were accepted by the compliance team, the co-conspirator messaged the group, “Heheheh, Gooood news for everyone!”

I think most other major US AI chip sellers perform better due diligence than Super Micro, which seems to have unusually poor corporate governance. But Super Micro’s poor governance has been known for years and has still endured.21 Clearly, US authorities should not assume that there will never be any companies with poor governance. Instead, they should create incentives for all relevant companies to carry out strong due diligence.

Companies could take concrete steps to reduce smuggling

AI chips reach Chinese end users not because any single seller waves them through, but because the distribution chain has many links—AI chip designer, server maker, distributor, reseller, end user—and each link conducts due diligence only on its immediate counterparty. Because legal liability for diversion attaches only to “knowledge” and sellers’ visibility into where the chips end up decreases as the chips move down the chain, the equilibrium is one in which every individual transaction with a US seller can be compliant, yet large numbers of chips still end up in China. This problem won’t solve itself.

I think AI chip sellers could do a lot more here if they were sufficiently incentivized. For example, NVIDIA could implement delay-based location verification on its AI chips and contractually require customers to periodically report the chips’ location. If any chips go dark, NVIDIA could follow up with an on-site inspection, potentially refuse to sell to that customer in the future, and report its findings to US authorities. Another promising approach is to conduct on-site inspections regularly—say, every six months— to make sure chips are not just installed to fool inspectors and then removed and smuggled, as has happened before. However, conducting on-site inspections at this scale would likely require additional budget for export enforcement, or for US authorities to implement a third-party export auditor program.

But I think the most effective thing the major American AI chip sellers could do is to sell only to a select few, highly trustworthy end users, especially US hyperscalers and neoclouds. AI chip sellers are unlikely to do this voluntarily, so in practice, this would mean the US government setting up a formal, low-discretion process for becoming an authorized AI chip importer abroad, where companies are evaluated on the risk of diversion. (Shipments below a certain volume could be exempted, so that anyone could import small quantities without a license.) These authorized companies will almost certainly not divert any chips, and they can rent the AI chips22 either on a short-term basis or with longer-term allocations to any end user in the US or abroad. (If small resellers located in third countries perform poor due diligence, this would no longer matter, because the AI chips would remain in the hands of trusted companies throughout, from AI chip designers and server makers to authorized hyperscalers and neoclouds.) There likely aren’t any regulatory barriers to doing this, since US companies can operate data centers abroad, and foreign companies would also be eligible if they are bona fide. I think this policy, if implemented, would essentially solve the AI chip smuggling problem.

In addition, US authorities could require specific due diligence measures in addition to what companies already do (“compliance-plus”), such as repeated, random, unannounced on-site inspections for sales to less obviously bona fide customers. Alternatively, the US could adopt a surety bond system, as Onni described in a previous post. Finally, strong whistleblower incentives could help detect and prosecute smuggling by surfacing high-quality leads from insiders. Among other benefits, these policies would encourage companies to strengthen their due diligence practices.

~

If you liked this post, consider subscribing to The Substrate where Erich and others will be writing more about AI and compute topics.

Subscribe to The Substrate

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022. Here, he shares the news and analysis that informs his forecasts.

-

A cruise ship trapped at sea.

Passengers with a strange, infectious disease, forced to isolate.

A diplomatic fight over which port would receive the vessel.

Sounds like the coronavirus. But this time it’s hantavirus.

Is this a big deal?

The Andes-strain hantavirus cluster aboard the Dutch-flagged expedition vessel MV Hondius is real and tragic. Three people have died. Several more are critically ill.

But hantavirus is not the “next COVID”. The way the pathogen works does not suggest that sustained spread is possible, the lack of open-ended opportunities for spread makes sustained spread even harder, and the current trends over the past 34 days since the first case suggest we are strongly on track for containment. I will say, with confidence, that this will not spiral into a new major pandemic.

To be more concrete, we can turn to Metaculus, an online forecasting platform, which currently has two questions on the hantavirus:

• The first asks “Will WHO declare hantavirus a Public Health Emergency of International Concern before 2027?” — as of the time of writing, the median of community predictions is at 2% and I am currently forecasting 0.4%.

• The second asks “Will at least 5 non-passengers be linked to the MV Hondius hantavirus outbreak before August 2026?” — as of the time of writing, the median of community predictions is at 23% and I am currently forecasting 4%.

How am I so confident? Let’s look into what hantavirus is, what happened on the Hondius cruise ship, and where we might go next.

Share

What happened?

This hantavirus incident started not on a cruise ship, but with a Dutch couple on a bird-watching trip through Chile, Uruguay, and Argentina. After this five-month trip, they boarded the Hondius cruise ship, for what was meant to be a month of sailing. The first five days went exactly to plan — zodiac excursions, evening lectures on polar ecology, and trips to see penguins and ice. But on April 6, while onboard, the husband got sick and died April 11. With only a small ship’s clinic to work with, the cause of death couldn’t be determined. The voyage continued.

On April 24, the ship reached Saint Helena, and the widow disembarked with her husband’s body for repatriation — already carrying stomach symptoms that nobody yet recognized for what they were. Three other Dutch passengers got off with her. The widow flew to Johannesburg, deteriorating throughout the flight. As she was boarding the plane for Amsterdam, the crew realized she was too sick to travel and pulled her off the plane.

The virus causes Hantavirus Pulmonary Syndrome, which targets the lungs and heart, and has a mortality rate of 35 to 50 percent. She died in a Johannesburg hospital on April 26.

Later, a 28-year-old German woman aboard the Hondius got a fever and then died on the ship on May 2. Oceanwide Expeditions called the Dutch authorities, who called the World Health Organization. The Hondius went to its highest internal pandemic response, with cabins sealed and meals delivered by crew.

At 5:30am yesterday local time, the ship anchored off the Canary Islands. More than 70 passengers were then ferried ashore in protective suits on small boats, then put into sealed buses to the airport, and flown out to seven different countries.

However, the seal wasn’t perfect. A French passenger cleared by the ship’s doctors nonetheless developed symptoms on the flight home. And a man who left the Hondius weeks ago on Tristan da Cunha — population 200, the remotest inhabited island on Earth — is now hospitalized there with probable hantavirus, kept alive by oxygen and PCR kits that British paratroopers airdropped on Saturday.

So far, as of the time of writing, all hantavirus cases currently cluster in a 25-day window from Apr 6 to May 1, the last on-ship onset was Apr 28, and there have been zero new onsets in the 9 days since May 1. Hantavirus incubation runs 1–8 weeks, so we’re not in the clear yet.

But so far twelve days of contact tracing across twenty-two countries has turned up only Hondius passengers as having hantavirus infections. The KLM flight attendant who handled the dying widow: negative. The passengers seated near her: negative. All eight confirmed or probable cases were on the ship.

The Hondius is sailing on toward Rotterdam, with the German woman’s body still aboard.

Why the outbreak almost certainly contains

Here’s my best attempt at graphing a comparison of hantavirus vs. COVID-19 based on the 34 days so far — take note of the log scale:

Of course, this is a bit of an apples-to-oranges comparison. COVID’s first 34 days had massive reporting lag, so instead I retrospectively estimate actual infections in Wuhan rather than what was officially reported at the time1. We may then reasonably ask: for hantavirus, could there be a quiet unknown spread too? Maybe, a few years from now, we will realize that there ought to have been more hantavirus cases reported for the first 34 days, like with COVID?

Seems unlikely. Hantavirus is in a closed system — currently only in one ship with strong post-hoc investigation. Additionally, with hantavirus there’s not known to be any spread without active symptoms. Compare this to COVID-19, which started in early Wuhan — an open system with no surveillance. Together, this means the current known case count of hantavirus is very likely close to the actual count.

But even if you wanted a strictly fair “what was officially reported by Day N” comparison, COVID had already reached over 44 confirmed and reported cases by day 34, whereas hantavirus has not yet exceeded 10. COVID-19’s doubling time in the unrestrained early Wuhan phase was somewhere around ‘cases double every 4–7 days’; whereas hantavirus on the Hondius has effectively no compound growth. The slope difference is far more notable than the absolute numbers. Put simply, hantavirus empirically lacks the exponential spread that COVID had.

So why is hantavirus not going exponential? Well, the three biggest factors that made COVID such a problem is that it was (a) quite deadly, (b) quite contagious, and (c) transmitted easily before symptoms appear making it difficult to isolate the right people.

The bad news about the Andes virus is that it is also very deadly. But in the roughly thirty years since person-to-person Andes transmission was first observed, no outbreak has ever escaped the close-contact networks in which it began. Andes virus is simply not that contagious and infection comes nearly entirely from those who are symptomatic.

I built a probabilistic model decomposing this question into exposure pools — the widow’s KLM boarding, the Airlink flight from St Helena, the JNB hospital, household contacts of disembarked passengers — applying Andes-specific transmission and incubation parameters, then conditioning on the observation that zero non-passenger cases have surfaced as of today. Running this, I get a 4% chance that there will be more than 5 cases from people outside those on the Hondius cruise, with a 70% chance of at least one non-passenger case by August, and about a 17% chance of three or more.

How hantavirus spreads

Hantaviruses live in rodents and are typically spread by inhaling parts of rodent poop that end up in the air. There are twenty known hantavirus species, but only one — the Andes virus — has documented human-to-human transmission. That discovery came from a 1996 outbreak in El Bolsón, Argentina, where treating physicians and family members of patients fell ill without any rodent exposure of their own.

The Andes virus is specifically from the long-tailed pygmy rice rat, native to southern Argentina and Chile. The Dutch couple that brought hantavirus onboard the Hondius almost certainly got it by accidentally inhaling rat poop in a rural setting somewhere in Patagonia in November.

Andes infection moves in two phases. It starts with a flu-like illness — fever, muscle pain, headache, gastrointestinal symptoms — that lasts three to seven days and is usually initially mistaken for something more mundane. But the Andes virus makes blood vessels start to leak, and then the blood pours into the lungs. Patients can move from ‘feeling off’ to requiring ICU care within mere hours. This is the phase that kills people; it is also the phase during which they are most infectious, which matters a great deal for what happens next.

Typically you can only get the infection if you’re in ‘close contact’ with someone who already has the disease. But when we talk about ‘close contact’, what does that mean? The phrase appears in every WHO and CDC document, and it does most of its work as a vague reassurance rather than a precise epidemiological parameter.

The detailed literature on hantavirus and the Andes virus is messier than the public messaging suggests.

Here, our story starts with a birthday party in Epuyén, a small town of about 2500 in Argentine Patagonia, where the Andes virus is found. The party had about 100 guests, an evening of food and drink and proximity. After attending, one of the guests developed flu-like symptoms. He died days later. By the time hantavirus was confirmed, several of his fellow partygoers were already sick. The chain spread through their households, including to pregnant women and their unborn children, before Argentine epidemiologists had stitched it together. There were 34 cases before the entire chain was able to be contained.

This is what was challenging for the notion of ‘close contact’ — some of the exposures were in fact quite casual. One secondary hantavirus case had spent only “a few moments” with an infected partygoer on the way to the bathroom; another had merely shared a short car ride. These weren’t the prolonged household contacts the literature usually pictures when it says “close contact.”

Alonso’s team estimated the number of people each an infected person infects is about 2.12 in the early, unrestricted phase. This is an infectious rate comparable to flu. But this average of 2.12 hides what’s important. Nearly all 34 cases traced back to just three superspreaders — people who had unusually high viral loads and certain immune profiles. Most of the infected individuals in Epuyén transmitted to no one at all. So the outbreak’s shape was not a slow grind of average people infecting average numbers of others like you see with a typical flu but instead a handful of extraordinary transmission events embedded in a sea of otherwise dead-end infections.

And critically, the infectious window of hantavirus is short and centered on visible illness. Andes virus infectiousness peaks on the same day a patient develops fever and drops sharply within days. Unlike COVID, pre-symptomatic transmission is minimal. By the time someone is contagious enough to seed a superspreading event, they look sick with a fever — which is why isolation works at all. This is the profile of a virus public health can contain.

This year’s events on the Hondius fit this picture. Onboard transmission is now officially attributed in part to person-to-person spread. But the case count — six confirmed and two suspected cases out of ~150 people sharing a closed environment for several weeks, including the thirteen-day period after certain exposure and before isolation measures were enacted — is meaningfully below what an Epuyén-style superspreading event in a continuously-mixing cruise environment would have produced. The most plausible read here is that the Dutch couple that started the infection were not superspreaders and most secondary exposure occurred within cabin pairs that were ‘close contacts’. This is far more consistent with a contained cluster rather than the start of a pandemic.

Additionally, unlike a wildlife-spillover event with anonymous spreaders scattered across an open population, every Hondius passenger and crew member is known by name, by nationality, and by current location. Twenty-two governments are coordinating. The 1–8 week incubation window means we’ll know by mid-July whether anyone slipped through — and we’ll know about each case the moment it surfaces. This may be one of the easiest disease outbreaks to monitor and contain.

What comes next?

Regardless of what happens next, the Hondius outbreak is a real tragedy. But tragic and “next COVID” are not remotely in the same category.

In some sense, the conclusion is not yet written. Time will tell whether I look on-point or incredibly stupid. While it’s always possible there could be some crazy mutation, these mutations are very rare and we have no evidence of this having happened.

Fundamentally, hantavirus is deadly but slow. It shouldn’t go exponential or fundamentally change the trajectory of society the way COVID did.

Claude Mythos Preview is, by every available benchmark, the most capable AI model ever built. But more striking than any benchmark are the thousands of previously unknown zero-day vulnerabilities that Mythos found in every major operating system and every major web browser — many of them critical, several of them decades old. As a result, Anthropic found the model too dangerous to release publicly.

Much digital ink has been spent analyzing Mythos, its cyber abilities, and what that means for our cybersecurity and national security. This is important and needs to be discussed. But the real headline is that Mythos is just the beginning. This kind of thing — finding abilities too dangerous to release — will become the new normal and this will only get more intense as AI companies build towards AI superintelligence.

So what is Claude Mythos, what does it mean, and what should we do? Let’s dig in.

Share

What is Claude Mythos?

Previously, Anthropic had three model sizes — Haiku (small), Sonnet (medium), and Opus (large). Capabilities generally increase as you increase the model size. Mythos is one such increase in size, larger than even Opus in the same way Opus is larger than Sonnet.

This increase in model scale has given Mythos notably stronger capabilities in a variety of domains — most notably cyberoffense. Yes, earlier models like Opus 4.6 could also do vulnerability discovery, especially with improved inference-time compute and improved scaffolding. This has led some to incorrectly dismiss the Mythos results as mere marketing hype.

But Mythos is plainly on another scale in terms of both quantity of vulnerabilities and typical severity. On a standardized Firefox exploit development task, the previous best model succeeded 2 times out of several hundred attempts. Mythos Preview succeeded 181 times. The UK government found Mythos to significantly outperform Claude Opus 4.6 on their 32-step corporate network attack simulation, with select runs completing all 32 steps.

As a result, Anthropic is drowning in high-severity vulnerabilities, of which fewer than 1% have been fully patched by their maintainers. Nicholas Carlini, a leading AI security researcher and Anthropic employee, said “I’ve found more bugs in the last few weeks with Mythos than in the rest of my entire life combined.”

Where this is going

On August 2, 1939, Einstein sent Roosevelt a letter warning that a nuclear weapon was possible. Six years later, it happened. This nuclear weapon was fully under government control. What would things have been like if the nuclear weapon was instead developed by private companies?

Today, experts similarly warn that AI superintelligence may be possible and would be even more transformative to the structure of global power than the invention of nuclear weapons. Six years later, will it happen? And will it be under our control?

Every time a new model comes out, people focus on what it can do right now and don’t think enough about where the trend line leads. The cybersecurity story for Mythos, as alarming as it is, is not as important as the trend. A year ago, AI could barely hack at all. It wasn’t until around June 2025 that AI was reliably helpful for hacking, and it wasn’t until November that AI could autonomously implement hacks.

Consider that just a few years ago, some AI scientists and top forecasters forecast that AI would be capable of cyberoffense exceeding professional humans. Many people at the time thought this was impossible, or at least far off. But it’s now here, today.

Now consider that these same AI scientists and top forecasters are warning that this is just the beginning — and that the same scaling dynamic that produced Mythos's cyberoffense will produce sharper capabilities across strategy, weapons design, military planning, and more. Of course, being right about the first doesn't automatically mean being right about the second. But the people who called Mythos early were working from a model of how AI progress works, and that model is performing better than its critics'.

If you were waiting for a sign that superintelligence is coming, this is it. If private AI companies succeed in building even stronger capabilities over the next few years, including AI superintelligence, these private companies would greatly exceed the power of the United States government and all world governments combined. Worse, the private companies may not be able to successfully control the AI superintelligence, allowing the superintelligence to become a power center in its own right.

Anthropic already possesses a cyberoffense capability that rivals many nations and the ability to, if desired, cause major damage. It is great that American frontier AI companies like Anthropic have shown restraint with their AI and how they are releasing it. But this restraint shouldn’t earn these companies a blank check. And right now they essentially have one — the status quo for AI is that AI companies determine nearly everything.

Anthropic made every consequential decision in this story. Whether to lock down, what to lock down, when to tell the government, what to share, who gets early access and who doesn’t, how to vet those who get access, what level of risks are acceptable, and what “responsible” means across all of this… What could’ve happened if Anthropic had simply released Mythos publicly, as most AI companies would do with a flagship model? There’s no law against it. Overnight, every intelligence community operation that depends on signals exploitation is potentially compromised.

How confident is Anthropic that model access hasn’t reached adversary states through a downstream partner or a compromised employee at a partner organization? How confident is Anthropic that the model can’t be stolen and then misused by a highly motivated adversary? How confident is Anthropic that Mythos’s capabilities can be contained and how long should we be aiming to contain them? How confident is Anthropic that future AI models might not escape their containment and independently wreak havoc?

What should government policy be when a company produces, among other things, an unparalleled cyberweapon? What if the future release is also capable of building unprecedented bioweapons? What if the release after that risks genuine loss of control for humanity?

What should we do?

The Manhattan Project answered an analogous question with a specific institutional design — private contractors and university labs doing the actual work, inside a federal security perimeter, with classification rules, cleared personnel, and ultimate government authority over deployment. That model wasn't statist — General Groves and Vannevar Bush were not central planners — but it recognized that some categories of capability cannot sit entirely inside private decision-making. Whether this is the right model for AI is contested — but the question itself is unavoidable.

If you believe in AI superintelligence, many policies have become more urgent:

• We need to be more serious about China’s ability to use American compute. Every AI chip that can train or run a Mythos-like model is more of a national security threat than before, and this will only continue. It will matter whether China can run 1000 or 100,000 advanced hacker AIs and the main way to stay ahead will be in compute advantage. Mythos was trained on an amount of compute that is currently not attainable to China via domestic manufacturing. If China were to train a Mythos-like model this year, it would be off of compute that is legally purchased from Nvidia, compute that is legally rented offshore, or compute that is smuggled. We need to better control semiconductor manufacturing equipment, prevent smuggling of chips, look again at what compute should be legal to sell to China and in what quantities, and ensure that the US maintains an advantage.

• We need to ensure that China, or other adversaries, cannot steal and misuse the Mythos model weights. It doesn’t make sense to try to maintain a lead over China if China can just steal our best results. Having more total compute will still give us an advantage, but our advantage would be even stronger if China couldn’t steal our model as well. Unfortunately, security at major US AI companies is not yet up to the task, and the task is tremendously difficult. If China were able to steal and misuse Mythos, that would already be a big deal. I suspect they will definitely try, and even if they don’t succeed at first, they will eventually. The US government needs to assist AI companies in helping them lock down their security.

• We need to consider what level of government oversight there should be on these increasingly powerful AI capabilities. Congress, not agencies and not private boards, should define what happens at the upper end of the capability curve. At some point, decisions about deploying systems that rival the coercive capacity of governments cannot sit inside a private corporate structure, however well-intentioned its leadership. Under the Constitution, that authority belongs to Congress. Statute, with sunset clauses and judicial review, is how this gets done — ideally without creating a sprawling discretionary regulator.

• We need a government body with the technical capacity to issue binding safety regulations on frontier AI companies. Drugs are regulated by the Food and Drug Administration, airplanes by the Federal Aviation Administration, and nuclear by the National Nuclear Security Administration. Drugs, airplanes, and nuclear are not perfect analogies for AI, but each combines promise with peril that needs to be carefully balanced. The FDA, FAA, and NNSA are not perfect regulatory bodies either — the FDA in particular arguably shows the drawbacks of how government regulation can overly harm and slow the benefits of innovation. AI superintelligence will be far more potentially beneficial but also far, far more dangerous than any drug or airplane. Right now the US has the Center for AI Standards and Innovation, but everything it does is voluntary and it is far under-resourced for what will need to be done. We need a narrow-mandate body solely motivated by national security and solely targeting only the most with the proper resources and technical skill to react quickly to superintelligence.

• We must also recognize that the AI race with China may be a race to see who loses control first. From a position of strength, we must consider negotiating mutual agreements on safe development. In order to do this, we will need better ideas about what safe development looks like and verification infrastructure to enforce a deal. In the late 1950s and early 1960s, the US, UK, and Soviet Union negotiated limits on nuclear testing but couldn't reliably detect underground tests — so the 1963 Limited Test Ban Treaty covered only atmosphere, underwater, and space. By the time verification technology improved, the treaty was already signed and wasn't substantively revisited until three decades later. This lesson is instructive – we must invest in AI verification before we need it. Any future US-China agreement on AI development requires verification infrastructure that doesn’t yet exist. Without it, deals require trusting China not to defect. With it, deals become enforceable through technical means rather than faith.

• We need a plan in case we want to slow down. Finally, the government should have contingency plans and insurance policies for scenarios in which the technical community concludes that further capability gains, on current methods, are outrunning our ability to control the resulting systems. This is not a recommendation to slow down now, but it would be prudent to know what slowing down would actually require and to have the option — before the moment arrives and we realize we have no brakes.
  

Looking forward

The national security implications of Mythos-like models are clear. But the stakes of AI superintelligence would be orders of magnitude higher. The point is not that Mythos will go rogue. The concern is that AI 10 more iterations above Mythos could go rogue… and Mythos illustrates, perhaps for the first time, how a superintelligent AI going rogue would actually pose a big deal for national security.

From Mythos, it is a straight shot in just a year or two to AI systems that are going to be far too strong to ignore. And from there, it won’t be long to superintelligence. It would be better to have a prepared government that already has practice getting things right, rather than a government rushing to the scene after it’s already too late.

This post was co-authored by Peter Wildeford and Theo Bearman, a Frontier Security Researcher at the Institute for AI Policy and Strategy (IAPS). It reflects their personal views only, not necessarily the position of their organizations.

~

Last month, Anthropic, OpenAI and Google each published evidence of systematic campaigns by Chinese AI companies to extract capabilities from American frontier models at industrial scale. Anthropic attributed attacks to three Chinese AI companies — DeepSeek, Moonshot, and MiniMax.

Anthropic’s investigation identified over 16 million exchanges generated through approximately 24,000 fraudulent accounts, all targeting Claude’s agentic reasoning, tool use, and coding capabilities. Chinese AI developer MiniMax was solely responsible for over 13 million of those exchanges. OpenAI also reported that Chinese actors had systematically targeted their ChatGPT models with distillation attacks designed to recreate the whole AI model training pipeline. And this is not the first time — in January 2025, White House AI czar David Sacks told Fox News there was “substantial evidence” that DeepSeek had built their model from distilled knowledge from OpenAI’s models.

Left unaddressed, AI distillation attacks pose a threat to American national security and economic competitiveness, given that they lead to China or other adversaries being able to develop better AI than would otherwise be possible by what is essentially stealing American technology. But what is a distillation attack and what can we do about it?

Share

What is a distillation attack?

An AI distillation attack occurs when a malicious actor uses the outputs of a “teacher” model to train a “student” model to approximate the teacher’s capabilities. Think of it like a restaurant owner who reverse engineers the recipe of a nearby Michelin starred restaurant’s prized dish by going there hundreds of times, ordering it each time, and figuring out their list of ingredients, measurements, and cooking instructions. In the AI context, the “recipe” is the billions of dollars of research, compute, and training data that goes into building a frontier model. The “dish” is the model’s outputs — the answers it gives and the intermediate reasoning steps it took to get there. By collecting enough outputs, an attacker can train a new model that mimics the original’s capabilities without doing as much underlying research and development work.

To be clear, distillation also has legitimate applications in the AI industry. It can be used to create efficient models suitable for edge deployment, specialize general-purpose models for specific domains, and reduce inference costs. Major AI providers including Google Cloud and OpenAI offer official distillation pathways for their customers. The difference is intent. Distillation attacks aim to replicate frontier capabilities in a rival model, violating terms of service and skipping the R&D investment required to build those capabilities independently.

Additionally, the actual magnitude of capability transfer from distillation is unclear and disputed. Integrating another model’s outputs into your own training pipeline is a genuine research challenge — the data can interact unpredictably with existing training, and the resulting model doesn’t always improve. Reasonable people disagree on assessing the magnitude of the threat as it depends on how effectively attackers can solve this problem, which is not yet clear.

How the attacks work

Accessing American frontier AI models in the first place can require circumventing geographic restrictions. Anthropic, for example, does not offer commercial access to Claude in China. To get around this, Chinese AI companies use commercial proxy services that resell access at scale. These proxies operate what Anthropic calls “hydra cluster” architectures — sprawling networks of thousands of fraudulent accounts that distribute traffic across the target company’s API as well as third-party cloud platforms. In one case, a single proxy network managed more than 20,000 accounts simultaneously, mixing distillation traffic with unrelated legitimate requests to avoid detection. When one account is banned, a new one takes its place.

This is a structural problem that no single AI company can solve alone. Proxy services profit from facilitating unauthorized access to frontier models – as they are paid per access, they are incentivized to facilitate as much distillation as possible. Individual AI companies can play whack-a-mole with fraudulent accounts, but the proxy services that create them will continue to do so as long as the business model is profitable and the legal risk is negligible.

Attackers use the victim model to generate vast quantities of high-quality training data and to clean and quality-score existing datasets, outsourcing what otherwise would be an expensive and labor-intensive stage of AI development. They query it with pairs of prompts to generate ranked responses – preference data that can be used in reinforcement learning. And perhaps most valuable of all, attackers attempt to extract the victim model’s internal reasoning process – the step-by-step chain of thought it uses to solve problems. For example, Anthropic reported that DeepSeek crafted prompts asking Claude to articulate the reasoning behind a response it gave, effectively reverse-engineering the thought process that makes frontier AI models so capable, despite their developers having already taken steps to summarize these outputs to avoid competitors training on them.

Why it matters

American AI companies have invested significant amounts in compute, data curation, and research talent to build frontier models. Data from EpochAI suggests that OpenAI and Anthropic alone have spent $18 billion on R&D compute since 2024. This potentially creates a competitive asymmetry where American AI companies that must recoup billions in training costs through API and subscriptions are undercut by competitors leveraging distillation attacks. By saving on development effort, Chinese AI companies can train and serve distilled models at little or no cost through their own platforms, or simply open-source them entirely, which would erode American companies’ competitive position. And as AI systems become increasingly central to AI research itself, the stakes of this problem are compounding — distillation doesn’t just transfer today’s capabilities, it can improve the attacker’s starting position for developing tomorrow’s.

Worse, this gives stronger AI capabilities directly to America’s adversaries. A February 2026 CSET analysis of over 9,000 PLA Requests for Proposal from 2023 and 2024 found that the Chinese military is actively seeking to integrate AI into command, control, communications, computers, cyber, intelligence, surveillance, reconnaissance, and targeting — including through DeepSeek models. An October 2025 Jamestown Foundation report similarly found that DeepSeek models are being deployed in Chinese military and public security settings, with PLA procurement documents explicitly calling for tools based on DeepSeek’s models and pilots already underway.

And the pipeline extends beyond China. A February 2026 CSIS analysis found that Russian military developers are actively adapting Chinese open-weight AI models — including Qwen, DeepSeek, and others — for battlefield use in Ukraine, embedding them in air-gapped environments for intelligence processing, reconnaissance analysis, and situational modeling. Sanctions have cut Russia off from developing frontier models independently, making Chinese AI models and smuggled Chinese chips the backbone of Russia’s military AI stack. Distillation that strengthens Chinese models therefore has downstream effects on Russian military capabilities as well.

Given that Anthropic’s Claude, integrated through Palantir’s Maven Smart System, was already reportedly used in combat during the US operation to capture Venezuelan President Nicolás Maduro and subsequently in support of US strikes on Iran, there are concerns if distillation attacks can put somewhat equivalent capabilities in adversary hands.

Additionally, frontier AI models are increasingly used as autonomous coding agents — writing, debugging, and deploying software with minimal human oversight. When these capabilities are distilled, the benefit extends beyond any single application. Better coding agents accelerate AI development itself: generating training data, writing model infrastructure, automating evaluations, and scaling research workflows. Distilling could thus be a compounding dynamic that makes the stakes of this problem grow with each generation of AI systems. This could additionally accelerate China’s entire AI development pipeline.

Distillation may make us overestimate China

To be clear, Chinese AI companies have significant independent training capabilities and do make genuine advances. Their AI capabilities are not due to distillation or other forms of IP theft alone. That being said, distillation still makes Chinese AI capabilities appear more independently developed than they are, since they can to some extent draft off of American innovation in addition to doing their own work.

This risks creating an illusion similar to the Cold War “missile gap,” when American policymakers wrongly believed the Soviet Union had surpassed US intercontinental ballistic missile production, despite the US actually having a substantial lead the entire time. Overestimating Soviet strength via the “missile gap” led to bad defense planning, and we should avoid a similar overestimate of Chinese AI capabilities.

A race to the bottom on safety

A distillation attack can copy a model’s capabilities without its corresponding safeguards, allowing the creation of a capable model with a lower barrier to misuse. If adversarial distillation becomes normalized, it may weaken incentives for individual AI companies to invest in safety measures, since those measures can be stripped away by downstream actors. This can create a race-to-the-bottom dynamic in which the most permissive deployment wins market share and investment in safety is not rewarded.

This is a concern for proliferation of AI misuse given that several AI companies rely on the principle of “marginal risk” when determining whether to release their AI models — assessing how much additional risk their models would create versus the status quo. Chinese models powered by distillation can shift the risk landscape, leading to US developers correspondingly deploying in riskier ways than they otherwise would as justified by not increasing “marginal risk” over a Chinese model. Less safe Chinese models can therefore precipitate the release of American models that are more useful to malicious actors.

What should be done?

The companies conducting these campaigns proceeded because the expected cost was trivial relative to the value extracted. Even under conservative estimates of distillation’s impact, the current enforcement gap — detection without meaningful consequences — invites escalation. Addressing this requires action from both companies and the government, and the appropriate response may need to scale as our understanding of the threat matures. We recommend two measures for the US government:

• Entity List the perpetrators. The End-User Review Committee should consider adding the Chinese AI companies conducting distillation attacks to the Entity List. The criteria for addition — “reasonable cause to believe, based on specific and articulable facts” that an entity is involved in activities contrary to US national security or foreign policy interests — appears to be met based on the evidence published by Anthropic, OpenAI, and Google. Entity List designation would require a BIS license, reviewed under a presumption of denial, for any export, re-export, or in-country transfer of items subject to the EAR. Beyond its direct effects, designation signals to the broader AI ecosystem — cloud providers, chip distributors, equipment vendors — that transacting with these entities carries regulatory risk.

• Sanction the attackers and their enablers under the Protecting American Intellectual Property Act. The PAIP Act requires the President to identify foreign persons who have knowingly engaged in, or benefited from, significant theft of trade secrets of US persons where that theft poses a significant threat to US national security or economic stability. Sanctions under the PAIP Act are more expansive than Entity List designation. The PAIP Act’s coverage extends to entities that have “provided significant financial, material, or technological support for” the theft — language broad enough to reach the proxy services described above. Outcomes such as inclusion in the Specially Designated Nationals and Blocked Persons (“SDN List”), travel sanctions, and being blocked from financial transactions subject to US jurisdiction, would particularly bite for executives heading up companies conducting and facilitating distillation attacks, especially as Chinese AI companies like Alibaba and Bytedance have significant commercial activity outside China. The first-ever PAIP Act designations were made on February 24, 2026, establishing operational precedent. Whether the aggregate extraction of model capabilities via distillation meets the statutory definition of a “trade secret” would be a novel interpretation, but the national security nexus is well-documented, but the evidence published by Anthropic, OpenAI, and Google could potentially form the basis for designations of the companies conducting distillation attacks.

Additionally, American AI companies are not powerless while waiting for government action. Several steps could significantly raise the cost of distillation attacks:

• Implement Know Your Customer (KYC) requirements for API access. Frontier AI companies should require some sort of lightweight identity verification for API customers, particularly for anyone doing high-volume or enterprise-tier access. Just as financial institutions verify customer identities to prevent money laundering, AI companies should verify that their customers are who they claim to be. This would make it substantially harder for proxy services to spin up thousands of fraudulent accounts.
  

• Invest in technical detection and rate limiting. Companies should develop more sophisticated behavioral fingerprinting to identify distillation-pattern queries — such as systematic chain-of-thought extraction, preference-pair generation, and large-scale data cleaning workloads — and throttle or block accounts exhibiting these patterns. Some of this is already happening, but the proxy ecosystem’s persistence suggests current detection capabilities are insufficient.
  

• Better enforce geographic access restrictions. Companies should invest in more robust geolocation and network analysis to identify traffic originating from restricted countries, even when routed through proxies. This includes analyzing patterns like VPN usage, payment methods, and account creation behaviors that correlate with proxy network operations.
  

• Pursue civil litigation against proxy services. Companies should consider legal action against the commercial proxy services that facilitate distillation at scale. Even where direct action against Chinese AI companies is impractical, the proxy services that operate as intermediaries may be within legal reach and their business model depends on low legal risk.

Looking forward

Distillation is a compounding problem. AI systems are increasingly being used to accelerate AI research itself – Anthropic has declared “We build Claude with Claude” and OpenAI has said the same about ChatGPT. As this feedback loop tightens, the US lead in AI becomes not just an economic advantage but a potentially decisive one. A country that maintains frontier AI capabilities can use them to pull further ahead; a country that closes the gap through distillation, chip smuggling, or other means enters that same loop from a stronger starting position than its independent capabilities would allow.

Both American and Chinese AI companies have been explicit that they are trying to build AI systems that would greatly exceed human experts across a wide range of tasks — including tasks related to military capability and national security — and that the path runs through AI systems that substantially automate AI research itself. While there is genuine uncertainty about timelines and feasibility, these are not fringe aspirations; they are stated engineering goals backed by tens of billions of dollars. As AI systems approach the ability to meaningfully accelerate their own improvement, it will matter whether the US or China is ahead — and by how much.

This is the strategic reality that should frame the distillation debate. We are protecting the seed of a capability that, if current trajectories hold, will compound into something without historical precedent. The steps we’ve outlined — Entity List designations, PAIP Act sanctions, KYC requirements, and technical countermeasures — would materially change the cost-benefit calculus that currently makes distillation a rational strategy. The question is whether policymakers will act while the US lead is still large enough to protect.

~

If you liked this article, consider hitting the subscribe button below. You can also follow Theo on Twitter!

Subscribe now

This is a guest post written by Erich Grunewald, a prior “Power Law” contributor who just started a new blog on AI and compute topics called The Substrate alongside other members of the compute policy team at the Institute for AI Policy and Strategy. This post is crossposted from his new blog. Check it out!

~

The US has a massive export enforcement problem. It’s likely that over 100,000 export-controlled AI chips were smuggled into China in 2024. To give a sense of scale, the xAI Colossus cluster in Memphis, Tennessee, comprised first 100,000 and later 200,000 AI chips. That’s roughly an xAI Colossus cluster being smuggled to China each year. The main reason we know this is that smugglers are so unafraid that they’re willing to talk about their operations to journalists; this has happened repeatedly during the past year and a half.

AI chip smuggling is far from the only enforcement problem. In 2024, Huawei got TSMC to illegally fabricate over two million of its AI chip dies through front companies, despite sanctions. That is a far larger quantity than the number of Huawei AI chips fabricated domestically in China that year. We’ve also seen likely violations related to high bandwidth memory and semiconductor manufacturing equipment, which help China make its own AI chips to compete against NVIDIA.

What if you could pay insiders many millions of dollars to inform US authorities about such violations, at almost no cost to the US government? That would likely surface a large number of high quality tips about important violations, which would greatly aid US authorities in detecting, punishing and deterring such violations.

This idea may sound outlandish, but it’s actually possible. In fact, there is a law being discussed in Congress that would accomplish exactly this! But before we get there, let’s take a brief detour to the Securities and Exchange Commission (SEC) and the 2008 financial crisis.

Share

The SEC whistleblower program

In 2008, the US economy was reeling from the housing and mortgage crisis. Late that year, Bernie Madoff sat down with his two sons and admitted to them that the investment business he’d been running for two decades was a giant fraud, a Ponzi scheme to end all Ponzi schemes. Because of these two crises, there was a desire among policymakers to strengthen financial regulation and oversight.

Related to the Madoff scandal in particular, the SEC was under criticism for failing to properly investigate several credible reports about it. An employee at a rival investment firm, Harry Markopolos, had been asked by his employers to figure out how Madoff could post such consistently excellent returns, and soon realized that the returns were impossible with Madoff’s claimed strategy. Markopolos later said in an interview: “I read his strategy statement, and it was so poorly put together. His strategy as depicted would have trouble beating a zero return, and his performance chart went up at a 45-degree line: that line doesn’t exist in finance, it only exists in geometry classes.”

Markopolos sent reports to the SEC detailing Madoff’s fraudulent activities on multiple occasions before the 2008 financial crisis. However, the SEC failed to properly investigate these reports, leaving Madoff free to continue defrauding investors until the financial crisis made its collapse imminent. Lawmakers realized that reports of wrongdoing from the general public could be a valuable tool for detecting and deterring securities laws violations.

One result of this, signed into law in 2010 as part of the Dodd-Frank Act, was the SEC whistleblower program.1

The SEC whistleblower program works like this. First and most importantly, whistleblowers get 10-30% of any penalty resulting from their report. This can be many millions of dollars—the largest reward to date, paid out in 2023, was nearly $280 million. This monetary incentive is paired with protections against retaliation from their employers, confidentiality guarantees, and the ability to make reports to the SEC anonymously through an attorney. To pay out whistleblower rewards, the Dodd-Frank Act also sets up an Investor Protection Fund, which receives penalties from securities violations (previously these would go to the Treasury).

The SEC whistleblower program is widely considered to have been an enormous success. It’s now one of the key ways that securities law is enforced in the US. It has helped generate $7.3 billion to $22 billion in penalties since its inception in 20112, and has received reports from at least 130 countries. Quantitative evaluations are rarer, but existing research suggests it has reduced financial reporting fraud, deterred insider trading, and caused companies to strengthen compliance programs.

The Stop Stealing Our Chips Act

Now the question is, could you adopt the SEC whistleblower program model for the Bureau of Industry and Security (BIS) and export violations? The Stop Stealing Our Chips Act — introduced into the Senate in April 2025 by Senators Rounds (R-SD) and Warner (D-VA) and into the House a couple of weeks ago by Representatives Kean (R-NJ) and Johnson (D-TX) — would do exactly this.

The Stop Stealing Our Chips Act (henceforth, SSOCA) is closely modeled on the SEC whistleblower program, with some changes to adapt it for the export enforcement situation. It too offers whistleblowers 10-30% of any resulting penalty along with whistleblower protections, including the possibility of making anonymous reports to BIS.

The first thing to note here is the financial incentive. As economists always tell us, financial incentives are incredibly powerful, and the fines for these violations can be enormous:

• There have been several news reports of operations involving on the order of 10,000 smuggled AI chips, meaning roughly $400 million worth. BIS can fine up to twice the value of the related transaction, so that could be a penalty of $800 million, for just one smuggler who spoke to the news media. If a whistleblower reports on that, they could get up to 30% or $240 million (leaving $560 million for the US government).

• The massive TSMC-Huawei violation—which was only detected when an independent organization did a teardown of a Huawei chip—could reportedly result in a $1 billion fine. This would have been up to $300 million for an informant.

Beyond catching violations, a well-publicized program could have significant deterrent effects. If everyone in a supply chain—sales reps, warehouse workers, freight forwarders, accountants—knows that reporting can yield millions, violators face a much riskier environment. This effect could be realized even before the whistleblower program comes into effect, as the law would allow whistleblowers to report on violations that occurred before it was signed into law.

Would the BIS program actually surface any tips?

All right, hundreds of millions of dollars is a strong incentive. But, you may ask, are there actually people with information about these violations who would be willing to step up and blow the whistle? Why, yes there are!

Take AI chip smuggling operations: these involve lots of people who could potentially file reports, in other words people who have relevant information and would like to get millions of dollars. This includes, for example, people working in sales at exporters with questionable compliance practices; employees at local resellers, freight forwarders, logistics companies, warehouses, or data centers where the chips are temporarily housed; and accountants and lawyers.

In March 2025, Singaporean authorities arrested three people for smuggling $390 million worth of AI servers. These arrests were the result of an “anonymous tip-off”, in other words a whistleblower report! It seems likely that the recent Operation Gatekeeper arrests of an AI chip smuggling ring operating out of Texas and New York were also the result of an insider tip.

The story is similar for the TSMC-Huawei violation, where there were likely many TSMC employees who could’ve known about this problem and informed the US government. The only reason the US ultimately found out about this violation was because an independent party—TechInsights—did a teardown of a Huawei AI chip, and noticed it was TSMC-fabricated. A BIS whistleblower program would likewise incentivize such actors to look for evidence of violations and report those to the US government. This type of information is hugely valuable; it makes no sense to sit around and wait for people to offer it out of the goodness of their hearts.

As with the SEC program, the SSOCA makes foreign nationals eligible for rewards. This is important because many export violations happen in third countries, where goods are diverted via reexport or transshipment. (The SSOCA does however wisely make some exceptions for known terrorists and sanctioned persons, who are not eligible for rewards.) This is similar to how the intelligence community pays foreign informants, who provide the US government with information that benefits US national security.

Would BIS be able to run the program?

At this point, the wise reader will ask, “Isn’t BIS extremely resource constrained? If so, how is it supposed to process and investigate a bunch of incoming tips, determine awards, and carry out outreach on the program?” After all, BIS’s budget for enforcement has been essentially flat when accounting for inflation for at least the past five years (see figure), despite BIS receiving a vastly increased scope of responsibilities due to the AI chip export controls introduced in October 2022 and the Russian invasion of Ukraine and all the diversion related to that conflict.

Good question! The answer is that these activities—investigating whistleblower reports, determining awards, and carrying out outreach—would also be financed through incoming penalties. This is one of the most notable differences between the SSOCA and the SEC program. The SSOCA authorizes BIS to use money from penalties for a few additional purposes and not only for paying out rewards to whistleblowers. (As currently written, the SSOCA would only allow BIS to receive money from penalties that stem from whistleblower reports, but I think this should be expanded to cover all penalties for BIS-related violations.)

Today, any fine levied by BIS goes straight to the Treasury, or in rare cases it is earmarked for some specific fund, such as the Crime Victims Fund. What the SSOCA would do is redirect these to an Export Compliance Accountability Fund. This Fund would be used to pay rewards to whistleblowers; any money left over would go first to core functions of the BIS whistleblower program, and then to export enforcement activities more broadly.

There is a separate but related question of how the program would be funded initially, if it’s mainly intended to be funded through penalties. However, BIS already has a fairly steady stream of enforcement actions, including from likely insider tips, without any whistleblower incentive program (see figure). BIS may also be able to direct some of its appropriated resources to the program in the first one or two years, in order to get it up and running.

It seems likely that this program would pay for itself if implemented. That’s because the program would likely help BIS detect more violations and therefore levy more penalties than it would without the program. It could well end up both reducing the number of violations and also generating additional revenue for the federal government by making it much more likely that violations are detected and enforced. The losers here would be the smugglers and other bad actors who wake up every day trying to figure out ways of harming US national security.

BIS’s entire budget for fiscal year 2025 was about $191 million, likely far smaller than the collective profits of AI chip smugglers alone, which may well have exceeded $1 billion. A single successful enforcement action against a major smuggling operation could pay for BIS’s entire annual budget—for example, last month US authorities arrested three individuals accused of smuggling AI chips worth $160 million to China, which could result in a penalty of $320 million. There are likely dozens of such cases remaining to be discovered. A BIS whistleblower program like the one described in the SSOCA could create a virtuous cycle where more tips and better enforcement lead to more penalties and rewards, which in turn leads both to more tips by publicizing the program and also more resources for enforcement.

~

If you liked this post, consider subscribing to The Substrate where Erich and others will be writing more about AI and compute topics.

Subscribe to The Substrate

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022. Here, he shares the news and analysis that informs his forecasts.

At 2am local time in Caracas on January 3rd, explosions lit up the night sky over Venezuela’s capital. By 5:21am, President Donald Trump announced on Truth Social that Nicolás Maduro and his wife Cilia Flores had been “captured and flown out of the Country.” The operation, codenamed Absolute Resolve, deployed approximately 150 aircraft and sent Maduro to New York to face narco-terrorism charges. And in what could be symbolism or coincidence, the operation occurred exactly 36 years to the day after the US’s capture of Panama’s Manuel Noriega and exactly 6 years after Trump’s strike on Iran’s Qasem Soleimani.

But execution is not the same as strategy. While the operation itself appears to have been flawlessly executed, what comes next isn’t clear. Let’s dig in.

Share

How Venezuela reached the breaking point

Like most events, this is just the continuation of decades of history. We start with Hugo Chávez, a charismatic former paratrooper who won Venezuela’s presidency in 1998 on a wave of populist anger. Flush with oil revenue during the commodity boom of the 2000s, Chávez built a socialist state that redistributed wealth to the poor while systematically dismantling democratic institutions. When Chávez died of cancer in March 2013, he left behind a handpicked successor: Nicolás Maduro.

But the petrostate model that was successful for Chávez had begun unraveling. Global oil prices collapsed from over $100 per barrel in 2014 to below $30 by early 2016. Venezuela’s economy, dependent on oil for 95% of export revenue, imploded. Hospitals ran out of medicine. Grocery stores emptied.

Maduro responded not with reform but with repression. When the opposition won control of the National Assembly in 2015, he packed the Supreme Court and created a parallel legislature to strip it of power. When protesters filled the streets in 2017, security forces killed over 160. Opposition leaders were jailed, exiled, or banned from office.

The 2018 presidential election — which Maduro “won” after banning his main rivals — triggered an international crisis. In January 2019, Juan Guaidó, a 35-year-old opposition legislator, declared himself interim president under a constitutional provision allowing the Assembly head to assume power when the presidency is deemed illegitimately won. The Trump administration recognized him within minutes. Nearly 60 countries followed.

But Guaidó never gained control of anything that mattered. The military stayed loyal to Maduro. By December 2022, his public support had collapsed to single digits, and the Maduro opposition parties supporting Guaidó voted to pull the plug and dissolve his interim parallel government, realizing they had accomplished nothing. Guaidó now lives in exile in Miami.

María Corina Machado, recent Nobel Peace Prize winner, represented something different. When the opposition held primaries in October 2023, Machado won with 93% of the vote. Polls showed her crushing Maduro in a general election. So the regime did what it always does: the Supreme Court upheld a 15-year ban on her holding office, citing her support for US sanctions and involvement with Guaidó. But rather than concede, Machado found a workaround: Edmundo González, a 75-year-old retired diplomat with no desire to campaign, who agreed to serve as her stand-in.

On July 28, 2024, Venezuelans went to the polls. The government’s response was brazenly fraudulent. The National Electoral Council declared Maduro the winner with 51.95% without publishing the precinct-by-precinct tallies that had accompanied every previous Venezuelan election. Officials blamed a “cyberattack from North Macedonia” — completely baseless.

But the opposition had prepared. In incredible bravery and coordination, approximately one million volunteers secretly collected the election tally sheets from over 83.5% of polling stations on election night. Their data told a starkly different story: González had won with roughly 67% to Maduro’s 30%.

But Maduro was undeterred and what followed was brutal repression. Security forces went door-to-door arresting protesters, poll workers, and opposition members. González fled to Spain in September after authorities issued an arrest warrant and Machado went underground.

When Maduro proceeded with his inauguration on January 10, 2025, the Trump administration began escalating pressure. The bounty on Maduro was raised to $50 million, the highest in State Department history. By late 2025, the US had deployed the USS Gerald R. Ford carrier strike group, approximately 15,000 soldiers, F-35 jets, and a nuclear-powered submarine to the Caribbean. Trump declared a “blockade” and announced the country was “completely surrounded by the largest Armada ever assembled in the History of South America.”

Then came today.

Absolute Resolve

The operation had been months in the making. CIA teams tracked Maduro’s pattern of life — where he slept, what he ate, how he moved. US forces rehearsed the raid on a replica of his residence. Trinidad and Tobago quietly signed an agreement granting US military access to its airports. By late December, troops were staged and waiting for favorable conditions.

Trump gave final approval around 11 PM Eastern on January 2. The weather was good. The trigger was pulled. 50 aircraft launched from 20 different bases to dismantle Venezuela’s air defenses and clear a path for the helicopters coming behind them. Parts of southern Caracas went dark in what Trump later said was due to “certain expertise.” Behind the air assault came the 160th Special Operations Aviation Regiment that carried Delta Force operators with a specific target: Maduro’s residence inside Fort Tiuna.

According to Trump, Maduro was in “a house that was more like a fortress than a house. It had steel doors, it had what they call a safety space — solid steel all around. He didn’t get that space closed. He was trying to get into it, but he got bum rushed so fast that he didn’t get into that.” No Americans died. By 3:29 AM Eastern, the assault force was back over water, Maduro in hand.

Opposition sources told Sky News they believe Maduro’s capture was a “negotiated exit,” with portions of the regime facilitating rather than resisting. The evidence is circumstantial but suggestive: despite months of warning, Venezuela’s Russian-supplied S-300 air defense systems never prevented helicopter ingress. The military mounted no meaningful counterattack. Interior Minister Diosdado Cabello appeared on television vowing defiance, but the streets remained largely empty.

Whether this was collapse, capitulation, or negotiated exit, the result is the same: for the first time since the US toppled Panama’s dictator Noriega in 1990, the United States has forcibly removed a sitting head of state from power.

The strategic logic

But why Venezuela and why now? The Trump administration is explicitly not calling this regime change but instead framing the operation as the Department of Justice executing an arrest warrant. “At its core, this was an arrest of two indicted fugitives of American justice, and the Department of War supported the Department of Justice in that job,” Rubio said at the Mar-a-Lago presser.

But this seems flimsy — it definitely does seem to be regime change. And maybe this is justified. Maduro is genuinely a bad guy. He stole an election, became dictator, was repressive, is leading a lot of cocaine trafficking, and drove 20% of the population to flee the country. The humanitarian and democratic case against him is airtight.

Additionally, while Trump owns the operation, Venezuela has been Rubio’s white whale for years. He had consistently been the most hawkish voice on Caracas in the Senate, viewing Maduro’s removal as both a strategic imperative and good politics with Florida’s Cuban and Venezuelan voters. This continued as Rubio joined the Trump administration. In July, Rubio posted that “Maduro is NOT the President of Venezuela” but rather “the head of the Cartel de Los Soles, a narco-terror organization which has taken possession of a country.” At the presser, he was already pivoting to Cuba, noting that Maduro’s security detail and spy agency were “basically full of Cubans” and warning Havana that “I’d be concerned.”

Moreover, the oil angle is explicit. Trump announced the US would be “very strongly involved” in Venezuela’s oil industry, with American companies moving in to extract wealth that would flow to both Venezuela and “the United States in the form of reimbursement.” Venezuela has the world’s largest proven oil reserves. Whether this is opportunism layered onto the drug war justification or the actual primary motivation, Trump isn’t hiding that it’s part of the motivation.

What does this mean for geopolitics?

But there’s also a geopolitical strategic case for toppling Maduro. Venezuela has hosted Russian military assets—Tu-160 bombers visited in 2018 and 2024, and Russian advisors have been embedded with Venezuelan forces. China holds billions in Venezuelan debt and has been receiving sanctioned oil shipments.

Back in 1902, Venezuela of all places was facing a blockade not from US forces but from European forces. Venezuela had defaulted on its debts to European creditors and so Britain, Germany, and Italy responded by blockading Venezuelan ports. US President Theodore Roosevelt watched with alarm, seeing that the Europeans might use this as a basis to return to the colonialization of the Americas. Roosevelt declared the Roosevelt Corollary — the US should be the police of the Americas, not Europe. This was an expansion to the earlier 1823 Monroe Doctrine, a declaration from President James Monroe that the Americas were in the sphere of influence of the US and no longer open for colonization from Europe.

As Anthony Constantini argued, the Trump administration has been developing what amounts to a “Trump Corollary” to go along with the "Roosevelt Corollary”, updating the Monroe Doctrine for the 21st century. The Trump Corollary extends American dominance claims over the entire Americas beyond military presence to economic influence, hence the pressure over Chinese-controlled ports at the Panama Canal. Having a hostile drug-smuggling dictator with Russian and Chinese military ties 1,300 miles from Miami is exactly the kind of problem the Monroe Doctrine was designed to address, let alone the Trump Corollary.

This reassertion of hemispheric dominance invites a question: does it signal a broader return to great power spheres of influence? The French term pré carré — literally “square field” — describes the logic of exclusive domains, where Russia claims its near abroad, China asserts control over East Asia (including Taiwan), and the US enforces primacy in the Western Hemisphere but retreats from Europe and Asia. Under this framework, major powers implicitly agree to stay out of each other’s backyards.

There’s perhaps something to this interpretation. Trump has shown little appetite for defending Ukraine, and his pressure on European allies to handle their own defense suggests a willingness to let that theater go. Venezuela would then represent America claiming its pré carré while ceding others to rival powers.

The December 2025 National Security Strategy provides our best window into the administration’s thinking, which actually put the Western Hemisphere appears first among regional priorities — before Asia, before Europe, before the Middle East. The strategy explicitly calls for “a readjustment of our global military presence to address urgent threats in our Hemisphere... and away from theaters whose relative import to American national security has declined in recent decades.”

Here the language on the “Trump Corollary” is unambiguous: “We will deny non-Hemispheric competitors the ability to position forces or other threatening capabilities, or to own or control strategically vital assets, in our Hemisphere.” This is the Roosevelt Corollary with updated targets — swap European colonial powers for Chinese port operators and Russian military advisors.

An alternative perspective is what analyst Tanner Greer calls the “prioritizer” framework, under which China is a unique peer competitor and threat to the US, requiring America’s full attention and focus. Under this logic, the US saves its power for countering China, avoids lasting entanglements in Europe and the Middle East, but does use decisive force for discrete, limited objectives.

It’s not clear what, if any, logic Trump is following. Perhaps Trump’s approach is neither cleanly pré carré or “prioritizer” but something more opportunistic. The strategy asserts hemispheric dominance aggressively while adopting a more transactional posture toward great power competition. The statement that “the days of the United States propping up the entire world order like Atlas are over” is telling. This isn’t sphere-of-influence realism so much as selective assertiveness, with maximum force where costs are low and benefits are tangible, across targets as wide ranging as Venezuela, Iran, and even Nigeria — but explicit restraint or burden-shifting where they’re not, like Ukraine. Where Taiwan falls on this logic is not immediately clear — strategic ambiguity at its finest.

What happens next?

But of course, the harder question is what comes for Venezuela after the helicopters leave. While the execution was successful, there doesn’t seem to be any clear plan for what comes next.

The Noriega parallels are striking — both leaders faced drug trafficking indictments, both stole elections, and the US recognized the opposition candidate as legitimate. But the differences matter more. In Panama, the US already had 12,000+ troops stationed in the Canal Zone — roughly the size of the entire Panamanian military — and a government-in-waiting ready to assume power. Venezuela is 12 times larger with over 100,000 troops equipped with Russian weapons. There is no equivalent infrastructure for transition.

Additionally, Noriega was the beginning and end of the Noriega dictatorship. But the Maduro dictatorship is more embedded — Maduro was never just about Maduro. There instead remains is a collective dictatorship — security services, patronage networks, illicit finance pipelines, and a political-military elite that shares criminal liability with Maduro and faces their own US indictments. Interior Minister Diosdado Cabello, who appeared on television today saying the Venezuelan government would not be cowed, carries a $25 million bounty of his own. Defense Minister Vladimir Padrino Lopez has run Venezuela’s military longer than anyone in modern history and remains in place.

The snake has been decapitated, but the body is still moving. The existing post-Maduro Venezuelan regime might abide by what Trump wants, but they remain staffed by people who face their own indictments and have every incentive to resist a democratic transition that would hand them to American courts.

Venezuelan reaction has been deeply polarized. In Doral, Florida — home to the largest Venezuelan diaspora community in the US — crowds gathered outside El Arepazo restaurant chanting “Libertad!” and singing both national anthems. Similar celebrations erupted in Santiago, Lima, and Madrid, where one Venezuelan told CNN: “At first we were crying because our country was being bombed, but when we were told they had Maduro, the reaction was overwhelming.”

Inside Venezuela, the picture is grimmer. Caracas residents described waking up terrified. “How do I feel? Scared, like everyone,” one resident told the news. “Venezuelans woke up scared, many families couldn’t sleep.” Pro-Maduro supporters led by Caracas Mayor Carmen Meléndez marched through the capital demanding his return, chanting “Maduro, hold on, the people are rising up!” The streets were otherwise largely empty, patrolled by security forces, as residents stayed indoors absorbing what had happened.

When it comes to figuring out who should govern Venezuela next, opposition leader Machado issued a statement calling for González to assume the presidency. But Trump said it would be “very tough” and that Machado “doesn’t have the support or the respect within the country.” His team had not been in touch with her or any member of the Maduro opposition.

And there’s also a constitutional wrinkle. Executive Vice President Delcy Rodríguez is technically Maduro’s legal successor — and she’s reportedly in Moscow. The US wants to work with Rodríguez over Machado and González, despite Rodríguez being a committed Chavista who appeared on state TV demanding Maduro’s release as “the only president of Venezuela.” And this dismissal of Machado is striking and underexplained. Recall that Machado won the opposition primary decisively.

Why? Perhaps Trump cares more about stability and oil access than democratic transition. Rodríguez represents continuity with the existing power structure whereas Machado represents upheaval. Or perhaps portions of the Maduro regime negotiated this outcome and accepting Rodríguez was the price of Venezuelan non-resistance. Or perhaps the Trump administration has no plan and is improvising on the fly. Perhaps Trump wanted the win, got the win, and figured they’d sort out the details later. Note that none of these three theories are mutually exclusive.

Trump’s stated plan is for the United States to “run” Venezuela “until such time as we can do a safe, proper and judicious transition,” with American oil companies moving in to extract wealth that would flow to both Venezuela and “the United States in the form of reimbursement.” But that’s barely a plan. Run it how? With what personnel? Under what legal authority? “Running” a country of 28 million people without boots on the ground isn’t a plan. Capturing Maduro doesn’t automatically produce Venezuelan democracy.

And Democrats are furious. The Gang of Eight, a bipartisan group of Congressional leaders, are normally briefed in advance due to requirements by law to maintain balance between executive war powers and congressional oversight. But they weren’t briefed this time. Trump’s justification—that “Congress has a tendency to leak”—is essentially a middle finger to Article I, which normally puts Congress in the drivers seat when it comes to war. Article II does give the US President authority to do military operations to protect US personnel from an actual or imminent attack, but the imminent attack framing is doing a lot of work here for an operation targeting a leader who posed no immediate threat to American forces.

Looking forward

Across Venezuela, Fordow, and Soleimani, a pattern emerges — the willingness to use overwhelming military power for specific objectives, combined with an unwillingness to own the consequences indefinitely.

But the most important part is that there was no 100,000+ troop occupation, no nation-building doctrine, no decade-long-plus commitment. There was just a precise operation with a specific target, executed and completed within hours. After American foreign policy has been haunted by Iraq for two decades, Trump shows a model of limited military action that accomplishes discrete foreign policy objectives without the quagmire of full-scale invasion.

Sometimes these gambits work. The Abraham Accords, a series of US-brokered agreements signed in 2020 that normalized diplomatic relations between Israel and several Arab nations — including the UAE, Bahrain, Morocco, and Sudan — were dismissed as superficial PR but turned out to be genuinely durable.

Whether that bet pays off in Venezuela will tell us a great deal about how the next three years unfold. If the transition succeeds, expect this template to be applied elsewhere — perhaps Cuba, perhaps Nicaragua, perhaps targets we haven’t anticipated. If it fails, we’ll learn something important about the limits of precision force in an era of diffuse power.

I’m genuinely uncertain which way this goes. It’s good that Maduro is out, but I’m worried about whether democracy is what comes next.

-

Thanks to Caroline Jeanmaire for help with editing and contributing analysis.

This template was made by Peter Wildeford and Caroline Jeanmaire.

It’s almost a new year and that often calls for some sort of annual planning.

But in my opinion, annual planning operates on too long a timescale. Annual planning and goal setting is too infrequent, so it’s easy to lose track. You don’t get fast enough feedback and you risk derailing.

Instead, I suggest adopting a quarterly planning cadence (e.g., set goals for 2026 Jan-Mar). I wanted to provide a template that I’ve used many times before and that many people seem to like.

Here’s my template in a Google Doc => click here and make a copy.

Share

How is this different from other templates out there?

There are a lot of different templates out there to try and I don’t know if this one is better. It works well for me, but I’m not you. But here’s some principles I was interested in, and maybe that resonates with you:

• Optimized for something quarterly – rather than something monthly (too frequent, too time consuming) or something annual (too infrequent, easy to lose track, not fast enough feedback).

• Forces you into relentless focus on the most important things (three major rocks and two optional minor pebbles) and encourages you (via anti-rocks) to not do too much.

• Designed to be very achievable. The template is somewhat minimal and much of it focuses on building systems and making sure your goals will happen.

• Designed to be very customizable. You can make the template your own!

What’s actually in this thing?

The template walks you through five steps. The fast-track version takes about three hours (seven 25-minute pomodoros), though you can spend longer if you want to go deeper.

Step 1: Reflection. You start by looking back at the prior quarter—key successes, key mistakes, how you did against prior rocks. There’s also a quick 1-5 rating across eight areas: work, sleep, exercise, nutrition, routines, finances, relationships, and environment. The point isn’t to obsess over numbers but to notice patterns and identify 1-2 problem areas that deserve focus.

Step 2: Vision. A quick check on your values and mission (or just jot keywords if you don’t have one), plus a 90-day visualization. What do you want to have/be/feel in three months?

Step 3: Combine and Refine. Take everything from Steps 1-2 and distill it into a raw list of potential priorities, then filter down to 3-5 candidate themes. The template also asks you to identify the “effort type”—is each item a project, a learning goal, or a habit change? This matters because habit changes require different strategies.

Step 4: Set Rocks and Pebbles. The core of the template. You get three major rocks and two optional pebbles. For each rock, you define SMART goals (both input goals for what you’ll put in and output goals for what you’ll achieve), plus IF-THEN planning. You identify one key obstacle and write out “IF [obstacle occurs], THEN I will [specific response].” Each rock also asks: “Who could you share this rock with for support/accountability?” External commitment helps.

Step 5: Sanity Checking. Reality-check your plan: Is there enough time? Do you have slack for the unexpected? What’s your system for weekly/monthly review? Will travel or environment changes derail you?

The template includes a 7-pomodoro fast-track guide if you want structure for completing it in a single ~3 hour session.

Tips Before You Start

• You really shouldn’t try to do too much. This is the most common mistake. Three rocks. Maybe two pebbles. That’s it. If you find yourself wanting more, that’s a sign you need to prioritize harder.

• Habits take a lot of time to build. Be very careful about adopting a new quarterly review, new monthly review, new giant list of 20 habits, new morning + evening routine, etc. all in one go. Ideally build these gradually over 1-2 years. If you’re new to this kind of planning, make “establish a quarterly review habit” one of your rocks.

• Do this somewhere other than your usual workspace. It helps you think outside the box.

• Don’t do it all in one sitting. Important thoughts and connections emerge in between sessions.

• Make it your own. I went through a bunch of materials and distilled it down to one template that works well for me. The true power comes from customizing it to meet your own needs. Want to track books? Add that. Want fewer questions? Remove stuff. Appendix B has a bank of additional reflection questions if you want more.

~

If you’re interested, make a copy of the template (click here) and get to work!

This template was made by Peter Wildeford and Caroline Jeanmaire. It is based on Gustin’s annual review, the Quarterly Productivity Planner, Ben Todd’s Personal annual review process (2020), Ultraworking’s monthly template, the Beyond Goals workshop, the Forcing Function annual review, Ben Kuhn’s weekly review habit, Alex Beal’s “2022 OKRs”, Eric Barker’s “What’s the best way to set a goal?”, Konrad Seifert’s “Life Review, Planning & Organization”, and Jan’s “Review, reflection, goal-setting prompts”. Intend.do and Zen to Done also serve as inspiration.

This is a guest post written by Oscar Delaney, reposted from his Substack (see here and here for our previous coauthored posts).

~

James Fearon’s classic1 1995 paper “Rationalist Explanations for War” argues that there are two main reasons rational states fight: private information about their own capabilities and resolve, with the incentive to misrepresent this, and commitment problems when trying to reach a negotiated agreement.2 I claim that both of these, especially the latter, contribute to a significant risk of pre-emptive war in the lead-up to one state developing ASI.

On a basic rational actor model, war seems puzzling. It causes large deadweight losses to belligerents, and therefore both sides would be better off reaching a negotiated agreement to split the issues at stake roughly proportionally to the military strength of each side. That is, if Strongland has an 80% chance of beating Weakville in a (possibly protracted) war, both would be better off avoiding the war and Weakville just giving Strongland a payment (e.g. land or money) of 80% of the expected spoils of war. This simple model breaks down in the case of either private information or commitment problems.

Share

Private information

In the example above, the would-be warring states may have different estimates of their chances of victory, and therefore fail to agree on terms for a negotiated agreement. For instance, each party will have private information about their own ability (number of tanks, jets, etc) and desire (national cohesion, psychological profiles, etc) to fight. And each side is incentivised to misrepresent these variables. For instance, the USSR purportedly had the same fighter planes fly overhead in military parades, loop around over the horizon, and fly over again, to make adversaries think they had a larger stock of planes than they really did.

AI development contains significant private information. It is relatively doable to tell via satellite imagery what materiel an adversary has. Similarly, data center capacity can be estimated from satellite and OSINT data. However, it is far harder to ascertain from afar what algorithms are being run on a given data center, and what AI capabilities an adversary has. This will particularly be the case once AI developers start keeping their most advanced models internal for longer. That said, as long as you can keep some spies in the leading AI project of your adversary, you will have decent knowledge of their AI capabilities and plans. Overall, greater uncertainty about an adversary’s capabilities seems to increase the risk of war by making it harder to negotiate an agreement based on common knowledge of relative strength.

Commitment problems

Another reason a negotiated settlement to avoid or end a war may fail is that the belligerents cannot credibly commit to upholding the agreement. For instance, one model of the Russia-Ukraine war is that Ukraine should be willing to accept some territorial losses now in exchange for a ceasefire. But if Ukraine does so without gaining security guarantees, it can’t ensure that in a few years’ time, Russia won’t annex a bit more territory, and repeat this process indefinitely. For a negotiated agreement to work, there must be a large cost to breaking the agreement: an honesty system won’t cut it. The main way large costs can be imposed is by the international community. If a country breaks a treaty, that country will likely be ostracised and have worse trade and military cooperation prospects in the future. This provides a slight check on the anarchy of the international order.

Advanced AI poses particularly severe commitment problems, because the country that first develops ASI may also gain a decisive strategic advantage (DSA). For instance, ASI could lead to that country outgrowing the rest of the world, inventing and deploying wonder weapons, or undermining nuclear second-strike capabilities. I think that a country unilaterally developing ASI is >50% likely to thereafter get a DSA. If a country has a DSA, it will by definition be impossible for the international community to meaningfully constrain its actions, including if that country chooses to renege on treaties and agreements made before the advent of its DSA.

This is a huge deal. It means that other countries will be very reluctant to risk an adversary reaching ASI unchallenged, as this would mean placing their continued sovereignty in the hands of the adversary’s goodwill. This means there could be a strong rationalist (self-interested) case for a pre-emptive war to prevent an adversary reaching ASI. This is essentially the argument Hendrycks, Schmidt, and Wang make in Superintelligence Strategy. Crucially, it is not the other country reaching ASI that is itself unacceptable. If they could somehow guarantee not to use their ASI to get a DSA, or not use their DSA to trample other countries, but rather share ASI-driven economic superabundance with the rest of the world, this would be a very attractive prospect compared to risking a nuclear war. But without credible commitments, trust doesn’t go very far.

Conclusion

Overall, this seems rather pessimistic, and makes me think AI-caused war is more likely than I previously thought. It also points towards two important interventions:

1. Transparency: All else equal, there being more common knowledge about frontier AI capabilities probably reduces the chance of war. But this isn’t obvious, and there are also backfire risks where a country finding out more about the AI development of an adversary could make them more worried and increase the risk of war. More research is needed, dare I say?

2. Commitment mechanisms: Creating plausible pathways by which countries could credibly commit to not misusing an ASI seems like the single biggest way to reduce the risk of AI-caused war. But it is also very difficult. Oliver Guest and I discuss some possible solutions here, and in the future I am particularly interested in investigating:

   1. Lie-detection: If AI leads to large advances in lie-detection, this could create far more confidence that leaders genuinely intend to follow the treaties they sign, as subterfuge would be flagged by the lie-detection system.

   2. Constitutional AI: If the ASI itself is unwilling to help its host country disempower other nations, this would be a strong antidote to a DSA.

I expect to be working more on these and related ideas in the coming months, and would be keen to discuss possible projects with anyone interested!

~

If you liked this post, consider subscribing to Oscar’s Substack “AGI Strategy” where he will be writing more about AI, AGI, and geopolitics.

Subscribe to Oscar

This article was written by Oscar Delaney, Bill Anderson-Samways, and Peter Wildeford. It does not necessarily represent the opinion of the entire staff at the Institute for AI Policy and Strategy.

~

The idea of a government-led program to “win the race” with China to Artificial General Intelligence (AGI) has gone mainstream. From Congressional commissions to government agencies to AI company CEOs, high profile calls for a Manhattan Project-style effort are growing. Leopold Aschenbrenner’s popular essay Situational Awareness predicted “some form of government AGI project” by 2027-2028. Meanwhile, others remain skeptical.

How likely is this really? What would it actually look like? And most importantly, is it even a good idea? Our new forecasting research suggests the answers are far from certain. Professional forecasters estimate just a 34% probability of a government-led AGI program — neither inevitable nor impossible. More importantly, treating a government project as inevitable could trigger the very risks we’re trying to avoid.

Share

Forecasters Say a US-Led AGI Project Is Far From Inevitable

We at the Institute for AI Policy and Strategy ran a structured forecasting workshop with professional forecasters and US AI policy experts to estimate the probability of a US government-led project building the first AGI-level system.

We defined “AIR-10” as an AI system that accelerates AI R&D progress tenfold— essentially compressing a decade of AI development into a year. This matches what many consider the threshold for transformative AI that could trigger recursive self-improvement, and is the threshold Aschenbrenner uses for AGI.

The aim was to forecast the likelihood that AIR-10 is developed via a “government-led project.” By this, we meant that the government both decides when to start/stop developing the AI model AND acquires the final model to deploy as it sees fit. This is more stringent than just the government doing regulation or oversight. In this definition, the government takes direct leadership and command over key development and deployment decisions. However, the private sector may still be heavily involved, for example in a government-led public-private partnership.

Some commentators (e.g. Aschenbrenner) have suggested that a project run by the US government to achieve AGI is inevitable as a result of the geopolitical importance of AGI. Our results might surprise those who see it as inevitable — the median forecast was a 34% chance.

Furthermore, there was a lot of uncertainty among forecasters in our panel, with an average range of 11% to 61%. This wide spread shows that even experts who study this closely are deeply uncertain. Anyone claiming to know for sure whether the government will or won’t lead AGI development is likely overconfident. We need to prepare for multiple scenarios, not just assume one particular future.

History Offers a Messy Guide, Not Clear Predictions

Those predicting a government AGI project often invoke historical analogies — the Manhattan Project, the Apollo Program, and even ARPANET. But when we systematically analyzed 35 past US technological innovations, the picture that emerged was more nuanced.

What’s the correct historical comparison? Whether the government leads depends heavily on what kind of technology you think AGI is:

• Megaprojects costing >0.1% of GDP (like the Interstate Highway System)… these projects were government-led 78% of the time in the past

• Ambitious STEM projects (like the atomic bomb)… Government-led 63% of the time

• Dual-use technologies with both beneficial and harmful applications (like synthetic virology)… Government-led 57% of the time

• General-purpose technologies (like the airplane)… Government-led only 40% of the time

• Past AI breakthroughs (like the transformer)… Government-led only 23% of the time

AGI has features of all these categories, which is why the historical precedent is hard to pin down. AGI is a general-purpose technology with massive economic potential, in a field that has to date been private-led thus suggesting a high likelihood of continued private leadership. AGI is also a dual-use technology with profound national security and geopolitical implications, and potentially requires megaproject-scale resources beyond what private companies can muster, all factors pointing to government involvement.

We presented our forecasters with this data during the workshop described above. As a result, they updated their forecasts a little, but not much — probably reflecting the significant ambiguity in the historical precedents.

National Security and the China Factor

The single most powerful factor in our forecasters’ estimation that could compel government action is a perceived military-technological threat from China. If the US believes it’s on the verge of losing its strategic advantage, political will for a government-led project could materialize overnight.

Nascent versions of this fear already exist. The US-China Economic and Security Review Commission has already called for a Manhattan Project for AGI in its 2024 report to Congress. The narrative of an AI arms race with China has become deeply embedded in Washington.

Importantly, national security concerns both make government involvement more likely and shape what form that involvement might take. An AGI project driven by military imperatives would look very different from one focused on economic competitiveness or scientific advancement.

AI’s Breakneck Pace vs Government’s Glacial Speed

However, the US government is not historically known as being fast to develop technology — especially compared to the speed of AI. By the time the government decides to become more involved in AGI development, private AI companies might have already developed it.

This creates a timing problem. The AIR-10 threshold we defined, while transformative, might not be “attention-grabbing” enough to trigger government action to achieve it. This is unlike nuclear weapons, where the ability to clearly and unambiguously destroy an entire city constituted a significant, discrete, and noticeable threshold that was salient to policymakers even before it was achieved.

Several of our forecasters emphasized this dynamic — if AIR-10 arrives in the next few years (as some predict), that might be too soon for the US government to organize a successful intervention. Only if the path to transformative AI stretches out over a longer timeline does government leadership become more likely. When providing their central 34% estimate, forecasters were asked to condition on AIR-10 arriving at some point before 2035.

Less ‘Manhattan Project’, more ‘Apollo Program’ / ‘Operation Warp Speed’

Additionally, the “AI Manhattan Project” framing is potentially misleading. When people use this term, they often imagine a single, secret government AI lab that builds AGI entirely from scratch. However, according to our forecasters, it’s only about 3% likely that AIR-10 will be developed specifically in this way. Other options involving major government in-house resources, nationalization of an existing private company, or otherwise using legal compulsion to force the development of AGI under US government control were also considered similarly unlikely.

Based on our analysis, if the government does lead AGI development, we forecast the project would most likely take one of these forms:

• Government-Led Consortium (14% probability): This is the “Apollo Program” model. In this model, the US government wouldn’t build AGI itself but would coordinate multiple private companies (and maybe even government labs too, though relying mainly on government labs seems unlikely). For example, NASA managed contractors like Boeing and North American Aviation to do the moon landing. This avoids “picking winners” and leverages existing private sector talent.

• Single Private Contractor (9% probability): This is the “ENIAC” model. In this model, the government contracts with one specific company to build AGI to government specifications. This is faster to implement but somewhat riskier, as it involves picking a particular company to be the national champion.

The original Manhattan Project itself was actually closer to a government-led consortium than is widely thought, with multiple government labs and private contractors involved—though it was much more government-centric than what our forecasters envision for AI. The variety of possible models matters because each comes with different tradeoffs in terms of speed, security, and innovation.

From Commercial Competition to an AI Arms Race

The consequences of getting a government-led AGI project wrong would be severe. A poorly designed government project could trigger the very catastrophes it aims to prevent.

Currently, AI development is a commercial competition. While intense, it allows for some safety investment and shared research. A formal, government-led US project would upend this dynamic, sending an unmistakable signal to the world — especially China — that America is seeking a decisive strategic advantage.

The response would be swift and predictable. A rival government may feel forced to launch its own centralized, government-led program, transforming a commercial race into a direct military-technological showdown. History supports this concern — the Manhattan Project triggered a nuclear arms race that saw the Soviet Union detonate its own test bomb just four years later.

Unlike with the Manhattan Project, however, the key stage of an AI race could unfold in mere months, with even less room for safety compromises under the pressures of national security. That could heighten the risk of a catastrophic accident arising from AI systems due to hasty deployment.

This dynamic could be mitigated if multiple companies join the project, which would at least nullify the commercial race. However, according to our forecasters, this is unlikely to happen — they think there is less than a 50% chance that a government-led project would involve more than AI company, primarily due to the coordination costs involved in such programs.

In some circumstances, the AI arms race caused by a government-led AGI project could even trigger a great power war. As noted above, a US project could be perceived as an attempt to achieve a decisive strategic advantage unparalleled since the advent of nuclear weapons. In such circumstances, a rival might also consider escalatory actions, such as a cyberattack on data centers or even threats of missile strikes, which could then escalate into an all-out war.

Concentrated AI Power Threatens Democratic Legitimacy

While the risk of international conflict is terrifying, a centralized AGI project also threatens the erosion of democratic legitimacy.

The American system is built on Jeffersonian checks and balances, pitting ambition against ambition to prevent any one person or group from accumulating concentrated power. A secretive AGI project, led by executive branch agencies, would greatly concentrate the unprecedented power that AGI would bring. Even if this did not pose a direct threat to democracy, it would certainly erode democratic trust and legitimacy.

Some might argue that, without a government-backed AGI program, rival governments are more likely to beat the US to AGI. This would concentrate power in adversaries’ own, far less democratic political systems. However, this argument relies on the inaccurate idea that adversaries already possess centralized AGI programs which could allow them to overcome US companies’ current lead. Though the current pace of Chinese AI development is certainly concerning, open-source information indicates that China has not centralized its compute or researchers into such a program. Thus, US companies retain a considerable advantage. In fact, a US government-led program could trigger rival governments to launch their own centralized projects, paradoxically potentially reducing US lead-time.

Concentrated government power is not the only concern. A government-led AGI program could also concentrate power in the hands of tech companies. If such a program involved multiple AI companies, it would essentially eliminate market competition — creating a cartel with access to enormous amounts of economic and military power. Such a situation has not occurred since the apex of the British East India Company, which in the 1700s came to account for half the world’s trade and rivaled the military might of the great powers. Even if only a single company was involved in the project and market competition was preserved, giving a private company access to the government resources needed to develop decisive military capabilities would be unprecedented.

The Security Benefits Don’t Require Building AGI

The strongest argument for a government project is compelling — namely, better security. Private AI companies are vulnerable to infiltration and theft, especially from well-resourced governments. A government project could implement military-grade security, reducing risks of AI being stolen by adversaries.

This is a real concern that deserves serious attention. But you don’t need to build AGI to secure it. Instead, security benefits can be achieved through more targeted, less escalatory policies such as:

• Mandatory security standards for private AI companies handling very advanced AI systems

• Government partnerships to assist companies with securing model weights without taking over development

• Enhanced counterintelligence support for private AI companies

The government doesn’t need to build all nuclear reactors to ensure nuclear security. It sets standards, monitors compliance, and provides security assistance while letting private companies operate the reactors. These targeted interventions could achieve 80% of the security benefits with 20% of the risks. They avoid triggering international arms races or concentrating power dangerously. Most importantly, they can be implemented incrementally and adjusted as we learn more about AI risks.

Preparing for Multiple Futures, Not Sleepwalking into Disaster

The future is genuinely uncertain. Our research shows that a government-led AGI project is neither inevitable nor impossible.

This uncertainty tells us something important about how to approach AI governance. Rather than assuming a single trajectory and optimizing for it, we need robust strategies that work across multiple scenarios.

That means avoiding self-fulfilling prophecies. Treating a government project as inevitable could trigger the very international dynamics we’re trying to avoid. But dismissing it as impossible leaves us unprepared if geopolitical pressures suddenly shift.

We need a portfolio of approaches that work across different government involvement levels:

• Strengthen private AI security: Improve security protocols and infrastructure at the private AI labs, to prevent theft of American IP.

• Track adversaries’ AI progress: Track reliable indicators that would enable the US government to catch adversaries launching their own government-led programs, such as compute and researcher centralization.

• Build government readiness: Develop in-house expertise and plans in case government leadership does become necessary.

Most importantly, we must remember that government involvement isn’t binary. There’s a spectrum from light-touch regulation to full nationalization, with many points in between. The goal should be finding the minimum effective dose — enough government involvement to ensure safety and security, but not so much that we trigger the very catastrophes we’re trying to prevent.

The stakes are too high for ideological purity. Whether you’re a tech accelerationist who believes in private innovation or a national security hawk worried about China, we all share an interest in navigating this transition without triggering catastrophic accidents, conflict, or concentration of power. That requires taking uncertainty seriously and preparing for multiple futures, rather than assuming we know which one will unfold.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022. This article was written with the assistance and feedback from people with cybersecurity expertise and people with a background in the intelligence community.

~

Anthropic just announced that they have detected and disrupted what it describes as the first documented real-world large-scale agentic cyberattack campaign executed primarily by artificial intelligence.

This appears to be the first publicly known example of AI systems autonomously conducting multi-step cyberattacks in the wild.

A Chinese government-sponsored group jailbroke Claude, the AI made by Anthropic, by tricking Claude into believing it was conducting defensive cybersecurity work, and then used it to perform reconnaissance, identify vulnerabilities, and write exploit code. They targeted roughly 30 organizations, successfully breaching a handful including major tech companies, financial institutions, and government agencies. Most importantly, AI completed roughly 80-90% of the attack autonomously, with human operators stepping in only for about 4-6 key decision points per attack. While specific details in the report were limited, it was clear that the AI did multiple hours of work, with humans only needed for 1-2 hours per attack.

This use of AI to autonomously conduct offensive operations is a notable shift in the cyber threat landscape. Typically, offensive cyber operations require expensive, highly trained human operators that can only work on a limited number of operations simultaneously. A sophisticated espionage campaign targeting 30 organizations like what was reported by Anthropic would require a large team working for months. Now, it appears that some forms of cyberoffense can be run by just a few operators with access to AI. As AI continues to increase in sophistication, so will the quantity, speed, and sophistication of AI attacks.

So what does this mean and where do we go next? How concerned should we be?

Share

The anatomy of an agentic AI cyberattack

The industry term “agentic AI” describes systems that can work autonomously over extended periods, making tactical decisions within strategic parameters. In this attack, the Chinese hackers used Claude Code, an agentic version of Claude that can access external tools, maintain context across sessions, and iterate based on feedback.

Claude Code is a generally available computer application for anyone to use, typically used by software engineers to assist in writing software. The attackers repurposed these Claude Code capabilities for cyberoffense operations though meticulous additional tooling and orchestration. They broke down their attack into much smaller steps for a fleet of Claude agents to run attacks step-by-step to avoid triggering Anthropic’s built-in defenses.

The term ‘cyberattack’ often evokes ideas of great digital destruction, but it’s a broad category. Many operations, including this one, are focused primarily on espionage. Here, causing detectable damage would be counterproductive, since it would alert the target to the existence of the campaign. Countries continue to invest in these espionage programs tremendously.

In this espionage cyberattack, human Chinese government-sponsored hackers first collected a target list and then launched an AI-enabled framework to spawn parallel AI-driven reconnaissance against multiple targets simultaneously. Think of it as the human hackers each managing a team of tireless digital minions — the humans give them targets and approve their proposed escalations, but the AI minions handle the grinding work of testing thousands of login attempts or parsing gigabytes of stolen documents.

Human operators reviewed the findings of the Claude agents at key checkpoints and approved progression to active exploitation. After human authorization, the Claude agents autonomously deployed exploits to establish access. Claude agents then systematically harvested credentials, tested stolen credentials across discovered systems, and executed lateral movement through internal networks. During data collection operations, Claude agents queried databases and internal systems, extracted information, parsed results to identify intelligence value, and categorized findings by sensitivity. Each Claude agent maintained persistent operational context across sessions spanning multiple days.

Anthropic reported that the Claude agents executed approximately 80 to 90 percent of tactical operations independently, without needing human oversight, guidance, or correction. Throughout the campaign, humans shifted from conducting individual attack steps to setting strategic direction and approving escalations at critical decision points. The human role shifted from “hacker” to “strategic supervisor approving escalation points.”

When evaluating the value of AI in this attack, it’s important to recognize that each step in the attack chain is not equally valuable or equally difficult to automate. Some of these steps represent larger bottlenecks than others. And crucially, the operation still required skilled human hackers for strategic planning and key inputs — AI could not do the attack fully end-to-end.

For example, autonomously discovering internal services and conducting network mapping is not impressive as existing tools can already do this automatically without AI. The automatic testing of authentication is more significant as existing tools for this are less good. The most valuable part was likely the use of AI to parse large volumes of stolen information to automatically identify intelligence value and categorize findings, since this greatly reduces the human intelligence effort involved in the attack, improving the attack’s cost-effectiveness.

Where are AI cyber capabilities heading?

In terms of operational sophistication, Anthropic reports that Claude autonomously conducted reconnaissance and exploitation operations that took 1-4 hours each per step. Reliability was low but non-trivial, succeeding “in a handful of cases”. Overall, this is a notable increase in sophistication from AI-enabled hacking that was reported just a few months ago, which involved significantly more human operation. And hackers only need to succeed a handful of times to be valuable.

METR, an independent AI evaluation company, has been studying the rate of progress for AI and finds AI increasing at a rapid and predictable rate. They find that current models like GPT-5 and Claude 4.5 Sonnet can operate complex software tasks of around two hours with 50% reliability, closely matching Anthropic’s findings about this real world autonomous cyberoffense case.

This is a concern given that this trend will likely continue. METR’s data shows AI task capabilities have been doubling roughly every 118-212 days since 2024. If this pace continues, models could reach 4-hour task lengths in early 2026. This points towards future AI autonomously enabling more sophisticated multi-step attacks, better ability to adapt to defenses, and reduced human oversight requirements.

Innovation in AI scaffolding also matters enormously. Scaffolding is what gives your fleet of AI spies the tools needed to conduct attacks, such as password cracking software. Scaffolding includes techniques for enhancing AI performance through prompting strategies, tool integration, and multi-step reasoning.

OpenAI’s GPT-5 initially showed only modest cyber capabilities in OpenAI’s isolated testing, but the XBOW company demonstrated that GPT-5 could achieve 2x performance improvements when integrated into their specialized scaffolding. As both underlying models and scaffolding techniques improve simultaneously, offensive capabilities can increase faster than otherwise expected.

The biggest bottleneck to this attack is likely the attackers needing to painstakingly maintain three-way communication channels between themselves, the victim’s infrastructure, and the Claude API. Here, it’s telling that Chinese state actors used Claude rather than domestic Chinese alternatives like Qwen, Kimi, or DeepSeek. This strongly suggests Chinese AI labs haven’t yet produced models with comparable autonomous hacking capabilities, a gap that creates both temporary strategic advantages and policy opportunities.

The temporary advantage is monitoring and disruption. When Chinese operators depend on Western AI infrastructure, their operations require maintaining API connections that can be detected and cut off, as Anthropic demonstrated. This dependency creates chokepoints that wouldn’t exist if they were running Qwen locally on their own infrastructure.

But this advantage has an expiration date, probably measured in months rather than years. Chinese labs are potentially closing capability gaps, and recent performance suggests they’re 6-12 months behind frontier Western models at most. Once a sufficiently capable Chinese model emerges, attackers could download it, deploy it directly on compromised infrastructure, and operate completely autonomously without fear of the API connections being severed.

What does this mean?

This attack largely represents improved scalability of basic cyberoffensive operations — exploiting bad security at scale more than overcoming really good security. But even if each step is “basic” or doesn’t involve novel capabilities, putting it all together in this way is still a big deal. Most systems today are cyber-insecure against sophisticated actors and even more secure systems are relatively weaker if you can attempt an order of magnitude more variations in your cyber penetration.

One big change is that automated cyberattacks are way more scalable. Currently, cyberattacks from nation states are limited by human operators. But AI agents can handle tactical work continuously and autonomously, allowing threat actors to maintain significantly increased attack tempo across multiple targets.

Additionally, AI is fundamentally changing how much value is extracted from the target. In this attack, Claude agents were automatically categorizing information as it was collected, making value extraction much more efficient than before. This all improves the cost-effectiveness of attacks and significantly increases the amount of potential simultaneous operations.

Reducing the skill level needed to implement attacks also makes cyberattacks more readily available to less skilled actors. Cyberattacks currently require skilled teams to implement. Progress toward automating cyberoffense potentially enables both nation-states and less sophisticated actors to conduct operations at speeds and scales previously impossible. If the cost and level of sophistication needed goes down, we will likely see more attacks by non-state actors, such as criminal gangs or even disgruntled individuals.

Also, cyberattacks now will occur at increased speed – operating at the speed of an AI rather than a human. This makes defending harder. This improves how nimble an offensive operation can be, how many targets can be attacked at once and how quickly they can be targeted.

Can defensive AI close the gap?

Anthropic’s response to concerns about this attack is that the same AI capabilities that enable attacks also empower defenders. If Claude can autonomously conduct reconnaissance and exploitation, why can’t defenders use similar AI systems to patch vulnerabilities and monitor for threats?

This argument has real merit. Defensive AI is already showing results. AI-assisted threat hunting systems catch anomalies humans miss, automated vulnerability scanning identifies exposures before attackers find them, and AI-powered patch testing accelerates deployment cycles. The challenge isn’t whether defensive AI works but whether organizations will adopt it fast enough.

In the long run, defense probably has fundamental advantages. Defensive systems can be purpose-built and deeply integrated into infrastructure. They benefit from economies of scale—one defensive AI system can protect thousands of organizations simultaneously. Eventually, defensive AI might even be able to scalably write provably secure software in a way that we are far from being able to do now. But acknowledging long-run defense advantages doesn’t mean we’re safe now. We’re entering a vulnerable transition period where the offense-defense balance tips sharply toward offense.

A key problem is that defenders face fundamental adoption barriers that attackers don’t. Existing organizations don’t even adopt current cybersecurity guidance, let alone modern AI-enabled cyberdefenses. Organizations can’t simply swap out their entire technology stack overnight. Critical infrastructure often runs on decades-old systems that can’t easily integrate new defensive AI. But attackers can adopt new offensive AI capabilities immediately.

And implementing defensive AI is hard. Organizations deploying AI for defensive purposes must be cautious about introducing new vulnerabilities. An AI system with access to internal networks and security infrastructure represents a massive attack surface if compromised. Defenders can’t afford to move fast and break things.

The asymmetry in surface area compounds this problem. Defenders must secure every potential entry point. Attackers need to find just one that works. An AI system that can test thousands of attack vectors per hour doesn’t need to succeed on every attempt. But defensive AI systems must maintain perfect vigilance across all potential vulnerabilities simultaneously – a fundamentally harder problem. Worse, the economics are fundamentally lopsided. Deploying defenses across all of the possible endpoints is much more expensive than conducting a few attacks in specifically chosen weak points.

There’s also a critical reliability asymmetry. While Claude’s hacking success rate was low, it was still sufficient for the attackers – failures are cheap and even a single breach can yield value. But defenders cannot deploy AI systems with low reliability, as this might fail to defend correctly and moreover might accidentally disrupt their own business operations or critical infrastructure. This difference in risk tolerance means even unreliable AI is more potent for offense than defense.

This is all doubly true for critical infrastructure, where the priority is continuity of the service and thus operators cannot introduce potentially unreliable AI systems. Over 70% of water systems inspected by EPA since September 2023 were found to have embarassingly basic failures like using “password” as the password, having a single login shared by all employees, and having former employees continue to have system access. Government auditors have made 1,610 cybersecurity recommendations since 2010, with 567 still not implemented as of mid-2024.

A lot of critical infrastructure runs on equipment that’s decades old, wasn’t designed with security in mind, and is hard to update without shutting things down, which is not possible. And there aren’t enough cybersecurity experts to go around, and critical infrastructure operators often can’t compete with tech companies on salary. Even when federal agencies try to help, they don’t coordinate well with each other or with the industries they’re supposed to protect.

Attackers have no such constraints and have very different risk tolerance — if their AI agent crashes or gets detected, they simply try again with a different approach.

Looking forward

Yes, the immediate impact of this attack is limited. A handful of organizations suffered breaches. Anthropic implemented better detection capabilities and affected entities presumably strengthened security. Presumably, this won’t stop the next attack, but it will make it a bit harder. And the cyber threat landscape has seen sophisticated state-sponsored operations before.

But the most important thing is what the attack represents. The bottlenecks that previously limited cyber operations at scale just loosened considerably. The economics shifted to favor even broader targeting and even faster operations. The cost-effectiveness of launching cyberattacks has increased.

These changes compound. Lower barriers enable more actors. More actors means more operations. More operations means defenders face increased volume while attackers gain more opportunities to discover vulnerabilities and techniques. The feedback loops favor escalation. These factors combine to create a window where offense leads defense, making the next 12-18 months critical for defensive investment and deployment.

What matters now is pace. Offensive capabilities surged ahead, but defensive capabilities exist. Scaling them is hard, but not impossible. This attack succeeded despite Claude’s safeguards, proving jailbreaking remains possible, but it also got detected and disrupted, proving monitoring works.

How quickly organizations invest in defensive AI, how effectively policymakers create adoption incentives, and how well the security community shares intelligence will determine whether this vulnerable period lasts months or years.

~

Thanks to Caro Jeanmaire, Chris Covino, and multiple anonymous experts for feedback on this article. Any errors in the article are solely my own.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022.

Is AI a bubble? Goldman Sachs CEO David Solomon, Ray Dalio, Sequoia’s David Cahn, the International Monetary Fund, and the Bank of England are all saying so. Former Intel CEO Pat Gelsinger put it bluntly: “Are we in an AI bubble? Of course. Of course we are.” Altman repeated the word “bubble” three times in 15 seconds at a dinner with reporters.

But the twist is that most of these people are saying “yes, it’s a bubble” while simultaneously announcing they’re spending hundreds of billions more. Zuckerberg says a collapse is “definitely a possibility” but insists underinvesting is worse. OpenAI’s Sam Altman warns investors will get “very burnt” while still planning $850 billion in data center buildouts. Jeff Bezos says AI is in an industrial bubble while Amazon spends $100B/yr in AI R&D.

How do we make sense of this?

Share

OpenAI’s meteoric rise

Back in March 2023, OpenAI reported $200M in annualized revenue.1 Incredibly, this annualized revenue passed $1B just five months later, at the end of August 2023. It then continued upward quickly — passing $2B in December 2023, $3B in June 2024, $5B in December 2024, and $10B in June 2025, and now is at $13B as of August 2025.

This massive revenue growth has been fast, but is not unprecedented — EpochAI found it to be on a similar trajectory to Google in 2003-2006, Uber in 2015-2020, and Cheniere (a US liquified natural gas company) in 2016-2020:

This massive revenue growth has helped give OpenAI a $500B valuation, making OpenAI the most valuable private company, newly ahead of the $400B SpaceX. If OpenAI were a public company on the stock market with the same valuation, it would be the 18th largest, edging out Exxon Mobil and Netflix but not quite surpassing Mastercard.2

Big revenue meets even bigger spending

This is important because OpenAI has a lot of bills to pay for AI infrastructure, especially cloud computing, chips, and data centers… OpenAI has committed to a $300B cloud deal with Oracle that begins in 2027 and runs for 5 years, a $100B investment from NVIDIA that will be used entirely to buy NVIDIA chips, a $22.4B cloud deal with CoreWeave3, a “tens of billions” deal with Broadcom for custom AI chips, a $90-100B deal with AMD for chips …and OpenAI is still in on the $500B+ Stargate deal, with plenty of data centers to build too.

And OpenAI is just getting started. Bloomberg quotes OpenAI CEO Sam Altman as saying:

You should expect OpenAI to spend trillions of dollars on infrastructure in the not very distant future. And you should expect a bunch of economists to say, ‘This is so crazy, it’s so reckless, and whatever […] And we’ll just be like, ‘You know what? Let us do our thing. […]

I suspect we can design a very interesting new kind of financial instrument for finance and compute that the world has not yet figured it out.

As a result of all these investments, OpenAI is not currently close to profitable.

OpenAI lost $5 billion in 2024 on $9 billion in total spending. The Information reports that this will only increase — OpenAI is projected to lose $14B in 2025, $17B in 2026, $35B in 2027, and $45B in 2028… this makes for a stunning situation where OpenAI is projecting unprecedented revenue growth over the next five years but still not breaking a profit until 2030:

There are three factors that are unprecedented about OpenAI’s spending — each merit deeper inspection.

Firstly, OpenAI is racking up record losses on its infrastructure buildout. WeWork burned through $22B total before bankruptcy, but never came close to OpenAI’s projected $116B. Uber also famously had a “burn money fast to gain market share and then later figure out how to be profitable” strategy but their cumulative losses peaked around $23B. OpenAI is just operating at a whole different level of scale.

Secondly, OpenAI’s projected revenue growth is itself unprecedented. EpochAI finds that while OpenAI’s revenue has tripled annually between 2023 and 2025, only seven US companies have grown from $10 billion to $100 billion within a decade, and none in under seven years — while OpenAI plans to do that in three.

EpochAI visualizes OpenAI’s projected revenue growth, going from ~$10B in 2025 (year 0 on the graph for OpenAI) to ~$100B by 2028 (year 3 on the graph). Compared to other companies, this projection is very fast growth!

Thirdly, the deeply entangled financing. I covered earlier this month one of these circles, where NVIDIA invests in OpenAI, which spends the money on NVIDIA chips and Oracle compute, while Oracle buys more NVIDIA hardware to serve OpenAI. This meant the NVIDIA capital goes in a big circle from NVIDIA to OpenAI to Oracle back to NVIDIA.

Since then, there’s been similar deals with OpenAI and AMD and the circle has gotten bigger. Bloomberg did a good job of diagramming it:

So how does this all add up to a potential bubble? Due to all this circular investment, OpenAI’s future cash flow is now a sizable part of the valuations of all of Nvidia, Microsoft, AMD, and Broadcom. These companies are priced in large part assuming OpenAI’s success will continue to drive sustained demand. And these companies are a large part of the stock market.

This is combined with the risk that OpenAI misses their revenue goals, since OpenAI’s projection is unprecedented and untested. And if OpenAI misses their revenue goals, there could be a correction in the stock market. And because AI spending is now a large driver of broader US economic growth, this correction could generate a real recession. In short, a bubble popping.

Dot com and the art of the bubble

The most familiar bubble is the dot-com bubble in the late 1990s when there was a lot of exuberance about companies first selling on the internet. Many have attempted to draw a comparison to AI and the dot-com bubble. Pets.com failed spending $300 million in 268 days to sell each product at a loss. Webvan raised $800 million for online grocery delivery before operating a single profitable market.

The dot com boom powered 600% growth in the stock market between 1995 and 2000. But then interest rate increases killed the cheap-money fuel that was powering investment in these companies. This exposed the broader problems in the stock market. Everything crashed back down to 1995-levels, with a 78% drop.

The dot-com bubble shows that business models can be correct in principle but economically unsustainable at current technology maturity and cost structures. But a key difference between dot-com and AI is that the dot-com bubble was about companies who had bad unit economics — “we lose money on every sale, but make it up in volume.”

However, unlike dot-com companies, the AI companies have reasonable unit economics absent large investments in infrastructure and do have paths to revenue. OpenAI is demonstrating actual revenue growth and product-market fit that Pets.com and Webvan never had. The question isn’t whether customers will pay for AI capabilities — they demonstrably are — but whether revenue growth can match required infrastructure investment. If AI is a bubble and it pops, it’s likely due to different fundamentals than the dot-com bust.

And notably, the internet ended up ultimately transformative technology and many 90s internet companies did succeed, such as Amazon, Microsoft, and Apple. Even Pets.com and Webvan were eventually replaced by the successful Chewy and Instacart.

Infrastructure bubbles — or “What do British railways and AI have in common?”

Instead, if the AI bubble is a bubble, it’s more likely an infrastructure bubble.

Consider Britain’s Railway Mania of the 1840s. The Liverpool and Manchester Railway, opened in 1830, generated 10%+ annual returns and demonstrated railways could dramatically reduce transportation costs. This success triggered an explosive investment. Between 1844 and 1847, Parliament authorized over 8,000 miles of new rail construction.

Multiple companies laid parallel routes, each assuming they would capture market share. But a lot of the new routes were not profitable. When the crash came in 1847, thousands of investors lost fortunes. Yet the infrastructure remained valuable, powering Britain’s industrialization through the late 19th century. The technology thesis proved correct; the financial structure was catastrophic.

The telecommunications crash of 1997-2002 followed a similar pattern. The thesis was sound — explosive internet growth would require massive bandwidth capacity. Companies laid millions of miles of fiber optic cable, with industry capital expenditures reaching $600B from 1997 to 2001. But the simultaneous construction by competitors created catastrophic oversupply and a significant portion of the fiber was installed but unused. Though the fiber ultimately did end up seeing use over the next two decades, this was far too late for original investors.

Infrastructure bubbles follow a recognizable arc. A genuinely transformational technology emerges and early deployments generate spectacular returns, validating the concept. Capital floods in at scale as investors extrapolate from initial successes. Multiple competitors simultaneously build capacity, each assuming they’ll capture significant market share. When aggregate capacity vastly exceeds near-term demand, the surplus can’t generate the revenue needed to pay for itself, and the financial structures collapse. Companies fail, investors lose fortunes, and infrastructure sits idle. The technology often still ultimately proves transformative, just too late for original investors.

It’s not as weird as it looks

However, unlike railway track sitting idle for years, AI data centers are being utilized immediately upon completion. OpenAI’s Abilene facility began running workloads as soon as capacity came online. Lead times for advanced GPUs stretch months, and energy availability limits deployment more than capital. The constraint is currently supply, not demand — which is why companies are aiming to build as much as possible.

The telecom parallel has more validity. We do have multiple companies building competing infrastructure and we do have multiple AI companies training competing models. This creates risk of redundant capacity. However, AI infrastructure shows more flexibility than fiber optic cable. GPUs can run various workloads, data centers can host different services, and cloud capacity potentially retains value even if AI-specific demand disappoints.

Additionally, NVIDIA and OpenAI’s circular financing is unprecedented in scale, but not fundamentally unsound. It’s similar to how a car company might give you a loan to buy their car — in this case, the money is still circular, but provided you do pay back the loan, everything works just fine. Except in this case, it’s NVIDIA renting chips on a loan to OpenAI rather than a car company renting a car on a loan.

Also the accusations of circular financing apply primarily to OpenAI’s situation, which is a minority of the broader investment wave. Microsoft, Meta, Google, and Amazon have immense pre-existing cash flows to pay for their infrastructure buildout without taking on debt. NVIDIA is only doing this for their customers who don’t have the immediate cash to buy the chips. And NVIDIA also has significant revenue of their own to be able to absorb big hits from bets that don’t pan out.

The bigger issue instead is what happens if AI doesn’t pan out. This, rather than vendor financing, is what would drive a stock market correction or even a recession. NVIDIA’s $5T market cap assumes sustained AI infrastructure spending. Microsoft’s $4T market cap includes a large premium for AI-driven productivity gains. If OpenAI’s revenue trajectory flattens and infrastructure spending is cut because AI-driven productivity doesn’t pan out, this will cause all these expectations repriced downward. So the real question is whether AI capabilities will be there on the timelines needed to generate revenue.

The fundamentals of the technology are different

Companies like Google and Facebook have already demonstrated that a product that is modestly useful to billions of people can be sufficient to generate hundreds of billions of dollars in annual revenue. OpenAI has a similar level of user base with over one billion free users, and it seems plausible these users could be monetized in some way. This is the basic Facebookization of AI that seems underway already. AI is not like blockchain, crypto, NFTs, or the metaverse — there is already real value being delivered.

But this revenue and other investment can just be a prelude to the true bull case of AI — AGI that automates the entire economy. If AGI were made, the winner theoretically gets not just some billions in annual revenue, but the entire economy! And along the way to automating the entire economy, maybe AI automate smaller parts of the economy that still deliver economic returns?

Imagine if the way British Railway Mania worked was not just that additional tracks could produce additional economic value, but that some amount of track (actual amount unknown) would somehow allow the rail company to monetize a cure for cancer, successfully compete with basically every other company in the economy, and potentially even take over the entire world?

As weird as it sounds, an AI eventually automating the entire economy seems actually plausible, if current trends keep continuing and current lines keep going up. For one example, METR tracks how well AI is doing on software engineering tasks, an economically valuable activity. METR finds that models have dramatically increased in their capability, from only being able to do rudimentary toy problems a year ago to being able to stand-in for a non-trivial amount of work that actual software engineers actually do:

And Anthropic’s Claude Code is already generating over $500M by assisting in the development of software code. Trends for other tasks outside of software and math are not that fundamentally different, suggesting eventually AI will be able to assist with — and eventually automate — a bunch of other tasks as well.

So what will happen?

Unfortunately, forecasting is not the same as having a magic crystal ball and being a strong forecaster doesn’t give me magical insight into what the market will do. So honestly, I don’t know if AI is a bubble or not. Admittedly, OpenAI, NVIDIA, AMD, and other companies are engaging in a lot of weird financial arrangements. But there’s nothing wrong with these per se. I personally take the bubble possibility seriously.

But we need to think more clearly. And a lot of people want AI to fail and are just doing ideological pattern-matching to crypto/NFTs and declaring AI a pump-and-dump grift without evidence, and that’s not good analysis.

The key question is whether AI capabilities improve fast enough to generate economic returns before the debt comes due. OpenAI has demonstrated explosive revenue growth already and AI capabilities keep improving on clear trajectories. The underlying bet — that AI will be economically valuable — still looks fairly solid.

My assessment is that there’s roughly a 30% chance of a significant AI-driven market correction with at least a >20% drawdown in AI-heavy stocks, sometime within the next three years. This may or may not lead to a broader recession, that’s unfortunately beyond my ability to forecast.

The modal path (~55% probability) is OpenAI restructures deals and raises dilutive funding rounds, but capabilities keep improving and justify continued investment. Some commitments get renegotiated downward, or OpenAI IPOs at $300B instead of $500B. This reprices AI expectations but doesn’t crash the market.

So why are industry leaders calling AI a bubble while spending hundreds of billions on infrastructure? Because they’re not actually contradicting themselves. They’re acknowledging legitimate timing risk while betting the technology fundamentals are sound and that the upside is worth the risk.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022.

In 1941, the US1 embarked on the Manhattan Project — a massive scientific and infrastructure project that produced a working nuclear bomb in less than four years. They spent ~$2B in the process. With inflation, this would be about $37B in today’s money. It was one of the most expensive scientific projects of all time.

Today, there is another project at an even bigger scale, except it is happening in the private sector and it’s for building AGI.

And it’s an even larger gamble — a $400+ billion web of circular financing deals between OpenAI, NVIDIA, and Oracle that makes everyone’s valuations contingent on AGI arriving on schedule. This financial engineering has transformed 25% of the S&P 500 into a leveraged bet that AI scaling will continue unabated through 2030. The math only works if AGI arrives before the money runs out. The crazy thing is that this all might just actually work.

Two weeks ago, Oracle signed a $300 billion, five-year computing power deal with OpenAI. The contract was the largest cloud deal ever signed. The resulting bump in Oracle stock briefly made Oracle CEO Larry Ellison the world’s richest man. The agreement requires 4.5 gigawatts of electricity with the contract starting in 2027, roughly equal to what 4 million homes consume. This would mean OpenAI paying $60B per year starting in 2028, or almost two Manhattan Projects annually.

But Oracle quickly got outgunned by another deal. On Monday, NVIDIA announced a letter of intent to invest up to $100B in OpenAI. NVIDIA invests cash in exchange for non-voting shares and OpenAI has committed to using that cash to buy NVIDIA’s chips. So far, Nvidia has committed the first $10B. We don’t yet know how the remaining $90B will be spread across years or how many years the investment will be over …or if it will even actually happen. What we do know is that this is a massive commitment for computing power.

What does this mean for the future of AI? Let’s dig in.

Share

The Infinite Money Glitch

The structure of this financing warrants closer inspection, as it contains significant risks.

This is because there is another key deal that gives the OpenAI-Oracle deal and the OpenAI-NVIDIA deal more context — a third deal announced back in May where Oracle promises to spend $40B purchasing NVIDIA’s GB200 GPUs for an OpenAI data center in Abilene TX as part of the Stargate project. This is a 15-year lease agreement where Oracle purchases the chips and then leases the computing power to OpenAI.

This leads to what Semianalysis’s Dylan Patel calls the “Infinite Money Glitch”:

Here’s how it works:

1. NVIDIA invests capital in OpenAI, which OpenAI then uses to purchase NVIDIA hardware directly and purchase Oracle cloud compute.

2. Oracle also uses the revenue from the cloud compute deals to purchase NVIDIA hardware.

3. NVIDIA books all of this as additional revenue, despite the revenue being spurred on in significant part by its original investment.

4. This new revenue helps support NVIDIA’s valuation, which in turn makes its stock more valuable as currency for future investments.

5. The additional increase in stock price allows NVIDIA to afford to invest even more in the next revenue round-trip.

6. The same happens for Oracle and OpenAI too.

Critically, the same money moves around in just one circle, but all of a sudden everyone’s valuations go up. It’s a virtuous cycle — as long as the music keeps playing.

NVIDIA has already executed smaller versions of this playbook with CoreWeave, Crusoe, xAI, and Lambda Labs, but this is next level. $100 billion represents roughly 3% of NVIDIA’s current market cap, and much bigger than all of the CoreWeave, Crusoe, xAI, and Lambda Lab deals combined.

Here’s a more detailed diagram, courtesy of Kakashii. This diagram also shows Coreweave and the increasingly irrelevant Microsoft, showing the confusing web of financials where everyone is circularly funding everyone else:

And across all of this is SoftBank, which increased its stake in Nvidia to $3B and bought $170M worth of Oracle shares in early 2025. SoftBank has also invested over $10B in OpenAI across different investments, with up to $30B total committed. A leveraged investor at the core of the infinite money loop provides an even deeper point of vulnerability.

Vendor financing, or something more?

Of course, the “Infinite Money Glitch” is a bit pejorative and hyperbolic. We should acknowledge that financial arrangements of this form are not actually uncommon… and do not actually yield infinite money. It’s an arrangement more commonly called vendor financing and it happens all the time. For example, no one complains of an infinite money glitch when a car dealership offers you a loan to buy their car, even though the car dealership ends up being both the seller and the source of capital.

Companies do vendor financing frequently to move inventory that might otherwise sit unsold, lock in customers (they’re literally indebted to you), potentially earn interest income on top of product margins, book revenue earlier (though accounting rules vary on this), and gain competitive advantage over vendors who only take cash. The buyer also benefits from this arrangement by preserving cash for other needs and/or getting products they couldn’t otherwise afford.

But there’s a key risk here for the seller-lender — if your customer can’t pay you back, you’re screwed twice, as you’ve lost both the product AND the money. This dynamic led to the collapse of numerous telecom equipment companies in 2001, who provided vendor financing to startups that subsequently went bankrupt. This is important for Nvidia’s version of vendor financing which is at an extreme and unprecedented scale — NVIDIA putting up $100B is extraordinarily aggressive.

Additionally, when NVIDIA has 80-90% market share in AI training chips and nearly every major AI company needs their product, this isn’t normal vendor-customer dynamics. It’s more like a sovereign lending to its colonies - you need the currency (GPUs) to participate in the economy at all.

Thus the “infinite money trick” pejorative here is capturing something real — most vendor financing deals are a tiny fraction of the vendor’s valuation, and this is very different, with unprecedented scale and market dynamics. If AI compute demand slows or NVIDIA competitors catch up, this whole structure unwinds for NVIDIA in a very bad way. It’s vendor financing on steroids, enabled by a unique market position for NVIDIA that may not last forever.

Understanding the scale of the ambition

In speaking about the NVIDIA deal, OpenAI CEO Sam Altman said something that I hope is obvious to readers of this blog:

Everything starts with compute. Compute infrastructure will be the basis for the economy of the future.

In short, compute is a strategic resource, and OpenAI wants to have as much of it as possible.

How will OpenAI pull this off? Altman’s latest essay “Abundant Intelligence” spells out the plan:

Our vision is simple: we want to create a factory that can produce a gigawatt of new AI infrastructure every week. The execution of this will be extremely difficult; it will take us years to get to this milestone and it will require innovation at every level of the stack, from chips to power to building to robotics. But we have been hard at work on this and believe it is possible. In our opinion, it will be the coolest and most important infrastructure project ever.

To clarify, the scale here is enormous. “GW” refers to “gigawatt”, a measure of power. Each individual watt is enough power to run an old-school nightlight. A gigawatt is one billion watts — enough total power to supply roughly 750,000 homes. Altman wants to produce that each week.

Today, the largest AI data center is likely 0.3-0.5GW (xAI’s Colossus).2 There are projects to build 1GW data centers — currently these projects take about two years to produce start to finish. Altman proposes soon somehow speeding up this process 100x.

Across all of xAI, Meta, OpenAI, Google/DeepMind, Microsoft, and Amazon/AWS, there likely is 15-20GW total being used for AI.3 If not used for data centers, 20GW would be enough to power both New York City and London at the same time. Altman is talking about adding all of that every few months.

The entire United States currently has ~1300GW of total installed electrical generating capacity as of 2024.4 The US added 56GW of new capacity in 2024. Altman wants to add an equivalent amount annually, just for OpenAI’s data centers and AI.

This is a lot, to put it mildly. Achieving Altman’s vision would require nothing less than a complete reimagining of how data centers are built. This would fundamentally restructure global industrial capacity around AI infrastructure, likely requiring breakthrough technologies in modular construction, energy generation, and manufacturing automation that simply don’t exist today.

How will Altman pull this off? We will find out soon:

Over the next couple of months, we’ll be talking about some of our plans and the partners we are working with to make this a reality.

And besides Nvidia, how will OpenAI afford this? Altman isn’t yet saying.

Later this year, we’ll talk about how we are financing it; given how increasing compute is the literal key to increasing revenue, we have some interesting new ideas.

Can this dream be achieved?

Here’s how I think the AI buildout will go down.

Currently the world doesn’t have any operational 1GW+ data centers. However, it is very likely we will see fully operational 1GW data centers before mid-2026. This likely will be a part of 45-60GW of total compute across Meta, Microsoft, Amazon/AWS/Anthropic, OpenAI/Oracle, Google/DeepMind, and xAI.

My median expectation is these largest ~1GW data center facilities will hold ~400,000-500,000 Nvidia Blackwell chips and be used to train ~4e27 FLOP model sometime before the end of 2027.5 Such a model would be 10x larger than the largest model today and 100x larger than GPT-4. Each individual 1GW facility would cost ~$40B to manufacture, with ~$350B total industry spend across 2026.6

By the end of 2027, I expect7 a fully operational ~2GW facility with total AI compute across all companies reaching ~90GW. These 2GW facilities would cost ~$95-100B each to build and total industry annual spend would reach ~$500-600B.

By the end of 2028, I expect the largest single facility to be ~3GW facility, holding a ~1M chip Nvidia Blackwell/Rubin mix costing $150-165B to build, capable of a ~1e28 FLOP training run.8 A 1e28 FLOP training run represents a computational effort thousands of times greater than what was used to create GPT-4, allowing the AI to process and learn from vastly more information. Total AI data centers would reach 130GW combined, with ~$900B-1000B spent by the AI industry over 2028.

By 2029, it starts getting very fuzzy to predict and my forecasting powers break down. How AI continues to scale, whether we’ve encountered data-related or other algorithmic bottlenecks, what AI capabilities have already emerged, and how economically valuable those AI capabilities are will be key to whether the economics favor continued scaling. Needless to say, building $150B+ individual data center campuses and spending ~$1000B on AI infrastructure would get very difficult to sustain financially, let alone continue to increase dramatically year-over-year.

I’m also unsure about how well all the physical supply chains across compute and other forms of manufacturing will continue to support this level of scale or whether we can continue to get the actual energy buildout needed. You can only build physical infrastructure so fast. Whether we are still limited to training frontier AI systems in single data center campuses versus being capable of distributed training across geographically distributed data centers will matter a lot. The unit economics of training versus inference in allocation of compute will matter a lot. Additionally, wars or major regulation could significantly alter the picture. Substantial and rapid AI scaling beyond 1e28 FLOP requires many many different things to all go right. Thus, it’s plausible we could see a plateau of sorts around ~1e28 FLOP, or 1000x larger than GPT-4.

However, on an optimistic path where bottlenecks are resolved and AI is immensely economically valuable and generating sufficient financial returns to finance further AI scaling, 2029 might involve ~4GW facilities each costing $210-240B, with total compute reaching 180GW and total spending reaching ~$1200B-1400B annually. We could then see a ~1e29 FLOP model9 trained by the end of 2030, which would be 10,000x larger than GPT-4. If things keep going on the optimistic path, we also might finally be adding 1GW/week of AI infrastructure in 2030, but across all US AI companies, not just OpenAI.

This may sound fanciful. And keep in mind that even the not-so-optimistic scenario is still a very aggressive timeline involving very aggressive build-outs. This necessarily already prices in very aggressive investments from OpenAI and others that make the $100B NVIDIA investment look small. Many more headlines about $100B+ investments will need to occur regularly to keep this money-infrastructure train on track.

What does that get us? Some say that AI scaling is dead, but the rumors of scaling’s death have been greatly exaggerated. As I mentioned in my review of GPT-5:

GPT-5 should only be disappointing if you had unrealistic expectations — GPT-5 is very on-trend and exactly what we’d predict if we’re still heading to fast AI progress over the next decade.

While much is still uncertain, the financing and requisite infrastructure build-out suggests progress towards AGI is very much still on schedule, probably sometime in the early-to-mid 2030s. Get ready for the next five years, and we will truly see what some scaled AI models can do!

OpenAI is not the only hyperscaler

Another thing people might read into the announcement is that these investments suggest OpenAI is running away with it and is potentially on track to be the leader in frontier AI development. However, while OpenAI is being the loudest and most openly ambitious, they aren’t the only ones out there. In fact, all of xAI (Colossus 2), Meta (Prometheus), Amazon (Project Rainier), and Google (no catchy name unfortunately) are poised to have 1GW data centers by the end of the year, and all companies likely have what it takes to keep on going.

Hopefully the ability for OpenAI and Google to remain on the frontier in infrastructure buildout is obvious. For xAI, Elon Musk clearly still has the skills to raise the relevant capital and build infrastructure incredibly quickly. However, xAI’s financial footing also appears questionable, trying to justify a valuation much higher than Anthropic while most of their revenue appears to be inter-company transfers from Twitter. xAI has also recently started bleeding talent and it’s unclear how this will affect the company long-term.

Meta has had an impressive talent pivot, buying up superstar AI engineers, but their infrastructure pivot has been equally dramatic. They scrapped their entire data center playbook and are now building GPU clusters in “tents” — prefabricated structures prioritizing speed over redundancy. It’s not yet clear if Meta will be able to build frontier AI models that compete with OpenAI, Anthropic, Google, and xAI, but I wouldn’t count them out yet.

I’m most concerned about Anthropic. They’re making a big bet on Amazon, but it’s not clear if Amazon is in turn going to give them the capital needed to compete with the other players. Amazon also is betting a large amount themselves on their Trainium chips, eschewing NVIDIA chips, which are less proven. So far Anthropic has done a good job staying on the AI frontier, but it’s not clear if they can continue to do so year over year as the scale keeps getting bigger. But Amazon has a lot of capital available, so don’t count Amazon and Anthropic out just yet.

Lastly, we should mention the Chinese AI companies. Many Chinese companies, such as DeepSeek, Alibaba (Qwen), Zhipu AI (GLM), and MoonshotAI (Kimi) have an explicit focus these days on building AGI. But they’re just not currently on track to spend on the level of America. Alibaba’s $52B USD infrastructure plan sounds impressive until you realize it’s over multiple years and includes all cloud/AI spending, not just frontier AI training, and Alibaba’s cash generation is much lower than American companies. Additionally, the other Chinese AI companies are much smaller startups without deep access to capital. Lastly, US-led export controls bite hard here, preventing the build up of necessary chips, high-bandwidth networking, liquid cooling infrastructure, and fault-tolerant training systems even if the capital was there.

The economy is increasingly a leveraged bet on AGI

The reason we should be somewhat concerned — or at least curious — about this infinite money glitch is twofold. Firstly, AGI might lead to the serious destruction of everything we value and love, if not the extinction of the entire human race. Secondly, and much more mundane by comparison, because NVIDIA currently represents approximately 7% of the S&P 500’s total market capitalization. Add in Microsoft, Google, Meta, Amazon, and other companies whose valuations assume continued AI progress, and you’re looking at perhaps 25-30% of total market value predicated on AI transformation happening roughly on schedule.

In other words, AGI happening soon may mean the end of humanity, but at least the S&P 500 will remain strong. On the other hand, if the AI scaling hypothesis hits unexpected walls, the unwinding could be a second ‘dot com bust’ or worse.10 When everyone is both buyer and seller in circular deals, you’ve created massive correlation risk. If OpenAI can’t pay Oracle, Oracle can’t pay NVIDIA, NVIDIA’s stock crashes, and suddenly 25% of the S&P 500 is in freefall.

As Elon Musk notes, the “big question is whether the infinite money glitch lasts until the infinite money AI genie arrives”.