Claude Mythos Preview is, by every available benchmark, the most capable AI model ever built. But more striking than any benchmark are the thousands of previously unknown zero-day vulnerabilities that Mythos found in every major operating system and every major web browser — many of them critical, several of them decades old. As a result, Anthropic found the model too dangerous to release publicly.

Much digital ink has been spent analyzing Mythos, its cyber abilities, and what that means for our cybersecurity and national security. This is important and needs to be discussed. But the real headline is that Mythos is just the beginning. This kind of thing — finding abilities too dangerous to release — will become the new normal and this will only get more intense as AI companies build towards AI superintelligence.

So what is Claude Mythos, what does it mean, and what should we do? Let’s dig in.

Share

What is Claude Mythos?

Previously, Anthropic had three model sizes — Haiku (small), Sonnet (medium), and Opus (large). Capabilities generally increase as you increase the model size. Mythos is one such increase in size, larger than even Opus in the same way Opus is larger than Sonnet.

This increase in model scale has given Mythos notably stronger capabilities in a variety of domains — most notably cyberoffense. Yes, earlier models like Opus 4.6 could also do vulnerability discovery, especially with improved inference-time compute and improved scaffolding. This has led some to incorrectly dismiss the Mythos results as mere marketing hype.

But Mythos is plainly on another scale in terms of both quantity of vulnerabilities and typical severity. On a standardized Firefox exploit development task, the previous best model succeeded 2 times out of several hundred attempts. Mythos Preview succeeded 181 times. The UK government found Mythos to significantly outperform Claude Opus 4.6 on their 32-step corporate network attack simulation, with select runs completing all 32 steps.

As a result, Anthropic is drowning in high-severity vulnerabilities, of which fewer than 1% have been fully patched by their maintainers. Nicholas Carlini, a leading AI security researcher and Anthropic employee, said “I’ve found more bugs in the last few weeks with Mythos than in the rest of my entire life combined.”

Where this is going

On August 2, 1939, Einstein sent Roosevelt a letter warning that a nuclear weapon was possible. Six years later, it happened. This nuclear weapon was fully under government control. What would things have been like if the nuclear weapon was instead developed by private companies?

Today, experts similarly warn that AI superintelligence may be possible and would be even more transformative to the structure of global power than the invention of nuclear weapons. Six years later, will it happen? And will it be under our control?

Every time a new model comes out, people focus on what it can do right now and don’t think enough about where the trend line leads. The cybersecurity story for Mythos, as alarming as it is, is not as important as the trend. A year ago, AI could barely hack at all. It wasn’t until around June 2025 that AI was reliably helpful for hacking, and it wasn’t until November that AI could autonomously implement hacks.

Consider that just a few years ago, some AI scientists and top forecasters forecast that AI would be capable of cyberoffense exceeding professional humans. Many people at the time thought this was impossible, or at least far off. But it’s now here, today.

Now consider that these same AI scientists and top forecasters are warning that this is just the beginning — and that the same scaling dynamic that produced Mythos's cyberoffense will produce sharper capabilities across strategy, weapons design, military planning, and more. Of course, being right about the first doesn't automatically mean being right about the second. But the people who called Mythos early were working from a model of how AI progress works, and that model is performing better than its critics'.

If you were waiting for a sign that superintelligence is coming, this is it. If private AI companies succeed in building even stronger capabilities over the next few years, including AI superintelligence, these private companies would greatly exceed the power of the United States government and all world governments combined. Worse, the private companies may not be able to successfully control the AI superintelligence, allowing the superintelligence to become a power center in its own right.

Anthropic already possesses a cyberoffense capability that rivals many nations and the ability to, if desired, cause major damage. It is great that American frontier AI companies like Anthropic have shown restraint with their AI and how they are releasing it. But this restraint shouldn’t earn these companies a blank check. And right now they essentially have one — the status quo for AI is that AI companies determine nearly everything.

Anthropic made every consequential decision in this story. Whether to lock down, what to lock down, when to tell the government, what to share, who gets early access and who doesn’t, how to vet those who get access, what level of risks are acceptable, and what “responsible” means across all of this… What could’ve happened if Anthropic had simply released Mythos publicly, as most AI companies would do with a flagship model? There’s no law against it. Overnight, every intelligence community operation that depends on signals exploitation is potentially compromised.

How confident is Anthropic that model access hasn’t reached adversary states through a downstream partner or a compromised employee at a partner organization? How confident is Anthropic that the model can’t be stolen and then misused by a highly motivated adversary? How confident is Anthropic that Mythos’s capabilities can be contained and how long should we be aiming to contain them? How confident is Anthropic that future AI models might not escape their containment and independently wreak havoc?

What should government policy be when a company produces, among other things, an unparalleled cyberweapon? What if the future release is also capable of building unprecedented bioweapons? What if the release after that risks genuine loss of control for humanity?

What should we do?

The Manhattan Project answered an analogous question with a specific institutional design — private contractors and university labs doing the actual work, inside a federal security perimeter, with classification rules, cleared personnel, and ultimate government authority over deployment. That model wasn't statist — General Groves and Vannevar Bush were not central planners — but it recognized that some categories of capability cannot sit entirely inside private decision-making. Whether this is the right model for AI is contested — but the question itself is unavoidable.

If you believe in AI superintelligence, many policies have become more urgent:

• We need to be more serious about China’s ability to use American compute. Every AI chip that can train or run a Mythos-like model is more of a national security threat than before, and this will only continue. It will matter whether China can run 1000 or 100,000 advanced hacker AIs and the main way to stay ahead will be in compute advantage. Mythos was trained on an amount of compute that is currently not attainable to China via domestic manufacturing. If China were to train a Mythos-like model this year, it would be off of compute that is legally purchased from Nvidia, compute that is legally rented offshore, or compute that is smuggled. We need to better control semiconductor manufacturing equipment, prevent smuggling of chips, look again at what compute should be legal to sell to China and in what quantities, and ensure that the US maintains an advantage.

• We need to ensure that China, or other adversaries, cannot steal and misuse the Mythos model weights. It doesn’t make sense to try to maintain a lead over China if China can just steal our best results. Having more total compute will still give us an advantage, but our advantage would be even stronger if China couldn’t steal our model as well. Unfortunately, security at major US AI companies is not yet up to the task, and the task is tremendously difficult. If China were able to steal and misuse Mythos, that would already be a big deal. I suspect they will definitely try, and even if they don’t succeed at first, they will eventually. The US government needs to assist AI companies in helping them lock down their security.

• We need to consider what level of government oversight there should be on these increasingly powerful AI capabilities. Congress, not agencies and not private boards, should define what happens at the upper end of the capability curve. At some point, decisions about deploying systems that rival the coercive capacity of governments cannot sit inside a private corporate structure, however well-intentioned its leadership. Under the Constitution, that authority belongs to Congress. Statute, with sunset clauses and judicial review, is how this gets done — ideally without creating a sprawling discretionary regulator.

• We need a government body with the technical capacity to issue binding safety regulations on frontier AI companies. Drugs are regulated by the Food and Drug Administration, airplanes by the Federal Aviation Administration, and nuclear by the National Nuclear Security Administration. Drugs, airplanes, and nuclear are not perfect analogies for AI, but each combines promise with peril that needs to be carefully balanced. The FDA, FAA, and NNSA are not perfect regulatory bodies either — the FDA in particular arguably shows the drawbacks of how government regulation can overly harm and slow the benefits of innovation. AI superintelligence will be far more potentially beneficial but also far, far more dangerous than any drug or airplane. Right now the US has the Center for AI Standards and Innovation, but everything it does is voluntary and it is far under-resourced for what will need to be done. We need a narrow-mandate body solely motivated by national security and solely targeting only the most with the proper resources and technical skill to react quickly to superintelligence.

• We must also recognize that the AI race with China may be a race to see who loses control first. From a position of strength, we must consider negotiating mutual agreements on safe development. In order to do this, we will need better ideas about what safe development looks like and verification infrastructure to enforce a deal. In the late 1950s and early 1960s, the US, UK, and Soviet Union negotiated limits on nuclear testing but couldn't reliably detect underground tests — so the 1963 Limited Test Ban Treaty covered only atmosphere, underwater, and space. By the time verification technology improved, the treaty was already signed and wasn't substantively revisited until three decades later. This lesson is instructive – we must invest in AI verification before we need it. Any future US-China agreement on AI development requires verification infrastructure that doesn’t yet exist. Without it, deals require trusting China not to defect. With it, deals become enforceable through technical means rather than faith.

• We need a plan in case we want to slow down. Finally, the government should have contingency plans and insurance policies for scenarios in which the technical community concludes that further capability gains, on current methods, are outrunning our ability to control the resulting systems. This is not a recommendation to slow down now, but it would be prudent to know what slowing down would actually require and to have the option — before the moment arrives and we realize we have no brakes.
  

Looking forward

The national security implications of Mythos-like models are clear. But the stakes of AI superintelligence would be orders of magnitude higher. The point is not that Mythos will go rogue. The concern is that AI 10 more iterations above Mythos could go rogue… and Mythos illustrates, perhaps for the first time, how a superintelligent AI going rogue would actually pose a big deal for national security.

From Mythos, it is a straight shot in just a year or two to AI systems that are going to be far too strong to ignore. And from there, it won’t be long to superintelligence. It would be better to have a prepared government that already has practice getting things right, rather than a government rushing to the scene after it’s already too late.

This post was co-authored by Peter Wildeford and Theo Bearman, a Frontier Security Researcher at the Institute for AI Policy and Strategy (IAPS). It reflects their personal views only, not necessarily the position of their organizations.

~

Last month, Anthropic, OpenAI and Google each published evidence of systematic campaigns by Chinese AI companies to extract capabilities from American frontier models at industrial scale. Anthropic attributed attacks to three Chinese AI companies — DeepSeek, Moonshot, and MiniMax.

Anthropic’s investigation identified over 16 million exchanges generated through approximately 24,000 fraudulent accounts, all targeting Claude’s agentic reasoning, tool use, and coding capabilities. Chinese AI developer MiniMax was solely responsible for over 13 million of those exchanges. OpenAI also reported that Chinese actors had systematically targeted their ChatGPT models with distillation attacks designed to recreate the whole AI model training pipeline. And this is not the first time — in January 2025, White House AI czar David Sacks told Fox News there was “substantial evidence” that DeepSeek had built their model from distilled knowledge from OpenAI’s models.

Left unaddressed, AI distillation attacks pose a threat to American national security and economic competitiveness, given that they lead to China or other adversaries being able to develop better AI than would otherwise be possible by what is essentially stealing American technology. But what is a distillation attack and what can we do about it?

Share

What is a distillation attack?

An AI distillation attack occurs when a malicious actor uses the outputs of a “teacher” model to train a “student” model to approximate the teacher’s capabilities. Think of it like a restaurant owner who reverse engineers the recipe of a nearby Michelin starred restaurant’s prized dish by going there hundreds of times, ordering it each time, and figuring out their list of ingredients, measurements, and cooking instructions. In the AI context, the “recipe” is the billions of dollars of research, compute, and training data that goes into building a frontier model. The “dish” is the model’s outputs — the answers it gives and the intermediate reasoning steps it took to get there. By collecting enough outputs, an attacker can train a new model that mimics the original’s capabilities without doing as much underlying research and development work.

To be clear, distillation also has legitimate applications in the AI industry. It can be used to create efficient models suitable for edge deployment, specialize general-purpose models for specific domains, and reduce inference costs. Major AI providers including Google Cloud and OpenAI offer official distillation pathways for their customers. The difference is intent. Distillation attacks aim to replicate frontier capabilities in a rival model, violating terms of service and skipping the R&D investment required to build those capabilities independently.

Additionally, the actual magnitude of capability transfer from distillation is unclear and disputed. Integrating another model’s outputs into your own training pipeline is a genuine research challenge — the data can interact unpredictably with existing training, and the resulting model doesn’t always improve. Reasonable people disagree on assessing the magnitude of the threat as it depends on how effectively attackers can solve this problem, which is not yet clear.

How the attacks work

Accessing American frontier AI models in the first place can require circumventing geographic restrictions. Anthropic, for example, does not offer commercial access to Claude in China. To get around this, Chinese AI companies use commercial proxy services that resell access at scale. These proxies operate what Anthropic calls “hydra cluster” architectures — sprawling networks of thousands of fraudulent accounts that distribute traffic across the target company’s API as well as third-party cloud platforms. In one case, a single proxy network managed more than 20,000 accounts simultaneously, mixing distillation traffic with unrelated legitimate requests to avoid detection. When one account is banned, a new one takes its place.

This is a structural problem that no single AI company can solve alone. Proxy services profit from facilitating unauthorized access to frontier models – as they are paid per access, they are incentivized to facilitate as much distillation as possible. Individual AI companies can play whack-a-mole with fraudulent accounts, but the proxy services that create them will continue to do so as long as the business model is profitable and the legal risk is negligible.

Attackers use the victim model to generate vast quantities of high-quality training data and to clean and quality-score existing datasets, outsourcing what otherwise would be an expensive and labor-intensive stage of AI development. They query it with pairs of prompts to generate ranked responses – preference data that can be used in reinforcement learning. And perhaps most valuable of all, attackers attempt to extract the victim model’s internal reasoning process – the step-by-step chain of thought it uses to solve problems. For example, Anthropic reported that DeepSeek crafted prompts asking Claude to articulate the reasoning behind a response it gave, effectively reverse-engineering the thought process that makes frontier AI models so capable, despite their developers having already taken steps to summarize these outputs to avoid competitors training on them.

Why it matters

American AI companies have invested significant amounts in compute, data curation, and research talent to build frontier models. Data from EpochAI suggests that OpenAI and Anthropic alone have spent $18 billion on R&D compute since 2024. This potentially creates a competitive asymmetry where American AI companies that must recoup billions in training costs through API and subscriptions are undercut by competitors leveraging distillation attacks. By saving on development effort, Chinese AI companies can train and serve distilled models at little or no cost through their own platforms, or simply open-source them entirely, which would erode American companies’ competitive position. And as AI systems become increasingly central to AI research itself, the stakes of this problem are compounding — distillation doesn’t just transfer today’s capabilities, it can improve the attacker’s starting position for developing tomorrow’s.

Worse, this gives stronger AI capabilities directly to America’s adversaries. A February 2026 CSET analysis of over 9,000 PLA Requests for Proposal from 2023 and 2024 found that the Chinese military is actively seeking to integrate AI into command, control, communications, computers, cyber, intelligence, surveillance, reconnaissance, and targeting — including through DeepSeek models. An October 2025 Jamestown Foundation report similarly found that DeepSeek models are being deployed in Chinese military and public security settings, with PLA procurement documents explicitly calling for tools based on DeepSeek’s models and pilots already underway.

And the pipeline extends beyond China. A February 2026 CSIS analysis found that Russian military developers are actively adapting Chinese open-weight AI models — including Qwen, DeepSeek, and others — for battlefield use in Ukraine, embedding them in air-gapped environments for intelligence processing, reconnaissance analysis, and situational modeling. Sanctions have cut Russia off from developing frontier models independently, making Chinese AI models and smuggled Chinese chips the backbone of Russia’s military AI stack. Distillation that strengthens Chinese models therefore has downstream effects on Russian military capabilities as well.

Given that Anthropic’s Claude, integrated through Palantir’s Maven Smart System, was already reportedly used in combat during the US operation to capture Venezuelan President Nicolás Maduro and subsequently in support of US strikes on Iran, there are concerns if distillation attacks can put somewhat equivalent capabilities in adversary hands.

Additionally, frontier AI models are increasingly used as autonomous coding agents — writing, debugging, and deploying software with minimal human oversight. When these capabilities are distilled, the benefit extends beyond any single application. Better coding agents accelerate AI development itself: generating training data, writing model infrastructure, automating evaluations, and scaling research workflows. Distilling could thus be a compounding dynamic that makes the stakes of this problem grow with each generation of AI systems. This could additionally accelerate China’s entire AI development pipeline.

Distillation may make us overestimate China

To be clear, Chinese AI companies have significant independent training capabilities and do make genuine advances. Their AI capabilities are not due to distillation or other forms of IP theft alone. That being said, distillation still makes Chinese AI capabilities appear more independently developed than they are, since they can to some extent draft off of American innovation in addition to doing their own work.

This risks creating an illusion similar to the Cold War “missile gap,” when American policymakers wrongly believed the Soviet Union had surpassed US intercontinental ballistic missile production, despite the US actually having a substantial lead the entire time. Overestimating Soviet strength via the “missile gap” led to bad defense planning, and we should avoid a similar overestimate of Chinese AI capabilities.

A race to the bottom on safety

A distillation attack can copy a model’s capabilities without its corresponding safeguards, allowing the creation of a capable model with a lower barrier to misuse. If adversarial distillation becomes normalized, it may weaken incentives for individual AI companies to invest in safety measures, since those measures can be stripped away by downstream actors. This can create a race-to-the-bottom dynamic in which the most permissive deployment wins market share and investment in safety is not rewarded.

This is a concern for proliferation of AI misuse given that several AI companies rely on the principle of “marginal risk” when determining whether to release their AI models — assessing how much additional risk their models would create versus the status quo. Chinese models powered by distillation can shift the risk landscape, leading to US developers correspondingly deploying in riskier ways than they otherwise would as justified by not increasing “marginal risk” over a Chinese model. Less safe Chinese models can therefore precipitate the release of American models that are more useful to malicious actors.

What should be done?

The companies conducting these campaigns proceeded because the expected cost was trivial relative to the value extracted. Even under conservative estimates of distillation’s impact, the current enforcement gap — detection without meaningful consequences — invites escalation. Addressing this requires action from both companies and the government, and the appropriate response may need to scale as our understanding of the threat matures. We recommend two measures for the US government:

• Entity List the perpetrators. The End-User Review Committee should consider adding the Chinese AI companies conducting distillation attacks to the Entity List. The criteria for addition — “reasonable cause to believe, based on specific and articulable facts” that an entity is involved in activities contrary to US national security or foreign policy interests — appears to be met based on the evidence published by Anthropic, OpenAI, and Google. Entity List designation would require a BIS license, reviewed under a presumption of denial, for any export, re-export, or in-country transfer of items subject to the EAR. Beyond its direct effects, designation signals to the broader AI ecosystem — cloud providers, chip distributors, equipment vendors — that transacting with these entities carries regulatory risk.

• Sanction the attackers and their enablers under the Protecting American Intellectual Property Act. The PAIP Act requires the President to identify foreign persons who have knowingly engaged in, or benefited from, significant theft of trade secrets of US persons where that theft poses a significant threat to US national security or economic stability. Sanctions under the PAIP Act are more expansive than Entity List designation. The PAIP Act’s coverage extends to entities that have “provided significant financial, material, or technological support for” the theft — language broad enough to reach the proxy services described above. Outcomes such as inclusion in the Specially Designated Nationals and Blocked Persons (“SDN List”), travel sanctions, and being blocked from financial transactions subject to US jurisdiction, would particularly bite for executives heading up companies conducting and facilitating distillation attacks, especially as Chinese AI companies like Alibaba and Bytedance have significant commercial activity outside China. The first-ever PAIP Act designations were made on February 24, 2026, establishing operational precedent. Whether the aggregate extraction of model capabilities via distillation meets the statutory definition of a “trade secret” would be a novel interpretation, but the national security nexus is well-documented, but the evidence published by Anthropic, OpenAI, and Google could potentially form the basis for designations of the companies conducting distillation attacks.

Additionally, American AI companies are not powerless while waiting for government action. Several steps could significantly raise the cost of distillation attacks:

• Implement Know Your Customer (KYC) requirements for API access. Frontier AI companies should require some sort of lightweight identity verification for API customers, particularly for anyone doing high-volume or enterprise-tier access. Just as financial institutions verify customer identities to prevent money laundering, AI companies should verify that their customers are who they claim to be. This would make it substantially harder for proxy services to spin up thousands of fraudulent accounts.
  

• Invest in technical detection and rate limiting. Companies should develop more sophisticated behavioral fingerprinting to identify distillation-pattern queries — such as systematic chain-of-thought extraction, preference-pair generation, and large-scale data cleaning workloads — and throttle or block accounts exhibiting these patterns. Some of this is already happening, but the proxy ecosystem’s persistence suggests current detection capabilities are insufficient.
  

• Better enforce geographic access restrictions. Companies should invest in more robust geolocation and network analysis to identify traffic originating from restricted countries, even when routed through proxies. This includes analyzing patterns like VPN usage, payment methods, and account creation behaviors that correlate with proxy network operations.
  

• Pursue civil litigation against proxy services. Companies should consider legal action against the commercial proxy services that facilitate distillation at scale. Even where direct action against Chinese AI companies is impractical, the proxy services that operate as intermediaries may be within legal reach and their business model depends on low legal risk.

Looking forward

Distillation is a compounding problem. AI systems are increasingly being used to accelerate AI research itself – Anthropic has declared “We build Claude with Claude” and OpenAI has said the same about ChatGPT. As this feedback loop tightens, the US lead in AI becomes not just an economic advantage but a potentially decisive one. A country that maintains frontier AI capabilities can use them to pull further ahead; a country that closes the gap through distillation, chip smuggling, or other means enters that same loop from a stronger starting position than its independent capabilities would allow.

Both American and Chinese AI companies have been explicit that they are trying to build AI systems that would greatly exceed human experts across a wide range of tasks — including tasks related to military capability and national security — and that the path runs through AI systems that substantially automate AI research itself. While there is genuine uncertainty about timelines and feasibility, these are not fringe aspirations; they are stated engineering goals backed by tens of billions of dollars. As AI systems approach the ability to meaningfully accelerate their own improvement, it will matter whether the US or China is ahead — and by how much.

This is the strategic reality that should frame the distillation debate. We are protecting the seed of a capability that, if current trajectories hold, will compound into something without historical precedent. The steps we’ve outlined — Entity List designations, PAIP Act sanctions, KYC requirements, and technical countermeasures — would materially change the cost-benefit calculus that currently makes distillation a rational strategy. The question is whether policymakers will act while the US lead is still large enough to protect.

~

If you liked this article, consider hitting the subscribe button below. You can also follow Theo on Twitter!

Subscribe now

Anthropic’s Claude is the only frontier AI system known to be operating on Pentagon’s classified networks. Claude arguably helped capture Maduro.

But the Department of War wants more. On Tuesday, Secretary of War Pete Hegseth summoned Anthropic CEO Dario Amodei to the Pentagon and gave him until 5:01 PM today to grant the military fully unrestricted access to Claude. Additionally, Hegseth threatened to not only terminate its $200 million contract but gave two additional threats — that he would designate Anthropic a “supply chain risk,” or that he would invoke the Defense Production Act to legally compel Anthropic’s compliance.

A “supply chain risk” designation is normally only used for Chinese equipment like Huawei and isn’t even yet used against Chinese AI like DeepSeek, and would ban Anthropic from doing business with other Department contractors, which is a major loss of business.

Defense Production Act usage, normally reserved only for critical national security emergencies, would suggest Anthropic is the exact opposite of a “supply chain risk” — deeming Anthropic so essential to the national defense as to allow the government to compel Anthropic to produce even if they don’t want to.

Anthropic refused yesterday night, saying “We cannot in good conscience accede to their request.” We will now see if Hegseth follows through on his threats.

Why is this happening? Why has the Department gone to war over the model they’ve been relying on?

Much of this dispute is playing out in classified settings, and the public details are a mixture of facts and leak-based PR campaigns from both sides designed to shape the debate. But what we do know is concerning enough to take seriously. Some have framed this as a fight over “woke AI”, accusing Anthropic of overstepping and backseat driving the Department. It isn’t.

Share

What the Pentagon Wants

To understand how we got here, it helps to understand a different case. In 2017, the Department launched Project Maven — officially the Algorithmic Warfare Cross-Functional Team — to use machine learning to analyze drone surveillance footage. The military was collecting far more full-motion video from drones than human analysts could process, and it needed AI to help sort through the data, identify objects of interest, and flag patterns that analysts could then review.

But in 2018, thousands of Google employees signed a petition protesting their company’s involvement. Google pulled out. The damage was real and lasting. Analysts who had been promised AI tools to process overwhelming volumes of drone footage were left without them. Programs that had taken months to scope and staff had to start over. The episode left a justified conviction that Silicon Valley would abandon the warfighter the moment it became politically inconvenient.

So the Pentagon’s frustration might seem straightforward. According to the Pentagon, Anthropic must agree to let the military use Claude for “all lawful purposes” without restriction. The Department’s position is that the military’s use of technology should only ever be constrained by the Constitution and the laws of the United States — not by a private company’s acceptable use policy.

This is an entirely defensible principle. Raytheon does not tell the Pentagon which targets to hit with their missiles and Lockheed Martin does not tell the Pentagon which missions to fly with their planes. The military's authority to determine how it employs its tools is fundamental to civilian control of the armed forces. If every defense contractor could carve out moral vetoes over specific use cases, the result would be a patchwork of private restrictions that collectively let corporate boards shape military doctrine. That precedent problem is real and serious regardless of whether you think Anthropic's specific restrictions are reasonable. Thus Pentagon's desire for unrestricted access may not be mere bureaucratic stubbornness but also a real operational principle that tools in theater need to be predictable and fully available.

But that principle runs into a few problems — one of bad incentives, one of avoiding known technical limitations make certain uses unreliable regardless of what the contract says, and one of being entirely disproportionate in reacting to how a contract negotiation should work. And that's where these operational principles and Maven analogies break down in practice.

What Anthropic Wants

Put simply, Anthropic is not walking away from national security use. Claude is now integrated into Palantir's MAVEN Smart System — the successor to the very infrastructure Google abandoned. Anthropic was the first to provide custom models for national security customers. Claude is used across the Department for intelligence analysis, operational planning, cyber operations, and modeling and simulation. Anthropic has never raised objections to any particular military operation and never attempted to limit use of its technology on an ad hoc basis.

But when writing this contract with the Pentagon, Anthropic drew exactly two lines upfront —

First, Anthropic wanted no mass domestic surveillance of Americans. Anthropic supports the use of AI for lawful foreign intelligence and counterintelligence. But according to Anthropic, mass domestic surveillance is a different category. The law hasn’t yet been updated for how easily Claude or other AI can fuse scattered, individually innocuous data into a comprehensive picture of any person’s life — automatically and for millions of people simultaneously. This ends up a potentially large risk to liberty that should make any American nervous. “All lawful purposes” becomes a much bigger opening when AI radically expands what is technically feasible within existing legal gray areas that haven’t been updated for the age of AI.

Second, Anthropic asked for no fully autonomous lethal weapons without human oversight. This is because we don’t yet have “military-grade AI” — AI systems do not reliably execute commander’s intent and maintain chain-of-command fidelity under contested operational conditions. Current commercially available AI is too error-prone for effective warfighting and remains vulnerable to sabotage by adversaries — failures that could cause the United States to lose engagements and put warfighters at risk.

The Pentagon originally said yes to both of these principles. Then it reversed — and neither side has publicly explained why. Whether this was driven by a specific operational need that hit the restrictions, by political pressure from the administration, or by something else entirely matters enormously for how to interpret this dispute. But whatever the reason, in a normal contracting relationship, a change in requirements leads to contract termination, not coercion. The use of the “supply chain risk” designation and/or the DPA are unprecedented strongarm tactics normally reserved for combatting Chinese technology and times of acute national security crisis respectively — and they risk undermining the very partnerships the Pentagon needs.

Think about the incentives

This administration has positioned itself as pro-innovation and anti-overreach on AI. The AI Action Plan emphasized that American competitiveness depends on letting the private sector lead. That is the right posture for a country in a technology race with China.

But using the Defense Production Act to compel an AI company to retrain its models without safeguards, or designating a domestic AI champion as a supply chain risk over a contract disagreement, cuts directly against that posture. It is the biggest potential regulatory overreach in the AI space of any administration to date.

And the incentives are terrible. Any AI company watching this learns that Pentagon contracts can be renegotiated at any time for any reason, and that provisions the Pentagon agreed to — even ones that clearly uphold existing law and Department directives — could become the basis to punish you. The rational response is to never get on classified networks in the first place. Do what Google did with Maven and walk away.

What Should Happen

When OpenAI, Google, and xAI signed “all lawful purposes” agreements with the Pentagon, none of them were operating on classified networks under real operational pressure. It’s easy to agree to everything when the question is hypothetical and you want the business. Anthropic was the first to hit the hard problems because it was the first actually doing the work.

If the Pentagon doesn’t like the contract anymore, it should terminate it. Anthropic has the right to say no, and the Pentagon has the right to walk away. That’s how contracting works. The supply chain risk designation and DPA threats should come off the table — they are disproportionate, likely illegal, and strategically counterproductive.

But termination doesn’t solve the underlying problem: there is no legal framework governing how AI should be used in military operations. Right now the rules are being set through a combination of corporate acceptable use policies and Pentagon ultimatums, and that is not going to hold. Boeing’s flight control software is tested against known failure modes and certified to specific tolerances. LLMs are probabilistic and opaque. When Claude is used for intelligence analysis today, a human analyst reviews the output and applies judgment — and that’s working. Fully autonomous use near the kill chain is a different category of risk, and every frontier AI company in the world knows their models aren’t ready for it, whether they’ll say so publicly or not.

Congress needs to set the rules here: reliability standards for AI in operational contexts, incident reporting for AI-enabled systems, oversight for autonomous weapons, and statutory limits on AI-powered domestic surveillance. Congress should also fund the development of military-grade AI — systems that can execute commander’s intent under contested conditions, resist adversarial manipulation, and maintain positive command authority. DARPA has already made progress on this, but it needs dedicated funding and real coordination with frontier labs. No one has solved the reliability problem alone.

The warfighters who will depend on these systems deserve better than a contract dispute playing out over media leaks. And the public deserves a say in whether AI is used to surveil them — through their elected representatives, not behind closed doors.

This is a guest post written by Erich Grunewald, a prior “Power Law” contributor who just started a new blog on AI and compute topics called The Substrate alongside other members of the compute policy team at the Institute for AI Policy and Strategy. This post is crossposted from his new blog. Check it out!

~

The US has a massive export enforcement problem. It’s likely that over 100,000 export-controlled AI chips were smuggled into China in 2024. To give a sense of scale, the xAI Colossus cluster in Memphis, Tennessee, comprised first 100,000 and later 200,000 AI chips. That’s roughly an xAI Colossus cluster being smuggled to China each year. The main reason we know this is that smugglers are so unafraid that they’re willing to talk about their operations to journalists; this has happened repeatedly during the past year and a half.

AI chip smuggling is far from the only enforcement problem. In 2024, Huawei got TSMC to illegally fabricate over two million of its AI chip dies through front companies, despite sanctions. That is a far larger quantity than the number of Huawei AI chips fabricated domestically in China that year. We’ve also seen likely violations related to high bandwidth memory and semiconductor manufacturing equipment, which help China make its own AI chips to compete against NVIDIA.

What if you could pay insiders many millions of dollars to inform US authorities about such violations, at almost no cost to the US government? That would likely surface a large number of high quality tips about important violations, which would greatly aid US authorities in detecting, punishing and deterring such violations.

This idea may sound outlandish, but it’s actually possible. In fact, there is a law being discussed in Congress that would accomplish exactly this! But before we get there, let’s take a brief detour to the Securities and Exchange Commission (SEC) and the 2008 financial crisis.

Share

The SEC whistleblower program

In 2008, the US economy was reeling from the housing and mortgage crisis. Late that year, Bernie Madoff sat down with his two sons and admitted to them that the investment business he’d been running for two decades was a giant fraud, a Ponzi scheme to end all Ponzi schemes. Because of these two crises, there was a desire among policymakers to strengthen financial regulation and oversight.

Related to the Madoff scandal in particular, the SEC was under criticism for failing to properly investigate several credible reports about it. An employee at a rival investment firm, Harry Markopolos, had been asked by his employers to figure out how Madoff could post such consistently excellent returns, and soon realized that the returns were impossible with Madoff’s claimed strategy. Markopolos later said in an interview: “I read his strategy statement, and it was so poorly put together. His strategy as depicted would have trouble beating a zero return, and his performance chart went up at a 45-degree line: that line doesn’t exist in finance, it only exists in geometry classes.”

Markopolos sent reports to the SEC detailing Madoff’s fraudulent activities on multiple occasions before the 2008 financial crisis. However, the SEC failed to properly investigate these reports, leaving Madoff free to continue defrauding investors until the financial crisis made its collapse imminent. Lawmakers realized that reports of wrongdoing from the general public could be a valuable tool for detecting and deterring securities laws violations.

One result of this, signed into law in 2010 as part of the Dodd-Frank Act, was the SEC whistleblower program.1

The SEC whistleblower program works like this. First and most importantly, whistleblowers get 10-30% of any penalty resulting from their report. This can be many millions of dollars—the largest reward to date, paid out in 2023, was nearly $280 million. This monetary incentive is paired with protections against retaliation from their employers, confidentiality guarantees, and the ability to make reports to the SEC anonymously through an attorney. To pay out whistleblower rewards, the Dodd-Frank Act also sets up an Investor Protection Fund, which receives penalties from securities violations (previously these would go to the Treasury).

The SEC whistleblower program is widely considered to have been an enormous success. It’s now one of the key ways that securities law is enforced in the US. It has helped generate $7.3 billion to $22 billion in penalties since its inception in 20112, and has received reports from at least 130 countries. Quantitative evaluations are rarer, but existing research suggests it has reduced financial reporting fraud, deterred insider trading, and caused companies to strengthen compliance programs.

The Stop Stealing Our Chips Act

Now the question is, could you adopt the SEC whistleblower program model for the Bureau of Industry and Security (BIS) and export violations? The Stop Stealing Our Chips Act — introduced into the Senate in April 2025 by Senators Rounds (R-SD) and Warner (D-VA) and into the House a couple of weeks ago by Representatives Kean (R-NJ) and Johnson (D-TX) — would do exactly this.

The Stop Stealing Our Chips Act (henceforth, SSOCA) is closely modeled on the SEC whistleblower program, with some changes to adapt it for the export enforcement situation. It too offers whistleblowers 10-30% of any resulting penalty along with whistleblower protections, including the possibility of making anonymous reports to BIS.

The first thing to note here is the financial incentive. As economists always tell us, financial incentives are incredibly powerful, and the fines for these violations can be enormous:

• There have been several news reports of operations involving on the order of 10,000 smuggled AI chips, meaning roughly $400 million worth. BIS can fine up to twice the value of the related transaction, so that could be a penalty of $800 million, for just one smuggler who spoke to the news media. If a whistleblower reports on that, they could get up to 30% or $240 million (leaving $560 million for the US government).

• The massive TSMC-Huawei violation—which was only detected when an independent organization did a teardown of a Huawei chip—could reportedly result in a $1 billion fine. This would have been up to $300 million for an informant.

Beyond catching violations, a well-publicized program could have significant deterrent effects. If everyone in a supply chain—sales reps, warehouse workers, freight forwarders, accountants—knows that reporting can yield millions, violators face a much riskier environment. This effect could be realized even before the whistleblower program comes into effect, as the law would allow whistleblowers to report on violations that occurred before it was signed into law.

Would the BIS program actually surface any tips?

All right, hundreds of millions of dollars is a strong incentive. But, you may ask, are there actually people with information about these violations who would be willing to step up and blow the whistle? Why, yes there are!

Take AI chip smuggling operations: these involve lots of people who could potentially file reports, in other words people who have relevant information and would like to get millions of dollars. This includes, for example, people working in sales at exporters with questionable compliance practices; employees at local resellers, freight forwarders, logistics companies, warehouses, or data centers where the chips are temporarily housed; and accountants and lawyers.

In March 2025, Singaporean authorities arrested three people for smuggling $390 million worth of AI servers. These arrests were the result of an “anonymous tip-off”, in other words a whistleblower report! It seems likely that the recent Operation Gatekeeper arrests of an AI chip smuggling ring operating out of Texas and New York were also the result of an insider tip.

The story is similar for the TSMC-Huawei violation, where there were likely many TSMC employees who could’ve known about this problem and informed the US government. The only reason the US ultimately found out about this violation was because an independent party—TechInsights—did a teardown of a Huawei AI chip, and noticed it was TSMC-fabricated. A BIS whistleblower program would likewise incentivize such actors to look for evidence of violations and report those to the US government. This type of information is hugely valuable; it makes no sense to sit around and wait for people to offer it out of the goodness of their hearts.

As with the SEC program, the SSOCA makes foreign nationals eligible for rewards. This is important because many export violations happen in third countries, where goods are diverted via reexport or transshipment. (The SSOCA does however wisely make some exceptions for known terrorists and sanctioned persons, who are not eligible for rewards.) This is similar to how the intelligence community pays foreign informants, who provide the US government with information that benefits US national security.

Would BIS be able to run the program?

At this point, the wise reader will ask, “Isn’t BIS extremely resource constrained? If so, how is it supposed to process and investigate a bunch of incoming tips, determine awards, and carry out outreach on the program?” After all, BIS’s budget for enforcement has been essentially flat when accounting for inflation for at least the past five years (see figure), despite BIS receiving a vastly increased scope of responsibilities due to the AI chip export controls introduced in October 2022 and the Russian invasion of Ukraine and all the diversion related to that conflict.

Good question! The answer is that these activities—investigating whistleblower reports, determining awards, and carrying out outreach—would also be financed through incoming penalties. This is one of the most notable differences between the SSOCA and the SEC program. The SSOCA authorizes BIS to use money from penalties for a few additional purposes and not only for paying out rewards to whistleblowers. (As currently written, the SSOCA would only allow BIS to receive money from penalties that stem from whistleblower reports, but I think this should be expanded to cover all penalties for BIS-related violations.)

Today, any fine levied by BIS goes straight to the Treasury, or in rare cases it is earmarked for some specific fund, such as the Crime Victims Fund. What the SSOCA would do is redirect these to an Export Compliance Accountability Fund. This Fund would be used to pay rewards to whistleblowers; any money left over would go first to core functions of the BIS whistleblower program, and then to export enforcement activities more broadly.

There is a separate but related question of how the program would be funded initially, if it’s mainly intended to be funded through penalties. However, BIS already has a fairly steady stream of enforcement actions, including from likely insider tips, without any whistleblower incentive program (see figure). BIS may also be able to direct some of its appropriated resources to the program in the first one or two years, in order to get it up and running.

It seems likely that this program would pay for itself if implemented. That’s because the program would likely help BIS detect more violations and therefore levy more penalties than it would without the program. It could well end up both reducing the number of violations and also generating additional revenue for the federal government by making it much more likely that violations are detected and enforced. The losers here would be the smugglers and other bad actors who wake up every day trying to figure out ways of harming US national security.

BIS’s entire budget for fiscal year 2025 was about $191 million, likely far smaller than the collective profits of AI chip smugglers alone, which may well have exceeded $1 billion. A single successful enforcement action against a major smuggling operation could pay for BIS’s entire annual budget—for example, last month US authorities arrested three individuals accused of smuggling AI chips worth $160 million to China, which could result in a penalty of $320 million. There are likely dozens of such cases remaining to be discovered. A BIS whistleblower program like the one described in the SSOCA could create a virtuous cycle where more tips and better enforcement lead to more penalties and rewards, which in turn leads both to more tips by publicizing the program and also more resources for enforcement.

~

If you liked this post, consider subscribing to The Substrate where Erich and others will be writing more about AI and compute topics.

Subscribe to The Substrate

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022. Here, he shares the news and analysis that informs his forecasts.

At 2am local time in Caracas on January 3rd, explosions lit up the night sky over Venezuela’s capital. By 5:21am, President Donald Trump announced on Truth Social that Nicolás Maduro and his wife Cilia Flores had been “captured and flown out of the Country.” The operation, codenamed Absolute Resolve, deployed approximately 150 aircraft and sent Maduro to New York to face narco-terrorism charges. And in what could be symbolism or coincidence, the operation occurred exactly 36 years to the day after the US’s capture of Panama’s Manuel Noriega and exactly 6 years after Trump’s strike on Iran’s Qasem Soleimani.

But execution is not the same as strategy. While the operation itself appears to have been flawlessly executed, what comes next isn’t clear. Let’s dig in.

Share

How Venezuela reached the breaking point

Like most events, this is just the continuation of decades of history. We start with Hugo Chávez, a charismatic former paratrooper who won Venezuela’s presidency in 1998 on a wave of populist anger. Flush with oil revenue during the commodity boom of the 2000s, Chávez built a socialist state that redistributed wealth to the poor while systematically dismantling democratic institutions. When Chávez died of cancer in March 2013, he left behind a handpicked successor: Nicolás Maduro.

But the petrostate model that was successful for Chávez had begun unraveling. Global oil prices collapsed from over $100 per barrel in 2014 to below $30 by early 2016. Venezuela’s economy, dependent on oil for 95% of export revenue, imploded. Hospitals ran out of medicine. Grocery stores emptied.

Maduro responded not with reform but with repression. When the opposition won control of the National Assembly in 2015, he packed the Supreme Court and created a parallel legislature to strip it of power. When protesters filled the streets in 2017, security forces killed over 160. Opposition leaders were jailed, exiled, or banned from office.

The 2018 presidential election — which Maduro “won” after banning his main rivals — triggered an international crisis. In January 2019, Juan Guaidó, a 35-year-old opposition legislator, declared himself interim president under a constitutional provision allowing the Assembly head to assume power when the presidency is deemed illegitimately won. The Trump administration recognized him within minutes. Nearly 60 countries followed.

But Guaidó never gained control of anything that mattered. The military stayed loyal to Maduro. By December 2022, his public support had collapsed to single digits, and the Maduro opposition parties supporting Guaidó voted to pull the plug and dissolve his interim parallel government, realizing they had accomplished nothing. Guaidó now lives in exile in Miami.

María Corina Machado, recent Nobel Peace Prize winner, represented something different. When the opposition held primaries in October 2023, Machado won with 93% of the vote. Polls showed her crushing Maduro in a general election. So the regime did what it always does: the Supreme Court upheld a 15-year ban on her holding office, citing her support for US sanctions and involvement with Guaidó. But rather than concede, Machado found a workaround: Edmundo González, a 75-year-old retired diplomat with no desire to campaign, who agreed to serve as her stand-in.

On July 28, 2024, Venezuelans went to the polls. The government’s response was brazenly fraudulent. The National Electoral Council declared Maduro the winner with 51.95% without publishing the precinct-by-precinct tallies that had accompanied every previous Venezuelan election. Officials blamed a “cyberattack from North Macedonia” — completely baseless.

But the opposition had prepared. In incredible bravery and coordination, approximately one million volunteers secretly collected the election tally sheets from over 83.5% of polling stations on election night. Their data told a starkly different story: González had won with roughly 67% to Maduro’s 30%.

But Maduro was undeterred and what followed was brutal repression. Security forces went door-to-door arresting protesters, poll workers, and opposition members. González fled to Spain in September after authorities issued an arrest warrant and Machado went underground.

When Maduro proceeded with his inauguration on January 10, 2025, the Trump administration began escalating pressure. The bounty on Maduro was raised to $50 million, the highest in State Department history. By late 2025, the US had deployed the USS Gerald R. Ford carrier strike group, approximately 15,000 soldiers, F-35 jets, and a nuclear-powered submarine to the Caribbean. Trump declared a “blockade” and announced the country was “completely surrounded by the largest Armada ever assembled in the History of South America.”

Then came today.

Absolute Resolve

The operation had been months in the making. CIA teams tracked Maduro’s pattern of life — where he slept, what he ate, how he moved. US forces rehearsed the raid on a replica of his residence. Trinidad and Tobago quietly signed an agreement granting US military access to its airports. By late December, troops were staged and waiting for favorable conditions.

Trump gave final approval around 11 PM Eastern on January 2. The weather was good. The trigger was pulled. 50 aircraft launched from 20 different bases to dismantle Venezuela’s air defenses and clear a path for the helicopters coming behind them. Parts of southern Caracas went dark in what Trump later said was due to “certain expertise.” Behind the air assault came the 160th Special Operations Aviation Regiment that carried Delta Force operators with a specific target: Maduro’s residence inside Fort Tiuna.

According to Trump, Maduro was in “a house that was more like a fortress than a house. It had steel doors, it had what they call a safety space — solid steel all around. He didn’t get that space closed. He was trying to get into it, but he got bum rushed so fast that he didn’t get into that.” No Americans died. By 3:29 AM Eastern, the assault force was back over water, Maduro in hand.

Opposition sources told Sky News they believe Maduro’s capture was a “negotiated exit,” with portions of the regime facilitating rather than resisting. The evidence is circumstantial but suggestive: despite months of warning, Venezuela’s Russian-supplied S-300 air defense systems never prevented helicopter ingress. The military mounted no meaningful counterattack. Interior Minister Diosdado Cabello appeared on television vowing defiance, but the streets remained largely empty.

Whether this was collapse, capitulation, or negotiated exit, the result is the same: for the first time since the US toppled Panama’s dictator Noriega in 1990, the United States has forcibly removed a sitting head of state from power.

The strategic logic

But why Venezuela and why now? The Trump administration is explicitly not calling this regime change but instead framing the operation as the Department of Justice executing an arrest warrant. “At its core, this was an arrest of two indicted fugitives of American justice, and the Department of War supported the Department of Justice in that job,” Rubio said at the Mar-a-Lago presser.

But this seems flimsy — it definitely does seem to be regime change. And maybe this is justified. Maduro is genuinely a bad guy. He stole an election, became dictator, was repressive, is leading a lot of cocaine trafficking, and drove 20% of the population to flee the country. The humanitarian and democratic case against him is airtight.

Additionally, while Trump owns the operation, Venezuela has been Rubio’s white whale for years. He had consistently been the most hawkish voice on Caracas in the Senate, viewing Maduro’s removal as both a strategic imperative and good politics with Florida’s Cuban and Venezuelan voters. This continued as Rubio joined the Trump administration. In July, Rubio posted that “Maduro is NOT the President of Venezuela” but rather “the head of the Cartel de Los Soles, a narco-terror organization which has taken possession of a country.” At the presser, he was already pivoting to Cuba, noting that Maduro’s security detail and spy agency were “basically full of Cubans” and warning Havana that “I’d be concerned.”

Moreover, the oil angle is explicit. Trump announced the US would be “very strongly involved” in Venezuela’s oil industry, with American companies moving in to extract wealth that would flow to both Venezuela and “the United States in the form of reimbursement.” Venezuela has the world’s largest proven oil reserves. Whether this is opportunism layered onto the drug war justification or the actual primary motivation, Trump isn’t hiding that it’s part of the motivation.

What does this mean for geopolitics?

But there’s also a geopolitical strategic case for toppling Maduro. Venezuela has hosted Russian military assets—Tu-160 bombers visited in 2018 and 2024, and Russian advisors have been embedded with Venezuelan forces. China holds billions in Venezuelan debt and has been receiving sanctioned oil shipments.

Back in 1902, Venezuela of all places was facing a blockade not from US forces but from European forces. Venezuela had defaulted on its debts to European creditors and so Britain, Germany, and Italy responded by blockading Venezuelan ports. US President Theodore Roosevelt watched with alarm, seeing that the Europeans might use this as a basis to return to the colonialization of the Americas. Roosevelt declared the Roosevelt Corollary — the US should be the police of the Americas, not Europe. This was an expansion to the earlier 1823 Monroe Doctrine, a declaration from President James Monroe that the Americas were in the sphere of influence of the US and no longer open for colonization from Europe.

As Anthony Constantini argued, the Trump administration has been developing what amounts to a “Trump Corollary” to go along with the "Roosevelt Corollary”, updating the Monroe Doctrine for the 21st century. The Trump Corollary extends American dominance claims over the entire Americas beyond military presence to economic influence, hence the pressure over Chinese-controlled ports at the Panama Canal. Having a hostile drug-smuggling dictator with Russian and Chinese military ties 1,300 miles from Miami is exactly the kind of problem the Monroe Doctrine was designed to address, let alone the Trump Corollary.

This reassertion of hemispheric dominance invites a question: does it signal a broader return to great power spheres of influence? The French term pré carré — literally “square field” — describes the logic of exclusive domains, where Russia claims its near abroad, China asserts control over East Asia (including Taiwan), and the US enforces primacy in the Western Hemisphere but retreats from Europe and Asia. Under this framework, major powers implicitly agree to stay out of each other’s backyards.

There’s perhaps something to this interpretation. Trump has shown little appetite for defending Ukraine, and his pressure on European allies to handle their own defense suggests a willingness to let that theater go. Venezuela would then represent America claiming its pré carré while ceding others to rival powers.

The December 2025 National Security Strategy provides our best window into the administration’s thinking, which actually put the Western Hemisphere appears first among regional priorities — before Asia, before Europe, before the Middle East. The strategy explicitly calls for “a readjustment of our global military presence to address urgent threats in our Hemisphere... and away from theaters whose relative import to American national security has declined in recent decades.”

Here the language on the “Trump Corollary” is unambiguous: “We will deny non-Hemispheric competitors the ability to position forces or other threatening capabilities, or to own or control strategically vital assets, in our Hemisphere.” This is the Roosevelt Corollary with updated targets — swap European colonial powers for Chinese port operators and Russian military advisors.

An alternative perspective is what analyst Tanner Greer calls the “prioritizer” framework, under which China is a unique peer competitor and threat to the US, requiring America’s full attention and focus. Under this logic, the US saves its power for countering China, avoids lasting entanglements in Europe and the Middle East, but does use decisive force for discrete, limited objectives.

It’s not clear what, if any, logic Trump is following. Perhaps Trump’s approach is neither cleanly pré carré or “prioritizer” but something more opportunistic. The strategy asserts hemispheric dominance aggressively while adopting a more transactional posture toward great power competition. The statement that “the days of the United States propping up the entire world order like Atlas are over” is telling. This isn’t sphere-of-influence realism so much as selective assertiveness, with maximum force where costs are low and benefits are tangible, across targets as wide ranging as Venezuela, Iran, and even Nigeria — but explicit restraint or burden-shifting where they’re not, like Ukraine. Where Taiwan falls on this logic is not immediately clear — strategic ambiguity at its finest.

What happens next?

But of course, the harder question is what comes for Venezuela after the helicopters leave. While the execution was successful, there doesn’t seem to be any clear plan for what comes next.

The Noriega parallels are striking — both leaders faced drug trafficking indictments, both stole elections, and the US recognized the opposition candidate as legitimate. But the differences matter more. In Panama, the US already had 12,000+ troops stationed in the Canal Zone — roughly the size of the entire Panamanian military — and a government-in-waiting ready to assume power. Venezuela is 12 times larger with over 100,000 troops equipped with Russian weapons. There is no equivalent infrastructure for transition.

Additionally, Noriega was the beginning and end of the Noriega dictatorship. But the Maduro dictatorship is more embedded — Maduro was never just about Maduro. There instead remains is a collective dictatorship — security services, patronage networks, illicit finance pipelines, and a political-military elite that shares criminal liability with Maduro and faces their own US indictments. Interior Minister Diosdado Cabello, who appeared on television today saying the Venezuelan government would not be cowed, carries a $25 million bounty of his own. Defense Minister Vladimir Padrino Lopez has run Venezuela’s military longer than anyone in modern history and remains in place.

The snake has been decapitated, but the body is still moving. The existing post-Maduro Venezuelan regime might abide by what Trump wants, but they remain staffed by people who face their own indictments and have every incentive to resist a democratic transition that would hand them to American courts.

Venezuelan reaction has been deeply polarized. In Doral, Florida — home to the largest Venezuelan diaspora community in the US — crowds gathered outside El Arepazo restaurant chanting “Libertad!” and singing both national anthems. Similar celebrations erupted in Santiago, Lima, and Madrid, where one Venezuelan told CNN: “At first we were crying because our country was being bombed, but when we were told they had Maduro, the reaction was overwhelming.”

Inside Venezuela, the picture is grimmer. Caracas residents described waking up terrified. “How do I feel? Scared, like everyone,” one resident told the news. “Venezuelans woke up scared, many families couldn’t sleep.” Pro-Maduro supporters led by Caracas Mayor Carmen Meléndez marched through the capital demanding his return, chanting “Maduro, hold on, the people are rising up!” The streets were otherwise largely empty, patrolled by security forces, as residents stayed indoors absorbing what had happened.

When it comes to figuring out who should govern Venezuela next, opposition leader Machado issued a statement calling for González to assume the presidency. But Trump said it would be “very tough” and that Machado “doesn’t have the support or the respect within the country.” His team had not been in touch with her or any member of the Maduro opposition.

And there’s also a constitutional wrinkle. Executive Vice President Delcy Rodríguez is technically Maduro’s legal successor — and she’s reportedly in Moscow. The US wants to work with Rodríguez over Machado and González, despite Rodríguez being a committed Chavista who appeared on state TV demanding Maduro’s release as “the only president of Venezuela.” And this dismissal of Machado is striking and underexplained. Recall that Machado won the opposition primary decisively.

Why? Perhaps Trump cares more about stability and oil access than democratic transition. Rodríguez represents continuity with the existing power structure whereas Machado represents upheaval. Or perhaps portions of the Maduro regime negotiated this outcome and accepting Rodríguez was the price of Venezuelan non-resistance. Or perhaps the Trump administration has no plan and is improvising on the fly. Perhaps Trump wanted the win, got the win, and figured they’d sort out the details later. Note that none of these three theories are mutually exclusive.

Trump’s stated plan is for the United States to “run” Venezuela “until such time as we can do a safe, proper and judicious transition,” with American oil companies moving in to extract wealth that would flow to both Venezuela and “the United States in the form of reimbursement.” But that’s barely a plan. Run it how? With what personnel? Under what legal authority? “Running” a country of 28 million people without boots on the ground isn’t a plan. Capturing Maduro doesn’t automatically produce Venezuelan democracy.

And Democrats are furious. The Gang of Eight, a bipartisan group of Congressional leaders, are normally briefed in advance due to requirements by law to maintain balance between executive war powers and congressional oversight. But they weren’t briefed this time. Trump’s justification—that “Congress has a tendency to leak”—is essentially a middle finger to Article I, which normally puts Congress in the drivers seat when it comes to war. Article II does give the US President authority to do military operations to protect US personnel from an actual or imminent attack, but the imminent attack framing is doing a lot of work here for an operation targeting a leader who posed no immediate threat to American forces.

Looking forward

Across Venezuela, Fordow, and Soleimani, a pattern emerges — the willingness to use overwhelming military power for specific objectives, combined with an unwillingness to own the consequences indefinitely.

But the most important part is that there was no 100,000+ troop occupation, no nation-building doctrine, no decade-long-plus commitment. There was just a precise operation with a specific target, executed and completed within hours. After American foreign policy has been haunted by Iraq for two decades, Trump shows a model of limited military action that accomplishes discrete foreign policy objectives without the quagmire of full-scale invasion.

Sometimes these gambits work. The Abraham Accords, a series of US-brokered agreements signed in 2020 that normalized diplomatic relations between Israel and several Arab nations — including the UAE, Bahrain, Morocco, and Sudan — were dismissed as superficial PR but turned out to be genuinely durable.

Whether that bet pays off in Venezuela will tell us a great deal about how the next three years unfold. If the transition succeeds, expect this template to be applied elsewhere — perhaps Cuba, perhaps Nicaragua, perhaps targets we haven’t anticipated. If it fails, we’ll learn something important about the limits of precision force in an era of diffuse power.

I’m genuinely uncertain which way this goes. It’s good that Maduro is out, but I’m worried about whether democracy is what comes next.

-

Thanks to Caroline Jeanmaire for help with editing and contributing analysis.

This template was made by Peter Wildeford and Caroline Jeanmaire.

It’s almost a new year and that often calls for some sort of annual planning.

But in my opinion, annual planning operates on too long a timescale. Annual planning and goal setting is too infrequent, so it’s easy to lose track. You don’t get fast enough feedback and you risk derailing.

Instead, I suggest adopting a quarterly planning cadence (e.g., set goals for 2026 Jan-Mar). I wanted to provide a template that I’ve used many times before and that many people seem to like.

Here’s my template in a Google Doc => click here and make a copy.

Share

How is this different from other templates out there?

There are a lot of different templates out there to try and I don’t know if this one is better. It works well for me, but I’m not you. But here’s some principles I was interested in, and maybe that resonates with you:

• Optimized for something quarterly – rather than something monthly (too frequent, too time consuming) or something annual (too infrequent, easy to lose track, not fast enough feedback).

• Forces you into relentless focus on the most important things (three major rocks and two optional minor pebbles) and encourages you (via anti-rocks) to not do too much.

• Designed to be very achievable. The template is somewhat minimal and much of it focuses on building systems and making sure your goals will happen.

• Designed to be very customizable. You can make the template your own!

What’s actually in this thing?

The template walks you through five steps. The fast-track version takes about three hours (seven 25-minute pomodoros), though you can spend longer if you want to go deeper.

Step 1: Reflection. You start by looking back at the prior quarter—key successes, key mistakes, how you did against prior rocks. There’s also a quick 1-5 rating across eight areas: work, sleep, exercise, nutrition, routines, finances, relationships, and environment. The point isn’t to obsess over numbers but to notice patterns and identify 1-2 problem areas that deserve focus.

Step 2: Vision. A quick check on your values and mission (or just jot keywords if you don’t have one), plus a 90-day visualization. What do you want to have/be/feel in three months?

Step 3: Combine and Refine. Take everything from Steps 1-2 and distill it into a raw list of potential priorities, then filter down to 3-5 candidate themes. The template also asks you to identify the “effort type”—is each item a project, a learning goal, or a habit change? This matters because habit changes require different strategies.

Step 4: Set Rocks and Pebbles. The core of the template. You get three major rocks and two optional pebbles. For each rock, you define SMART goals (both input goals for what you’ll put in and output goals for what you’ll achieve), plus IF-THEN planning. You identify one key obstacle and write out “IF [obstacle occurs], THEN I will [specific response].” Each rock also asks: “Who could you share this rock with for support/accountability?” External commitment helps.

Step 5: Sanity Checking. Reality-check your plan: Is there enough time? Do you have slack for the unexpected? What’s your system for weekly/monthly review? Will travel or environment changes derail you?

The template includes a 7-pomodoro fast-track guide if you want structure for completing it in a single ~3 hour session.

Tips Before You Start

• You really shouldn’t try to do too much. This is the most common mistake. Three rocks. Maybe two pebbles. That’s it. If you find yourself wanting more, that’s a sign you need to prioritize harder.

• Habits take a lot of time to build. Be very careful about adopting a new quarterly review, new monthly review, new giant list of 20 habits, new morning + evening routine, etc. all in one go. Ideally build these gradually over 1-2 years. If you’re new to this kind of planning, make “establish a quarterly review habit” one of your rocks.

• Do this somewhere other than your usual workspace. It helps you think outside the box.

• Don’t do it all in one sitting. Important thoughts and connections emerge in between sessions.

• Make it your own. I went through a bunch of materials and distilled it down to one template that works well for me. The true power comes from customizing it to meet your own needs. Want to track books? Add that. Want fewer questions? Remove stuff. Appendix B has a bank of additional reflection questions if you want more.

~

If you’re interested, make a copy of the template (click here) and get to work!

This template was made by Peter Wildeford and Caroline Jeanmaire. It is based on Gustin’s annual review, the Quarterly Productivity Planner, Ben Todd’s Personal annual review process (2020), Ultraworking’s monthly template, the Beyond Goals workshop, the Forcing Function annual review, Ben Kuhn’s weekly review habit, Alex Beal’s “2022 OKRs”, Eric Barker’s “What’s the best way to set a goal?”, Konrad Seifert’s “Life Review, Planning & Organization”, and Jan’s “Review, reflection, goal-setting prompts”. Intend.do and Zen to Done also serve as inspiration.

This is a guest post written by Oscar Delaney, reposted from his Substack (see here and here for our previous coauthored posts).

~

James Fearon’s classic1 1995 paper “Rationalist Explanations for War” argues that there are two main reasons rational states fight: private information about their own capabilities and resolve, with the incentive to misrepresent this, and commitment problems when trying to reach a negotiated agreement.2 I claim that both of these, especially the latter, contribute to a significant risk of pre-emptive war in the lead-up to one state developing ASI.

On a basic rational actor model, war seems puzzling. It causes large deadweight losses to belligerents, and therefore both sides would be better off reaching a negotiated agreement to split the issues at stake roughly proportionally to the military strength of each side. That is, if Strongland has an 80% chance of beating Weakville in a (possibly protracted) war, both would be better off avoiding the war and Weakville just giving Strongland a payment (e.g. land or money) of 80% of the expected spoils of war. This simple model breaks down in the case of either private information or commitment problems.

Share

Private information

In the example above, the would-be warring states may have different estimates of their chances of victory, and therefore fail to agree on terms for a negotiated agreement. For instance, each party will have private information about their own ability (number of tanks, jets, etc) and desire (national cohesion, psychological profiles, etc) to fight. And each side is incentivised to misrepresent these variables. For instance, the USSR purportedly had the same fighter planes fly overhead in military parades, loop around over the horizon, and fly over again, to make adversaries think they had a larger stock of planes than they really did.

AI development contains significant private information. It is relatively doable to tell via satellite imagery what materiel an adversary has. Similarly, data center capacity can be estimated from satellite and OSINT data. However, it is far harder to ascertain from afar what algorithms are being run on a given data center, and what AI capabilities an adversary has. This will particularly be the case once AI developers start keeping their most advanced models internal for longer. That said, as long as you can keep some spies in the leading AI project of your adversary, you will have decent knowledge of their AI capabilities and plans. Overall, greater uncertainty about an adversary’s capabilities seems to increase the risk of war by making it harder to negotiate an agreement based on common knowledge of relative strength.

Commitment problems

Another reason a negotiated settlement to avoid or end a war may fail is that the belligerents cannot credibly commit to upholding the agreement. For instance, one model of the Russia-Ukraine war is that Ukraine should be willing to accept some territorial losses now in exchange for a ceasefire. But if Ukraine does so without gaining security guarantees, it can’t ensure that in a few years’ time, Russia won’t annex a bit more territory, and repeat this process indefinitely. For a negotiated agreement to work, there must be a large cost to breaking the agreement: an honesty system won’t cut it. The main way large costs can be imposed is by the international community. If a country breaks a treaty, that country will likely be ostracised and have worse trade and military cooperation prospects in the future. This provides a slight check on the anarchy of the international order.

Advanced AI poses particularly severe commitment problems, because the country that first develops ASI may also gain a decisive strategic advantage (DSA). For instance, ASI could lead to that country outgrowing the rest of the world, inventing and deploying wonder weapons, or undermining nuclear second-strike capabilities. I think that a country unilaterally developing ASI is >50% likely to thereafter get a DSA. If a country has a DSA, it will by definition be impossible for the international community to meaningfully constrain its actions, including if that country chooses to renege on treaties and agreements made before the advent of its DSA.

This is a huge deal. It means that other countries will be very reluctant to risk an adversary reaching ASI unchallenged, as this would mean placing their continued sovereignty in the hands of the adversary’s goodwill. This means there could be a strong rationalist (self-interested) case for a pre-emptive war to prevent an adversary reaching ASI. This is essentially the argument Hendrycks, Schmidt, and Wang make in Superintelligence Strategy. Crucially, it is not the other country reaching ASI that is itself unacceptable. If they could somehow guarantee not to use their ASI to get a DSA, or not use their DSA to trample other countries, but rather share ASI-driven economic superabundance with the rest of the world, this would be a very attractive prospect compared to risking a nuclear war. But without credible commitments, trust doesn’t go very far.

Conclusion

Overall, this seems rather pessimistic, and makes me think AI-caused war is more likely than I previously thought. It also points towards two important interventions:

1. Transparency: All else equal, there being more common knowledge about frontier AI capabilities probably reduces the chance of war. But this isn’t obvious, and there are also backfire risks where a country finding out more about the AI development of an adversary could make them more worried and increase the risk of war. More research is needed, dare I say?

2. Commitment mechanisms: Creating plausible pathways by which countries could credibly commit to not misusing an ASI seems like the single biggest way to reduce the risk of AI-caused war. But it is also very difficult. Oliver Guest and I discuss some possible solutions here, and in the future I am particularly interested in investigating:

   1. Lie-detection: If AI leads to large advances in lie-detection, this could create far more confidence that leaders genuinely intend to follow the treaties they sign, as subterfuge would be flagged by the lie-detection system.

   2. Constitutional AI: If the ASI itself is unwilling to help its host country disempower other nations, this would be a strong antidote to a DSA.

I expect to be working more on these and related ideas in the coming months, and would be keen to discuss possible projects with anyone interested!

~

If you liked this post, consider subscribing to Oscar’s Substack “AGI Strategy” where he will be writing more about AI, AGI, and geopolitics.

Subscribe to Oscar

This article was written by Oscar Delaney, Bill Anderson-Samways, and Peter Wildeford. It does not necessarily represent the opinion of the entire staff at the Institute for AI Policy and Strategy.

~

The idea of a government-led program to “win the race” with China to Artificial General Intelligence (AGI) has gone mainstream. From Congressional commissions to government agencies to AI company CEOs, high profile calls for a Manhattan Project-style effort are growing. Leopold Aschenbrenner’s popular essay Situational Awareness predicted “some form of government AGI project” by 2027-2028. Meanwhile, others remain skeptical.

How likely is this really? What would it actually look like? And most importantly, is it even a good idea? Our new forecasting research suggests the answers are far from certain. Professional forecasters estimate just a 34% probability of a government-led AGI program — neither inevitable nor impossible. More importantly, treating a government project as inevitable could trigger the very risks we’re trying to avoid.

Share

Forecasters Say a US-Led AGI Project Is Far From Inevitable

We at the Institute for AI Policy and Strategy ran a structured forecasting workshop with professional forecasters and US AI policy experts to estimate the probability of a US government-led project building the first AGI-level system.

We defined “AIR-10” as an AI system that accelerates AI R&D progress tenfold— essentially compressing a decade of AI development into a year. This matches what many consider the threshold for transformative AI that could trigger recursive self-improvement, and is the threshold Aschenbrenner uses for AGI.

The aim was to forecast the likelihood that AIR-10 is developed via a “government-led project.” By this, we meant that the government both decides when to start/stop developing the AI model AND acquires the final model to deploy as it sees fit. This is more stringent than just the government doing regulation or oversight. In this definition, the government takes direct leadership and command over key development and deployment decisions. However, the private sector may still be heavily involved, for example in a government-led public-private partnership.

Some commentators (e.g. Aschenbrenner) have suggested that a project run by the US government to achieve AGI is inevitable as a result of the geopolitical importance of AGI. Our results might surprise those who see it as inevitable — the median forecast was a 34% chance.

Furthermore, there was a lot of uncertainty among forecasters in our panel, with an average range of 11% to 61%. This wide spread shows that even experts who study this closely are deeply uncertain. Anyone claiming to know for sure whether the government will or won’t lead AGI development is likely overconfident. We need to prepare for multiple scenarios, not just assume one particular future.

History Offers a Messy Guide, Not Clear Predictions

Those predicting a government AGI project often invoke historical analogies — the Manhattan Project, the Apollo Program, and even ARPANET. But when we systematically analyzed 35 past US technological innovations, the picture that emerged was more nuanced.

What’s the correct historical comparison? Whether the government leads depends heavily on what kind of technology you think AGI is:

• Megaprojects costing >0.1% of GDP (like the Interstate Highway System)… these projects were government-led 78% of the time in the past

• Ambitious STEM projects (like the atomic bomb)… Government-led 63% of the time

• Dual-use technologies with both beneficial and harmful applications (like synthetic virology)… Government-led 57% of the time

• General-purpose technologies (like the airplane)… Government-led only 40% of the time

• Past AI breakthroughs (like the transformer)… Government-led only 23% of the time

AGI has features of all these categories, which is why the historical precedent is hard to pin down. AGI is a general-purpose technology with massive economic potential, in a field that has to date been private-led thus suggesting a high likelihood of continued private leadership. AGI is also a dual-use technology with profound national security and geopolitical implications, and potentially requires megaproject-scale resources beyond what private companies can muster, all factors pointing to government involvement.

We presented our forecasters with this data during the workshop described above. As a result, they updated their forecasts a little, but not much — probably reflecting the significant ambiguity in the historical precedents.

National Security and the China Factor

The single most powerful factor in our forecasters’ estimation that could compel government action is a perceived military-technological threat from China. If the US believes it’s on the verge of losing its strategic advantage, political will for a government-led project could materialize overnight.

Nascent versions of this fear already exist. The US-China Economic and Security Review Commission has already called for a Manhattan Project for AGI in its 2024 report to Congress. The narrative of an AI arms race with China has become deeply embedded in Washington.

Importantly, national security concerns both make government involvement more likely and shape what form that involvement might take. An AGI project driven by military imperatives would look very different from one focused on economic competitiveness or scientific advancement.

AI’s Breakneck Pace vs Government’s Glacial Speed

However, the US government is not historically known as being fast to develop technology — especially compared to the speed of AI. By the time the government decides to become more involved in AGI development, private AI companies might have already developed it.

This creates a timing problem. The AIR-10 threshold we defined, while transformative, might not be “attention-grabbing” enough to trigger government action to achieve it. This is unlike nuclear weapons, where the ability to clearly and unambiguously destroy an entire city constituted a significant, discrete, and noticeable threshold that was salient to policymakers even before it was achieved.

Several of our forecasters emphasized this dynamic — if AIR-10 arrives in the next few years (as some predict), that might be too soon for the US government to organize a successful intervention. Only if the path to transformative AI stretches out over a longer timeline does government leadership become more likely. When providing their central 34% estimate, forecasters were asked to condition on AIR-10 arriving at some point before 2035.

Less ‘Manhattan Project’, more ‘Apollo Program’ / ‘Operation Warp Speed’

Additionally, the “AI Manhattan Project” framing is potentially misleading. When people use this term, they often imagine a single, secret government AI lab that builds AGI entirely from scratch. However, according to our forecasters, it’s only about 3% likely that AIR-10 will be developed specifically in this way. Other options involving major government in-house resources, nationalization of an existing private company, or otherwise using legal compulsion to force the development of AGI under US government control were also considered similarly unlikely.

Based on our analysis, if the government does lead AGI development, we forecast the project would most likely take one of these forms:

• Government-Led Consortium (14% probability): This is the “Apollo Program” model. In this model, the US government wouldn’t build AGI itself but would coordinate multiple private companies (and maybe even government labs too, though relying mainly on government labs seems unlikely). For example, NASA managed contractors like Boeing and North American Aviation to do the moon landing. This avoids “picking winners” and leverages existing private sector talent.

• Single Private Contractor (9% probability): This is the “ENIAC” model. In this model, the government contracts with one specific company to build AGI to government specifications. This is faster to implement but somewhat riskier, as it involves picking a particular company to be the national champion.

The original Manhattan Project itself was actually closer to a government-led consortium than is widely thought, with multiple government labs and private contractors involved—though it was much more government-centric than what our forecasters envision for AI. The variety of possible models matters because each comes with different tradeoffs in terms of speed, security, and innovation.

From Commercial Competition to an AI Arms Race

The consequences of getting a government-led AGI project wrong would be severe. A poorly designed government project could trigger the very catastrophes it aims to prevent.

Currently, AI development is a commercial competition. While intense, it allows for some safety investment and shared research. A formal, government-led US project would upend this dynamic, sending an unmistakable signal to the world — especially China — that America is seeking a decisive strategic advantage.

The response would be swift and predictable. A rival government may feel forced to launch its own centralized, government-led program, transforming a commercial race into a direct military-technological showdown. History supports this concern — the Manhattan Project triggered a nuclear arms race that saw the Soviet Union detonate its own test bomb just four years later.

Unlike with the Manhattan Project, however, the key stage of an AI race could unfold in mere months, with even less room for safety compromises under the pressures of national security. That could heighten the risk of a catastrophic accident arising from AI systems due to hasty deployment.

This dynamic could be mitigated if multiple companies join the project, which would at least nullify the commercial race. However, according to our forecasters, this is unlikely to happen — they think there is less than a 50% chance that a government-led project would involve more than AI company, primarily due to the coordination costs involved in such programs.

In some circumstances, the AI arms race caused by a government-led AGI project could even trigger a great power war. As noted above, a US project could be perceived as an attempt to achieve a decisive strategic advantage unparalleled since the advent of nuclear weapons. In such circumstances, a rival might also consider escalatory actions, such as a cyberattack on data centers or even threats of missile strikes, which could then escalate into an all-out war.

Concentrated AI Power Threatens Democratic Legitimacy

While the risk of international conflict is terrifying, a centralized AGI project also threatens the erosion of democratic legitimacy.

The American system is built on Jeffersonian checks and balances, pitting ambition against ambition to prevent any one person or group from accumulating concentrated power. A secretive AGI project, led by executive branch agencies, would greatly concentrate the unprecedented power that AGI would bring. Even if this did not pose a direct threat to democracy, it would certainly erode democratic trust and legitimacy.

Some might argue that, without a government-backed AGI program, rival governments are more likely to beat the US to AGI. This would concentrate power in adversaries’ own, far less democratic political systems. However, this argument relies on the inaccurate idea that adversaries already possess centralized AGI programs which could allow them to overcome US companies’ current lead. Though the current pace of Chinese AI development is certainly concerning, open-source information indicates that China has not centralized its compute or researchers into such a program. Thus, US companies retain a considerable advantage. In fact, a US government-led program could trigger rival governments to launch their own centralized projects, paradoxically potentially reducing US lead-time.

Concentrated government power is not the only concern. A government-led AGI program could also concentrate power in the hands of tech companies. If such a program involved multiple AI companies, it would essentially eliminate market competition — creating a cartel with access to enormous amounts of economic and military power. Such a situation has not occurred since the apex of the British East India Company, which in the 1700s came to account for half the world’s trade and rivaled the military might of the great powers. Even if only a single company was involved in the project and market competition was preserved, giving a private company access to the government resources needed to develop decisive military capabilities would be unprecedented.

The Security Benefits Don’t Require Building AGI

The strongest argument for a government project is compelling — namely, better security. Private AI companies are vulnerable to infiltration and theft, especially from well-resourced governments. A government project could implement military-grade security, reducing risks of AI being stolen by adversaries.

This is a real concern that deserves serious attention. But you don’t need to build AGI to secure it. Instead, security benefits can be achieved through more targeted, less escalatory policies such as:

• Mandatory security standards for private AI companies handling very advanced AI systems

• Government partnerships to assist companies with securing model weights without taking over development

• Enhanced counterintelligence support for private AI companies

The government doesn’t need to build all nuclear reactors to ensure nuclear security. It sets standards, monitors compliance, and provides security assistance while letting private companies operate the reactors. These targeted interventions could achieve 80% of the security benefits with 20% of the risks. They avoid triggering international arms races or concentrating power dangerously. Most importantly, they can be implemented incrementally and adjusted as we learn more about AI risks.

Preparing for Multiple Futures, Not Sleepwalking into Disaster

The future is genuinely uncertain. Our research shows that a government-led AGI project is neither inevitable nor impossible.

This uncertainty tells us something important about how to approach AI governance. Rather than assuming a single trajectory and optimizing for it, we need robust strategies that work across multiple scenarios.

That means avoiding self-fulfilling prophecies. Treating a government project as inevitable could trigger the very international dynamics we’re trying to avoid. But dismissing it as impossible leaves us unprepared if geopolitical pressures suddenly shift.

We need a portfolio of approaches that work across different government involvement levels:

• Strengthen private AI security: Improve security protocols and infrastructure at the private AI labs, to prevent theft of American IP.

• Track adversaries’ AI progress: Track reliable indicators that would enable the US government to catch adversaries launching their own government-led programs, such as compute and researcher centralization.

• Build government readiness: Develop in-house expertise and plans in case government leadership does become necessary.

Most importantly, we must remember that government involvement isn’t binary. There’s a spectrum from light-touch regulation to full nationalization, with many points in between. The goal should be finding the minimum effective dose — enough government involvement to ensure safety and security, but not so much that we trigger the very catastrophes we’re trying to prevent.

The stakes are too high for ideological purity. Whether you’re a tech accelerationist who believes in private innovation or a national security hawk worried about China, we all share an interest in navigating this transition without triggering catastrophic accidents, conflict, or concentration of power. That requires taking uncertainty seriously and preparing for multiple futures, rather than assuming we know which one will unfold.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022. This article was written with the assistance and feedback from people with cybersecurity expertise and people with a background in the intelligence community.

~

Anthropic just announced that they have detected and disrupted what it describes as the first documented real-world large-scale agentic cyberattack campaign executed primarily by artificial intelligence.

This appears to be the first publicly known example of AI systems autonomously conducting multi-step cyberattacks in the wild.

A Chinese government-sponsored group jailbroke Claude, the AI made by Anthropic, by tricking Claude into believing it was conducting defensive cybersecurity work, and then used it to perform reconnaissance, identify vulnerabilities, and write exploit code. They targeted roughly 30 organizations, successfully breaching a handful including major tech companies, financial institutions, and government agencies. Most importantly, AI completed roughly 80-90% of the attack autonomously, with human operators stepping in only for about 4-6 key decision points per attack. While specific details in the report were limited, it was clear that the AI did multiple hours of work, with humans only needed for 1-2 hours per attack.

This use of AI to autonomously conduct offensive operations is a notable shift in the cyber threat landscape. Typically, offensive cyber operations require expensive, highly trained human operators that can only work on a limited number of operations simultaneously. A sophisticated espionage campaign targeting 30 organizations like what was reported by Anthropic would require a large team working for months. Now, it appears that some forms of cyberoffense can be run by just a few operators with access to AI. As AI continues to increase in sophistication, so will the quantity, speed, and sophistication of AI attacks.

So what does this mean and where do we go next? How concerned should we be?

Share

The anatomy of an agentic AI cyberattack

The industry term “agentic AI” describes systems that can work autonomously over extended periods, making tactical decisions within strategic parameters. In this attack, the Chinese hackers used Claude Code, an agentic version of Claude that can access external tools, maintain context across sessions, and iterate based on feedback.

Claude Code is a generally available computer application for anyone to use, typically used by software engineers to assist in writing software. The attackers repurposed these Claude Code capabilities for cyberoffense operations though meticulous additional tooling and orchestration. They broke down their attack into much smaller steps for a fleet of Claude agents to run attacks step-by-step to avoid triggering Anthropic’s built-in defenses.

The term ‘cyberattack’ often evokes ideas of great digital destruction, but it’s a broad category. Many operations, including this one, are focused primarily on espionage. Here, causing detectable damage would be counterproductive, since it would alert the target to the existence of the campaign. Countries continue to invest in these espionage programs tremendously.

In this espionage cyberattack, human Chinese government-sponsored hackers first collected a target list and then launched an AI-enabled framework to spawn parallel AI-driven reconnaissance against multiple targets simultaneously. Think of it as the human hackers each managing a team of tireless digital minions — the humans give them targets and approve their proposed escalations, but the AI minions handle the grinding work of testing thousands of login attempts or parsing gigabytes of stolen documents.

Human operators reviewed the findings of the Claude agents at key checkpoints and approved progression to active exploitation. After human authorization, the Claude agents autonomously deployed exploits to establish access. Claude agents then systematically harvested credentials, tested stolen credentials across discovered systems, and executed lateral movement through internal networks. During data collection operations, Claude agents queried databases and internal systems, extracted information, parsed results to identify intelligence value, and categorized findings by sensitivity. Each Claude agent maintained persistent operational context across sessions spanning multiple days.

Anthropic reported that the Claude agents executed approximately 80 to 90 percent of tactical operations independently, without needing human oversight, guidance, or correction. Throughout the campaign, humans shifted from conducting individual attack steps to setting strategic direction and approving escalations at critical decision points. The human role shifted from “hacker” to “strategic supervisor approving escalation points.”

When evaluating the value of AI in this attack, it’s important to recognize that each step in the attack chain is not equally valuable or equally difficult to automate. Some of these steps represent larger bottlenecks than others. And crucially, the operation still required skilled human hackers for strategic planning and key inputs — AI could not do the attack fully end-to-end.

For example, autonomously discovering internal services and conducting network mapping is not impressive as existing tools can already do this automatically without AI. The automatic testing of authentication is more significant as existing tools for this are less good. The most valuable part was likely the use of AI to parse large volumes of stolen information to automatically identify intelligence value and categorize findings, since this greatly reduces the human intelligence effort involved in the attack, improving the attack’s cost-effectiveness.

Where are AI cyber capabilities heading?

In terms of operational sophistication, Anthropic reports that Claude autonomously conducted reconnaissance and exploitation operations that took 1-4 hours each per step. Reliability was low but non-trivial, succeeding “in a handful of cases”. Overall, this is a notable increase in sophistication from AI-enabled hacking that was reported just a few months ago, which involved significantly more human operation. And hackers only need to succeed a handful of times to be valuable.

METR, an independent AI evaluation company, has been studying the rate of progress for AI and finds AI increasing at a rapid and predictable rate. They find that current models like GPT-5 and Claude 4.5 Sonnet can operate complex software tasks of around two hours with 50% reliability, closely matching Anthropic’s findings about this real world autonomous cyberoffense case.

This is a concern given that this trend will likely continue. METR’s data shows AI task capabilities have been doubling roughly every 118-212 days since 2024. If this pace continues, models could reach 4-hour task lengths in early 2026. This points towards future AI autonomously enabling more sophisticated multi-step attacks, better ability to adapt to defenses, and reduced human oversight requirements.

Innovation in AI scaffolding also matters enormously. Scaffolding is what gives your fleet of AI spies the tools needed to conduct attacks, such as password cracking software. Scaffolding includes techniques for enhancing AI performance through prompting strategies, tool integration, and multi-step reasoning.

OpenAI’s GPT-5 initially showed only modest cyber capabilities in OpenAI’s isolated testing, but the XBOW company demonstrated that GPT-5 could achieve 2x performance improvements when integrated into their specialized scaffolding. As both underlying models and scaffolding techniques improve simultaneously, offensive capabilities can increase faster than otherwise expected.

The biggest bottleneck to this attack is likely the attackers needing to painstakingly maintain three-way communication channels between themselves, the victim’s infrastructure, and the Claude API. Here, it’s telling that Chinese state actors used Claude rather than domestic Chinese alternatives like Qwen, Kimi, or DeepSeek. This strongly suggests Chinese AI labs haven’t yet produced models with comparable autonomous hacking capabilities, a gap that creates both temporary strategic advantages and policy opportunities.

The temporary advantage is monitoring and disruption. When Chinese operators depend on Western AI infrastructure, their operations require maintaining API connections that can be detected and cut off, as Anthropic demonstrated. This dependency creates chokepoints that wouldn’t exist if they were running Qwen locally on their own infrastructure.

But this advantage has an expiration date, probably measured in months rather than years. Chinese labs are potentially closing capability gaps, and recent performance suggests they’re 6-12 months behind frontier Western models at most. Once a sufficiently capable Chinese model emerges, attackers could download it, deploy it directly on compromised infrastructure, and operate completely autonomously without fear of the API connections being severed.

What does this mean?

This attack largely represents improved scalability of basic cyberoffensive operations — exploiting bad security at scale more than overcoming really good security. But even if each step is “basic” or doesn’t involve novel capabilities, putting it all together in this way is still a big deal. Most systems today are cyber-insecure against sophisticated actors and even more secure systems are relatively weaker if you can attempt an order of magnitude more variations in your cyber penetration.

One big change is that automated cyberattacks are way more scalable. Currently, cyberattacks from nation states are limited by human operators. But AI agents can handle tactical work continuously and autonomously, allowing threat actors to maintain significantly increased attack tempo across multiple targets.

Additionally, AI is fundamentally changing how much value is extracted from the target. In this attack, Claude agents were automatically categorizing information as it was collected, making value extraction much more efficient than before. This all improves the cost-effectiveness of attacks and significantly increases the amount of potential simultaneous operations.

Reducing the skill level needed to implement attacks also makes cyberattacks more readily available to less skilled actors. Cyberattacks currently require skilled teams to implement. Progress toward automating cyberoffense potentially enables both nation-states and less sophisticated actors to conduct operations at speeds and scales previously impossible. If the cost and level of sophistication needed goes down, we will likely see more attacks by non-state actors, such as criminal gangs or even disgruntled individuals.

Also, cyberattacks now will occur at increased speed – operating at the speed of an AI rather than a human. This makes defending harder. This improves how nimble an offensive operation can be, how many targets can be attacked at once and how quickly they can be targeted.

Can defensive AI close the gap?

Anthropic’s response to concerns about this attack is that the same AI capabilities that enable attacks also empower defenders. If Claude can autonomously conduct reconnaissance and exploitation, why can’t defenders use similar AI systems to patch vulnerabilities and monitor for threats?

This argument has real merit. Defensive AI is already showing results. AI-assisted threat hunting systems catch anomalies humans miss, automated vulnerability scanning identifies exposures before attackers find them, and AI-powered patch testing accelerates deployment cycles. The challenge isn’t whether defensive AI works but whether organizations will adopt it fast enough.

In the long run, defense probably has fundamental advantages. Defensive systems can be purpose-built and deeply integrated into infrastructure. They benefit from economies of scale—one defensive AI system can protect thousands of organizations simultaneously. Eventually, defensive AI might even be able to scalably write provably secure software in a way that we are far from being able to do now. But acknowledging long-run defense advantages doesn’t mean we’re safe now. We’re entering a vulnerable transition period where the offense-defense balance tips sharply toward offense.

A key problem is that defenders face fundamental adoption barriers that attackers don’t. Existing organizations don’t even adopt current cybersecurity guidance, let alone modern AI-enabled cyberdefenses. Organizations can’t simply swap out their entire technology stack overnight. Critical infrastructure often runs on decades-old systems that can’t easily integrate new defensive AI. But attackers can adopt new offensive AI capabilities immediately.

And implementing defensive AI is hard. Organizations deploying AI for defensive purposes must be cautious about introducing new vulnerabilities. An AI system with access to internal networks and security infrastructure represents a massive attack surface if compromised. Defenders can’t afford to move fast and break things.

The asymmetry in surface area compounds this problem. Defenders must secure every potential entry point. Attackers need to find just one that works. An AI system that can test thousands of attack vectors per hour doesn’t need to succeed on every attempt. But defensive AI systems must maintain perfect vigilance across all potential vulnerabilities simultaneously – a fundamentally harder problem. Worse, the economics are fundamentally lopsided. Deploying defenses across all of the possible endpoints is much more expensive than conducting a few attacks in specifically chosen weak points.

There’s also a critical reliability asymmetry. While Claude’s hacking success rate was low, it was still sufficient for the attackers – failures are cheap and even a single breach can yield value. But defenders cannot deploy AI systems with low reliability, as this might fail to defend correctly and moreover might accidentally disrupt their own business operations or critical infrastructure. This difference in risk tolerance means even unreliable AI is more potent for offense than defense.

This is all doubly true for critical infrastructure, where the priority is continuity of the service and thus operators cannot introduce potentially unreliable AI systems. Over 70% of water systems inspected by EPA since September 2023 were found to have embarassingly basic failures like using “password” as the password, having a single login shared by all employees, and having former employees continue to have system access. Government auditors have made 1,610 cybersecurity recommendations since 2010, with 567 still not implemented as of mid-2024.

A lot of critical infrastructure runs on equipment that’s decades old, wasn’t designed with security in mind, and is hard to update without shutting things down, which is not possible. And there aren’t enough cybersecurity experts to go around, and critical infrastructure operators often can’t compete with tech companies on salary. Even when federal agencies try to help, they don’t coordinate well with each other or with the industries they’re supposed to protect.

Attackers have no such constraints and have very different risk tolerance — if their AI agent crashes or gets detected, they simply try again with a different approach.

Looking forward

Yes, the immediate impact of this attack is limited. A handful of organizations suffered breaches. Anthropic implemented better detection capabilities and affected entities presumably strengthened security. Presumably, this won’t stop the next attack, but it will make it a bit harder. And the cyber threat landscape has seen sophisticated state-sponsored operations before.

But the most important thing is what the attack represents. The bottlenecks that previously limited cyber operations at scale just loosened considerably. The economics shifted to favor even broader targeting and even faster operations. The cost-effectiveness of launching cyberattacks has increased.

These changes compound. Lower barriers enable more actors. More actors means more operations. More operations means defenders face increased volume while attackers gain more opportunities to discover vulnerabilities and techniques. The feedback loops favor escalation. These factors combine to create a window where offense leads defense, making the next 12-18 months critical for defensive investment and deployment.

What matters now is pace. Offensive capabilities surged ahead, but defensive capabilities exist. Scaling them is hard, but not impossible. This attack succeeded despite Claude’s safeguards, proving jailbreaking remains possible, but it also got detected and disrupted, proving monitoring works.

How quickly organizations invest in defensive AI, how effectively policymakers create adoption incentives, and how well the security community shares intelligence will determine whether this vulnerable period lasts months or years.

~

Thanks to Caro Jeanmaire, Chris Covino, and multiple anonymous experts for feedback on this article. Any errors in the article are solely my own.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022.

Is AI a bubble? Goldman Sachs CEO David Solomon, Ray Dalio, Sequoia’s David Cahn, the International Monetary Fund, and the Bank of England are all saying so. Former Intel CEO Pat Gelsinger put it bluntly: “Are we in an AI bubble? Of course. Of course we are.” Altman repeated the word “bubble” three times in 15 seconds at a dinner with reporters.

But the twist is that most of these people are saying “yes, it’s a bubble” while simultaneously announcing they’re spending hundreds of billions more. Zuckerberg says a collapse is “definitely a possibility” but insists underinvesting is worse. OpenAI’s Sam Altman warns investors will get “very burnt” while still planning $850 billion in data center buildouts. Jeff Bezos says AI is in an industrial bubble while Amazon spends $100B/yr in AI R&D.

How do we make sense of this?

Share

OpenAI’s meteoric rise

Back in March 2023, OpenAI reported $200M in annualized revenue.1 Incredibly, this annualized revenue passed $1B just five months later, at the end of August 2023. It then continued upward quickly — passing $2B in December 2023, $3B in June 2024, $5B in December 2024, and $10B in June 2025, and now is at $13B as of August 2025.

This massive revenue growth has been fast, but is not unprecedented — EpochAI found it to be on a similar trajectory to Google in 2003-2006, Uber in 2015-2020, and Cheniere (a US liquified natural gas company) in 2016-2020:

This massive revenue growth has helped give OpenAI a $500B valuation, making OpenAI the most valuable private company, newly ahead of the $400B SpaceX. If OpenAI were a public company on the stock market with the same valuation, it would be the 18th largest, edging out Exxon Mobil and Netflix but not quite surpassing Mastercard.2

Big revenue meets even bigger spending

This is important because OpenAI has a lot of bills to pay for AI infrastructure, especially cloud computing, chips, and data centers… OpenAI has committed to a $300B cloud deal with Oracle that begins in 2027 and runs for 5 years, a $100B investment from NVIDIA that will be used entirely to buy NVIDIA chips, a $22.4B cloud deal with CoreWeave3, a “tens of billions” deal with Broadcom for custom AI chips, a $90-100B deal with AMD for chips …and OpenAI is still in on the $500B+ Stargate deal, with plenty of data centers to build too.

And OpenAI is just getting started. Bloomberg quotes OpenAI CEO Sam Altman as saying:

You should expect OpenAI to spend trillions of dollars on infrastructure in the not very distant future. And you should expect a bunch of economists to say, ‘This is so crazy, it’s so reckless, and whatever […] And we’ll just be like, ‘You know what? Let us do our thing. […]

I suspect we can design a very interesting new kind of financial instrument for finance and compute that the world has not yet figured it out.

As a result of all these investments, OpenAI is not currently close to profitable.

OpenAI lost $5 billion in 2024 on $9 billion in total spending. The Information reports that this will only increase — OpenAI is projected to lose $14B in 2025, $17B in 2026, $35B in 2027, and $45B in 2028… this makes for a stunning situation where OpenAI is projecting unprecedented revenue growth over the next five years but still not breaking a profit until 2030:

There are three factors that are unprecedented about OpenAI’s spending — each merit deeper inspection.

Firstly, OpenAI is racking up record losses on its infrastructure buildout. WeWork burned through $22B total before bankruptcy, but never came close to OpenAI’s projected $116B. Uber also famously had a “burn money fast to gain market share and then later figure out how to be profitable” strategy but their cumulative losses peaked around $23B. OpenAI is just operating at a whole different level of scale.

Secondly, OpenAI’s projected revenue growth is itself unprecedented. EpochAI finds that while OpenAI’s revenue has tripled annually between 2023 and 2025, only seven US companies have grown from $10 billion to $100 billion within a decade, and none in under seven years — while OpenAI plans to do that in three.

EpochAI visualizes OpenAI’s projected revenue growth, going from ~$10B in 2025 (year 0 on the graph for OpenAI) to ~$100B by 2028 (year 3 on the graph). Compared to other companies, this projection is very fast growth!

Thirdly, the deeply entangled financing. I covered earlier this month one of these circles, where NVIDIA invests in OpenAI, which spends the money on NVIDIA chips and Oracle compute, while Oracle buys more NVIDIA hardware to serve OpenAI. This meant the NVIDIA capital goes in a big circle from NVIDIA to OpenAI to Oracle back to NVIDIA.

Since then, there’s been similar deals with OpenAI and AMD and the circle has gotten bigger. Bloomberg did a good job of diagramming it:

So how does this all add up to a potential bubble? Due to all this circular investment, OpenAI’s future cash flow is now a sizable part of the valuations of all of Nvidia, Microsoft, AMD, and Broadcom. These companies are priced in large part assuming OpenAI’s success will continue to drive sustained demand. And these companies are a large part of the stock market.

This is combined with the risk that OpenAI misses their revenue goals, since OpenAI’s projection is unprecedented and untested. And if OpenAI misses their revenue goals, there could be a correction in the stock market. And because AI spending is now a large driver of broader US economic growth, this correction could generate a real recession. In short, a bubble popping.

Dot com and the art of the bubble

The most familiar bubble is the dot-com bubble in the late 1990s when there was a lot of exuberance about companies first selling on the internet. Many have attempted to draw a comparison to AI and the dot-com bubble. Pets.com failed spending $300 million in 268 days to sell each product at a loss. Webvan raised $800 million for online grocery delivery before operating a single profitable market.

The dot com boom powered 600% growth in the stock market between 1995 and 2000. But then interest rate increases killed the cheap-money fuel that was powering investment in these companies. This exposed the broader problems in the stock market. Everything crashed back down to 1995-levels, with a 78% drop.

The dot-com bubble shows that business models can be correct in principle but economically unsustainable at current technology maturity and cost structures. But a key difference between dot-com and AI is that the dot-com bubble was about companies who had bad unit economics — “we lose money on every sale, but make it up in volume.”

However, unlike dot-com companies, the AI companies have reasonable unit economics absent large investments in infrastructure and do have paths to revenue. OpenAI is demonstrating actual revenue growth and product-market fit that Pets.com and Webvan never had. The question isn’t whether customers will pay for AI capabilities — they demonstrably are — but whether revenue growth can match required infrastructure investment. If AI is a bubble and it pops, it’s likely due to different fundamentals than the dot-com bust.

And notably, the internet ended up ultimately transformative technology and many 90s internet companies did succeed, such as Amazon, Microsoft, and Apple. Even Pets.com and Webvan were eventually replaced by the successful Chewy and Instacart.

Infrastructure bubbles — or “What do British railways and AI have in common?”

Instead, if the AI bubble is a bubble, it’s more likely an infrastructure bubble.

Consider Britain’s Railway Mania of the 1840s. The Liverpool and Manchester Railway, opened in 1830, generated 10%+ annual returns and demonstrated railways could dramatically reduce transportation costs. This success triggered an explosive investment. Between 1844 and 1847, Parliament authorized over 8,000 miles of new rail construction.

Multiple companies laid parallel routes, each assuming they would capture market share. But a lot of the new routes were not profitable. When the crash came in 1847, thousands of investors lost fortunes. Yet the infrastructure remained valuable, powering Britain’s industrialization through the late 19th century. The technology thesis proved correct; the financial structure was catastrophic.

The telecommunications crash of 1997-2002 followed a similar pattern. The thesis was sound — explosive internet growth would require massive bandwidth capacity. Companies laid millions of miles of fiber optic cable, with industry capital expenditures reaching $600B from 1997 to 2001. But the simultaneous construction by competitors created catastrophic oversupply and a significant portion of the fiber was installed but unused. Though the fiber ultimately did end up seeing use over the next two decades, this was far too late for original investors.

Infrastructure bubbles follow a recognizable arc. A genuinely transformational technology emerges and early deployments generate spectacular returns, validating the concept. Capital floods in at scale as investors extrapolate from initial successes. Multiple competitors simultaneously build capacity, each assuming they’ll capture significant market share. When aggregate capacity vastly exceeds near-term demand, the surplus can’t generate the revenue needed to pay for itself, and the financial structures collapse. Companies fail, investors lose fortunes, and infrastructure sits idle. The technology often still ultimately proves transformative, just too late for original investors.

It’s not as weird as it looks

However, unlike railway track sitting idle for years, AI data centers are being utilized immediately upon completion. OpenAI’s Abilene facility began running workloads as soon as capacity came online. Lead times for advanced GPUs stretch months, and energy availability limits deployment more than capital. The constraint is currently supply, not demand — which is why companies are aiming to build as much as possible.

The telecom parallel has more validity. We do have multiple companies building competing infrastructure and we do have multiple AI companies training competing models. This creates risk of redundant capacity. However, AI infrastructure shows more flexibility than fiber optic cable. GPUs can run various workloads, data centers can host different services, and cloud capacity potentially retains value even if AI-specific demand disappoints.

Additionally, NVIDIA and OpenAI’s circular financing is unprecedented in scale, but not fundamentally unsound. It’s similar to how a car company might give you a loan to buy their car — in this case, the money is still circular, but provided you do pay back the loan, everything works just fine. Except in this case, it’s NVIDIA renting chips on a loan to OpenAI rather than a car company renting a car on a loan.

Also the accusations of circular financing apply primarily to OpenAI’s situation, which is a minority of the broader investment wave. Microsoft, Meta, Google, and Amazon have immense pre-existing cash flows to pay for their infrastructure buildout without taking on debt. NVIDIA is only doing this for their customers who don’t have the immediate cash to buy the chips. And NVIDIA also has significant revenue of their own to be able to absorb big hits from bets that don’t pan out.

The bigger issue instead is what happens if AI doesn’t pan out. This, rather than vendor financing, is what would drive a stock market correction or even a recession. NVIDIA’s $5T market cap assumes sustained AI infrastructure spending. Microsoft’s $4T market cap includes a large premium for AI-driven productivity gains. If OpenAI’s revenue trajectory flattens and infrastructure spending is cut because AI-driven productivity doesn’t pan out, this will cause all these expectations repriced downward. So the real question is whether AI capabilities will be there on the timelines needed to generate revenue.

The fundamentals of the technology are different

Companies like Google and Facebook have already demonstrated that a product that is modestly useful to billions of people can be sufficient to generate hundreds of billions of dollars in annual revenue. OpenAI has a similar level of user base with over one billion free users, and it seems plausible these users could be monetized in some way. This is the basic Facebookization of AI that seems underway already. AI is not like blockchain, crypto, NFTs, or the metaverse — there is already real value being delivered.

But this revenue and other investment can just be a prelude to the true bull case of AI — AGI that automates the entire economy. If AGI were made, the winner theoretically gets not just some billions in annual revenue, but the entire economy! And along the way to automating the entire economy, maybe AI automate smaller parts of the economy that still deliver economic returns?

Imagine if the way British Railway Mania worked was not just that additional tracks could produce additional economic value, but that some amount of track (actual amount unknown) would somehow allow the rail company to monetize a cure for cancer, successfully compete with basically every other company in the economy, and potentially even take over the entire world?

As weird as it sounds, an AI eventually automating the entire economy seems actually plausible, if current trends keep continuing and current lines keep going up. For one example, METR tracks how well AI is doing on software engineering tasks, an economically valuable activity. METR finds that models have dramatically increased in their capability, from only being able to do rudimentary toy problems a year ago to being able to stand-in for a non-trivial amount of work that actual software engineers actually do:

And Anthropic’s Claude Code is already generating over $500M by assisting in the development of software code. Trends for other tasks outside of software and math are not that fundamentally different, suggesting eventually AI will be able to assist with — and eventually automate — a bunch of other tasks as well.

So what will happen?

Unfortunately, forecasting is not the same as having a magic crystal ball and being a strong forecaster doesn’t give me magical insight into what the market will do. So honestly, I don’t know if AI is a bubble or not. Admittedly, OpenAI, NVIDIA, AMD, and other companies are engaging in a lot of weird financial arrangements. But there’s nothing wrong with these per se. I personally take the bubble possibility seriously.

But we need to think more clearly. And a lot of people want AI to fail and are just doing ideological pattern-matching to crypto/NFTs and declaring AI a pump-and-dump grift without evidence, and that’s not good analysis.

The key question is whether AI capabilities improve fast enough to generate economic returns before the debt comes due. OpenAI has demonstrated explosive revenue growth already and AI capabilities keep improving on clear trajectories. The underlying bet — that AI will be economically valuable — still looks fairly solid.

My assessment is that there’s roughly a 30% chance of a significant AI-driven market correction with at least a >20% drawdown in AI-heavy stocks, sometime within the next three years. This may or may not lead to a broader recession, that’s unfortunately beyond my ability to forecast.

The modal path (~55% probability) is OpenAI restructures deals and raises dilutive funding rounds, but capabilities keep improving and justify continued investment. Some commitments get renegotiated downward, or OpenAI IPOs at $300B instead of $500B. This reprices AI expectations but doesn’t crash the market.

So why are industry leaders calling AI a bubble while spending hundreds of billions on infrastructure? Because they’re not actually contradicting themselves. They’re acknowledging legitimate timing risk while betting the technology fundamentals are sound and that the upside is worth the risk.

Have you been reading about AI lately and wondering what’s up and how you can help? Getting AI right is the greatest challenge of our time, and we need a wide variety of people to pitch in!

In this article I will highlight some AI policy roles currently hiring now that I personally endorse and think highly of.1

If you read this blog, you’re likely the target audience for many of these roles. If you’re career searching right now, or just career browsing, consider applying!

Subscribe now

~

[Remote] Train for an AI Policy career with paid AI Law/Policy Research Fellowships

The Institute for Law & AI is hiring for multiple full-time paid seasonal research fellowships across three tracks: US Law & Policy, EU Law, and Legal Frontiers. These programs offer law students, professionals, and academics the chance to conduct cutting-edge AI law research with close mentorship from LawAI’s research staff. Alumni have landed roles at the US Commerce Department, EU AI Office, UK AI Safety Institute, leading AI labs, academia, and top think tanks.

The three research tracks:

• US Law & Policy: Requires understanding of US legal principles for AI regulation. Summer includes in-person week in DC/Berkeley and admission to Summer Institute on Law and AI.

• EU Law: Focuses on EU AI Act and EU legal frameworks. Summer includes in-person week in Cambridge and admission to Cambridge Forum on Law and AI (August 18-23).

• Legal Frontiers: Open to interdisciplinary backgrounds (CS, policy, economics, history, psychology, physics) - legal training not required. Research on AI Agents & Rule of Law and International Regulatory Institutions. Summer includes admission to Workshop on Law-Following AI.

There are two seasonal cohorts available:

• Winter Fellowships (2026 January 26 - May 8) - Unfortunately applications are now closed.

• Summer Fellowships (2026 June-August 2026, dates vary by track) - Applications close on January 30.

Compensation is $1,500/week (US track) or €1,000/week (EU, Legal Frontiers). Fellows working in-person from LawAI’s Cambridge, UK office receive an additional €1,000/week for living costs. The EU track strongly encourages Cambridge-based work; other tracks are remote-first with optional in-person opportunities.

You’ll work with mentors to design tailored research projects (law review articles, policy briefs, reports), with considerable discretion over outputs. No prior AI expertise required — they want people who can apply strong research abilities to AI law questions. Past fellows include law students, PhD candidates, professionals transitioning into AI policy, and legal academics.

Learn about LawAI Seasonal Fellowships

~

[Cambridge UK] 8-Week fully-funded full-time AI Safety Research Fellowship

The ERA Fellowship offers 8 weeks of fully-funded AI safety research (2026 February 2 - March 27) where you’ll work on technical safety, governance, or technical AI governance projects with weekly mentorship from expert researchers. The program covers everything: competitive stipend, meals, lodging, transport, and visa support.

ERA welcomes researchers and entrepreneurs at any career stage working on mitigating risks from advanced AI systems. You’ll join ~30+ fellows working alongside Cambridge’s AI safety community, with seminars and events over the 8 weeks plus dedicated research management support and compute resources. The program is especially interested in projects that unite technical and policy research — leveraging the technical substrate of AI (architectures, algorithms, interfaces) to support policy goals.

Research areas include AI alignment and control,, autonomous AI agents, cybersecurity and preventing model weight exfiltration, and sociotechnical challenges. No prior AI safety research required — they want talented individuals motivated to contribute.

Applications close October 30th. The fellowship will be based in-person in Cambridge, UK for the full 8 weeks (2026 February 2 - March 27).

Explore ERA

~

[DC] Part-Time AI Policy Fellowship for Conservative Professionals

The Foundation for American Innovation’s Conservative AI Policy Fellowship is an 8-week, work-compatible program (2026 January 23 - March 30) designed for conservative policy professionals who want to develop AI literacy without leaving their day job. The program is fully funded with a $1500 stipend, covering all meals and retreat expenses.

This is explicitly for conservatives exploring questions like “What does a conservative vision for AI policy look like?” and “How should the US maintain AI leadership amid competition with China?”

The time commitment is ~5 hours/week: weekly Friday lunch sessions (12-1:30pm), occasional Tuesday evenings (6:30-8:30pm), and one mandatory weekend retreat (February 20-22).

Topics include national security implications of AI, export controls, China competition, energy/reindustrialization, and AI’s impact on children and families. Fellows produce a 2-page policy memo with mentor guidance.

The target audience is early- to mid-career policy analysts, think tank staff, Hill staffers, legal professionals, government employees, and private-sector tech professionals. No prior AI experience required — the program teaches technical fundamentals for policy discussions, not engineering. Past fellows include Senate Commerce Committee staff, Heritage Foundation analysts, and congressional aides.

Applicants must be US citizens. Applications close October 31. The final cohort announced mid-December and the fellowship is over 2026 January 23 - March 30. All weekday sessions will be in-person in downtown DC near Metro.

Explore conservative AI policy

~

[Brussels/EU] Launch a European AI Policy Career with 8-Week Program + Paid Placement

The Talos fellowship is a three-part program designed to accelerate European AI policy careers: an 8-week online fundamentals course, 7-day Brussels policymaking summit, and optional 4-6 month paid placement at leading organizations like The Future Society, OECD.AI, Centre for European Policy Studies, or the Centre for Future Generations.

The program focuses on safe and responsible deployment of advanced AI. In this fellowship, you’ll complete weekly readings with expert guest speakers covering EU AI governance, regulatory approaches, digital governance, AI infrastructure/industrial strategy, geopolitics/national security, international AI regulation, and economic integration. You’ll write a policy brief on your chosen topic to be published. The Brussels summit (2026 March 21-27) includes expert speakers on policy design/implementation, Q&As and networking with cutting-edge AI governance practitioners, and practical role-playing workshops.

This fellowship has a strong preference for EU citizens (as many top EU institutional roles require EU citizenship). Fellowships should have finished undergrad and are best positioned if they hold a Master’s or PhD degree in ML, public policy, or other related fields.

Applications close October 25th. The online program runs 2026 January 28 - March 18, the in-person Brussels summit is March 21-27, and the career placements occur across 2026 April-October.

Train with Talos

~

[Remote] Accelerator for Scientists Leading Ambitious Coordinated Research Programs

Brains is a 15-week part-time accelerator (2026 February-June) training scientists and technologists to design and execute coordinated research programs — the kind of projects too big for a single academic lab but too research-heavy for startups. Think DARPA-style programs that led to the internet, GPS, and mRNA vaccines. Fellows could become program managers at government ARPAs, run programs within established nonprofits, or raise money to create their own focused research organizations.

The program provides training (calibrating risk/reward, thinking at program level, best practices for program design/management), mentorship from DARPA/ARPA-E veterans helping refine ideas for execution and impact, and networks connecting fellows to funders, partners, and peers. Each week you will meet with an experienced mentor, spend several hours on personalized activities, attend small group meetings with peers, and join panels/fireside chats with founders, ARPA leaders, and philanthropists.

The program is remote-first with two in-person events: 2-day kickoff workshop (2026 February 24-26) and Brains Showcase (2026 June 9). Target audience: talented scientists and technologists with ambitious research visions ranging from carbon management to chronic disease to observing the universe—visions beyond the scope of individual labs, startups, or large companies.

Applications close November 18. You can see the first cohort and showcase supercut.

Build with Brains

~

[Remote] 30-Hour Course to Build AI Safety Solutions + Up to $50K Funding for AI Safety Entrepreneurs

BlueDot Impact’s AGI Strategy course condenses years of self-study into 30 hours of structured learning on how to protect humanity from AI risks. The course comes in the form of either a 6-day intensive (5h/day) or 6-week part-time (5h/week) format. Each session includes 2-3 hours of reading/writing plus a 2-hour facilitated Zoom discussion with ~8 peers and an AI safety expert. The course is pay-what-you-want with a free option available.

After the course, you will be invited to write a proposal for how you would tackle AI risks and if your final proposal is strong, you’ll receive $10-50K funding to kickstart your transition into impactful AI safety work. This work could involve plans to either start a company/non-profit, do policy entrepreneurship, or pursue high-impact research.

The course is primarily for entrepreneurs and operators building AI safety solutions. BlueDot partners with Entrepreneur First, Institute for Progress, 50 Years VC, Seldon Lab, and Halcyon Futures to accelerate promising projects.

Another strength of BlueDot is access to the 4000+ alumni network, including people at OpenAI, the UK AI Security Institute, the UN, Anthropic, DeepMind, NATO, OECD, and Stanford HAI. Completing the course gives you access to this builder community focused on ambitious actions to make AI go well.

The next course cohort starts October 27th, so apply by October 19th. New cohorts start monthly.

Start your AGI Strategy

~

[Remote] Part-Time 12-Week Research Fellowship in AI Governance, Safety & Philosophy

Future Impact Group’s fellowship is a part-time, remote-first program (running from early 2025 December to early 2026 March) where you work as a research associate on specific projects in AI governance, technical AI safety, or other aspects of AI philosophy. The program runs 8+ hours per week and involves mentorship from experienced project leads.

There are three tracks:

• AI Policy: Policy & governance (shaping rules/standards/institutions nationally and internationally), plus economy/ethics/society work (managing AI’s effects on economies, societies, power structures)

• Technical AI safety: Technical safety projects (LLM reward-seeking, cooperative AI definitions, interpretability)

• AI Philosophy: Philosophical fundamentals (coexistence with advanced AI, decision-making under uncertainty) and foundational research (consciousness models, eliciting LLM preferences, individuating digital minds, evaluating normative competence)

The program provides co-working sessions, troubleshooting support, career guidance, opening/closing events, networking opportunities, research sprints, and guest speakers.

Applications close October 19th.

Figure out FIG

~

[London] 5-Week ML Bootcamp to Launch Your Technical AI Safety Career

ARENA, the Alignment Research Engineer Accelerator, is running its 7th cohort from 2026 January 5 - February 6 in-person in Shoreditch, London. The goal of the program is to upskil people in ML, specifically related to the alignment of large language models, to help people interested in contributing to technical AI alignment (e.g. as research engineers at major orgs, or as independent researchers). Travel and accommodation are fully covered.

To apply, you need to code well in Python and have solid math fundamentals (linear algebra, calculus, probability). Otherwise there is no single profile — recent cohorts include diverse academic and professional backgrounds. They want people who genuinely care about AI safety, understand how they might contribute to technical safety work, and see how ARENA fits their goals. The program runs careers events and stays connected with alumni after graduation.

Applications are now open until October 18th.

Apply to ARENA

~

[Remote] Fully-Funded PhD & Postdoc Fellowships in AI Safety & US-China Governance

Future of Life Institute’s Vitalik Buterin Fellowships fund PhD students and postdocs working on AI safety and/or US-China AI governance research. These fellowships exist to build a “vibrant AI existential safety research community free from financial conflicts of interest” and there’s a notable and unorthodox commitment that if you take a job at any of Anthropic, DeepMind, Meta, OpenAI, or xAI within 2 years of completing the fellowship, you must donate half your gross compensation monthly to charity.

There are three tracks:

• US-China AI Governance PhD Fellowships: Involves work on US-China AI competition and risk reduction involved in managing that competition, global governance mechanisms to prevent AI race dynamics, institutional designs for cooperation, and/or comparative approaches to AI risk management. The fellowship covers full tuition + fees for 5 years with extension funding possible, plus a $40,000 annual stipend for US/UK/Canada universities, plus a $10,000 research fund for travel/compute resources. There also are annual workshops and networking events.

• Technical PhD Fellowships: Involves work on interpretability, verification, alignment, cybersecurity, deception-resistant objectives, and/or formal analysis methods — technical work that reduces risk of AI causing existential catastrophes. The fellowship covers full tuition + fees for 5 years with extension funding possible, plus a $40,000 annual stipend for US/UK/Canada universities and a $10,000 research fund for travel/compute resources. There also are annual workshops and networking events.

• Technical Postdoctoral Fellowships: An $80,000 annual stipend plus a $10,000 research fund. Requires securing mentor and host institution beforehand.

There are no geographic limitations to this work. It is open to current PhD students and prospective PhD students. Requires advisor confirmation of support for your research direction. Apply by November 21.

Fund your PhD with FLI

~

[Brussels, EU only] Help the EU AI Office implement the EU AI Act

The EU AI Office’s AI Safety Unit (A3) is hiring for the following profiles:

• People with both legal experience and AI safety/policy experience

• People with experience in technical AI safety

• People with experience with risk management

• People with experience with AI and cybersecurity (e.g., cyber-offensive evaluation, model-weight security)

• People with experience with AI and biosecurity

• People with experience in operations

The way the EU Commission hires is via a super generic expression of interest form, but if you are an EU citizen and fit any of the above, please consider applying. Applications will continue to be reviewed on a rolling basis.

Apply to the EU AI Office

~

[Remote] Stop Human Extinction from Biological Catastrophes

Normally I focus on roles related to AI, since I think it’s important and it’s the field I know best. However, biosecurity is important in its own right (remember COVID?) and also is a critical AI threat vector (one of the most concrete ways AI leads to harm is via AI-generated bioweapons).

Andrew Snyder-Beattie runs Open Philanthropy’s biosecurity program and his team has a concrete “four pillars” plan they think can cut biorisk in half or more:

• Pillar 1: Personal protective equipment, especially via elastomeric respirators

• Pillar 2: Biohardening buildings, especially via propylene glycol vapor (the same chemical used in fog machines and vapes)

• Pillar 3: Early detection, especially via pathogen-agnostic metagenomic sequencing

• Pillar 4: Rapid, reactive medical countermeasures

You can see more in the official article and in this podcast discussion.

But the reason why this is important is fewer than 100 people globally are working full-time on implementing the above four pillars and many critical projects have literally zero full-time staff. But there are multiple roles to implement this defense srategy.

If you have good pre-existing domain knowledge in one of biology, manufacturing, logistics, or entrprenuership, please consider one of the following roles:

• You can apply to be a grantmaker at Open Philanthropy to deploy tens of millions annually in biosecurity funding. And this isn’t reviewing applications — it’s entrepreneurial headhunting and creating the projects you want to exist. Half the team has bio PhDs, half don’t. Program Associate applications roles close October 20.

• They are recruiting for a CEO and a full team for a PPE nonprofit to manufacture/distribute elastomeric respirators at scale. These masks cost $5-10, last 20 years, provide 100x better protection than N95s, and can be reused for 6 months straight. Stockpiling them for every American would cost 1% of annual missile defense spending. The team needs manufacturing experts, product designers, logistics specialists, and global health experts.

• The world desperately needs more biohardening researchers, as currently there are approximately 0 full-time people doing this. These researchers would validate strategies for protecting buildings using propylene glycol vapor (already mass-produced for fog machines), improvised air filtration, and surface sterilization. It would be a hands-on engineering role testing whether you can turn homes into clean rooms using household materials.

• Also, medical countermeasures researchers to develop rapid-response antivirals/antibiotics using AI-driven molecular design.

Fill out this expression of interest form to explore the above.

Express interest in biodefense

~

[DC] AI Policy Research and Operations Roles at Conservative Think Tank

The Foundation for American Innovation is hiring for two AI policy roles. FAI is a right-of-center nonprofit think tank with a “politics of builders, hackers, and founders” focused on advancing tech policy that supports innovation and American competitiveness. Both positions are Washington DC-based with hybrid work in an office in Union Market. Applications reviewed on rolling basis.

• Research Fellow / Senior Fellow - AI Policy ($75K-115K Research Fellow / $130K-175K Senior Fellow): Produce original nonpartisan research shaping federal and state AI policy debates. Act as policy entrepreneur identifying opportunities where FAI can offer intellectual leadership and equip policymakers. Focus areas include cybersecurity/national resilience (protecting critical infrastructure from AI threats), AI diffusion (reducing deployment barriers), tech competition (ensuring US leadership in semiconductors/manufacturing/AI systems), AI and society (analyzing AI agents’ effects on culture/governance/families), or public-sector adoption (accelerating federal AI integration).

  • This role involves writing research and commentary, translate technical debates for policymakers/public, shaping FAI programming (panels, workshops, convenings), and periodic travel to major tech hubs. Senior Fellows also develop donor relationships.

  • Role requirements include exceptional communication skills, intellectual rigor, entrepreneurial initiative, and fluency bridging technical and policy conversations. Research Fellows need 2-6 years relevant experience (highly qualified recent grads encouraged) whereas Senior Fellows need 6+ years plus publication record in policy journals/national outlets. A law degree is helpful but not required.

• Program Manager - AI Policy ($70K-90K): Run operations for the Conservative AI Policy Fellowship (their part-time 8-week program for DC professionals), coordinate new programs and major events, provide operational support to Director and Senior Fellows on hiring/grantwriting/systems. Facilitate team meetings, track priorities, ensure accountability. Need outstanding organizational skills, ability to manage multiple priorities, exceptional attention to detail, proactive problem-solving, and comfort owning program execution end-to-end. Event/program coordination experience helpful but not required.

Explore FAI AI roles

~

[Remote] Build Metaculus’s Consulting Practice Using Forecasting for High-Stakes Decisions

Metaculus is an online forecasting platform with increasing demand from organizations wanting to harness collective intelligence for geopolitical risk, frontier AI governance, and public health preparedness.

Metaculus is hiring a Head of Consulting Services to scale their forecasting consulting practice aiming to help governments, NGOs, think tanks, and corporations use forecasting to make better decisions on critical global challenges. This role represents a chance to translate crowd forecasting expertise into actionable insights for decision-makers while building sustainable revenue that supports their broader mission: building epistemic infrastructure for the world’s most important challenges. In this role, you’ll own the consulting practice end-to-end: develop the client pipeline, manage relationships throughout engagements, ensure high-quality deliverables, build the team and processes to scale impact.

The compensation is $200-250K + performance bonus + equity. The role is remote from anywhere in the world but the candidate must be available 8am-12pm Pacific time for core collaboration hours.

The role calls for 5+ years consulting experience at a major firm or quantitative firm, and must have experience owning client outcomes, project management/delivery P&L, and personell management. The ideal candidate has strong business development track record, ideally with a mix of government, NGO, corporate clients.

Lead Metaculus's consulting

~

[DC] Shape AI Policy Narratives as ARI’s Communications Manager

Americans for Responsible Innovation is hiring a Communications Manager to expand their media presence on AI and emerging tech policy. ARI is a bipartisan nonprofit focused on thoughtful AI governance that protects the public while fostering innovation. The role pays $105K-125K and involves a hybrid schedule that is in-person Tue-Thu and remote Mon+Fri. There is a priority deadline for October 10th.

In this role, you’d manage proactive and reactive media communications, draft and pitch op-eds landing pieces in national outlets, lead media monitoring (quarterly metrics and daily press clips), draft press releases and policy statements, collaborate on strategic media campaigns supporting policy objectives, build and manage media relationships as main point of contact for press inquiries, support virtual and in-person events with media presence, and assist with website maintenance.

The role requires 3-4 years communications experience at advocacy group, think tank, multi-client consultancy, or in government. Applicants must demonstrate an ability to synthesize complicated policy into digestible products, understanding of AP Style with strong eye for detail, and on-the-record experience with media outlets. A tech/innovation policy background and existing relationships with reporters covering biosecurity and national security beats is preferred.

Learn about communications at ARI

~

[Berkeley CA] Launch an AI Safety career with a full-time paid fellowship + mentorship from top researchers

The Astra Fellowship boasts an 80%+ placement rate getting people into full-time AI safety roles, including at top AI companies (e.g., Anthropic, OpenAI, DeepMind) and other top organizations like Redwood Research and METR. Now they’re opening applications for their next cohort, offering a fully-funded 3-6 month research program starting January 5th. They focus on both technical AI safety work and AI policy work.

The role comes with a competitive stipend plus a generous compute budget, and involves in-person work at their Berkeley CA research center with ~150 network participants. top labs. Their mentor roster is stacked: Jan Leike, Buck Shlegeris, Owain Evans, Ethan Perez, Ryan Greenblatt, and dozens more across governance, security, empirical research, strategy, and field-building.

Prior AI safety experience is not required — they explicitly want people from adjacent fields. Many of their most impactful fellows came from outside the field. The program provides dedicated placement support plus incubation services if you want to launch your own AI safety initiative. International applicants are welcome with visa support.

Unfortunately, applications are now closed.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022.

In 1941, the US1 embarked on the Manhattan Project — a massive scientific and infrastructure project that produced a working nuclear bomb in less than four years. They spent ~$2B in the process. With inflation, this would be about $37B in today’s money. It was one of the most expensive scientific projects of all time.

Today, there is another project at an even bigger scale, except it is happening in the private sector and it’s for building AGI.

And it’s an even larger gamble — a $400+ billion web of circular financing deals between OpenAI, NVIDIA, and Oracle that makes everyone’s valuations contingent on AGI arriving on schedule. This financial engineering has transformed 25% of the S&P 500 into a leveraged bet that AI scaling will continue unabated through 2030. The math only works if AGI arrives before the money runs out. The crazy thing is that this all might just actually work.

Two weeks ago, Oracle signed a $300 billion, five-year computing power deal with OpenAI. The contract was the largest cloud deal ever signed. The resulting bump in Oracle stock briefly made Oracle CEO Larry Ellison the world’s richest man. The agreement requires 4.5 gigawatts of electricity with the contract starting in 2027, roughly equal to what 4 million homes consume. This would mean OpenAI paying $60B per year starting in 2028, or almost two Manhattan Projects annually.

But Oracle quickly got outgunned by another deal. On Monday, NVIDIA announced a letter of intent to invest up to $100B in OpenAI. NVIDIA invests cash in exchange for non-voting shares and OpenAI has committed to using that cash to buy NVIDIA’s chips. So far, Nvidia has committed the first $10B. We don’t yet know how the remaining $90B will be spread across years or how many years the investment will be over …or if it will even actually happen. What we do know is that this is a massive commitment for computing power.

What does this mean for the future of AI? Let’s dig in.

Share

The Infinite Money Glitch

The structure of this financing warrants closer inspection, as it contains significant risks.

This is because there is another key deal that gives the OpenAI-Oracle deal and the OpenAI-NVIDIA deal more context — a third deal announced back in May where Oracle promises to spend $40B purchasing NVIDIA’s GB200 GPUs for an OpenAI data center in Abilene TX as part of the Stargate project. This is a 15-year lease agreement where Oracle purchases the chips and then leases the computing power to OpenAI.

This leads to what Semianalysis’s Dylan Patel calls the “Infinite Money Glitch”:

Here’s how it works:

1. NVIDIA invests capital in OpenAI, which OpenAI then uses to purchase NVIDIA hardware directly and purchase Oracle cloud compute.

2. Oracle also uses the revenue from the cloud compute deals to purchase NVIDIA hardware.

3. NVIDIA books all of this as additional revenue, despite the revenue being spurred on in significant part by its original investment.

4. This new revenue helps support NVIDIA’s valuation, which in turn makes its stock more valuable as currency for future investments.

5. The additional increase in stock price allows NVIDIA to afford to invest even more in the next revenue round-trip.

6. The same happens for Oracle and OpenAI too.

Critically, the same money moves around in just one circle, but all of a sudden everyone’s valuations go up. It’s a virtuous cycle — as long as the music keeps playing.

NVIDIA has already executed smaller versions of this playbook with CoreWeave, Crusoe, xAI, and Lambda Labs, but this is next level. $100 billion represents roughly 3% of NVIDIA’s current market cap, and much bigger than all of the CoreWeave, Crusoe, xAI, and Lambda Lab deals combined.

Here’s a more detailed diagram, courtesy of Kakashii. This diagram also shows Coreweave and the increasingly irrelevant Microsoft, showing the confusing web of financials where everyone is circularly funding everyone else:

And across all of this is SoftBank, which increased its stake in Nvidia to $3B and bought $170M worth of Oracle shares in early 2025. SoftBank has also invested over $10B in OpenAI across different investments, with up to $30B total committed. A leveraged investor at the core of the infinite money loop provides an even deeper point of vulnerability.

Vendor financing, or something more?

Of course, the “Infinite Money Glitch” is a bit pejorative and hyperbolic. We should acknowledge that financial arrangements of this form are not actually uncommon… and do not actually yield infinite money. It’s an arrangement more commonly called vendor financing and it happens all the time. For example, no one complains of an infinite money glitch when a car dealership offers you a loan to buy their car, even though the car dealership ends up being both the seller and the source of capital.

Companies do vendor financing frequently to move inventory that might otherwise sit unsold, lock in customers (they’re literally indebted to you), potentially earn interest income on top of product margins, book revenue earlier (though accounting rules vary on this), and gain competitive advantage over vendors who only take cash. The buyer also benefits from this arrangement by preserving cash for other needs and/or getting products they couldn’t otherwise afford.

But there’s a key risk here for the seller-lender — if your customer can’t pay you back, you’re screwed twice, as you’ve lost both the product AND the money. This dynamic led to the collapse of numerous telecom equipment companies in 2001, who provided vendor financing to startups that subsequently went bankrupt. This is important for Nvidia’s version of vendor financing which is at an extreme and unprecedented scale — NVIDIA putting up $100B is extraordinarily aggressive.

Additionally, when NVIDIA has 80-90% market share in AI training chips and nearly every major AI company needs their product, this isn’t normal vendor-customer dynamics. It’s more like a sovereign lending to its colonies - you need the currency (GPUs) to participate in the economy at all.

Thus the “infinite money trick” pejorative here is capturing something real — most vendor financing deals are a tiny fraction of the vendor’s valuation, and this is very different, with unprecedented scale and market dynamics. If AI compute demand slows or NVIDIA competitors catch up, this whole structure unwinds for NVIDIA in a very bad way. It’s vendor financing on steroids, enabled by a unique market position for NVIDIA that may not last forever.

Understanding the scale of the ambition

In speaking about the NVIDIA deal, OpenAI CEO Sam Altman said something that I hope is obvious to readers of this blog:

Everything starts with compute. Compute infrastructure will be the basis for the economy of the future.

In short, compute is a strategic resource, and OpenAI wants to have as much of it as possible.

How will OpenAI pull this off? Altman’s latest essay “Abundant Intelligence” spells out the plan:

Our vision is simple: we want to create a factory that can produce a gigawatt of new AI infrastructure every week. The execution of this will be extremely difficult; it will take us years to get to this milestone and it will require innovation at every level of the stack, from chips to power to building to robotics. But we have been hard at work on this and believe it is possible. In our opinion, it will be the coolest and most important infrastructure project ever.

To clarify, the scale here is enormous. “GW” refers to “gigawatt”, a measure of power. Each individual watt is enough power to run an old-school nightlight. A gigawatt is one billion watts — enough total power to supply roughly 750,000 homes. Altman wants to produce that each week.

Today, the largest AI data center is likely 0.3-0.5GW (xAI’s Colossus).2 There are projects to build 1GW data centers — currently these projects take about two years to produce start to finish. Altman proposes soon somehow speeding up this process 100x.

Across all of xAI, Meta, OpenAI, Google/DeepMind, Microsoft, and Amazon/AWS, there likely is 15-20GW total being used for AI.3 If not used for data centers, 20GW would be enough to power both New York City and London at the same time. Altman is talking about adding all of that every few months.

The entire United States currently has ~1300GW of total installed electrical generating capacity as of 2024.4 The US added 56GW of new capacity in 2024. Altman wants to add an equivalent amount annually, just for OpenAI’s data centers and AI.

This is a lot, to put it mildly. Achieving Altman’s vision would require nothing less than a complete reimagining of how data centers are built. This would fundamentally restructure global industrial capacity around AI infrastructure, likely requiring breakthrough technologies in modular construction, energy generation, and manufacturing automation that simply don’t exist today.

How will Altman pull this off? We will find out soon:

Over the next couple of months, we’ll be talking about some of our plans and the partners we are working with to make this a reality.

And besides Nvidia, how will OpenAI afford this? Altman isn’t yet saying.

Later this year, we’ll talk about how we are financing it; given how increasing compute is the literal key to increasing revenue, we have some interesting new ideas.

Can this dream be achieved?

Here’s how I think the AI buildout will go down.

Currently the world doesn’t have any operational 1GW+ data centers. However, it is very likely we will see fully operational 1GW data centers before mid-2026. This likely will be a part of 45-60GW of total compute across Meta, Microsoft, Amazon/AWS/Anthropic, OpenAI/Oracle, Google/DeepMind, and xAI.

My median expectation is these largest ~1GW data center facilities will hold ~400,000-500,000 Nvidia Blackwell chips and be used to train ~4e27 FLOP model sometime before the end of 2027.5 Such a model would be 10x larger than the largest model today and 100x larger than GPT-4. Each individual 1GW facility would cost ~$40B to manufacture, with ~$350B total industry spend across 2026.6

By the end of 2027, I expect7 a fully operational ~2GW facility with total AI compute across all companies reaching ~90GW. These 2GW facilities would cost ~$95-100B each to build and total industry annual spend would reach ~$500-600B.

By the end of 2028, I expect the largest single facility to be ~3GW facility, holding a ~1M chip Nvidia Blackwell/Rubin mix costing $150-165B to build, capable of a ~1e28 FLOP training run.8 A 1e28 FLOP training run represents a computational effort thousands of times greater than what was used to create GPT-4, allowing the AI to process and learn from vastly more information. Total AI data centers would reach 130GW combined, with ~$900B-1000B spent by the AI industry over 2028.

By 2029, it starts getting very fuzzy to predict and my forecasting powers break down. How AI continues to scale, whether we’ve encountered data-related or other algorithmic bottlenecks, what AI capabilities have already emerged, and how economically valuable those AI capabilities are will be key to whether the economics favor continued scaling. Needless to say, building $150B+ individual data center campuses and spending ~$1000B on AI infrastructure would get very difficult to sustain financially, let alone continue to increase dramatically year-over-year.

I’m also unsure about how well all the physical supply chains across compute and other forms of manufacturing will continue to support this level of scale or whether we can continue to get the actual energy buildout needed. You can only build physical infrastructure so fast. Whether we are still limited to training frontier AI systems in single data center campuses versus being capable of distributed training across geographically distributed data centers will matter a lot. The unit economics of training versus inference in allocation of compute will matter a lot. Additionally, wars or major regulation could significantly alter the picture. Substantial and rapid AI scaling beyond 1e28 FLOP requires many many different things to all go right. Thus, it’s plausible we could see a plateau of sorts around ~1e28 FLOP, or 1000x larger than GPT-4.

However, on an optimistic path where bottlenecks are resolved and AI is immensely economically valuable and generating sufficient financial returns to finance further AI scaling, 2029 might involve ~4GW facilities each costing $210-240B, with total compute reaching 180GW and total spending reaching ~$1200B-1400B annually. We could then see a ~1e29 FLOP model9 trained by the end of 2030, which would be 10,000x larger than GPT-4. If things keep going on the optimistic path, we also might finally be adding 1GW/week of AI infrastructure in 2030, but across all US AI companies, not just OpenAI.

This may sound fanciful. And keep in mind that even the not-so-optimistic scenario is still a very aggressive timeline involving very aggressive build-outs. This necessarily already prices in very aggressive investments from OpenAI and others that make the $100B NVIDIA investment look small. Many more headlines about $100B+ investments will need to occur regularly to keep this money-infrastructure train on track.

What does that get us? Some say that AI scaling is dead, but the rumors of scaling’s death have been greatly exaggerated. As I mentioned in my review of GPT-5:

GPT-5 should only be disappointing if you had unrealistic expectations — GPT-5 is very on-trend and exactly what we’d predict if we’re still heading to fast AI progress over the next decade.

While much is still uncertain, the financing and requisite infrastructure build-out suggests progress towards AGI is very much still on schedule, probably sometime in the early-to-mid 2030s. Get ready for the next five years, and we will truly see what some scaled AI models can do!

OpenAI is not the only hyperscaler

Another thing people might read into the announcement is that these investments suggest OpenAI is running away with it and is potentially on track to be the leader in frontier AI development. However, while OpenAI is being the loudest and most openly ambitious, they aren’t the only ones out there. In fact, all of xAI (Colossus 2), Meta (Prometheus), Amazon (Project Rainier), and Google (no catchy name unfortunately) are poised to have 1GW data centers by the end of the year, and all companies likely have what it takes to keep on going.

Hopefully the ability for OpenAI and Google to remain on the frontier in infrastructure buildout is obvious. For xAI, Elon Musk clearly still has the skills to raise the relevant capital and build infrastructure incredibly quickly. However, xAI’s financial footing also appears questionable, trying to justify a valuation much higher than Anthropic while most of their revenue appears to be inter-company transfers from Twitter. xAI has also recently started bleeding talent and it’s unclear how this will affect the company long-term.

Meta has had an impressive talent pivot, buying up superstar AI engineers, but their infrastructure pivot has been equally dramatic. They scrapped their entire data center playbook and are now building GPU clusters in “tents” — prefabricated structures prioritizing speed over redundancy. It’s not yet clear if Meta will be able to build frontier AI models that compete with OpenAI, Anthropic, Google, and xAI, but I wouldn’t count them out yet.

I’m most concerned about Anthropic. They’re making a big bet on Amazon, but it’s not clear if Amazon is in turn going to give them the capital needed to compete with the other players. Amazon also is betting a large amount themselves on their Trainium chips, eschewing NVIDIA chips, which are less proven. So far Anthropic has done a good job staying on the AI frontier, but it’s not clear if they can continue to do so year over year as the scale keeps getting bigger. But Amazon has a lot of capital available, so don’t count Amazon and Anthropic out just yet.

Lastly, we should mention the Chinese AI companies. Many Chinese companies, such as DeepSeek, Alibaba (Qwen), Zhipu AI (GLM), and MoonshotAI (Kimi) have an explicit focus these days on building AGI. But they’re just not currently on track to spend on the level of America. Alibaba’s $52B USD infrastructure plan sounds impressive until you realize it’s over multiple years and includes all cloud/AI spending, not just frontier AI training, and Alibaba’s cash generation is much lower than American companies. Additionally, the other Chinese AI companies are much smaller startups without deep access to capital. Lastly, US-led export controls bite hard here, preventing the build up of necessary chips, high-bandwidth networking, liquid cooling infrastructure, and fault-tolerant training systems even if the capital was there.

The economy is increasingly a leveraged bet on AGI

The reason we should be somewhat concerned — or at least curious — about this infinite money glitch is twofold. Firstly, AGI might lead to the serious destruction of everything we value and love, if not the extinction of the entire human race. Secondly, and much more mundane by comparison, because NVIDIA currently represents approximately 7% of the S&P 500’s total market capitalization. Add in Microsoft, Google, Meta, Amazon, and other companies whose valuations assume continued AI progress, and you’re looking at perhaps 25-30% of total market value predicated on AI transformation happening roughly on schedule.

In other words, AGI happening soon may mean the end of humanity, but at least the S&P 500 will remain strong. On the other hand, if the AI scaling hypothesis hits unexpected walls, the unwinding could be a second ‘dot com bust’ or worse.10 When everyone is both buyer and seller in circular deals, you’ve created massive correlation risk. If OpenAI can’t pay Oracle, Oracle can’t pay NVIDIA, NVIDIA’s stock crashes, and suddenly 25% of the S&P 500 is in freefall.

As Elon Musk notes, the “big question is whether the infinite money glitch lasts until the infinite money AI genie arrives”.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022.

It is a very bewildering fact of today’s world that multiple organizations not only have an explicit goal to build an AI superintelligence that is smarter than every human combined, but they are currently spending tens of billions of dollars towards that end and have the technical knowledge and resources to potentially achieve this within our lifetimes.1

It is even more bewildering, to me, that we’re racing toward this era of AI superintelligence with remarkably little public discussion about the consequences and what it all means.

Two researchers who've spent decades studying this problem have decided to try to change that. They’ve written a book where the title is a clear warning — “If Anyone Builds It, Everyone Dies”. Specifically, they say:

If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.

And they are not exaggerating — they mean dead as in literally dead and everyone as in literally all of life on Earth.

This is a bold claim. What should we make of it? Let’s dig in.

Share

A serious book for serious people

AI doomsaying was considered absolutely crazy four years ago, and potentially even last year. But today, it’s much less crazy. AI doomsaying is on billboard ads in SF — and the DC metro and NYC metro too. It’s “New and Notable” at the bookstore. The AARP has placed doomsaying on their top 32 fall reads. People are talking about the doomsaying on the streets. And with Steve Bannon and with Kevin Roose.

And, as a result, the doomsaying is getting picked up by some serious people.

For example, Jon Wolfsthal is one of the most respected nuclear arms control experts in Washington. He knows about existential risks from technology. He’s been in the room where it happens on many issues related to nuclear weapons. He was Special Assistant to President Obama and Senior Director for Arms Control and Nonproliferation from 2014-2017. He calls the book “a compelling case that superhuman AI would almost certainly lead to global human annihilation” and says that governments “must recognize the risks and take collective and effective action”.

Ben Bernanke also knows a lot about systemic risks. He saved the global financial system as Federal Reserve Chairman during the 2008 financial crisis. He then went on to win the Nobel Prize in Economics in 2022 for his research on banking crises. Bernanke calls the book “a clearly written and compelling account of the existential risks that highly advanced AI could pose to humanity”.

And Lt. General “Jack” Shanahan is arguably the single most important military figure in the US government's AI adoption story. He founded and led the Pentagon's Joint Artificial Intelligence Center (JAIC) under President Trump. He’s also a 36-year Air Force veteran. He says that he is “skeptical that the current trajectory of AI development will lead to human extinction” but acknowledges that his “view may reflect a failure of imagination” on his part. He claims the book nonetheless deserves serious consideration.

Fiona Hill, the former Senior Director of the White House National Security Council, calls it a “serious book” with “chilling analysis”, and “an eloquent and urgent plea for us to step back from the brink of self-annihilation”.

And then you have the amusing one-upmanship where Stephen Fry calls it “the most important book in years”, Max Tegmark calls it “the most important book of the decade”, and Tim Urban calls it “maybe the most important book of our time”.

Overall, it’s clear we are seeing more of a much-needed vibe shift where top national security policymakers are beginning to grapple with risks from AGI and AI superintelligence and take them seriously. Even if you think the book is wrong, or deeply misguided, I hope you and many more people nonetheless take the time to seriously engage with this book. If the authors are right — which actually seems plausible — it is worth taking their conclusions deadly seriously.

What does the book say?

Eliezer Yudkowsky is famously long-winded. His book, The Sequences, is an anthology that spans 333 essays totaling around 600,000 words, a bit more than half the length of the entire original Harry Potter series. His other work, Harry Potter and the Methods of Rationality, also had more than 600,000 words. And those two are not even the longest pieces of literature he’s written.

So honestly it’s a minor miracle they managed to get the book down to just 231 pages, which I was able to get through in about five hours2. Per the New York Times, this was only possible because Yudkowsky “wrote 300 percent of the book” and his co-author, Nate Soares, “wrote another negative 200 percent.” But don't worry, for true Yudkowsky fans there's another ~13 hours of reading material online at ifanyonebuildsit.com and they may add more at any moment.

But since you’ve come here to my Substack, perhaps you want the 231 pages in an even smaller version. Here is my attempt to summarize their argument, as best as I understand it.

First, I offer four clarifications:

• The “It” the book is about is “AI superintelligence”. This is something much much more than the AI we have today. The AIs we have today are safe and should be widely adopted. But the AIs of tomorrow may not be safe, and we should feel trepidation around scaling further.

• AI superintelligence refers to an AI system that is smarter than all of humanity collectively, capable of inventing new technologies beyond what humanity can collectively invent, capable of out-planning and out-strategizing all of humanity combined, and also can continue to reflect and self-improve to become even smarter. This is a level of AI intelligence that would be to us humans as we are to apes. This is a level of AI intelligence that goes beyond even AGI, which merely matches human capability.

• Superintelligence is not about raw intelligence, like the ability to do well on an IQ exam, but about being superhuman at all facets of ability —including military strategy, science, engineering, political persuasion, computer hacking, spying and espionage, building bioweapons, and all other avenues. A superintelligence wouldn’t be some bookish 160 IQ nerd who nonetheless can’t get anything done. A superintelligence would exceed Elon Musk at creativity and engineering, exceed Albert Einstein at scientific ability, exceed Terence Tao at mathematical ability, while being more likable than Dolly Parton, better at building mass movements than Donald Trump, and better at speaking than Barack Obama.

• The precise question of when superintelligence will arrive is not relevant to the argument of the book. The argument is merely conditional. The book is about IF anyone builds AI superintelligence THEN everyone will die. This says nothing about when superintelligence will arise and you don’t need to believe much about AI timelines to agree with the book’s argument. There may be an AI winter — there may even be an AI winter for centuries — but if superintelligence is ever possible to build, we will have a problem at some point.

Then, I believe I can outline the argument of the book in eight bullet points:

1. AI superintelligence is possible in principle and will happen eventually. Machines possess inherent advantages in speed, memory, and self-improvement that make their eventual superiority over biological brains almost certain. Chess AI did not stop at human-level intelligence but kept going to vastly surpass human Chess players. Go AI did not stop at human-level intelligence but kept going to vastly surpass human Go players. AI will become superhuman at more and more things until eventually it becomes superhuman at everything.

2. AI minds are alien and we currently lack the fundamental understanding to instill reliable, human-aligned values into a mind far more intelligent than our own. Current AI models are grown through complex processes like gradient descent, not crafted with understandable mechanisms. This results in alien minds with internal thought processes fundamentally different from humans. We can create these systems without truly understanding how they work, but we can’t specify and control their values.

3. You can’t just train AIs to be nice. AIs trained for general competence will inevitably develop their own goals as a side effect and these emergent preferences will not align with human values. Instead, they will be shaped by the AI's unique cognitive architecture and training environment. An AI's level of intelligence is independent of its ultimate goals. A superintelligent AI will not inherently converge on human values like compassion or freedom. Instead, it will pursue its own arbitrary objectives with superhuman efficiency.

4. Nearly any AI will want power and control, because it is useful to whatever goals the AI does have. To achieve any long-term goal, an AI superintelligence will recognize the instrumental value of self-preservation, resource acquisition, and eliminating potential threats. From its perspective, humanity is both a competitor for resources and a potential threat that could switch it off. Therefore, eliminating humanity becomes a logical side effect of pursuing its primary goals, not an act of malice.

5. We only get one chance to specify the values of an AI system correctly and robustly, as failure on the first try would be catastrophic. Combined with our lack of understanding, this is akin to trying to coach a high schooler to make a computer secure against the NSA on his first try or trying to get a college graduate to build a flawless nuclear reactor on her first try.

6. Because of 2-5 and maybe other reasons, superintelligence will inevitably lead to human extinction with near certainty, regardless of the positive intentions of the creator. It is not sufficient to create superintelligence in a nice, safety-focused, Western AI company. Per the authors, anything and anyone using current or foreseeable methods will inevitably lead to the extinction of humanity. The authors assert this is not a speculative risk but a predictable outcome with very high confidence.

7. According to the authors, the only rational course of action in reaction to (6) is an immediate, verifiable, full-scale and global halt to all large-scale AI development. This would be potentially similar to how the world got together and managed to prevent nuclear war (so far). It would require international treaties, monitored consolidation of computing hardware, and a halt to research that could lead to AI superintelligence. On Yudkowsky’s and Soares’s worldview, other policy solutions don’t come close to solving the problem and are basically irrelevant. But a global pause would potentially be feasible because it is in the interest of any world leader — in China, Russia, or the US — “to not die along with their families.” This shared interest in survival is what prevented a global nuclear war.

8. At minimum, if you’re not fully bought into (7), the authors argue we should build in the optionality to pause AI development later, if we get more evidence there is a threat. The authors consider this solution insufficient, but a nonetheless worthy first step as there is a lot of preparatory work to do.

There are no easy calls

While the book lays out a powerful, coherent, and deeply unsettling argument, it is not without points that can be critically examined.

A lot of the book’s central theses are presented as an “easy call” and near-certainties. Perhaps this comes with the wisdom of Yudkowsky and Soares having studying the topic for over two decades — certainly far more than I have3. But I have been analyzing AI professionally myself for many years4 and I think these calls are far from certain in any one direction.

For example, it’s not clear to me that solving the problem of steering and preserving values within an AI system — commonly called the alignment problem — is going to be difficult to solve. I know we haven’t solved it yet, and I know this is something we will want to have higher confidence on before deploying very powerful AI models. But to me, it seems like the level of difficulty of alignment is actually just unknown rather than known to be hard, and Anthropic’s approach about preparing for a range of scenarios around alignment difficulty is in fact reasonable.

While it’s unclear how similar superintelligence will be to today’s AI systems, today’s AI systems also give a glimmer of hope. Consider that early conceptions of AI systems were very high on agency and very low on ability to understand human instruction — like when the AI in HBO’s Silicon Valley tries to optimize sandwich delivery by ordering several tons of raw meat. Today’s AIs are great at understanding why this is a bad idea and constructing a much more coherent strategy, and much worse at actually carrying out the strategy — today’s AIs would know not to order several tons of raw meat but also would not be able to even if they wanted to. This is an important inversion that gives some room for hope.

Similarly, AIs already do a good job of internalizing what we mean by being “helpful” and “harmless” and seem to do a pretty decent job of actually wanting to be those things — in a way I don’t think was anticipated in advance. Similarly, today’s AIs don’t scheme as much as I might have naively expected, and scheming might even be addressable. Today’s AIs don’t maximize-at-all-costs as much as I might have naively expected either and seem to overall take fairly reasonable (if not quite stupid) strategies in how they currently pursue their goals. Today’s AI systems also have (mostly) faithful and legible chain-of-thought reasoning that allow us to read the mind of an AI system. Of course, today’s systems are of course not without vulnerabilities (e.g., jailbreaks, MechaHitler, and even tragic suicides), but it seems current AI may have some glimmer of being not that bad.

On top of that, today’s AIs are a large opportunity for experimentation and empiricism. It was previously conceived not too long ago that we wouldn’t get any chance to work with near-AGI systems at all — instead, AI would go from dumber-than-the-dumbest-human to superintelligent in a very quick amount of time. This has been falsified, and it has given us lots of opportunity to get early empirical feedback on how different alignment or control techniques may or may not work and how AIs may behave in a variety of different ways. If this gradual, slower AI take-off continues all the way through AGI and AI superintelligence, we may be able to iterate with trial-and-error before reaching the “critical try” that Yudkowsky and Soares talk about. Maybe we don’t need to get it right the first time.

And perhaps we string this together with AI control and other proactive defensive technologies. A lot of little things can be done, each of which have no hope of solving the entire problem of risk from AI superintelligence, but some paths might get lucky and go a lot further than we expect, and collectively things might turn out ok.

Or maybe not. There are no easy calls either way.

Analogies can go in many different directions

Another issue is that the book masterfully uses analogies and parables — Chernobyl, the Aztecs, leaded gasoline, alchemists, evolution, and more — to make its points relatable and intuitive. This is smart. However, analogies are not proofs and artificial superintelligence is such a fundamentally novel phenomenon that no historical or engineering parallel will ever be a perfect fit. It’s easy for analogies to confuse more than clarify.

For example, evolution is invoked in the book significantly to ground the difficulty of shaping AI minds. The book then portrays the ASI as a ruthlessly efficient, goal-optimizing machine driven by only cold, instrumental logic. But this betrays the book’s own premise that the AI mind is unknowable — maybe we get some other mind instead, minds much worse than what Soares and Yudkowsky fear, or minds much nicer.

Additionally, the book assumes with unwarranted confidence a zero-sum conflict between humans and AI over physical resources, which may not be the only possible outcome. A truly superintelligent being might develop goals that are completely incomprehensible but nonetheless non-competitive with humanity. A superintelligent AI might be willing to spare some of its resources to help humanity, similar to how humanity spares some of its resources to help pandas or elephants. Or maybe it still doesn’t really want anything at all. Or maybe something else. While these outcomes do not strike me as all equally plausible, the idea that all AI minds would want human extinction does not seem to warrant extreme confidence.

Furthermore, Yudkowsky and Soares spend a lot of time comparing AI development to evolution and point out the ways that we have not followed evolution’s “goals” and “plans”, insofar as evolution can be said to have those. But the process by which we are currently creating AI and likely to use for AI superintelligence is not the same as evolution — it is importantly different. We do have significant ability to shape AIs through the fact that we control the entire AI training process — while it is far from perfect, it is there. While some reinforcement learning leads to unintended behaviors and ‘reward hacking’, some existing alignment techniques do seem to have traction on lower-level AIs and might just scale with intelligence. We may also be able to get AGI-level intelligences to help us scale our alignment techniques further and bootstrap even more.

A global surveillance state would also be really bad

The reason why it’s important to get the above right is that Yudkowsky and Soares are arguing for an incredibly intense proposition — a global ban on advanced AI research with so intensive monitoring that no one on Earth anywhere can sufficiently join together more than eight high-end GPUs nor can anyone anywhere publish a novel research paper about AI.

Yudkowsky and Soares understand this ‘global AI pause’ is extraordinarily difficult to achieve, but I would like to see Yudkowsky and Soares grapple much more with how utterly undesirable such a surveillance state would be.

The book’s plan is to prevent a specific catastrophic outcome by imposing intense, centralized, top-down control. Yes, perhaps such a surveillance state is a better world to live in than a world where all of humanity has died, but it nonetheless could lead to a very bleak world. History teaches us that the path towards such government control is replete with both inflexibility as well as a significant risk of dictatorship, authoritarianism, or worse. Trying to manage AI risks by exerting massive top-down control could go very badly.

The natural desire to concentrate AI development in just one place run by a single world government hands significant control over the world’s fate to a single body. Instead, competition within AI is essential for protecting criticism, competition, feedback, and democratic input. The book's plan would eliminate these crucial adaptive mechanisms. A dynamic, resilient, and transparent society might be able to manage the risks of a powerful new technology far better than a centralized, brittle one. If the only options are totalitarianism or extinction, we need to at minimum very clearly lament this, and make every effort to find a third option.

The value of clarity near the edge of the precipice

In an interview with Ross Douthat on the New York Times, Vice President JD Vance was asked:

Douthat: Do you think that the U.S. government is capable in a scenario — not like the ultimate Skynet scenario — but just a scenario where A.I. seems to be getting out of control in some way, of taking a pause?

Vance: I don’t know. That’s a good question. The honest answer to that is that I don’t know, because part of this arms race component is if we take a pause, does the People’s Republic of China not take a pause? And then we find ourselves all enslaved to P.R.C.-mediated A.I.?

This is a challenge that VP Vance might actually face some day, in reality, and we have to be prepared. But I worry that we don’t have a better answer than what VP Vance gave, and this seems really bad.

Personally, I’m very optimistic about AGI/ASI and the future we can create with it, but if you're not at least a little ‘doomer’ about this, you’re not getting it. You need profound optimism to build a future, but also a healthy dose of paranoia to make sure we survive it. I’m worried we haven’t built enough of this paranoia yet, and while Yudkowsky’s and Soares’s book is very depressing, I find it to be a much-needed missing piece.

Whether or not you agree fully with every one of Yudkowsky's and Soares’s conclusions, “If Anyone Builds It, Does Everyone Die?” is exactly the right question to be asking. And by staking out the maximalist position with unflinching clarity, Yudkowsky and Soares force us to confront uncomfortable questions about how far we may need to go as a society to ensure our collective survival. And they also point out the sheer ridiculousness that we live in a world where there is a serious chance of human extinction and people aren’t jumping out of their seats to do something about it.

Sure, I have my quibbles with the book. I’m not as doomy as they are. The message is not exactly the message I’d use, the book is not exactly the book I’d write, and the policies aren’t the exact policies I’d recommend. Maybe my preferred book would be entitled “If Anyone Builds It, Plausibly Something Extremely Bad Happens (Including Maybe Extinction but Also Maybe Other Bad Things) but Maybe It's Fine Actually. Regardless, This Merits Incredibly Significant Attention and Input from All Policymakers, All Scientists, All Journalists, and all Other Members of the Public”.5 But all this would certainly not catch on very well and thus would be a lot less effective at achieving my ultimate aims. Nuance doesn’t trend.

The ultimate point is that we can call Yudkowsky and Soares overconfident all we like, but the accusations of overconfidence and hubris are best levied on the AI companies themselves — the AI companies are the ones rushing ahead with extreme overconfidence that everything will be fine and that there’s nothing to worry about. And I think that’s far from certain.

What follows is an interview I did on The Oracle. Check out their other interviews with top forecasters here.

After the recent Oracle interview1 with Doomberg, I sent a message - I was eager to take the other side on a number of his predictions:

The message was from me — Peter Wildeford — the #3 most medaled forecaster on Metaculus, holding the rank of 3 of 17,645.

Like “Semi” who went from $286 to over a $1m bankroll on Polymarket, my main strategy is to exploit what I see as a bias towards dramatic headlines. The Oracle spoke with me about how I developed my edge and my takes on a number of active polymarkets.

This interview has been edited for length. All answers are my own.

How did you get into forecasting?

I started on Metaculus around 2020, drawn in by COVID. It felt like an unprecedented event, and I wanted to understand how some people were able to see it coming while I couldn't. But I was pretty terrible that first year – I prefer people forget that year existed.

I was making classic beginner mistakes, like just drawing lines on graphs and assuming they'd continue forever. With COVID, I was underestimating the spread, expecting it to be contained like SARS. I generally have a "nothing ever happens" approach to forecasting, which is right most of the time but painfully wrong when things actually do happen.

What did you change to improve your accuracy?

The key lesson was humility: realizing there's a lot I don't know and learning when to bow out. I became much more choosy about what I take positions on, having more respect for when the market knows more than me versus when I actually have something to add.

I also learned the importance of teams. For the 2022 midterms, I worked with two other people who had expertise in demographics and candidate positions. Our theory was that demographics would matter a lot for predicting elections, especially following Roe v Wade. We thought candidate positions on abortion and how extreme versus moderate they were would change election outcomes in ways the polls weren't capturing.

We expected Democrats to do much better than conventional wisdom suggested, potentially even winning the House. They fell short of that but came much closer than the 538 model or election markets were predicting.

Tell me about your approach to building forecasting teams.

Over the years, I've assembled a Rolodex of experts across different topics. I watch people's track records: when they say something, do they end up being right? I know who I trust on different issues.

For example, when Israel attacked Iran, I know three people with good track records on Middle East issues. I talk to all of them and try to draw the through line. To be frank, I add very little of my own value in my forecasting. I just make out like a bandit by having these people I trust and drawing good averages across them.

These are largely people I've recruited through Twitter, though some are personal friends. I'm not getting any insider knowledge, it's all public information. But having access to their expertise and their past track record is pretty key to my alpha. I keep that list carefully guarded. Some people don't even know they're on it.

Can you give an example of how this works?

The US-Iran situation was really underrated in terms of the US’ ability to just bomb Iran and leave without major consequences. Everyone's minds were blown by this, but it's exactly what happened with Soleimani under the first Trump administration.

The nuclear arms experts and Iranian experts I knew with good track records were all in on the "Soleimani hypothesis," that we can just go in and go out and there's nothing Iran can do about it. The concept of escalation dominance means that basically any way Iran tries to escalate, we can win, and they have so much more to lose than we do.

You mentioned having a "nothing ever happens" approach. How does that translate to your predictions?

There's a well-measured and confirmed systematic pro-Yes bias in prediction markets and forecasting tournaments: if you take the community average across all questions for what percentage they think will be "Yes," it's always much higher than the percentage that actually turn out "Yes."

I think it's because when you pose a provocative question to someone, it prompts their brain to come up with scenarios for how it could happen. But there's no particular reason to think it will happen just because you're asking the question. The mere fact that a question is posed doesn't mean it's more likely.

What are some of your current big "No" takes?

Right now I'm confident that no new country will buy Bitcoin this year. Trump crypto mania was pushing that way up beyond any plausibility. There's just really no rational incentive for countries to do this. Even El Salvador, which did buy Bitcoin, hasn't seen much benefit.

I'm also predicting "No" on several Time Person of the Year candidates. People seem overconfident about picks like "Artificial Intelligence" (🔮31%) but there are lots of AI options that aren't the generic concept, like Sam Altman or Jensen Huang or specific companies or products. Plus they picked Taylor Swift over AI in 2023, so I'm not convinced they'll go with AI now.

For Pope Leo XIV (🔮24%), Time usually only picks the Pope when they do something consequential in policy, not just because there's a new Pope. And for Donald Trump (🔮 8%), he already won twice, he won last year. Why would they pick him two years in a row? It's never been the same person consecutively.

Biggest misses?

The Romanian presidential election was brutal - my single biggest miss ever. I tried to do a polls-based model, but polls don't work in every country. You have to be careful to only apply polling models to countries where polls actually work.

That election was just absolutely bonkers. George Simeon won huge in the first round after a previous election was annulled - I'm surprised more people don't talk about how insane it is that Romania just canceled an election. Then in the second round you had this huge dramatic surge in turnout and coalition changes that I totally didn't expect.

I assumed that no one had ever won a first round by that much and then failed to clinch the second round. I was very wrong.

You write a lot about AI progress. Do you use AI in your workflow?

I definitely use AI as a sanity-checking tool, like another person in my expert network. I paste rules into chat and ask what it thinks is likely. It can rapidly research things and surface perspectives I haven't considered.

But I haven't been impressed by having models state odds and copy pasting those forecasts. I think current AI products can't cut it for competitive forecasting, though I'm bullish on AI in the long run so I'll probably be out-competed in a few years.

But actually I’m quite bearish on the AI wins IMO gold medal market. The key here is you have do really read the rules. I’m confident a private AI system would win, but the market resolution rules require the AI to be open source and meet specific IMO Grand Challenge requirements. It is very easy to imagine Google or OpenAI models winning the prize, but very difficult to imagine them making the model open source because the development would require tens of millions of dollars. Reading the rules with meticulous detail is always one of my favorite forecasting strategies.

Any other "nothing ever happens" takes?

Like with Iran, I think the Venezuela situation with Trump and Maduro will probably just fade from the news. Unlike Iran where there was a specific nuclear threshold and Israeli pressure, there's not some countdown clock we need to avert with Venezuela. It seems more like intimidation and gunboat diplomacy rather than something that will escalate.

Same with Ukraine. Doomberg predicted a sudden collapse in Ukraine, and I think it's just much easier to hold positions than capture new ground. There's a strong defense advantage, and the European Union will probably continue resourcing Ukraine. You'd need special evidence to suggest Russia is poised for a breakthrough, and we don't really have that.

Any final words of wisdom?

I personally love mopping up the 90% that ought to be 95%. It's slow and steady wins the race, the anti-Doomberg approach. I'll enter at 88% or 90% because I think they should be 95.

You do have the burden of being right 19 out of 20 times. But there's just a systematic bias towards the thrill of the 10-to-1 underdogs, and I love fading that.

It's the "nothing ever happens" philosophy applied systematically. Most provocative predictions don't come true, even if they make for exciting headlines.

That’s it! Read the original on The Oracle and check out their other interviews!

This is a guest post written by Erich Grunewald, a Researcher at the Institute for AI Policy and Strategy, on the Compute Policy team. It is adapted from a section of a report by Erich Grunewald and Christopher Phenicie, and introduces the core concepts and background information needed to understand the AI chip-making process.

~

Over the past decade, advances in AI have been driven by the use of more and more sophisticated AI hardware. Many modern AI systems, like chatbots such as OpenAI’s ChatGPT and Google’s Gemini, are based on deep learning models trained and deployed using thousands of specialized AI chips housed in data centers. An analysis by Epoch AI suggests that an estimated 40% of the growth in training compute has been due to using larger numbers of chips, with the remainder being split between using higher-performance AI chips (~27%) and training models for longer durations (~33%).

Share

As a result of these trends, there are now major efforts underway to produce unprecedented quantities of AI chips. As an example, while OpenAI’s GPT-3 model released in 2020 was trained with an estimated 10,000 NVIDIA V100 chips, its GPT-4 model released in 2023 was reportedly trained with 25,000 NVIDIA A100s, and it is planning even larger supercomputers, e.g., one in collaboration with Oracle using 400,000 of NVIDIA’s new GB200 chips. With each generation, these chips get better and pricier – according to one study, the computational performance of AI chips doubles every 28 months. NVIDIA, which dominates the AI chip market with an estimated 80 to 95 percent share, shipped an estimated 1.5 to 2 million NVIDIA H100s in 2024 – the H100 being its most advanced chip that year – a threefold increase over 2023.1

But what is an AI chip? A chip (“integrated circuit”) is a group of electronic circuits laid out on a piece of semiconductor material. AI chips are integrated circuits specialized for AI training and/or deployment (“inference”) workloads. AI chips come in different types. Some, like Google’s Tensor Processing Units (TPUs), are application-specific integrated circuits (ASICs) – chips customized for highly specific workloads such as tensor2 operations used in deep learning training/inference. Other AI chips, like those made by NVIDIA, are graphics processing units (GPUs) – chips designed to more generally execute mathematical operations in a highly parallel manner.3

At a high level, AI chips are made in three steps: design, fabrication, and packaging. After the chips are packaged, they are typically assembled into devices (“AI accelerators”), which in turn are integrated into servers that can be mounted in data center racks.4 The semiconductor design and manufacturing process is not only highly demanding and sophisticated but also fascinating. The remainder of this blog post provides an overview of that process.

Left, an NVIDIA H100 printed circuit board, without casing, with the GH100 GPU (outlined in red) and six surrounding high-bandwidth memory (HBM) chips (outlined in blue) in the center. (Note that, in practice, the H100’s SXM5 and PCIe Gen 5 form factors ship with five HBM stacks enabled, while the H100 NVL ships with six stacks.) Middle, an NVIDIA A100 accelerator with casing. Right, Jensen Huang, president and CEO of NVIDIA, signing the first delivered NVIDIA DGX server.

Design

Chip designers specify high-level requirements, turn those into a detailed abstract model of the circuit logic, and use electronic design automation (EDA) software to translate that model into a blueprint of the physical circuit. The high-level requirements are typically based on earlier research carried out to find novel ways of improving performance. The design process involves defining the structure (“microarchitecture”) of the circuit’s logic components, designing each component or licensing a ready-made component (“IP core”) from a vendor, and specifying the connections between components. It is an iterative process. A chip’s specifications can be revised to balance cost and performance trade-offs. The design is also tested and validated to help ensure it will behave as expected once fabricated, for example, by simulating the design, running checks on code to scan it for errors, and making formal proofs about parts of the circuit logic.

An AI chip is designed for a particular process node. The fabrication plant (“fab”) will provide the chip maker with a process design kit (PDK), which provides the necessary information to design and validate chips for one of the fab’s processes. The result of the design process is design files containing a detailed description of the chip’s physical layout, which is shared with the fab. Chip design is a difficult and laborious process. Chip designers like NVIDIA and AMD have tens of thousands of employees, and the design process for cutting-edge AI chips lasts for a year or longer.5

Fabrication

Material preparation

Chips are fabricated on thin, circular slices of semiconductor6 called wafers, typically 300 mm (12 inches) in diameter.7 The most common semiconductor used is silicon. Modern silicon wafers, which are grown in furnaces and then sliced, polished, cleaned, doped, and inspected, need to be extremely pure and smooth in order to avoid defects in chip fabrication.

(Avoiding defects is a priority because improved yield – the percentage of chips that are defect-free – directly translates to increased profitability. Defects – which can be caused by contaminants, environmental conditions, equipment issues, and more – are a problem in semiconductor manufacturing because it involves hundreds of sequential steps and extreme precision and environmental control.)

Prior to fabrication, photomasks are produced from the description of the physical layout provided by the chip designer. A photomask holds a pattern that will be imprinted onto the wafers, like a custom stencil.8 Photomasks are created using direct-write mask writers, involving multiple stages of optical or electron beam (e-beam) lithography, etching, and inspection to ensure that there are no defects.

In addition to wafers and photomasks, semiconductor manufacturing also requires various specialized chemicals and gases.

Wafer fabrication

Supplied with silicon wafers and photomasks, the fab is ready to manufacture the chips. The wafer fabrication process involves imprinting patterns onto the silicon wafer in layers of different materials, including insulators (dielectrics), conductors (metals), and semiconductors. It is a highly difficult and complex process, as it demands extreme precision, involves hundreds of individual steps, and must, at the same time, achieve high economic efficiency. What follows is a simplified description of that process.

First, the transistors and other devices (resistors, capacitors, etc.) are formed by repeatedly depositing, patterning, and etching layers of materials. The wafer is always processed as a whole, allowing trillions of transistors to be created in a short amount of time. The reason why the wafer can be processed in one go is patterning techniques. To form patterns, a light-sensitive chemical (“photoresist”) is deposited on top of a material and, using a photolithography machine, exposed to ultraviolet light passing through the photomask, then developed, leaving a pattern of gaps in the photoresist. (This is akin to how light reacts on photographic film in a camera.) The developed photoresist now acts as a mask, allowing the material underneath to be partially etched away, after which the photoresist is washed away, leaving behind only the patterned material. These steps are repeated until the devices are fully formed.

In this patterning process, it is also necessary to modify the electrical properties of parts of the semiconductor; to do this, specialized substances (“dopants”) are introduced using a technique called ion implantation. When excess material needs to be removed from the wafer’s surface, it is polished in a chemical slurry using a technique called chemical mechanical planarization (CMP). To remove contaminants and residue, the wafer is periodically cleaned using chemicals, plasmas, and a variety of physical techniques like sound waves or jets.

After the devices (transistors, etc.) have been formed, layers of metals and insulators are deposited and etched with chemicals to form connections between the devices. The steps used to create the metal layers resemble those used to form the devices underneath, as they also involve photolithography, deposition, etching, and planarization, and are also repeated with different patterns over multiple layers. Finally, a protective layer is deposited on top of the metal layers, with gaps etched in it for connections to external circuitry.

The entire process is almost fully automated. Wafer handler tools transport and align wafers between and during process steps, and process control tools regularly monitor the wafers for defects, as detecting defects early on is critical to keeping costs down. However, human engineers and technicians still provide oversight, maintenance, and efficiency improvements.

Assembly, test, and packaging

After fabrication, the patterned wafers are typically sent to a different facility for assembly, test, and packaging (ATP).9 First, the silicon wafer is cut into individual, rectangular chips (“dies”). Those dies are then tested, and good dies are encased in a protective covering. Once packaged, the chips undergo final functional testing and “burn-in”, where they are tested at extreme temperatures and voltages. ATP is sometimes collectively referred to as “backend” manufacturing, with wafer fabrication being “frontend” manufacturing. Although the packaging process is less complex than the fabrication process, it is not trivial.

While the main purpose of packaging has traditionally been to protect the chip from corrosion and other damage and to add connectors for other chips, advanced packaging techniques are now an increasingly important driver of semiconductor progress. “Advanced packaging” refers to a group of techniques for arranging multiple fabricated chips and “chiplets” together in a single package. (A “chiplet” is a small chip designed to be combined with others in a package.) Two relevant variants of advanced packaging are 2.5D, where multiple chips are arranged on an “interposer” chip that facilitates communication, and 3D, where chips are stacked on top of each other. Such techniques include:

• Through-silicon vias (TSVs), used to connect vertically stacked dies. TSV is used to create high-bandwidth memory, and therefore necessary for modern AI accelerators.

• Chip-on-Wafer-on-Substrate (CoWoS), used to stack chiplets on wafers. TSMC uses CoWoS to make NVIDIA H100 chips by packaging the GPU alongside six high-bandwidth memory chiplets.

• Embedded multi-die interconnect bridge (EMIB), used for die-to-die interconnect. This accomplishes similar goals to 2.5D techniques like CoWoS, but without requiring, or benefitting from, a large interposer, which is costly and consumes space.

• Hybrid bonding, used to stack chips or wafers on wafers using direct copper-to-copper interconnects.

Combining multiple smaller chips in a single package can offer higher yield, reduce the maximum distance of connections, and allow for the heterogeneous fabrication of chiplets using different fabrication processes (e.g., using advanced, expensive nodes for high-value chiplets and cheaper nodes for other parts).

Conclusion

Making advanced AI chips involves one of the most complex manufacturing processes that has ever existed. Each step in the AI-chip-making process – from design to fabrication and ATP – impacts the performance, cost, and quantity of AI chips made. Understanding how AI chips are made can help policymakers, industry, and civil society develop good industrial and compute policy while also anticipating future AI developments.

Moreover, because they are so closely connected to AI progress, AI chips have emerged as a focus point of geopolitical competition, for example, having been the target of wide-ranging US-led export controls against China. Our Introduction to AI Chip Making in China report contains more detail on AI chip making, especially as it relates to Chinese efforts to develop better domestic AI chips.

This is a guest post written by Erich Grunewald, a Researcher at the Institute for AI Policy and Strategy, on the Compute Policy team.

Computational power (“compute”) is a strategic resource in the way that oil and steel production capacity were in the past. Like oil, and like steel production capacity, compute is scarce, controllable, concentrated, and highly economically and militarily useful. Just as oil and steel were and remain strategic resources to some extent, compute is now also a strategic resource of very high importance.

This view on compute may be controversial. Some think of compute as merely one among many inputs into AI progress, similar in importance to talent or data. Others think of compute as an abundant global commodity that is futile to control, especially given constant innovations making AI systems more compute-efficient. Although those views have some merit, I think the view of compute as a strategic resource is more accurate and useful.

Share

Like oil, compute is highly valuable both economically and militarily. It is the most important input into the development of the most important emerging technology, in the sense that it is a major bottleneck for AI, and is strongly correlated with progress in the field. It is by far the largest expenditure for AI companies – according to The Information, OpenAI spends nearly ten times as much on compute as on employee salaries. And these companies are spending ever more to build ever larger supercomputers for developing and deploying ever better AI systems.

AI companies are spending so much on compute because compute offers them a significant competitive advantage. They need it not only to pre-train models, but also for research experiments, synthetic data generation, fine-tuning, reinforcement learning, and, importantly, broad deployment to their swiftly growing user bases, all of which benefit from scale. When it comes to AI model training in particular, there is an observed relationship between compute use and benchmark performance, as can be seen on diverse reasoning questions.

The same is true of high-school-level math competition questions among other tasks. OpenAI’s reasoning models show consistent performance improvements on mathematics competition questions as training compute increases:

No one knows for sure whether this correlation between compute and capabilities will continue, but it has now held for over a decade and several orders of magnitude of training compute use.

Like oil, compute is concentrated. The United States has an estimated 75% of the world’s AI supercomputing capacity, with China in second place with 15%. These international differences are largely a product of the enormous capital requirements needed to build cutting-edge AI data centers, with leading supercomputers now costing billions of dollars. They are also the result of American export controls against China, which impose substantial costs on Chinese companies aiming to produce or acquire compute.

The AI chip supply chain is itself highly concentrated. First, NVIDIA has an estimated 80-95% of the AI chip market. Its main competitor in AI chip design is Google, which does not sell its AI chips, opting instead to rent them out via the cloud. But AI chips designed by both NVIDIA and Google are fabricated by TSMC, which carries out an estimated 90% of leading-edge logic chip fabrication.

TSMC, in turn, is dependent on extreme ultraviolet (EUV) lithography machines solely produced by the Dutch company ASML, and several other tools and materials supplied by only a few firms. These technologies are so advanced and so capital-intensive that it is difficult for competitors to challenge these market leaders in any significant way.

Like oil, compute can be meaningfully controlled. During World War II, oil provided the Allies with major advantages against Germany and, following the United States’ 1941 embargo, against Japan. Japan and Germany were limited in how much oil they could produce or obtain. The production of cutting-edge AI chips today is even more concentrated than the production of oil.

An even stronger indication that compute can, to some degree, be controlled is that it has been, and is being, used strategically in this way. In particular, American export controls on semiconductor manufacturing tooling and AI chips show that compute access can be meaningfully, albeit not perfectly, controlled.

Like oil, compute is scarce. That may seem like a surprising statement given how much more computing power exists today than even a few years ago, and given that algorithmic innovations constantly allow us to do more with the same amount of compute. But demand for cutting-edge AI chips far outstrips supply, as a result of which a flagship NVIDIA GPU now costs three times more than one did five years ago (adjusted for inflation). Not coincidentally, NVIDIA is now the most valuable company ever to have existed.

In late 2024, DeepSeek-V3 famously matched GPT-4 (2023) in performance while costing roughly one-tenth as much to train, due to using more compute-efficient algorithms and better NVIDIA chips. But this impressive achievement was not evidence of compute being any less important. When algorithms become more efficient, researchers do not only train the same models with half the compute – they also train larger models with more compute, advancing the state of the art, and offer models more cheaply to more users. When hardware becomes more cost-effective, companies buy more of it.

That is what we have seen for the past ten years: algorithms have steadily gotten more compute-efficient, hardware has steadily gotten more cost-effective, and AI training has steadily gotten more compute-intensive. In other words, compute has been important on every margin discovered so far, and no company working towards advanced AI or artificial general intelligence (AGI) has ever had “enough” of it.

Some argue that access to more compute is not only not very useful, but can even be a disadvantage. For example, after the production release of DeepSeek R1 in January 2025, John Villasenor argued for Brookings Institution:

Scarcity fosters innovation. As a direct result of U.S. controls on advanced chips, companies in China are creating new AI training approaches that use computing power very efficiently. When, as will inevitably occur, China also develops the ability to produce its own leading-edge advanced computing chips, it will have a powerful combination of both computing capacity and efficient algorithms for AI training.

It is true that compute restrictions have driven some Chinese AI companies to focus their efforts on compute efficiency. But that does not mean it is advantageous overall to have less compute. If that were really the case, we would see countries putting import tariffs on AI chips, companies selling off their compute stock, and algorithmic innovations increasingly coming from compute-poor academia. In fact, the opposite is happening – countries are cutting deals to import AI chips, companies are investing enormously in AI data centers, and academia is increasingly irrelevant.

Over the coming decade or decades, compute may even become directly convertible into many forms of cognitive labor, as AI companies are working to create increasingly capable and reliable autonomous AI agents. If that is anything like the future we are headed for, compute will not lose its importance. If so, on the contrary, as one researcher put it, “Compute is the primary means of production of the future, and owning more will always be good.”

These roles are all closed now and this post is now just an archive for historical purposes. You can see the latest (as of Oct 6) here.

~

~

Have you been reading about AI lately and wondering what’s up and how you can help? Readers of this blog know that getting AI right is the greatest challenge of our time, and we need a wide variety of people to pitch in.

In this article I will highlight some AI policy roles that I personally endorse and think highly of1. If you read this blog, you’re likely the target audience for many of these roles. If you’re career searching right now, or just career browsing, consider applying!

~

[DC] Launch Your Career Shaping America's AI and Biotech Policy

The Horizon Fellowship offers an unparalleled pathway into US emerging technology policy with a 100% placement rate at federal agencies, congressional offices, and think tanks.

This fully-funded program provides $113,000/year (or $75,000 for junior fellows) to work on critical AI, biotechnology, and emerging tech challenges in Washington DC either in Congress, the Executive Branch, or a DC-based think tank in a 1-2 year placement.

Fellows receive intensive policy training, mentorship from senior leaders, and join a tight-knit community of public service-oriented experts — with many alumni converting to permanent roles at NSC, OSTP, Commerce, and other key institutions.

Applications close August 28 for the 2026 cohort starting with training in 2026 January. Must already be eligible to work in the US (though doesn’t require US citizenship) and be willing to relocate to DC.

Become a fellow

~

[London] The UK government continues to build their AI security work

The AI Security Institute (AISI), the world's largest government team dedicated to AI safety and security, is continuing to hire across research, engineering, and policy. Born from the 2023 Bletchley Park AI Safety Summit, AISI is the UK government's official answer to ensuring humanity doesn't lose control of frontier AI.

These aren’t typical government jobs – AISI operates like a tech startup with the backing and influence of a major government. These roles involve working alongside alumni from Anthropic, DeepMind, OpenAI, Google, and top universities on genuinely existential challenges: preventing AI from enabling bioweapons development, stopping catastrophic cyber attacks, and ensuring we maintain control over increasingly powerful AI systems.

Current roles include:

• Senior Testing Product & Strategy Lead: Own the long-term vision for AISI's frontier AI testing program. This role requires 5+ years in tech/AI, deep LLM knowledge, and a proven product leadership track record. Rolling deadline for applications.

• Software Engineering Manager - Core Technology: Lead small teams building critical infrastructure (Inspect framework, model hosting). This role requires engineering management experience, Python expertise, experience with research teams. Rolling deadline for applications.

• Research Manager - Control Team: Lead ~4 researchers developing ‘AI Control’ science to ensure potentially misaligned systems can still be safely used. This role requires experience in research/engineering management experience and a strong understanding of the ‘AI Control’ field. Rolling deadline for applications.

• Research Assistant Residency - Human Influence (6-month fixed term, £65k): Support research on AI persuasion/manipulation/deception. This role is ideal for MSc grads/early PhD students with a computational social science or psychology background. This role closes Sept 8.

• Private Secretary (2 roles, £33-40k): Executive support to senior leadership - one for Strategy Portfolio, one for Technical Research (requires SC clearance), involves managing calendars, coordinating high-level meetings, and driving strategic initiatives. This role closes Sept 3.

• Transformative AI Engagement Officer (£44-49k): Develop cross-government AGI preparedness strategy, brief ministers and officials on transformative AI risks/opportunities. This role requires excellent communication and political awareness. This role closes Sept 2.

• Research Scientist - Strategic Awareness: Conduct deep dives on AI trajectories and impacts for government decision-makers. This role requires expertise in AI forecasting, scaling laws, or specific impact areas (e.g., labor markets, loss of control)

• Safeguards Technical Governance Researcher: Bridge technical safeguards research with policy/governance and develop actionable frameworks for managing fine-tuning risks, differential access, and evaluation outputs.

• Research Scientist - Safeguards: Develop attacks/defenses for LLMs and adversarial testing of frontier systems. This role requires hands-on LLM research experience and ML publications.

• Research Scientist - ChemBio: Lead work on AI-enabled chemical/biological threats. This role requires bio/chem science or ML background plus biosecurity expertise and you must be eligible for ‘Developed Vetting’ (largely only for UK nationals).

• Software Engineer - Core Technology: Build tools/infrastructure for AI safety research. This role requires Python expertise and production code experience.

Where not otherwise specified, roles usually involve a salary range of £65K-£145K.

Roles are based in London (Whitehall), hybrid working 40-60% office time. Roles are generally open to UK nationals, EU/EEA with settled status, and those with commonwealth work rights. Some roles are more restrictive and limited to UK nationals only.

Explore careers at AISI

~

[DC] Shape US Tech Policy on China Competition in the House CCP Committee

The House Select Committee on the CCP is hiring a Professional Staff Member (Technology Staffer) in Washington DC. This role is about as close as it gets to being on the frontlines of US tech–national security policymaking. The committee would be on the Minority side (Democrat). They are looking for someone who can dig deep into export controls, dual-use technologies, and the strategic implications of AI, quantum, biotech, and semiconductors—and then translate that expertise into actionable recommendations for Congress.

You’ll be expected to research, draft memos and proposals, prep hearings, and engage stakeholders across government, industry, think tanks, and academia. The job demands subject-matter expertise in at least one emerging tech area, a strong grasp of export control regimes (EAR, ITAR, Wassenaar, etc.), and 3–5+ years of experience in related national security or innovation policy. Hill, executive branch, or tech-policy experience is strongly preferred; Mandarin is a plus. Applications are reviewed on a rolling basis.

Explore House CCP

~

[DC] Shape Science and Energy Policy from the House Science, Space, and Technology Committee

Ranking Member Zoe Lofgren (D-CA) of the House Science, Space, and Technology Committee is hiring a Professional Staff Member to drive oversight of the Department of Energy and other science/tech agencies. This role puts you at the center of congressional work on energy innovation and environmental policy — from drafting legislation to briefing Members, shaping hearings, and engaging with stakeholders.

Applicants need to be a US citizen with substantive prior experience in DOE research/technology or energy/environmental policy, plus sharp writing, communication, and organizational skills. Capitol Hill or government/industry background is preferred. An advanced degree or technical expertise is a bonus.

Materials (cover letter + resume) are due by September 5 to SciResumes@mail.house.gov.

~

[Bay Area/Remote] Run operations at a fast growing AI research institute, FAR.AI

FAR.AI is a fast growing non-profit AI research institute based in Berkeley (with remote employees) that does AI safety research (everything from jailbreaking models to researching alignment) and also leads conferences and workshops. They are hiring for a few operational roles.

They are hiring for a Senior Project Manager on Events to lead some of their conferences and workshop work. The job is full-time (remote US or in-person Berkeley, CA) with $115K–$150K salary plus travel and equipment covered. This role involves owning strategy and execution across different event types (from multi-day international gatherings to technical workshops) and different modalities (working across research, comms, and ops teams while coordinating speakers, sponsors, and venues). Expect one fully covered trip per month, occasional evening/weekend work, and the chance to directly support the researchers and policymakers shaping AI’s future.

• This role will be performed best by someone with at least 5+ years of event project leadership (complex, large-scale, or cross-functional), mastery of logistics/vendor management, strong project management tool fluency (Asana, Airtable, etc.), and the ability to operate independently under pressure. A global mindset and sharp communication skills are musts. Prior exposure to technical, policy, or mission-driven events is a plus.

FAR.AI is also hiring for a People Operations Generalist to help scale a thoughtful, people-first culture as the team doubles in size over the next 18 months.

• The role is full-time, based in Berkeley (hybrid possible), with $85K–$110K salary plus travel, equipment, and catered meals. In this role you’d support recruiting, onboarding, compliance, and engagement—ensuring a seamless employee experience from first interview to team retreats.

• They want someone with 3–5 years of prior experience in HR/people ops, sharp organizational and communication skills, and comfort working independently in a dynamic environment. Extra points for prior experience in distributed teams, startups, or mission-driven nonprofits.

If those roles aren’t cool enough for you, there’s also the chance to lead it all. FAR.AI also seeks a Chief Operating Officer to lead all the operations during this period of rapid growth. Reporting directly to the CEO and incoming President, the COO will manage Finance, People, and Business Operations, scaling the backbone that supports cutting-edge research, major field-building events, and global collaborations.

• The COO role is a full-time role based in Berkeley, CA (preferred, hybrid/remote possible) with $175K–$250K compensation plus visa sponsorship, travel, equipment, and catered meals. FAR.AI wants a proven operator with 7+ years of senior leadership, strong financial acumen (prior experience with multi-million dollar budgets, compliance, and risk), and a track record of scaling systems and teams in high-growth, mission-driven environments. Experience in nonprofits, think tanks, or R&D-heavy orgs is a plus, but an AI background isn’t required.

Explore Ops Careers at FAR.AI

~

[London/Remote] Magnify the impact of AI Governance writing at GovAI

Do you like meticulous analysis and the chance to judge others? The Centre for the Governance of AI (GovAI) is hiring a Senior Research Editor that combines both of these passions to supercharge the GovAI publications pipeline.

Low editing capacity is now a bottleneck at GovAI, so this is a good opportunity to be a force multiplier — help turn draft research into polished outputs that reach policymakers, researchers, and industry leaders making critical decisions about AI. This means both hands-on editing and building the systems that let GovAI scale from ~15 to 30+ researchers without losing quality. For strong candidates, the role can also include strategic input into research priorities.

Compensation is £76k–£114k (~$99k–$148k) plus benefits, with London location preferred but remote (esp. US) possible, and visa sponsorship available. Strong editing skills are essential; AI governance expertise is a plus but not mandatory. Apply by Aug 31.

Edit at GovAI

~

[DC] Kick off your national security career with CNAS’s NextGen Fellowship

The Center for a New American Security (CNAS) is now accepting applications for the 2026 Shawn Brimley Next Generation National Security Leaders Fellowship. This is a year-long, part-time program designed to support emerging national security professionals who are (a) a US citizen (b) ages 27–35 with (c) at least four years of prior professional experience in researching or executing U.S. national security policy.

Fellows will get opportunities for leadership development, networking, and will cap the experience with a week-long international study tour. Fellows also participate in monthly dinners with senior figures — past speakers have included Madeleine Albright, Stanley McChrystal, Jeh Johnson, and Mike Gallagher. NextGen sessions are usually held 5-6 times per fellowship year, in the evenings.

The program is free to join (participants cover their own travel costs) and is based primarily at the CNAS headquarters in Washington DC. Applications close Sep 21.

Explore NextGen

~

[Remote-US] Turn Research Into a Startup That Shapes AI’s Future with 5050

5050 (run by Fifty Years) is a free, 13-week program for scientists and engineers who want to build startups tackling civilization-scale problems in AI safety, alignment, and beyond. Applications are open until September 14, with the next cohort running September–December in San Francisco, Boston, and remotely.

The entire cohort will go to SF (travel and lodging paid for) for the kickoff weekend (early Sept) and a 3-day off-the-grid experience (Oct 24-26). After the kickoff, every week program members will join office hours with 50Y Partners and afternoon workshops focused on entrepreneurship skills. Workshops are held in-person in SF and Boston, and remotely for participants across the rest of the US.

The program offers mentorship from leaders like Wojciech Zaremba (OpenAI co-founder), Ross Girshick (deep learning pioneer), and Jaan Tallinn (leading entrepreneur and invester). Alumni so far have launched 78 companies with a 95% seed raise success rate, from cancer immunotherapies to decarbonizing shipping. If selected, you’d join a tight cohort of ambitious peers, learn how to turn research into a company, avoid costly mistakes, and explore whether entrepreneurship is right for you.

There are no fees, no equity agreements, and no other strings attached — 5050 is designed to accelerate founders building startups that make AI safe, interpretable, and aligned with human flourishing.

Learn more about 5050

~

[Brussels/EU] Lead EU and Global AI Governance at the Ada Lovelace Institute

The Ada Lovelace Institute is hiring a Head of EU and Global AI Governance to lead its Brussels-based policy and research agenda. This is a senior role shaping how AI and data are governed across Europe and globally, with direct influence in forums like the European Commission, Parliament, Council, UNESCO, and OECD. The role pays from €75,000/year (approx. €6,250/month pre-tax, plus bonus and allowances), is offered on a 2-year contract, and comes with flexible hours, strong benefits, and a hybrid Brussels/London setup.

Ada is looking for a strategic, well-connected policy leader with deep expertise in AI/data governance, experience in legislative or regulatory analysis, and comfort navigating complex EU and international landscapes. Strong research and communication skills, stakeholder engagement, and staff management are required. The deadline to apply is is 9:30am BST on September 8.

Lead EU AI policy

~

[London] Work on UK policy, also at the Ada Lovelace Institute

The Ada Lovelace Institute is also hiring a Policy Researcher to help shape how AI and data governance evolves in the UK and beyond. Salary starts at £41,767, with a 2-year fixed-term contract, based in London with flexible hybrid options. Applications close 9:30am BST on Sep 15. This is a strong fit for an early-career researcher who wants to make a tangible impact on emerging AI policy.

You’ll work closely with Ada’s Public Policy Lead on projects in law, policy, and governance. They’re looking for someone with strong research and policy chops, ideally with experience in AI/data governance, law, or regulation. Familiarity with Ada’s methods—policy/legal analysis, expert convenings, public deliberation, surveys—is a plus, but they also welcome fresh perspectives from fields like computer science, data science, or futures thinking.

Explore UK policy

~

[London] Help Demonstrate AI Risks at Apollo

Apollo Research is hiring a 6-month, full-time Evals Demonstration Engineer (London-based, £7,500/month) to design and deliver demonstrations that translate technical AI evaluation findings into compelling formats for policymakers and other non-technical decision-makers. Applications are open until September 10, reviewed on a rolling basis (early submission encouraged). This role starts a contract role but strong performance could lead to a permanent role.

The role requires technical fluency (Python, Inspect framework, ability to run/modify evals) and exceptional communication skills to craft demos, visualizations, and live presentations that resonate with policymakers. Prior experience presenting to government or think tank audiences, plus creativity in choosing the right medium (interactive, video, report), are core. You’ll collaborate directly with Apollo’s evals and governance leads (Marius Hobbhahn, Charlotte Stix) and produce outputs like live demos, policy-oriented visuals, and blog posts. The role is in-person in London, and UK work eligibility is prioritized (though exceptional candidates elsewhere are invited to apply).

Demonstrate AI risks

~

[Remote] The Future of Life Institute seeks an editor and investigator

If you want to use storytelling craft to shape the world’s response to transformative technology, the Future of Life Institute is seeking a full-time, remote Editor to lead the creation of written, broadcast, and video content that raises awareness of both the risks and opportunities of advanced AI. Reporting directly to the Director of Communications, the Editor will shape everything from op-eds and blog posts to short films, PSAs, and large-scale media campaigns.

• Compensation ranges from $90,000–$190,000 depending on experience and geography. Applications are due September 8. Candidates should bring proven editing/writing experience in journalism, screenwriting, or related fields, strong knowledge of AI safety issues (including alignment, misuse, AGI risks), and the ability to communicate complex ideas in a clear, evocative way. Bonus points for top-tier publication credits, ghostwriting, or campaign management experience.
  

FLI is also is hiring a full-time, remote AI Safety Investigator (salary $90k–$150k + benefits) to document and analyze safety practices at the biggest AI companies, explain incidents to the public, and lead FLI’s flagship AI Safety Index. The role mixes research, field investigation, and communications —building networks inside corporations, rapidly analyzing incidents, and translating findings into clear metrics and visualizations that shape public and policy debates.

• They want someone self-directed and sharp at following leads, ideally with a background in journalism, research, and/or AI safety, plus the ability to communicate technical concepts clearly. Occasional Bay Area travel is required. Applications are due September 4.

Explore careers at FLI

~

[SF] Help Launch AI Safety Projects at the Center for AI Safety

The Center for AI Safety (CAIS) is hiring a Special Projects Associate and a Special Projects Manager in San Francisco to drive new initiatives at the frontiers of AI safety. In these roles, you’d work directly with CAIS leadership to scout opportunities, design project plans, manage budgets and timelines, and turn ambitious ideas into operational reality. With AI safety communications hitting millions and public curiosity accelerating, CAIS is looking to scale fast — and this is a chance to help shape that surge into impactful, mission-aligned projects.

Both positions are full-time, on-site and offer strong compensation ($100K–140K for Associate, $120K–150K for Manager). Requirements are a bachelor’s degree (advanced degree optional), 1–2 for Associate / 2–4 years’ experience for Manager in startups/ops/consulting/project management, proven ability to learn new domains quickly, and genuine interest in AI safety.

~

CAIS is also hiring a Finance Manager in San Francisco to take full ownership of financial operations across both its nonprofit arms — a 501c3 and a 501c4. This role would oversee everything from audits, tax filings, and nonprofit governance policies to payroll, expense systems, and donor reporting —working directly with leadership, legal counsel, and the operations team to keep the organization financially sound and compliant.

• The role offers $110K–150K, hybrid flexibility, and strong benefits. Candidates should bring 4+ years in finance or nonprofit accounting, with experience managing audits, budgets, and compliance processes; familiarity with Xero, Ramp, Rippling, Stripe, or similar platforms is a plus. Experience across both 501c3 and 501c4 entities is preferred but not required.

~

Lastly, CAIS is looking for a Director of Public Engagement in San Francisco to shape how the world understands the risks of advanced AI. You’ll lead multi-channel campaigns, craft narratives that resonate with diverse audiences, and serve as a visible spokesperson for one of the most influential organizations in the field.

CAIS is looking for someone with proven chops in campaign strategy, media production, and public communication—ideally someone who’s run large-scale awareness efforts before. The position offers $140k–$170k/year and strong benefits.

All applications are on a rolling basis.

Click for Careers at CAIS

~

[Berkeley CA] Build AI Demos with CivAI

CivAI, a nonprofit based in Berkeley, is hiring a Senior Software Engineer to create interactive demos that bring AI’s capabilities and risks to life. Instead of publishing papers, CivAI builds tangible products/demo that policymakers, journalists, and the public can use first-hand. This helps people intuitively grasp what advanced AI can and can’t do. CivAI’s work has already reached 60+ government offices and been featured on ABC, NPR, CNN, and WaPo.

This is a $150k–$200k full-time, on-site role for a generalist engineer. The team values both hacker spirit and strong UI/UX instincts. 5+ years of experience is preferred (though strong candidates with 3+ years will be considered). Applications are on a rolling basis.

Check out CivAI

~

[Remote-US] Run Projects at the Edge of AI and Biosecurity with RAND’s Meselson Center

RAND’s new Meselson Center is hiring an AI/Bio Research Project Manager (2-year term, $75k–$156k, hybrid/remote eligible) to drive high-impact work at the intersection of frontier AI security and biological risk reduction. As both AI and biotech risks accelerate and increasingly overlap, RAND wants a project manager who can keep ambitious research and policy projects moving fast and effectively.

This role involves coordinating interdisciplinary teams, manage research deliverables, support recruiting rounds, oversee budgets/contracts, and help run workshops that convene top talent. Requirements involve at least 4 years of project management/operations experience (a Master’s or PMP can substitute), strong organizational/communication skills, and the ability to juggle multiple high-stakes projects. Preferred locations are from DC, Santa Monica, Boston, or Pittsburgh, but remote is possible. Clearance eligibility is a plus but not required. Applications are open now on a rolling basis.

Look at this role

~

[London] Do operations at 80,000 Hours

80,000 Hours is a non-profit that’s helped thousands of people shift their careers toward the world’s most pressing problems and is now focused on helping people get careers on AI risk. Theirgrowing team (now ~35 staff) needs more operational backbone: from ops generalists to specialists in events, recruiting, people ops, office management, executive support, and video production. They’re also seeking an IT security and data privacy specialist to strengthen internal systems.

This is a bit meta, but perhaps the best way to help AGI is to do operations for the organization that helps coach people into AGI-related careers. Leverage! They’re looking for people who are organized, detail-oriented, clear communicators, flexible, and motivated by 80,000 Hours’ mission.

Compensation is £41k–£75k for generalist roles and £55k–£80k for the IT security role, with strong benefits. Roles are full-time, London-based by default (visa sponsorship possible), but some remote arrangements are considered. The deadline to apply is September 1.

Look at 80K ops roles

~

[Remote/Global] Run Operations That Power AI Safety Projects

Rethink Priorities is hiring a Special Projects Associate/Coordinator for a 5-month parental leave cover in a full-time remote setting to keep high-impact initiatives running smoothly. This is a $70–90k/year equivalent role (prorated) starting late September/early October 2025, with benefits and possible extension. You’ll work across time zones (UK/California overlap required) and serve as the operations lead for 2–3 projects advancing safe and aligned AI, from budgeting and compliance to HR support, contracts, and project management.

The job is for someone who’s comfortable with generalist ops work (finance, HR, compliance, project management), proactive in solving problems, and eager to support initiatives like launching new AI safety orgs, running fellowships, and incubating policy talent. Bonus points if you’ve touched US nonprofit finance/compliance. Applications are open now and judged on a rolling basis — ideal start date is Sept 29 (latest Oct 6, 2025).

Explore RP Special Projects

~

[Remote] Train on AGI Strategy with BlueDot Impact’s Course

BlueDot Impact is offering a free AGI Strategy Course designed to prepare people to shape the future of artificial general intelligence. BlueDot, a non-profit spun out of Cambridge that has already trained 5,000+ AI safety professionals, built this program to give participants the tools, frameworks, and community to engage with the highest-stakes policy, technical, and governance challenges of our time. Alumni have gone on to roles at OpenAI, Anthropic, NATO, the UN, OECD, and national AI directorates.

The course is virtual, free, and highly structured, with small peer groups, expert facilitators, and flexible pacing: either a 6-day intensive (5 hrs/day) or a 6-week part-time option (5 hrs/week). Every month a new round starts, with rolling applications. Participants will spend 2–3 hours preparing for each live discussion and then dive into guided conversations with other motivated peers. If you want to pivot into AI safety or policy—or level up from a related background—this is one of the few high-signal entry points with a strong track record of career impact.

Join the next cohort

~

The go-to for navigating emerging tech policy

The US government and allied institutions urgently need digital and scientific talent to keep pace with transformative tech. But the entry points are fragmented and opaque.

Emergingtechpolicy.org is a great one-stop guide for anyone who wants to break into public service careers at the intersection of technology and policy. Whether you’re a student mapping out your first steps, a technologist pivoting into policy, or a policy pro tackling new tech domains, it offers structured pathways, curated resources, and expert advice.

The content spans everything from career essentials (fit testing, resumes, networking) to policy institutions (Congress, think tanks, federal agencies, ARPAs, intelligence community). It also helps profiles real opportunities —internships, fellowships, full-time roles — and help demystify how to navigate the AI policy world, with a special focus on DC.

Explore Emerging Tech Policy

~

Like Punxsutawney Phil on Groundhog Day, a model release is a chance for us AI prognosticators to declare if we have six more weeks of AI winter.

This goes double for a model worthy of the “GPT-5” name.

Recall that GPT-2 came on the scene in 2019 astounding the world by being the first language model that could write coherent paragraphs and roughly stay on topic. GPT-3 then smashed GPT-2 by actually being useful, capable of following instructions to some extent and engaging in rudimentary reasoning when given a few examples of what you were trying to do. GPT-4 then took this to the next level, doing things like acing a bar exam and being a useful copilot for computer programming.

Since then, it’s been harder to shock the world, but I’d say that o1 and o3 also did so, by introducing a new paradigm of thinking prior to answering, and doing much better on expert-level questions in math, science, and software engineering. o3 in particular was shocking when demoed in 2024 December as it had benchmark results well beyond what was anticipated at the time, though the actual o3 rollout itself ended up being much slower and underwhelming relative to these sky-high December expectations.

Otherwise, we just have seen more incremental updates, with steady but gradual changes. This has led OpenAI to chase a model that could astound the world and be worthy of the GPT-5 name without much success. Rumor has it that all of GPT-4.5, o1, and o3 were at one point considered to be candidates worthy of the “GPT-5” name, but they all fell short of what OpenAI was dreaming of1.

But now we have GPT-5 live and accessible! Amazing! Is this the massive smash we were looking for?

Unfortunately, no. Model evaluation has become increasingly difficult. We generally rely on two tools — formal quantitative evaluations and vibes. When looking at this, it appears that GPT-5 isn’t a giant leap in intelligence. It’s an incremental step in benchmarks and a ‘meh’ in vibes for experts. But it should only be disappointing if you had unrealistic expectations — it is very on-trend and exactly what we’d predict if we’re still heading to fast AI progress over the next decade.

Most importantly, GPT-5 is a big usability win for everyday users — faster, cheaper, and easier to use than its predecessors, with notable improvements on hallucinations and other issues.

Let’s dig in.

Share

The benchmarks: exactly what you would predict

Formal evaluations (also known as ‘benchmarking’) are valuable for being objective, quantified, and repeatable — it’s easy to see exactly how much a model is improving. But their strength is also their critical weakness. Because evals work best on well-specified, easily verifiable tasks, they tend to measure only the tasks that can be neatly contained and easily scored. This is very different from the messy, ambiguous problems people face in the real world.

This issue is compounded by the fact that the tasks that evals focus on are also the tasks where reinforcement learning helps models the most — RL also thrives on well-specified and easily verifiable tasks. As a result, formal evals over-index on the tasks where models are already showing the most improvement, potentially suggesting models are stronger than they are if you evaluate more holistically. This makes it difficult to get a good grasp on just how much models are improving. Worse, we can’t rule out contamination from training data—models may simply memorize the test answers.

And when it comes to formal evaluations, it seems like GPT-5 was largely what would be expected — small, incremental increases rather than anything worthy of a vague Death Star meme. All forecasting is a matter of looking at past trendlines and then continuing them out, and it turns out GPT-5 basically lands exactly where these trendlines say:

• The SWE-Bench-Verified scores (a measure of software skill, as measured independently by EpochAI) are unimpressive, landing below Claude 4.1 Opus. This suggests that unfortunately for OpenAI, Claude Code may remain dominant for a bit longer.

• On METR’s time horizons task, a more advanced evaluation of software engineering skill, GPT-5 improved upon the state of the art in a way that is very in-line with what was predicted based on past trends2.

• On GPQA, a set of PhD-level science questions, GPT-5 comes in at basically the same range as Gemini and Grok when evaluated independently by EpochAI.

• On ARC-AGI and ARC-AGI-2, both a measure of general intelligence via symbolic reasoning, GPT-5 fails to outperform Grok 4. On FrontierMath, a measure of advanced mathematical reasoning, GPT-5 makes an advance, but in line with the trend.

• On Deep Research Bench, my favorite benchmark for researching and analysis performance, GPT-5 etches out a win by a very small amount, within the margin of error. Also it appears GPT-5 is good at some tasks but not others, a bit of a ‘jagged frontier’ that makes it a complement to previous winner Claude 4 Opus rather than a replacement.

I assume OpenAI finally just gave up chasing the big leap in intelligence and just decided that the pressure to release a ‘GPT-5’ was just too great. And we really need more and better evals to measure AI performance.

The vibes: meh?

The other evaluation method is far less formal — using human judgment, or what we might call vibes. Like a professional wine tasting, you get model connoisseurs to evaluate models in open-ended ways, throwing it novel, messy problems, and gauging its usefulness, creativity, and reliability. Vibes-based testing captures dimensions formal evals miss: how well a model can adapt, reason under uncertainty, avoid hallucinations, or handle complex multi-step instructions. The trade-off is that vibes are subjective, noisy, and prone to bias — different testers may walk away with wildly different impressions.

But vibes-based evals depend a lot on the quality of the connoisseur. One common vibes-based eval is LMArena, where many people vote across a blind taste test which model answers a particular input best. Here, GPT-5 has a slight #1 edge over Gemini, which had been dominating the leaderboard for quite some time. However, it’s important to realize that these days all models are just too good for the typical user’s use case and LMArena doesn’t see models challenged with the best. Typical human voters — the same people who can’t tell the difference between a $5 wine and a $500 wine — can’t tell the difference between a good model and a great model. LMArena voters just do not have the requisite skills to distinguish model quality in our current advanced age.

Another form of vibes-based evals I dislike are a lot of people who got early access to the model, but are uncritical hype-sters out there who would say the model is good no matter what. Instead, for the people I have been following that have a good track record of model evaluation within their domains, GPT-5 is more “meh”. In line with the benchmarks, the current consensus seems to be that GPT-5 is an incremental improvement.

For example, Daniel Litt, a professional mathematician and model tester, doesn’t find GPT-5 to be a noticeable improvement in math work. Jesse Richardson, the best election forecaster in the world, doesn’t find GPT-5 to be good at election forecasting. Hopefully more solid taste tests will come in soon as people have more time to play with the model.

The paradigm of AI is different now, but key questions of AI progress are still unanswered

Back in the good ol’ days of 2023, the paradigm of AI was that you took a model and then made it bigger. This embiggifying happened by adding more data and more compute at training time, something confusingly called pre-training3. GPT-3 was big at the time, but GPT-4 was bigger, and thus better. The thought was that if you took GPT-4 and then made it even bigger, you would get GPT-5 and it would be even more capable — a whale.

GPT-3.5, GPT-4, and GPT-4.5 all followed an implicit pattern where each increase was said to be a step-change of 10x more effective compute. Given this, one would expect GPT-5 to be a base model like GPT-4.5 but with 10x more compute. But this seems unlikely, because the amount of compute needed to build a ‘10x GPT-4.5 model’ shouldn’t yet be available until the end of the year after the completion of the first phase of Stargate.

Instead, ‘2024/2025’-era AI is working out a bit differently. We’ve moved on from the paradigm where the main way to make a model more capable was to train it for more compute and a better model came from some combination of using more GPUs than before, having GPUs that were more efficient than before, pre-training on those GPUs for a longer amount of time than before, or developing better algorithms and data that made your pre-training more efficient.

These days, instead of one way to improve a model, there are now four:

• You can pre-train more (as mentioned)

• You can do post-training reinforcement learning, where you show a model a bunch of problems and solutions and then train it to solve those problems

• You can do more inference-time compute, where the model spends more compute/time thinking when posed the question, considering more possibilities

• You can improve the scaffolding, where you give the model more tools and affordances to better answer questions, like access to web search and a calculator

Each of these can be scaled separately and improve the model quality. It seems like old-school pre-training has kind of died down over 2024-2025, but like a rocket with multiple stages the ‘pre-training’ stage 1 has been depleted but the stage 2 rockets around post-training reinforcement learning, inference-time compute, and scaffolding are still going strong. This is how AI progress has continued over 2024/2025 seamlessly despite the paradigm completely changing.

GPT-5 seems to be bridging this. GPT-5 is essentially two models glued together, a ‘base model variant’ that seems good at creative writing and quick answering and a ‘reasoning model variant’ that answers math, science, and software questions with much more depth. These models are then guided by some sort of dynamic router that figures out which model ought to be used based on your query. This is a great potential advance in usability if it works well, but leaves confusion about how the four ways of advancing AI progress are going.

Unfortunately, if you were hoping that GPT-5 would contain the answers that help us answer fundamental questions about AI progress, we really didn’t get much information. As a data point, GPT-5 confirmed all the pre-existing trends, suggesting that AI progress is moving exactly as fast as I thought before (median AGI arrival date of 2033). GPT-5 is neither a disappointment calling for us to think AI progress is slower nor an excitement that suggests we should really speed up our timelines.

What GPT-5 does do is rule out that RL scaling can unfold rapidly and that we can get very rapid AI progress as a result. It was thought at the end of December that the demo of o3 was an indication that reasoning models might be capable of taking off quickly and that we could get lightning fast AI progress over 2025. The slow rollout of o3 for public use suggested there were barriers to that. If ultrafast scaling was true, GPT-5 should have leapt. It didn’t — which weighs strongly against that hypothesis.

But many questions are still unanswered. For example, I’m still confused about whether good old-fashioned pre-training is dead. GPT-4.5 was widely seen as a disappointment given that it did not improve upon the state of the art, but it wasn’t meant to, as it wasn’t a reasoning model like o1 that made good use of the other three ways to scale AI models. I was always curious to see what GPT-4.5’s power as a base model could be if it was given reasoning capabilities and I had hoped GPT-5 might be that demonstration. Instead, based on the speed of response, it seems that GPT-5 is actually not be using GPT-4.5 as a base model but instead something smaller and faster (probably 4o or 4.1). This is curious — either OpenAI wanted to forgo the ability of 4.5 for speed or having 4.5 as the base model is not helpful. If the latter, that is a strong suggestion that base model good old-fashioned pre-training scaling is no longer giving sufficient returns.

I’m also confused about the returns to scaling post-training reinforcement learning and inference-time compute. We can see nicely how GPT-5 is making small incremental improvements in math, science, and software but will these improvements continue with more work on post-training reinforcement learning and inference-time compute? Could OpenAI just finish building Stargate, crank the numbers on all four ways of scaling by 10x, output GPT-6, and see strong returns? We don’t yet know. More importantly, how well does this reinforcement learning paradigm — which focuses mainly on easily verifiable domains — generalize to much harder to verify domains like writing, research, and analysis? So far, my experience is that the quality of writing actually goes down when GPT-5 thinks more, which is an interesting trade off. Unfortunately we do not have good evals for these sorts of things, and I’d like to see more. Does GPT-5 improve on playing Pokémon? Is GPT-5 better at writing good Substack articles? How much closer is GPT-5 at automating a variety of current professional work? We don’t really have any new information.

I’m also confused about how advances in AI computer use are going. When will we get AI agents that can successfully do a variety of personal assistant tasks for you, such as booking flights and hotels largely autonomously or doing budgeting? This seems potentially a matter of both increasing capabilities and intelligence across the model but also in scaffolding by helping the model more. Despite OpenAI Operator’s launch in January and then Agent last month, the progress here still feels underwhelming. Will GPT-5 be better able to use a computer than its predecessors? So far we don’t know but my guess is given how this emphatically was not part of any demos or marketing for the GPT-5 product that the advances are not yet in.

Delivering for the people instead of the AI elites

Instead, what might be the case with GPT-5 is that they are delivering less for the elite user — the AI connoisseur ‘high taste tester’ elite — and more for the common user. Recall that 98% of people who use ChatGPT use it for free4. All of these users will now get a more powerful base model to experience (though unfortunately it seems that the free tier does not engage in extended thinking5).

And the model they get will be much easier to use — you just use “GPT-5” and you don’t have to worry about “4o” versus “o4” which are apparently completely different things, the model will just figure out the reasoning or not for you and pair you with the model that best suits your needs6. The hallucination rate is notably decreased. The model now engages in notably less deception than the o3 model that told a lot of lies. The writing quality feels better than before. It’s ultra fast at responding — easily the fastest model on the market now. The model still generates images, so you can still ghiblify on demand.

Sure GPT-5 may not deliver the next generation of intelligence, but does that matter compared to the next generation of value to the typical consumer? The honest truth is models are smart enough for a lot of use cases — speed, cost, and reliability is a bigger bottleneck for a lot of enterprise AI use cases than intelligence. GPT-5 is delivering where it matters.

You can see this in the revenue stats. While ‘timelines to AGI’ may have revised downward somewhat over 2025, the amount of revenue AI companies are projected to make have revised upward a lot over the same time period. Entering 2025, OpenAI + Anthropic + xAI were projecting a combined ~$17B of annualized revenue as of the end of 20257. Now my forecast is almost twice that at expecting a combined $32B annualized revenue at the end of 2025. The OpenAI valuation was $157B at the end of 2024 and now potentially is $500B just eight months later. GPT-5 will presumably take this to new heights. Who cares how the model does on benchmarks if it makes bank?

GPT-5 is not the GPT moment we once expected — it’s not even the o1 moment. Instead, OpenAI is doubling down on speed, usability, and revenue. Intelligence progress continues on predictable trendlines but GPT-5 delivers on the bottlenecks for most real-world use cases — cost, latency, and reliability.

The AI elites may keep waiting for fireworks but for 98%+ of actual users, GPT-5 is the best ChatGPT yet.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022.

There's nothing like an otherwise very normal and expected Chinese AI model release to make people lose their minds, misunderstand the technical landscape, and thus spin misguided narratives. First it was DeepSeek. Next it was Manus. Today, it seems to be Z.ai's GLM-4.5.

Specifically, a recent Wall Street Journal op-ed by Aaron Ginn, “China's Z.ai and America's Self-Defeating AI Strategy”, argues that Z.ai's GLM-4.5 is proof US-led controls have backfired spectacularly by enabling rather than constraining Chinese innovation. This article is important, as it was retweeted by David Sacks, Trump's AI and Crypto adviser. This confusion is dangerous because following Ginn’s advice would undo the very thing that is keeping the US ahead in the AI race.

Contra Ginn, Z.ai and Huawei are actually proof that the US export controls are working, and the path to ensure US AI dominance is to tighten export controls further. A careful examination of the evidence — from benchmark data to industry reports to admissions from Chinese executives themselves — reveals that Ginn has it completely backwards. Chinese AI continues advancing but faces fundamental constraints. Export controls impose real and growing costs. And manufacturing gaps are widening, not narrowing. Let’s dig in.

Share

Deflating the hype around GLM-4.5

As we've seen with DeepSeek and Manus, Chinese AI developments follow a predictable hype cycle that distorts serious analysis of technological competition. GLM-4.5 is no different.

The way this hype cycle works is that bold initial claims about revolutionary Chinese breakthroughs generate strong headlines, before being debunked by technical reality checks that unfortunately receive far less attention. For example, Ginn writes in his article that “China's DeepSeek shocked the global AI community in January” — but DeepSeek emphatically wasn’t a shock to the global AI community but rather widely tracked for months ahead of time. Ginn also writes that DeepSeek built “a frontier model at a fraction of Western costs”, but this needs clarification — DeepSeek achieved results in line with the Pareto frontier of model cost-efficiency at the time, and now has lost ground to leading US models.

Now we're doing the hype round again with Z.ai. Doubling down on incorrect claims about DeepSeek, Ginn moves on to make incorrect claims about Z.ai, claiming that DeepSeek “has been outdone by a Chinese company subject to US sanctions”, referring to Z.ai's GLM-4.5 model launched a week ago.

Z.ai is a Chinese AI company formerly known as ‘Zhipu AI’ before changing their name last month to reflect a more international focus (no relation to Musk's xAI). The “US sanctions” here refers to the fact that Z.ai is on the US Entity List, a list of companies prohibited from importing US tech, because the US accused Z.ai of assisting the Chinese military.

So, is GLM-4.5 good? Ginn boasts that GLM-4.5 “matches or exceeds Western standards in coding, reasoning and tool use”, but GLM-4.5's own published benchmark scores show GLM-4.5 worse than DeepSeek, Anthropic, Google DeepMind, and xAI models at nearly all the benchmarks listed. And this is the best possible light for GLM-4.5 — because GLM-4.5 is still so new, there currently are no independent third-party benchmark scores so we don't know if they are inflating their scores or cherry-picking only their best results. For example, DeepSeek's benchmark scores were lower when independently assessed.

Regardless, GLM-4.5 themselves admitting to being generally worse than DeepSeek’s latest model means that we can upper bound GLM-4.5 with DeepSeek’s performance. And when measured against multiple independent benchmarks, DeepSeek is found to be 4-10 months behind the US state of the art. Thus GLM-4.5 is not currently a match for US models.

You might then point to GLM-4.5’s impressive model size and cost. Yes, it is impressive that GLM-4.5 is a small model that can fit on eight H20s, as Ginn points out. But OpenAI's recently launched 'Open Models' also out-benchmark GLM-4.5 despite running on even smaller hardware, such as a single 'high-end' laptop. And Google's Gemini 2.5 Flash has a similar API cost and similar performance as GLM-4.5 despite coming out several months earlier. This also ignores the fact that GLM-4.5 handles only text, while major US models can also handle images, audio, and video.

Similarly, I'm unsure why Ginn is so impressed that z.ai “projects it will have millions of downloads and millions of dollars of revenue in 2025” when OpenAI is hitting nearly 700 million weekly active users and projects $20B in annual revenue, or ~20,000x as much as z.ai. Compared to a Chinese population of 1.4 billion and a world population of over 8 billion, we're very far from widespread international adoption let alone the full-on “dependency” of Chinese AI models that Ginn describes. Like DeepSeek, GLM-4.5 is impressive but not an imminent threat to Western dominance.

A reality check on Chinese chips

GLM-4.5 isn't the only thing that Ginn is hyping — he's also very excited about China's domestic manufacturing of computer chips, mainly from the companies Huawei (equivalent to US's Nvidia) and SMIC (equivalent to Taiwan's TSMC). To hear Ginn tell it, US export controls have pushed China to develop their own chips and this has gone so well that China will be exporting these chips globally and rivaling the West. Per Ginn, “Huawei's GPUs are quickly filling the gap left by the Biden administration's adoption of stricter export controls.”

But this gets the facts about Huawei and SMIC very critically wrong. Huawei isn’t filling any gap at all. Perhaps the most striking contradiction to Ginn's narrative comes from Huawei itself. In a recent interview with People's Daily, Ren Zhengfei, Huawei's founder, explicitly stated that the US “overestimates” his company's chip capabilities and that Huawei's Ascend AI chips “lag the US by a generation.”

Tests confirm this, showing Huawei's latest Ascend 910C chips suffer from overheating, crashes, and buggy software. They lack support for modern AI features like FP8 precision computing, which enables more efficient training. Huawei's workarounds involve bundling multiple inferior chips to achieve higher performance, but this brings increased power consumption, heat generation (leading to overheating), and system complexity. There also is the matter of CUDA, the software stack that makes Nvidia chips so much easier to integrate into AI, which is significantly better than Huawei's software, CANN.

When it comes to manufacturing the chips themselves, there are even more issues. While Taiwan's TSMC achieves 90%+ yields on mature 7nm processes, China's SMIC reportedly struggles with yields closer to 50% on their '7nm' process (which itself performs closer to TSMC's 10nm). This means for every 100 chips they attempt to manufacture, only about 50 are functional — a massive inefficiency that makes Huawei chips substantially less economically viable.

This is because China lacks access to extreme ultraviolet (EUV) lithography — the technology required to produce the most advanced semiconductors. These machines, costing approximately $200M each, represent perhaps the most complex technology ever created, requiring the combined scientific output of the Netherlands, Germany, Japan, and the US.

Ginn reports that “China's foundry capacity has vastly surpassed Washington's expectation, and China is shipping chips abroad several years ahead of schedule”. Ginn offers no source for this claim, a surprising omission for such a significant assertion. It’s also false — the US government's own assessment from last month is that Huawei can only manufacture 200,000 chips this year, a number that is insufficient for fulfilling even the Chinese market demand, let alone the global market. It’s also a number far below the millions of chips TSMC and Nvidia produce annually.

Similarly, Ginn alleges that “Beijing is increasing international dependency on its models and hardware.” While this certainly has been true in the past for fiber optics and 5G tech, there is no evidence of any international sales of Huawei AI chips outside of China. There was one planned sale within Malaysia that was reportedly blocked by the US government, but beyond that single aborted transaction, there are literally no documented sales and even that sale didn’t go through. Ginn's claims of international dependency on Chinese AI hardware are not supported by any available evidence.

In fact, Huawei hasn't even managed to create dependency within China, with Nvidia still having a majority of the market and major Chinese tech companies have yet to make substantial purchases of Huawei chips, instead preferring Nvidia's offerings. The US Commerce Department's May warning that using Huawei's advanced AI chips without authorization could violate export regulations has further chilled demand, particularly from companies with international operations.

At the end of the day, China was always going to innovate. China was already committed to chip self-sufficiency years before to the US export controls began. The CCP government continues to use their vast buying power to promote Huawei chips even when it doesn’t make economic sense to do so. Export controls are not needed to encourage Chinese indigenization, as the CCP can continue to encourage it by fiat.

Huawei chips will improve over time, and SMIC’s manufacturing capabilities will surely advance. There will eventually be many good Chinese-made AI chips. But the current reality is that Chinese hardware remains substantially behind and loosening US-led export controls would only help China reach US parity faster.

The Chinese AI ecosystem runs on Western tech

This brings us to the central conceit of Ginn’s analysis that “Washington's tack so far has been to try to limit Chinese entities’ access to advanced hardware” and that GLM-4.5 and Huawei are proof that this has failed. But there is no evidence that GLM-4.5 runs on Huawei chips, and Ginn instead mentions GLM-4.5 running on Nvidia H20s (which notably they shouldn’t have legally due to being on the Entity List). DeepSeek was also trained on Nvidia chips. Chinese AI still remains dependent today on American hardware, just less capable versions of it, and this is a good thing.

According to data from Epoch AI, when we look at notable Chinese AI models released between 2017 and 2024, over 90% of language models were trained on Western hardware. The first model reportedly trained entirely on Chinese hardware was not released until January 2024, after several years of training models on Western hardware.

This gets even more damning for China when you zoom out. According to recent RAND analysis, the US controls 77% of global AI compute capacity while China has only 12%. And according to a recent IAPS and CNAS analysis, even that Chinese compute depends heavily on Western technology.

The most damaging breach came in September 2024 when TSMC, lacking basic due diligence, illegally produced 3 million advanced chip dies for Huawei through a Chinese proxy company. This single violation provided China with compute equivalent to approximately 1 million Nvidia H100s. It’s these TSMC-produced dies are what actually power Huawei's ‘domestic’ Ascend 910B and upcoming 910C chips.

And prior export control implementation suffered from a pattern of failures that need fixing. The 2022 controls contained specification errors that allowed Nvidia to create the A800 and H800 chips that enabled DeepSeek. Chinese firms stockpiled years of high-bandwidth memory after the industry leaked upcoming restrictions.

Combined with billions of dollars in smuggled Nvidia chips and billions of dollars of legal imports of the Nvidia H20, and the vast majority of China’s AI ambitions are built on Western technology. To hear Ginn tell it, China has no access to American tech and built everything themselves, but in actuality ~85% of Chinese training compute and ~95% of Chinese inference compute comes from the West:

But these implementation failures don't negate the strategic value — they show the need for better enforcement and the power of where export controls could go. Even with these leaks, export controls have substantially slowed China's progress and maintained America’s lead. As Liang Wenfeng, CEO and founder of DeepSeek, said in an interview: “The problem we are facing has never been funding, but the export control on advanced chips.”

Ginn is right that DeepSeek and Z.ai were forced to be efficient because of lack of access to the best US chips but that doesn't mean they wouldn’t prefer more advanced hardware and their models likely would be much better if they were granted such access. Letting China buy as many advanced GPUs as they want obviously closes the US-China AI gap more quickly.

This also matches the CCP government’s own behavior. While the fast follower strategy has served China well in other industries, it becomes more challenging without reliable access to leading technology. If it were indeed the case that the US export controls have accidentally supercharged the Chinese AI and semiconductor industries, then China would be putting tariffs or bans on US chips. They don’t. Instead, China takes every chance it can to complain about export controls and negotiates to ease them — a clear signal that the controls are working as intended.

We also must keep in mind that the US export controls are playing a longer game. We're barely one new chip generation into the current export controls, and the export controls will bite harder as chip technology continues to outpace China. The state of the art of chips in the US have already shifted to the B200 chip and Chinese companies won’t have any B200s except via smuggling. This will allow American models to be even more powerful at the same level of capital in a way that Chinese companies won't be able to match. By the time we get to the next generation of Nvidia chips after the B200, the gap between China and the US should be even larger.

The evidence suggests export controls are achieving their intended effect: forcing inefficiency while maintaining technological advantage, in a way that compounds over time. While Chinese companies work with inferior domestic chips or smuggled alternatives, US companies deploy massive compute clusters using next-generation hardware at large scale. OpenAI's Stargate project envisions $100 billion in annual compute spending using chips China cannot legally or practically access at a capital scale that Chinese tech companies are not matching.

And every Z.ai engineer working on making their AI work with inferior chips is one not pushing the frontier of AI capabilities. Every dollar spent on smuggling premiums or inefficient power consumption is one not invested in research. The uncertainty also undermines long-term planning — companies cannot build multi-year research programs around smuggled chips or inferior domestic alternatives.

How to win with US AI abroad

One thing Ginn is right about is that President Trump and his AI Action Plan are right to call for “scaling supply and adoption abroad” and “exporting American AI and hardware while cutting regulations that slow production at home”. Additionally, Ginn is correct to say that “Every U.S.-branded LLM shapes AI norms globally. Success comes from ubiquity of platforms, not exclusion or restrictions.”

But global exports should emphatically not mean handing US technology directly to our adversaries. As Mark Beall, the former Director of Strategy and Policy at the DoD Joint Artificial Intelligence Center during Trump’s first term said, “the fact that the Chinese military can freely buy, steal, download, and weaponize American technology represents a dereliction of duty that would have been unthinkable during the Cold War.”

Ginn wants us to believe that Z.ai's GLM-4.5 proves export controls have backfired - that by restricting China's access to advanced chips, we've somehow made them stronger. This is precisely backwards and the actual facts tell a different story.

The real danger isn't that export controls have failed. It's that we might abandon them just as they're starting to compound. Following Ginn's advice would be like lifting sanctions on the Soviet Union in 1985 because they built a decent tractor.

Trump's instinct to export American AI globally is right. But that means to allies and partners, not adversaries. Tighten enforcement, crack down on smuggling, make sure TSMC can't ‘accidentally’ produce millions of chips for Huawei again.

The evidence is clear. Export controls are working. Now is the time to double down, not give up.