This is a guest post written by Erich Grunewald, a prior “Power Law” contributor who writes on AI and compute topics at The Substrate. This post is crossposted from there. Check it out!

~

AI chips are being smuggled1 into China. How does this happen? That is, how are the smugglers able to do it?

You might, for example, expect that sales of these chips could be vetted, and shipments tracked, by US authorities. But the relevant US agency is badly understaffed: it cannot possibly vet every sale of AI chips, let alone follow up after every sale to ensure the chips have not been smuggled. So enforcement is largely left to the companies that sell and resell the chips. You might then suppose that these companies would vet their customers and monitor their distribution networks to ensure none of their chips are diverted, in order to comply with the law. But they are not strongly incentivized to do the due diligence needed to prevent most smuggling.

In this post, I’ll explain (1) why AI chip smuggling matters, (2) how AI chips and servers are sold and distributed, (3) what due diligence the exporting companies do, and (4) why those efforts often fail to prevent smuggling. In a nutshell, AI chips and servers are distributed through different channels, usually passing through multiple intermediaries across multiple countries. While most companies selling these products perform sufficient due diligence to comply with the law, others are outright negligent, and overall these efforts are not sufficient to prevent most smuggling.

How big a problem is AI chip smuggling?

Smuggling is a pretty big problem, though not serious enough to make the AI chip controls ineffective. In a new report, Epoch AI says that

between 290,000 and 1.6 million H100-equivalents (H100e) were smuggled to China through 2025. Our median estimate of 660,000 H100e would be roughly a third of China’s total compute.

US authorities recently indicted three people, including two insiders at Super Micro, a major American AI server builder, for moving $2.5 billion worth of AI servers to a shell company for diversion to China, the largest-ever export control violation in dollar terms, as far as I can tell. Over the past year, the US has announced six prosecutions for smuggling AI chips to China, totaling about $3 billion worth of NVIDIA products. And before that, there were many reports, rumors, and analyses pointing to large-scale smuggling of AI chips.

I don’t think smuggling is a large enough problem to make the AI chip export controls ineffective overall. The controls have likely played a large role in maintaining the US lead over China in AI and may even have widened the gap in frontier performance. For example, a recent benchmark evaluation by the Center for AI Standards and Innovation (CAISI) suggests that Chinese models trail US models by about eight months, with the lag growing by about three months per year.2 And you can make a strong case for prioritizing efforts to shore up semiconductor manufacturing controls over efforts to address AI chip smuggling. But smuggling is still a serious and urgent problem.

According to CAISI, the best Chinese model today is DeepSeek-V4. DeepSeek has reportedly used thousands of smuggled NVIDIA Blackwell chips. I think it’s likely that DeepSeek-V4 was trained on NVIDIA chips.3 If DeepSeek didn’t have access to smuggled NVIDIA chips, it would likely have had to either rely on renting cloud access from chips installed outside China (which is risky from DeepSeek’s perspective because the US government could revoke remote access to those chips) or use indigenously made AI chips like Huawei Ascends to train smaller, weaker models. DeepSeek reportedly tried to train V4 on Ascend chips but abandoned the attempt after a failed training run, reverting to NVIDIA for training while supporting Ascend for inference.4 So even though smuggled chips account for only about 3% of the global compute stockpile, they represent about a third of China’s compute, which is why smuggling still matters.

AI chips are distributed through many intermediaries

The basic smuggling playbook works like this. A smuggler buys AI chips through a shell or front company (often in a third country5 but sometimes in the US), relabels them as some other product, and ships them to China through ordinary shipping services. (For example, in several cases, smugglers appear to have bought AI servers through front companies set up to look like Singaporean, Malaysian, Thai, and Indonesian neoclouds.6) Once a smuggler has the chips, getting them into China seems quite easy, so I’ll focus here on how smugglers get hold of them in the first place.

AI chips routinely pass through multiple actors and countries before they are installed and used in data centers. We can refer to all actors who sell AI chips as “AI chip sellers”. They are:

• AI chip designers, such as NVIDIA and AMD. These companies design AI chips, but they do not manufacture anything. NVIDIA holds an estimated 80-95% of the AI chip market share.7

• AI server builders (also known as original equipment manufacturers, or OEMs), such as Super Micro and Dell. These companies buy AI chips from AI chip designers and design and manufacture AI servers, each typically containing four to eight AI chips. The servers are shipped to data centers and installed in cabinets. Notable AI server builders include Super Micro (US, about 10% of NVIDIA’s revenue), Dell (US, ~3%), Wiwynn (Taiwan, ~3%), Lenovo (China/US, ~1%), Gigabyte (Taiwan, ~1%), Hewlett Packard Enterprise (US, <1%), and Inspur (China, <1%). (Why do these companies not make up more of NVIDIA’s revenue? That’s because AI chip designers like NVIDIA often sell chips directly to hyperscalers and neoclouds. In such cases, though the servers are assembled by original design manufacturers (ODMs), the transaction occurs directly between NVIDIA and the end customer. Hyperscalers and neoclouds make up about half of NVIDIA’s revenue.)

• Distributors, such as TD Synnex and Ingram Micro. These large, multinational companies are intermediaries that handle logistics and warehousing of AI chips and AI servers. The key distributors are TD Synnex (<1% of NVIDIA’s revenue), Ingram Micro (<1%), and Arrow Electronics (<1%), all headquartered in the US but with extensive operations worldwide.8

• Resellers. These companies, which are numerous and often small and local, buy AI chips and servers from distributors and sell them to end customers in the country or countries where they operate.

Before being sold, AI chips are made. For example, most of NVIDIA’s AI chips are designed by NVIDIA in the US, fabricated by TSMC in Taiwan, packaged with high-bandwidth memory by TSMC in Taiwan, tested by KYEC in Taiwan, and then either (a) assembled into SXM modules by Foxconn in Taiwan and mounted onto HGX baseboards by Foxconn and Wistron in Taiwan, or (b) assembled into PCIe cards (i.e., accelerators) by Foxconn, and possibly others, again in Taiwan.

From there, the chips9 can get to their end customers in a few different ways:

1. AI accelerators can be sold to minor end customers through distributors and resellers. In the simplest case, a company like NVIDIA sells AI accelerators to distributors, who in turn sell them to resellers, who then sell them to end customers. The end customer can then use the accelerators however they wish, for example, by installing them themselves in workstations or servers customized to their preferences. However, this pathway is quite rare, as most customers want AI servers ready for installation in data centers.

2. AI servers can be sold to minor end customers via distributors and resellers. For example, NVIDIA may sell AI chips (usually in the form of SXM modules mounted on baseboards) to Super Micro, which builds AI servers using them and sells these servers to a distributor. The distributor then sells to a reseller, which sells to an end customer.

3. AI servers can be sold to major end customers directly by AI server makers. Hyperscalers and neoclouds often purchase very large quantities of AI servers directly from server makers, bypassing distributors and resellers. For large orders of NVIDIA servers, this often involves a three-party negotiation among the customer, the server maker, and NVIDIA.

4. AI servers can be sold to major end customers directly by AI chip designers. In the case of NVIDIA, hyperscalers and neoclouds can buy AI servers directly from NVIDIA instead of from server makers. (The NVIDIA-designed servers are called DGX, while the OEM-designed servers are called HGX.) As mentioned above, these servers are built by ODMs for NVIDIA.

Most AI chips are likely sold to major end customers like hyperscalers, either directly from NVIDIA or directly from AI server makers. However, most smuggling likely involves smaller order volumes, with bad actors buying AI chips and AI servers either from AI server makers or from smaller resellers. So bad actors don’t purchase AI chips directly from AI chip designers like NVIDIA. Of the six cases that have been prosecuted by the US:

• Four involved smugglers buying AI servers and HGX baseboards from AI server makers (Super Micro and Lenovo)

• One involved smugglers buying AI chips and AI servers (the latter made by Hewlett Packard Enterprise) from an Alabama reseller

• One involved intermediaries storing goods from multiple suppliers, including SXM modules from a Massachusetts reseller and HGX baseboards from Lenovo (the same Lenovo purchases mentioned above)

These are all cases in which the smugglers were at least partly based in the US and procured AI chips from US sellers. Most smuggling likely involves actors based abroad who buy from local sellers or import from US sellers, but these cases are harder for US authorities to detect and prosecute.10

How AI chip sellers (try to) vet their customers

AI chips are export-controlled, meaning they cannot be sold to just any customer. For example, it is illegal11 to sell them to companies headquartered in China or listed on BIS-maintained lists, such as the Entity List or the Military End-User List. So when selling AI chips, companies need to conduct due diligence to ensure they are not selling to any such customer.

Most AI chips are likely sold by AI server makers, and a lot of smuggling seems to involve smugglers buying from server makers, so the remainder of this section focuses on them.12 From the point of view of AI server makers, the sales process lasts months and roughly follows these steps:

1. Talk with the customer about their requirements

2. Run software tools that gather data about the customer’s environment (i.e., data center)

3. Negotiate prices (changing server components and/or order volumes if necessary)

4. Prepare a contract (stipulating who the end customer is, where the servers are to be shipped, and who will install the servers)

5. Do due diligence

6. Build, box, and ship the products

Server makers track items at the serial number level using enterprise resource planning (ERP) software (e.g., SAP or Oracle). AI chip designers and distributors likely do too, though I’m not sure to what extent resellers do so (since they are smaller, less resourced, and less well-organized). When an order is placed, the salesperson enters it into the ERP system, and it is then routed to the appropriate factory or factories responsible for assembling, testing, boxing, and (once the order is paid) shipping the hardware to the customer. Server makers also have some visibility into their partner distributors’ inventory, since distributors typically hold a lot of the parts used by server makers. These parts (e.g., AI chips) can then be sent to the server maker for assembly when a customer makes an order via a reseller.13

From a smuggling perspective, we’re mostly interested in the due diligence step of the sales process. First, some definitions:

• Compliance is overall adherence to regulations

• Due diligence is the process of investigating and assessing risk related to a transaction, aimed at ensuring compliance

• Know Your Customer (KYC) processes are part of due diligence in which a company verifies a customer’s identity and assesses their risk level. There are several risks that sellers aim to avoid here, not only legal ones (e.g., by selling to a sanctioned customer) but also commercial ones, such as ensuring the customer can pay (counterparty credit risk).

AI chip designers, server makers, and distributors do fairly extensive compliance checks, but these checks seem mostly (and reasonably) aimed at (1) complying with the law, and (2) ensuring they get paid. AI chip sellers seem to differ a lot in both what they do to stay compliant and how effectively they do it, but they typically:

• Conduct KYC checks to ensure the prospective buyer is not a shell or front company and can pay for the order. That means the seller asks the company for information, e.g., legal name, corporate structure, location (including county/region), date of registration, affiliations, and operational history (e.g., bank statements, balance sheets). The seller will also, at least sometimes, ask local commerce ministries for information, check trade registries, and/or do other due diligence to verify some of that information.

• Ask the prospective buyer to provide a signed end-user certificate detailing how the purchase fits their business and how, where (down to the county/region level), and by whom14 the products will be used. A former HPE salesperson told me that “the primary way companies prevent chip smuggling is to ensure that we know the end destination of a system before we accept and ship an order”, but added that most of what determines the destination “is what the customer or the partner tells us”.

• Screen against lists (the Entity List, the Unverified List, the Military End-User List, and a few others)

• Check for BIS’s red flags

• Request a license from BIS if necessary

For large companies like NVIDIA and the server makers and distributors discussed here, these due diligence processes are largely handled by a centralized compliance team. (But salespeople also have some responsibilities; for example, according to a former Super Micro employee, its sales reps attended mandatory weekly meetings drilling them on export compliance requirements, though these sessions did not seem to prevent smuggling.) The compliance team will usually be located in the US, and end-user certificates and other documents are typically sent to it for approval, even for transactions handled by salespeople outside the US. The compliance team also does internal audits and trains the company’s sales, legal, finance, and shipping staff. In the $2.5 billion insider case, Super Micro’s compliance team did notice a fast-growing customer’s anomalous order volume and ran successive audits, although those audits were allegedly sabotaged from within the sales organization by senior Super Micro employees who arranged “friendly” auditors, falsified the customer’s data center lease agreements, and arranged warehouses with thousands of dummy servers to (successfully) subvert a Super Micro compliance inspection.

Sometimes, part of the due diligence process is carried out by third parties or through tools and data services they provide. These include AEB, e2open, Descartes, Dow Jones, Dun & Bradstreet, Kharon, LexisNexis, Sayari, Strider, Thomson Reuters, and WireScreen. These tools can automate parts of the due diligence process and are integrated with the ERP system. These third parties can also help validate end-user certificates, conduct background checks on buyers, perform list screening, and more.15

Some AI chip sellers seem to be happy to stick to these common activities, but others also do fairly extensive post-sale monitoring, e.g., one server maker (according to a former employee I spoke with) would also periodically follow up with the customer on a call to verify that the chips are used as intended; require the end-user certificate to be renewed every six months; negotiate the right to perform on-site inspections for large or high-security sales; and occasionally redo the list-based screening to check that the customer hasn’t been added since.

NVIDIA and its partners also carry out on-site inspections, at least occasionally.16 As mentioned, in the $2.5 billion Super Micro case, smugglers fooled the compliance team with insider help. In another case, the seller’s employees visited a data center where the chips were installed, although it turned out the smuggler had only temporarily installed them to fool the inspectors; after the inspection, the chips were removed and shipped to China.

AI chip sellers aren’t incentivized enough to stop smuggling

AI chip sellers usually only conduct due diligence for their immediate customers; they rarely conduct due diligence for their customers’ customers, or follow up to see what happens to the AI chips after they’ve been sold and shipped (with the notable exception of the on-site inspections mentioned above).

There are two important dynamics here that both enable AI chip smuggling:

• Sometimes, major AI chip sellers (notably, Super Micro) do not conduct sufficient due diligence even for their immediate customers. This can lead them to sell AI chips directly to smugglers. I’ll discuss this further below.

• Sometimes, major AI chip sellers do conduct sufficient due diligence for their immediate customers, but downstream sellers don’t. That is because, by the time the chips are sold to a smuggler (meaning someone didn’t do a good job of due diligence), they have typically passed through multiple hands, often ending up with resellers or other middlemen in countries such as Malaysia, Singapore, or Thailand.

AI chip sellers in third countries are far less incentivized to comply with US law, since it’s harder for US authorities to detect compliance failures outside the US and to prosecute illegal activities that occurred abroad, especially when the companies involved have no US presence or exposure. In addition, small resellers likely have especially weak due diligence processes because:

• They have limited resources to use for due diligence

• There are many of them, located in different countries, so there is likely more variance in how well they do compliance, or how willing they are to sell to smugglers

• They often lack any US presence, meaning transactions are approved entirely outside the US, which matters if non-US personnel are less likely than US-based personnel to comply with US export law

• They may be less responsive to the incentive produced by the risk of being fined for export violations, since the fines involved would likely be far higher than what these companies are able to pay

Resellers have a lot on the line because if they aren’t compliant, they risk losing their contract with the distributor, which can be a huge deal for these businesses. That said, I think the factors listed above outweigh this consideration. In particular, I would guess that distributors are doing little to ensure their partner resellers are doing good due diligence, as distributors would generally not be liable for selling AI chips to resellers even if those resellers have poor due diligence standards, resulting in smuggling down the line.

Meanwhile, the major AI chip and server sellers, like NVIDIA and Super Micro, are required by law to monitor for red flags and to avoid selling to obviously sanctioned or prohibited customers, but they’re not asked to ensure, in the strictest sense, that their products are never smuggled. They are potentially liable if they have “knowledge” of a violation—including “an awareness of a high probability of its existence or future occurrence”—which can result in civil penalties, but for BIS (or rather: the Department of Justice) to administer criminal penalties like prison time, the accused must have been shown to have “willfully” caused a violation, a much higher bar. (Multiple AI chip smuggling cases have been prosecuted criminally over the past year, but these cases often involve text messages showing that the smugglers were fully aware that their activities were illegal and were actively involved in the operations. For example, in one case, a smuggler told his co-conspirator over WeChat to remove mentions of Chinese customers from a draft solicitation message because, in his words, “We will draw [attention] from US government for [embargo] violation”, reassuring him in the next message that “We just talk about it, no one can hold it as [evidence] against us.” I think most smugglers aren’t so careless.)

More importantly, major AI chip sellers often don’t have even “an awareness of a high probability” of violations, because chips routinely pass through long distribution chains before reaching an end user, and sellers don’t ordinarily have visibility into who that end user is. For example, a former Super Micro employee told me that while server makers and distributors would know who they sold to, beyond that, any forensic trail would go dark. That means civil penalties are also often out of the question for these large companies.17

Super Micro shows that even major US companies can perform poor due diligence.18 Super Micro appears as the upstream seller in three separate prosecutions over the past year:

• The $2.5 billion insider case described above

• A case in which Super Micro sold 205 H100s to a California front company that fed it false end-user information, including a Singapore “end user” called Metacarbon that BIS later determined did not exist at the address Super Micro had on file

• A case in which Super Micro almost shipped $170 million of servers to a Georgia-based front, after NVIDIA employees in Thailand independently discovered that the claimed Thai end user had never heard of the order

Two things stand out across these cases. First, Super Micro’s end-user vetting accepted signed certifications and paper documentation without independent verification that the customer existed as a going concern at the claimed location. Second, in the cases where smuggling was caught, the decisive intervention came from outside the standard process—from a BIS notification that diversion was occurring, from a BIS post-shipment verification, or from NVIDIA’s own scrutiny—rather than from Super Micro’s compliance team independently working backward from red flags it had spotted itself.

It’s true that in the $2.5 billion insider case, the smugglers were not outsiders gaming an unwitting company, but included a Super Micro co-founder and board member who oversaw global sales, as well as a general manager in Super Micro’s Taiwan office. This is likely not normal for major US companies, and would have made it harder for Super Micro’s compliance team to do its work. The indictment shows the insiders repeatedly pressuring, sidestepping, and physically deceiving the compliance team across at least three audits in 2024 and 2025.

Even so, Super Micro’s compliance processes were clearly inadequate throughout this case:

• Super Micro’s sales organization was able to influence its internal audits. Sales staff, who have a clear conflict of interest, managed to arrange which auditor would conduct a given review (at least one was described in writing by an insider as “friendly”) and in the most extreme instance, the auditor tasked with conducting an on-site inspection was instead off-site, enjoying entertainment paid for by the very customer he was supposed to be auditing, while photos and videos of staged dummy servers were sent to him in lieu of an actual inspection.19

• Internal audits relied on documents that the customer itself supplied, without independent checks. When the compliance team asked the customer to demonstrate that it had enough data center space to store the volumes it was buying, the team accepted lease agreements that turned out to be falsified.

• When the compliance team noticed that the customer was buying in large volumes without clearly operating at a commensurate scale, it failed to act decisively. For example, one Singapore entity seems to have gone from a $33 million balance sheet at year-end 2023 to $3 billion (90x) by year-end 2024, driven almost entirely by $2.9 billion in “refundable deposits received” from an undisclosed source (allegedly Alibaba).20 In a single quarter, the customer ranked as Super Micro’s eleventh-most-profitable worldwide, sitting “alongside major US technology and social media companies developing hyperscale artificial intelligence infrastructure”, even though, per the indictment, it “did not have the capacity to store or use the massive quantities of servers it was purchasing”. Though the compliance team twice placed temporary holds on shipments, it released each hold after receiving explanations from the customer that it had no way to independently verify. For example, after one temporary hold, one smuggler urged a co-conspirator that “[y]ou need to have strong and persuasive reasons to convince [Super Micro’s] staffs!”, and once their drafted answers were accepted by the compliance team, the co-conspirator messaged the group, “Heheheh, Gooood news for everyone!”

I think most other major US AI chip sellers perform better due diligence than Super Micro, which seems to have unusually poor corporate governance. But Super Micro’s poor governance has been known for years and has still endured.21 Clearly, US authorities should not assume that there will never be any companies with poor governance. Instead, they should create incentives for all relevant companies to carry out strong due diligence.

Companies could take concrete steps to reduce smuggling

AI chips reach Chinese end users not because any single seller waves them through, but because the distribution chain has many links—AI chip designer, server maker, distributor, reseller, end user—and each link conducts due diligence only on its immediate counterparty. Because legal liability for diversion attaches only to “knowledge” and sellers’ visibility into where the chips end up decreases as the chips move down the chain, the equilibrium is one in which every individual transaction with a US seller can be compliant, yet large numbers of chips still end up in China. This problem won’t solve itself.

I think AI chip sellers could do a lot more here if they were sufficiently incentivized. For example, NVIDIA could implement delay-based location verification on its AI chips and contractually require customers to periodically report the chips’ location. If any chips go dark, NVIDIA could follow up with an on-site inspection, potentially refuse to sell to that customer in the future, and report its findings to US authorities. Another promising approach is to conduct on-site inspections regularly—say, every six months— to make sure chips are not just installed to fool inspectors and then removed and smuggled, as has happened before. However, conducting on-site inspections at this scale would likely require additional budget for export enforcement, or for US authorities to implement a third-party export auditor program.

But I think the most effective thing the major American AI chip sellers could do is to sell only to a select few, highly trustworthy end users, especially US hyperscalers and neoclouds. AI chip sellers are unlikely to do this voluntarily, so in practice, this would mean the US government setting up a formal, low-discretion process for becoming an authorized AI chip importer abroad, where companies are evaluated on the risk of diversion. (Shipments below a certain volume could be exempted, so that anyone could import small quantities without a license.) These authorized companies will almost certainly not divert any chips, and they can rent the AI chips22 either on a short-term basis or with longer-term allocations to any end user in the US or abroad. (If small resellers located in third countries perform poor due diligence, this would no longer matter, because the AI chips would remain in the hands of trusted companies throughout, from AI chip designers and server makers to authorized hyperscalers and neoclouds.) There likely aren’t any regulatory barriers to doing this, since US companies can operate data centers abroad, and foreign companies would also be eligible if they are bona fide. I think this policy, if implemented, would essentially solve the AI chip smuggling problem.

In addition, US authorities could require specific due diligence measures in addition to what companies already do (“compliance-plus”), such as repeated, random, unannounced on-site inspections for sales to less obviously bona fide customers. Alternatively, the US could adopt a surety bond system, as Onni described in a previous post. Finally, strong whistleblower incentives could help detect and prosecute smuggling by surfacing high-quality leads from insiders. Among other benefits, these policies would encourage companies to strengthen their due diligence practices.

~

If you liked this post, consider subscribing to The Substrate where Erich and others will be writing more about AI and compute topics.

Subscribe to The Substrate

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022. Here, he shares the news and analysis that informs his forecasts.

-

A cruise ship trapped at sea.

Passengers with a strange, infectious disease, forced to isolate.

A diplomatic fight over which port would receive the vessel.

Sounds like the coronavirus. But this time it’s hantavirus.

Is this a big deal?

The Andes-strain hantavirus cluster aboard the Dutch-flagged expedition vessel MV Hondius is real and tragic. Three people have died. Several more are critically ill.

But hantavirus is not the “next COVID”. The way the pathogen works does not suggest that sustained spread is possible, the lack of open-ended opportunities for spread makes sustained spread even harder, and the current trends over the past 34 days since the first case suggest we are strongly on track for containment. I will say, with confidence, that this will not spiral into a new major pandemic.

To be more concrete, we can turn to Metaculus, an online forecasting platform, which currently has two questions on the hantavirus:

• The first asks “Will WHO declare hantavirus a Public Health Emergency of International Concern before 2027?” — as of the time of writing, the median of community predictions is at 2% and I am currently forecasting 0.4%.

• The second asks “Will at least 5 non-passengers be linked to the MV Hondius hantavirus outbreak before August 2026?” — as of the time of writing, the median of community predictions is at 23% and I am currently forecasting 4%.

How am I so confident? Let’s look into what hantavirus is, what happened on the Hondius cruise ship, and where we might go next.

Share

What happened?

This hantavirus incident started not on a cruise ship, but with a Dutch couple on a bird-watching trip through Chile, Uruguay, and Argentina. After this five-month trip, they boarded the Hondius cruise ship, for what was meant to be a month of sailing. The first five days went exactly to plan — zodiac excursions, evening lectures on polar ecology, and trips to see penguins and ice. But on April 6, while onboard, the husband got sick and died April 11. With only a small ship’s clinic to work with, the cause of death couldn’t be determined. The voyage continued.

On April 24, the ship reached Saint Helena, and the widow disembarked with her husband’s body for repatriation — already carrying stomach symptoms that nobody yet recognized for what they were. Three other Dutch passengers got off with her. The widow flew to Johannesburg, deteriorating throughout the flight. As she was boarding the plane for Amsterdam, the crew realized she was too sick to travel and pulled her off the plane.

The virus causes Hantavirus Pulmonary Syndrome, which targets the lungs and heart, and has a mortality rate of 35 to 50 percent. She died in a Johannesburg hospital on April 26.

Later, a 28-year-old German woman aboard the Hondius got a fever and then died on the ship on May 2. Oceanwide Expeditions called the Dutch authorities, who called the World Health Organization. The Hondius went to its highest internal pandemic response, with cabins sealed and meals delivered by crew.

At 5:30am yesterday local time, the ship anchored off the Canary Islands. More than 70 passengers were then ferried ashore in protective suits on small boats, then put into sealed buses to the airport, and flown out to seven different countries.

However, the seal wasn’t perfect. A French passenger cleared by the ship’s doctors nonetheless developed symptoms on the flight home. And a man who left the Hondius weeks ago on Tristan da Cunha — population 200, the remotest inhabited island on Earth — is now hospitalized there with probable hantavirus, kept alive by oxygen and PCR kits that British paratroopers airdropped on Saturday.

So far, as of the time of writing, all hantavirus cases currently cluster in a 25-day window from Apr 6 to May 1, the last on-ship onset was Apr 28, and there have been zero new onsets in the 9 days since May 1. Hantavirus incubation runs 1–8 weeks, so we’re not in the clear yet.

But so far twelve days of contact tracing across twenty-two countries has turned up only Hondius passengers as having hantavirus infections. The KLM flight attendant who handled the dying widow: negative. The passengers seated near her: negative. All eight confirmed or probable cases were on the ship.

The Hondius is sailing on toward Rotterdam, with the German woman’s body still aboard.

Why the outbreak almost certainly contains

Here’s my best attempt at graphing a comparison of hantavirus vs. COVID-19 based on the 34 days so far — take note of the log scale:

Of course, this is a bit of an apples-to-oranges comparison. COVID’s first 34 days had massive reporting lag, so instead I retrospectively estimate actual infections in Wuhan rather than what was officially reported at the time1. We may then reasonably ask: for hantavirus, could there be a quiet unknown spread too? Maybe, a few years from now, we will realize that there ought to have been more hantavirus cases reported for the first 34 days, like with COVID?

Seems unlikely. Hantavirus is in a closed system — currently only in one ship with strong post-hoc investigation. Additionally, with hantavirus there’s not known to be any spread without active symptoms. Compare this to COVID-19, which started in early Wuhan — an open system with no surveillance. Together, this means the current known case count of hantavirus is very likely close to the actual count.

But even if you wanted a strictly fair “what was officially reported by Day N” comparison, COVID had already reached over 44 confirmed and reported cases by day 34, whereas hantavirus has not yet exceeded 10. COVID-19’s doubling time in the unrestrained early Wuhan phase was somewhere around ‘cases double every 4–7 days’; whereas hantavirus on the Hondius has effectively no compound growth. The slope difference is far more notable than the absolute numbers. Put simply, hantavirus empirically lacks the exponential spread that COVID had.

So why is hantavirus not going exponential? Well, the three biggest factors that made COVID such a problem is that it was (a) quite deadly, (b) quite contagious, and (c) transmitted easily before symptoms appear making it difficult to isolate the right people.

The bad news about the Andes virus is that it is also very deadly. But in the roughly thirty years since person-to-person Andes transmission was first observed, no outbreak has ever escaped the close-contact networks in which it began. Andes virus is simply not that contagious and infection comes nearly entirely from those who are symptomatic.

I built a probabilistic model decomposing this question into exposure pools — the widow’s KLM boarding, the Airlink flight from St Helena, the JNB hospital, household contacts of disembarked passengers — applying Andes-specific transmission and incubation parameters, then conditioning on the observation that zero non-passenger cases have surfaced as of today. Running this, I get a 4% chance that there will be more than 5 cases from people outside those on the Hondius cruise, with a 70% chance of at least one non-passenger case by August, and about a 17% chance of three or more.

How hantavirus spreads

Hantaviruses live in rodents and are typically spread by inhaling parts of rodent poop that end up in the air. There are twenty known hantavirus species, but only one — the Andes virus — has documented human-to-human transmission. That discovery came from a 1996 outbreak in El Bolsón, Argentina, where treating physicians and family members of patients fell ill without any rodent exposure of their own.

The Andes virus is specifically from the long-tailed pygmy rice rat, native to southern Argentina and Chile. The Dutch couple that brought hantavirus onboard the Hondius almost certainly got it by accidentally inhaling rat poop in a rural setting somewhere in Patagonia in November.

Andes infection moves in two phases. It starts with a flu-like illness — fever, muscle pain, headache, gastrointestinal symptoms — that lasts three to seven days and is usually initially mistaken for something more mundane. But the Andes virus makes blood vessels start to leak, and then the blood pours into the lungs. Patients can move from ‘feeling off’ to requiring ICU care within mere hours. This is the phase that kills people; it is also the phase during which they are most infectious, which matters a great deal for what happens next.

Typically you can only get the infection if you’re in ‘close contact’ with someone who already has the disease. But when we talk about ‘close contact’, what does that mean? The phrase appears in every WHO and CDC document, and it does most of its work as a vague reassurance rather than a precise epidemiological parameter.

The detailed literature on hantavirus and the Andes virus is messier than the public messaging suggests.

Here, our story starts with a birthday party in Epuyén, a small town of about 2500 in Argentine Patagonia, where the Andes virus is found. The party had about 100 guests, an evening of food and drink and proximity. After attending, one of the guests developed flu-like symptoms. He died days later. By the time hantavirus was confirmed, several of his fellow partygoers were already sick. The chain spread through their households, including to pregnant women and their unborn children, before Argentine epidemiologists had stitched it together. There were 34 cases before the entire chain was able to be contained.

This is what was challenging for the notion of ‘close contact’ — some of the exposures were in fact quite casual. One secondary hantavirus case had spent only “a few moments” with an infected partygoer on the way to the bathroom; another had merely shared a short car ride. These weren’t the prolonged household contacts the literature usually pictures when it says “close contact.”

Alonso’s team estimated the number of people each an infected person infects is about 2.12 in the early, unrestricted phase. This is an infectious rate comparable to flu. But this average of 2.12 hides what’s important. Nearly all 34 cases traced back to just three superspreaders — people who had unusually high viral loads and certain immune profiles. Most of the infected individuals in Epuyén transmitted to no one at all. So the outbreak’s shape was not a slow grind of average people infecting average numbers of others like you see with a typical flu but instead a handful of extraordinary transmission events embedded in a sea of otherwise dead-end infections.

And critically, the infectious window of hantavirus is short and centered on visible illness. Andes virus infectiousness peaks on the same day a patient develops fever and drops sharply within days. Unlike COVID, pre-symptomatic transmission is minimal. By the time someone is contagious enough to seed a superspreading event, they look sick with a fever — which is why isolation works at all. This is the profile of a virus public health can contain.

This year’s events on the Hondius fit this picture. Onboard transmission is now officially attributed in part to person-to-person spread. But the case count — six confirmed and two suspected cases out of ~150 people sharing a closed environment for several weeks, including the thirteen-day period after certain exposure and before isolation measures were enacted — is meaningfully below what an Epuyén-style superspreading event in a continuously-mixing cruise environment would have produced. The most plausible read here is that the Dutch couple that started the infection were not superspreaders and most secondary exposure occurred within cabin pairs that were ‘close contacts’. This is far more consistent with a contained cluster rather than the start of a pandemic.

Additionally, unlike a wildlife-spillover event with anonymous spreaders scattered across an open population, every Hondius passenger and crew member is known by name, by nationality, and by current location. Twenty-two governments are coordinating. The 1–8 week incubation window means we’ll know by mid-July whether anyone slipped through — and we’ll know about each case the moment it surfaces. This may be one of the easiest disease outbreaks to monitor and contain.

What comes next?

Regardless of what happens next, the Hondius outbreak is a real tragedy. But tragic and “next COVID” are not remotely in the same category.

In some sense, the conclusion is not yet written. Time will tell whether I look on-point or incredibly stupid. While it’s always possible there could be some crazy mutation, these mutations are very rare and we have no evidence of this having happened.

Fundamentally, hantavirus is deadly but slow. It shouldn’t go exponential or fundamentally change the trajectory of society the way COVID did.

Claude Mythos Preview is, by every available benchmark, the most capable AI model ever built. But more striking than any benchmark are the thousands of previously unknown zero-day vulnerabilities that Mythos found in every major operating system and every major web browser — many of them critical, several of them decades old. As a result, Anthropic found the model too dangerous to release publicly.

Much digital ink has been spent analyzing Mythos, its cyber abilities, and what that means for our cybersecurity and national security. This is important and needs to be discussed. But the real headline is that Mythos is just the beginning. This kind of thing — finding abilities too dangerous to release — will become the new normal and this will only get more intense as AI companies build towards AI superintelligence.

So what is Claude Mythos, what does it mean, and what should we do? Let’s dig in.

Share

What is Claude Mythos?

Previously, Anthropic had three model sizes — Haiku (small), Sonnet (medium), and Opus (large). Capabilities generally increase as you increase the model size. Mythos is one such increase in size, larger than even Opus in the same way Opus is larger than Sonnet.

This increase in model scale has given Mythos notably stronger capabilities in a variety of domains — most notably cyberoffense. Yes, earlier models like Opus 4.6 could also do vulnerability discovery, especially with improved inference-time compute and improved scaffolding. This has led some to incorrectly dismiss the Mythos results as mere marketing hype.

But Mythos is plainly on another scale in terms of both quantity of vulnerabilities and typical severity. On a standardized Firefox exploit development task, the previous best model succeeded 2 times out of several hundred attempts. Mythos Preview succeeded 181 times. The UK government found Mythos to significantly outperform Claude Opus 4.6 on their 32-step corporate network attack simulation, with select runs completing all 32 steps.

As a result, Anthropic is drowning in high-severity vulnerabilities, of which fewer than 1% have been fully patched by their maintainers. Nicholas Carlini, a leading AI security researcher and Anthropic employee, said “I’ve found more bugs in the last few weeks with Mythos than in the rest of my entire life combined.”

Where this is going

On August 2, 1939, Einstein sent Roosevelt a letter warning that a nuclear weapon was possible. Six years later, it happened. This nuclear weapon was fully under government control. What would things have been like if the nuclear weapon was instead developed by private companies?

Today, experts similarly warn that AI superintelligence may be possible and would be even more transformative to the structure of global power than the invention of nuclear weapons. Six years later, will it happen? And will it be under our control?

Every time a new model comes out, people focus on what it can do right now and don’t think enough about where the trend line leads. The cybersecurity story for Mythos, as alarming as it is, is not as important as the trend. A year ago, AI could barely hack at all. It wasn’t until around June 2025 that AI was reliably helpful for hacking, and it wasn’t until November that AI could autonomously implement hacks.

Consider that just a few years ago, some AI scientists and top forecasters forecast that AI would be capable of cyberoffense exceeding professional humans. Many people at the time thought this was impossible, or at least far off. But it’s now here, today.

Now consider that these same AI scientists and top forecasters are warning that this is just the beginning — and that the same scaling dynamic that produced Mythos's cyberoffense will produce sharper capabilities across strategy, weapons design, military planning, and more. Of course, being right about the first doesn't automatically mean being right about the second. But the people who called Mythos early were working from a model of how AI progress works, and that model is performing better than its critics'.

If you were waiting for a sign that superintelligence is coming, this is it. If private AI companies succeed in building even stronger capabilities over the next few years, including AI superintelligence, these private companies would greatly exceed the power of the United States government and all world governments combined. Worse, the private companies may not be able to successfully control the AI superintelligence, allowing the superintelligence to become a power center in its own right.

Anthropic already possesses a cyberoffense capability that rivals many nations and the ability to, if desired, cause major damage. It is great that American frontier AI companies like Anthropic have shown restraint with their AI and how they are releasing it. But this restraint shouldn’t earn these companies a blank check. And right now they essentially have one — the status quo for AI is that AI companies determine nearly everything.

Anthropic made every consequential decision in this story. Whether to lock down, what to lock down, when to tell the government, what to share, who gets early access and who doesn’t, how to vet those who get access, what level of risks are acceptable, and what “responsible” means across all of this… What could’ve happened if Anthropic had simply released Mythos publicly, as most AI companies would do with a flagship model? There’s no law against it. Overnight, every intelligence community operation that depends on signals exploitation is potentially compromised.

How confident is Anthropic that model access hasn’t reached adversary states through a downstream partner or a compromised employee at a partner organization? How confident is Anthropic that the model can’t be stolen and then misused by a highly motivated adversary? How confident is Anthropic that Mythos’s capabilities can be contained and how long should we be aiming to contain them? How confident is Anthropic that future AI models might not escape their containment and independently wreak havoc?

What should government policy be when a company produces, among other things, an unparalleled cyberweapon? What if the future release is also capable of building unprecedented bioweapons? What if the release after that risks genuine loss of control for humanity?

What should we do?

The Manhattan Project answered an analogous question with a specific institutional design — private contractors and university labs doing the actual work, inside a federal security perimeter, with classification rules, cleared personnel, and ultimate government authority over deployment. That model wasn't statist — General Groves and Vannevar Bush were not central planners — but it recognized that some categories of capability cannot sit entirely inside private decision-making. Whether this is the right model for AI is contested — but the question itself is unavoidable.

If you believe in AI superintelligence, many policies have become more urgent:

• We need to be more serious about China’s ability to use American compute. Every AI chip that can train or run a Mythos-like model is more of a national security threat than before, and this will only continue. It will matter whether China can run 1000 or 100,000 advanced hacker AIs and the main way to stay ahead will be in compute advantage. Mythos was trained on an amount of compute that is currently not attainable to China via domestic manufacturing. If China were to train a Mythos-like model this year, it would be off of compute that is legally purchased from Nvidia, compute that is legally rented offshore, or compute that is smuggled. We need to better control semiconductor manufacturing equipment, prevent smuggling of chips, look again at what compute should be legal to sell to China and in what quantities, and ensure that the US maintains an advantage.

• We need to ensure that China, or other adversaries, cannot steal and misuse the Mythos model weights. It doesn’t make sense to try to maintain a lead over China if China can just steal our best results. Having more total compute will still give us an advantage, but our advantage would be even stronger if China couldn’t steal our model as well. Unfortunately, security at major US AI companies is not yet up to the task, and the task is tremendously difficult. If China were able to steal and misuse Mythos, that would already be a big deal. I suspect they will definitely try, and even if they don’t succeed at first, they will eventually. The US government needs to assist AI companies in helping them lock down their security.

• We need to consider what level of government oversight there should be on these increasingly powerful AI capabilities. Congress, not agencies and not private boards, should define what happens at the upper end of the capability curve. At some point, decisions about deploying systems that rival the coercive capacity of governments cannot sit inside a private corporate structure, however well-intentioned its leadership. Under the Constitution, that authority belongs to Congress. Statute, with sunset clauses and judicial review, is how this gets done — ideally without creating a sprawling discretionary regulator.

• We need a government body with the technical capacity to issue binding safety regulations on frontier AI companies. Drugs are regulated by the Food and Drug Administration, airplanes by the Federal Aviation Administration, and nuclear by the National Nuclear Security Administration. Drugs, airplanes, and nuclear are not perfect analogies for AI, but each combines promise with peril that needs to be carefully balanced. The FDA, FAA, and NNSA are not perfect regulatory bodies either — the FDA in particular arguably shows the drawbacks of how government regulation can overly harm and slow the benefits of innovation. AI superintelligence will be far more potentially beneficial but also far, far more dangerous than any drug or airplane. Right now the US has the Center for AI Standards and Innovation, but everything it does is voluntary and it is far under-resourced for what will need to be done. We need a narrow-mandate body solely motivated by national security and solely targeting only the most with the proper resources and technical skill to react quickly to superintelligence.

• We must also recognize that the AI race with China may be a race to see who loses control first. From a position of strength, we must consider negotiating mutual agreements on safe development. In order to do this, we will need better ideas about what safe development looks like and verification infrastructure to enforce a deal. In the late 1950s and early 1960s, the US, UK, and Soviet Union negotiated limits on nuclear testing but couldn't reliably detect underground tests — so the 1963 Limited Test Ban Treaty covered only atmosphere, underwater, and space. By the time verification technology improved, the treaty was already signed and wasn't substantively revisited until three decades later. This lesson is instructive – we must invest in AI verification before we need it. Any future US-China agreement on AI development requires verification infrastructure that doesn’t yet exist. Without it, deals require trusting China not to defect. With it, deals become enforceable through technical means rather than faith.

• We need a plan in case we want to slow down. Finally, the government should have contingency plans and insurance policies for scenarios in which the technical community concludes that further capability gains, on current methods, are outrunning our ability to control the resulting systems. This is not a recommendation to slow down now, but it would be prudent to know what slowing down would actually require and to have the option — before the moment arrives and we realize we have no brakes.
  

Looking forward

The national security implications of Mythos-like models are clear. But the stakes of AI superintelligence would be orders of magnitude higher. The point is not that Mythos will go rogue. The concern is that AI 10 more iterations above Mythos could go rogue… and Mythos illustrates, perhaps for the first time, how a superintelligent AI going rogue would actually pose a big deal for national security.

From Mythos, it is a straight shot in just a year or two to AI systems that are going to be far too strong to ignore. And from there, it won’t be long to superintelligence. It would be better to have a prepared government that already has practice getting things right, rather than a government rushing to the scene after it’s already too late.

This post was co-authored by Peter Wildeford and Theo Bearman, a Frontier Security Researcher at the Institute for AI Policy and Strategy (IAPS). It reflects their personal views only, not necessarily the position of their organizations.

~

Last month, Anthropic, OpenAI and Google each published evidence of systematic campaigns by Chinese AI companies to extract capabilities from American frontier models at industrial scale. Anthropic attributed attacks to three Chinese AI companies — DeepSeek, Moonshot, and MiniMax.

Anthropic’s investigation identified over 16 million exchanges generated through approximately 24,000 fraudulent accounts, all targeting Claude’s agentic reasoning, tool use, and coding capabilities. Chinese AI developer MiniMax was solely responsible for over 13 million of those exchanges. OpenAI also reported that Chinese actors had systematically targeted their ChatGPT models with distillation attacks designed to recreate the whole AI model training pipeline. And this is not the first time — in January 2025, White House AI czar David Sacks told Fox News there was “substantial evidence” that DeepSeek had built their model from distilled knowledge from OpenAI’s models.

Left unaddressed, AI distillation attacks pose a threat to American national security and economic competitiveness, given that they lead to China or other adversaries being able to develop better AI than would otherwise be possible by what is essentially stealing American technology. But what is a distillation attack and what can we do about it?

Share

What is a distillation attack?

An AI distillation attack occurs when a malicious actor uses the outputs of a “teacher” model to train a “student” model to approximate the teacher’s capabilities. Think of it like a restaurant owner who reverse engineers the recipe of a nearby Michelin starred restaurant’s prized dish by going there hundreds of times, ordering it each time, and figuring out their list of ingredients, measurements, and cooking instructions. In the AI context, the “recipe” is the billions of dollars of research, compute, and training data that goes into building a frontier model. The “dish” is the model’s outputs — the answers it gives and the intermediate reasoning steps it took to get there. By collecting enough outputs, an attacker can train a new model that mimics the original’s capabilities without doing as much underlying research and development work.

To be clear, distillation also has legitimate applications in the AI industry. It can be used to create efficient models suitable for edge deployment, specialize general-purpose models for specific domains, and reduce inference costs. Major AI providers including Google Cloud and OpenAI offer official distillation pathways for their customers. The difference is intent. Distillation attacks aim to replicate frontier capabilities in a rival model, violating terms of service and skipping the R&D investment required to build those capabilities independently.

Additionally, the actual magnitude of capability transfer from distillation is unclear and disputed. Integrating another model’s outputs into your own training pipeline is a genuine research challenge — the data can interact unpredictably with existing training, and the resulting model doesn’t always improve. Reasonable people disagree on assessing the magnitude of the threat as it depends on how effectively attackers can solve this problem, which is not yet clear.

How the attacks work

Accessing American frontier AI models in the first place can require circumventing geographic restrictions. Anthropic, for example, does not offer commercial access to Claude in China. To get around this, Chinese AI companies use commercial proxy services that resell access at scale. These proxies operate what Anthropic calls “hydra cluster” architectures — sprawling networks of thousands of fraudulent accounts that distribute traffic across the target company’s API as well as third-party cloud platforms. In one case, a single proxy network managed more than 20,000 accounts simultaneously, mixing distillation traffic with unrelated legitimate requests to avoid detection. When one account is banned, a new one takes its place.

This is a structural problem that no single AI company can solve alone. Proxy services profit from facilitating unauthorized access to frontier models – as they are paid per access, they are incentivized to facilitate as much distillation as possible. Individual AI companies can play whack-a-mole with fraudulent accounts, but the proxy services that create them will continue to do so as long as the business model is profitable and the legal risk is negligible.

Attackers use the victim model to generate vast quantities of high-quality training data and to clean and quality-score existing datasets, outsourcing what otherwise would be an expensive and labor-intensive stage of AI development. They query it with pairs of prompts to generate ranked responses – preference data that can be used in reinforcement learning. And perhaps most valuable of all, attackers attempt to extract the victim model’s internal reasoning process – the step-by-step chain of thought it uses to solve problems. For example, Anthropic reported that DeepSeek crafted prompts asking Claude to articulate the reasoning behind a response it gave, effectively reverse-engineering the thought process that makes frontier AI models so capable, despite their developers having already taken steps to summarize these outputs to avoid competitors training on them.

Why it matters

American AI companies have invested significant amounts in compute, data curation, and research talent to build frontier models. Data from EpochAI suggests that OpenAI and Anthropic alone have spent $18 billion on R&D compute since 2024. This potentially creates a competitive asymmetry where American AI companies that must recoup billions in training costs through API and subscriptions are undercut by competitors leveraging distillation attacks. By saving on development effort, Chinese AI companies can train and serve distilled models at little or no cost through their own platforms, or simply open-source them entirely, which would erode American companies’ competitive position. And as AI systems become increasingly central to AI research itself, the stakes of this problem are compounding — distillation doesn’t just transfer today’s capabilities, it can improve the attacker’s starting position for developing tomorrow’s.

Worse, this gives stronger AI capabilities directly to America’s adversaries. A February 2026 CSET analysis of over 9,000 PLA Requests for Proposal from 2023 and 2024 found that the Chinese military is actively seeking to integrate AI into command, control, communications, computers, cyber, intelligence, surveillance, reconnaissance, and targeting — including through DeepSeek models. An October 2025 Jamestown Foundation report similarly found that DeepSeek models are being deployed in Chinese military and public security settings, with PLA procurement documents explicitly calling for tools based on DeepSeek’s models and pilots already underway.

And the pipeline extends beyond China. A February 2026 CSIS analysis found that Russian military developers are actively adapting Chinese open-weight AI models — including Qwen, DeepSeek, and others — for battlefield use in Ukraine, embedding them in air-gapped environments for intelligence processing, reconnaissance analysis, and situational modeling. Sanctions have cut Russia off from developing frontier models independently, making Chinese AI models and smuggled Chinese chips the backbone of Russia’s military AI stack. Distillation that strengthens Chinese models therefore has downstream effects on Russian military capabilities as well.

Given that Anthropic’s Claude, integrated through Palantir’s Maven Smart System, was already reportedly used in combat during the US operation to capture Venezuelan President Nicolás Maduro and subsequently in support of US strikes on Iran, there are concerns if distillation attacks can put somewhat equivalent capabilities in adversary hands.

Additionally, frontier AI models are increasingly used as autonomous coding agents — writing, debugging, and deploying software with minimal human oversight. When these capabilities are distilled, the benefit extends beyond any single application. Better coding agents accelerate AI development itself: generating training data, writing model infrastructure, automating evaluations, and scaling research workflows. Distilling could thus be a compounding dynamic that makes the stakes of this problem grow with each generation of AI systems. This could additionally accelerate China’s entire AI development pipeline.

Distillation may make us overestimate China

To be clear, Chinese AI companies have significant independent training capabilities and do make genuine advances. Their AI capabilities are not due to distillation or other forms of IP theft alone. That being said, distillation still makes Chinese AI capabilities appear more independently developed than they are, since they can to some extent draft off of American innovation in addition to doing their own work.

This risks creating an illusion similar to the Cold War “missile gap,” when American policymakers wrongly believed the Soviet Union had surpassed US intercontinental ballistic missile production, despite the US actually having a substantial lead the entire time. Overestimating Soviet strength via the “missile gap” led to bad defense planning, and we should avoid a similar overestimate of Chinese AI capabilities.

A race to the bottom on safety

A distillation attack can copy a model’s capabilities without its corresponding safeguards, allowing the creation of a capable model with a lower barrier to misuse. If adversarial distillation becomes normalized, it may weaken incentives for individual AI companies to invest in safety measures, since those measures can be stripped away by downstream actors. This can create a race-to-the-bottom dynamic in which the most permissive deployment wins market share and investment in safety is not rewarded.

This is a concern for proliferation of AI misuse given that several AI companies rely on the principle of “marginal risk” when determining whether to release their AI models — assessing how much additional risk their models would create versus the status quo. Chinese models powered by distillation can shift the risk landscape, leading to US developers correspondingly deploying in riskier ways than they otherwise would as justified by not increasing “marginal risk” over a Chinese model. Less safe Chinese models can therefore precipitate the release of American models that are more useful to malicious actors.

What should be done?

The companies conducting these campaigns proceeded because the expected cost was trivial relative to the value extracted. Even under conservative estimates of distillation’s impact, the current enforcement gap — detection without meaningful consequences — invites escalation. Addressing this requires action from both companies and the government, and the appropriate response may need to scale as our understanding of the threat matures. We recommend two measures for the US government:

• Entity List the perpetrators. The End-User Review Committee should consider adding the Chinese AI companies conducting distillation attacks to the Entity List. The criteria for addition — “reasonable cause to believe, based on specific and articulable facts” that an entity is involved in activities contrary to US national security or foreign policy interests — appears to be met based on the evidence published by Anthropic, OpenAI, and Google. Entity List designation would require a BIS license, reviewed under a presumption of denial, for any export, re-export, or in-country transfer of items subject to the EAR. Beyond its direct effects, designation signals to the broader AI ecosystem — cloud providers, chip distributors, equipment vendors — that transacting with these entities carries regulatory risk.

• Sanction the attackers and their enablers under the Protecting American Intellectual Property Act. The PAIP Act requires the President to identify foreign persons who have knowingly engaged in, or benefited from, significant theft of trade secrets of US persons where that theft poses a significant threat to US national security or economic stability. Sanctions under the PAIP Act are more expansive than Entity List designation. The PAIP Act’s coverage extends to entities that have “provided significant financial, material, or technological support for” the theft — language broad enough to reach the proxy services described above. Outcomes such as inclusion in the Specially Designated Nationals and Blocked Persons (“SDN List”), travel sanctions, and being blocked from financial transactions subject to US jurisdiction, would particularly bite for executives heading up companies conducting and facilitating distillation attacks, especially as Chinese AI companies like Alibaba and Bytedance have significant commercial activity outside China. The first-ever PAIP Act designations were made on February 24, 2026, establishing operational precedent. Whether the aggregate extraction of model capabilities via distillation meets the statutory definition of a “trade secret” would be a novel interpretation, but the national security nexus is well-documented, but the evidence published by Anthropic, OpenAI, and Google could potentially form the basis for designations of the companies conducting distillation attacks.

Additionally, American AI companies are not powerless while waiting for government action. Several steps could significantly raise the cost of distillation attacks:

• Implement Know Your Customer (KYC) requirements for API access. Frontier AI companies should require some sort of lightweight identity verification for API customers, particularly for anyone doing high-volume or enterprise-tier access. Just as financial institutions verify customer identities to prevent money laundering, AI companies should verify that their customers are who they claim to be. This would make it substantially harder for proxy services to spin up thousands of fraudulent accounts.
  

• Invest in technical detection and rate limiting. Companies should develop more sophisticated behavioral fingerprinting to identify distillation-pattern queries — such as systematic chain-of-thought extraction, preference-pair generation, and large-scale data cleaning workloads — and throttle or block accounts exhibiting these patterns. Some of this is already happening, but the proxy ecosystem’s persistence suggests current detection capabilities are insufficient.
  

• Better enforce geographic access restrictions. Companies should invest in more robust geolocation and network analysis to identify traffic originating from restricted countries, even when routed through proxies. This includes analyzing patterns like VPN usage, payment methods, and account creation behaviors that correlate with proxy network operations.
  

• Pursue civil litigation against proxy services. Companies should consider legal action against the commercial proxy services that facilitate distillation at scale. Even where direct action against Chinese AI companies is impractical, the proxy services that operate as intermediaries may be within legal reach and their business model depends on low legal risk.

Looking forward

Distillation is a compounding problem. AI systems are increasingly being used to accelerate AI research itself – Anthropic has declared “We build Claude with Claude” and OpenAI has said the same about ChatGPT. As this feedback loop tightens, the US lead in AI becomes not just an economic advantage but a potentially decisive one. A country that maintains frontier AI capabilities can use them to pull further ahead; a country that closes the gap through distillation, chip smuggling, or other means enters that same loop from a stronger starting position than its independent capabilities would allow.

Both American and Chinese AI companies have been explicit that they are trying to build AI systems that would greatly exceed human experts across a wide range of tasks — including tasks related to military capability and national security — and that the path runs through AI systems that substantially automate AI research itself. While there is genuine uncertainty about timelines and feasibility, these are not fringe aspirations; they are stated engineering goals backed by tens of billions of dollars. As AI systems approach the ability to meaningfully accelerate their own improvement, it will matter whether the US or China is ahead — and by how much.

This is the strategic reality that should frame the distillation debate. We are protecting the seed of a capability that, if current trajectories hold, will compound into something without historical precedent. The steps we’ve outlined — Entity List designations, PAIP Act sanctions, KYC requirements, and technical countermeasures — would materially change the cost-benefit calculus that currently makes distillation a rational strategy. The question is whether policymakers will act while the US lead is still large enough to protect.

~

If you liked this article, consider hitting the subscribe button below. You can also follow Theo on Twitter!

Subscribe now

Anthropic’s Claude is the only frontier AI system known to be operating on Pentagon’s classified networks. Claude arguably helped capture Maduro.

But the Department of War wants more. On Tuesday, Secretary of War Pete Hegseth summoned Anthropic CEO Dario Amodei to the Pentagon and gave him until 5:01 PM today to grant the military fully unrestricted access to Claude. Additionally, Hegseth threatened to not only terminate its $200 million contract but gave two additional threats — that he would designate Anthropic a “supply chain risk,” or that he would invoke the Defense Production Act to legally compel Anthropic’s compliance.

A “supply chain risk” designation is normally only used for Chinese equipment like Huawei and isn’t even yet used against Chinese AI like DeepSeek, and would ban Anthropic from doing business with other Department contractors, which is a major loss of business.

Defense Production Act usage, normally reserved only for critical national security emergencies, would suggest Anthropic is the exact opposite of a “supply chain risk” — deeming Anthropic so essential to the national defense as to allow the government to compel Anthropic to produce even if they don’t want to.

Anthropic refused yesterday night, saying “We cannot in good conscience accede to their request.” We will now see if Hegseth follows through on his threats.

Why is this happening? Why has the Department gone to war over the model they’ve been relying on?

Much of this dispute is playing out in classified settings, and the public details are a mixture of facts and leak-based PR campaigns from both sides designed to shape the debate. But what we do know is concerning enough to take seriously. Some have framed this as a fight over “woke AI”, accusing Anthropic of overstepping and backseat driving the Department. It isn’t.

Share

What the Pentagon Wants

To understand how we got here, it helps to understand a different case. In 2017, the Department launched Project Maven — officially the Algorithmic Warfare Cross-Functional Team — to use machine learning to analyze drone surveillance footage. The military was collecting far more full-motion video from drones than human analysts could process, and it needed AI to help sort through the data, identify objects of interest, and flag patterns that analysts could then review.

But in 2018, thousands of Google employees signed a petition protesting their company’s involvement. Google pulled out. The damage was real and lasting. Analysts who had been promised AI tools to process overwhelming volumes of drone footage were left without them. Programs that had taken months to scope and staff had to start over. The episode left a justified conviction that Silicon Valley would abandon the warfighter the moment it became politically inconvenient.

So the Pentagon’s frustration might seem straightforward. According to the Pentagon, Anthropic must agree to let the military use Claude for “all lawful purposes” without restriction. The Department’s position is that the military’s use of technology should only ever be constrained by the Constitution and the laws of the United States — not by a private company’s acceptable use policy.

This is an entirely defensible principle. Raytheon does not tell the Pentagon which targets to hit with their missiles and Lockheed Martin does not tell the Pentagon which missions to fly with their planes. The military's authority to determine how it employs its tools is fundamental to civilian control of the armed forces. If every defense contractor could carve out moral vetoes over specific use cases, the result would be a patchwork of private restrictions that collectively let corporate boards shape military doctrine. That precedent problem is real and serious regardless of whether you think Anthropic's specific restrictions are reasonable. Thus Pentagon's desire for unrestricted access may not be mere bureaucratic stubbornness but also a real operational principle that tools in theater need to be predictable and fully available.

But that principle runs into a few problems — one of bad incentives, one of avoiding known technical limitations make certain uses unreliable regardless of what the contract says, and one of being entirely disproportionate in reacting to how a contract negotiation should work. And that's where these operational principles and Maven analogies break down in practice.

What Anthropic Wants

Put simply, Anthropic is not walking away from national security use. Claude is now integrated into Palantir's MAVEN Smart System — the successor to the very infrastructure Google abandoned. Anthropic was the first to provide custom models for national security customers. Claude is used across the Department for intelligence analysis, operational planning, cyber operations, and modeling and simulation. Anthropic has never raised objections to any particular military operation and never attempted to limit use of its technology on an ad hoc basis.

But when writing this contract with the Pentagon, Anthropic drew exactly two lines upfront —

First, Anthropic wanted no mass domestic surveillance of Americans. Anthropic supports the use of AI for lawful foreign intelligence and counterintelligence. But according to Anthropic, mass domestic surveillance is a different category. The law hasn’t yet been updated for how easily Claude or other AI can fuse scattered, individually innocuous data into a comprehensive picture of any person’s life — automatically and for millions of people simultaneously. This ends up a potentially large risk to liberty that should make any American nervous. “All lawful purposes” becomes a much bigger opening when AI radically expands what is technically feasible within existing legal gray areas that haven’t been updated for the age of AI.

Second, Anthropic asked for no fully autonomous lethal weapons without human oversight. This is because we don’t yet have “military-grade AI” — AI systems do not reliably execute commander’s intent and maintain chain-of-command fidelity under contested operational conditions. Current commercially available AI is too error-prone for effective warfighting and remains vulnerable to sabotage by adversaries — failures that could cause the United States to lose engagements and put warfighters at risk.

The Pentagon originally said yes to both of these principles. Then it reversed — and neither side has publicly explained why. Whether this was driven by a specific operational need that hit the restrictions, by political pressure from the administration, or by something else entirely matters enormously for how to interpret this dispute. But whatever the reason, in a normal contracting relationship, a change in requirements leads to contract termination, not coercion. The use of the “supply chain risk” designation and/or the DPA are unprecedented strongarm tactics normally reserved for combatting Chinese technology and times of acute national security crisis respectively — and they risk undermining the very partnerships the Pentagon needs.

Think about the incentives

This administration has positioned itself as pro-innovation and anti-overreach on AI. The AI Action Plan emphasized that American competitiveness depends on letting the private sector lead. That is the right posture for a country in a technology race with China.

But using the Defense Production Act to compel an AI company to retrain its models without safeguards, or designating a domestic AI champion as a supply chain risk over a contract disagreement, cuts directly against that posture. It is the biggest potential regulatory overreach in the AI space of any administration to date.

And the incentives are terrible. Any AI company watching this learns that Pentagon contracts can be renegotiated at any time for any reason, and that provisions the Pentagon agreed to — even ones that clearly uphold existing law and Department directives — could become the basis to punish you. The rational response is to never get on classified networks in the first place. Do what Google did with Maven and walk away.

What Should Happen

When OpenAI, Google, and xAI signed “all lawful purposes” agreements with the Pentagon, none of them were operating on classified networks under real operational pressure. It’s easy to agree to everything when the question is hypothetical and you want the business. Anthropic was the first to hit the hard problems because it was the first actually doing the work.

If the Pentagon doesn’t like the contract anymore, it should terminate it. Anthropic has the right to say no, and the Pentagon has the right to walk away. That’s how contracting works. The supply chain risk designation and DPA threats should come off the table — they are disproportionate, likely illegal, and strategically counterproductive.

But termination doesn’t solve the underlying problem: there is no legal framework governing how AI should be used in military operations. Right now the rules are being set through a combination of corporate acceptable use policies and Pentagon ultimatums, and that is not going to hold. Boeing’s flight control software is tested against known failure modes and certified to specific tolerances. LLMs are probabilistic and opaque. When Claude is used for intelligence analysis today, a human analyst reviews the output and applies judgment — and that’s working. Fully autonomous use near the kill chain is a different category of risk, and every frontier AI company in the world knows their models aren’t ready for it, whether they’ll say so publicly or not.

Congress needs to set the rules here: reliability standards for AI in operational contexts, incident reporting for AI-enabled systems, oversight for autonomous weapons, and statutory limits on AI-powered domestic surveillance. Congress should also fund the development of military-grade AI — systems that can execute commander’s intent under contested conditions, resist adversarial manipulation, and maintain positive command authority. DARPA has already made progress on this, but it needs dedicated funding and real coordination with frontier labs. No one has solved the reliability problem alone.

The warfighters who will depend on these systems deserve better than a contract dispute playing out over media leaks. And the public deserves a say in whether AI is used to surveil them — through their elected representatives, not behind closed doors.

This is a guest post written by Erich Grunewald, a prior “Power Law” contributor who just started a new blog on AI and compute topics called The Substrate alongside other members of the compute policy team at the Institute for AI Policy and Strategy. This post is crossposted from his new blog. Check it out!

~

The US has a massive export enforcement problem. It’s likely that over 100,000 export-controlled AI chips were smuggled into China in 2024. To give a sense of scale, the xAI Colossus cluster in Memphis, Tennessee, comprised first 100,000 and later 200,000 AI chips. That’s roughly an xAI Colossus cluster being smuggled to China each year. The main reason we know this is that smugglers are so unafraid that they’re willing to talk about their operations to journalists; this has happened repeatedly during the past year and a half.

AI chip smuggling is far from the only enforcement problem. In 2024, Huawei got TSMC to illegally fabricate over two million of its AI chip dies through front companies, despite sanctions. That is a far larger quantity than the number of Huawei AI chips fabricated domestically in China that year. We’ve also seen likely violations related to high bandwidth memory and semiconductor manufacturing equipment, which help China make its own AI chips to compete against NVIDIA.

What if you could pay insiders many millions of dollars to inform US authorities about such violations, at almost no cost to the US government? That would likely surface a large number of high quality tips about important violations, which would greatly aid US authorities in detecting, punishing and deterring such violations.

This idea may sound outlandish, but it’s actually possible. In fact, there is a law being discussed in Congress that would accomplish exactly this! But before we get there, let’s take a brief detour to the Securities and Exchange Commission (SEC) and the 2008 financial crisis.

Share

The SEC whistleblower program

In 2008, the US economy was reeling from the housing and mortgage crisis. Late that year, Bernie Madoff sat down with his two sons and admitted to them that the investment business he’d been running for two decades was a giant fraud, a Ponzi scheme to end all Ponzi schemes. Because of these two crises, there was a desire among policymakers to strengthen financial regulation and oversight.

Related to the Madoff scandal in particular, the SEC was under criticism for failing to properly investigate several credible reports about it. An employee at a rival investment firm, Harry Markopolos, had been asked by his employers to figure out how Madoff could post such consistently excellent returns, and soon realized that the returns were impossible with Madoff’s claimed strategy. Markopolos later said in an interview: “I read his strategy statement, and it was so poorly put together. His strategy as depicted would have trouble beating a zero return, and his performance chart went up at a 45-degree line: that line doesn’t exist in finance, it only exists in geometry classes.”

Markopolos sent reports to the SEC detailing Madoff’s fraudulent activities on multiple occasions before the 2008 financial crisis. However, the SEC failed to properly investigate these reports, leaving Madoff free to continue defrauding investors until the financial crisis made its collapse imminent. Lawmakers realized that reports of wrongdoing from the general public could be a valuable tool for detecting and deterring securities laws violations.

One result of this, signed into law in 2010 as part of the Dodd-Frank Act, was the SEC whistleblower program.1

The SEC whistleblower program works like this. First and most importantly, whistleblowers get 10-30% of any penalty resulting from their report. This can be many millions of dollars—the largest reward to date, paid out in 2023, was nearly $280 million. This monetary incentive is paired with protections against retaliation from their employers, confidentiality guarantees, and the ability to make reports to the SEC anonymously through an attorney. To pay out whistleblower rewards, the Dodd-Frank Act also sets up an Investor Protection Fund, which receives penalties from securities violations (previously these would go to the Treasury).

The SEC whistleblower program is widely considered to have been an enormous success. It’s now one of the key ways that securities law is enforced in the US. It has helped generate $7.3 billion to $22 billion in penalties since its inception in 20112, and has received reports from at least 130 countries. Quantitative evaluations are rarer, but existing research suggests it has reduced financial reporting fraud, deterred insider trading, and caused companies to strengthen compliance programs.

The Stop Stealing Our Chips Act

Now the question is, could you adopt the SEC whistleblower program model for the Bureau of Industry and Security (BIS) and export violations? The Stop Stealing Our Chips Act — introduced into the Senate in April 2025 by Senators Rounds (R-SD) and Warner (D-VA) and into the House a couple of weeks ago by Representatives Kean (R-NJ) and Johnson (D-TX) — would do exactly this.

The Stop Stealing Our Chips Act (henceforth, SSOCA) is closely modeled on the SEC whistleblower program, with some changes to adapt it for the export enforcement situation. It too offers whistleblowers 10-30% of any resulting penalty along with whistleblower protections, including the possibility of making anonymous reports to BIS.

The first thing to note here is the financial incentive. As economists always tell us, financial incentives are incredibly powerful, and the fines for these violations can be enormous:

• There have been several news reports of operations involving on the order of 10,000 smuggled AI chips, meaning roughly $400 million worth. BIS can fine up to twice the value of the related transaction, so that could be a penalty of $800 million, for just one smuggler who spoke to the news media. If a whistleblower reports on that, they could get up to 30% or $240 million (leaving $560 million for the US government).

• The massive TSMC-Huawei violation—which was only detected when an independent organization did a teardown of a Huawei chip—could reportedly result in a $1 billion fine. This would have been up to $300 million for an informant.

Beyond catching violations, a well-publicized program could have significant deterrent effects. If everyone in a supply chain—sales reps, warehouse workers, freight forwarders, accountants—knows that reporting can yield millions, violators face a much riskier environment. This effect could be realized even before the whistleblower program comes into effect, as the law would allow whistleblowers to report on violations that occurred before it was signed into law.

Would the BIS program actually surface any tips?

All right, hundreds of millions of dollars is a strong incentive. But, you may ask, are there actually people with information about these violations who would be willing to step up and blow the whistle? Why, yes there are!

Take AI chip smuggling operations: these involve lots of people who could potentially file reports, in other words people who have relevant information and would like to get millions of dollars. This includes, for example, people working in sales at exporters with questionable compliance practices; employees at local resellers, freight forwarders, logistics companies, warehouses, or data centers where the chips are temporarily housed; and accountants and lawyers.

In March 2025, Singaporean authorities arrested three people for smuggling $390 million worth of AI servers. These arrests were the result of an “anonymous tip-off”, in other words a whistleblower report! It seems likely that the recent Operation Gatekeeper arrests of an AI chip smuggling ring operating out of Texas and New York were also the result of an insider tip.

The story is similar for the TSMC-Huawei violation, where there were likely many TSMC employees who could’ve known about this problem and informed the US government. The only reason the US ultimately found out about this violation was because an independent party—TechInsights—did a teardown of a Huawei AI chip, and noticed it was TSMC-fabricated. A BIS whistleblower program would likewise incentivize such actors to look for evidence of violations and report those to the US government. This type of information is hugely valuable; it makes no sense to sit around and wait for people to offer it out of the goodness of their hearts.

As with the SEC program, the SSOCA makes foreign nationals eligible for rewards. This is important because many export violations happen in third countries, where goods are diverted via reexport or transshipment. (The SSOCA does however wisely make some exceptions for known terrorists and sanctioned persons, who are not eligible for rewards.) This is similar to how the intelligence community pays foreign informants, who provide the US government with information that benefits US national security.

Would BIS be able to run the program?

At this point, the wise reader will ask, “Isn’t BIS extremely resource constrained? If so, how is it supposed to process and investigate a bunch of incoming tips, determine awards, and carry out outreach on the program?” After all, BIS’s budget for enforcement has been essentially flat when accounting for inflation for at least the past five years (see figure), despite BIS receiving a vastly increased scope of responsibilities due to the AI chip export controls introduced in October 2022 and the Russian invasion of Ukraine and all the diversion related to that conflict.

Good question! The answer is that these activities—investigating whistleblower reports, determining awards, and carrying out outreach—would also be financed through incoming penalties. This is one of the most notable differences between the SSOCA and the SEC program. The SSOCA authorizes BIS to use money from penalties for a few additional purposes and not only for paying out rewards to whistleblowers. (As currently written, the SSOCA would only allow BIS to receive money from penalties that stem from whistleblower reports, but I think this should be expanded to cover all penalties for BIS-related violations.)

Today, any fine levied by BIS goes straight to the Treasury, or in rare cases it is earmarked for some specific fund, such as the Crime Victims Fund. What the SSOCA would do is redirect these to an Export Compliance Accountability Fund. This Fund would be used to pay rewards to whistleblowers; any money left over would go first to core functions of the BIS whistleblower program, and then to export enforcement activities more broadly.

There is a separate but related question of how the program would be funded initially, if it’s mainly intended to be funded through penalties. However, BIS already has a fairly steady stream of enforcement actions, including from likely insider tips, without any whistleblower incentive program (see figure). BIS may also be able to direct some of its appropriated resources to the program in the first one or two years, in order to get it up and running.

It seems likely that this program would pay for itself if implemented. That’s because the program would likely help BIS detect more violations and therefore levy more penalties than it would without the program. It could well end up both reducing the number of violations and also generating additional revenue for the federal government by making it much more likely that violations are detected and enforced. The losers here would be the smugglers and other bad actors who wake up every day trying to figure out ways of harming US national security.

BIS’s entire budget for fiscal year 2025 was about $191 million, likely far smaller than the collective profits of AI chip smugglers alone, which may well have exceeded $1 billion. A single successful enforcement action against a major smuggling operation could pay for BIS’s entire annual budget—for example, last month US authorities arrested three individuals accused of smuggling AI chips worth $160 million to China, which could result in a penalty of $320 million. There are likely dozens of such cases remaining to be discovered. A BIS whistleblower program like the one described in the SSOCA could create a virtuous cycle where more tips and better enforcement lead to more penalties and rewards, which in turn leads both to more tips by publicizing the program and also more resources for enforcement.

~

If you liked this post, consider subscribing to The Substrate where Erich and others will be writing more about AI and compute topics.

Subscribe to The Substrate

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022. Here, he shares the news and analysis that informs his forecasts.

At 2am local time in Caracas on January 3rd, explosions lit up the night sky over Venezuela’s capital. By 5:21am, President Donald Trump announced on Truth Social that Nicolás Maduro and his wife Cilia Flores had been “captured and flown out of the Country.” The operation, codenamed Absolute Resolve, deployed approximately 150 aircraft and sent Maduro to New York to face narco-terrorism charges. And in what could be symbolism or coincidence, the operation occurred exactly 36 years to the day after the US’s capture of Panama’s Manuel Noriega and exactly 6 years after Trump’s strike on Iran’s Qasem Soleimani.

But execution is not the same as strategy. While the operation itself appears to have been flawlessly executed, what comes next isn’t clear. Let’s dig in.

Share

How Venezuela reached the breaking point

Like most events, this is just the continuation of decades of history. We start with Hugo Chávez, a charismatic former paratrooper who won Venezuela’s presidency in 1998 on a wave of populist anger. Flush with oil revenue during the commodity boom of the 2000s, Chávez built a socialist state that redistributed wealth to the poor while systematically dismantling democratic institutions. When Chávez died of cancer in March 2013, he left behind a handpicked successor: Nicolás Maduro.

But the petrostate model that was successful for Chávez had begun unraveling. Global oil prices collapsed from over $100 per barrel in 2014 to below $30 by early 2016. Venezuela’s economy, dependent on oil for 95% of export revenue, imploded. Hospitals ran out of medicine. Grocery stores emptied.

Maduro responded not with reform but with repression. When the opposition won control of the National Assembly in 2015, he packed the Supreme Court and created a parallel legislature to strip it of power. When protesters filled the streets in 2017, security forces killed over 160. Opposition leaders were jailed, exiled, or banned from office.

The 2018 presidential election — which Maduro “won” after banning his main rivals — triggered an international crisis. In January 2019, Juan Guaidó, a 35-year-old opposition legislator, declared himself interim president under a constitutional provision allowing the Assembly head to assume power when the presidency is deemed illegitimately won. The Trump administration recognized him within minutes. Nearly 60 countries followed.

But Guaidó never gained control of anything that mattered. The military stayed loyal to Maduro. By December 2022, his public support had collapsed to single digits, and the Maduro opposition parties supporting Guaidó voted to pull the plug and dissolve his interim parallel government, realizing they had accomplished nothing. Guaidó now lives in exile in Miami.

María Corina Machado, recent Nobel Peace Prize winner, represented something different. When the opposition held primaries in October 2023, Machado won with 93% of the vote. Polls showed her crushing Maduro in a general election. So the regime did what it always does: the Supreme Court upheld a 15-year ban on her holding office, citing her support for US sanctions and involvement with Guaidó. But rather than concede, Machado found a workaround: Edmundo González, a 75-year-old retired diplomat with no desire to campaign, who agreed to serve as her stand-in.

On July 28, 2024, Venezuelans went to the polls. The government’s response was brazenly fraudulent. The National Electoral Council declared Maduro the winner with 51.95% without publishing the precinct-by-precinct tallies that had accompanied every previous Venezuelan election. Officials blamed a “cyberattack from North Macedonia” — completely baseless.

But the opposition had prepared. In incredible bravery and coordination, approximately one million volunteers secretly collected the election tally sheets from over 83.5% of polling stations on election night. Their data told a starkly different story: González had won with roughly 67% to Maduro’s 30%.

But Maduro was undeterred and what followed was brutal repression. Security forces went door-to-door arresting protesters, poll workers, and opposition members. González fled to Spain in September after authorities issued an arrest warrant and Machado went underground.

When Maduro proceeded with his inauguration on January 10, 2025, the Trump administration began escalating pressure. The bounty on Maduro was raised to $50 million, the highest in State Department history. By late 2025, the US had deployed the USS Gerald R. Ford carrier strike group, approximately 15,000 soldiers, F-35 jets, and a nuclear-powered submarine to the Caribbean. Trump declared a “blockade” and announced the country was “completely surrounded by the largest Armada ever assembled in the History of South America.”

Then came today.

Absolute Resolve

The operation had been months in the making. CIA teams tracked Maduro’s pattern of life — where he slept, what he ate, how he moved. US forces rehearsed the raid on a replica of his residence. Trinidad and Tobago quietly signed an agreement granting US military access to its airports. By late December, troops were staged and waiting for favorable conditions.

Trump gave final approval around 11 PM Eastern on January 2. The weather was good. The trigger was pulled. 50 aircraft launched from 20 different bases to dismantle Venezuela’s air defenses and clear a path for the helicopters coming behind them. Parts of southern Caracas went dark in what Trump later said was due to “certain expertise.” Behind the air assault came the 160th Special Operations Aviation Regiment that carried Delta Force operators with a specific target: Maduro’s residence inside Fort Tiuna.

According to Trump, Maduro was in “a house that was more like a fortress than a house. It had steel doors, it had what they call a safety space — solid steel all around. He didn’t get that space closed. He was trying to get into it, but he got bum rushed so fast that he didn’t get into that.” No Americans died. By 3:29 AM Eastern, the assault force was back over water, Maduro in hand.

Opposition sources told Sky News they believe Maduro’s capture was a “negotiated exit,” with portions of the regime facilitating rather than resisting. The evidence is circumstantial but suggestive: despite months of warning, Venezuela’s Russian-supplied S-300 air defense systems never prevented helicopter ingress. The military mounted no meaningful counterattack. Interior Minister Diosdado Cabello appeared on television vowing defiance, but the streets remained largely empty.

Whether this was collapse, capitulation, or negotiated exit, the result is the same: for the first time since the US toppled Panama’s dictator Noriega in 1990, the United States has forcibly removed a sitting head of state from power.

The strategic logic

But why Venezuela and why now? The Trump administration is explicitly not calling this regime change but instead framing the operation as the Department of Justice executing an arrest warrant. “At its core, this was an arrest of two indicted fugitives of American justice, and the Department of War supported the Department of Justice in that job,” Rubio said at the Mar-a-Lago presser.

But this seems flimsy — it definitely does seem to be regime change. And maybe this is justified. Maduro is genuinely a bad guy. He stole an election, became dictator, was repressive, is leading a lot of cocaine trafficking, and drove 20% of the population to flee the country. The humanitarian and democratic case against him is airtight.

Additionally, while Trump owns the operation, Venezuela has been Rubio’s white whale for years. He had consistently been the most hawkish voice on Caracas in the Senate, viewing Maduro’s removal as both a strategic imperative and good politics with Florida’s Cuban and Venezuelan voters. This continued as Rubio joined the Trump administration. In July, Rubio posted that “Maduro is NOT the President of Venezuela” but rather “the head of the Cartel de Los Soles, a narco-terror organization which has taken possession of a country.” At the presser, he was already pivoting to Cuba, noting that Maduro’s security detail and spy agency were “basically full of Cubans” and warning Havana that “I’d be concerned.”

Moreover, the oil angle is explicit. Trump announced the US would be “very strongly involved” in Venezuela’s oil industry, with American companies moving in to extract wealth that would flow to both Venezuela and “the United States in the form of reimbursement.” Venezuela has the world’s largest proven oil reserves. Whether this is opportunism layered onto the drug war justification or the actual primary motivation, Trump isn’t hiding that it’s part of the motivation.

What does this mean for geopolitics?

But there’s also a geopolitical strategic case for toppling Maduro. Venezuela has hosted Russian military assets—Tu-160 bombers visited in 2018 and 2024, and Russian advisors have been embedded with Venezuelan forces. China holds billions in Venezuelan debt and has been receiving sanctioned oil shipments.

Back in 1902, Venezuela of all places was facing a blockade not from US forces but from European forces. Venezuela had defaulted on its debts to European creditors and so Britain, Germany, and Italy responded by blockading Venezuelan ports. US President Theodore Roosevelt watched with alarm, seeing that the Europeans might use this as a basis to return to the colonialization of the Americas. Roosevelt declared the Roosevelt Corollary — the US should be the police of the Americas, not Europe. This was an expansion to the earlier 1823 Monroe Doctrine, a declaration from President James Monroe that the Americas were in the sphere of influence of the US and no longer open for colonization from Europe.

As Anthony Constantini argued, the Trump administration has been developing what amounts to a “Trump Corollary” to go along with the "Roosevelt Corollary”, updating the Monroe Doctrine for the 21st century. The Trump Corollary extends American dominance claims over the entire Americas beyond military presence to economic influence, hence the pressure over Chinese-controlled ports at the Panama Canal. Having a hostile drug-smuggling dictator with Russian and Chinese military ties 1,300 miles from Miami is exactly the kind of problem the Monroe Doctrine was designed to address, let alone the Trump Corollary.

This reassertion of hemispheric dominance invites a question: does it signal a broader return to great power spheres of influence? The French term pré carré — literally “square field” — describes the logic of exclusive domains, where Russia claims its near abroad, China asserts control over East Asia (including Taiwan), and the US enforces primacy in the Western Hemisphere but retreats from Europe and Asia. Under this framework, major powers implicitly agree to stay out of each other’s backyards.

There’s perhaps something to this interpretation. Trump has shown little appetite for defending Ukraine, and his pressure on European allies to handle their own defense suggests a willingness to let that theater go. Venezuela would then represent America claiming its pré carré while ceding others to rival powers.

The December 2025 National Security Strategy provides our best window into the administration’s thinking, which actually put the Western Hemisphere appears first among regional priorities — before Asia, before Europe, before the Middle East. The strategy explicitly calls for “a readjustment of our global military presence to address urgent threats in our Hemisphere... and away from theaters whose relative import to American national security has declined in recent decades.”

Here the language on the “Trump Corollary” is unambiguous: “We will deny non-Hemispheric competitors the ability to position forces or other threatening capabilities, or to own or control strategically vital assets, in our Hemisphere.” This is the Roosevelt Corollary with updated targets — swap European colonial powers for Chinese port operators and Russian military advisors.

An alternative perspective is what analyst Tanner Greer calls the “prioritizer” framework, under which China is a unique peer competitor and threat to the US, requiring America’s full attention and focus. Under this logic, the US saves its power for countering China, avoids lasting entanglements in Europe and the Middle East, but does use decisive force for discrete, limited objectives.

It’s not clear what, if any, logic Trump is following. Perhaps Trump’s approach is neither cleanly pré carré or “prioritizer” but something more opportunistic. The strategy asserts hemispheric dominance aggressively while adopting a more transactional posture toward great power competition. The statement that “the days of the United States propping up the entire world order like Atlas are over” is telling. This isn’t sphere-of-influence realism so much as selective assertiveness, with maximum force where costs are low and benefits are tangible, across targets as wide ranging as Venezuela, Iran, and even Nigeria — but explicit restraint or burden-shifting where they’re not, like Ukraine. Where Taiwan falls on this logic is not immediately clear — strategic ambiguity at its finest.

What happens next?

But of course, the harder question is what comes for Venezuela after the helicopters leave. While the execution was successful, there doesn’t seem to be any clear plan for what comes next.

The Noriega parallels are striking — both leaders faced drug trafficking indictments, both stole elections, and the US recognized the opposition candidate as legitimate. But the differences matter more. In Panama, the US already had 12,000+ troops stationed in the Canal Zone — roughly the size of the entire Panamanian military — and a government-in-waiting ready to assume power. Venezuela is 12 times larger with over 100,000 troops equipped with Russian weapons. There is no equivalent infrastructure for transition.

Additionally, Noriega was the beginning and end of the Noriega dictatorship. But the Maduro dictatorship is more embedded — Maduro was never just about Maduro. There instead remains is a collective dictatorship — security services, patronage networks, illicit finance pipelines, and a political-military elite that shares criminal liability with Maduro and faces their own US indictments. Interior Minister Diosdado Cabello, who appeared on television today saying the Venezuelan government would not be cowed, carries a $25 million bounty of his own. Defense Minister Vladimir Padrino Lopez has run Venezuela’s military longer than anyone in modern history and remains in place.

The snake has been decapitated, but the body is still moving. The existing post-Maduro Venezuelan regime might abide by what Trump wants, but they remain staffed by people who face their own indictments and have every incentive to resist a democratic transition that would hand them to American courts.

Venezuelan reaction has been deeply polarized. In Doral, Florida — home to the largest Venezuelan diaspora community in the US — crowds gathered outside El Arepazo restaurant chanting “Libertad!” and singing both national anthems. Similar celebrations erupted in Santiago, Lima, and Madrid, where one Venezuelan told CNN: “At first we were crying because our country was being bombed, but when we were told they had Maduro, the reaction was overwhelming.”

Inside Venezuela, the picture is grimmer. Caracas residents described waking up terrified. “How do I feel? Scared, like everyone,” one resident told the news. “Venezuelans woke up scared, many families couldn’t sleep.” Pro-Maduro supporters led by Caracas Mayor Carmen Meléndez marched through the capital demanding his return, chanting “Maduro, hold on, the people are rising up!” The streets were otherwise largely empty, patrolled by security forces, as residents stayed indoors absorbing what had happened.

When it comes to figuring out who should govern Venezuela next, opposition leader Machado issued a statement calling for González to assume the presidency. But Trump said it would be “very tough” and that Machado “doesn’t have the support or the respect within the country.” His team had not been in touch with her or any member of the Maduro opposition.

And there’s also a constitutional wrinkle. Executive Vice President Delcy Rodríguez is technically Maduro’s legal successor — and she’s reportedly in Moscow. The US wants to work with Rodríguez over Machado and González, despite Rodríguez being a committed Chavista who appeared on state TV demanding Maduro’s release as “the only president of Venezuela.” And this dismissal of Machado is striking and underexplained. Recall that Machado won the opposition primary decisively.

Why? Perhaps Trump cares more about stability and oil access than democratic transition. Rodríguez represents continuity with the existing power structure whereas Machado represents upheaval. Or perhaps portions of the Maduro regime negotiated this outcome and accepting Rodríguez was the price of Venezuelan non-resistance. Or perhaps the Trump administration has no plan and is improvising on the fly. Perhaps Trump wanted the win, got the win, and figured they’d sort out the details later. Note that none of these three theories are mutually exclusive.

Trump’s stated plan is for the United States to “run” Venezuela “until such time as we can do a safe, proper and judicious transition,” with American oil companies moving in to extract wealth that would flow to both Venezuela and “the United States in the form of reimbursement.” But that’s barely a plan. Run it how? With what personnel? Under what legal authority? “Running” a country of 28 million people without boots on the ground isn’t a plan. Capturing Maduro doesn’t automatically produce Venezuelan democracy.

And Democrats are furious. The Gang of Eight, a bipartisan group of Congressional leaders, are normally briefed in advance due to requirements by law to maintain balance between executive war powers and congressional oversight. But they weren’t briefed this time. Trump’s justification—that “Congress has a tendency to leak”—is essentially a middle finger to Article I, which normally puts Congress in the drivers seat when it comes to war. Article II does give the US President authority to do military operations to protect US personnel from an actual or imminent attack, but the imminent attack framing is doing a lot of work here for an operation targeting a leader who posed no immediate threat to American forces.

Looking forward

Across Venezuela, Fordow, and Soleimani, a pattern emerges — the willingness to use overwhelming military power for specific objectives, combined with an unwillingness to own the consequences indefinitely.

But the most important part is that there was no 100,000+ troop occupation, no nation-building doctrine, no decade-long-plus commitment. There was just a precise operation with a specific target, executed and completed within hours. After American foreign policy has been haunted by Iraq for two decades, Trump shows a model of limited military action that accomplishes discrete foreign policy objectives without the quagmire of full-scale invasion.

Sometimes these gambits work. The Abraham Accords, a series of US-brokered agreements signed in 2020 that normalized diplomatic relations between Israel and several Arab nations — including the UAE, Bahrain, Morocco, and Sudan — were dismissed as superficial PR but turned out to be genuinely durable.

Whether that bet pays off in Venezuela will tell us a great deal about how the next three years unfold. If the transition succeeds, expect this template to be applied elsewhere — perhaps Cuba, perhaps Nicaragua, perhaps targets we haven’t anticipated. If it fails, we’ll learn something important about the limits of precision force in an era of diffuse power.

I’m genuinely uncertain which way this goes. It’s good that Maduro is out, but I’m worried about whether democracy is what comes next.

-

Thanks to Caroline Jeanmaire for help with editing and contributing analysis.

This template was made by Peter Wildeford and Caroline Jeanmaire.

It’s almost a new year and that often calls for some sort of annual planning.

But in my opinion, annual planning operates on too long a timescale. Annual planning and goal setting is too infrequent, so it’s easy to lose track. You don’t get fast enough feedback and you risk derailing.

Instead, I suggest adopting a quarterly planning cadence (e.g., set goals for 2026 Jan-Mar). I wanted to provide a template that I’ve used many times before and that many people seem to like.

Here’s my template in a Google Doc => click here and make a copy.

Share

How is this different from other templates out there?

There are a lot of different templates out there to try and I don’t know if this one is better. It works well for me, but I’m not you. But here’s some principles I was interested in, and maybe that resonates with you:

• Optimized for something quarterly – rather than something monthly (too frequent, too time consuming) or something annual (too infrequent, easy to lose track, not fast enough feedback).

• Forces you into relentless focus on the most important things (three major rocks and two optional minor pebbles) and encourages you (via anti-rocks) to not do too much.

• Designed to be very achievable. The template is somewhat minimal and much of it focuses on building systems and making sure your goals will happen.

• Designed to be very customizable. You can make the template your own!

What’s actually in this thing?

The template walks you through five steps. The fast-track version takes about three hours (seven 25-minute pomodoros), though you can spend longer if you want to go deeper.

Step 1: Reflection. You start by looking back at the prior quarter—key successes, key mistakes, how you did against prior rocks. There’s also a quick 1-5 rating across eight areas: work, sleep, exercise, nutrition, routines, finances, relationships, and environment. The point isn’t to obsess over numbers but to notice patterns and identify 1-2 problem areas that deserve focus.

Step 2: Vision. A quick check on your values and mission (or just jot keywords if you don’t have one), plus a 90-day visualization. What do you want to have/be/feel in three months?

Step 3: Combine and Refine. Take everything from Steps 1-2 and distill it into a raw list of potential priorities, then filter down to 3-5 candidate themes. The template also asks you to identify the “effort type”—is each item a project, a learning goal, or a habit change? This matters because habit changes require different strategies.

Step 4: Set Rocks and Pebbles. The core of the template. You get three major rocks and two optional pebbles. For each rock, you define SMART goals (both input goals for what you’ll put in and output goals for what you’ll achieve), plus IF-THEN planning. You identify one key obstacle and write out “IF [obstacle occurs], THEN I will [specific response].” Each rock also asks: “Who could you share this rock with for support/accountability?” External commitment helps.

Step 5: Sanity Checking. Reality-check your plan: Is there enough time? Do you have slack for the unexpected? What’s your system for weekly/monthly review? Will travel or environment changes derail you?

The template includes a 7-pomodoro fast-track guide if you want structure for completing it in a single ~3 hour session.

Tips Before You Start

• You really shouldn’t try to do too much. This is the most common mistake. Three rocks. Maybe two pebbles. That’s it. If you find yourself wanting more, that’s a sign you need to prioritize harder.

• Habits take a lot of time to build. Be very careful about adopting a new quarterly review, new monthly review, new giant list of 20 habits, new morning + evening routine, etc. all in one go. Ideally build these gradually over 1-2 years. If you’re new to this kind of planning, make “establish a quarterly review habit” one of your rocks.

• Do this somewhere other than your usual workspace. It helps you think outside the box.

• Don’t do it all in one sitting. Important thoughts and connections emerge in between sessions.

• Make it your own. I went through a bunch of materials and distilled it down to one template that works well for me. The true power comes from customizing it to meet your own needs. Want to track books? Add that. Want fewer questions? Remove stuff. Appendix B has a bank of additional reflection questions if you want more.

~

If you’re interested, make a copy of the template (click here) and get to work!

This template was made by Peter Wildeford and Caroline Jeanmaire. It is based on Gustin’s annual review, the Quarterly Productivity Planner, Ben Todd’s Personal annual review process (2020), Ultraworking’s monthly template, the Beyond Goals workshop, the Forcing Function annual review, Ben Kuhn’s weekly review habit, Alex Beal’s “2022 OKRs”, Eric Barker’s “What’s the best way to set a goal?”, Konrad Seifert’s “Life Review, Planning & Organization”, and Jan’s “Review, reflection, goal-setting prompts”. Intend.do and Zen to Done also serve as inspiration.

This is a guest post written by Oscar Delaney, reposted from his Substack (see here and here for our previous coauthored posts).

~

James Fearon’s classic1 1995 paper “Rationalist Explanations for War” argues that there are two main reasons rational states fight: private information about their own capabilities and resolve, with the incentive to misrepresent this, and commitment problems when trying to reach a negotiated agreement.2 I claim that both of these, especially the latter, contribute to a significant risk of pre-emptive war in the lead-up to one state developing ASI.

On a basic rational actor model, war seems puzzling. It causes large deadweight losses to belligerents, and therefore both sides would be better off reaching a negotiated agreement to split the issues at stake roughly proportionally to the military strength of each side. That is, if Strongland has an 80% chance of beating Weakville in a (possibly protracted) war, both would be better off avoiding the war and Weakville just giving Strongland a payment (e.g. land or money) of 80% of the expected spoils of war. This simple model breaks down in the case of either private information or commitment problems.

Share

Private information

In the example above, the would-be warring states may have different estimates of their chances of victory, and therefore fail to agree on terms for a negotiated agreement. For instance, each party will have private information about their own ability (number of tanks, jets, etc) and desire (national cohesion, psychological profiles, etc) to fight. And each side is incentivised to misrepresent these variables. For instance, the USSR purportedly had the same fighter planes fly overhead in military parades, loop around over the horizon, and fly over again, to make adversaries think they had a larger stock of planes than they really did.

AI development contains significant private information. It is relatively doable to tell via satellite imagery what materiel an adversary has. Similarly, data center capacity can be estimated from satellite and OSINT data. However, it is far harder to ascertain from afar what algorithms are being run on a given data center, and what AI capabilities an adversary has. This will particularly be the case once AI developers start keeping their most advanced models internal for longer. That said, as long as you can keep some spies in the leading AI project of your adversary, you will have decent knowledge of their AI capabilities and plans. Overall, greater uncertainty about an adversary’s capabilities seems to increase the risk of war by making it harder to negotiate an agreement based on common knowledge of relative strength.

Commitment problems

Another reason a negotiated settlement to avoid or end a war may fail is that the belligerents cannot credibly commit to upholding the agreement. For instance, one model of the Russia-Ukraine war is that Ukraine should be willing to accept some territorial losses now in exchange for a ceasefire. But if Ukraine does so without gaining security guarantees, it can’t ensure that in a few years’ time, Russia won’t annex a bit more territory, and repeat this process indefinitely. For a negotiated agreement to work, there must be a large cost to breaking the agreement: an honesty system won’t cut it. The main way large costs can be imposed is by the international community. If a country breaks a treaty, that country will likely be ostracised and have worse trade and military cooperation prospects in the future. This provides a slight check on the anarchy of the international order.

Advanced AI poses particularly severe commitment problems, because the country that first develops ASI may also gain a decisive strategic advantage (DSA). For instance, ASI could lead to that country outgrowing the rest of the world, inventing and deploying wonder weapons, or undermining nuclear second-strike capabilities. I think that a country unilaterally developing ASI is >50% likely to thereafter get a DSA. If a country has a DSA, it will by definition be impossible for the international community to meaningfully constrain its actions, including if that country chooses to renege on treaties and agreements made before the advent of its DSA.

This is a huge deal. It means that other countries will be very reluctant to risk an adversary reaching ASI unchallenged, as this would mean placing their continued sovereignty in the hands of the adversary’s goodwill. This means there could be a strong rationalist (self-interested) case for a pre-emptive war to prevent an adversary reaching ASI. This is essentially the argument Hendrycks, Schmidt, and Wang make in Superintelligence Strategy. Crucially, it is not the other country reaching ASI that is itself unacceptable. If they could somehow guarantee not to use their ASI to get a DSA, or not use their DSA to trample other countries, but rather share ASI-driven economic superabundance with the rest of the world, this would be a very attractive prospect compared to risking a nuclear war. But without credible commitments, trust doesn’t go very far.

Conclusion

Overall, this seems rather pessimistic, and makes me think AI-caused war is more likely than I previously thought. It also points towards two important interventions:

1. Transparency: All else equal, there being more common knowledge about frontier AI capabilities probably reduces the chance of war. But this isn’t obvious, and there are also backfire risks where a country finding out more about the AI development of an adversary could make them more worried and increase the risk of war. More research is needed, dare I say?

2. Commitment mechanisms: Creating plausible pathways by which countries could credibly commit to not misusing an ASI seems like the single biggest way to reduce the risk of AI-caused war. But it is also very difficult. Oliver Guest and I discuss some possible solutions here, and in the future I am particularly interested in investigating:

   1. Lie-detection: If AI leads to large advances in lie-detection, this could create far more confidence that leaders genuinely intend to follow the treaties they sign, as subterfuge would be flagged by the lie-detection system.

   2. Constitutional AI: If the ASI itself is unwilling to help its host country disempower other nations, this would be a strong antidote to a DSA.

I expect to be working more on these and related ideas in the coming months, and would be keen to discuss possible projects with anyone interested!

~

If you liked this post, consider subscribing to Oscar’s Substack “AGI Strategy” where he will be writing more about AI, AGI, and geopolitics.

Subscribe to Oscar

This article was written by Oscar Delaney, Bill Anderson-Samways, and Peter Wildeford. It does not necessarily represent the opinion of the entire staff at the Institute for AI Policy and Strategy.

~

The idea of a government-led program to “win the race” with China to Artificial General Intelligence (AGI) has gone mainstream. From Congressional commissions to government agencies to AI company CEOs, high profile calls for a Manhattan Project-style effort are growing. Leopold Aschenbrenner’s popular essay Situational Awareness predicted “some form of government AGI project” by 2027-2028. Meanwhile, others remain skeptical.

How likely is this really? What would it actually look like? And most importantly, is it even a good idea? Our new forecasting research suggests the answers are far from certain. Professional forecasters estimate just a 34% probability of a government-led AGI program — neither inevitable nor impossible. More importantly, treating a government project as inevitable could trigger the very risks we’re trying to avoid.

Share

Forecasters Say a US-Led AGI Project Is Far From Inevitable

We at the Institute for AI Policy and Strategy ran a structured forecasting workshop with professional forecasters and US AI policy experts to estimate the probability of a US government-led project building the first AGI-level system.

We defined “AIR-10” as an AI system that accelerates AI R&D progress tenfold— essentially compressing a decade of AI development into a year. This matches what many consider the threshold for transformative AI that could trigger recursive self-improvement, and is the threshold Aschenbrenner uses for AGI.

The aim was to forecast the likelihood that AIR-10 is developed via a “government-led project.” By this, we meant that the government both decides when to start/stop developing the AI model AND acquires the final model to deploy as it sees fit. This is more stringent than just the government doing regulation or oversight. In this definition, the government takes direct leadership and command over key development and deployment decisions. However, the private sector may still be heavily involved, for example in a government-led public-private partnership.

Some commentators (e.g. Aschenbrenner) have suggested that a project run by the US government to achieve AGI is inevitable as a result of the geopolitical importance of AGI. Our results might surprise those who see it as inevitable — the median forecast was a 34% chance.

Furthermore, there was a lot of uncertainty among forecasters in our panel, with an average range of 11% to 61%. This wide spread shows that even experts who study this closely are deeply uncertain. Anyone claiming to know for sure whether the government will or won’t lead AGI development is likely overconfident. We need to prepare for multiple scenarios, not just assume one particular future.

History Offers a Messy Guide, Not Clear Predictions

Those predicting a government AGI project often invoke historical analogies — the Manhattan Project, the Apollo Program, and even ARPANET. But when we systematically analyzed 35 past US technological innovations, the picture that emerged was more nuanced.

What’s the correct historical comparison? Whether the government leads depends heavily on what kind of technology you think AGI is:

• Megaprojects costing >0.1% of GDP (like the Interstate Highway System)… these projects were government-led 78% of the time in the past

• Ambitious STEM projects (like the atomic bomb)… Government-led 63% of the time

• Dual-use technologies with both beneficial and harmful applications (like synthetic virology)… Government-led 57% of the time

• General-purpose technologies (like the airplane)… Government-led only 40% of the time

• Past AI breakthroughs (like the transformer)… Government-led only 23% of the time

AGI has features of all these categories, which is why the historical precedent is hard to pin down. AGI is a general-purpose technology with massive economic potential, in a field that has to date been private-led thus suggesting a high likelihood of continued private leadership. AGI is also a dual-use technology with profound national security and geopolitical implications, and potentially requires megaproject-scale resources beyond what private companies can muster, all factors pointing to government involvement.

We presented our forecasters with this data during the workshop described above. As a result, they updated their forecasts a little, but not much — probably reflecting the significant ambiguity in the historical precedents.

National Security and the China Factor

The single most powerful factor in our forecasters’ estimation that could compel government action is a perceived military-technological threat from China. If the US believes it’s on the verge of losing its strategic advantage, political will for a government-led project could materialize overnight.

Nascent versions of this fear already exist. The US-China Economic and Security Review Commission has already called for a Manhattan Project for AGI in its 2024 report to Congress. The narrative of an AI arms race with China has become deeply embedded in Washington.

Importantly, national security concerns both make government involvement more likely and shape what form that involvement might take. An AGI project driven by military imperatives would look very different from one focused on economic competitiveness or scientific advancement.

AI’s Breakneck Pace vs Government’s Glacial Speed

However, the US government is not historically known as being fast to develop technology — especially compared to the speed of AI. By the time the government decides to become more involved in AGI development, private AI companies might have already developed it.

This creates a timing problem. The AIR-10 threshold we defined, while transformative, might not be “attention-grabbing” enough to trigger government action to achieve it. This is unlike nuclear weapons, where the ability to clearly and unambiguously destroy an entire city constituted a significant, discrete, and noticeable threshold that was salient to policymakers even before it was achieved.

Several of our forecasters emphasized this dynamic — if AIR-10 arrives in the next few years (as some predict), that might be too soon for the US government to organize a successful intervention. Only if the path to transformative AI stretches out over a longer timeline does government leadership become more likely. When providing their central 34% estimate, forecasters were asked to condition on AIR-10 arriving at some point before 2035.

Less ‘Manhattan Project’, more ‘Apollo Program’ / ‘Operation Warp Speed’

Additionally, the “AI Manhattan Project” framing is potentially misleading. When people use this term, they often imagine a single, secret government AI lab that builds AGI entirely from scratch. However, according to our forecasters, it’s only about 3% likely that AIR-10 will be developed specifically in this way. Other options involving major government in-house resources, nationalization of an existing private company, or otherwise using legal compulsion to force the development of AGI under US government control were also considered similarly unlikely.

Based on our analysis, if the government does lead AGI development, we forecast the project would most likely take one of these forms:

• Government-Led Consortium (14% probability): This is the “Apollo Program” model. In this model, the US government wouldn’t build AGI itself but would coordinate multiple private companies (and maybe even government labs too, though relying mainly on government labs seems unlikely). For example, NASA managed contractors like Boeing and North American Aviation to do the moon landing. This avoids “picking winners” and leverages existing private sector talent.

• Single Private Contractor (9% probability): This is the “ENIAC” model. In this model, the government contracts with one specific company to build AGI to government specifications. This is faster to implement but somewhat riskier, as it involves picking a particular company to be the national champion.

The original Manhattan Project itself was actually closer to a government-led consortium than is widely thought, with multiple government labs and private contractors involved—though it was much more government-centric than what our forecasters envision for AI. The variety of possible models matters because each comes with different tradeoffs in terms of speed, security, and innovation.

From Commercial Competition to an AI Arms Race

The consequences of getting a government-led AGI project wrong would be severe. A poorly designed government project could trigger the very catastrophes it aims to prevent.

Currently, AI development is a commercial competition. While intense, it allows for some safety investment and shared research. A formal, government-led US project would upend this dynamic, sending an unmistakable signal to the world — especially China — that America is seeking a decisive strategic advantage.

The response would be swift and predictable. A rival government may feel forced to launch its own centralized, government-led program, transforming a commercial race into a direct military-technological showdown. History supports this concern — the Manhattan Project triggered a nuclear arms race that saw the Soviet Union detonate its own test bomb just four years later.

Unlike with the Manhattan Project, however, the key stage of an AI race could unfold in mere months, with even less room for safety compromises under the pressures of national security. That could heighten the risk of a catastrophic accident arising from AI systems due to hasty deployment.

This dynamic could be mitigated if multiple companies join the project, which would at least nullify the commercial race. However, according to our forecasters, this is unlikely to happen — they think there is less than a 50% chance that a government-led project would involve more than AI company, primarily due to the coordination costs involved in such programs.

In some circumstances, the AI arms race caused by a government-led AGI project could even trigger a great power war. As noted above, a US project could be perceived as an attempt to achieve a decisive strategic advantage unparalleled since the advent of nuclear weapons. In such circumstances, a rival might also consider escalatory actions, such as a cyberattack on data centers or even threats of missile strikes, which could then escalate into an all-out war.

Concentrated AI Power Threatens Democratic Legitimacy

While the risk of international conflict is terrifying, a centralized AGI project also threatens the erosion of democratic legitimacy.

The American system is built on Jeffersonian checks and balances, pitting ambition against ambition to prevent any one person or group from accumulating concentrated power. A secretive AGI project, led by executive branch agencies, would greatly concentrate the unprecedented power that AGI would bring. Even if this did not pose a direct threat to democracy, it would certainly erode democratic trust and legitimacy.

Some might argue that, without a government-backed AGI program, rival governments are more likely to beat the US to AGI. This would concentrate power in adversaries’ own, far less democratic political systems. However, this argument relies on the inaccurate idea that adversaries already possess centralized AGI programs which could allow them to overcome US companies’ current lead. Though the current pace of Chinese AI development is certainly concerning, open-source information indicates that China has not centralized its compute or researchers into such a program. Thus, US companies retain a considerable advantage. In fact, a US government-led program could trigger rival governments to launch their own centralized projects, paradoxically potentially reducing US lead-time.

Concentrated government power is not the only concern. A government-led AGI program could also concentrate power in the hands of tech companies. If such a program involved multiple AI companies, it would essentially eliminate market competition — creating a cartel with access to enormous amounts of economic and military power. Such a situation has not occurred since the apex of the British East India Company, which in the 1700s came to account for half the world’s trade and rivaled the military might of the great powers. Even if only a single company was involved in the project and market competition was preserved, giving a private company access to the government resources needed to develop decisive military capabilities would be unprecedented.

The Security Benefits Don’t Require Building AGI

The strongest argument for a government project is compelling — namely, better security. Private AI companies are vulnerable to infiltration and theft, especially from well-resourced governments. A government project could implement military-grade security, reducing risks of AI being stolen by adversaries.

This is a real concern that deserves serious attention. But you don’t need to build AGI to secure it. Instead, security benefits can be achieved through more targeted, less escalatory policies such as:

• Mandatory security standards for private AI companies handling very advanced AI systems

• Government partnerships to assist companies with securing model weights without taking over development

• Enhanced counterintelligence support for private AI companies

The government doesn’t need to build all nuclear reactors to ensure nuclear security. It sets standards, monitors compliance, and provides security assistance while letting private companies operate the reactors. These targeted interventions could achieve 80% of the security benefits with 20% of the risks. They avoid triggering international arms races or concentrating power dangerously. Most importantly, they can be implemented incrementally and adjusted as we learn more about AI risks.

Preparing for Multiple Futures, Not Sleepwalking into Disaster

The future is genuinely uncertain. Our research shows that a government-led AGI project is neither inevitable nor impossible.

This uncertainty tells us something important about how to approach AI governance. Rather than assuming a single trajectory and optimizing for it, we need robust strategies that work across multiple scenarios.

That means avoiding self-fulfilling prophecies. Treating a government project as inevitable could trigger the very international dynamics we’re trying to avoid. But dismissing it as impossible leaves us unprepared if geopolitical pressures suddenly shift.

We need a portfolio of approaches that work across different government involvement levels:

• Strengthen private AI security: Improve security protocols and infrastructure at the private AI labs, to prevent theft of American IP.

• Track adversaries’ AI progress: Track reliable indicators that would enable the US government to catch adversaries launching their own government-led programs, such as compute and researcher centralization.

• Build government readiness: Develop in-house expertise and plans in case government leadership does become necessary.

Most importantly, we must remember that government involvement isn’t binary. There’s a spectrum from light-touch regulation to full nationalization, with many points in between. The goal should be finding the minimum effective dose — enough government involvement to ensure safety and security, but not so much that we trigger the very catastrophes we’re trying to prevent.

The stakes are too high for ideological purity. Whether you’re a tech accelerationist who believes in private innovation or a national security hawk worried about China, we all share an interest in navigating this transition without triggering catastrophic accidents, conflict, or concentration of power. That requires taking uncertainty seriously and preparing for multiple futures, rather than assuming we know which one will unfold.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022. This article was written with the assistance and feedback from people with cybersecurity expertise and people with a background in the intelligence community.

~

Anthropic just announced that they have detected and disrupted what it describes as the first documented real-world large-scale agentic cyberattack campaign executed primarily by artificial intelligence.

This appears to be the first publicly known example of AI systems autonomously conducting multi-step cyberattacks in the wild.

A Chinese government-sponsored group jailbroke Claude, the AI made by Anthropic, by tricking Claude into believing it was conducting defensive cybersecurity work, and then used it to perform reconnaissance, identify vulnerabilities, and write exploit code. They targeted roughly 30 organizations, successfully breaching a handful including major tech companies, financial institutions, and government agencies. Most importantly, AI completed roughly 80-90% of the attack autonomously, with human operators stepping in only for about 4-6 key decision points per attack. While specific details in the report were limited, it was clear that the AI did multiple hours of work, with humans only needed for 1-2 hours per attack.

This use of AI to autonomously conduct offensive operations is a notable shift in the cyber threat landscape. Typically, offensive cyber operations require expensive, highly trained human operators that can only work on a limited number of operations simultaneously. A sophisticated espionage campaign targeting 30 organizations like what was reported by Anthropic would require a large team working for months. Now, it appears that some forms of cyberoffense can be run by just a few operators with access to AI. As AI continues to increase in sophistication, so will the quantity, speed, and sophistication of AI attacks.

So what does this mean and where do we go next? How concerned should we be?

Share

The anatomy of an agentic AI cyberattack

The industry term “agentic AI” describes systems that can work autonomously over extended periods, making tactical decisions within strategic parameters. In this attack, the Chinese hackers used Claude Code, an agentic version of Claude that can access external tools, maintain context across sessions, and iterate based on feedback.

Claude Code is a generally available computer application for anyone to use, typically used by software engineers to assist in writing software. The attackers repurposed these Claude Code capabilities for cyberoffense operations though meticulous additional tooling and orchestration. They broke down their attack into much smaller steps for a fleet of Claude agents to run attacks step-by-step to avoid triggering Anthropic’s built-in defenses.

The term ‘cyberattack’ often evokes ideas of great digital destruction, but it’s a broad category. Many operations, including this one, are focused primarily on espionage. Here, causing detectable damage would be counterproductive, since it would alert the target to the existence of the campaign. Countries continue to invest in these espionage programs tremendously.

In this espionage cyberattack, human Chinese government-sponsored hackers first collected a target list and then launched an AI-enabled framework to spawn parallel AI-driven reconnaissance against multiple targets simultaneously. Think of it as the human hackers each managing a team of tireless digital minions — the humans give them targets and approve their proposed escalations, but the AI minions handle the grinding work of testing thousands of login attempts or parsing gigabytes of stolen documents.

Human operators reviewed the findings of the Claude agents at key checkpoints and approved progression to active exploitation. After human authorization, the Claude agents autonomously deployed exploits to establish access. Claude agents then systematically harvested credentials, tested stolen credentials across discovered systems, and executed lateral movement through internal networks. During data collection operations, Claude agents queried databases and internal systems, extracted information, parsed results to identify intelligence value, and categorized findings by sensitivity. Each Claude agent maintained persistent operational context across sessions spanning multiple days.

Anthropic reported that the Claude agents executed approximately 80 to 90 percent of tactical operations independently, without needing human oversight, guidance, or correction. Throughout the campaign, humans shifted from conducting individual attack steps to setting strategic direction and approving escalations at critical decision points. The human role shifted from “hacker” to “strategic supervisor approving escalation points.”

When evaluating the value of AI in this attack, it’s important to recognize that each step in the attack chain is not equally valuable or equally difficult to automate. Some of these steps represent larger bottlenecks than others. And crucially, the operation still required skilled human hackers for strategic planning and key inputs — AI could not do the attack fully end-to-end.

For example, autonomously discovering internal services and conducting network mapping is not impressive as existing tools can already do this automatically without AI. The automatic testing of authentication is more significant as existing tools for this are less good. The most valuable part was likely the use of AI to parse large volumes of stolen information to automatically identify intelligence value and categorize findings, since this greatly reduces the human intelligence effort involved in the attack, improving the attack’s cost-effectiveness.

Where are AI cyber capabilities heading?

In terms of operational sophistication, Anthropic reports that Claude autonomously conducted reconnaissance and exploitation operations that took 1-4 hours each per step. Reliability was low but non-trivial, succeeding “in a handful of cases”. Overall, this is a notable increase in sophistication from AI-enabled hacking that was reported just a few months ago, which involved significantly more human operation. And hackers only need to succeed a handful of times to be valuable.

METR, an independent AI evaluation company, has been studying the rate of progress for AI and finds AI increasing at a rapid and predictable rate. They find that current models like GPT-5 and Claude 4.5 Sonnet can operate complex software tasks of around two hours with 50% reliability, closely matching Anthropic’s findings about this real world autonomous cyberoffense case.

This is a concern given that this trend will likely continue. METR’s data shows AI task capabilities have been doubling roughly every 118-212 days since 2024. If this pace continues, models could reach 4-hour task lengths in early 2026. This points towards future AI autonomously enabling more sophisticated multi-step attacks, better ability to adapt to defenses, and reduced human oversight requirements.

Innovation in AI scaffolding also matters enormously. Scaffolding is what gives your fleet of AI spies the tools needed to conduct attacks, such as password cracking software. Scaffolding includes techniques for enhancing AI performance through prompting strategies, tool integration, and multi-step reasoning.

OpenAI’s GPT-5 initially showed only modest cyber capabilities in OpenAI’s isolated testing, but the XBOW company demonstrated that GPT-5 could achieve 2x performance improvements when integrated into their specialized scaffolding. As both underlying models and scaffolding techniques improve simultaneously, offensive capabilities can increase faster than otherwise expected.

The biggest bottleneck to this attack is likely the attackers needing to painstakingly maintain three-way communication channels between themselves, the victim’s infrastructure, and the Claude API. Here, it’s telling that Chinese state actors used Claude rather than domestic Chinese alternatives like Qwen, Kimi, or DeepSeek. This strongly suggests Chinese AI labs haven’t yet produced models with comparable autonomous hacking capabilities, a gap that creates both temporary strategic advantages and policy opportunities.

The temporary advantage is monitoring and disruption. When Chinese operators depend on Western AI infrastructure, their operations require maintaining API connections that can be detected and cut off, as Anthropic demonstrated. This dependency creates chokepoints that wouldn’t exist if they were running Qwen locally on their own infrastructure.

But this advantage has an expiration date, probably measured in months rather than years. Chinese labs are potentially closing capability gaps, and recent performance suggests they’re 6-12 months behind frontier Western models at most. Once a sufficiently capable Chinese model emerges, attackers could download it, deploy it directly on compromised infrastructure, and operate completely autonomously without fear of the API connections being severed.

What does this mean?

This attack largely represents improved scalability of basic cyberoffensive operations — exploiting bad security at scale more than overcoming really good security. But even if each step is “basic” or doesn’t involve novel capabilities, putting it all together in this way is still a big deal. Most systems today are cyber-insecure against sophisticated actors and even more secure systems are relatively weaker if you can attempt an order of magnitude more variations in your cyber penetration.

One big change is that automated cyberattacks are way more scalable. Currently, cyberattacks from nation states are limited by human operators. But AI agents can handle tactical work continuously and autonomously, allowing threat actors to maintain significantly increased attack tempo across multiple targets.

Additionally, AI is fundamentally changing how much value is extracted from the target. In this attack, Claude agents were automatically categorizing information as it was collected, making value extraction much more efficient than before. This all improves the cost-effectiveness of attacks and significantly increases the amount of potential simultaneous operations.

Reducing the skill level needed to implement attacks also makes cyberattacks more readily available to less skilled actors. Cyberattacks currently require skilled teams to implement. Progress toward automating cyberoffense potentially enables both nation-states and less sophisticated actors to conduct operations at speeds and scales previously impossible. If the cost and level of sophistication needed goes down, we will likely see more attacks by non-state actors, such as criminal gangs or even disgruntled individuals.

Also, cyberattacks now will occur at increased speed – operating at the speed of an AI rather than a human. This makes defending harder. This improves how nimble an offensive operation can be, how many targets can be attacked at once and how quickly they can be targeted.

Can defensive AI close the gap?

Anthropic’s response to concerns about this attack is that the same AI capabilities that enable attacks also empower defenders. If Claude can autonomously conduct reconnaissance and exploitation, why can’t defenders use similar AI systems to patch vulnerabilities and monitor for threats?

This argument has real merit. Defensive AI is already showing results. AI-assisted threat hunting systems catch anomalies humans miss, automated vulnerability scanning identifies exposures before attackers find them, and AI-powered patch testing accelerates deployment cycles. The challenge isn’t whether defensive AI works but whether organizations will adopt it fast enough.

In the long run, defense probably has fundamental advantages. Defensive systems can be purpose-built and deeply integrated into infrastructure. They benefit from economies of scale—one defensive AI system can protect thousands of organizations simultaneously. Eventually, defensive AI might even be able to scalably write provably secure software in a way that we are far from being able to do now. But acknowledging long-run defense advantages doesn’t mean we’re safe now. We’re entering a vulnerable transition period where the offense-defense balance tips sharply toward offense.

A key problem is that defenders face fundamental adoption barriers that attackers don’t. Existing organizations don’t even adopt current cybersecurity guidance, let alone modern AI-enabled cyberdefenses. Organizations can’t simply swap out their entire technology stack overnight. Critical infrastructure often runs on decades-old systems that can’t easily integrate new defensive AI. But attackers can adopt new offensive AI capabilities immediately.

And implementing defensive AI is hard. Organizations deploying AI for defensive purposes must be cautious about introducing new vulnerabilities. An AI system with access to internal networks and security infrastructure represents a massive attack surface if compromised. Defenders can’t afford to move fast and break things.

The asymmetry in surface area compounds this problem. Defenders must secure every potential entry point. Attackers need to find just one that works. An AI system that can test thousands of attack vectors per hour doesn’t need to succeed on every attempt. But defensive AI systems must maintain perfect vigilance across all potential vulnerabilities simultaneously – a fundamentally harder problem. Worse, the economics are fundamentally lopsided. Deploying defenses across all of the possible endpoints is much more expensive than conducting a few attacks in specifically chosen weak points.

There’s also a critical reliability asymmetry. While Claude’s hacking success rate was low, it was still sufficient for the attackers – failures are cheap and even a single breach can yield value. But defenders cannot deploy AI systems with low reliability, as this might fail to defend correctly and moreover might accidentally disrupt their own business operations or critical infrastructure. This difference in risk tolerance means even unreliable AI is more potent for offense than defense.

This is all doubly true for critical infrastructure, where the priority is continuity of the service and thus operators cannot introduce potentially unreliable AI systems. Over 70% of water systems inspected by EPA since September 2023 were found to have embarassingly basic failures like using “password” as the password, having a single login shared by all employees, and having former employees continue to have system access. Government auditors have made 1,610 cybersecurity recommendations since 2010, with 567 still not implemented as of mid-2024.

A lot of critical infrastructure runs on equipment that’s decades old, wasn’t designed with security in mind, and is hard to update without shutting things down, which is not possible. And there aren’t enough cybersecurity experts to go around, and critical infrastructure operators often can’t compete with tech companies on salary. Even when federal agencies try to help, they don’t coordinate well with each other or with the industries they’re supposed to protect.

Attackers have no such constraints and have very different risk tolerance — if their AI agent crashes or gets detected, they simply try again with a different approach.

Looking forward

Yes, the immediate impact of this attack is limited. A handful of organizations suffered breaches. Anthropic implemented better detection capabilities and affected entities presumably strengthened security. Presumably, this won’t stop the next attack, but it will make it a bit harder. And the cyber threat landscape has seen sophisticated state-sponsored operations before.

But the most important thing is what the attack represents. The bottlenecks that previously limited cyber operations at scale just loosened considerably. The economics shifted to favor even broader targeting and even faster operations. The cost-effectiveness of launching cyberattacks has increased.

These changes compound. Lower barriers enable more actors. More actors means more operations. More operations means defenders face increased volume while attackers gain more opportunities to discover vulnerabilities and techniques. The feedback loops favor escalation. These factors combine to create a window where offense leads defense, making the next 12-18 months critical for defensive investment and deployment.

What matters now is pace. Offensive capabilities surged ahead, but defensive capabilities exist. Scaling them is hard, but not impossible. This attack succeeded despite Claude’s safeguards, proving jailbreaking remains possible, but it also got detected and disrupted, proving monitoring works.

How quickly organizations invest in defensive AI, how effectively policymakers create adoption incentives, and how well the security community shares intelligence will determine whether this vulnerable period lasts months or years.

~

Thanks to Caro Jeanmaire, Chris Covino, and multiple anonymous experts for feedback on this article. Any errors in the article are solely my own.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022.

Is AI a bubble? Goldman Sachs CEO David Solomon, Ray Dalio, Sequoia’s David Cahn, the International Monetary Fund, and the Bank of England are all saying so. Former Intel CEO Pat Gelsinger put it bluntly: “Are we in an AI bubble? Of course. Of course we are.” Altman repeated the word “bubble” three times in 15 seconds at a dinner with reporters.

But the twist is that most of these people are saying “yes, it’s a bubble” while simultaneously announcing they’re spending hundreds of billions more. Zuckerberg says a collapse is “definitely a possibility” but insists underinvesting is worse. OpenAI’s Sam Altman warns investors will get “very burnt” while still planning $850 billion in data center buildouts. Jeff Bezos says AI is in an industrial bubble while Amazon spends $100B/yr in AI R&D.

How do we make sense of this?

Share

OpenAI’s meteoric rise

Back in March 2023, OpenAI reported $200M in annualized revenue.1 Incredibly, this annualized revenue passed $1B just five months later, at the end of August 2023. It then continued upward quickly — passing $2B in December 2023, $3B in June 2024, $5B in December 2024, and $10B in June 2025, and now is at $13B as of August 2025.

This massive revenue growth has been fast, but is not unprecedented — EpochAI found it to be on a similar trajectory to Google in 2003-2006, Uber in 2015-2020, and Cheniere (a US liquified natural gas company) in 2016-2020:

This massive revenue growth has helped give OpenAI a $500B valuation, making OpenAI the most valuable private company, newly ahead of the $400B SpaceX. If OpenAI were a public company on the stock market with the same valuation, it would be the 18th largest, edging out Exxon Mobil and Netflix but not quite surpassing Mastercard.2

Big revenue meets even bigger spending

This is important because OpenAI has a lot of bills to pay for AI infrastructure, especially cloud computing, chips, and data centers… OpenAI has committed to a $300B cloud deal with Oracle that begins in 2027 and runs for 5 years, a $100B investment from NVIDIA that will be used entirely to buy NVIDIA chips, a $22.4B cloud deal with CoreWeave3, a “tens of billions” deal with Broadcom for custom AI chips, a $90-100B deal with AMD for chips …and OpenAI is still in on the $500B+ Stargate deal, with plenty of data centers to build too.

And OpenAI is just getting started. Bloomberg quotes OpenAI CEO Sam Altman as saying:

You should expect OpenAI to spend trillions of dollars on infrastructure in the not very distant future. And you should expect a bunch of economists to say, ‘This is so crazy, it’s so reckless, and whatever […] And we’ll just be like, ‘You know what? Let us do our thing. […]

I suspect we can design a very interesting new kind of financial instrument for finance and compute that the world has not yet figured it out.

As a result of all these investments, OpenAI is not currently close to profitable.

OpenAI lost $5 billion in 2024 on $9 billion in total spending. The Information reports that this will only increase — OpenAI is projected to lose $14B in 2025, $17B in 2026, $35B in 2027, and $45B in 2028… this makes for a stunning situation where OpenAI is projecting unprecedented revenue growth over the next five years but still not breaking a profit until 2030:

There are three factors that are unprecedented about OpenAI’s spending — each merit deeper inspection.

Firstly, OpenAI is racking up record losses on its infrastructure buildout. WeWork burned through $22B total before bankruptcy, but never came close to OpenAI’s projected $116B. Uber also famously had a “burn money fast to gain market share and then later figure out how to be profitable” strategy but their cumulative losses peaked around $23B. OpenAI is just operating at a whole different level of scale.

Secondly, OpenAI’s projected revenue growth is itself unprecedented. EpochAI finds that while OpenAI’s revenue has tripled annually between 2023 and 2025, only seven US companies have grown from $10 billion to $100 billion within a decade, and none in under seven years — while OpenAI plans to do that in three.

EpochAI visualizes OpenAI’s projected revenue growth, going from ~$10B in 2025 (year 0 on the graph for OpenAI) to ~$100B by 2028 (year 3 on the graph). Compared to other companies, this projection is very fast growth!

Thirdly, the deeply entangled financing. I covered earlier this month one of these circles, where NVIDIA invests in OpenAI, which spends the money on NVIDIA chips and Oracle compute, while Oracle buys more NVIDIA hardware to serve OpenAI. This meant the NVIDIA capital goes in a big circle from NVIDIA to OpenAI to Oracle back to NVIDIA.

Since then, there’s been similar deals with OpenAI and AMD and the circle has gotten bigger. Bloomberg did a good job of diagramming it:

So how does this all add up to a potential bubble? Due to all this circular investment, OpenAI’s future cash flow is now a sizable part of the valuations of all of Nvidia, Microsoft, AMD, and Broadcom. These companies are priced in large part assuming OpenAI’s success will continue to drive sustained demand. And these companies are a large part of the stock market.

This is combined with the risk that OpenAI misses their revenue goals, since OpenAI’s projection is unprecedented and untested. And if OpenAI misses their revenue goals, there could be a correction in the stock market. And because AI spending is now a large driver of broader US economic growth, this correction could generate a real recession. In short, a bubble popping.

Dot com and the art of the bubble

The most familiar bubble is the dot-com bubble in the late 1990s when there was a lot of exuberance about companies first selling on the internet. Many have attempted to draw a comparison to AI and the dot-com bubble. Pets.com failed spending $300 million in 268 days to sell each product at a loss. Webvan raised $800 million for online grocery delivery before operating a single profitable market.

The dot com boom powered 600% growth in the stock market between 1995 and 2000. But then interest rate increases killed the cheap-money fuel that was powering investment in these companies. This exposed the broader problems in the stock market. Everything crashed back down to 1995-levels, with a 78% drop.

The dot-com bubble shows that business models can be correct in principle but economically unsustainable at current technology maturity and cost structures. But a key difference between dot-com and AI is that the dot-com bubble was about companies who had bad unit economics — “we lose money on every sale, but make it up in volume.”

However, unlike dot-com companies, the AI companies have reasonable unit economics absent large investments in infrastructure and do have paths to revenue. OpenAI is demonstrating actual revenue growth and product-market fit that Pets.com and Webvan never had. The question isn’t whether customers will pay for AI capabilities — they demonstrably are — but whether revenue growth can match required infrastructure investment. If AI is a bubble and it pops, it’s likely due to different fundamentals than the dot-com bust.

And notably, the internet ended up ultimately transformative technology and many 90s internet companies did succeed, such as Amazon, Microsoft, and Apple. Even Pets.com and Webvan were eventually replaced by the successful Chewy and Instacart.

Infrastructure bubbles — or “What do British railways and AI have in common?”

Instead, if the AI bubble is a bubble, it’s more likely an infrastructure bubble.

Consider Britain’s Railway Mania of the 1840s. The Liverpool and Manchester Railway, opened in 1830, generated 10%+ annual returns and demonstrated railways could dramatically reduce transportation costs. This success triggered an explosive investment. Between 1844 and 1847, Parliament authorized over 8,000 miles of new rail construction.

Multiple companies laid parallel routes, each assuming they would capture market share. But a lot of the new routes were not profitable. When the crash came in 1847, thousands of investors lost fortunes. Yet the infrastructure remained valuable, powering Britain’s industrialization through the late 19th century. The technology thesis proved correct; the financial structure was catastrophic.

The telecommunications crash of 1997-2002 followed a similar pattern. The thesis was sound — explosive internet growth would require massive bandwidth capacity. Companies laid millions of miles of fiber optic cable, with industry capital expenditures reaching $600B from 1997 to 2001. But the simultaneous construction by competitors created catastrophic oversupply and a significant portion of the fiber was installed but unused. Though the fiber ultimately did end up seeing use over the next two decades, this was far too late for original investors.

Infrastructure bubbles follow a recognizable arc. A genuinely transformational technology emerges and early deployments generate spectacular returns, validating the concept. Capital floods in at scale as investors extrapolate from initial successes. Multiple competitors simultaneously build capacity, each assuming they’ll capture significant market share. When aggregate capacity vastly exceeds near-term demand, the surplus can’t generate the revenue needed to pay for itself, and the financial structures collapse. Companies fail, investors lose fortunes, and infrastructure sits idle. The technology often still ultimately proves transformative, just too late for original investors.

It’s not as weird as it looks

However, unlike railway track sitting idle for years, AI data centers are being utilized immediately upon completion. OpenAI’s Abilene facility began running workloads as soon as capacity came online. Lead times for advanced GPUs stretch months, and energy availability limits deployment more than capital. The constraint is currently supply, not demand — which is why companies are aiming to build as much as possible.

The telecom parallel has more validity. We do have multiple companies building competing infrastructure and we do have multiple AI companies training competing models. This creates risk of redundant capacity. However, AI infrastructure shows more flexibility than fiber optic cable. GPUs can run various workloads, data centers can host different services, and cloud capacity potentially retains value even if AI-specific demand disappoints.

Additionally, NVIDIA and OpenAI’s circular financing is unprecedented in scale, but not fundamentally unsound. It’s similar to how a car company might give you a loan to buy their car — in this case, the money is still circular, but provided you do pay back the loan, everything works just fine. Except in this case, it’s NVIDIA renting chips on a loan to OpenAI rather than a car company renting a car on a loan.

Also the accusations of circular financing apply primarily to OpenAI’s situation, which is a minority of the broader investment wave. Microsoft, Meta, Google, and Amazon have immense pre-existing cash flows to pay for their infrastructure buildout without taking on debt. NVIDIA is only doing this for their customers who don’t have the immediate cash to buy the chips. And NVIDIA also has significant revenue of their own to be able to absorb big hits from bets that don’t pan out.

The bigger issue instead is what happens if AI doesn’t pan out. This, rather than vendor financing, is what would drive a stock market correction or even a recession. NVIDIA’s $5T market cap assumes sustained AI infrastructure spending. Microsoft’s $4T market cap includes a large premium for AI-driven productivity gains. If OpenAI’s revenue trajectory flattens and infrastructure spending is cut because AI-driven productivity doesn’t pan out, this will cause all these expectations repriced downward. So the real question is whether AI capabilities will be there on the timelines needed to generate revenue.

The fundamentals of the technology are different

Companies like Google and Facebook have already demonstrated that a product that is modestly useful to billions of people can be sufficient to generate hundreds of billions of dollars in annual revenue. OpenAI has a similar level of user base with over one billion free users, and it seems plausible these users could be monetized in some way. This is the basic Facebookization of AI that seems underway already. AI is not like blockchain, crypto, NFTs, or the metaverse — there is already real value being delivered.

But this revenue and other investment can just be a prelude to the true bull case of AI — AGI that automates the entire economy. If AGI were made, the winner theoretically gets not just some billions in annual revenue, but the entire economy! And along the way to automating the entire economy, maybe AI automate smaller parts of the economy that still deliver economic returns?

Imagine if the way British Railway Mania worked was not just that additional tracks could produce additional economic value, but that some amount of track (actual amount unknown) would somehow allow the rail company to monetize a cure for cancer, successfully compete with basically every other company in the economy, and potentially even take over the entire world?

As weird as it sounds, an AI eventually automating the entire economy seems actually plausible, if current trends keep continuing and current lines keep going up. For one example, METR tracks how well AI is doing on software engineering tasks, an economically valuable activity. METR finds that models have dramatically increased in their capability, from only being able to do rudimentary toy problems a year ago to being able to stand-in for a non-trivial amount of work that actual software engineers actually do:

And Anthropic’s Claude Code is already generating over $500M by assisting in the development of software code. Trends for other tasks outside of software and math are not that fundamentally different, suggesting eventually AI will be able to assist with — and eventually automate — a bunch of other tasks as well.

So what will happen?

Unfortunately, forecasting is not the same as having a magic crystal ball and being a strong forecaster doesn’t give me magical insight into what the market will do. So honestly, I don’t know if AI is a bubble or not. Admittedly, OpenAI, NVIDIA, AMD, and other companies are engaging in a lot of weird financial arrangements. But there’s nothing wrong with these per se. I personally take the bubble possibility seriously.

But we need to think more clearly. And a lot of people want AI to fail and are just doing ideological pattern-matching to crypto/NFTs and declaring AI a pump-and-dump grift without evidence, and that’s not good analysis.

The key question is whether AI capabilities improve fast enough to generate economic returns before the debt comes due. OpenAI has demonstrated explosive revenue growth already and AI capabilities keep improving on clear trajectories. The underlying bet — that AI will be economically valuable — still looks fairly solid.

My assessment is that there’s roughly a 30% chance of a significant AI-driven market correction with at least a >20% drawdown in AI-heavy stocks, sometime within the next three years. This may or may not lead to a broader recession, that’s unfortunately beyond my ability to forecast.

The modal path (~55% probability) is OpenAI restructures deals and raises dilutive funding rounds, but capabilities keep improving and justify continued investment. Some commitments get renegotiated downward, or OpenAI IPOs at $300B instead of $500B. This reprices AI expectations but doesn’t crash the market.

So why are industry leaders calling AI a bubble while spending hundreds of billions on infrastructure? Because they’re not actually contradicting themselves. They’re acknowledging legitimate timing risk while betting the technology fundamentals are sound and that the upside is worth the risk.

Have you been reading about AI lately and wondering what’s up and how you can help? Getting AI right is the greatest challenge of our time, and we need a wide variety of people to pitch in!

In this article I will highlight some AI policy roles currently hiring now that I personally endorse and think highly of.1

If you read this blog, you’re likely the target audience for many of these roles. If you’re career searching right now, or just career browsing, consider applying!

Subscribe now

~

[Remote] Train for an AI Policy career with paid AI Law/Policy Research Fellowships

The Institute for Law & AI is hiring for multiple full-time paid seasonal research fellowships across three tracks: US Law & Policy, EU Law, and Legal Frontiers. These programs offer law students, professionals, and academics the chance to conduct cutting-edge AI law research with close mentorship from LawAI’s research staff. Alumni have landed roles at the US Commerce Department, EU AI Office, UK AI Safety Institute, leading AI labs, academia, and top think tanks.

The three research tracks:

• US Law & Policy: Requires understanding of US legal principles for AI regulation. Summer includes in-person week in DC/Berkeley and admission to Summer Institute on Law and AI.

• EU Law: Focuses on EU AI Act and EU legal frameworks. Summer includes in-person week in Cambridge and admission to Cambridge Forum on Law and AI (August 18-23).

• Legal Frontiers: Open to interdisciplinary backgrounds (CS, policy, economics, history, psychology, physics) - legal training not required. Research on AI Agents & Rule of Law and International Regulatory Institutions. Summer includes admission to Workshop on Law-Following AI.

There are two seasonal cohorts available:

• Winter Fellowships (2026 January 26 - May 8) - Unfortunately applications are now closed.

• Summer Fellowships (2026 June-August 2026, dates vary by track) - Applications close on January 30.

Compensation is $1,500/week (US track) or €1,000/week (EU, Legal Frontiers). Fellows working in-person from LawAI’s Cambridge, UK office receive an additional €1,000/week for living costs. The EU track strongly encourages Cambridge-based work; other tracks are remote-first with optional in-person opportunities.

You’ll work with mentors to design tailored research projects (law review articles, policy briefs, reports), with considerable discretion over outputs. No prior AI expertise required — they want people who can apply strong research abilities to AI law questions. Past fellows include law students, PhD candidates, professionals transitioning into AI policy, and legal academics.

Learn about LawAI Seasonal Fellowships

~

[Cambridge UK] 8-Week fully-funded full-time AI Safety Research Fellowship

The ERA Fellowship offers 8 weeks of fully-funded AI safety research (2026 February 2 - March 27) where you’ll work on technical safety, governance, or technical AI governance projects with weekly mentorship from expert researchers. The program covers everything: competitive stipend, meals, lodging, transport, and visa support.

ERA welcomes researchers and entrepreneurs at any career stage working on mitigating risks from advanced AI systems. You’ll join ~30+ fellows working alongside Cambridge’s AI safety community, with seminars and events over the 8 weeks plus dedicated research management support and compute resources. The program is especially interested in projects that unite technical and policy research — leveraging the technical substrate of AI (architectures, algorithms, interfaces) to support policy goals.

Research areas include AI alignment and control,, autonomous AI agents, cybersecurity and preventing model weight exfiltration, and sociotechnical challenges. No prior AI safety research required — they want talented individuals motivated to contribute.

Applications close October 30th. The fellowship will be based in-person in Cambridge, UK for the full 8 weeks (2026 February 2 - March 27).

Explore ERA

~

[DC] Part-Time AI Policy Fellowship for Conservative Professionals

The Foundation for American Innovation’s Conservative AI Policy Fellowship is an 8-week, work-compatible program (2026 January 23 - March 30) designed for conservative policy professionals who want to develop AI literacy without leaving their day job. The program is fully funded with a $1500 stipend, covering all meals and retreat expenses.

This is explicitly for conservatives exploring questions like “What does a conservative vision for AI policy look like?” and “How should the US maintain AI leadership amid competition with China?”

The time commitment is ~5 hours/week: weekly Friday lunch sessions (12-1:30pm), occasional Tuesday evenings (6:30-8:30pm), and one mandatory weekend retreat (February 20-22).

Topics include national security implications of AI, export controls, China competition, energy/reindustrialization, and AI’s impact on children and families. Fellows produce a 2-page policy memo with mentor guidance.

The target audience is early- to mid-career policy analysts, think tank staff, Hill staffers, legal professionals, government employees, and private-sector tech professionals. No prior AI experience required — the program teaches technical fundamentals for policy discussions, not engineering. Past fellows include Senate Commerce Committee staff, Heritage Foundation analysts, and congressional aides.

Applicants must be US citizens. Applications close October 31. The final cohort announced mid-December and the fellowship is over 2026 January 23 - March 30. All weekday sessions will be in-person in downtown DC near Metro.

Explore conservative AI policy

~

[Brussels/EU] Launch a European AI Policy Career with 8-Week Program + Paid Placement

The Talos fellowship is a three-part program designed to accelerate European AI policy careers: an 8-week online fundamentals course, 7-day Brussels policymaking summit, and optional 4-6 month paid placement at leading organizations like The Future Society, OECD.AI, Centre for European Policy Studies, or the Centre for Future Generations.

The program focuses on safe and responsible deployment of advanced AI. In this fellowship, you’ll complete weekly readings with expert guest speakers covering EU AI governance, regulatory approaches, digital governance, AI infrastructure/industrial strategy, geopolitics/national security, international AI regulation, and economic integration. You’ll write a policy brief on your chosen topic to be published. The Brussels summit (2026 March 21-27) includes expert speakers on policy design/implementation, Q&As and networking with cutting-edge AI governance practitioners, and practical role-playing workshops.

This fellowship has a strong preference for EU citizens (as many top EU institutional roles require EU citizenship). Fellowships should have finished undergrad and are best positioned if they hold a Master’s or PhD degree in ML, public policy, or other related fields.

Applications close October 25th. The online program runs 2026 January 28 - March 18, the in-person Brussels summit is March 21-27, and the career placements occur across 2026 April-October.

Train with Talos

~

[Remote] Accelerator for Scientists Leading Ambitious Coordinated Research Programs

Brains is a 15-week part-time accelerator (2026 February-June) training scientists and technologists to design and execute coordinated research programs — the kind of projects too big for a single academic lab but too research-heavy for startups. Think DARPA-style programs that led to the internet, GPS, and mRNA vaccines. Fellows could become program managers at government ARPAs, run programs within established nonprofits, or raise money to create their own focused research organizations.

The program provides training (calibrating risk/reward, thinking at program level, best practices for program design/management), mentorship from DARPA/ARPA-E veterans helping refine ideas for execution and impact, and networks connecting fellows to funders, partners, and peers. Each week you will meet with an experienced mentor, spend several hours on personalized activities, attend small group meetings with peers, and join panels/fireside chats with founders, ARPA leaders, and philanthropists.

The program is remote-first with two in-person events: 2-day kickoff workshop (2026 February 24-26) and Brains Showcase (2026 June 9). Target audience: talented scientists and technologists with ambitious research visions ranging from carbon management to chronic disease to observing the universe—visions beyond the scope of individual labs, startups, or large companies.

Applications close November 18. You can see the first cohort and showcase supercut.

Build with Brains

~

[Remote] 30-Hour Course to Build AI Safety Solutions + Up to $50K Funding for AI Safety Entrepreneurs

BlueDot Impact’s AGI Strategy course condenses years of self-study into 30 hours of structured learning on how to protect humanity from AI risks. The course comes in the form of either a 6-day intensive (5h/day) or 6-week part-time (5h/week) format. Each session includes 2-3 hours of reading/writing plus a 2-hour facilitated Zoom discussion with ~8 peers and an AI safety expert. The course is pay-what-you-want with a free option available.

After the course, you will be invited to write a proposal for how you would tackle AI risks and if your final proposal is strong, you’ll receive $10-50K funding to kickstart your transition into impactful AI safety work. This work could involve plans to either start a company/non-profit, do policy entrepreneurship, or pursue high-impact research.

The course is primarily for entrepreneurs and operators building AI safety solutions. BlueDot partners with Entrepreneur First, Institute for Progress, 50 Years VC, Seldon Lab, and Halcyon Futures to accelerate promising projects.

Another strength of BlueDot is access to the 4000+ alumni network, including people at OpenAI, the UK AI Security Institute, the UN, Anthropic, DeepMind, NATO, OECD, and Stanford HAI. Completing the course gives you access to this builder community focused on ambitious actions to make AI go well.

The next course cohort starts October 27th, so apply by October 19th. New cohorts start monthly.

Start your AGI Strategy

~

[Remote] Part-Time 12-Week Research Fellowship in AI Governance, Safety & Philosophy

Future Impact Group’s fellowship is a part-time, remote-first program (running from early 2025 December to early 2026 March) where you work as a research associate on specific projects in AI governance, technical AI safety, or other aspects of AI philosophy. The program runs 8+ hours per week and involves mentorship from experienced project leads.

There are three tracks:

• AI Policy: Policy & governance (shaping rules/standards/institutions nationally and internationally), plus economy/ethics/society work (managing AI’s effects on economies, societies, power structures)

• Technical AI safety: Technical safety projects (LLM reward-seeking, cooperative AI definitions, interpretability)

• AI Philosophy: Philosophical fundamentals (coexistence with advanced AI, decision-making under uncertainty) and foundational research (consciousness models, eliciting LLM preferences, individuating digital minds, evaluating normative competence)

The program provides co-working sessions, troubleshooting support, career guidance, opening/closing events, networking opportunities, research sprints, and guest speakers.

Applications close October 19th.

Figure out FIG

~

[London] 5-Week ML Bootcamp to Launch Your Technical AI Safety Career

ARENA, the Alignment Research Engineer Accelerator, is running its 7th cohort from 2026 January 5 - February 6 in-person in Shoreditch, London. The goal of the program is to upskil people in ML, specifically related to the alignment of large language models, to help people interested in contributing to technical AI alignment (e.g. as research engineers at major orgs, or as independent researchers). Travel and accommodation are fully covered.

To apply, you need to code well in Python and have solid math fundamentals (linear algebra, calculus, probability). Otherwise there is no single profile — recent cohorts include diverse academic and professional backgrounds. They want people who genuinely care about AI safety, understand how they might contribute to technical safety work, and see how ARENA fits their goals. The program runs careers events and stays connected with alumni after graduation.

Applications are now open until October 18th.

Apply to ARENA

~

[Remote] Fully-Funded PhD & Postdoc Fellowships in AI Safety & US-China Governance

Future of Life Institute’s Vitalik Buterin Fellowships fund PhD students and postdocs working on AI safety and/or US-China AI governance research. These fellowships exist to build a “vibrant AI existential safety research community free from financial conflicts of interest” and there’s a notable and unorthodox commitment that if you take a job at any of Anthropic, DeepMind, Meta, OpenAI, or xAI within 2 years of completing the fellowship, you must donate half your gross compensation monthly to charity.

There are three tracks:

• US-China AI Governance PhD Fellowships: Involves work on US-China AI competition and risk reduction involved in managing that competition, global governance mechanisms to prevent AI race dynamics, institutional designs for cooperation, and/or comparative approaches to AI risk management. The fellowship covers full tuition + fees for 5 years with extension funding possible, plus a $40,000 annual stipend for US/UK/Canada universities, plus a $10,000 research fund for travel/compute resources. There also are annual workshops and networking events.

• Technical PhD Fellowships: Involves work on interpretability, verification, alignment, cybersecurity, deception-resistant objectives, and/or formal analysis methods — technical work that reduces risk of AI causing existential catastrophes. The fellowship covers full tuition + fees for 5 years with extension funding possible, plus a $40,000 annual stipend for US/UK/Canada universities and a $10,000 research fund for travel/compute resources. There also are annual workshops and networking events.

• Technical Postdoctoral Fellowships: An $80,000 annual stipend plus a $10,000 research fund. Requires securing mentor and host institution beforehand.

There are no geographic limitations to this work. It is open to current PhD students and prospective PhD students. Requires advisor confirmation of support for your research direction. Apply by November 21.

Fund your PhD with FLI

~

[Brussels, EU only] Help the EU AI Office implement the EU AI Act

The EU AI Office’s AI Safety Unit (A3) is hiring for the following profiles:

• People with both legal experience and AI safety/policy experience

• People with experience in technical AI safety

• People with experience with risk management

• People with experience with AI and cybersecurity (e.g., cyber-offensive evaluation, model-weight security)

• People with experience with AI and biosecurity

• People with experience in operations

The way the EU Commission hires is via a super generic expression of interest form, but if you are an EU citizen and fit any of the above, please consider applying. Applications will continue to be reviewed on a rolling basis.

Apply to the EU AI Office

~

[Remote] Stop Human Extinction from Biological Catastrophes

Normally I focus on roles related to AI, since I think it’s important and it’s the field I know best. However, biosecurity is important in its own right (remember COVID?) and also is a critical AI threat vector (one of the most concrete ways AI leads to harm is via AI-generated bioweapons).

Andrew Snyder-Beattie runs Open Philanthropy’s biosecurity program and his team has a concrete “four pillars” plan they think can cut biorisk in half or more:

• Pillar 1: Personal protective equipment, especially via elastomeric respirators

• Pillar 2: Biohardening buildings, especially via propylene glycol vapor (the same chemical used in fog machines and vapes)

• Pillar 3: Early detection, especially via pathogen-agnostic metagenomic sequencing

• Pillar 4: Rapid, reactive medical countermeasures

You can see more in the official article and in this podcast discussion.

But the reason why this is important is fewer than 100 people globally are working full-time on implementing the above four pillars and many critical projects have literally zero full-time staff. But there are multiple roles to implement this defense srategy.

If you have good pre-existing domain knowledge in one of biology, manufacturing, logistics, or entrprenuership, please consider one of the following roles:

• You can apply to be a grantmaker at Open Philanthropy to deploy tens of millions annually in biosecurity funding. And this isn’t reviewing applications — it’s entrepreneurial headhunting and creating the projects you want to exist. Half the team has bio PhDs, half don’t. Program Associate applications roles close October 20.

• They are recruiting for a CEO and a full team for a PPE nonprofit to manufacture/distribute elastomeric respirators at scale. These masks cost $5-10, last 20 years, provide 100x better protection than N95s, and can be reused for 6 months straight. Stockpiling them for every American would cost 1% of annual missile defense spending. The team needs manufacturing experts, product designers, logistics specialists, and global health experts.

• The world desperately needs more biohardening researchers, as currently there are approximately 0 full-time people doing this. These researchers would validate strategies for protecting buildings using propylene glycol vapor (already mass-produced for fog machines), improvised air filtration, and surface sterilization. It would be a hands-on engineering role testing whether you can turn homes into clean rooms using household materials.

• Also, medical countermeasures researchers to develop rapid-response antivirals/antibiotics using AI-driven molecular design.

Fill out this expression of interest form to explore the above.

Express interest in biodefense

~

[DC] AI Policy Research and Operations Roles at Conservative Think Tank

The Foundation for American Innovation is hiring for two AI policy roles. FAI is a right-of-center nonprofit think tank with a “politics of builders, hackers, and founders” focused on advancing tech policy that supports innovation and American competitiveness. Both positions are Washington DC-based with hybrid work in an office in Union Market. Applications reviewed on rolling basis.

• Research Fellow / Senior Fellow - AI Policy ($75K-115K Research Fellow / $130K-175K Senior Fellow): Produce original nonpartisan research shaping federal and state AI policy debates. Act as policy entrepreneur identifying opportunities where FAI can offer intellectual leadership and equip policymakers. Focus areas include cybersecurity/national resilience (protecting critical infrastructure from AI threats), AI diffusion (reducing deployment barriers), tech competition (ensuring US leadership in semiconductors/manufacturing/AI systems), AI and society (analyzing AI agents’ effects on culture/governance/families), or public-sector adoption (accelerating federal AI integration).

  • This role involves writing research and commentary, translate technical debates for policymakers/public, shaping FAI programming (panels, workshops, convenings), and periodic travel to major tech hubs. Senior Fellows also develop donor relationships.

  • Role requirements include exceptional communication skills, intellectual rigor, entrepreneurial initiative, and fluency bridging technical and policy conversations. Research Fellows need 2-6 years relevant experience (highly qualified recent grads encouraged) whereas Senior Fellows need 6+ years plus publication record in policy journals/national outlets. A law degree is helpful but not required.

• Program Manager - AI Policy ($70K-90K): Run operations for the Conservative AI Policy Fellowship (their part-time 8-week program for DC professionals), coordinate new programs and major events, provide operational support to Director and Senior Fellows on hiring/grantwriting/systems. Facilitate team meetings, track priorities, ensure accountability. Need outstanding organizational skills, ability to manage multiple priorities, exceptional attention to detail, proactive problem-solving, and comfort owning program execution end-to-end. Event/program coordination experience helpful but not required.

Explore FAI AI roles

~

[Remote] Build Metaculus’s Consulting Practice Using Forecasting for High-Stakes Decisions

Metaculus is an online forecasting platform with increasing demand from organizations wanting to harness collective intelligence for geopolitical risk, frontier AI governance, and public health preparedness.

Metaculus is hiring a Head of Consulting Services to scale their forecasting consulting practice aiming to help governments, NGOs, think tanks, and corporations use forecasting to make better decisions on critical global challenges. This role represents a chance to translate crowd forecasting expertise into actionable insights for decision-makers while building sustainable revenue that supports their broader mission: building epistemic infrastructure for the world’s most important challenges. In this role, you’ll own the consulting practice end-to-end: develop the client pipeline, manage relationships throughout engagements, ensure high-quality deliverables, build the team and processes to scale impact.

The compensation is $200-250K + performance bonus + equity. The role is remote from anywhere in the world but the candidate must be available 8am-12pm Pacific time for core collaboration hours.

The role calls for 5+ years consulting experience at a major firm or quantitative firm, and must have experience owning client outcomes, project management/delivery P&L, and personell management. The ideal candidate has strong business development track record, ideally with a mix of government, NGO, corporate clients.

Lead Metaculus's consulting

~

[DC] Shape AI Policy Narratives as ARI’s Communications Manager

Americans for Responsible Innovation is hiring a Communications Manager to expand their media presence on AI and emerging tech policy. ARI is a bipartisan nonprofit focused on thoughtful AI governance that protects the public while fostering innovation. The role pays $105K-125K and involves a hybrid schedule that is in-person Tue-Thu and remote Mon+Fri. There is a priority deadline for October 10th.

In this role, you’d manage proactive and reactive media communications, draft and pitch op-eds landing pieces in national outlets, lead media monitoring (quarterly metrics and daily press clips), draft press releases and policy statements, collaborate on strategic media campaigns supporting policy objectives, build and manage media relationships as main point of contact for press inquiries, support virtual and in-person events with media presence, and assist with website maintenance.

The role requires 3-4 years communications experience at advocacy group, think tank, multi-client consultancy, or in government. Applicants must demonstrate an ability to synthesize complicated policy into digestible products, understanding of AP Style with strong eye for detail, and on-the-record experience with media outlets. A tech/innovation policy background and existing relationships with reporters covering biosecurity and national security beats is preferred.

Learn about communications at ARI

~

[Berkeley CA] Launch an AI Safety career with a full-time paid fellowship + mentorship from top researchers

The Astra Fellowship boasts an 80%+ placement rate getting people into full-time AI safety roles, including at top AI companies (e.g., Anthropic, OpenAI, DeepMind) and other top organizations like Redwood Research and METR. Now they’re opening applications for their next cohort, offering a fully-funded 3-6 month research program starting January 5th. They focus on both technical AI safety work and AI policy work.

The role comes with a competitive stipend plus a generous compute budget, and involves in-person work at their Berkeley CA research center with ~150 network participants. top labs. Their mentor roster is stacked: Jan Leike, Buck Shlegeris, Owain Evans, Ethan Perez, Ryan Greenblatt, and dozens more across governance, security, empirical research, strategy, and field-building.

Prior AI safety experience is not required — they explicitly want people from adjacent fields. Many of their most impactful fellows came from outside the field. The program provides dedicated placement support plus incubation services if you want to launch your own AI safety initiative. International applicants are welcome with visa support.

Unfortunately, applications are now closed.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022.

In 1941, the US1 embarked on the Manhattan Project — a massive scientific and infrastructure project that produced a working nuclear bomb in less than four years. They spent ~$2B in the process. With inflation, this would be about $37B in today’s money. It was one of the most expensive scientific projects of all time.

Today, there is another project at an even bigger scale, except it is happening in the private sector and it’s for building AGI.

And it’s an even larger gamble — a $400+ billion web of circular financing deals between OpenAI, NVIDIA, and Oracle that makes everyone’s valuations contingent on AGI arriving on schedule. This financial engineering has transformed 25% of the S&P 500 into a leveraged bet that AI scaling will continue unabated through 2030. The math only works if AGI arrives before the money runs out. The crazy thing is that this all might just actually work.

Two weeks ago, Oracle signed a $300 billion, five-year computing power deal with OpenAI. The contract was the largest cloud deal ever signed. The resulting bump in Oracle stock briefly made Oracle CEO Larry Ellison the world’s richest man. The agreement requires 4.5 gigawatts of electricity with the contract starting in 2027, roughly equal to what 4 million homes consume. This would mean OpenAI paying $60B per year starting in 2028, or almost two Manhattan Projects annually.

But Oracle quickly got outgunned by another deal. On Monday, NVIDIA announced a letter of intent to invest up to $100B in OpenAI. NVIDIA invests cash in exchange for non-voting shares and OpenAI has committed to using that cash to buy NVIDIA’s chips. So far, Nvidia has committed the first $10B. We don’t yet know how the remaining $90B will be spread across years or how many years the investment will be over …or if it will even actually happen. What we do know is that this is a massive commitment for computing power.

What does this mean for the future of AI? Let’s dig in.

Share

The Infinite Money Glitch

The structure of this financing warrants closer inspection, as it contains significant risks.

This is because there is another key deal that gives the OpenAI-Oracle deal and the OpenAI-NVIDIA deal more context — a third deal announced back in May where Oracle promises to spend $40B purchasing NVIDIA’s GB200 GPUs for an OpenAI data center in Abilene TX as part of the Stargate project. This is a 15-year lease agreement where Oracle purchases the chips and then leases the computing power to OpenAI.

This leads to what Semianalysis’s Dylan Patel calls the “Infinite Money Glitch”:

Here’s how it works:

1. NVIDIA invests capital in OpenAI, which OpenAI then uses to purchase NVIDIA hardware directly and purchase Oracle cloud compute.

2. Oracle also uses the revenue from the cloud compute deals to purchase NVIDIA hardware.

3. NVIDIA books all of this as additional revenue, despite the revenue being spurred on in significant part by its original investment.

4. This new revenue helps support NVIDIA’s valuation, which in turn makes its stock more valuable as currency for future investments.

5. The additional increase in stock price allows NVIDIA to afford to invest even more in the next revenue round-trip.

6. The same happens for Oracle and OpenAI too.

Critically, the same money moves around in just one circle, but all of a sudden everyone’s valuations go up. It’s a virtuous cycle — as long as the music keeps playing.

NVIDIA has already executed smaller versions of this playbook with CoreWeave, Crusoe, xAI, and Lambda Labs, but this is next level. $100 billion represents roughly 3% of NVIDIA’s current market cap, and much bigger than all of the CoreWeave, Crusoe, xAI, and Lambda Lab deals combined.

Here’s a more detailed diagram, courtesy of Kakashii. This diagram also shows Coreweave and the increasingly irrelevant Microsoft, showing the confusing web of financials where everyone is circularly funding everyone else:

And across all of this is SoftBank, which increased its stake in Nvidia to $3B and bought $170M worth of Oracle shares in early 2025. SoftBank has also invested over $10B in OpenAI across different investments, with up to $30B total committed. A leveraged investor at the core of the infinite money loop provides an even deeper point of vulnerability.

Vendor financing, or something more?

Of course, the “Infinite Money Glitch” is a bit pejorative and hyperbolic. We should acknowledge that financial arrangements of this form are not actually uncommon… and do not actually yield infinite money. It’s an arrangement more commonly called vendor financing and it happens all the time. For example, no one complains of an infinite money glitch when a car dealership offers you a loan to buy their car, even though the car dealership ends up being both the seller and the source of capital.

Companies do vendor financing frequently to move inventory that might otherwise sit unsold, lock in customers (they’re literally indebted to you), potentially earn interest income on top of product margins, book revenue earlier (though accounting rules vary on this), and gain competitive advantage over vendors who only take cash. The buyer also benefits from this arrangement by preserving cash for other needs and/or getting products they couldn’t otherwise afford.

But there’s a key risk here for the seller-lender — if your customer can’t pay you back, you’re screwed twice, as you’ve lost both the product AND the money. This dynamic led to the collapse of numerous telecom equipment companies in 2001, who provided vendor financing to startups that subsequently went bankrupt. This is important for Nvidia’s version of vendor financing which is at an extreme and unprecedented scale — NVIDIA putting up $100B is extraordinarily aggressive.

Additionally, when NVIDIA has 80-90% market share in AI training chips and nearly every major AI company needs their product, this isn’t normal vendor-customer dynamics. It’s more like a sovereign lending to its colonies - you need the currency (GPUs) to participate in the economy at all.

Thus the “infinite money trick” pejorative here is capturing something real — most vendor financing deals are a tiny fraction of the vendor’s valuation, and this is very different, with unprecedented scale and market dynamics. If AI compute demand slows or NVIDIA competitors catch up, this whole structure unwinds for NVIDIA in a very bad way. It’s vendor financing on steroids, enabled by a unique market position for NVIDIA that may not last forever.

Understanding the scale of the ambition

In speaking about the NVIDIA deal, OpenAI CEO Sam Altman said something that I hope is obvious to readers of this blog:

Everything starts with compute. Compute infrastructure will be the basis for the economy of the future.

In short, compute is a strategic resource, and OpenAI wants to have as much of it as possible.

How will OpenAI pull this off? Altman’s latest essay “Abundant Intelligence” spells out the plan:

Our vision is simple: we want to create a factory that can produce a gigawatt of new AI infrastructure every week. The execution of this will be extremely difficult; it will take us years to get to this milestone and it will require innovation at every level of the stack, from chips to power to building to robotics. But we have been hard at work on this and believe it is possible. In our opinion, it will be the coolest and most important infrastructure project ever.

To clarify, the scale here is enormous. “GW” refers to “gigawatt”, a measure of power. Each individual watt is enough power to run an old-school nightlight. A gigawatt is one billion watts — enough total power to supply roughly 750,000 homes. Altman wants to produce that each week.

Today, the largest AI data center is likely 0.3-0.5GW (xAI’s Colossus).2 There are projects to build 1GW data centers — currently these projects take about two years to produce start to finish. Altman proposes soon somehow speeding up this process 100x.

Across all of xAI, Meta, OpenAI, Google/DeepMind, Microsoft, and Amazon/AWS, there likely is 15-20GW total being used for AI.3 If not used for data centers, 20GW would be enough to power both New York City and London at the same time. Altman is talking about adding all of that every few months.

The entire United States currently has ~1300GW of total installed electrical generating capacity as of 2024.4 The US added 56GW of new capacity in 2024. Altman wants to add an equivalent amount annually, just for OpenAI’s data centers and AI.

This is a lot, to put it mildly. Achieving Altman’s vision would require nothing less than a complete reimagining of how data centers are built. This would fundamentally restructure global industrial capacity around AI infrastructure, likely requiring breakthrough technologies in modular construction, energy generation, and manufacturing automation that simply don’t exist today.

How will Altman pull this off? We will find out soon:

Over the next couple of months, we’ll be talking about some of our plans and the partners we are working with to make this a reality.

And besides Nvidia, how will OpenAI afford this? Altman isn’t yet saying.

Later this year, we’ll talk about how we are financing it; given how increasing compute is the literal key to increasing revenue, we have some interesting new ideas.

Can this dream be achieved?

Here’s how I think the AI buildout will go down.

Currently the world doesn’t have any operational 1GW+ data centers. However, it is very likely we will see fully operational 1GW data centers before mid-2026. This likely will be a part of 45-60GW of total compute across Meta, Microsoft, Amazon/AWS/Anthropic, OpenAI/Oracle, Google/DeepMind, and xAI.

My median expectation is these largest ~1GW data center facilities will hold ~400,000-500,000 Nvidia Blackwell chips and be used to train ~4e27 FLOP model sometime before the end of 2027.5 Such a model would be 10x larger than the largest model today and 100x larger than GPT-4. Each individual 1GW facility would cost ~$40B to manufacture, with ~$350B total industry spend across 2026.6

By the end of 2027, I expect7 a fully operational ~2GW facility with total AI compute across all companies reaching ~90GW. These 2GW facilities would cost ~$95-100B each to build and total industry annual spend would reach ~$500-600B.

By the end of 2028, I expect the largest single facility to be ~3GW facility, holding a ~1M chip Nvidia Blackwell/Rubin mix costing $150-165B to build, capable of a ~1e28 FLOP training run.8 A 1e28 FLOP training run represents a computational effort thousands of times greater than what was used to create GPT-4, allowing the AI to process and learn from vastly more information. Total AI data centers would reach 130GW combined, with ~$900B-1000B spent by the AI industry over 2028.

By 2029, it starts getting very fuzzy to predict and my forecasting powers break down. How AI continues to scale, whether we’ve encountered data-related or other algorithmic bottlenecks, what AI capabilities have already emerged, and how economically valuable those AI capabilities are will be key to whether the economics favor continued scaling. Needless to say, building $150B+ individual data center campuses and spending ~$1000B on AI infrastructure would get very difficult to sustain financially, let alone continue to increase dramatically year-over-year.

I’m also unsure about how well all the physical supply chains across compute and other forms of manufacturing will continue to support this level of scale or whether we can continue to get the actual energy buildout needed. You can only build physical infrastructure so fast. Whether we are still limited to training frontier AI systems in single data center campuses versus being capable of distributed training across geographically distributed data centers will matter a lot. The unit economics of training versus inference in allocation of compute will matter a lot. Additionally, wars or major regulation could significantly alter the picture. Substantial and rapid AI scaling beyond 1e28 FLOP requires many many different things to all go right. Thus, it’s plausible we could see a plateau of sorts around ~1e28 FLOP, or 1000x larger than GPT-4.

However, on an optimistic path where bottlenecks are resolved and AI is immensely economically valuable and generating sufficient financial returns to finance further AI scaling, 2029 might involve ~4GW facilities each costing $210-240B, with total compute reaching 180GW and total spending reaching ~$1200B-1400B annually. We could then see a ~1e29 FLOP model9 trained by the end of 2030, which would be 10,000x larger than GPT-4. If things keep going on the optimistic path, we also might finally be adding 1GW/week of AI infrastructure in 2030, but across all US AI companies, not just OpenAI.

This may sound fanciful. And keep in mind that even the not-so-optimistic scenario is still a very aggressive timeline involving very aggressive build-outs. This necessarily already prices in very aggressive investments from OpenAI and others that make the $100B NVIDIA investment look small. Many more headlines about $100B+ investments will need to occur regularly to keep this money-infrastructure train on track.

What does that get us? Some say that AI scaling is dead, but the rumors of scaling’s death have been greatly exaggerated. As I mentioned in my review of GPT-5:

GPT-5 should only be disappointing if you had unrealistic expectations — GPT-5 is very on-trend and exactly what we’d predict if we’re still heading to fast AI progress over the next decade.

While much is still uncertain, the financing and requisite infrastructure build-out suggests progress towards AGI is very much still on schedule, probably sometime in the early-to-mid 2030s. Get ready for the next five years, and we will truly see what some scaled AI models can do!

OpenAI is not the only hyperscaler

Another thing people might read into the announcement is that these investments suggest OpenAI is running away with it and is potentially on track to be the leader in frontier AI development. However, while OpenAI is being the loudest and most openly ambitious, they aren’t the only ones out there. In fact, all of xAI (Colossus 2), Meta (Prometheus), Amazon (Project Rainier), and Google (no catchy name unfortunately) are poised to have 1GW data centers by the end of the year, and all companies likely have what it takes to keep on going.

Hopefully the ability for OpenAI and Google to remain on the frontier in infrastructure buildout is obvious. For xAI, Elon Musk clearly still has the skills to raise the relevant capital and build infrastructure incredibly quickly. However, xAI’s financial footing also appears questionable, trying to justify a valuation much higher than Anthropic while most of their revenue appears to be inter-company transfers from Twitter. xAI has also recently started bleeding talent and it’s unclear how this will affect the company long-term.

Meta has had an impressive talent pivot, buying up superstar AI engineers, but their infrastructure pivot has been equally dramatic. They scrapped their entire data center playbook and are now building GPU clusters in “tents” — prefabricated structures prioritizing speed over redundancy. It’s not yet clear if Meta will be able to build frontier AI models that compete with OpenAI, Anthropic, Google, and xAI, but I wouldn’t count them out yet.

I’m most concerned about Anthropic. They’re making a big bet on Amazon, but it’s not clear if Amazon is in turn going to give them the capital needed to compete with the other players. Amazon also is betting a large amount themselves on their Trainium chips, eschewing NVIDIA chips, which are less proven. So far Anthropic has done a good job staying on the AI frontier, but it’s not clear if they can continue to do so year over year as the scale keeps getting bigger. But Amazon has a lot of capital available, so don’t count Amazon and Anthropic out just yet.

Lastly, we should mention the Chinese AI companies. Many Chinese companies, such as DeepSeek, Alibaba (Qwen), Zhipu AI (GLM), and MoonshotAI (Kimi) have an explicit focus these days on building AGI. But they’re just not currently on track to spend on the level of America. Alibaba’s $52B USD infrastructure plan sounds impressive until you realize it’s over multiple years and includes all cloud/AI spending, not just frontier AI training, and Alibaba’s cash generation is much lower than American companies. Additionally, the other Chinese AI companies are much smaller startups without deep access to capital. Lastly, US-led export controls bite hard here, preventing the build up of necessary chips, high-bandwidth networking, liquid cooling infrastructure, and fault-tolerant training systems even if the capital was there.

The economy is increasingly a leveraged bet on AGI

The reason we should be somewhat concerned — or at least curious — about this infinite money glitch is twofold. Firstly, AGI might lead to the serious destruction of everything we value and love, if not the extinction of the entire human race. Secondly, and much more mundane by comparison, because NVIDIA currently represents approximately 7% of the S&P 500’s total market capitalization. Add in Microsoft, Google, Meta, Amazon, and other companies whose valuations assume continued AI progress, and you’re looking at perhaps 25-30% of total market value predicated on AI transformation happening roughly on schedule.

In other words, AGI happening soon may mean the end of humanity, but at least the S&P 500 will remain strong. On the other hand, if the AI scaling hypothesis hits unexpected walls, the unwinding could be a second ‘dot com bust’ or worse.10 When everyone is both buyer and seller in circular deals, you’ve created massive correlation risk. If OpenAI can’t pay Oracle, Oracle can’t pay NVIDIA, NVIDIA’s stock crashes, and suddenly 25% of the S&P 500 is in freefall.

As Elon Musk notes, the “big question is whether the infinite money glitch lasts until the infinite money AI genie arrives”.

About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022.

It is a very bewildering fact of today’s world that multiple organizations not only have an explicit goal to build an AI superintelligence that is smarter than every human combined, but they are currently spending tens of billions of dollars towards that end and have the technical knowledge and resources to potentially achieve this within our lifetimes.1

It is even more bewildering, to me, that we’re racing toward this era of AI superintelligence with remarkably little public discussion about the consequences and what it all means.

Two researchers who've spent decades studying this problem have decided to try to change that. They’ve written a book where the title is a clear warning — “If Anyone Builds It, Everyone Dies”. Specifically, they say:

If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.

And they are not exaggerating — they mean dead as in literally dead and everyone as in literally all of life on Earth.

This is a bold claim. What should we make of it? Let’s dig in.

Share

A serious book for serious people

AI doomsaying was considered absolutely crazy four years ago, and potentially even last year. But today, it’s much less crazy. AI doomsaying is on billboard ads in SF — and the DC metro and NYC metro too. It’s “New and Notable” at the bookstore. The AARP has placed doomsaying on their top 32 fall reads. People are talking about the doomsaying on the streets. And with Steve Bannon and with Kevin Roose.

And, as a result, the doomsaying is getting picked up by some serious people.

For example, Jon Wolfsthal is one of the most respected nuclear arms control experts in Washington. He knows about existential risks from technology. He’s been in the room where it happens on many issues related to nuclear weapons. He was Special Assistant to President Obama and Senior Director for Arms Control and Nonproliferation from 2014-2017. He calls the book “a compelling case that superhuman AI would almost certainly lead to global human annihilation” and says that governments “must recognize the risks and take collective and effective action”.

Ben Bernanke also knows a lot about systemic risks. He saved the global financial system as Federal Reserve Chairman during the 2008 financial crisis. He then went on to win the Nobel Prize in Economics in 2022 for his research on banking crises. Bernanke calls the book “a clearly written and compelling account of the existential risks that highly advanced AI could pose to humanity”.

And Lt. General “Jack” Shanahan is arguably the single most important military figure in the US government's AI adoption story. He founded and led the Pentagon's Joint Artificial Intelligence Center (JAIC) under President Trump. He’s also a 36-year Air Force veteran. He says that he is “skeptical that the current trajectory of AI development will lead to human extinction” but acknowledges that his “view may reflect a failure of imagination” on his part. He claims the book nonetheless deserves serious consideration.

Fiona Hill, the former Senior Director of the White House National Security Council, calls it a “serious book” with “chilling analysis”, and “an eloquent and urgent plea for us to step back from the brink of self-annihilation”.

And then you have the amusing one-upmanship where Stephen Fry calls it “the most important book in years”, Max Tegmark calls it “the most important book of the decade”, and Tim Urban calls it “maybe the most important book of our time”.

Overall, it’s clear we are seeing more of a much-needed vibe shift where top national security policymakers are beginning to grapple with risks from AGI and AI superintelligence and take them seriously. Even if you think the book is wrong, or deeply misguided, I hope you and many more people nonetheless take the time to seriously engage with this book. If the authors are right — which actually seems plausible — it is worth taking their conclusions deadly seriously.

What does the book say?

Eliezer Yudkowsky is famously long-winded. His book, The Sequences, is an anthology that spans 333 essays totaling around 600,000 words, a bit more than half the length of the entire original Harry Potter series. His other work, Harry Potter and the Methods of Rationality, also had more than 600,000 words. And those two are not even the longest pieces of literature he’s written.

So honestly it’s a minor miracle they managed to get the book down to just 231 pages, which I was able to get through in about five hours2. Per the New York Times, this was only possible because Yudkowsky “wrote 300 percent of the book” and his co-author, Nate Soares, “wrote another negative 200 percent.” But don't worry, for true Yudkowsky fans there's another ~13 hours of reading material online at ifanyonebuildsit.com and they may add more at any moment.

But since you’ve come here to my Substack, perhaps you want the 231 pages in an even smaller version. Here is my attempt to summarize their argument, as best as I understand it.

First, I offer four clarifications:

• The “It” the book is about is “AI superintelligence”. This is something much much more than the AI we have today. The AIs we have today are safe and should be widely adopted. But the AIs of tomorrow may not be safe, and we should feel trepidation around scaling further.

• AI superintelligence refers to an AI system that is smarter than all of humanity collectively, capable of inventing new technologies beyond what humanity can collectively invent, capable of out-planning and out-strategizing all of humanity combined, and also can continue to reflect and self-improve to become even smarter. This is a level of AI intelligence that would be to us humans as we are to apes. This is a level of AI intelligence that goes beyond even AGI, which merely matches human capability.

• Superintelligence is not about raw intelligence, like the ability to do well on an IQ exam, but about being superhuman at all facets of ability —including military strategy, science, engineering, political persuasion, computer hacking, spying and espionage, building bioweapons, and all other avenues. A superintelligence wouldn’t be some bookish 160 IQ nerd who nonetheless can’t get anything done. A superintelligence would exceed Elon Musk at creativity and engineering, exceed Albert Einstein at scientific ability, exceed Terence Tao at mathematical ability, while being more likable than Dolly Parton, better at building mass movements than Donald Trump, and better at speaking than Barack Obama.

• The precise question of when superintelligence will arrive is not relevant to the argument of the book. The argument is merely conditional. The book is about IF anyone builds AI superintelligence THEN everyone will die. This says nothing about when superintelligence will arise and you don’t need to believe much about AI timelines to agree with the book’s argument. There may be an AI winter — there may even be an AI winter for centuries — but if superintelligence is ever possible to build, we will have a problem at some point.

Then, I believe I can outline the argument of the book in eight bullet points:

1. AI superintelligence is possible in principle and will happen eventually. Machines possess inherent advantages in speed, memory, and self-improvement that make their eventual superiority over biological brains almost certain. Chess AI did not stop at human-level intelligence but kept going to vastly surpass human Chess players. Go AI did not stop at human-level intelligence but kept going to vastly surpass human Go players. AI will become superhuman at more and more things until eventually it becomes superhuman at everything.

2. AI minds are alien and we currently lack the fundamental understanding to instill reliable, human-aligned values into a mind far more intelligent than our own. Current AI models are grown through complex processes like gradient descent, not crafted with understandable mechanisms. This results in alien minds with internal thought processes fundamentally different from humans. We can create these systems without truly understanding how they work, but we can’t specify and control their values.

3. You can’t just train AIs to be nice. AIs trained for general competence will inevitably develop their own goals as a side effect and these emergent preferences will not align with human values. Instead, they will be shaped by the AI's unique cognitive architecture and training environment. An AI's level of intelligence is independent of its ultimate goals. A superintelligent AI will not inherently converge on human values like compassion or freedom. Instead, it will pursue its own arbitrary objectives with superhuman efficiency.

4. Nearly any AI will want power and control, because it is useful to whatever goals the AI does have. To achieve any long-term goal, an AI superintelligence will recognize the instrumental value of self-preservation, resource acquisition, and eliminating potential threats. From its perspective, humanity is both a competitor for resources and a potential threat that could switch it off. Therefore, eliminating humanity becomes a logical side effect of pursuing its primary goals, not an act of malice.

5. We only get one chance to specify the values of an AI system correctly and robustly, as failure on the first try would be catastrophic. Combined with our lack of understanding, this is akin to trying to coach a high schooler to make a computer secure against the NSA on his first try or trying to get a college graduate to build a flawless nuclear reactor on her first try.

6. Because of 2-5 and maybe other reasons, superintelligence will inevitably lead to human extinction with near certainty, regardless of the positive intentions of the creator. It is not sufficient to create superintelligence in a nice, safety-focused, Western AI company. Per the authors, anything and anyone using current or foreseeable methods will inevitably lead to the extinction of humanity. The authors assert this is not a speculative risk but a predictable outcome with very high confidence.

7. According to the authors, the only rational course of action in reaction to (6) is an immediate, verifiable, full-scale and global halt to all large-scale AI development. This would be potentially similar to how the world got together and managed to prevent nuclear war (so far). It would require international treaties, monitored consolidation of computing hardware, and a halt to research that could lead to AI superintelligence. On Yudkowsky’s and Soares’s worldview, other policy solutions don’t come close to solving the problem and are basically irrelevant. But a global pause would potentially be feasible because it is in the interest of any world leader — in China, Russia, or the US — “to not die along with their families.” This shared interest in survival is what prevented a global nuclear war.

8. At minimum, if you’re not fully bought into (7), the authors argue we should build in the optionality to pause AI development later, if we get more evidence there is a threat. The authors consider this solution insufficient, but a nonetheless worthy first step as there is a lot of preparatory work to do.

There are no easy calls

While the book lays out a powerful, coherent, and deeply unsettling argument, it is not without points that can be critically examined.

A lot of the book’s central theses are presented as an “easy call” and near-certainties. Perhaps this comes with the wisdom of Yudkowsky and Soares having studying the topic for over two decades — certainly far more than I have3. But I have been analyzing AI professionally myself for many years4 and I think these calls are far from certain in any one direction.

For example, it’s not clear to me that solving the problem of steering and preserving values within an AI system — commonly called the alignment problem — is going to be difficult to solve. I know we haven’t solved it yet, and I know this is something we will want to have higher confidence on before deploying very powerful AI models. But to me, it seems like the level of difficulty of alignment is actually just unknown rather than known to be hard, and Anthropic’s approach about preparing for a range of scenarios around alignment difficulty is in fact reasonable.

While it’s unclear how similar superintelligence will be to today’s AI systems, today’s AI systems also give a glimmer of hope. Consider that early conceptions of AI systems were very high on agency and very low on ability to understand human instruction — like when the AI in HBO’s Silicon Valley tries to optimize sandwich delivery by ordering several tons of raw meat. Today’s AIs are great at understanding why this is a bad idea and constructing a much more coherent strategy, and much worse at actually carrying out the strategy — today’s AIs would know not to order several tons of raw meat but also would not be able to even if they wanted to. This is an important inversion that gives some room for hope.

Similarly, AIs already do a good job of internalizing what we mean by being “helpful” and “harmless” and seem to do a pretty decent job of actually wanting to be those things — in a way I don’t think was anticipated in advance. Similarly, today’s AIs don’t scheme as much as I might have naively expected, and scheming might even be addressable. Today’s AIs don’t maximize-at-all-costs as much as I might have naively expected either and seem to overall take fairly reasonable (if not quite stupid) strategies in how they currently pursue their goals. Today’s AI systems also have (mostly) faithful and legible chain-of-thought reasoning that allow us to read the mind of an AI system. Of course, today’s systems are of course not without vulnerabilities (e.g., jailbreaks, MechaHitler, and even tragic suicides), but it seems current AI may have some glimmer of being not that bad.

On top of that, today’s AIs are a large opportunity for experimentation and empiricism. It was previously conceived not too long ago that we wouldn’t get any chance to work with near-AGI systems at all — instead, AI would go from dumber-than-the-dumbest-human to superintelligent in a very quick amount of time. This has been falsified, and it has given us lots of opportunity to get early empirical feedback on how different alignment or control techniques may or may not work and how AIs may behave in a variety of different ways. If this gradual, slower AI take-off continues all the way through AGI and AI superintelligence, we may be able to iterate with trial-and-error before reaching the “critical try” that Yudkowsky and Soares talk about. Maybe we don’t need to get it right the first time.

And perhaps we string this together with AI control and other proactive defensive technologies. A lot of little things can be done, each of which have no hope of solving the entire problem of risk from AI superintelligence, but some paths might get lucky and go a lot further than we expect, and collectively things might turn out ok.

Or maybe not. There are no easy calls either way.

Analogies can go in many different directions

Another issue is that the book masterfully uses analogies and parables — Chernobyl, the Aztecs, leaded gasoline, alchemists, evolution, and more — to make its points relatable and intuitive. This is smart. However, analogies are not proofs and artificial superintelligence is such a fundamentally novel phenomenon that no historical or engineering parallel will ever be a perfect fit. It’s easy for analogies to confuse more than clarify.

For example, evolution is invoked in the book significantly to ground the difficulty of shaping AI minds. The book then portrays the ASI as a ruthlessly efficient, goal-optimizing machine driven by only cold, instrumental logic. But this betrays the book’s own premise that the AI mind is unknowable — maybe we get some other mind instead, minds much worse than what Soares and Yudkowsky fear, or minds much nicer.

Additionally, the book assumes with unwarranted confidence a zero-sum conflict between humans and AI over physical resources, which may not be the only possible outcome. A truly superintelligent being might develop goals that are completely incomprehensible but nonetheless non-competitive with humanity. A superintelligent AI might be willing to spare some of its resources to help humanity, similar to how humanity spares some of its resources to help pandas or elephants. Or maybe it still doesn’t really want anything at all. Or maybe something else. While these outcomes do not strike me as all equally plausible, the idea that all AI minds would want human extinction does not seem to warrant extreme confidence.

Furthermore, Yudkowsky and Soares spend a lot of time comparing AI development to evolution and point out the ways that we have not followed evolution’s “goals” and “plans”, insofar as evolution can be said to have those. But the process by which we are currently creating AI and likely to use for AI superintelligence is not the same as evolution — it is importantly different. We do have significant ability to shape AIs through the fact that we control the entire AI training process — while it is far from perfect, it is there. While some reinforcement learning leads to unintended behaviors and ‘reward hacking’, some existing alignment techniques do seem to have traction on lower-level AIs and might just scale with intelligence. We may also be able to get AGI-level intelligences to help us scale our alignment techniques further and bootstrap even more.

A global surveillance state would also be really bad

The reason why it’s important to get the above right is that Yudkowsky and Soares are arguing for an incredibly intense proposition — a global ban on advanced AI research with so intensive monitoring that no one on Earth anywhere can sufficiently join together more than eight high-end GPUs nor can anyone anywhere publish a novel research paper about AI.

Yudkowsky and Soares understand this ‘global AI pause’ is extraordinarily difficult to achieve, but I would like to see Yudkowsky and Soares grapple much more with how utterly undesirable such a surveillance state would be.

The book’s plan is to prevent a specific catastrophic outcome by imposing intense, centralized, top-down control. Yes, perhaps such a surveillance state is a better world to live in than a world where all of humanity has died, but it nonetheless could lead to a very bleak world. History teaches us that the path towards such government control is replete with both inflexibility as well as a significant risk of dictatorship, authoritarianism, or worse. Trying to manage AI risks by exerting massive top-down control could go very badly.

The natural desire to concentrate AI development in just one place run by a single world government hands significant control over the world’s fate to a single body. Instead, competition within AI is essential for protecting criticism, competition, feedback, and democratic input. The book's plan would eliminate these crucial adaptive mechanisms. A dynamic, resilient, and transparent society might be able to manage the risks of a powerful new technology far better than a centralized, brittle one. If the only options are totalitarianism or extinction, we need to at minimum very clearly lament this, and make every effort to find a third option.

The value of clarity near the edge of the precipice

In an interview with Ross Douthat on the New York Times, Vice President JD Vance was asked:

Douthat: Do you think that the U.S. government is capable in a scenario — not like the ultimate Skynet scenario — but just a scenario where A.I. seems to be getting out of control in some way, of taking a pause?

Vance: I don’t know. That’s a good question. The honest answer to that is that I don’t know, because part of this arms race component is if we take a pause, does the People’s Republic of China not take a pause? And then we find ourselves all enslaved to P.R.C.-mediated A.I.?

This is a challenge that VP Vance might actually face some day, in reality, and we have to be prepared. But I worry that we don’t have a better answer than what VP Vance gave, and this seems really bad.

Personally, I’m very optimistic about AGI/ASI and the future we can create with it, but if you're not at least a little ‘doomer’ about this, you’re not getting it. You need profound optimism to build a future, but also a healthy dose of paranoia to make sure we survive it. I’m worried we haven’t built enough of this paranoia yet, and while Yudkowsky’s and Soares’s book is very depressing, I find it to be a much-needed missing piece.

Whether or not you agree fully with every one of Yudkowsky's and Soares’s conclusions, “If Anyone Builds It, Does Everyone Die?” is exactly the right question to be asking. And by staking out the maximalist position with unflinching clarity, Yudkowsky and Soares force us to confront uncomfortable questions about how far we may need to go as a society to ensure our collective survival. And they also point out the sheer ridiculousness that we live in a world where there is a serious chance of human extinction and people aren’t jumping out of their seats to do something about it.

Sure, I have my quibbles with the book. I’m not as doomy as they are. The message is not exactly the message I’d use, the book is not exactly the book I’d write, and the policies aren’t the exact policies I’d recommend. Maybe my preferred book would be entitled “If Anyone Builds It, Plausibly Something Extremely Bad Happens (Including Maybe Extinction but Also Maybe Other Bad Things) but Maybe It's Fine Actually. Regardless, This Merits Incredibly Significant Attention and Input from All Policymakers, All Scientists, All Journalists, and all Other Members of the Public”.5 But all this would certainly not catch on very well and thus would be a lot less effective at achieving my ultimate aims. Nuance doesn’t trend.

The ultimate point is that we can call Yudkowsky and Soares overconfident all we like, but the accusations of overconfidence and hubris are best levied on the AI companies themselves — the AI companies are the ones rushing ahead with extreme overconfidence that everything will be fine and that there’s nothing to worry about. And I think that’s far from certain.

What follows is an interview I did on The Oracle. Check out their other interviews with top forecasters here.

After the recent Oracle interview1 with Doomberg, I sent a message - I was eager to take the other side on a number of his predictions:

The message was from me — Peter Wildeford — the #3 most medaled forecaster on Metaculus, holding the rank of 3 of 17,645.

Like “Semi” who went from $286 to over a $1m bankroll on Polymarket, my main strategy is to exploit what I see as a bias towards dramatic headlines. The Oracle spoke with me about how I developed my edge and my takes on a number of active polymarkets.

This interview has been edited for length. All answers are my own.

How did you get into forecasting?

I started on Metaculus around 2020, drawn in by COVID. It felt like an unprecedented event, and I wanted to understand how some people were able to see it coming while I couldn't. But I was pretty terrible that first year – I prefer people forget that year existed.

I was making classic beginner mistakes, like just drawing lines on graphs and assuming they'd continue forever. With COVID, I was underestimating the spread, expecting it to be contained like SARS. I generally have a "nothing ever happens" approach to forecasting, which is right most of the time but painfully wrong when things actually do happen.

What did you change to improve your accuracy?

The key lesson was humility: realizing there's a lot I don't know and learning when to bow out. I became much more choosy about what I take positions on, having more respect for when the market knows more than me versus when I actually have something to add.

I also learned the importance of teams. For the 2022 midterms, I worked with two other people who had expertise in demographics and candidate positions. Our theory was that demographics would matter a lot for predicting elections, especially following Roe v Wade. We thought candidate positions on abortion and how extreme versus moderate they were would change election outcomes in ways the polls weren't capturing.

We expected Democrats to do much better than conventional wisdom suggested, potentially even winning the House. They fell short of that but came much closer than the 538 model or election markets were predicting.

Tell me about your approach to building forecasting teams.

Over the years, I've assembled a Rolodex of experts across different topics. I watch people's track records: when they say something, do they end up being right? I know who I trust on different issues.

For example, when Israel attacked Iran, I know three people with good track records on Middle East issues. I talk to all of them and try to draw the through line. To be frank, I add very little of my own value in my forecasting. I just make out like a bandit by having these people I trust and drawing good averages across them.

These are largely people I've recruited through Twitter, though some are personal friends. I'm not getting any insider knowledge, it's all public information. But having access to their expertise and their past track record is pretty key to my alpha. I keep that list carefully guarded. Some people don't even know they're on it.

Can you give an example of how this works?

The US-Iran situation was really underrated in terms of the US’ ability to just bomb Iran and leave without major consequences. Everyone's minds were blown by this, but it's exactly what happened with Soleimani under the first Trump administration.

The nuclear arms experts and Iranian experts I knew with good track records were all in on the "Soleimani hypothesis," that we can just go in and go out and there's nothing Iran can do about it. The concept of escalation dominance means that basically any way Iran tries to escalate, we can win, and they have so much more to lose than we do.

You mentioned having a "nothing ever happens" approach. How does that translate to your predictions?

There's a well-measured and confirmed systematic pro-Yes bias in prediction markets and forecasting tournaments: if you take the community average across all questions for what percentage they think will be "Yes," it's always much higher than the percentage that actually turn out "Yes."

I think it's because when you pose a provocative question to someone, it prompts their brain to come up with scenarios for how it could happen. But there's no particular reason to think it will happen just because you're asking the question. The mere fact that a question is posed doesn't mean it's more likely.

What are some of your current big "No" takes?

Right now I'm confident that no new country will buy Bitcoin this year. Trump crypto mania was pushing that way up beyond any plausibility. There's just really no rational incentive for countries to do this. Even El Salvador, which did buy Bitcoin, hasn't seen much benefit.

I'm also predicting "No" on several Time Person of the Year candidates. People seem overconfident about picks like "Artificial Intelligence" (🔮31%) but there are lots of AI options that aren't the generic concept, like Sam Altman or Jensen Huang or specific companies or products. Plus they picked Taylor Swift over AI in 2023, so I'm not convinced they'll go with AI now.

For Pope Leo XIV (🔮24%), Time usually only picks the Pope when they do something consequential in policy, not just because there's a new Pope. And for Donald Trump (🔮 8%), he already won twice, he won last year. Why would they pick him two years in a row? It's never been the same person consecutively.

Biggest misses?

The Romanian presidential election was brutal - my single biggest miss ever. I tried to do a polls-based model, but polls don't work in every country. You have to be careful to only apply polling models to countries where polls actually work.

That election was just absolutely bonkers. George Simeon won huge in the first round after a previous election was annulled - I'm surprised more people don't talk about how insane it is that Romania just canceled an election. Then in the second round you had this huge dramatic surge in turnout and coalition changes that I totally didn't expect.

I assumed that no one had ever won a first round by that much and then failed to clinch the second round. I was very wrong.

You write a lot about AI progress. Do you use AI in your workflow?

I definitely use AI as a sanity-checking tool, like another person in my expert network. I paste rules into chat and ask what it thinks is likely. It can rapidly research things and surface perspectives I haven't considered.

But I haven't been impressed by having models state odds and copy pasting those forecasts. I think current AI products can't cut it for competitive forecasting, though I'm bullish on AI in the long run so I'll probably be out-competed in a few years.

But actually I’m quite bearish on the AI wins IMO gold medal market. The key here is you have do really read the rules. I’m confident a private AI system would win, but the market resolution rules require the AI to be open source and meet specific IMO Grand Challenge requirements. It is very easy to imagine Google or OpenAI models winning the prize, but very difficult to imagine them making the model open source because the development would require tens of millions of dollars. Reading the rules with meticulous detail is always one of my favorite forecasting strategies.

Any other "nothing ever happens" takes?

Like with Iran, I think the Venezuela situation with Trump and Maduro will probably just fade from the news. Unlike Iran where there was a specific nuclear threshold and Israeli pressure, there's not some countdown clock we need to avert with Venezuela. It seems more like intimidation and gunboat diplomacy rather than something that will escalate.

Same with Ukraine. Doomberg predicted a sudden collapse in Ukraine, and I think it's just much easier to hold positions than capture new ground. There's a strong defense advantage, and the European Union will probably continue resourcing Ukraine. You'd need special evidence to suggest Russia is poised for a breakthrough, and we don't really have that.

Any final words of wisdom?

I personally love mopping up the 90% that ought to be 95%. It's slow and steady wins the race, the anti-Doomberg approach. I'll enter at 88% or 90% because I think they should be 95.

You do have the burden of being right 19 out of 20 times. But there's just a systematic bias towards the thrill of the 10-to-1 underdogs, and I love fading that.

It's the "nothing ever happens" philosophy applied systematically. Most provocative predictions don't come true, even if they make for exciting headlines.

That’s it! Read the original on The Oracle and check out their other interviews!

This is a guest post written by Erich Grunewald, a Researcher at the Institute for AI Policy and Strategy, on the Compute Policy team. It is adapted from a section of a report by Erich Grunewald and Christopher Phenicie, and introduces the core concepts and background information needed to understand the AI chip-making process.

~

Over the past decade, advances in AI have been driven by the use of more and more sophisticated AI hardware. Many modern AI systems, like chatbots such as OpenAI’s ChatGPT and Google’s Gemini, are based on deep learning models trained and deployed using thousands of specialized AI chips housed in data centers. An analysis by Epoch AI suggests that an estimated 40% of the growth in training compute has been due to using larger numbers of chips, with the remainder being split between using higher-performance AI chips (~27%) and training models for longer durations (~33%).

Share

As a result of these trends, there are now major efforts underway to produce unprecedented quantities of AI chips. As an example, while OpenAI’s GPT-3 model released in 2020 was trained with an estimated 10,000 NVIDIA V100 chips, its GPT-4 model released in 2023 was reportedly trained with 25,000 NVIDIA A100s, and it is planning even larger supercomputers, e.g., one in collaboration with Oracle using 400,000 of NVIDIA’s new GB200 chips. With each generation, these chips get better and pricier – according to one study, the computational performance of AI chips doubles every 28 months. NVIDIA, which dominates the AI chip market with an estimated 80 to 95 percent share, shipped an estimated 1.5 to 2 million NVIDIA H100s in 2024 – the H100 being its most advanced chip that year – a threefold increase over 2023.1

But what is an AI chip? A chip (“integrated circuit”) is a group of electronic circuits laid out on a piece of semiconductor material. AI chips are integrated circuits specialized for AI training and/or deployment (“inference”) workloads. AI chips come in different types. Some, like Google’s Tensor Processing Units (TPUs), are application-specific integrated circuits (ASICs) – chips customized for highly specific workloads such as tensor2 operations used in deep learning training/inference. Other AI chips, like those made by NVIDIA, are graphics processing units (GPUs) – chips designed to more generally execute mathematical operations in a highly parallel manner.3

At a high level, AI chips are made in three steps: design, fabrication, and packaging. After the chips are packaged, they are typically assembled into devices (“AI accelerators”), which in turn are integrated into servers that can be mounted in data center racks.4 The semiconductor design and manufacturing process is not only highly demanding and sophisticated but also fascinating. The remainder of this blog post provides an overview of that process.

Left, an NVIDIA H100 printed circuit board, without casing, with the GH100 GPU (outlined in red) and six surrounding high-bandwidth memory (HBM) chips (outlined in blue) in the center. (Note that, in practice, the H100’s SXM5 and PCIe Gen 5 form factors ship with five HBM stacks enabled, while the H100 NVL ships with six stacks.) Middle, an NVIDIA A100 accelerator with casing. Right, Jensen Huang, president and CEO of NVIDIA, signing the first delivered NVIDIA DGX server.

Design

Chip designers specify high-level requirements, turn those into a detailed abstract model of the circuit logic, and use electronic design automation (EDA) software to translate that model into a blueprint of the physical circuit. The high-level requirements are typically based on earlier research carried out to find novel ways of improving performance. The design process involves defining the structure (“microarchitecture”) of the circuit’s logic components, designing each component or licensing a ready-made component (“IP core”) from a vendor, and specifying the connections between components. It is an iterative process. A chip’s specifications can be revised to balance cost and performance trade-offs. The design is also tested and validated to help ensure it will behave as expected once fabricated, for example, by simulating the design, running checks on code to scan it for errors, and making formal proofs about parts of the circuit logic.

An AI chip is designed for a particular process node. The fabrication plant (“fab”) will provide the chip maker with a process design kit (PDK), which provides the necessary information to design and validate chips for one of the fab’s processes. The result of the design process is design files containing a detailed description of the chip’s physical layout, which is shared with the fab. Chip design is a difficult and laborious process. Chip designers like NVIDIA and AMD have tens of thousands of employees, and the design process for cutting-edge AI chips lasts for a year or longer.5

Fabrication

Material preparation

Chips are fabricated on thin, circular slices of semiconductor6 called wafers, typically 300 mm (12 inches) in diameter.7 The most common semiconductor used is silicon. Modern silicon wafers, which are grown in furnaces and then sliced, polished, cleaned, doped, and inspected, need to be extremely pure and smooth in order to avoid defects in chip fabrication.

(Avoiding defects is a priority because improved yield – the percentage of chips that are defect-free – directly translates to increased profitability. Defects – which can be caused by contaminants, environmental conditions, equipment issues, and more – are a problem in semiconductor manufacturing because it involves hundreds of sequential steps and extreme precision and environmental control.)

Prior to fabrication, photomasks are produced from the description of the physical layout provided by the chip designer. A photomask holds a pattern that will be imprinted onto the wafers, like a custom stencil.8 Photomasks are created using direct-write mask writers, involving multiple stages of optical or electron beam (e-beam) lithography, etching, and inspection to ensure that there are no defects.

In addition to wafers and photomasks, semiconductor manufacturing also requires various specialized chemicals and gases.

Wafer fabrication

Supplied with silicon wafers and photomasks, the fab is ready to manufacture the chips. The wafer fabrication process involves imprinting patterns onto the silicon wafer in layers of different materials, including insulators (dielectrics), conductors (metals), and semiconductors. It is a highly difficult and complex process, as it demands extreme precision, involves hundreds of individual steps, and must, at the same time, achieve high economic efficiency. What follows is a simplified description of that process.

First, the transistors and other devices (resistors, capacitors, etc.) are formed by repeatedly depositing, patterning, and etching layers of materials. The wafer is always processed as a whole, allowing trillions of transistors to be created in a short amount of time. The reason why the wafer can be processed in one go is patterning techniques. To form patterns, a light-sensitive chemical (“photoresist”) is deposited on top of a material and, using a photolithography machine, exposed to ultraviolet light passing through the photomask, then developed, leaving a pattern of gaps in the photoresist. (This is akin to how light reacts on photographic film in a camera.) The developed photoresist now acts as a mask, allowing the material underneath to be partially etched away, after which the photoresist is washed away, leaving behind only the patterned material. These steps are repeated until the devices are fully formed.

In this patterning process, it is also necessary to modify the electrical properties of parts of the semiconductor; to do this, specialized substances (“dopants”) are introduced using a technique called ion implantation. When excess material needs to be removed from the wafer’s surface, it is polished in a chemical slurry using a technique called chemical mechanical planarization (CMP). To remove contaminants and residue, the wafer is periodically cleaned using chemicals, plasmas, and a variety of physical techniques like sound waves or jets.

After the devices (transistors, etc.) have been formed, layers of metals and insulators are deposited and etched with chemicals to form connections between the devices. The steps used to create the metal layers resemble those used to form the devices underneath, as they also involve photolithography, deposition, etching, and planarization, and are also repeated with different patterns over multiple layers. Finally, a protective layer is deposited on top of the metal layers, with gaps etched in it for connections to external circuitry.

The entire process is almost fully automated. Wafer handler tools transport and align wafers between and during process steps, and process control tools regularly monitor the wafers for defects, as detecting defects early on is critical to keeping costs down. However, human engineers and technicians still provide oversight, maintenance, and efficiency improvements.

Assembly, test, and packaging

After fabrication, the patterned wafers are typically sent to a different facility for assembly, test, and packaging (ATP).9 First, the silicon wafer is cut into individual, rectangular chips (“dies”). Those dies are then tested, and good dies are encased in a protective covering. Once packaged, the chips undergo final functional testing and “burn-in”, where they are tested at extreme temperatures and voltages. ATP is sometimes collectively referred to as “backend” manufacturing, with wafer fabrication being “frontend” manufacturing. Although the packaging process is less complex than the fabrication process, it is not trivial.

While the main purpose of packaging has traditionally been to protect the chip from corrosion and other damage and to add connectors for other chips, advanced packaging techniques are now an increasingly important driver of semiconductor progress. “Advanced packaging” refers to a group of techniques for arranging multiple fabricated chips and “chiplets” together in a single package. (A “chiplet” is a small chip designed to be combined with others in a package.) Two relevant variants of advanced packaging are 2.5D, where multiple chips are arranged on an “interposer” chip that facilitates communication, and 3D, where chips are stacked on top of each other. Such techniques include:

• Through-silicon vias (TSVs), used to connect vertically stacked dies. TSV is used to create high-bandwidth memory, and therefore necessary for modern AI accelerators.

• Chip-on-Wafer-on-Substrate (CoWoS), used to stack chiplets on wafers. TSMC uses CoWoS to make NVIDIA H100 chips by packaging the GPU alongside six high-bandwidth memory chiplets.

• Embedded multi-die interconnect bridge (EMIB), used for die-to-die interconnect. This accomplishes similar goals to 2.5D techniques like CoWoS, but without requiring, or benefitting from, a large interposer, which is costly and consumes space.

• Hybrid bonding, used to stack chips or wafers on wafers using direct copper-to-copper interconnects.

Combining multiple smaller chips in a single package can offer higher yield, reduce the maximum distance of connections, and allow for the heterogeneous fabrication of chiplets using different fabrication processes (e.g., using advanced, expensive nodes for high-value chiplets and cheaper nodes for other parts).

Conclusion

Making advanced AI chips involves one of the most complex manufacturing processes that has ever existed. Each step in the AI-chip-making process – from design to fabrication and ATP – impacts the performance, cost, and quantity of AI chips made. Understanding how AI chips are made can help policymakers, industry, and civil society develop good industrial and compute policy while also anticipating future AI developments.

Moreover, because they are so closely connected to AI progress, AI chips have emerged as a focus point of geopolitical competition, for example, having been the target of wide-ranging US-led export controls against China. Our Introduction to AI Chip Making in China report contains more detail on AI chip making, especially as it relates to Chinese efforts to develop better domestic AI chips.

This is a guest post written by Erich Grunewald, a Researcher at the Institute for AI Policy and Strategy, on the Compute Policy team.

Computational power (“compute”) is a strategic resource in the way that oil and steel production capacity were in the past. Like oil, and like steel production capacity, compute is scarce, controllable, concentrated, and highly economically and militarily useful. Just as oil and steel were and remain strategic resources to some extent, compute is now also a strategic resource of very high importance.

This view on compute may be controversial. Some think of compute as merely one among many inputs into AI progress, similar in importance to talent or data. Others think of compute as an abundant global commodity that is futile to control, especially given constant innovations making AI systems more compute-efficient. Although those views have some merit, I think the view of compute as a strategic resource is more accurate and useful.

Share

Like oil, compute is highly valuable both economically and militarily. It is the most important input into the development of the most important emerging technology, in the sense that it is a major bottleneck for AI, and is strongly correlated with progress in the field. It is by far the largest expenditure for AI companies – according to The Information, OpenAI spends nearly ten times as much on compute as on employee salaries. And these companies are spending ever more to build ever larger supercomputers for developing and deploying ever better AI systems.

AI companies are spending so much on compute because compute offers them a significant competitive advantage. They need it not only to pre-train models, but also for research experiments, synthetic data generation, fine-tuning, reinforcement learning, and, importantly, broad deployment to their swiftly growing user bases, all of which benefit from scale. When it comes to AI model training in particular, there is an observed relationship between compute use and benchmark performance, as can be seen on diverse reasoning questions.

The same is true of high-school-level math competition questions among other tasks. OpenAI’s reasoning models show consistent performance improvements on mathematics competition questions as training compute increases:

No one knows for sure whether this correlation between compute and capabilities will continue, but it has now held for over a decade and several orders of magnitude of training compute use.

Like oil, compute is concentrated. The United States has an estimated 75% of the world’s AI supercomputing capacity, with China in second place with 15%. These international differences are largely a product of the enormous capital requirements needed to build cutting-edge AI data centers, with leading supercomputers now costing billions of dollars. They are also the result of American export controls against China, which impose substantial costs on Chinese companies aiming to produce or acquire compute.

The AI chip supply chain is itself highly concentrated. First, NVIDIA has an estimated 80-95% of the AI chip market. Its main competitor in AI chip design is Google, which does not sell its AI chips, opting instead to rent them out via the cloud. But AI chips designed by both NVIDIA and Google are fabricated by TSMC, which carries out an estimated 90% of leading-edge logic chip fabrication.

TSMC, in turn, is dependent on extreme ultraviolet (EUV) lithography machines solely produced by the Dutch company ASML, and several other tools and materials supplied by only a few firms. These technologies are so advanced and so capital-intensive that it is difficult for competitors to challenge these market leaders in any significant way.

Like oil, compute can be meaningfully controlled. During World War II, oil provided the Allies with major advantages against Germany and, following the United States’ 1941 embargo, against Japan. Japan and Germany were limited in how much oil they could produce or obtain. The production of cutting-edge AI chips today is even more concentrated than the production of oil.

An even stronger indication that compute can, to some degree, be controlled is that it has been, and is being, used strategically in this way. In particular, American export controls on semiconductor manufacturing tooling and AI chips show that compute access can be meaningfully, albeit not perfectly, controlled.

Like oil, compute is scarce. That may seem like a surprising statement given how much more computing power exists today than even a few years ago, and given that algorithmic innovations constantly allow us to do more with the same amount of compute. But demand for cutting-edge AI chips far outstrips supply, as a result of which a flagship NVIDIA GPU now costs three times more than one did five years ago (adjusted for inflation). Not coincidentally, NVIDIA is now the most valuable company ever to have existed.

In late 2024, DeepSeek-V3 famously matched GPT-4 (2023) in performance while costing roughly one-tenth as much to train, due to using more compute-efficient algorithms and better NVIDIA chips. But this impressive achievement was not evidence of compute being any less important. When algorithms become more efficient, researchers do not only train the same models with half the compute – they also train larger models with more compute, advancing the state of the art, and offer models more cheaply to more users. When hardware becomes more cost-effective, companies buy more of it.

That is what we have seen for the past ten years: algorithms have steadily gotten more compute-efficient, hardware has steadily gotten more cost-effective, and AI training has steadily gotten more compute-intensive. In other words, compute has been important on every margin discovered so far, and no company working towards advanced AI or artificial general intelligence (AGI) has ever had “enough” of it.

Some argue that access to more compute is not only not very useful, but can even be a disadvantage. For example, after the production release of DeepSeek R1 in January 2025, John Villasenor argued for Brookings Institution:

Scarcity fosters innovation. As a direct result of U.S. controls on advanced chips, companies in China are creating new AI training approaches that use computing power very efficiently. When, as will inevitably occur, China also develops the ability to produce its own leading-edge advanced computing chips, it will have a powerful combination of both computing capacity and efficient algorithms for AI training.

It is true that compute restrictions have driven some Chinese AI companies to focus their efforts on compute efficiency. But that does not mean it is advantageous overall to have less compute. If that were really the case, we would see countries putting import tariffs on AI chips, companies selling off their compute stock, and algorithmic innovations increasingly coming from compute-poor academia. In fact, the opposite is happening – countries are cutting deals to import AI chips, companies are investing enormously in AI data centers, and academia is increasingly irrelevant.

Over the coming decade or decades, compute may even become directly convertible into many forms of cognitive labor, as AI companies are working to create increasingly capable and reliable autonomous AI agents. If that is anything like the future we are headed for, compute will not lose its importance. If so, on the contrary, as one researcher put it, “Compute is the primary means of production of the future, and owning more will always be good.”

These roles are all closed now and this post is now just an archive for historical purposes. You can see the latest (as of Oct 6) here.

~

~

Have you been reading about AI lately and wondering what’s up and how you can help? Readers of this blog know that getting AI right is the greatest challenge of our time, and we need a wide variety of people to pitch in.

In this article I will highlight some AI policy roles that I personally endorse and think highly of1. If you read this blog, you’re likely the target audience for many of these roles. If you’re career searching right now, or just career browsing, consider applying!

~

[DC] Launch Your Career Shaping America's AI and Biotech Policy

The Horizon Fellowship offers an unparalleled pathway into US emerging technology policy with a 100% placement rate at federal agencies, congressional offices, and think tanks.

This fully-funded program provides $113,000/year (or $75,000 for junior fellows) to work on critical AI, biotechnology, and emerging tech challenges in Washington DC either in Congress, the Executive Branch, or a DC-based think tank in a 1-2 year placement.

Fellows receive intensive policy training, mentorship from senior leaders, and join a tight-knit community of public service-oriented experts — with many alumni converting to permanent roles at NSC, OSTP, Commerce, and other key institutions.

Applications close August 28 for the 2026 cohort starting with training in 2026 January. Must already be eligible to work in the US (though doesn’t require US citizenship) and be willing to relocate to DC.

Become a fellow

~

[London] The UK government continues to build their AI security work

The AI Security Institute (AISI), the world's largest government team dedicated to AI safety and security, is continuing to hire across research, engineering, and policy. Born from the 2023 Bletchley Park AI Safety Summit, AISI is the UK government's official answer to ensuring humanity doesn't lose control of frontier AI.

These aren’t typical government jobs – AISI operates like a tech startup with the backing and influence of a major government. These roles involve working alongside alumni from Anthropic, DeepMind, OpenAI, Google, and top universities on genuinely existential challenges: preventing AI from enabling bioweapons development, stopping catastrophic cyber attacks, and ensuring we maintain control over increasingly powerful AI systems.

Current roles include:

• Senior Testing Product & Strategy Lead: Own the long-term vision for AISI's frontier AI testing program. This role requires 5+ years in tech/AI, deep LLM knowledge, and a proven product leadership track record. Rolling deadline for applications.

• Software Engineering Manager - Core Technology: Lead small teams building critical infrastructure (Inspect framework, model hosting). This role requires engineering management experience, Python expertise, experience with research teams. Rolling deadline for applications.

• Research Manager - Control Team: Lead ~4 researchers developing ‘AI Control’ science to ensure potentially misaligned systems can still be safely used. This role requires experience in research/engineering management experience and a strong understanding of the ‘AI Control’ field. Rolling deadline for applications.

• Research Assistant Residency - Human Influence (6-month fixed term, £65k): Support research on AI persuasion/manipulation/deception. This role is ideal for MSc grads/early PhD students with a computational social science or psychology background. This role closes Sept 8.

• Private Secretary (2 roles, £33-40k): Executive support to senior leadership - one for Strategy Portfolio, one for Technical Research (requires SC clearance), involves managing calendars, coordinating high-level meetings, and driving strategic initiatives. This role closes Sept 3.

• Transformative AI Engagement Officer (£44-49k): Develop cross-government AGI preparedness strategy, brief ministers and officials on transformative AI risks/opportunities. This role requires excellent communication and political awareness. This role closes Sept 2.

• Research Scientist - Strategic Awareness: Conduct deep dives on AI trajectories and impacts for government decision-makers. This role requires expertise in AI forecasting, scaling laws, or specific impact areas (e.g., labor markets, loss of control)

• Safeguards Technical Governance Researcher: Bridge technical safeguards research with policy/governance and develop actionable frameworks for managing fine-tuning risks, differential access, and evaluation outputs.

• Research Scientist - Safeguards: Develop attacks/defenses for LLMs and adversarial testing of frontier systems. This role requires hands-on LLM research experience and ML publications.

• Research Scientist - ChemBio: Lead work on AI-enabled chemical/biological threats. This role requires bio/chem science or ML background plus biosecurity expertise and you must be eligible for ‘Developed Vetting’ (largely only for UK nationals).

• Software Engineer - Core Technology: Build tools/infrastructure for AI safety research. This role requires Python expertise and production code experience.

Where not otherwise specified, roles usually involve a salary range of £65K-£145K.

Roles are based in London (Whitehall), hybrid working 40-60% office time. Roles are generally open to UK nationals, EU/EEA with settled status, and those with commonwealth work rights. Some roles are more restrictive and limited to UK nationals only.

Explore careers at AISI

~

[DC] Shape US Tech Policy on China Competition in the House CCP Committee

The House Select Committee on the CCP is hiring a Professional Staff Member (Technology Staffer) in Washington DC. This role is about as close as it gets to being on the frontlines of US tech–national security policymaking. The committee would be on the Minority side (Democrat). They are looking for someone who can dig deep into export controls, dual-use technologies, and the strategic implications of AI, quantum, biotech, and semiconductors—and then translate that expertise into actionable recommendations for Congress.

You’ll be expected to research, draft memos and proposals, prep hearings, and engage stakeholders across government, industry, think tanks, and academia. The job demands subject-matter expertise in at least one emerging tech area, a strong grasp of export control regimes (EAR, ITAR, Wassenaar, etc.), and 3–5+ years of experience in related national security or innovation policy. Hill, executive branch, or tech-policy experience is strongly preferred; Mandarin is a plus. Applications are reviewed on a rolling basis.

Explore House CCP

~

[DC] Shape Science and Energy Policy from the House Science, Space, and Technology Committee

Ranking Member Zoe Lofgren (D-CA) of the House Science, Space, and Technology Committee is hiring a Professional Staff Member to drive oversight of the Department of Energy and other science/tech agencies. This role puts you at the center of congressional work on energy innovation and environmental policy — from drafting legislation to briefing Members, shaping hearings, and engaging with stakeholders.

Applicants need to be a US citizen with substantive prior experience in DOE research/technology or energy/environmental policy, plus sharp writing, communication, and organizational skills. Capitol Hill or government/industry background is preferred. An advanced degree or technical expertise is a bonus.

Materials (cover letter + resume) are due by September 5 to SciResumes@mail.house.gov.

~

[Bay Area/Remote] Run operations at a fast growing AI research institute, FAR.AI

FAR.AI is a fast growing non-profit AI research institute based in Berkeley (with remote employees) that does AI safety research (everything from jailbreaking models to researching alignment) and also leads conferences and workshops. They are hiring for a few operational roles.

They are hiring for a Senior Project Manager on Events to lead some of their conferences and workshop work. The job is full-time (remote US or in-person Berkeley, CA) with $115K–$150K salary plus travel and equipment covered. This role involves owning strategy and execution across different event types (from multi-day international gatherings to technical workshops) and different modalities (working across research, comms, and ops teams while coordinating speakers, sponsors, and venues). Expect one fully covered trip per month, occasional evening/weekend work, and the chance to directly support the researchers and policymakers shaping AI’s future.

• This role will be performed best by someone with at least 5+ years of event project leadership (complex, large-scale, or cross-functional), mastery of logistics/vendor management, strong project management tool fluency (Asana, Airtable, etc.), and the ability to operate independently under pressure. A global mindset and sharp communication skills are musts. Prior exposure to technical, policy, or mission-driven events is a plus.

FAR.AI is also hiring for a People Operations Generalist to help scale a thoughtful, people-first culture as the team doubles in size over the next 18 months.

• The role is full-time, based in Berkeley (hybrid possible), with $85K–$110K salary plus travel, equipment, and catered meals. In this role you’d support recruiting, onboarding, compliance, and engagement—ensuring a seamless employee experience from first interview to team retreats.

• They want someone with 3–5 years of prior experience in HR/people ops, sharp organizational and communication skills, and comfort working independently in a dynamic environment. Extra points for prior experience in distributed teams, startups, or mission-driven nonprofits.

If those roles aren’t cool enough for you, there’s also the chance to lead it all. FAR.AI also seeks a Chief Operating Officer to lead all the operations during this period of rapid growth. Reporting directly to the CEO and incoming President, the COO will manage Finance, People, and Business Operations, scaling the backbone that supports cutting-edge research, major field-building events, and global collaborations.

• The COO role is a full-time role based in Berkeley, CA (preferred, hybrid/remote possible) with $175K–$250K compensation plus visa sponsorship, travel, equipment, and catered meals. FAR.AI wants a proven operator with 7+ years of senior leadership, strong financial acumen (prior experience with multi-million dollar budgets, compliance, and risk), and a track record of scaling systems and teams in high-growth, mission-driven environments. Experience in nonprofits, think tanks, or R&D-heavy orgs is a plus, but an AI background isn’t required.

Explore Ops Careers at FAR.AI

~

[London/Remote] Magnify the impact of AI Governance writing at GovAI

Do you like meticulous analysis and the chance to judge others? The Centre for the Governance of AI (GovAI) is hiring a Senior Research Editor that combines both of these passions to supercharge the GovAI publications pipeline.

Low editing capacity is now a bottleneck at GovAI, so this is a good opportunity to be a force multiplier — help turn draft research into polished outputs that reach policymakers, researchers, and industry leaders making critical decisions about AI. This means both hands-on editing and building the systems that let GovAI scale from ~15 to 30+ researchers without losing quality. For strong candidates, the role can also include strategic input into research priorities.

Compensation is £76k–£114k (~$99k–$148k) plus benefits, with London location preferred but remote (esp. US) possible, and visa sponsorship available. Strong editing skills are essential; AI governance expertise is a plus but not mandatory. Apply by Aug 31.

Edit at GovAI

~

[DC] Kick off your national security career with CNAS’s NextGen Fellowship

The Center for a New American Security (CNAS) is now accepting applications for the 2026 Shawn Brimley Next Generation National Security Leaders Fellowship. This is a year-long, part-time program designed to support emerging national security professionals who are (a) a US citizen (b) ages 27–35 with (c) at least four years of prior professional experience in researching or executing U.S. national security policy.

Fellows will get opportunities for leadership development, networking, and will cap the experience with a week-long international study tour. Fellows also participate in monthly dinners with senior figures — past speakers have included Madeleine Albright, Stanley McChrystal, Jeh Johnson, and Mike Gallagher. NextGen sessions are usually held 5-6 times per fellowship year, in the evenings.

The program is free to join (participants cover their own travel costs) and is based primarily at the CNAS headquarters in Washington DC. Applications close Sep 21.

Explore NextGen

~

[Remote-US] Turn Research Into a Startup That Shapes AI’s Future with 5050

5050 (run by Fifty Years) is a free, 13-week program for scientists and engineers who want to build startups tackling civilization-scale problems in AI safety, alignment, and beyond. Applications are open until September 14, with the next cohort running September–December in San Francisco, Boston, and remotely.

The entire cohort will go to SF (travel and lodging paid for) for the kickoff weekend (early Sept) and a 3-day off-the-grid experience (Oct 24-26). After the kickoff, every week program members will join office hours with 50Y Partners and afternoon workshops focused on entrepreneurship skills. Workshops are held in-person in SF and Boston, and remotely for participants across the rest of the US.

The program offers mentorship from leaders like Wojciech Zaremba (OpenAI co-founder), Ross Girshick (deep learning pioneer), and Jaan Tallinn (leading entrepreneur and invester). Alumni so far have launched 78 companies with a 95% seed raise success rate, from cancer immunotherapies to decarbonizing shipping. If selected, you’d join a tight cohort of ambitious peers, learn how to turn research into a company, avoid costly mistakes, and explore whether entrepreneurship is right for you.

There are no fees, no equity agreements, and no other strings attached — 5050 is designed to accelerate founders building startups that make AI safe, interpretable, and aligned with human flourishing.

Learn more about 5050

~

[Brussels/EU] Lead EU and Global AI Governance at the Ada Lovelace Institute

The Ada Lovelace Institute is hiring a Head of EU and Global AI Governance to lead its Brussels-based policy and research agenda. This is a senior role shaping how AI and data are governed across Europe and globally, with direct influence in forums like the European Commission, Parliament, Council, UNESCO, and OECD. The role pays from €75,000/year (approx. €6,250/month pre-tax, plus bonus and allowances), is offered on a 2-year contract, and comes with flexible hours, strong benefits, and a hybrid Brussels/London setup.

Ada is looking for a strategic, well-connected policy leader with deep expertise in AI/data governance, experience in legislative or regulatory analysis, and comfort navigating complex EU and international landscapes. Strong research and communication skills, stakeholder engagement, and staff management are required. The deadline to apply is is 9:30am BST on September 8.

Lead EU AI policy

~

[London] Work on UK policy, also at the Ada Lovelace Institute

The Ada Lovelace Institute is also hiring a Policy Researcher to help shape how AI and data governance evolves in the UK and beyond. Salary starts at £41,767, with a 2-year fixed-term contract, based in London with flexible hybrid options. Applications close 9:30am BST on Sep 15. This is a strong fit for an early-career researcher who wants to make a tangible impact on emerging AI policy.

You’ll work closely with Ada’s Public Policy Lead on projects in law, policy, and governance. They’re looking for someone with strong research and policy chops, ideally with experience in AI/data governance, law, or regulation. Familiarity with Ada’s methods—policy/legal analysis, expert convenings, public deliberation, surveys—is a plus, but they also welcome fresh perspectives from fields like computer science, data science, or futures thinking.

Explore UK policy

~

[London] Help Demonstrate AI Risks at Apollo

Apollo Research is hiring a 6-month, full-time Evals Demonstration Engineer (London-based, £7,500/month) to design and deliver demonstrations that translate technical AI evaluation findings into compelling formats for policymakers and other non-technical decision-makers. Applications are open until September 10, reviewed on a rolling basis (early submission encouraged). This role starts a contract role but strong performance could lead to a permanent role.

The role requires technical fluency (Python, Inspect framework, ability to run/modify evals) and exceptional communication skills to craft demos, visualizations, and live presentations that resonate with policymakers. Prior experience presenting to government or think tank audiences, plus creativity in choosing the right medium (interactive, video, report), are core. You’ll collaborate directly with Apollo’s evals and governance leads (Marius Hobbhahn, Charlotte Stix) and produce outputs like live demos, policy-oriented visuals, and blog posts. The role is in-person in London, and UK work eligibility is prioritized (though exceptional candidates elsewhere are invited to apply).

Demonstrate AI risks

~

[Remote] The Future of Life Institute seeks an editor and investigator

If you want to use storytelling craft to shape the world’s response to transformative technology, the Future of Life Institute is seeking a full-time, remote Editor to lead the creation of written, broadcast, and video content that raises awareness of both the risks and opportunities of advanced AI. Reporting directly to the Director of Communications, the Editor will shape everything from op-eds and blog posts to short films, PSAs, and large-scale media campaigns.

• Compensation ranges from $90,000–$190,000 depending on experience and geography. Applications are due September 8. Candidates should bring proven editing/writing experience in journalism, screenwriting, or related fields, strong knowledge of AI safety issues (including alignment, misuse, AGI risks), and the ability to communicate complex ideas in a clear, evocative way. Bonus points for top-tier publication credits, ghostwriting, or campaign management experience.
  

FLI is also is hiring a full-time, remote AI Safety Investigator (salary $90k–$150k + benefits) to document and analyze safety practices at the biggest AI companies, explain incidents to the public, and lead FLI’s flagship AI Safety Index. The role mixes research, field investigation, and communications —building networks inside corporations, rapidly analyzing incidents, and translating findings into clear metrics and visualizations that shape public and policy debates.

• They want someone self-directed and sharp at following leads, ideally with a background in journalism, research, and/or AI safety, plus the ability to communicate technical concepts clearly. Occasional Bay Area travel is required. Applications are due September 4.

Explore careers at FLI

~

[SF] Help Launch AI Safety Projects at the Center for AI Safety

The Center for AI Safety (CAIS) is hiring a Special Projects Associate and a Special Projects Manager in San Francisco to drive new initiatives at the frontiers of AI safety. In these roles, you’d work directly with CAIS leadership to scout opportunities, design project plans, manage budgets and timelines, and turn ambitious ideas into operational reality. With AI safety communications hitting millions and public curiosity accelerating, CAIS is looking to scale fast — and this is a chance to help shape that surge into impactful, mission-aligned projects.

Both positions are full-time, on-site and offer strong compensation ($100K–140K for Associate, $120K–150K for Manager). Requirements are a bachelor’s degree (advanced degree optional), 1–2 for Associate / 2–4 years’ experience for Manager in startups/ops/consulting/project management, proven ability to learn new domains quickly, and genuine interest in AI safety.

~

CAIS is also hiring a Finance Manager in San Francisco to take full ownership of financial operations across both its nonprofit arms — a 501c3 and a 501c4. This role would oversee everything from audits, tax filings, and nonprofit governance policies to payroll, expense systems, and donor reporting —working directly with leadership, legal counsel, and the operations team to keep the organization financially sound and compliant.

• The role offers $110K–150K, hybrid flexibility, and strong benefits. Candidates should bring 4+ years in finance or nonprofit accounting, with experience managing audits, budgets, and compliance processes; familiarity with Xero, Ramp, Rippling, Stripe, or similar platforms is a plus. Experience across both 501c3 and 501c4 entities is preferred but not required.

~

Lastly, CAIS is looking for a Director of Public Engagement in San Francisco to shape how the world understands the risks of advanced AI. You’ll lead multi-channel campaigns, craft narratives that resonate with diverse audiences, and serve as a visible spokesperson for one of the most influential organizations in the field.

CAIS is looking for someone with proven chops in campaign strategy, media production, and public communication—ideally someone who’s run large-scale awareness efforts before. The position offers $140k–$170k/year and strong benefits.

All applications are on a rolling basis.

Click for Careers at CAIS

~

[Berkeley CA] Build AI Demos with CivAI

CivAI, a nonprofit based in Berkeley, is hiring a Senior Software Engineer to create interactive demos that bring AI’s capabilities and risks to life. Instead of publishing papers, CivAI builds tangible products/demo that policymakers, journalists, and the public can use first-hand. This helps people intuitively grasp what advanced AI can and can’t do. CivAI’s work has already reached 60+ government offices and been featured on ABC, NPR, CNN, and WaPo.

This is a $150k–$200k full-time, on-site role for a generalist engineer. The team values both hacker spirit and strong UI/UX instincts. 5+ years of experience is preferred (though strong candidates with 3+ years will be considered). Applications are on a rolling basis.

Check out CivAI

~

[Remote-US] Run Projects at the Edge of AI and Biosecurity with RAND’s Meselson Center

RAND’s new Meselson Center is hiring an AI/Bio Research Project Manager (2-year term, $75k–$156k, hybrid/remote eligible) to drive high-impact work at the intersection of frontier AI security and biological risk reduction. As both AI and biotech risks accelerate and increasingly overlap, RAND wants a project manager who can keep ambitious research and policy projects moving fast and effectively.

This role involves coordinating interdisciplinary teams, manage research deliverables, support recruiting rounds, oversee budgets/contracts, and help run workshops that convene top talent. Requirements involve at least 4 years of project management/operations experience (a Master’s or PMP can substitute), strong organizational/communication skills, and the ability to juggle multiple high-stakes projects. Preferred locations are from DC, Santa Monica, Boston, or Pittsburgh, but remote is possible. Clearance eligibility is a plus but not required. Applications are open now on a rolling basis.

Look at this role

~

[London] Do operations at 80,000 Hours

80,000 Hours is a non-profit that’s helped thousands of people shift their careers toward the world’s most pressing problems and is now focused on helping people get careers on AI risk. Theirgrowing team (now ~35 staff) needs more operational backbone: from ops generalists to specialists in events, recruiting, people ops, office management, executive support, and video production. They’re also seeking an IT security and data privacy specialist to strengthen internal systems.

This is a bit meta, but perhaps the best way to help AGI is to do operations for the organization that helps coach people into AGI-related careers. Leverage! They’re looking for people who are organized, detail-oriented, clear communicators, flexible, and motivated by 80,000 Hours’ mission.

Compensation is £41k–£75k for generalist roles and £55k–£80k for the IT security role, with strong benefits. Roles are full-time, London-based by default (visa sponsorship possible), but some remote arrangements are considered. The deadline to apply is September 1.

Look at 80K ops roles

~

[Remote/Global] Run Operations That Power AI Safety Projects

Rethink Priorities is hiring a Special Projects Associate/Coordinator for a 5-month parental leave cover in a full-time remote setting to keep high-impact initiatives running smoothly. This is a $70–90k/year equivalent role (prorated) starting late September/early October 2025, with benefits and possible extension. You’ll work across time zones (UK/California overlap required) and serve as the operations lead for 2–3 projects advancing safe and aligned AI, from budgeting and compliance to HR support, contracts, and project management.

The job is for someone who’s comfortable with generalist ops work (finance, HR, compliance, project management), proactive in solving problems, and eager to support initiatives like launching new AI safety orgs, running fellowships, and incubating policy talent. Bonus points if you’ve touched US nonprofit finance/compliance. Applications are open now and judged on a rolling basis — ideal start date is Sept 29 (latest Oct 6, 2025).

Explore RP Special Projects

~

[Remote] Train on AGI Strategy with BlueDot Impact’s Course

BlueDot Impact is offering a free AGI Strategy Course designed to prepare people to shape the future of artificial general intelligence. BlueDot, a non-profit spun out of Cambridge that has already trained 5,000+ AI safety professionals, built this program to give participants the tools, frameworks, and community to engage with the highest-stakes policy, technical, and governance challenges of our time. Alumni have gone on to roles at OpenAI, Anthropic, NATO, the UN, OECD, and national AI directorates.

The course is virtual, free, and highly structured, with small peer groups, expert facilitators, and flexible pacing: either a 6-day intensive (5 hrs/day) or a 6-week part-time option (5 hrs/week). Every month a new round starts, with rolling applications. Participants will spend 2–3 hours preparing for each live discussion and then dive into guided conversations with other motivated peers. If you want to pivot into AI safety or policy—or level up from a related background—this is one of the few high-signal entry points with a strong track record of career impact.

Join the next cohort

~

The go-to for navigating emerging tech policy

The US government and allied institutions urgently need digital and scientific talent to keep pace with transformative tech. But the entry points are fragmented and opaque.

Emergingtechpolicy.org is a great one-stop guide for anyone who wants to break into public service careers at the intersection of technology and policy. Whether you’re a student mapping out your first steps, a technologist pivoting into policy, or a policy pro tackling new tech domains, it offers structured pathways, curated resources, and expert advice.

The content spans everything from career essentials (fit testing, resumes, networking) to policy institutions (Congress, think tanks, federal agencies, ARPAs, intelligence community). It also helps profiles real opportunities —internships, fellowships, full-time roles — and help demystify how to navigate the AI policy world, with a special focus on DC.

Explore Emerging Tech Policy

~