Pacing the Frontier

1300+ AI company employees are afraid of what they are building towards

As I understand it, the openly stated plan of the leading AI companies is to:

1. hire software engineers and AI researchers to build AIs that can automate software engineering and AI research

2. use automated AI researchers to go >100x faster at figuring out how to automate everything else.

3. End up with AI superintelligence that would be smarter than everyone at everything

This seems very much on track, and companies are spending tens of billions of dollars towards this goal. At some point, plausibly within 24 months or less, skilled engineers may stop having anything useful to add to AI research, similar to how chess grandmasters have nothing useful to add to AI-played chess games.

This is, to put it lightly, a very risky plan. No one knows what safeguards we need before we hand control over to the AIs. No one knows if the right safeguards will be ready in time. What happens if company CEOs are approaching AI superintelligence and the CEOs (or the government) determine that the safeguards are not good enough to keep catastrophic risks at acceptably low levels?

It would be nice if we could — for some temporary period of time — agree not to pass certain critical and unprecedented capability thresholds until we have better safeguards. In other words, it would be nice to pace the progress of the frontier.

This is why over 1300 AI company employees across OpenAI, Google, Meta, Anthropic, SSI, Hugging Face, xAI, Inherent, Nvidia, Microsoft, and other companies have signed a statement entitled “Pacing the Frontier” — including Thinking Machines Chief Scientist John Schulman; OpenAI Chief Scientist Jakub Pachocki; Anthropic Chief Scientist Jared Kaplan; Meta AI Chief Scientist Shengjia Zhao; Google DeepMind Chief Scientist Shane Legg¹; Ilya Sutskever, former OpenAI and current CEO of SSI; and Dario Amodei, CEO of Anthropic. This statement was also endorsed by OpenAI directly and Anthropic.²

The statement notes that “AI could help create a dramatically better future, but that outcome is not guaranteed”, that “[t]he world’s leading AI companies believe they could be close to automating AI research” and thus “there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.”

These employees then say that “industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight”, calling for an “effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

I take this statement as a call for help. The signatories don’t want to slow AI down now, but they don’t even have the ability to slow down if they wanted to. They are headed towards some really crazy places and they’re not sure this will be safe. Things are moving too quickly. The race dynamics are too fierce. There are no ways to be able to make credible commitments and coordinate.

If we take these signatories seriously, how do we accomplish this mission? How do we build the tools to pace the frontier?

What does it actually mean to pace the frontier?

What to do can be complex and depends a lot on your specific threat models, the preferences of the various stakeholders (companies, US government, China), and a bunch of other variables. But proposals to “pace the frontier” have a few key features in common. These common elements tend to include:

1. Identifying an agreement. A common agreement involves thinking about a particular threshold that we do not want to reach — for now, absent good safeguards. Examples include “recursive self-improvement”.

2. Operationalizing that agreement to make it more concrete or measurable. Examples include “we need to monitor data centers that have more than X H100-equivalents to ensure they don’t do Y” or “we need to ensure that no data centers are used to create a model above a certain size.”

3. Coordinating around your agreement, and verifying people are following it. How do we actually detect noncompliance or verify that everyone is following through? Examples include national technical means (e.g., activities of intelligence communities, open-source intelligence), inspections (of data centers, fabs, or developer facilities), and verification technologies (e.g., technology that automatically verifies that clusters are only performing inference and not performing training above a certain size).

Ok, but what would we agree to? There is still a lot of conceptual and macrostrategic work to be done here. But here are two of the most common proposals:

1. No “recursive self-improvement” or “superintelligence” until better safeguards are developed — or unless the risk of a project outside the agreement racing ahead becomes intolerable. The goal is to ensure that innovation in safety and security research can still occur, but there are certain danger thresholds that are not allowed to be passed until breakthroughs in safety/security work are achieved.

   
   One important variable to consider in this setup is the risk of a hidden or “dark” superintelligence project getting off the ground — a project that is outside the agreement that we can’t control. Simply put, you can die if you scale to superintelligence before you know how to control it. But you can also die if you’re too cautious and someone else develops superintelligence without being detected by your monitoring/verification.³

2. Spend at least X% of your compute on safety/security work. In this setup, companies/governments agree to spend X% of their compute on safety/security work. Suppose that under today’s conditions, race dynamics cause companies to spend 90%+ of their compute on capabilities advancements, leaving <10% of compute for focused safety/security work. An arrangement to “pace the frontier” could identify categories of work that count as “safety/security” and require >20% or even >50% of compute to be spent on such areas.

What about China?

In May 2025, right after Pope Leo XIV was inaugurated, Ross Douthat had a sit-down interview with Vice President JD Vance for the New York Times in Rome. Douthat asked Vance a range of questions, but one caught my eye – Douthat asked “Do you think that the U.S. government is capable in a scenario, not like the ultimate Skynet scenario, but just a scenario where AI seems to be getting out of control in some way, of taking a pause?”

VP Vance replied: “The honest answer to that is that I don’t know, because part of this arms race component is if we take a pause, does the People’s Republic of China not take a pause? And then we find ourselves all enslaved to PRC-mediated AI?”

This “China question” is a reasonable question - I don’t want China building AI superintelligence. The proposals described earlier are in general terms and could be implemented between companies (e.g., OpenAI and Anthropic), at a national level (e.g., the US government implements this across the US industry), and/or internationally (e.g., between the US and China).

If you don’t have China on board, there is only so much time you can buy to pace the frontier. If you have China on board and you have good enough verification, you might be able to pace the frontier for a longer period of time.

In order for such a coordinated slowdown to fully work, if it were ever desirable, you would likely need all frontier AI companies across the US and China to agree to stop under the same conditions and to be able to verify that the other parties are also doing so. This would generally require getting the US and China to trust each other to agree to stop and then getting the US and China to each be able to enforce that halt domestically.

But how would they trust each other? The answer from arms control history is: they wouldn’t — and they shouldn’t have to. When Ronald Reagan negotiated nuclear reductions with the Soviets, his refrain, borrowed from a Russian proverb, was “trust, but verify.” The INF Treaty didn’t work because Washington and Moscow trusted each other. It worked because each side had inspectors physically stationed at the other’s missile facilities and satellites overhead.

A US-China agreement on AI would need the same approach. Each side would need the ability to confirm — with high confidence, on an ongoing basis, and without taking anyone’s word for it — that the other side is meeting their obligations.

There are open questions about whether there will ever be enough political will for US-China coordination around superintelligence and whether the monitoring and verification techniques will be good enough. I’m not going to claim that we will definitely find ourselves in worlds where America’s desire to pace the frontier is so strong that it’s willing to engage in a deal with China. But I think it’s plausible enough that it’s worth preparing for.

Five years ago, many of the smartest people in the AI space thought the US government would stay completely unaware of and uninterested in managing AI progress. The belief was that frontier AI companies would reach recursive self-improvement before the government cared, and that the government would never intervene to block the release of models. The belief also was that public would be so supportive of AI after seeing all of its tangible benefits. This ended up being wrong.

Even a month ago, I think very few people would’ve predicted that over 1000 AI company employees would sign a statement advocating for pacing the frontier, and that such a statement would be endorsed by both OpenAI and Anthropic.

This makes me think we need to have a healthy amount of uncertainty about how the future of superintelligence politics will play out. We should prepare for worlds in which our political leaders want to control the pace of frontier AI progress, especially as we approach extremely dangerous thresholds like recursive improvement.

Building the verification infrastructure in time

The good news is that frontier AI development runs on compute, and compute is physical. Advanced AI chips are designed by a handful of companies, fabricated almost entirely at TSMC, dependent on lithography machines from a single Dutch firm, and consumed in enormous, power-hungry, hard-to-hide quantities. In arms control terms, compute plays the role that fissile material played in the nuclear world — scarce, countable, chokepointed input that verification can anchor on.

Data centers at frontier scale draw hundreds of megawatts, and nearly all of that energy exits as heat, visible to infrared satellites. Power grid data, cooling infrastructure, network buildouts, and procurement patterns all leak information. This is the same category of tools the intelligence community already uses to track missile sites and enrichment facilities.

With negotiated access, you can do on-site inspections of declared data centers. There could be stationed personnel, the way US and Russian inspectors lived at each other’s missile plants under INF. And because the AI chip supply chain is so concentrated, cross-referencing records from fabs, packaging plants, and integrators could produce a reasonable census of how much frontier compute exists worldwide and where it went. If the census says a country has a million accelerators and inspectors can only find 800,000, that gap is itself the evidence.

And there may be more advanced methods that are possible. Modern AI chips already ship with confidential computing features that include “remote attestation” — the ability for a chip to cryptographically report on its own configuration. These techniques, and other technological techniques, could potentially be built upon to enable more custom verification solutions.

But if we ever were in a position where we wanted to make a deal with China, it’s not guaranteed that we would have this technology ready in time. For example, when we were in the Cold War and wanted to ban underground nuclear tests with the Soviets, we realized we didn’t have any way to reliably detect such tests, since it was impossible at the time to distinguish them from naturally occurring earthquakes. So there was no treaty on underground testing. Later on, an international network of seismometers was invented, a key verification technology that enabled underground tests to be verified.

Thus, if we did want the option to verifiably slow down AI development in the future, we would need to build the infrastructure now – verification systems that let frontier developers confirm rivals have genuinely stopped or slowed, and where no one is defecting quietly.

So what can we actually do to make it more likely that we have the tools needed to pace the frontier? In my view, this work is among the most important work in all of AI policy. I plan to write more on this topic, but for now, here’s an overview of some promising directions:

1. Building better verification technologies. Verification technologies can help improve the robustness or decrease the cost of verification setups.

   1. Prototyping or retrofitting inference-only clusters. One especially promising area for technical work involves figuring out how to build, prototype, or retrofit inference-only clusters (and the technologies that would make them possible). These are data centers that we can verify are only capable of running existing models (called inference), rather than doing the high-quality training needed to train even more capable models. There are many open technical questions. For example: network taps that let inspectors monitor the traffic flowing through a cluster, workload records that log what a data center computed in a reproducible format, and partial recomputation, where inspectors rerun random samples of that logged work to check the records are honest.⁴

2. Identifying “dark” compute. Most verification can only verify what you know exists — verifying properties of known or declared compute. But could countries or companies be hiding compute we don’t know about? How do we know that there is not a secret data center out there pursuing recursive improvement or superintelligence?

   1. Intelligence agencies. What should intelligence agencies be prioritizing? How can national intelligence services get better estimates of total compute or “dark compute?”

   2. Supply chain audits. Shavit's compute-monitoring framework proposed using supply-chain records from fabs and chip component suppliers to build a registry of who owns frontier chips, and RAND's verification taxonomy treats supply-chain audits as one of the more feasible near-term mechanisms. How would these work, and how much total compute could be estimated from these methods?

   3. Satellites and thermal imagery. Various groups like The Compute Visibility Institute and Federation of American Scientists have proposed that geospatial analysis could be used to detect dark compute or hidden data centers. What are the best ways to use these tools to detect dark compute, and what are the limitations of these approaches? What concealment strategies most effectively get around these tools?

3. China, geopolitics, and macrostrategy. So far, the “pacing the frontier” crowd consists disproportionately of technical people with technical interests in areas like AI and semiconductors. These are extremely smart people! But there are often skillsets that are underrepresented. Experts in China policy, international relations, arms control, diplomacy, and other related areas could be extremely valuable for creating new proposals and refining existing ones. Example questions:

   1. China.

      1. How is China likely to react if it becomes “superintelligence-pilled”?

      2. Which stakeholders in China would matter most for forming their overall strategy around ASI topics? How are they likely to see the world or make decisions?

      3. What would China want out of a deal with the US?

      4. Which entities would China trust most to develop or validate verification technologies?

   2. Geopolitics.

      1. What kinds of assumptions are made about China or geopolitics in AI2040 or Superintelligence Strategy (MAIM)? Are there critical assumptions they get wrong? Are there better ways of understanding and mapping this?

      2. What are alternative or new visions of what “pacing the frontier” could look like on an international scale?

   3. “Minimum viable slapdash deals”. Some work has focused on “grand bargains” or “comprehensive treaties” between the US and China. But what does an initial phase look like? What would the US President do if he wanted to pause for a few months just to learn more, assess the situation, and figure out what to do? What could we verify (before the government validates and trusts technical verification methods)?

In the 20th century, the world had to figure out new technical tools and geopolitical strategies to manage nuclear weapons. Concepts like mutually-assured destruction, second-strike capability, and strategic stability emerged. New verification tools like photoreconnaissance satellites and seismic monitoring were developed, tested, and validated to support verification setups. We will need similar new concepts and technologies for verifying AI. There are, of course, many ways in which the AI situation is not like the nuclear situation, but we can draw some inspiration from history.

There is also a role for Washington right now, well before any agreement is on the table. In the Cold War, the US did not wait for a treaty to become politically viable before building the ability to verify one. Starting in 1959, the VELA program funded detection research precisely so that verification would be ready if the politics ever were. The AI equivalents are concrete. Fund verification R&D through DARPA, NIST, and the national labs. Task the intelligence community with producing estimates of global compute and where it lives. None of this commits the US to slowing anything down — it just ensures that if a President ever wants the option Vance was asked about, the tools actually exist.

If you’re interested in pursuing or supporting work on technical or governance tools to pace the frontier, please feel free to reach out. There’s a lot of urgent work to be done. The frontier is advancing rapidly. And it’s not going to pace itself.

1. Furthermore, Demis Hassabis, the CEO of Google DeepMind, wrote “A Framework for Frontier AI and the Dawning of a New Age”, stating that AI would soon be “perhaps 10x of the Industrial Revolution at 10x the speed” and that “advances on the frontier are outpacing our understanding of the technology” and that “coordinating a slowdown in development among the Frontier Labs” might at some point become necessary.

Hassabis also answered a question from an interviewer where he was asked “In a perfect world, if you knew that every other company would pause, if every country would pause, would you advocate for that?” and replied “I think so”.

2. This also matches earlier statements – previously, OpenAI, in “Built to benefit everyone: our plan”, wrote that they want to “make it possible for the world to take coordinated action, including slowing frontier development when needed, so societal resilience, safety, and alignment can keep pace.”

Also, In “When AI builds itself”, Marina Favaro and Jack Clark from Anthropic agree that the speed of AI development may become a societal issue and stated that “it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology” – and that if “such systems existed” that “would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret”, then Anthropic “expect[s] that [they] would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.”

3. Currently, the best writeup of detecting covert AI projects (including estimates of how long we would be able to do this for) is presented by the AI2040 team.

4. This research agenda on inference-only clusters covers these and other directions.