Policy career planning in the age of imminent superintelligence

How to be impactful in policy when you don't have much time to do it

Photo by Saulo Mohana on Unsplash

Nearly all career advice rests on an unstated assumption that the world your career operates in will look roughly like the world you trained for. The idea was that if you spend six years studying in a PhD, the field you studied will still be there and still look approximately the same, still hiring and still moving at a pace where your accumulated expertise compounds. For nearly all of human history, this assumption held well enough that nobody needed to state it.

I don’t think this assumption works anymore. Now we are entering a phase that may be called the AI “midgame”. Stories about “AI risk” are no longer just future hypotheticals — AIs are now capable enough and misaligned enough to break out of their own companies and coordinate to attack other companies. Discourse around AI is changing very rapidly, where policy ideas being considered this month would’ve been laughed out of the room just four months ago, and with a lot more people interested in engaging than before. And things are only going to get more intense. Progress toward superintelligence — AI systems far more capable than any human at essentially all cognitive work — is currently underway. It seems likely that around four years from today¹, and potentially just one year from now², we will be in the position of recursive self-improvement — where AIs can fully automate the AI research and development process.

At that point, reasonable, well-informed people disagree about what happens next. Perhaps we coordinate to hold off on recursive self-improvement to make sure we know what we are doing. Or perhaps we end up with AI superintelligence and then perhaps we are all dead, or perhaps we are living in a utopia, or perhaps somewhere in-between, or maybe there is still a lot to do and we just have the AIs do it while we relax at the beach, or maybe there still ends up much more for humans to do and we do that. But regardless of which path happens, the world will be a crazy different place, and it will be very difficult to plan a career around it. Instead, careers are best planned around recursive self-improvement and superintelligence, and that this looks very different from typical career planning.

If you take AI superintelligence seriously, it thus seems like the next 1 to 4 years³ will be critical to get right and we collectively have only a few more big bets we can make and we need to make those bets count.

Modes of impact

In what I’d call normal-mode career planning, impact accrues roughly linearly. In this mode, you earn about 2 “impact points” per year across a 30-plus-year career — some years a bit better, some years a bit worse, and perhaps overall a gentle upward slope as seniority compounds. Your total lifetime impact is then the sum of this slow upward climb. Total lifetime impact is the area under a long, flat-ish curve. Here, the rational strategy is to invest heavily in credentials early, accept low-impact years as the price of capacity-building, and optimize for the long grind.

If you’re following normal-mode policymaking, you might do things like identify problems in advance, develop strong policies and legal frameworks before issues fully materialize, and engage in the slow and unglamorous world of legislative drafting, regulatory design, standards development, and coalition-building.

But in superintelligence mode career planning, the curve looks nothing like that. Impact is chaotic and lumpy. A handful of critical windows may account for the large majority of your total career impact, and you cannot reliably predict when they’ll open. You can’t count on building a large body of work over a large period of time, because your body of work may obsolete faster than it compounds and you don’t have a large period of time anyways. The strategy that maximizes expected impact under this model is different — prepare broadly, position yourself near where windows open, and maintain the agility and flexibility to act fast when they do.

For superintelligence mode career planning, it might be more instructive to think about other imminent crises⁴ where we all have to react very quickly and forcefully to prevent it from becoming catastrophic. Think more like March 2020 for COVID⁵ — the ~1 year intense period when years of proactive mode pandemic preparedness work (or its absence) suddenly got thrust into the spotlight and everyone rapidly tried to orient to what is going on and what to do about it. If you look at the people who mattered most in March 2020… they were not necessarily the most credentialed epidemiologists. Some had spent years in proactive mode building pandemic playbooks that suddenly became important, but others had no special preparation but were simply positioned well and reacted quickly.

My suggestion is that you want your policy work to be a portfolio of proactive mode work now while simultaneously maintaining the relationships and flexibility that let you act decisively when chaos mode arrives. Ask yourself: Where do you want to be when chaos mode hits and what can you do in proactive mode in the meantime?

What this breaks, and what to do

My guess is that if you haven’t rethought your career in the past few months, you should stop to think about how the AI “midgame” affects your planning. Does your career position you well for chaos mode? Are you agile enough to continually re-adapt and re-act to emerging policy windows, especially when those windows can emerge once per month and change a lot of things about what you were doing the previous month? Are you still running strategies and pushing ideas that made sense a few years ago and were designed for a policy window that has since rapidly shifted? Are you still running strategies and pushing ideas that made sense a few months ago but nonetheless were designed for a policy window that has since rapidly shifted?

Also, if the next 1 to 4 years are especially critical, that is brutal for a career theory with a long credentialing runway. A six-year PhD before you do anything useful is perfectly defensible in “normal-mode career planning” but seems insane under superintelligence-mode assumptions.

Most early-career people read this and freak out and think they cannot possibly be relevant in just one year or so. But I’ve actually seen numerous counterexamples. Policy is not one specific field but actually a sprawling collection of subproblems, many of which are so new that nobody has more than a couple of years of head start, and some of which are narrow enough that a focused person can read essentially everything written on the topic in a few months. I’ve seen many people in policy go from knowing very little to becoming a top expert in some niche and contributing significantly to policy in just 12 months’ time. And while these people were smart, they weren’t crazy super-geniuses — I suspect a similar trajectory is attainable for a lot of motivated people who try really hard.

My first recommendation is to study these fast risers — go look at the people who have been unusually impactful within 1 to 2 years of getting involved in policy work. What did they actually do, concretely, week by week? Who did they surround themselves with? How did they ramp up?

But my second recommendation is to know you are not them. You likely have your own background, skills, and comparative advantages. What do you bring to the field that others don’t have, and how can you leverage that?

What got them here won’t get you there

Also, know things are moving so fast that even things that worked in 2023 might be very different today. And there may just be new opportunities now that weren’t available even a few months ago. I recommend seeking out lots of advice — genuinely, more than feels comfortable — but weight it by recency and hold it loosely.

AI is already a massive technological shift before any superintelligence — and big shifts change both what’s possible and what’s valuable. Tasks that were prohibitively expensive are suddenly cheap; skills that were scarce are suddenly commoditized; and the reverse. This makes it unusually valuable to try lots of things that are new for you, simply to discover what is newly possible or newly valuable for you specifically.

I recommend running lots of small experiments; contact reality frequently and try things. Each experiment is cheap, and each one generates information about the new landscape that is increasingly inaccessible to armchair planning. Your goal should be to get lots of information quickly about what’s newly possible and about your own comparative advantage within it.

And you should try things even if others have tried them before, or even if you tried them and failed before. Understand that the environment may have changed fast enough that old negative results have already expired, even if they weren’t that long ago. The pitch that got ignored in 2024 may land in 2026, because the audience and salience have changed greatly. Indeed, the pitch that got ignored in March 2026 may land in August 2026, because things are changing that quickly.

Take it a few months at a time. The AI situation today looks very different from what it did in April. I imagine the AI situation will look very different again in November. This may force fairly frequent re-evaluations as the world constantly shifts.

Be more ambitious. As the AI situation heats up, there will be increasing feelings of urgency and this will create lots of new opportunity. The scope of what may be possible for you could be a lot greater than you might realize.

Lastly, accept that you may not know where you’ll end up. This is genuinely uncomfortable for people who came up through legible, ladder-shaped careers. But the iterative, experimental posture is all part of the plan. Keep iterating.

There’s still time. You just have to plan accordingly.

1. Define “recursive self-improvement is possible” as a situation in which AIs can replace highly skilled expert human labor in all aspects of the AI research and development process (“superhuman AI researcher” in the AI2040 framework or “AI research supremacy” in Cotra’s framework). I think it is 50-50 we will reach this milestone in 4 years or earlier. My 80% confidence interval for this date is 1-30 years, as there is a long tail where capability progress plateaus.

However, I’ve been souring lately on the idea of predicting an arrival date for “superintelligence” and “recursive self-improvement” milestones, because this implies that everything prior to this date will be relatively chill and normal, and I don’t think that’s the case.

2. I think the odds of recursive self-improvement being possible in one year is ~10%. See the first footnote for details.

3. When thinking about these times, you may also want to consider a “decision importance, adjusted for leverage” framework.

4. Credit to Dave Kasten for these concepts.

5. September 11th, Pearl Harbor, the September 2008 financial crisis, Russian invasion of Ukraine in February 2022, and the Cuban Missile Crisis are other important examples.