I have two problems with this essay. The easy one is how one-dimensional and bluntly oppositional it relates to AI -- the whole category of superintelligence is Sauron, and every lab is equal in their evil pursuits of his Ring. They're just The Bad Guys of the story, and the Good Guys here to save us are the wisdom of our forefathers. It's just facile, steamrolling every AI lab into the same comic book mold
The deeper problem is a status quo bias, and the uncritical trust that everything was fine, would be fine, if not for the superintelligent Villains lurking just around the corner. "You can decline to buy an iPhone; or if it harms you, just sue!" That's a regulatory regime that barely kept up with Ford's Model T; ask a Bostonian in 1970 whether they could decline to live in a city occupied by motor vehicles. Ask a Bostonian in 2000 if they can decline occupation by the Internet. Ask in 2020 about Twitter. More sharply, ask a Muslim in 2017 Myanmar whether they would like to share a home with Facebook. A society can be reshaped by soft power, and dominated through entirely consentual exchanges. This was the American Way for a long time, as it at happens.
[Jeremiah Johnson already has made the argument, imo persuasively, that social media is already a Ring of power](https://www.infinitescroll.us/p/the-one-ring-of-posting). We have a poaster President in his second term. For years we've had stories of parents trying to keep their kids off social media, but "my son is the only one without a phone, he's the odd duck out at school". For years, we've had stories of elders getting their personalities changed by media like cable news, or scammed by pig butchers.
I remember my college commencement speaker told me that the singularity was coming, in the ballpark of 2030. That technology was accelerating, and would keep accelerating. AI is just its latest face. It's a delusion to think the genie was ever contained in the bottle. Or that it was ever escapable.
Ultimately if we are poised to create superintelligence without being able to control it, this would be an unacceptable risk to the world. The government needs to be in a position to both see if this is happening, correctly assess the situation, and stop it.
I think most people who think about AI risk agree with that, but how you get from where we are today to that is challenging. Debatably the government becoming much more interested in frontier AI recently is a positive development but it still lacks eyes and power inside the labs themselves where it seems likely that there will be warning signs, hopefully well in advance, but maybe not, of super intelligence.
Man - I fully buy AI risk and I'm conceptually and morally and philosophically 100% on your side, but this is just the worst possible solution.
1. There's no means for "the American people" to do anything about it
2. Why would you even want them to do anything about it?? Do you see how they run their lives? Do you see who they repeatedly put into office?? Literally nothing the masses can come up with would work OR be a good idea
3. Because there is no viable road for 1 to happen, this just immediately devolves to "the government controls AI." This is literally the worst possible solution, because the odds of extinction go up thousands or millionsfold with an NSA + military dominated AI race, and that is the ONLY option if "the government" takes control.
It's obviously millions of times better for "a bunch of smart nerds" to be in charge of safety protocols, fine tuning, and what is and isn't released than "the government." Look at Anthropic's track record so far! You think NSA + military is going to do anything but massively worse?? They already "enemy of the state-ed" Anthropic just for not letting Claude be used to surveil every single American 24/7.
And once the NSA + military is in charge of the AI labs, what do you think China is going to do? The same? Yeah, so would I. So then you get the two dominant world powers BOTH AI racing as fast as possible, in the worst possible environment, with the worst conceivable leadership and priorities and end goals.
You are just fully and openly recommending a path that is guaranteed to increase our existential risk millionsfold.
PB and Phil, "Trust Anthropic's track record" and "trust Altman over the average voter"... The essay's claim isn't that AI companies are villains... it's that we cannot rely on a single actor's good character. In LOTR terms, Boromir was sincere too but it didn't work out. And the whole point of Madison's "auxiliary precautions" is that you don't get to find out how the sincere man behaves under pressure... you build structure so it doesn't matter. Altman's OpenAI has already shed its nonprofit structure under commercial pressure and I don't expect Anthropic to hold up much better in the long run.
PB, on the government point specifically — accountability is not the same as nationalization. Civilian control of the military never meant Congress designs the rifles or that generals get replaced by senators... the army stayed expert-run; it just lived on a two-year lease from the people. If the NSA scenario is what you fear, constructing democratic structure early is exactly how you prevent it.
Phil, Lee Kuan Yew is survivorship bias... for every LKY there's a shelf of Mobutus and Marcoses, and monarchy's fatal flaw was never the first king; it's what happens in the long run. An AI monarchy would have no exit ramp at all. But honestly, the deeper problem isn't that the wise monarch might turn out badly. It's that rule without consent is illegitimate even when it's benevolent. This is the argument of 1776!
And I'd push back on the premise: what the public is actually asking for isn't "let us design the transformers." It's "don't build things you can't demonstrate you can control"... which is arguably the most rational position held by anyone in this entire debate, including the experts. We've run the "just trust the smartest guys in the room" experiment before. It gave us Enron and it gave us Vietnam... Raw intelligence has never been the same thing as wise judgment, and it's definitely never been the same thing as genuine accountability.
Diana - this is the right question, and the honest answer is the tools exist but the incentives don't. Participatory budgeting has real traction with younger residents in some cities, Ireland's citizens' assemblies showed deliberative bodies can handle genuinely hard questions, and Taiwan's vTaiwan/Pol.is process is the closest thing to what you're describing... real-time, scaled constituent input that actually shaped legislation. The bottleneck isn't technology; it's that most legislators currently pay no price for ignoring these channels.
But I don't want to leave this on a cynical note, because the boring old tools still work and normal people badly underrate them. Call your representative's office... they tally every call, and on an issue like AI where they're hearing almost nothing... even twenty constituents is a signal. Show up to a town hall and ask about AI. Write. Protest. These things matter! Congressional offices move on remarkably thin evidence of public attention, and right now the AI companies are supplying too much of it. You don't need an MS in Policy to be part of the input - that's rather the point!
My thoughts exactly. This is a great essay that raises many important issues—but if the public mind has become so ignorant and debased as to be incapable of reason and self-rule (see the recent elections of radical "defund the police and freeze the rent" socialists in New York and countless other indicators that democracy is failing), then perhaps the only safe path, to prevent the self-immolation of ignorant voters is for a benevolent AI dictator to emerge, like Lee Kuan Yew of Singapore, who ruled with authoritarian power for three decades and left his country one of the most successful and flourishing republics on earth.
At this point, I have more confidence in Sam Altman, Dario Amodei, or Demis Hassabis than in the average American citizen to chart a wise and sustainable course. Democracy only works with an educated, rational citizenry, which it seems we no longer have. Perhaps we'd be better off with something closer to an AI-based monarchy run by silicon valley entrepreneurs to save us from ourselves.
That’s a very interesting point you make here. What do you think leads to the failure of the average American citizen to chart wise and sustainable course?
I’ve been thinking about systems in place that enable young people (19-29 e.g.) to provide input/feedback on local (district/city/town-level) policy or budget allocation decisions.
There are poorly attended public hearings, or an option to upvote a bill on NY state website. If I didn’t get my MS in Policy, I would not have known about the bill upvote option. From my understanding those are the 2 options available for an average NY resident to participate in decision-making.
Then there is the reliance on news to bring legislative developments into the public sphere, but after a short “breaking news” piece there’s no guidance provided on what an average citizen can do to support or object what they’ve just heard about. Not to say that it’s media’s role.
The U.S. being the birth place of social media, I’m surprised how / why local representatives have not invested in effective at scale tools publicized via social media to capture real-time feedback from its constituents on policy decisions?
Peter — thank you for the reply. You make a fair point that the old tools still work. A small number of calls can genuinely move the needle. However, I have to push back on whether these tools are still the best way to engage the public given the participation rates.
Participatory budgeting is a great. Yet, it only represents a small fraction of total municipal or state budgets in the US. The vast majority of policy decisions are still handled through traditional channels. These channels feel isolating.
When you call a representative or speak at a town hall, you act in a silo. You make a call, but then you never know who else in your community shares your concerns. Even with an MPA navigating these disjointed systems feels opaque and frankly lonely.
A marketed to community, digital portal solves this problem through visibility. It functions like social media for civic action. You can see exactly what your neighbors care about. You can upvote a specific bill or proposal. You can discover critical issues you might not have considered. This builds a sense of collective momentum empowering public engagement.
Ukraine’s Diia government app proves that this works for important masters. It handles much more than small administrative choices like renaming a street. In 2022, over 1.7 million Ukrainians used the app to vote on civilian firearms legislation. The results directly shaped national security policy. Later that year, almost 780,000 people voted to replace the Soviet emblem on the Motherland monument with the Ukrainian trident. The government followed this public mandate and physically updated the monument.
Recently, the government even used Diia to let citizens name Ukraine's first state artificial intelligence model. Diia shows that we have the technology to make civic engagement a transparent, collective experience. This then empowers people to be more engaged in policy making, and governance in general.
The "auxiliary precautions" framing travels further than AI, I think. We already hand real authority to epistemic powers — forecasters, pundits, agency models — on the strength of asserted credibility, with almost none of the adversarial machinery you describe. No two-year lease, no binding oath to a prior commitment, nothing stopping a confident voice from quietly revising what it "always said" once the outcome is in. Your Madisonian point — that a benevolent authority answerable to no one is still unaccountable — seems to bite hardest exactly where we least notice it, because the forecaster who's usually right feels trustworthy enough that nobody demands the structure. Would you extend the constitutional analogy down to individual expert credibility, or is that problem different in kind?
A bit in-jokish to use em-dashes where commas, colons and semicolons are more appropriate, in the first few paragraphs. But hey, I'll take a smile where I can get one — these days.
As a generally pro-AI person I thought this piece was deft because of how many dimensions along which it helpfully reframes the problem. A technology capable of upending governance is not a mere market participant, and we have precedents for this. Likewise the mechanics of inhibiting the harm of such entities cannot rest on nominal barriers inviting them to action, but on oppositional superstructures where great forces coming at opposing vectors tend to cancel out.
1776 is an interesting and unexpected frame, and the Washington-at-Annapolis move is genuinely moving. Power's signature act relinquishing it is an instinct the current moment lacks.
Two places where I think 1776 gets us only halfway.
First, your Madisonian machinery is vertical: it answers the accountability question. How do the people assert authority over the lab leaders holding the Ring? But there's a horizontal problem the founding can't reach. Even if America hypothetically domesticated its labs tomorrow the capability doesn't stop at the water's edge. China's labs keep racing. Your nuclear analogy holds the answer you left out: the standing army was bound by the Constitution, but nuclear weapons required the NPT. One is national accountability. The other is international architecture. 1776 gives us the first and is silent on the second, because in 1776 there was no peer power operating on different terms.
Second, you note that Madisonian checks assume "contending powers are roughly comparable," and that a superintelligence breaks the balancing machinery — and then you reassure us the Ring is held by humans "for at least a few more years." But that's where the real clock is. The window in which these leaders can still be obliged to answer isn't measured in constitutional time. It's measured in model releases. Madison studied every confederacy in history before Philadelphia. This age may not grant that luxury: capability is compounding faster than any deliberative body can convene.
So 1776 gives us the accountability half and the nuclear age gives us the coordination half; and this is the first time in history both problems have arrived in the same leaders, at the same moment, at a speed that may not allow us to solve either. I worked through exactly this convergence in a piece a few weeks ago — Five Get Into Trouble — if it's useful: https://rajeshachanta.substack.com/p/five-get-into-trouble?utm_source=publication-search. The Ring analogy you close on is one I keep returning to as well. Boromir was sincere too.
I have two problems with this essay. The easy one is how one-dimensional and bluntly oppositional it relates to AI -- the whole category of superintelligence is Sauron, and every lab is equal in their evil pursuits of his Ring. They're just The Bad Guys of the story, and the Good Guys here to save us are the wisdom of our forefathers. It's just facile, steamrolling every AI lab into the same comic book mold
The deeper problem is a status quo bias, and the uncritical trust that everything was fine, would be fine, if not for the superintelligent Villains lurking just around the corner. "You can decline to buy an iPhone; or if it harms you, just sue!" That's a regulatory regime that barely kept up with Ford's Model T; ask a Bostonian in 1970 whether they could decline to live in a city occupied by motor vehicles. Ask a Bostonian in 2000 if they can decline occupation by the Internet. Ask in 2020 about Twitter. More sharply, ask a Muslim in 2017 Myanmar whether they would like to share a home with Facebook. A society can be reshaped by soft power, and dominated through entirely consentual exchanges. This was the American Way for a long time, as it at happens.
[Jeremiah Johnson already has made the argument, imo persuasively, that social media is already a Ring of power](https://www.infinitescroll.us/p/the-one-ring-of-posting). We have a poaster President in his second term. For years we've had stories of parents trying to keep their kids off social media, but "my son is the only one without a phone, he's the odd duck out at school". For years, we've had stories of elders getting their personalities changed by media like cable news, or scammed by pig butchers.
I remember my college commencement speaker told me that the singularity was coming, in the ballpark of 2030. That technology was accelerating, and would keep accelerating. AI is just its latest face. It's a delusion to think the genie was ever contained in the bottle. Or that it was ever escapable.
(Slightly off topic)
I like to say that the Singularity already happened back in 1876, when Thomas Edison invented the industrial research laboratory. ;)
Do you have policy proposals for how to deal with superintelligence?
Ultimately if we are poised to create superintelligence without being able to control it, this would be an unacceptable risk to the world. The government needs to be in a position to both see if this is happening, correctly assess the situation, and stop it.
I think most people who think about AI risk agree with that, but how you get from where we are today to that is challenging. Debatably the government becoming much more interested in frontier AI recently is a positive development but it still lacks eyes and power inside the labs themselves where it seems likely that there will be warning signs, hopefully well in advance, but maybe not, of super intelligence.
MIRI's Technical Governance Team has done some thinking about how to get to a world where government can prevent superintelligence from being created: https://techgov.intelligence.org/research/an-international-agreement-to-prevent-the-premature-creation-of-artificial-superintelligence
Man - I fully buy AI risk and I'm conceptually and morally and philosophically 100% on your side, but this is just the worst possible solution.
1. There's no means for "the American people" to do anything about it
2. Why would you even want them to do anything about it?? Do you see how they run their lives? Do you see who they repeatedly put into office?? Literally nothing the masses can come up with would work OR be a good idea
3. Because there is no viable road for 1 to happen, this just immediately devolves to "the government controls AI." This is literally the worst possible solution, because the odds of extinction go up thousands or millionsfold with an NSA + military dominated AI race, and that is the ONLY option if "the government" takes control.
It's obviously millions of times better for "a bunch of smart nerds" to be in charge of safety protocols, fine tuning, and what is and isn't released than "the government." Look at Anthropic's track record so far! You think NSA + military is going to do anything but massively worse?? They already "enemy of the state-ed" Anthropic just for not letting Claude be used to surveil every single American 24/7.
And once the NSA + military is in charge of the AI labs, what do you think China is going to do? The same? Yeah, so would I. So then you get the two dominant world powers BOTH AI racing as fast as possible, in the worst possible environment, with the worst conceivable leadership and priorities and end goals.
You are just fully and openly recommending a path that is guaranteed to increase our existential risk millionsfold.
PB and Phil, "Trust Anthropic's track record" and "trust Altman over the average voter"... The essay's claim isn't that AI companies are villains... it's that we cannot rely on a single actor's good character. In LOTR terms, Boromir was sincere too but it didn't work out. And the whole point of Madison's "auxiliary precautions" is that you don't get to find out how the sincere man behaves under pressure... you build structure so it doesn't matter. Altman's OpenAI has already shed its nonprofit structure under commercial pressure and I don't expect Anthropic to hold up much better in the long run.
PB, on the government point specifically — accountability is not the same as nationalization. Civilian control of the military never meant Congress designs the rifles or that generals get replaced by senators... the army stayed expert-run; it just lived on a two-year lease from the people. If the NSA scenario is what you fear, constructing democratic structure early is exactly how you prevent it.
Phil, Lee Kuan Yew is survivorship bias... for every LKY there's a shelf of Mobutus and Marcoses, and monarchy's fatal flaw was never the first king; it's what happens in the long run. An AI monarchy would have no exit ramp at all. But honestly, the deeper problem isn't that the wise monarch might turn out badly. It's that rule without consent is illegitimate even when it's benevolent. This is the argument of 1776!
And I'd push back on the premise: what the public is actually asking for isn't "let us design the transformers." It's "don't build things you can't demonstrate you can control"... which is arguably the most rational position held by anyone in this entire debate, including the experts. We've run the "just trust the smartest guys in the room" experiment before. It gave us Enron and it gave us Vietnam... Raw intelligence has never been the same thing as wise judgment, and it's definitely never been the same thing as genuine accountability.
Diana - this is the right question, and the honest answer is the tools exist but the incentives don't. Participatory budgeting has real traction with younger residents in some cities, Ireland's citizens' assemblies showed deliberative bodies can handle genuinely hard questions, and Taiwan's vTaiwan/Pol.is process is the closest thing to what you're describing... real-time, scaled constituent input that actually shaped legislation. The bottleneck isn't technology; it's that most legislators currently pay no price for ignoring these channels.
But I don't want to leave this on a cynical note, because the boring old tools still work and normal people badly underrate them. Call your representative's office... they tally every call, and on an issue like AI where they're hearing almost nothing... even twenty constituents is a signal. Show up to a town hall and ask about AI. Write. Protest. These things matter! Congressional offices move on remarkably thin evidence of public attention, and right now the AI companies are supplying too much of it. You don't need an MS in Policy to be part of the input - that's rather the point!
My thoughts exactly. This is a great essay that raises many important issues—but if the public mind has become so ignorant and debased as to be incapable of reason and self-rule (see the recent elections of radical "defund the police and freeze the rent" socialists in New York and countless other indicators that democracy is failing), then perhaps the only safe path, to prevent the self-immolation of ignorant voters is for a benevolent AI dictator to emerge, like Lee Kuan Yew of Singapore, who ruled with authoritarian power for three decades and left his country one of the most successful and flourishing republics on earth.
At this point, I have more confidence in Sam Altman, Dario Amodei, or Demis Hassabis than in the average American citizen to chart a wise and sustainable course. Democracy only works with an educated, rational citizenry, which it seems we no longer have. Perhaps we'd be better off with something closer to an AI-based monarchy run by silicon valley entrepreneurs to save us from ourselves.
See https://blog.peterwildeford.com/p/the-alignment-problem-of-1776/comment/297280733
That’s a very interesting point you make here. What do you think leads to the failure of the average American citizen to chart wise and sustainable course?
I’ve been thinking about systems in place that enable young people (19-29 e.g.) to provide input/feedback on local (district/city/town-level) policy or budget allocation decisions.
There are poorly attended public hearings, or an option to upvote a bill on NY state website. If I didn’t get my MS in Policy, I would not have known about the bill upvote option. From my understanding those are the 2 options available for an average NY resident to participate in decision-making.
Then there is the reliance on news to bring legislative developments into the public sphere, but after a short “breaking news” piece there’s no guidance provided on what an average citizen can do to support or object what they’ve just heard about. Not to say that it’s media’s role.
The U.S. being the birth place of social media, I’m surprised how / why local representatives have not invested in effective at scale tools publicized via social media to capture real-time feedback from its constituents on policy decisions?
See https://blog.peterwildeford.com/p/the-alignment-problem-of-1776/comment/297280733
Peter — thank you for the reply. You make a fair point that the old tools still work. A small number of calls can genuinely move the needle. However, I have to push back on whether these tools are still the best way to engage the public given the participation rates.
Participatory budgeting is a great. Yet, it only represents a small fraction of total municipal or state budgets in the US. The vast majority of policy decisions are still handled through traditional channels. These channels feel isolating.
When you call a representative or speak at a town hall, you act in a silo. You make a call, but then you never know who else in your community shares your concerns. Even with an MPA navigating these disjointed systems feels opaque and frankly lonely.
A marketed to community, digital portal solves this problem through visibility. It functions like social media for civic action. You can see exactly what your neighbors care about. You can upvote a specific bill or proposal. You can discover critical issues you might not have considered. This builds a sense of collective momentum empowering public engagement.
Ukraine’s Diia government app proves that this works for important masters. It handles much more than small administrative choices like renaming a street. In 2022, over 1.7 million Ukrainians used the app to vote on civilian firearms legislation. The results directly shaped national security policy. Later that year, almost 780,000 people voted to replace the Soviet emblem on the Motherland monument with the Ukrainian trident. The government followed this public mandate and physically updated the monument.
Recently, the government even used Diia to let citizens name Ukraine's first state artificial intelligence model. Diia shows that we have the technology to make civic engagement a transparent, collective experience. This then empowers people to be more engaged in policy making, and governance in general.
The "auxiliary precautions" framing travels further than AI, I think. We already hand real authority to epistemic powers — forecasters, pundits, agency models — on the strength of asserted credibility, with almost none of the adversarial machinery you describe. No two-year lease, no binding oath to a prior commitment, nothing stopping a confident voice from quietly revising what it "always said" once the outcome is in. Your Madisonian point — that a benevolent authority answerable to no one is still unaccountable — seems to bite hardest exactly where we least notice it, because the forecaster who's usually right feels trustworthy enough that nobody demands the structure. Would you extend the constitutional analogy down to individual expert credibility, or is that problem different in kind?
A bit in-jokish to use em-dashes where commas, colons and semicolons are more appropriate, in the first few paragraphs. But hey, I'll take a smile where I can get one — these days.
I’ve been a big fan of em dashes long before LLMs made them infamous.
As a generally pro-AI person I thought this piece was deft because of how many dimensions along which it helpfully reframes the problem. A technology capable of upending governance is not a mere market participant, and we have precedents for this. Likewise the mechanics of inhibiting the harm of such entities cannot rest on nominal barriers inviting them to action, but on oppositional superstructures where great forces coming at opposing vectors tend to cancel out.
1776 is an interesting and unexpected frame, and the Washington-at-Annapolis move is genuinely moving. Power's signature act relinquishing it is an instinct the current moment lacks.
Two places where I think 1776 gets us only halfway.
First, your Madisonian machinery is vertical: it answers the accountability question. How do the people assert authority over the lab leaders holding the Ring? But there's a horizontal problem the founding can't reach. Even if America hypothetically domesticated its labs tomorrow the capability doesn't stop at the water's edge. China's labs keep racing. Your nuclear analogy holds the answer you left out: the standing army was bound by the Constitution, but nuclear weapons required the NPT. One is national accountability. The other is international architecture. 1776 gives us the first and is silent on the second, because in 1776 there was no peer power operating on different terms.
Second, you note that Madisonian checks assume "contending powers are roughly comparable," and that a superintelligence breaks the balancing machinery — and then you reassure us the Ring is held by humans "for at least a few more years." But that's where the real clock is. The window in which these leaders can still be obliged to answer isn't measured in constitutional time. It's measured in model releases. Madison studied every confederacy in history before Philadelphia. This age may not grant that luxury: capability is compounding faster than any deliberative body can convene.
So 1776 gives us the accountability half and the nuclear age gives us the coordination half; and this is the first time in history both problems have arrived in the same leaders, at the same moment, at a speed that may not allow us to solve either. I worked through exactly this convergence in a piece a few weeks ago — Five Get Into Trouble — if it's useful: https://rajeshachanta.substack.com/p/five-get-into-trouble?utm_source=publication-search. The Ring analogy you close on is one I keep returning to as well. Boromir was sincere too.
Thank you for thinking to write this. Encouraging, sobering and motivating. One finds one’s self gazing lovingly at the ring from time to time.
Thank you. If only our government-funded schools would teach the basics of logic, rhetoric, and philosophy.