Discussion about this post

User's avatar
Scott Alexander's avatar

Thanks for this. A few questions, from the perspective of someone who might be doing grantmaking work:

1. Is the "Right now, a lot of smart people are working on work that really doesn’t need to happen now" section a direct request to fund these things less? I had previously been trying to get them more funding, on the grounds that I didn't think the ~3 months we might have during this kind of "scramble" was enough to get the job done, on the grounds that it might be useful to have smart people from within the AI safety community rather than random defense-tech people as the nucleus around whom these projects converged, and on the grounds that it might make the government more likely to support large-scale versions of these projects if someone can present them with ready-to-go small-scale versions as proof of concept. Do you find these reasons unconvincing?

2. If not verification projects, what kind of projects/groups would you prioritize to make sure this sort of scramble goes well, beyond the obvious?

3. Can you explain the idea of a pause that doesn't kick in now but "stops before RSI"? Labs already claim to be doing RSI, and RSI is a spectrum from partial automation to full automation. Is your idea to stop before full automation? Couldn't companies circumvent that by only automating 99%? How would you define RSI for a task like this?

David Spies's avatar

I'm having trouble imagining anything that could plausibly look like "missiles in Cuba" to set this off, at least not until after it's likely too late to do anything. Or rather, the sort of thing I'm imagining is "lab was training/evaluating an AI and it secretly broke out of the box, went rogue and started hacking things on the open internet" but that happened already a couple times

Do you have a story of what leads to getting serious?

9 more comments...

Ready for more?