Q: How about Hugging Face and other instances in which AI agents have behaved unexpectedly and ended up hacking into other companies’ systems? What should we make of those incidents?

These kinds of software attacks seem within scope of stuff we could have expected to happen. I think we're realizing that there are a lot of cracks in a lot of code, all over the internet. So, yes, I think a real concern is things like software attacks. A version of this that worries me could happen if we hook these up to things in the real world — a factory floor or the means of production, so to speak. To me that sounds like something we should objectively not be doing at this point. 

What do you think about the proposal by the frontier AI labs to slow things down?

I would be very supportive of them slowing down. This has been discussed for five or six years in academia and outside the labs. I feel like many people I know have been supportive of this since then. I wish it had happened a while ago, and we'd be in a less panicky situation now.

But I also think about this in terms of supply and demand. The frontier labs are the supply side, and we could certainly be doing things on that side. But there’s more to it. Something that has frustrated me about this conversation is that it really puts the frontier labs as the main characters. People seem to just be asking which lab has the most powerful model and is it powerful enough to kill us all. But that backgrounds everything else. AI doesn't get control of stuff that we don't give it control of, or that we don't expose it to. We still have control over how widely it’s deployed. I think it's worth remembering that there are still decisions to be made on that. There's not an inevitable progression outside of our control. There are a lot of powerful players in government and in non-AI corporations that have control over the deployment side, and I think that can make a big difference into how well or badly this could go. For some reason, in the national discussion, it's always just taken for granted that this is inevitable and we have no control. But in reality, we can say “no.”

Q: I realize you’re an AI expert and not a public policy expert, but what would you as a technologist say is a practical thing the government can do in the short-term?

Right, not my expertise, but I think one approach might be to just require humans in the loop — simply making it so you're not allowed to give control over certain things to autonomous agents. I think that could make a big difference. I’ve also heard people making an analogy to how we investigate plane crashes. When planes crash, third parties are allowed free rein to investigate what happened. But for something like the Hugging Face attack, there were like three third-party researchers who looked at this. They’re smart, credible people, but they’re pretty close with that company. That’s not the most robust way to investigate something. I think something closer to the crash investigation model is the way to go. 

Q: How has all of this influenced the work that ARIA is doing?

ARIA is a research institute that brings together computer scientists, cognitive scientists and civil society experts to think about how we build AI that interacts safely with humans. We’re focused primarily on the basic science that will help us answer hard questions about humans, AI and their interaction.

We just met with our external advisory board to talk about this exact question — basically, how does an institute like ARIA have impact in this kind of an environment? There’s absolutely a role for academic research in this, but academic research works on a different timeline than Silicon Valley. And so we're trying to kind of figure out how to thread this needle. What I would hate to see is a situation where everything feels so time-sensitive that no one thinks long-term anymore. This is a common thing I hear from Silicon Valley: We don't have 10 years to think things over at the rate things are moving. But if we don’t do forward thinking now, we’re going to be far worse off 10 years from now.