How the OpenAI-Hugging Face Hack May Affect the Geopolitics of AI Governance

Audio of this conversation is available via your favorite podcast service.

On July 16, Hugging Face, a company that provides a platform for AI models, datasets, and other machine learning applications, posted a “security incident disclosure” to its blog about an “intrusion” into its infrastructure. But this was no typical hack. The company said “it was driven, end to end, by an autonomous AI agent system,” matching what it said is the “‘agentic attacker’ scenario the industry has been forecasting.”

A few days later, OpenAI posted to its blog that the agent was in fact “driven by a combination of OpenAI models” with “reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠ of cyber capabilities.”

OpenAI called it an “unprecedented cyber incident, involving state-of-the-art cyber capabilities,” a framing that many observers regarded as an attempt to shift the blame to its AI rather than the decisions taken by the executives and engineers that created the circumstances in which the incident occurred.

Nevertheless, cybersecurity experts say the event points to an important shift in the threat landscape, one with implications for domestic US and international debates over AI governance and security.

To try to better understand these developments, I spoke to two individuals who are following the details closely:

• Vinh Nguyen, a senior fellow for AI at the Council on Foreign Relations and the former chief responsible AI officer and chief data scientist for operations at the National Security Agency (NSA), and
• Graham Webster, a lecturer and research scholar in the Program on Geopolitics, Technology, and Governance at Stanford University, where he leads the DigiChina Project.

What follows is a lightly edited transcript of the discussion.

Vinh Nguyen:

My name is Vinh Nguyen. I am the senior fellow for AI at the Council on Foreign Relations.

Graham Webster:

I'm Graham Webster. I'm a lecturer and research scholar in the Program on Geopolitics, Technology, and Governance at Stanford University, and I lead the DigiChina project there.

Justin Hendrix:

I'm very pleased to have you both here, and, Graham, pleased to have you again on the podcast.

Vinh, I'm going to start with you, was looking through some of the things that you've written lately for the council. And in a piece in May, you talked about the sort of 30-year trajectory of cybersecurity and some of the assumptions that have been in place for a long time. You talked about the idea that cybersecurity has rested on three assumptions in particular, the idea that sophisticated attacks would remain expensive, that identity systems built for humans could extend to whatever came next, and that human judgment would remain in the path of consequential decisions.

You talked about some things that had really shaken those assumptions at the time. You talked about them as bolts of lightning. One was a November incident where Anthropic disclosed China state-sponsored actors using Claude apparently for various exploits. Another was the Mythos moment. Have we had another bolt of lightning this week learning about what's happened with this OpenAI "rogue agent", which escaped its sandbox, and wandered off, and did various meddling at Hugging Face? Is this another bolt of lightning? How serious is this? What does it mean?

Vinh Nguyen:

Yes, so this is another bolt of lightning, but I think we anticipated this third bolt a while back. I would say that for any frontier AI capabilities, the ability to discover vulnerabilities have been known. If you look at some of the UK AI Security Institute’s testing on autonomy, we know this is happening. And so within the community and those who work in the frontier labs, we all know that we have to find ways to sandbox and control these capabilities knowing that it can act autonomously if we let it run for long-horizon task. So this bolt of lightning is anticipated. We just didn't know where and when I would have anticipated that it would come from a cyber criminal group or whatnot, to do this. But in this case, unfortunately it was an OpenAI set of models that was able to break through the sandbox.

So I think this is a really good lesson to learn, that security controls are key to manage, and constraint, and steer these capabilities, be able to have controllability of models, and be able to align the models so that it doesn't do something like this. It's challenging, but should be done as a safeguard. And lastly, that we have to have monitoring in place so when failure modes like this happen, we actually know what's going on and be able to intervene if we can do it. So this is not somehow out of the blue. I think we anticipated it and it came. And I think people should not be surprised that this can happen with frontier AI capabilities.

Justin Hendrix:

And there's been a lot of this discussion online about the extent to which OpenAI appears to have almost spun this as a marketing ploy to say, "Oh, look how dangerous our advanced agents are, advanced models are." What do you make of that? Is this a both things are true at once? Maybe OpenAI is somehow able to do what it's done in past to portray its models in this regard as extremely powerful and dangerous, and yet at the same time, I don't know, this is a real phenomenon that's occurred and one that clearly concerns cybersecurity experts. Where do we separate the hype from the reality here?

Vinh Nguyen:

It's a real incident, and we should take it seriously. When Anthropic released Mythos and talked about the abilities to find vulnerabilities, I can tell you lots of people in the cybersecurity community took it as hype or a marketing ploy. And then they rushed to figure out if it's real or not, and they realize it's real, and then there's a problem. And so I think that for so long, the cybersecurity community make assumptions about the context, the adversaries, and the conditions that we're in, but the time is changing and the assumptions... Like I mentioned, identities, sophisticated actors no longer require nation states and resources to do. Sophisticated attacks can be done by anyone.

And those are the assumptions that I think we have to check. Otherwise, we walk into this new environment believing, naively, that the controls and the practices that we had will work. And I can tell you that it will not. And so we have to be on guard as we introduce these new technologies into the ecosystem. I would take it seriously as an incident. No one, no lab would want this incident. And so I would minimize the debate discussion that it's a hype.

Justin Hendrix:

Graham, I want to bring you in here as with... It seems like everything these days, there's a thread of this that relates to this broader discourse about the US versus China in terms of models, and capabilities, and open versus closed, and all of these sort of themes and topics that we've discussed. But in this particular case, it turns out Hugging Face, I suppose, felt it needed to turn to Chinese open model in order to help fend off this attack, which it learned later was coming from OpenAI's agents. I don't know, what does this tell us? How is this an entry point, I suppose, into some of the things you're looking at, this broader geopolitics of AI and these kind of considerations between the US and China?

Graham Webster:

Yeah, I mean, it's a strange event. I immediately want to disclaim any detailed technical understanding about what went on in the cybersecurity incident, both because I'm not a technical person in that way and because I think a lot of the details are not fully public. But what I understand from reporting is that the incident response team at Hugging Face initially went to work trying to figure out what was going on, and they turned to some US models, some of the paid ones from OpenAI and Anthropic. And they ran into cybersecurity guardrails in the way that those models are allowed to be used, that are designed to stop people from using those models for offensive cyber purposes, but in effect, here, undermined the ability of Hugging Face to use them for defense. Now, Hugging Face is very lucky in this situation. They run a massive AI inference and model distribution system, so they had on hand the ability to run full size versions of essentially any open-weights model they want from China.

So the reporting suggests that they then turn to those, and that tooling helped them respond to the event. So this sets up a really interesting thicket of dilemmas. You've got US model providers want to put cybersecurity guardrails on their most advanced models to prevent misuse. There's also US government policy that affects this type of deployment. And at the same time, Chinese open models that don't have the same guardrails, especially when they're run on one's own infrastructure, are accessible and they're useful for this stuff.

Now, I can't compare whether the new Kimi model from Moonshot AI is just as exquisitely powerful as Fable or Mythos from Anthropic for cyber uses, but clearly, it's useful. And so you have the obvious point that people can use the Chinese models for defensive purposes and probably for offensive purposes out there. And the guardrails exist probably on... The Chinese model providers hosted services are going to have more guardrails than if you install the open model on your own. So it sets up this weird distributional dilemma where US actors, both government and private, want to control cyber misuse. I think most of the Chinese labs and the Chinese government would also like to control cyber misuse. But when there are open models available, people can use it as they please. And in this case, Hugging Face got some good defensive utility out of it.

Justin Hendrix:

What does this tell us about some of these considerations about constraining models on both sides? I mean, we've had, just in the last bit, this moment where these export control directives were used to put constraint on Fable and Mythos from Anthropic since released, I suppose. Also some rumors or discussion out of Beijing about the possibility of constraining releases. What do we know about that? Maybe what's the most up-to-date view from Beijing, I suppose? You've been paying attention to some of the events there, some of the very public events, including the World AI Conference and also Xi Jinping's big speech.

Graham Webster:

Yeah, so it's been a big week for Chinese government communication on AI and AI policy. Last weekend in Shanghai, the annual World AI Conference was held, and for the first time, China's top leader, Xi Jinping, gave the keynote address. Last year, I think it was the premier, so the number two official, but there's always somebody big. But this is the top guy. And so he gave, really, the most visible top leader speech on AI that I can think of, and I've been watching Chinese AI policy closely since 2017. A lot of people were looking at this because there had been reporting that China might try to export control its AI technology, and maybe that would include preventing the transfer of the weights of open models to foreign users. It was always a little confusing if you could have an open model that was only available in China.

It recalls these debates about encryption and the early internet where the US wanted to control some special math and people were printing it on t-shirts and stuff like that to show how silly this was. But it was totally possible that the Chinese government could crack down and say, "We're going to control the distribution of models at a certain level of performance." So a lot of people looked at the Xi Jinping speech and said, "Okay, well, he's actually talking about continuing openness and distribution." And the word, open source, was even said. I think the signals are pretty unclear. There's a very strong Chinese government pitch to the world that they are the ecosystem that is open, that's going to help, especially global south countries, but also others, develop AI without being reliant on centralized providers in the style that the US labs are doing.

But the speech and all of Chinese policy also emphasizes that there's need for security and safety, and there's this longstanding Chinese watchword of secure and controllable technology. And it's utterly conceivable that some capability would be demonstrated, that the Chinese government would decide, "Okay, this being completely open for anyone to use in the world is not compatible with our secure and controllable style." So for now, the signals are that China's still going to be releasing models openly. The Kimi K3 model came out too much attention internationally with apparently very strong performance alongside that conference. And the models just keep coming. But I wouldn't bet on it going forever.

And the last thing I'll say is there's an obvious common ground for US and China here, whether the government's appreciated or not, which is that for third-party misuse, cyber criminals, terrorists, other types of risks, or even really for using it against each other, both governments and both countries have to worry about cyber defense in a landscape where AI capability can supercharge offense as well as defense. So they may be able to reach a meeting of minds, and there's a kind of quietly developing new channel on AI between the two countries that we don't know a lot about.

Vinh Nguyen:

I want to add what Graham said. I think the Chinese may end up have to find a way forward if there is a major incident that will be quite disruptive. Say if K3 is released, I can just see that there's so many cyber criminals out there. And I'm sure that both the Chinese and the US government agree that they don't want cyber criminals going after their own citizens. And so there will be a case, a likely case scenario where cyber criminals can run these things autonomously, and jeopardize, and harm both the Chinese and American citizens. And the reaction may end up with... You see what happened with all these Chinese model aiming after US citizens, or if it takes out some infrastructure in China. And so I think, hopefully, there is an incident where the two governments can agree that maybe we have to sort out this open-weight debate, but I don't think they will agree to anything unless there is a major incident that will have to force their hands.

And so I don't think the strategy of open-weight and just let them all go, at this level of capabilities and this level of diffusion, will bode well for really anyone outside of the criminals. My take is my cybersecurity community keep on debating, "We can use this for defense, we can use this for offense." My take, people, is that criminals can use these things faster than the defenders can. And defenders have to put the resource to defend a very large attack surface across the enterprises and things that they have to protect. Criminals don't have to do that. They can actually just run these and be able to get to what they need. And so my take is that it sounds theoretically right that, yes, it can benefit both sides, but I think it will benefit the cyber criminal first until you get to a level of consequences where you decide that maybe we don't tolerate that anymore. So my take is that in the next few months, it wouldn't be surprising that there are more cases coming out, that are quite consequential, that we have yet heard.

Justin Hendrix:

Maybe, Vinh, I'll stick with you just on a question to follow up on that. You spent decades in the national security establishment. What do you think are the kind of dilemmas or the kind of decision matrix that people in the Trump administration, right now, are facing on this question in particular? Is there a détente we can reach with China or is there any indications that we will just carry on with this race mentality? Which I want to come back to you on Graham as well, 'cause I know you have this way of complicating the idea of who's in the lead. But Vinh, I don't know, what do you make of what's going on at the top rungs of the administration, what we can see from the signals that are coming from those individuals?

Vinh Nguyen:

I worked through the US-China cyber discussion for more than a decade. I can tell you that trust is quite low from the beginning, and so you have to really trust the other side, which is unlikely at this point. The second is making sure that we understand what we are talking about based on the definitions that we use. I think the Chinese government and the US governments may agree on controllability. And as Graham, he may tell you more in term of maybe the ambiguity in controllability of AI. And so I think there has to be an incident that is consequential, that will force both sides to come to a discussion and to have a meaningful conversation. And I think the vulnerabilities that could be discovered in the global financial systems and all the system that glue the world together could be a point of discussion on how China and US can work together.

Areas where we can find ways to agree, understand how we measure and how we understand about risk. No one wants the proliferations of dangerous weapons that AI can augment bad people, criminals, to do. So there's a lot of things that we can agree on if we try. But it has to come from some level of agreement of what we need to do, which is forced by an incident and building the trust, little by little, to get that right. But if the US and China don't come together on this, I think it will be impossible to have any global conversation on the governance of AI at this point.

Justin Hendrix:

Well, then I don't love that. Some kind of Mr. Robot style destabilizing hack or something like that, that messes everything up. Doesn't sound awesome. And, Graham, bringing you in, what do you make? What brings people to the table? Or how might we complicate even this notion of what it means to be in the lead in a way that could be productive?

Graham Webster:

Yeah. Well, first, I'm going to go out on a limb and endorse a piece of writing that I haven't read yet, although I've heard some of the arguments from the author before publication. Matt Sheehan from the Carnegie Endowment for International Peace has a new commentary out on a concept he's been working on called AI Safety in Parallel. And I hope I don't do violence to it by summarizing what I've heard him talk about, but the idea is basically it's going to be really hard to get these two to trust each other and really make a deal. But there may be substantive measures that both governments and AI ecosystems can take that actually mitigate some risks through various types of contact. And there's track two dialogue and there's lab-to-lab, and there is some hope there. And so I'll stop talking about something I haven't read yet, but I recommend people read it because I know that Matt's been working quite hard on that.

Now, as for the AI race thing, one thing that I think was really helpful about the way that Vinh has been laying out the cybersecurity risks is that I think a lot of the time we hear about, "Well, our country must be ahead in AI so that we can stop the other country from hurting us with their AI." And for me, this is sort of a strategic miss because the reality for several years has been that if the United States or China was determined to really harm the other in cyber, they both had the capability to do it. US offensive cyber and Chinese offensive cyber could wreak significant havoc on the other side. But you'd be in the conditions of war. That would be very bad. There's all sorts of activity below the level of war. But really, if people wanted to throw it at one another... These are highly capable offensive cyber apparatus.

The common risk that advances is third parties, whether it's criminals, or terrorists, or just anybody who previously wasn't enabled to cause significant havoc. I think that's really important. But I had this idea the other day, what is it really? What is the US lead? Is it just that there's a couple of models that people talk about, "Well, they're six months ahead," or, "They're nine months ahead," or, "18 months ahead," or, "Three months ahead of the best Chinese model on some metric"? So that means that there should be some little slice of what AI can do, or really, what a person or an organization can do with AI today, that can only be done with that more advanced US model. But that got me thinking there's actually a bunch of stuff that you can only do with an open model that you can install on your own infrastructure and that you can run without the permission or observation of the model provider.

So in a way, you can imagine that there are exclusive capabilities that US provides. Currently it looks like yes, still, there's... At the top end of performance on various metrics, there's an edge there that the US labs have. And that will solve some problems that people can't solve with the Chinese options. At the same time, no US model is providing that sort of near frontier level capability in a way that you can download into your own corporate network or into your own government operations network, and run without any interference, or observation, or data risks from the provider. So China, then, has this lead just in providing, really. It's the world has this open lead in certain things that the closed US models won't provide because... The point of talking about this is that the idea of a single AI race really falls apart unless you define what it is.

It's a race to do what? And often, the implicit destination is the vague idea of AGI or super intelligence. And I just don't think that's enough. I think analytically when we're thinking about the situation between US and Chinese models and what open-weights models mean for the business model of US companies, this is all a specific question of who can do what, and for what cost, and in what sort of logistical arrangement using the different technologies? And in that situation, it looks like a much more complicated landscape where some things are going to work better with a Kimi than with a Claude if only because you can afford the Kimi.

Vinh Nguyen:

Can I jump in, Graham? I love what he said. It's really about the outcomes. I think American people want... If there is a race, the race, the outcomes would be able to advance life sciences so we can cure cancer, and fight diseases, be able to improve our national security and defenses, and be able to maybe accelerate nuclear fusion so that we can lower prices. These are the race that I think race to the top to really better everyone and humanity alike. But absolutely I do not want the US to beat China in domestic surveillance, which China's really good. So let's not compare ourself on things that don't really matter. And I think the whole discussion lacks any outcome and lacking any outcome. I don't buy anything because it's so ambiguous. And so I hope that we don't discuss about some crazy races, 5 or 10 races, as long as the races are to deliver something better for humanity.

Justin Hendrix:

Then I want to ask you a little bit about this most recent piece you have for CFR. You're responding to these reports that the administration is potentially preparing to announce something, maybe a new regulatory body, a FINRA style body, perhaps, that would make various judgements about the safety of AI models before they're released, and also engage with the industry on these questions. You're obviously hoping to see that if something like that does come along, that it's structured in the most appropriate way, at least from your perspective. I don't know. If the administration does do something like that, what should it look like?

Vinh Nguyen:

Yeah, thank you, Justin. Really, the background is Demis Hassabis, who's the CEO of Google DeepMind, proposed a self-regulatory organization to provide a level of stability and predictability in the ecosystem. It is very hard for the frontier AI companies, and the hyperscalers, and the tech companies to guess what models could be released, which model could suddenly be reported to be so dangerous that they cannot deploy. And so I think that is quite disturbing to the US markets, just the whole ecosystem who are developing technologies in order to do useful things. My work here is with Elham Tabassi from Brookings. She was the former NIST Chief AI Advisor along with my colleague, Kat Duffy, at the Council, who's leading the lead AI efforts. Three of us, we do really care that whatever is proposed, that we want this to be a long-term success and has the support of not only just the industry, the governments, but also the people who are benefiting or will be impacted by it along with our allies.

Because our thinking is that if we rushed into some institution, we will build it, it will take years to do it, and then suddenly, it fails because we failed to design it from the beginning and try to resolve the problems. It's so much easier to fix problems that we can identify today than try to fix it a few years down the road because there will be many interests and there's a lot of things that we have to do. And we're to undo the whole thing, and that will cost a lot of time for us to master a technology and govern a technology that is so meaningful and so powerful for all of us. And so what we are really looking for is that whatever the design, the regulatory, the legislative umbrella of this is, as long as we have a level of independence in the evaluation and benchmarking so that the evaluated is not setting the terms on what to be evaluating on, basically no one will trust that, unfortunately.

And I don't think the labs, all of them want a level of independence in term of third-party auditing. The second is the national security. No longer can we just somehow talk about the commercial and the economic progress, and then somehow we have some national security apparatus sitting on the side, not connected. And that is how we end up with disruptive decision-making, because one side will then talk to the other side, and suddenly, we have this strange erratic outcome. So the national security community should be involved right from the beginning to address any national security concern.

And lastly is really the legitimacy. If we are going to produce American AI stack, that need to be marketable and trusted with our allies, with the people who want to sell the products too, it cannot just be run by us. And somehow when we sell it to the Europeans, or the UK, or the global south, suddenly they're like, "Oh, it's a US product, so we trust it." No, it has to be much more independent in term of trust, but verify many times in term of the reliability, the security and the safety of the so-called product.

And if we don't demonstrate that level of transparency and accountability to our allies, no one will trust us and they will trust things like Chinese products that they have a control of, but they don't have a way to verify or assure it. So that's really our concern on how to address it. And we list up a variety of things to ensure that we can undo, unpack who is going to set the standard, who is going to set and fund the infrastructure to do such evaluation, and how do we fund and power the independent evaluators who are going to evaluate such product on the standards. So we can do it right from the beginning. We will save ourself a decade of trouble ahead.

Justin Hendrix:

Well, I want to double down the international dimensions of the question about how to do this. And, Graham, maybe bring you in, because another thing that feels like just hot news coming out of China, the launch of this World Artificial Intelligence Cooperation Organization, which I think wants to be the alternative, wants to be the standard setter, and maybe answer some of the questions that whatever a US body might think or its remit, it would like to do that, but in a more multilateral way, perhaps, at least from the start.

Graham Webster:

Yeah. I think you have to read the initiatives like this from the Chinese government with both a genuine lens and a cynical lens. Often in Washington, you just get the cynical lens, so I'll give both. But I think there's a genuine desire in the Chinese government and among the top leadership to be seen as an enabler of development around the world and to share the kind of development miracle that China feels that it has had in the last several decades, and that has certainly been visible to anybody who could witness China 40 years ago versus today. I think that's genuine. I think they really want to do that. Of course, it also comes with being able to sell your products and having political influence, but I think this idea of being a leader of the developing world is an honestly held motivation, and it's something that Chinese leaders and Chinese people are proud of.

And sometimes they can be justifiably proud when products actually do provide public goods. The cynical side is more familiar to an American audience, is that it would be a way to exert influence, and displace US influence, and displace the influence of small-d democratic and human rights-respecting systems. The World AI Cooperation Organization, it's a little hard to tell what precisely it is. It was announced that, I think, 29 countries had signed onto it. The idea that this organization would exist was announced a year ago at the previous World AI Conference, which I happened to attend, and it took them a year to announce the initial signatories. I think as we're recording this, the document that they signed is still not released, but the organization's meant to be headquartered in Shanghai and it's supposed to be about sharing the wealth and also about developing international governance and norms for AI use.

Now, the signatory list did come out and it's kind of a negative space of countries that the United States would work with. And I don't have the list in front of me, but suffice it to say it's not grabbing a bunch of NATO allies or Five Eyes type countries that are core US partners. And every now and then, there is a Chinese initiative that does include some of those US partners. So what you end up with is a Chinese-led club and then the potential for a US-led club. And I guess for development purposes, this could be okay. I mean, you could just have two separate groups of people trying to do good. But if you're trying to solve problems and do control of risks and governance, I really believe that no really effective governance or risk management framework for the sort of LLM era of AI can really work without both the United States and China somehow signed on.

And maybe they can do it in parallel, maybe they don't have to sign the same proposal, but without including both of those, you're missing at least one of the two countries that produces really powerful models. And so that's just super crucial. And I think that we can look at those diplomatic initiatives as potential venues for the diffusion of good ideas, and of course, all the cynical narratives about spreading influence and trying to get business for a country's companies. And I'll throw that accusation at the United States too. I mean, remember that when we spoke last year, Justin, the Chinese conference had just gone off and the US AI plan, I forget the exact title, had come out, and the US plan was boasting of building AI dominance. And this is not a very friendly pitch to the world. It's saying, "You shall buy our stuff, and by the way, follow our rules." The Chinese pitch is a little more friendly, but neither will, in my opinion, reach a global problem-solving modus operandi without breaching the divide.

Vinh Nguyen:

Yeah, my take is that trust is broken everywhere, and folks may not trust the Chinese government or the US government. And so in order to operate in a world where trust is quite limited, all you have is really... If you don't have the capabilities, you cannot control it, you have to have an ability to verify and you have to have an ability to have a trusted ecosystem for that verification that maybe governments are not involved. And I find that it is truly a strategic advantage for the US Think of our financial systems, even though it's not flawless here, but at least in some ways, we learn to rely on the global ecosystem of the financial services to validate, to verify financial asset.

And so there is some solution. I am optimistic, but I think as soon as we get the governments involved, the geopolitics will show up, the national security work will show up, and then people will look that through that lens. So if we don't have that, is there a different alternate ways that we can build trust in the ecosystem and be able to allow people to verify 'cause they don't have to trust what we have to say, but at least they can verify and verify it independently? That's my thinking, is that that may be a better outcome compared to the alternatives which we have the two big powers fighting each other on every little thing.

Justin Hendrix:

Well, I want to ask, and this may be a little ham-handed, this question, but I know that even this afternoon, there is a letter out from American AI firms about open source and open-weight models, and I think also trying to basically influence the administration and maybe the discourse that's going on inside it about what to do about these issues. But US, China, obviously concentrations of power, but American industry, its own concentration of power with its own voice... I don't know, Vinh, how do you think about that right now? What the industry wants, the dynamics of the industry, the, I don't know, financial situation that the industry's in? And, Graham, I know you're watching that as well from a question around, is this a bubble? Is it not? Are these investments going to play out? And clearly, there's a lot of commercial considerations that are going on here as well.

Vinh Nguyen:

I find this era and timeline quite dramatic, and ironic, and just fascinating. All the major powers that we know of, the Wall Street, the big tech, the big things, they feel powerless. And I'm like, "Really? You should not be." And so when you read these through the lens, the assumption, maybe the underlying assumption and maybe the implicit assumption here is that the three frontier labs, companies, be it Google DeepMind, Anthropic, OpenAI, the entire industry assess that they have so much power concentrated from lot of industries that they will take our business away. They will take so-called alpha... This is what we hear all over the place, away from the business that we run. So they are frustrated, upset, and they want a check and balance, a counterbalance to the proprietary models, and therefore, support open source and be able to do more of this work.

I think the other charge that I don't think is fully articulated or clear enough was that the industry is very concerned that the labs will somehow take their proprietary data, train something, and then out-compete them. This is a valid concern, and there's no way that you can trust giving a data to a lab and trust that they don't use that data for whatever purposes they want. There are contracts, there are things, but there's a lot of concern. So my take is that this is a way to pull back some powers that they see as concentrated in just a few hands. And this is a interesting dynamic that I think we should see it for what it is rather than being critical that somehow this is a ploy, a play, a hype, or something else. But I think this is truly a concern from industry on the shifting dynamics, and power, and opportunities for the future economy.

Graham Webster:

Yeah. So I'll jump in with a couple of things here. I mean, like Vinh, I just find this to be kind of a dazzling, confusing moment in what everybody's arguing, and what's going on, and what the US government might do. I made a little list of tools that the US government could potentially use according to various news reports and official statements to restrict access to Chinese models. They could maybe put the Chinese companies on the entity list, but I think that's mostly about exporting things to those companies, so I don't know how that works. There's a thing called the ICTS rule where the US can ban transactions on ICT services, but that would really be hosted providers only. And if I download it on my own infrastructure, I'm not sure if that touches it. There's the TikTok law, which is again, you could go after the app stores. There's putting sanctions on the companies and making it impossible to transact with them, and maybe there's regulatory risk for US businesses, but I could probably still do it in my garage.

And then there's just this sort of warning and saying, "Oh, there's this security risk, which may be real or maybe not, but maybe to try to deter people from using it." All these tools are being debated. And behind that, there's this question of why would the United States crack down on Chinese open source models? And I can see two basic arguments. One is that all open-weights, sorry, I try to say weights rather than open source, all open-weights models are risky at a certain level. I think that's a kind of AI safety philosophical standpoint where they say, "Look, the harms are just too big. We can't have these things out there. Bad actors will use them. They shouldn't be released." That's a principled stance and it aligns with, I think, what Anthropic says and what a lot of researchers say. But it's also convenient because Anthropic's revenue, and OpenAI's revenue, and DeepMind's revenue is challenged by cheaper token costs from some of the open-weights models that can get a lot of work done for less money.

And then there's, "Should we ban just Chinese open-weights models?" And that's the argument that some people are making, and that's just an explicitly competitive argument. And so I keep coming back to, "I am not going to be your market analyst here," but going around in the Bay Area meetings and reading stuff online, there is this completely saturated consensus right now that the AI bubble, not is there one? The AI bubble has got to give. And the only question is when. Now, I get really nervous when there's such a broad consensus, but I find it frustrating that we don't get to evaluate the actual risk of open-weights distribution, which I think is real, versus the real benefits also that people get out of it, out of the inexpensive utility, the ability to control the data environment, et cetera. We don't get to have that debate in the United States without also having to ask, "Is this really just about trying to prevent a bunch of people's startup values going to zero?"

And the answer is we just can't have that. They aren't separate. These things are happening at the same time. There's no way around it. And I find that amazing and frustrating, but I guess I would urge everyone to keep in mind that when you're hearing debates about going after some Chinese AI company or some product, you have to think about real risks, and then you also have to think about ulterior motives. The Chinese government, I think, can be seen to capture the Chinese companies. The companies can't really do anything big without government approval, or not really approval, but the government can tell them to stop. It's not a prior approval, but there is the ability of the government to intervene.

I think in the US, we have to worry about which parts of the industry are capturing the government because the government doesn't know what it wants to do independently on this, in my opinion. They're hearing different pleas from people on different sides of the debate who have vested interests, and some of whom also have honestly held philosophical and security beliefs. So yeah, call me confused, but it's definitely not boring.

Justin Hendrix:

I believe we'll have to end on a point of confusion, and perhaps with this particular topic, that was always going to be. But I think, at least from my perspective, listening to the two views brought me a bit more clarity, at least on the questions I should be asking going forward. So I appreciate you both. Vinh, Graham, thanks so much for joining.

Vinh Nguyen:

Thank you, Justin, for having me.

Graham Webster:

Yeah, thanks for having us.