California Becomes Frontline in Battle Over AI Companions

Audio of this conversation is available via your favorite podcast service.

California could soon pass the most significant guardrails for artificial companions in the nation, sparking a lobbying brawl between consumer advocates and tech industry groups.

Advocates have largely rallied behind AB1064, which would restrict minors’ access to AI companions unless they meet certain safety criteria, while several industry groups have embraced SB243, a more limited proposal requiring that companies make it clearer to users that they are interacting with AI.

After clearing the state's legislature, the bills await a decision from California Gov. Gavin Newsom (D), who could sign or veto them.

In a recent report for Tech Policy Press, associate editor Cristiano Lima-Strong detailed how groups are pouring tens if not hundreds of thousands of dollars into the lobbying fight, which has gained steam amid mounting scrutiny of the products.

I spoke to Cristiano about the findings, and what the state's legislative battle could mean for AI regulation in the United States.

This reporting was supported by a grant from the Tarbell Center for AI Journalism.

What follows is a lightly edited transcript of the discussion.

Justin Hendrix:

Hey, Cristiano.

Cristiano Lima-Strong:

Hey Justin. How are you doing?

Justin Hendrix:

Doing okay. Pleased to get to talk to you today about this piece that you published this week. This was the result of a special effort. You took a trip out to California, you did a lot of research on this. Walk me through your reporting process for this piece.

Cristiano Lima-Strong:

Yeah. Sure. So we've been interested for a while in AI Companions and the new legislation that we're seeing come out on this. Certainly there is a lot of attention in Washington and discussions starting to pick up about the safety of these tools. There's been a number of reports coming out in recent months tying tools such as these to self-harm, instances of them actively encouraging users, in some cases, very young users to engage in self-harm or other detrimental behavior. And so there's been this response that we've seen more and more AI safety legislation picking up. And so we've been monitoring the situation and recently the fight in California over two particular bills really started to pick up. And so when we saw an opportunity to go out there and track some of the lobbying, the negotiations going on over these bills, we jumped on it and that's what led to this piece.

Justin Hendrix:

Take us inside it a little bit. What's it like in Sacramento right now? What goes on? What does it feel like when the legislature is in session and negotiating over these types of laws?

Cristiano Lima-Strong:

Yeah. It's wildly different from what it often feels like in Washington in some ways. These bills have been introduced within the past year or so, and they go through a similar committee process where they're taken up, their hearings held, they're voted on, but in the grand scheme of things, they advance quite quickly as do a lot of state legislatures compared to bills in Washington, which often, especially in the tech space get introduced and then languish for years or there are discussions back and forth. Things in California and particularly around tech can materialize quite quickly. And so we saw these two bills. In the past there have been bills around social media safety, around privacy that have taken a few different sessions to get through, but we saw these move fairly quickly and in advance. And so just in the past month was the end of California's legislative session where for the two bills that we were looking at, the upper chamber had approved one, the lower chamber had approved another and they had to swap and debate them and then kick them back for final approval. And so we're now at the stage where they've both cleared both chambers and are on the governor's desk awaiting a potential signature or a veto.

Justin Hendrix:

So those bills in the lower house, as you mentioned, AB 1064 and the Senate SB 243, can you give us the basics of what these laws do?

Cristiano Lima-Strong:

So AB 1064 is seen as the more stringent measure. It's the one that a lot of child online safety advocates and tech critics have embraced. Basically it would ban companies from offering AI companions to kids unless that they are "not foreseeably capable of engaging in an array of harmful conduct, including encouraging users towards self-harm, engaging in sexually explicit role play." And so that bill, as I mentioned, it's the one that a lot of sort folks in the consumer advocacy space have rallied around to get past. The other main contender dealing with this issue, SB 243, that's more of a transparency and disclosure measure. Basically, it would require companies to provide an active disclosure to users when they are interacting with an and companion in instances where they might not know that that's what's going on. There's been a few measures like that that have passed in other states, whereas 1064 the child safety advocates believe it would be the most significant rules to go into effect if it is indeed signed.

Justin Hendrix:

So you introduce us to a cast of organizations and other characters in this piece who are involved in lobbying efforts around these bills. Can you just set the field for us a little bit? What does it look like? How many organizations appear to be contesting this space and how would you characterize what it looks like in terms of the competition over these bills?

Cristiano Lima-Strong:

So we're about a half dozen to a dozen organizations. I would say are very actively involved and there are some nuances between them, but speaking broadly, there is the consumer advocacy camp that, as I mentioned, has been supportive throughout of AB 1064, which is seen as the more protective, more sweeping measure. A lot of those groups had also been supportive at the onset of SB 243, and their support has waned to the point where in recent weeks a lot of groups actually withdrew it altogether. So that's where they come down on those two bills. Then there are an array of tech and business groups. So trade associations such as TechNet, the Chamber of Progress, CCIA, and this newer group called the American Innovators Network that's backed by Andreessen Horowitz and Y Combinator that represents, as they say, little tech, some smaller tech companies. Those camps started out vehemently opposed to both of the measures, but some of them have come around to support 243 as a number of the restrictions in that bill have been paired back over the process of getting them to the floor and passed out of both chambers.

Justin Hendrix:

This campaign, on both sides, it looks like it's not just hand-to-hand lobbying, you're also seeing some digital advertising and other forms of influence.

Cristiano Lima-Strong:

Yeah. Absolutely. So particularly TechNet and the American Innovators Network ... TechNet I should mention like CCIA and the Chamber of Progress, their members and their funders are a lot of the major tech companies as well. Some smaller ones. The Amazons and Apples of the world. And so the TechNet and American Innovators Network in particular were more aggressive in some of their public messaging, taking out tens of thousands of dollars, potentially more in ads on Instagram and Facebook, particularly taking aim at negative ads targeting AB 1064. And part of what they've argued is that ... In their letters to lawmakers, they've said that the definitions are too broad, that they would catch a whole bunch of AI chatbots, AI tools that aren't strictly companions, which is the intended scope of the two bills. And you've heard a lot of the same rhetoric that you've heard around some of the other AI bills being proposed in the state, that it would chill innovation.

But particularly what we've seen when it comes to 1064, which again focuses on young people using these companions is warnings about it's going to hamper innovation in schools, it's going to harm education. There were some [inaudible 00:07:47] taken out a warning about a potential tilling effect around the use of AI companions in healthcare. And so touching on core bread and butter issues that people care about and saying that these bills would have a harmful effect in those spaces as well. And we've seen that in addition to the traditional lobbying that you've talked about.

Justin Hendrix:

And while I think your piece does a good job of trying to provide a balanced view of what the different parties are arguing and to the points over which they're contesting these bills. Is it a fair fight? Do you see any asymmetry in the resources at play in this particular legislative skirmish?

Cristiano Lima-Strong:

Well, so if you look at sheer numbers, for example, on lobbying figures ...I looked at a handful or a few groups from each camp that were the most active on this, and the tech groups registered significantly more lobbying in the first half of the year. Now they're also active on a whole bunch of bills. And so we don't know precisely dollar for dollar how much each camp spent, but certainly they have infinitely more resources than the consumer advocates have in terms of bringing this fight to bear and pushing for changes in the legislation on the ground. One of the groups that was active in this fight, Tech Oversight California that just recently launched, it's a branch in the state. This was their first year actively lobbying in the state, whereas a lot of these tech groups, they've had a ground game in California for over a decade. And so certainly not a one-to-one in terms of firepower.

Justin Hendrix:

And of course, this is the homeless Silicon Valley. It is the backyard of many of the tech firms and the venture capitalists whose interests are represented in this fight. Talk to me a little bit about the governor, the role of the governor. You were able to uncover a little bit of evidence about his role behind the scenes in some of this. Tell me a little bit about that. And also, I don't know what we know about what role he might play in the next stage of this.

Cristiano Lima-Strong:

Yeah. Well, so this is the big question now because the legislature passed both bills. Passed the one that's backed by consumer advocates, and it passed the one that industry groups had embraced. And so now they're both sitting on Governor Gavin Newsom's desk and he could choose to sign one or the other, which of course would draw backlash from the other side. He could choose to sign both. There's aspects in which that could potentially raise legal issues because they do have some overlapping features. 1064 is mostly aimed at minors and at youth, restricting their access to AI companions, whereas SP 243 is targeted at use by all consumers, but it has some expanded protections for minors. But what we found out through our reporting is that some of the changes that were made to SP 243 were made in consultation with Newsom's office.

And again, these are changes that caused a lot of child safety advocates and consumer advocates to balk and to revoke their support. And that led to a lot of industry groups to start to embrace 243 as an alternative to the more stringent 1064. And of course, some of the context here in the background here is that last year in California, there was this huge battle over SP 1047, which was at the time the most sweeping proposal to restrict AI models that we'd seen in the state, and it passed through the legislature, and ultimately the governor decided to veto it, which is something that industry had called for. And so this is setting up as another test of the governor. Is he going to choose to take up the bill that's favored by consumer advocates and child safety advocates, or is he going to side with the one that tech industry groups have backed as he did last year with 1047?

Now we know that he consulted on 243. If you're a supporter of 1064 and you hear that you're probably not feeling as great about your chances, but there are some other factors here. We touched on in our report that the governor's wife, the first partner, she is been an ally to some of the groups in this space that have been advocating for 1064. She's participated in a number of panels with Common Sense Media, which has been one of the most outspoken proponents of 1064, and has given some encouraging remarks about their efforts now. She ultimately does not hold the pen to make the signatures and approve these laws. It comes down to the governor. And so we'll have to see how far does that alliance go and where is he going to come down on this fight? And will it be the same as last year with 1047?

Justin Hendrix:

Now, you point out that California wouldn't be the first state to enact new laws on AI companions. You point to New York where I sit, where Kathy Hochul has assigned a law that requires AI companions to deploy a safety protocol when users discuss self-harm, suicide. What can you tell us about the way California lawmakers think about their role in terms of setting precedent for the rest of the country? A lot of states look to California, the federal government looks at California concerns itself with California, and whether it's laws become a standard as they have in other contexts, do lawmakers there think about that when they're having these debates or are they mostly just focused on their own politics?

Cristiano Lima-Strong:

Yeah, absolutely. I think they are. I spoke to Assemblymember Rebecca Bauer-Kahan, who introduced 1064, and she said as much, she said basically, Congress doesn't seem to be acting on this anytime soon, so there's effectively not protections on the books. So she said she'd love it if other states picked this up as a model. States under Steve Padilla who is behind SB 243 said something similar. And he particularly talked about the urgency behind this issue and wanting to have something on the books and protections to spread. We have seen also instances in the past of companies changing their practices at times nationwide in response to rules that have passed in California. And so I think that that's something that legislators in California think about, and they very much take pride in being leaders in passing rules that other states might replicate. But it's still an open question whether this is going to happen here.

As you alluded to, there have been a number of measures passed in other states with AI companions. Several of them are very similar to SB 243. So focusing more on transparency around potential deception if someone isn't aware they're interacting with AI. 1064 would be more of laying a marker in the debate that I'm sure if they were to pass advocates would push for other versions in other states, but we'll have to see if that's how it unfolds. California in the past decade, of course, passed its landmark privacy law, and there was a lot of debate at the time of, is this going to become the standard in the US. We haven't really seen that pan out. We've seen a separate model that industry groups tend to prefer instead serve as a template for a lot of other states. We saw California pass, of course, the age appropriate design code dealing with harms to kids online. There have been versions of that that have passed in other states, but we haven't necessarily seen it spark standards throughout the country. And so we'll have to see if these bills are able to make it through the finish line, how significant is it actually going to be in terms of inspiring lawmakers in other states to take up these causes.

Justin Hendrix:

Just two last questions for you. One is whether there was any awareness or discussion there about the federal government, there's been this proposal to potentially try to preempt all state laws on AI, or I suppose to CC enforcement of them through a moratorium. Any discussion about that? Any chatter about what that could mean for these types of efforts?

Cristiano Lima-Strong:

Yeah. I was coming in expecting to hear more on that, but I didn't much honestly, and I asked assembly member Rebecca Bauer-Kahan directly about this, and she said that it wasn't really something that was on her mind. It is a different process in states. Something that State Senator Steve Padilla talked about was measures that can be amended given that the California legislature passes dozens, hundreds of measures in a given session. They're also not afraid to go back and tweak things. So it's a bit more of a living process. And I think also we've seen that California as a state is also not afraid to challenge the administration and to go to court to defend its laws. And so if we were to see state preemption of AI legislation go through, that could tee up a battle for states, and you'd have to imagine that California would be at the forefront of that. Having said that, I think there's also part of this where given that some of these measures deal with child safety protections, they also have allies in Washington. We've seen Senator Marsha Blackburn, a Republican from Tennessee, for example, raising concerns about how moratorium would impact potential laws potentially like these. And so I think that that's probably of comfort to legislators as they're trying to pass some of these measures as well even knowing that a moratorium or some preemption could eventually happen.

Justin Hendrix:

Anything on the cutting room floor or anything you wish you had more time to report on or to get to just generally?

Cristiano Lima-Strong:

So we published this report shortly after these two bills made it through the legislature. And so certainly a lot of the lobbyists I was talking to, the groups are active on this. I said, "Oh, is this a break for you now?" And of course not now because they need to turn their attention to the governor and getting their message to the governor's mansion. And so I think probably as we're speaking, all of these groups are mobilizing to try to persuade the governor to go one way or another. And I'll also note a lot of lobbying that we talked about happening, a lot of the disclosure filings are so far only available for the first half of the year. It's likely that probably the most significant activity that happened on this hasn't even been made public yet. So I think it's something that we could come back to in a few months' time. And once we've seen the governor's eventual decision, we'll have a clear picture of how we got from point A to point B.

Justin Hendrix:

Cristiano, thank you for this report. Thanks for taking the trip to Sacramento.

Cristiano Lima-Strong:

Yeah, thanks so much.