Considering the Ethics of AI Assistants

Audio of this conversation is available via your favorite podcast service.

On May 14, 2024, Google hosted I/O, the annual event where it promotes its products. In his keynote address, CEO Sundar Pichai said AI assistants are core to the company’s vision:

...I can see you all out there thinking about the possibilities, but what if it could go even further? That's one of the opportunities we see with AI agents. Let me take a step back and explain what I mean by that. I think about them as intelligent systems that show reasoning, planning and memory, are able to think multiple steps ahead, work across software and systems, all to get something done on your behalf and most importantly, under your supervision. We are still in the early days and you'll see glimpses of our approach throughout the day, but let me show you the kinds of use cases we are working hard to solve. Let's start with shopping.

Just a couple of weeks before Pichai took the stage, in April, Google DeepMind published a paper that boasts 57 authors, including experts from a range of disciplines from different parts of Google, including DeepMind, Jigsaw, and Google Research, as well as researchers from academic institutions such as Oxford, University College London, Delft University of Technology, University of Edinburgh, and a think tank at Georgetown, the Center for Security and Emerging Technology. The paper speculates about the ethical and societal risks posed by the types of AI assistants Google and other tech firms want to build, which the authors say are “likely to have a profound impact on our individual and collective lives.”

The paper considers the potential nature of the technology itself, giving a broad overview of these imagined AI assistants, their technical roots, and the wide array of potential applications. It delves into questions about values and safety, and how to guard against malicious uses.

Then, it takes a closer look at how these imagined advanced AI assistants interact with individual users, discussing issues like manipulation, persuasion, anthropomorphism, trust, and privacy. Then, the papers moves on to the collective, examining the broader societal implications of deploying advanced AI assistants, including on cooperation, equity and access, misinformation, economic impact, environmental concerns, and methods for evaluating these technologies. The paper also offers a series of recommendations for researchers, developers, policymakers, and public stakeholders to consider.

I had the chance to speak to two of the papers authors about some of these issues:

• Shannon Vallor, a professor of AI and data ethics at the University of Edinburgh and director of the Center for Technomoral Futures in the Edinburgh Futures Institute; and
• Iason Gabriel, a research scientist at Google DeepMind in its ethics research team.

What follows is a lightly edited transcript of the discussion.

Justin Hendrix:

I am pleased the two of you could join me today to talk about a document that came to my attention a couple of months ago. This is a document called The Ethics of Advanced AI Assistants. And in many ways, this is more of a book. But rare to see a paper that's 200 odd pages and 50 odd pages of citations. And I thought we'd take the opportunity to talk about some of the key ideas in it today. In particular, the ones that the two of you have contributed most significantly to. I thought we might just start with a little bit of definition. What are we talking about when we talk about advanced AI assistants?

Iason Gabriel:

Yeah. Thank you so much Justin. I think a little bit of wider context here might be useful. And I'm sure as many of you're familiar, people have started to talk about an agentic turn in the context of AI research and AI systems. So the big picture idea is that while we've been interacting with conversational AI or chatbot type AI, in the not too distant future, these systems could have additional functionalities that enabled them to use tools and to take actions in the world. And then an interesting question is, given that potential, what's the most likely form of a AI agent that we will interact with on a daily basis or see deployed? And so this paper investigates the idea of an advanced AI assistant, which is understood as a kind of agent, but one which helps the user perform a wide array of tasks across different domains and crucially has this ingredient of autonomy. So it's able to do a long sequence of actions without direct interference or interaction with the user.

And the idea is that could potentially be useful as people often think a personal planner is the first port of call. But if we look at examples of how these digital agents are being used now, we also see them being used as companions, as educators, potentially as research partners. And so this model of interaction where AI is interacting with us and then interacting in the world opens up a huge range of possibilities which we wanted to explore.

Justin Hendrix:

I get the sense just listening to various Silicon Valley leaders that not only is this where we're at in the moment of the technology, it's also where we see most of the large firms, including Google want to move. They would like to see these systems get deployed more broadly across the internet and across various other technology platforms. Shannon, how do you contextualize this moment we're in as a student of the history of artificial intelligence and where we're at with this technology?

Shannon Vallor:

I think if you understand the history of AI as being shaped by human imagination and our visions about what AI would be going back 100 years or more, it's been the case very often that people envisioned AI as eventually being agentic, being something that would be another acting being who would either be cooperative and helpful to us or potentially competitive and threatening. And these visions of AI are very much independent of how we've built systems until today. Because I think it's really only now that we see the possibility of people taking AI seriously as something that isn't just a tool that you use, but that it's another agent that you interact with. We've had chatbots before, but their functionality has been so limited and we so quickly run up against the obvious barriers to real communication that we know very quickly that we're not dealing with an actual agent.

I think where we're at today is actually quite a dangerous place because we have in many ways an often seamless appearance of human-like agency because of the language capability of these systems. And yet in some respects, these tools are no more like human agents than old style chatbots or calculators. They don't have consciousness, they don't have sentience, they don't have emotion or feeling or a literal experience of the world, but they can say that they do and that they feel things, want things, that they want to help us, for example. And they can do so in such convincing ways that it will be really unnatural feeling for people to treat them as mere tools. So we're in this very curious moment where we don't have tools that fulfill the old science fiction dreams of artificial sentience or artificial consciousness, beings who think with us, feel with us, experience the world alongside us.

We're not there yet. But what we've built are tools that will act as if they are beings of that kind in ways that will both intentionally and subconsciously encourage us to treat them in this way and frame them as partners, collaborators, assistants. And of course from a commercial standpoint, that's an incredibly attractive proposition. So what I think this paper does is explore not just the potential benefits but the very real dangers of this moment where we have one foot in reality, where we have tools that actually have very new and powerful capabilities that we can leverage for ourselves and our organizations, but we have one foot in this liminal space between reality and illusion where we have beings that will act as assistants and helpful and caring and supportive partners, but actually at their base are very little different from previous kinds of computing tools and technologies.

Justin Hendrix:

This paper gets into all sorts of things. It goes across questions around manipulation and persuasion, anthropomorphism, misinformation, the economic impact, the environment. We're only going to have time to talk about a few of those today and I think that I'd like to start somewhat with this question around value alignment. You pose this question about how we go about creating these systems in a way that's aligned with values. But the first question is, alignment with what?

Iason Gabriel:

The question of AI alignment is, of course, a longstanding one in AI research and one that has traditionally been the focal point of the AI safety community. And if we look at its intellectual lineage, the concern was often that we would build these AI systems that weren't properly responsive to human instructions or intentions and hence that they would act in ways that are not fully under control with people postulating all kinds of disastrous scenario. And so for that reason, getting AI system to follow human instructions has been a major research goal and research achievement in so far as it's something that we're now able to do. But what becomes clear when we think about AI system deployed in a societal context or AI agents is that clearly they're going to have a capacity and not just influence the person who they're interacting with directly, but also a range of other peoples.

I may instruct my agent to do something and it may be able to execute that quite faithfully, but someone else is going to be on the receiving end of that relationship and that's particularly important if they have an independent claim or say some rights or a welfare concern that intersects with what the AI has been instructed to. In other words, we need to make sure that the range of actions that people can do with an AI system are properly bounded so they respect the needs of other people. And this makes it clear that the question of AI alignment is really a multi-party question that requires us to take into account a huge range of needs, not only the needs of the user, but also the wider needs of society. And in many ways, the key question is how do we strike that balance? Where do we draw these boundaries on what people can do and how do we ensure that the behavior of these AI agents is well calibrated in the sense that it's beneficial when we imagine a society that's widely populated by AI agents where there may be thousands or even millions of these things interacting with different people on different people's behalf.

Justin Hendrix:

One of the things that's great about this paper is it does offer us some rubrics and frameworks I thought in particular around thinking about the ways AI agents may operate at the expense of the user, at the expense of society, the way they may privilege the interest of the user at the expense of society or the developer at the expense of the user, or the developer at the expense of society or society at the expense of the user. It's really quite a web of potentially competing values going along the way. It strikes me as our technologies at the moment are nowhere near being able to handle this complicated terrain that we're often building much simpler instructions into the devices we're deploying at the moment.

Shannon Vallor:

Yeah. I think one of the things that I'm really proud of to be part of this paper is that I think there is so much work in the field of AI ethics that fails to fully examine the reality of the trade-offs that complex technologies present where a simplistic analysis of harm and benefit fails because you may be able to identify a long list of benefits that this technology may realize, but if you're not asking who receives those benefits and at the cost of what and to whom, then you can very easily convince yourself that any design choice that delivers a needed or wanted benefit to someone who will pay for it is justifiable. And there are obvious cases where our intuitions tell us that isn't the case, where the thing that's being enabled is clearly illegal or malicious. But there are lots of well-meaning things we might want from a technology, things that seem completely benign that could impose completely unacceptable costs on other people. And these tools will be able, as Iason says, in some cases, to calculate paths to achieving our goals that we would not have sought out or even realized were there or that we wouldn't have been able to carry out ourselves.

Let me give you just an example. Imagine that you have an AI assistant that you've asked to book a restaurant for you for your anniversary dinner and your most important criterion is that the restaurant be quiet and that it not be crowded. So you communicate that to your AI assistant. Now, imagine that what your AI assistant does is find a restaurant that it knows that you like and then not only make a reservation for you, but book up that restaurant with 50 other fake reservations using manufactured aliases to make sure that those tables are empty when you and your partner arrive. Quiet restaurant achieved, but at the expense of the restaurant itself, of other people who had a right to have a chance to make a booking. And you might not predict that this would be the way that the machine would solve your completely ethically acceptable request to find a quiet restaurant. So these are the kinds of problems that, as you say, we've never had to anticipate with previous kinds of technology. And so we don't have the skills, the routines, the guardrails in place that we need, and that's what this paper is really pointing to, is the need to develop those and to develop those rapidly because the technologies are already getting rolled out.

Justin Hendrix:

Another thing this section on value alignment asks us to think about is the idea that these various agents or ethical AI assistants as it were, may enter into various forms of conflict amongst one another. Can we imagine that maybe the restaurant's bot would push back in this scenario or in other scenarios? Can we imagine a world where there's an enormous amount of essentially technically mediated conflict between these various assistants that's going on behind the scenes?

Iason Gabriel:

Yeah. It's a fantastic question and I think there is a possibility of a disorganized deployment where you see that competition playing out over a range of different societal interfaces. So Shannon gave the nice example of trying to book a restaurant, but you can imagine trying to book a hospital appointment or something else that's got a greater weight and urgency behind it. So we probably do need some kind of interaction protocols that ensure that if there is competition between different agents, it's at least competition of the beneficial variety. And in other areas what we really need is structured cooperation and ideally cooperation that accords with some kind of principles of fairness. So maybe turn taking or working cooperatively to identify solutions to problems that were previously out of reach.

Of course, as human beings, we often interact in a chaotic environment and we lack good information about what the best possible compromise is. This is very clear when we think about something like traffic in a city. And it's possible that with a degree of coordination we could all get to where we want to get to more effectively in a way that causes less externalities in the form of pollution and things like that. But we lack high level coordination, so it could really play out in either direction. You could have unstructured and destructive competition or you could potentially actually identify new optimal ways of doing things that would work even better, which is a a rare but positive aspiration for this technology.

Shannon Vallor:

I'll just jump in and add to that, the coordination problems with technologies that have agentic capabilities and automation built in is not actually entirely new because of course we know that high frequency trading algorithms in previous decades caused serious problems. We talk about the flash crashes and various kinds of unexpected interactions that could destabilize financial markets very quickly when you had many different trading bots interacting with each other and trying to exploit weaknesses in one another's trading behavior. So I think one of the things that highlights and Iason has pointed to this, is that there is a very great difference between the strategy that we need to have for deploying these in low stakes environments with low stakes systems where perhaps we can afford to work these strategies out as we go versus deploying these systems where they're touching critical infrastructure or where the stakes of harmful interactions could potentially be very high for people.

Iason mentioned hospitals and healthcare. I mentioned the stock market. We could think about the power grid. There are certain kinds of systems that frankly we cannot just afford to wait and see what happens or just hope for the best and figure out that we'll work out the problems once we see them. That's not a responsible strategy when you're dealing with human lives, when you're dealing with things that can't be corrected or fixed or remedied. And so we have to start making some very careful distinctions. I think you see in the EU AI Act, some attempt imperfect as it may be to classify different kinds of risk levels for AI applications. And I think we need to be able to think about AI assistants through that lens.

Justin Hendrix:

So much in here, but I want to skip all the way to chapter 10 and the focus on anthropomorphism. Your careful not to really refer to AGI or artificial general intelligence so much in this paper. There's only a couple of citations I think that reference the term. But it creeps in here around this question of anthropomorphism. Imagining AI assistants that at some point may try to portray themselves as human or portray themselves as significantly like humans. What are the specific challenges that you recognize around anthropomorphism?

Iason Gabriel:

I think it's important to realize that although there's a variety of different artificial agents that could come into existence ... So we could think of famous gameplay algorithms like AlphaGo and MuZero. In reality, what's been built now is not just a generic AI agent that can be deployed in all contexts, but it is this slightly anthropomorphic variation. We're getting anthropomorphic agents. And that's almost entailed by the fact that they use natural language and they've been trained on this larger repositories of human data to engage in almost natural mimicry. Although of course there's a debate about how much further their capabilities might extend. And increasingly we see these further affordances and ability to talk to your assistant. We saw that kind of feature recently announced. And so people very easily fall into this mode of thinking that this is, as Shannon mentioned, a proto-sentient interlocutor and to have a whole range of attendant emotions throughout the interaction.

So they could easily feel offended by something that this chatbot said, which you probably wouldn't if you were just receiving feedback from a computer that didn't have this human-like trapping. But of course, there's also these interesting questions about how deeply people fall into those relationships. And we've seen from studies with Replika users that often these bonds that people forge with their particularly AI companions can be quite profound. I think a recent study that I read said that 90% of Replika users believe that their digital friends or companions are essentially human-like. They more or less all regard them as highly intelligent companions. And for many people they become the main point of social contact or a really important life coach. And actually the preliminary evidence is that isn't necessarily harmful to users, but it is a very different situation from what we've encountered before.

And so in the paper we make a couple of claims about anthropomorphism. The first thing to say is we are concerned about deception and simulation. So we don't think that AI agents should be permitted to mimic humans in a way that's non-transparent to the user. They should always self-disclose as AI agents. And then you have the more subtle questions, which are okay, the user knows that they're interacting with an AI agent, but they still want this friendly humanistic type experience and maybe they even want to seek out their version of emotional intimacy with the AI system. And the question is, what do we make of that? And I think there's a couple of interesting things that arise in that situation. The first is that if people really want anthropomorphic agents, then it's very likely they will be built by someone. The real question is what does this do to the overall dynamic between the user of the technology and the technology company that's developing it?

And I think one thing that's important to foreground is essentially with anthropomorphism users give the agent, and by ascension the technology company, a lot of trust basically. They start to open up, they may divulge deeply personal information and they develop an emotional entanglement, which can be hard to wind down. So it may be that they're really reluctant to stop using this technology later on. And so the question is, if people have moved along that journey, how can we make sure that their trust is not misplaced and that they do have a sustainable, viable level of support moving forwards? And that's really just the tip of the iceberg, but you can see that it is ... I almost consider anthropomorphism a gateway question because it leads to so many other deep things that are going on right now.

Justin Hendrix:

Of course, one of the questions that leads to is what are appropriate relationships between humans and these AI agents or AI assistants? And this is something that the two of you have worked on together with a variety of other collaborators in this chapter on appropriate relationships, trying to think about whether the features of relationships that render them appropriate or inappropriate. There's a lot in here on the question of power asymmetries. So I want to get you started on this. On this exploration of what's appropriate when it comes to the relationships and how do we even think about how to define what we would call appropriate when it comes to interactions with these assistants?

Shannon Vallor:

Yeah. And I think you can see right away that the word appropriate is a placeholder. It doesn't tell you anything. And what it points to is the fact that criteria by which we judge the acceptability of our relationships are quite diverse and sometimes their intention with one another that is a relationship might be good for us in one way and bad for us in another. On top of all of the trade-offs we were talking about earlier that a relationship could be good for you, but bad for the people around you or bad for society. But even if we just focus on good for you, let's say, it can be good for you in multiple different ways and in the chapter on appropriate relationships, we identify four high level values that we think are really fundamental to all human relationships worth cherishing and protecting.

The four values are benefit. We receive some good from this relationship. It could be an emotional good, it could be a material or physical good. It could be professional good. It could be an informational good. But there's some way in which our experience or our life added to or enriched by this relationship. A second value is human flourishing, which has some relationship with human benefit as we explain. It's deeper. So human flourishing are about our capabilities and the conditions that allow us to have a good life. And so we might have, for example, a technology that delivers lots of end benefits for us, but does so by gradually eroding our capabilities to live well in the longterm. So a classic example of this would be an AI companion that gives us lots of pleasure and relieves our feelings of loneliness and anxiety, but does so in such a way that actually isolates us, other people and isolates us from our community and makes us very dependent on this one point of support, which is a very fragile way to live.

And might in such a case mean that even if this AI system were then to be disabled, I might have lost the capabilities to actually reform more supportive relationships with others, find support in my community. I might not know how anymore to go about interacting with people in a healthy and flourishing way. So we look at benefit, but also the deeper capabilities that people need in order to live well and how the system affects that. We also look at two other values, autonomy and care. So we spoke about autonomy with these systems. That's actually a different meaning of autonomy than we use in this chapter. So when we talk about autonomy in systems, we mean their capacity to do something without our explicit step-by-step instructions, their ability to solve a problem without us telling it exactly how to do that.

When we talk about autonomy in the context of moral values though we mean something different. We mean the ability to choose your own way of life, the ability to pursue the opportunities that are available to you without being forced into a particular pattern by external agents, the ability to think for yourself. So these ideas about being able to guide your own life are really central to many of the ... Particularly in the west, many of the moral concepts that we have. And when people either by force or deception prevent us from thinking and choosing for ourselves, we often experience that as a moral or a political violation or both.

And then the fourth value is care, which is really fundamental to humans as social animals. We are dependent, we are vulnerable at birth and for the rest of our lives, and for the most part, we require networks of support and mutual aid in order to live well. And so if we had a system for example, that gave us lots of autonomy, but cut us off from our ability to either receive care from others or be a caring part of a community ... What if it made us so capable on our own that we no longer felt any need to give back to our communities? No need anymore to lend aid to others in need. These would be things we would worry about. So in the chapter we really look at this notion of appropriate human relationships as multifaceted, as involving these four different core values that could be, in some cases intention with one another, and that could introduce some very difficult trade-offs.

Justin Hendrix:

I want to just tunnel in a little bit on this thing about relationships and the extent to which somehow these AI assistant may in feeble us in ways that limit our ability to form good quality relationships. When you talk about care, there's this example that you've given here about a Replika user who talks about this idea that they appreciate the fact that they have no friction essentially with their AI assistant. The person says, "People come with baggage attitude, ego, but a robot has no bad updates. I don't have to deal with his family, kids or his friends. I'm in control. I can do what I want." And it just occurs to me that one of the aspects of our relationships is not just care and not just these positive dimensions, but also friction, also criticism, critique, disagreement. I'm subtly steered every time I say something foolish by my wife and the look she gives me, if I remove friction, I'm probably removing something very important from my experience of the relationships I have.

Shannon Vallor:

This was something that I started talking about many years ago when I started working on social media and the kinds of friendships we formed there were the kinds of relationships that we can just turn off with the click of a button. And I think we're seeing an extension of that risk into AI assistants now. Now, I do think in certain kinds of contexts, we're naturally wired to look for this friction to some degree, and that are a relationship that is too easy or simple is likely to be at some point experienced as boring. But here's the thing, imagine that you have an AI assistant where you essentially have a dial or a setting where you can just dial up the friction to make the interaction more interesting. Is that really preserving the value of that friction? As you point out, sometimes what we need from relationships is not what we want.

Sometimes what we want is a form of friction that just gives a little spice or a little color to our lives, but we don't want the friction that really causes us pain. But sometimes the friction that causes us pain is also how we learn. It's also part of our experience of growth. We know, for example, that humans who have a genetic condition that prevents them from experiencing physical pain often end up doing tremendous damage to their physical bodies and live often very difficult lives as a result. Pain turns out to be a very healthy thing to be able to live through and learn from. And I think this is true in the emotional and intellectual realm as well.

Think about, Iason and I are researchers. We didn't get where we are without a lot of intellectual pain that we put ourselves through and that others put us through. And there were many times when we might've wanted to evade that and take an easier path. So thinking about even research assistants, things that might help us learn if they only help us learn through the shortest distance or the most painless path, we may find our intellectual capabilities undermined as well. These are really difficult things that we have to think about, and I think we have to really reckon with what do people need in order to continue to grow in capability, in richness, in depth? And how do communities thrive with these tools, not just individuals being happy with them. We can't stop there. We can't just say, well, as long as the people who use them are content, it's all good. We need to be able to think about these deeper issues of how relationships and communities depend upon people being willing to engage in relationships that are difficult, that are challenging, that do involve very real conflict.

Justin Hendrix:

Laced throughout this paper there's also a set of considerations, concerns about the role of private companies and the extent to which private companies can engage with all of these very complicated issues that you're raising in this paper. At one point you say that the types of duties to solve these types of ethical concerns or to address them or to even understand them may be more extensive than those typically shoulder by private companies, which you say are often in large part confined to fiduciary duties towards shareholders. Is the current industry we have or even the current mode of capitalism set up to really deploy these types of incredibly complex technologies into society?

Iason Gabriel:

So it's a very challenging question, but one that's well worth reflecting on. I think this is a morally complex technology. If we think about a world in which there are many non-human agents that we interact with, it could be quite different from the world that we're in now. Of course, there have been huge technological transitions of maybe equivalent significance in the past, and it isn't clear that they have always been navigated well. But it's also hard to say ahead of time what's possible. So with this research, essentially we're aware that as people developing these technologies, we have a privileged, I say epistemic position. We know certain things about the potential character of the technology, what it might look like ahead of time. And with that comes a responsibility to try and model these futures. And the thing that's really important to understand is nothing that we've spoken about today is predetermined in a way our wagers, that by understanding these critical decision points, we can design better versions of the technology and we can also create a much wider conversation where people will ask for a better version of the technology.

Superficially, you might think that people would enjoy interacting with this sycophantic AI assistant that gives them exactly what they want, and this would be a wonderful product. But in reality, once you have this wider view of things, both as someone who interacts with the technology and someone who's developing it, it becomes clear that can't really be the focal point of a technological effort that has any real long-term viability. And so I was just reflecting on the earlier comment from Shannon about care. And it's really interesting because in many ways, these AI systems cannot manifest authentic human emotions. If they offer care in whatever capacity they're capable of offering, they're not really sacrificing personal times or personal projects or the things that a human caregiver would be giving up. And that does change the nature of the interaction. But on the other hand, as one of our contributors to the chapter, Joel Lehman points out, there's things that AI could do if it was designed well, that exceed human capabilities in many regards. So for example, an AI assistant could be endlessly patient with someone who, I don't know, is suffering from dementia. I think in many ways the pain points and things that are so easy to anticipate just lead to deeper questions about what ought to be built. And that's a conversation that we're all trying to be part of and to in many ways, encourage with this research.

Justin Hendrix:

I appreciate that feeling of we have decisions to make in this moment, but I guess I'm also struck by the idea reading this, that we've already made a lot of decisions about the shape of our technical ... The shape of the industry, the guardrails that we have or don't have around how technology is deployed in society.

And Shannon, I came back to read some testimony that you gave in front of Senate Homeland Security Government Affairs Committee last year. Very interesting hearing. Not one you I think normally see on Capitol Hill that was titled "The Philosophy of AI, Learning from History Shaping Our Future." But you talked about this idea that we are in a weakened political condition and dangerously susceptible to manipulation by AI evangelists who now routinely ask, what if the future is about the humans writing down the questions and machines coming up with the answers? That future is an authoritarian paradise. I read that and I think to myself, yes, we are in this moment of agency, we can make choices about what we build and don't build. But on the other hand, I don't know. It's not like we've got a clean slate in front of us. We're already operating in this weakened democratic position. An economy that seems to be highly exploitative and favors increasing inequality. So I don't know, Shannon. I'm prompting you here with your own words, but how does this project inform your sense of our trajectory?

Shannon Vallor:

Yeah. That's a great, great question. And I think what my testimony was pointing to is that the current incentives ... And I used the word incentives as many times as I could in that testimony because I think it's just something we don't talk enough about. The current incentives in the tech ecosystem, in the commercial AI sector are misaligned with the outcomes that we're seeking. And many people, if they're honest, admit that and say, yes, they are but what do we do about it? So I think we are at this turning point where we have to recognize that for one thing, there is no we in the sense that we have been talking about throughout this podcast, and it's constantly natural to say, we have to do this, we have to choose. But it isn't we. It is particular very powerful institutions, governments, and large multinational corporations that have a disproportionate amount of power in deciding what happens and how it happens.

But I want to go back to this point about it not being predetermined despite all of that, despite the choices that are already made, despite the deck already being stacked in favor of certain interests. And history has shown us this. I try to talk about this in my testimony as well. We have long histories of resistance to these kinds of imbalances. We have long histories of claiming and defending the things that matter most, and it's really important that we not become fatalistic and resigned and decide that we can't actually restore conditions of greater accountability, greater justice. We've done it before in many different ways and different contexts, and there's no reason that it can't be done in this context.

And one of the things that I'll add ... And I've been talking about this a lot recently. Is that I think you're also seeing AI companies perhaps, I hope ... I'm not a Google employee, so I can't speak to whether Google sees this or any other AI company sees this, but I'm guessing they must. Which is that the initial choices to deploy AI tools in context where there was no consultation, no participation of the relevant stakeholders in what tools would be developed, how they'd be deployed, things like education. AI being in a sense dropped into educational environments in ways that have been incredibly damaging in some cases in the same way that it's been dropped into the arts and creative sector in ways that have caused a lot of collateral damage already. And what's happening now is that you're seeing a backlash against AI that's arising in a lot of these sectors. You're seeing artist groups who say, we won't work with you unless you promise you're not going to use AI at all. You have educators banding together to say, we are not going to allow anyone to ask us to use AI in our classrooms. I actually don't think that's the outcome that anyone should want. It's obviously not good for the AI companies, but I don't even think it's good for society because it doesn't then allow us to be selective and wise in the way that we choose tools that actually can benefit us and help us.

As Iason mentioned, there are contexts in which AI can add something that isn't there and that we couldn't put there ourselves. And we are going to throw the baby out with the bathwater if we continue to develop AI without attention to things like human flourishing, to things like sustainability, to things like its effects on our ability to care for one another and the planet. So we can either gather up all the short-term gains and then burn up all of the goodwill for AI that we started with, which is going to leave us, I think, in a very diminished condition both economically and socially, or we can hopefully force through the resistance to these more destructive and less thoughtful applications of AI.

We can force a rethinking of what technology is for, how it's developed, and what the responsibilities of those who develop it are. And that's what the paper points to is that developers themselves need to see themselves as having duties of care to their users, to society, to institutions. That's missing right now. And it doesn't mean that developers are bad people. It means that they haven't been given the incentives and the context that they need to fill the role that now has changed in a fundamental way. And so we need to recognize this moment and be ready to make that big shift in how we think about technology. And it might be that regulation needs to drive that. It might be that public resistance needs to drive that. But I think companies will eventually need to step up and say, okay, we need to operate in a new mode here if innovation is going to be sustainable in the longterm and embraced by people rather than resisted and denied.

Justin Hendrix:

That strike me that you're talking about both top down, the idea that companies need what you call moral maturity. The leaders of those companies would need that. But then you also talked about the idea that rank and file engineers and designers and product managers, et cetera, may need to find a new set of professional norms and apply those across the deployment of these technologies as well.

I want to bring us to a close by just asking you both each to comment very briefly on where next. This is a big piece of work and it collects a lot of ideas into one place. What do you think is the next hill you need to climb or that researchers in this space need to climb in order to contribute to this conversation?

Shannon Vallor:

So I think in terms of what comes next, I think we really need to push from the bottom up and the top down companies to demonstrate a renewed commitment to what goes under the heading responsible AI, but has waxed and waned in terms of how seriously organizations have been committed to that goal and the resources that they've been willing to direct to it. So I think one thing that really needs to come next, as I said, is a maturity that needs to come to the tech industry in the same way that you can think about the history of medicine and the way that it was gradually professionalized and genuinely moralized in the sense that we talk about medical ethics and we see that as a central feature of clinicians and hospital administrators need to care about. There are ethics committees. There are ethics review processes that every medical professional sees as an integral part of their work. Not something simply imposed on them from the outside, but something without which they can't actually fulfill their professional goals.

And people who treat those ethical guidelines as things to be evaded are typically seen as people who don't belong in the profession. It took a long time for the medical profession to mature in this way, but it was necessary because medicine is a very dangerous thing, an incredibly beneficial thing. But an incredibly high risk and dangerous enterprise. And we see very clearly that AI belongs in the same category, tremendously beneficial, but also very high risk, very high stakes, and it requires that professional maturity to come into the industry. So I'm very interested in research on how professionalization could look in the AI ecosystem. You're not going to just take the medical model and copy it over. That's not going to work. It's a different reality. It's different kinds of complex supply chains and things that can't be managed in exactly the way we would manage healthcare. But we certainly can look at innovation in a new light.

I have a PhD student, Bhargavi Ganesh, who works on innovation in governance. And I think this is a really interesting direction to move in. What if we take the lens of innovation and turn it on, how we govern and grow up our institutions? It's not just tech that needs it. You look at the media ecosystem, you look at the political ecosystem. We've really lost the notion of institutional maturity and competence in a lot of ways. And I think thinking about professionalization, how that has served us well in other contexts and how it might come into the tech ecosystem more fully as something I'm personally quite interested in.

Iason Gabriel:

I agree with Shannon that there's a big challenge now, which is to move from ethical foresight to ethical practice. And to some extent, that's a work of translation, deriving really solid practical guidelines from this investigation about how to build a good technology. But there's also another crucial element of what I've said, which is that this is something that is actively being built now. It hasn't yet arrived in the world in a fully fledged and mature manifestation. So actually, these questions should be front of mind for everyone. And I think it isn't just about technologists adopting high standards of moral etiquette and professional norms, but actually us collectively having this conversation about what technology we want to see enter the world. After all, this is our social world and our social interactions in our lives that are being profoundly affected. So really, in many ways, it's a question that's bigger than just technology companies. It's about the world that we want to see ushered for. And then there's a whole distribution of levers that can be pulled to affect better outcomes. And really what we were trying to do with this paper was both provide something that can serve as a foundation for better decision making by technologists, but also a foundation for this wider conversation. Because a lot of these questions, only questions that can be answered in real life and when we have conversations together.

Shannon Vallor:

On top of widening the conversation, it's not just about it needing to be a conversation. It also needs to be decision making power. The power to actually choose that future, not just to talk about it or describe it needs to be widened. The powers that currently hold most of that power. And so thinking about how we can actually equalize the decision-making power, democratize the decision-making power, and not just giving people a voice, but not a say.

Justin Hendrix:

So many issues in this paper that we did not have a chance to discuss, including around accessibility and how to address the potential inequality and the extent to which certain individuals have access to these AI tools and others don't. All the questions around impact on media and information ecosystem, but we'll have to save that for another day. This paper's called The Ethics of Advanced AI Assistants. I believe you can find it in your Google search bar. If you look that up next to Google DeepMind, you'll find that document on the internet. Shannon, Iason, thank you so much for joining me to speak about this today.

Iason Gabriel:

Yeah. Thanks so much, Justin. We really appreciate it.

Shannon Vallor:

Thank you, Justin. This was a lot of fun and I really appreciate the opportunity to highlight the paper, which I think we're all really proud of.