Promising Opportunities, Distinct Risks: AI and Digital Public Squares

Audio of this conversation is available via your favorite podcast service.

Could AI help design better, more democratic platforms and online environments for public discourse? What are the opportunities, challenges, and risks of deploying AI in contexts where people are engaged in political discussion?

Today’s guests are among the more than two dozen authors of a new paper on AI and the future of digital public squares:

• Audrey Tang, Taiwan's Cyber Ambassador and former Digital Minister
• Ravi Iyer, managing director of the USC Marshall School Neely Center for Ethical Leadership and Decision Making
• Beth Goldberg, head of research and development at Google's Jigsaw, and a lecturer at Yale School of Public Policy.

What follows is a lightly edited transcript of the discussion.

Audrey Tang:

I'm Audrey Tang, Taiwan's cyber ambassador, and previously digital minister.

Ravi Iyer:

Ravi Iyer, managing director of the University of Southern California Marshall School's Neely Center.

Beth Goldberg:

I'm Beth Goldberg. I'm the head of the research and development team at Jigsaw. We're a semi-autonomous unit at Google. I'm also a lecturer at Yale School of Public Policy.

Justin Hendrix:

I'm excited to speak to all three of you tonight, and we're going to talk a little bit about a paper that you all are among the 27 authors of called AI and the Future of Digital Public Squares. It's got ideas drawn from a discussion that involved at least 50 other thinkers and technologists last year, April 3rd, 2024, in New York City. I should disclose that I was one of the individuals in the room for that discussion. I learned a lot from it and looking forward to hearing where you all ended up with the ideas that you developed there. But of course, this is really an effort to think about whether AI can potentially be used in pro-democratic ways. Can it help the health of the public square? So I want to find out first from each of you how you came at this work, what perspective you brought to that room, of course, in April 2024, but ultimately then to the paper. And Audrey, perhaps I'll start with you.

Audrey Tang:

I'll start from 10 years ago, in 2014, when I began this work. In that year, only 9% of Taiwan's citizens trusted our government. Meaning whenever President Ma Ying-jeou spoke, over 20 million people disbelieved him in a country of 24 million. And so, in March 2014, a trade pact with Beijing was rushed through, which would have invited them into our network and communication infrastructure. And so there was massive outrage amplified by social media that had just arrived on the scene. And so I helped young students peacefully occupy our parliament for three weeks. It's called the Sunflower Movement. So it's not just a digital public square; it's also literally turning the parliament into a public square, and we called ourselves not protesters but demonstrators to demonstrate new possibilities for democracy to emerge in the digital age.

And so, concretely speaking, we helped, for example, live streaming debates, building conversation networks, posting daily transcripts from open deliberations involving half a million people on the streets, and many more online. And then it just worked. After three weeks, the speaker of Parliament agreed to the people's policy demands. And so it was a very rare occupy that converged instead of diverged. And so ever since I've been working to make digital equivalents of that process so that people very polarized, very divided, can nevertheless agree on what's uncommon among them, a common ground or what I call uncommon ground.

Justin Hendrix:

And let me ask you before we come to Ravi, has your faith in technology as a tool to create a healthier public square or to do democratic engagement, has it been shaken to some extent over the last few years? Certainly, we've seen major social media platforms present various challenges to democracy, various challenges to public discourse. How do you think about that versus the type of technologies that you've been working on?

Audrey Tang:

A key starting point from the Sunflower experience was that polarization is not an inevitable feature of social media. It is a consequence, a direct consequence of how platforms are designed. So our idea is to pilot new bridging tools, such as the open-source platform Polis starting in 2015, to tackle the heated debate on Uber and so on and so forth. And already by 2020, we're seeing Taiwan's trust approval rate over 70%. So from 9% to over 70%. All it took is this consistent use of bridging systems to reflect people's will together through what I call pro-social media and not anti-social media. So to your question, no, I think Taiwan's experience showed that it is possible to build social networks that are pro-social.

Justin Hendrix:

Ravi, that's a good place to bring you in. Sounds like Audrey's observations are somewhat similar to yours. After both spending time inside Meta and now, more recently, in the work you're doing. How do you come to the questions in this paper on AI and the future of digital public squares?

Ravi Iyer:

Yeah, so I mean, I started my career as a programmer who didn't really know what he was doing with his life, and I eventually got a degree in social psychology and ended up studying moral psychology, which is the study of why do people do what they do, what are the values they have, why do they come to have the values they have? I ended up working with Jon Haidt a bunch because we used to describe how people came to their values as opposed to judging them, which other groups were doing. And then we ended up doing a lot of work on polarization. My first work on polarization really was bringing liberals and conservatives together in the U.S. and trying to study before and after what changed with them. What we learned was that people don't really change their attitudes, but if you bring them together under the right circumstances, they do change their attitudes about the other side, right? They still have the same political opinions, but they believe that the people on the other side are maybe not evil, not as bad as they think they could be, and they could have a beer with them.

Eventually. I started working at Meta. I had this career in academia working on polarization. I also worked at a company called Ranker, and I had a dual tech and academic career. And somebody said, "Well, why don't you work at Meta? You can work on polarization. You can do a lot of good in the world." So I worked at Meta for about five years. I started off trying to define what is the kind of content that is dividing people and how do we get rid of it. And I think the answer I came to was, that's actually the wrong question. There's always going to be conflict in the world. If you try to moderate your way out of it, you end up doing really awful things. You end up with the backlash that we see, and you end up over-enforcing on people's authentic beliefs.

The actual question that we should be asking is are we incentivizing divisiveness on these systems? Are we almost paying people with attention to be more divisive? I was there when BuzzFeed famously came to us and said, "We're writing more divisive things because you make us. We're doing it because it does well in your algorithm." Ben Sasse, a conservative politician, wrote a whole book about how politicians feel this push to be more divisive. And so I think we can design spaces that push people to be better, that people are pushing stuff that they're proud of, not things that they're doing just because the algorithm tells them to.

Justin Hendrix:

Beth, I want to come to you, and I think about Jigsaw and think about the legacy of Jigsaw, where it started. Love to hear your perspective on how you've come to this particular moment, this particular work. But I just think about back to its founding Jared Cohen, the ex-US State Department policy wonk. The mission to use technology to tackle geopolitics. It seems like gone through some twists and turns over the years in terms of what exactly that means, but this still seems, to some extent, right in the zone of that initial concern.

Beth Goldberg:

Yeah, I couldn't agree more, asking the question of how AI and particularly large language models can help us be more democratic online feels part and parcel to why Jigsaw was created in the first place. I joined Jigsaw myself about seven years ago now, and I've been lucky to spend those years in a really interdisciplinary team. I think one of the things that's unique about us is at Jigsaw, we've got engineers, product folks, we've got researchers, and designers. We also work in really close partnership with a wide array of folks in academia and civil society. And so we've been coming at some of these questions, like using natural language processing for how to improve conversation spaces online, for many years now. And we've been asking how can we design tools that will help people want to engage in digital public squares and not just lurk or leave the public square when something becomes too toxic or too hostile.

And we define digital public squares here really broadly, right? This is not just traditional social media. We're also talking about comment sections. We're talking about private messaging apps. We're talking about anywhere that we're having large-scale conversations today and doing the substance of democratic deliberation. And in those spaces, I think we came to this conclusion that we're really not fulfilling this opportunity to have open democratic discourse. But we saw an emerging set of opportunities recently with the advent of large language models where we can really start to reshape and redesign these spaces. Not just to have more voices participate in this democratic discourse but actually people that have more agency and more influence on those decisions. Engage in more meaningful ways.

So I don't need to tell folks listening to this particular podcast that our digital public squares today are not exactly these utopian ideals of pluralistic democratic discourse. You're not exactly encouraging people like myself to stop lurking and start engaging, so I'm experiencing this too. But we wanted to really harness these opportunities with large language models that I mentioned, but we didn't want to do it alone. We wanted to develop the opportunities and mitigate the attendant risks that come with them with a whole range of experts across sectors.

So that brings me to the event you mentioned, Justin. Last April, we brought about 70 experts together to New York, and these were folks from academia, researchers, civil society media, different tech company engineers and designers, and even some folks from governments like Audrey was there, as well as Ravi. And we asked really big questions about the different ways in which large language models could play an instructive role in democratic discourse. We narrowed in on four different areas that we can talk a little bit more about today, but 27 of us then went on to write a position paper, which, let me tell you, 27 co-authors, what a choice. But I'm really glad we did it because we had this real diversity of opinions and perspective,s and we were able to pull together not just a collaborative research agenda of where research fields should go, but also really prioritize opportunities for us as technologists, as civil society, as decision-makers to think about how we can be redesigning our digital public squares.

Justin Hendrix:

So my listeners probably definitely agree that there are problems on the internet at the moment, but there are probably multiple listeners thinking, "LLMs are the solution?" There are lots of problems with LLMs as well around bias and original sin in their training data. All sorts of questions about the ways that they might harden some of the problems that we face simply by their deployment across the internet. How do you think about that in going into a project like this, knowing that today's LLMs certainly aren't perfect and, in fact, in some cases, are harmful? How do you think about that as part of the condition for this set of questions?

Beth Goldberg:

There's a whole range of mitigations that we're thinking about and that we thought about as a group for this paper and at that convening. Some of them are more technical, some of them are more social. When we think about some of the social mitigations, this starts with really deep and broad listening. Can we do more ethnographic-style studies or deep interviews to understand people's relationships with AI and how they understand it? How the AI can be better interpreted, can be better explained to them. And then we also need to do broader listening to make sure that more people are actually able to use this stuff. It's not just getting concentrated in the hands of the few.

I think some of the solutions are about the ways that we're building to be more transparent and more explainable in terms of why the AI is generating what it's generating. And then, lastly, I think there are ways to build that are more interoperable and open source so that people can learn from each other, stress test what one another is building, and contextualize these tools for different geographies and cultures more natively. So we'll be able to actually build for different populations more effectively that way.

Audrey Tang:

I agree that not only do the tools themselves need to be open source, but the ability to work with open-source language models goes a long way to address some of the fears around essentially handing control over to proprietary language model providers. And I also would like to say that, for example, in Polis, the tool that we use to uncover uncommon ground, 99% of this is good old-fashioned AI. So it's K-means clustering, it's principal component analysis. It really is not about language models. The only one page that uses language models as we speak on Polis GitHub is the narrative reporting. They use it in an extremely narrow way; that is to say, they only use a paraphraser so that multiple statements by multiple citizens can be summarized into a shorter statement, but all the output bits come from the input. So you're not forcing the language model to hallucinate. And so I think this kind of very judicious use of re-narrowing of the scope of the originally maybe two broad capabilities of language model also shows a more reliable way to do evaluations and so on.

Ravi Iyer:

I've done a lot of work on comment ranking, so I wish when I had done that work that, I had access to the LLMs of today. I think the central observation is that people don't really like crappy comments. The way that people often will rank comments is like, what's the comment that got the most likes or what's the comment that gets the most replies? And oftentimes, the comment that gets the most likes or replies actually ends up being a crappy comment. If I say, "Eff you, politician," I get a lot of likes. Maybe some people will argue with me. You just don't have a good signal of what's a better comment.

Whereas LLMs can actually give you a signal. It can give you like, okay, that's a comment that is expressing emotional support, or curiosity, or it's a very thoughtful comment. You can ask an LLM these questions and it used to be really expensive to develop these... You could build a classifier for these kinds of constructs, but it was really expensive, and you had to convince a ton of people to do it. Whereas now with LLMs, it's a lot cheaper, it's open source. So I think you no longer have to live in a world where you're optimizing for replies and likes. You can now optimize for all these things that LLMs can give you.

Justin Hendrix:

And so maybe we'll dig into some of those mitigations a little bit more as we talk about some of the application areas that you imagine in this paper. There are four that you look at: collective dialogue systems, what you call bridging systems, which we've already mentioned today, community moderation, which is on the top of many people's minds at the moment, and then, of course, proof of humanity systems. So we'll have to get into a little bit on each one of these. I think we'll spend most of our time on this idea of collective dialogue and bridging, but maybe, Beth, collective dialogue systems. What is involved in collective dialogue systems?

Beth Goldberg:

We're lucky to have Audrey Tang here, who's actually implemented a lot of these in practice, but I can give a bit of a high-level overview. These are actually systems that take a lot of inspiration from peacebuilders and those who've done large-scale negotiations in the past. But for example, these types of collective dialogue systems are taking technology platforms, some that you might've heard of like Polis that Audrey mentioned. And they're taking that qualitative richness, a focus group, or a large-scale negotiation, and they're combining it with the quantitative scale polling, where you're able to get a large number of people's opinions all at once.

What makes them deliberative is when you're able to then iterate and have people work on exchanging those ideas with each other on finding these emerging areas of agreement, and then negotiating them, refining them with one another until you ultimately get to a place where you're really surfacing common ground and areas of disagreement between large groups of people. So there are lots of different types of technology platforms that can get there, some that are much more formal and look a little bit more like maybe an offline citizen assembly that just has some digital components. And then there are modes that look a little bit more just large-scale surveys with snapshots of people's opinions that get upvoted and downvoted almost like a subreddit.

I think these are super, super valuable, these collective dialogue systems are super valuable for helping us gather collective feedback. The collective intelligence of a whole group of people on a scale and with a level of nuance that was totally unimaginable before large language models. They're overcoming a lot of challenges of cost, of information overload for decision-makers and participants. And they're really making the most of large language models' ability to parse, analyze, summarize, and find the common ground in huge quantities of data.

I'll just flag that we've got a pilot that Jigsaw's running now with some partners running one of these collective dialogues in Kentucky, and I think it's a really cool example of how this technology can be applied in practice. It's a purple town, and they're having this conversation right now about what the future of their community should look like. And it's the type of thing that if you had a citizen assembly or a town hall, you'd get maybe a hundred people in the room. But they're able to have many more times to engage digitally by communicating with each other on Polis, just like Audrey used in Taiwan. These large language models are going to help make sense of and moderate this much larger conversation and then elevate those voices to decision-makers in the community. So that's a bit of an example of how you could actually run these collective dialogues in practice.

Justin Hendrix:

Now, Audrey, maybe I'll just ask a little more about your experience. You mentioned Polis. There are also other mentions of different systems in here: Remesh, Make.org, All Our Ideas, CrowdSmart, Citizen Lab, and other shots on the goal when it comes to collective dialogue systems. Is there a way in which you think about what LLMs open up for these systems, and I don't know, potentially any bounds that you see for them? How big can these dialogue systems get? Where do they start to break down and just become the internet?

Audrey Tang:

Definitely. So I think a lot in terms of pro-social spaces because this anti-social behavior is not a function of any particular person or even any particular community, but rather a function of how they're structured online. And the great thing about conversation networks is that it emphasizes the dialogue. So, for example, the four of us are now getting real-time feedback as to which of our statements resonates with the room. And compare that with the more viral source of social media where people can talk alone to their screen and only the most viral, I don't know, 30-second clips get amplified. And so you see a very narrow slice sample of what is the most outraged, or more strange, or things like that, but you don't see the resonance of it.

And to your question, I think language models can help to provide this listening experience so that from surprising validators, from people who are unlike me, but actually we do have common ground, it has the way to paraphrase their output, their content in a way that I can understand in my frame. And I think that is very... This transcultural capability is, I think, one of the most powerful. But constantly we need to push back against the intuition of this entire in silico deliberation in that we don't have to even listen to one another anymore. We just talk to our avatars and our LLMs, and they do the deliberations for us.

I think we need to push back that a little bit because that's going to the gym, wanting to exercise our civic muscles, but instead we're just celebrating our robots doing the weightlifting for us. It's quite interesting, pretty spectacular perhaps, but at the end of the day, the civic muscle doesn't grow from it. So I think that is one of the bounds that we need to constantly weigh to see what kind of work is translational in nature that enhances human civic muscles and what kind of work could actually make it a trophy.

Justin Hendrix:

Ravi, anything to jump in on here on collective dialogue systems?

Ravi Iyer:

I think the simple analogy I sometimes use in the work I used to do when we brought liberals and conservatives together is you can design a space and start with what people agree on, or you can start with what people disagree on. When we used to bring liberals and conservatives together, every group used to start not by discussing healthcare or something that was contentious. They would start by talking about their families, talk about the local sports team, talk about the things that we all have in common, and then you could get to the stuff that you disagree on.

And in some ways, the social media systems are designed to start with what we disagree on. That's what you get first. Whereas a lot of these collective dialogue systems start with what you agree on, what are the things that diverse groups of people both agree on? For example, with community notes, they surface notes that diverse groups of people agree on. And I think when you take that inspiration from the collective dialogue space and you import it into the social media space and you start thinking, what's a comment that diverse groups of people would agree upon? What if I put that first instead of one that everyone is replying to? How does that transform a space, and how does that bring us closer together?

Justin Hendrix:

Let's go into bridging, which is related, but this is a set of ideas that could potentially be used outside of collective dialogue systems and other systems for dialogue. But Ravi, I'll maybe stick with you. You've done a lot of work on bridging. How do you see LLMs being used for bridging?

Ravi Iyer:

A lot of what I was just talking about as far as structuring a space so that it starts with what you agree upon is what the practice of bridging is online. So the simple version of bridging is to look for things that diverse groups of people both have some positive opinion about and try to surface those things as opposed to the things that people are arguing about. And one thing that LLMs can do, and AI can do is help figure out what are those things that diverse groups of people do agree upon. It can do it both by simulating the people, it can also simulate the constructs that those diverse groups of people agree upon. So, the kinds of things that diverse groups of people find positive tend to be things that are more thoughtful, that are more curious, and that are more supportive. Whether it's trying to simulate the people and their reactions or whether it's trying to actually simulate the constructs that those people would respond positively to AI and LLMs can help create a space where we start with what we agree upon as opposed to what we disagree upon.

Justin Hendrix:

You had a number of the folks that I think of as some of the experts on bridging in this conversation. There are folks among the authors, including Aviv Ovadya and Jonathan Stray, who have done a lot of work on depolarization. Beth, are you already using some of these bridging concepts in the work you're doing in Kentucky or elsewhere?

Beth Goldberg:

Yeah, we are. So some of the systems that Ravi just described are part of these collective dialogue systems. Some of the ways in which these dialogue platforms are identifying common ground is they are actually bridging. They're bridging these opinion groups or different groups who tend to disagree on those really polarizing topics, and they'll identify, oh, okay, maybe you guys all disagreed on the healthcare question, but there's actually a lot of overlap on this question about how do we invest in education? And so they're giving, in a way, decision-makers maybe some of the lower-hanging fruit of saying, "Hey, here's where there's already a built-in consensus in your community on where folks agree."

The other space that we're already engaging in outside of Kentucky is something that we're calling content-based bridging. So it's related to what Ravi just described of looking at what bridges groups and individuals, but that sometimes has the drawback of just identifying the stuff that everyone can agree on that's fluffy. So think maybe groups are really polarized, but they all like cat memes, but we want to avoid a world where we're just upranking cat memes. So we spend a lot of time with researchers who look at the types of features of language that tend to connect people to one another.

So what are comments that build empathy or comments that build connections between groups? And that's stuff like curiosity, compassion, even reasoning, and nuance. And so we were able to use large language models to actually build models to identify these nuanced attributes of speech, and then we can go ahead and uprank those. So if we've got comments that have a lot of nuance and reasoning, maybe a lot of curiosity and compassion, we can reward you for those types of comments and actually uprank those. It's not something that Jigsaw is deploying ourselves, but we've been working with a number of partners who run, say, comment spaces or community spaces, and they're testing them out in their own communities now and saying, can we bridge divides in our community by changing the reward system, right? Redesigning what our public square is rewarding and upranking.

Justin Hendrix:

Let's talk just a little bit about community moderation. Clearly, there's a lot in the news at the moment about community moderation after Meta's announcement on January 7th that it would try to move more towards the community moderation system looking a little like X. But I want to put that out of my mind a little bit right now just to hear what you think may be possible with community moderation in the future. Because you're trying to think through what might be possible with new technology, but certainly, also in new contexts. So not always situating the way we're thinking about community moderation necessarily in the examples that we have before us. But Ravi, I know you're somewhat a fan of the idea that community moderation could work better and could perhaps even work in coordination with better fact-checking. What do you think when it comes to community moderation, what can LLMs do to help?

Ravi Iyer:

Instead of the word community moderation, I might just use the term feedback. And instead of having it centralized where there's one group of people who decides what goes up or down or what people can and can't say, I think it's much better if there's a large group of people providing that it's better for scale, it's better for legitimacy. And we all live in a world where we get feedback all the time. I see someone nodding, or I see someone shaking their head, and I'm constantly adjusting my behavior to the feedback I get. And it's actually weird when we go into an online environment, and we don't get that feedback. And that's where we get some of the problems we have in online discourse where people are not getting signals that they're turning people off. They're dominating a conversation.

And so we can create community feedback that, look, this is not something that people are appreciating. This is the voice that we want to elevate. And I think community moderation is just the start of that, but I think the paradigm of content moderation is not going to solve the problems that we need it to solve. And hopefully, if we start to move more towards this feedback system, we can get to where we want to go.

Justin Hendrix:

Audrey, what about you? Has community moderation played a big role in the experiments that you've been working on?

Audrey Tang:

Yeah, definitely. I just published a paper with my co-authors called Pro-Social Media that summarizes what we have learned in Taiwan when it comes to open-source communities that build their own pro-social media and what that may have a broader impact if adopted by especially decentralized, but also now the increasingly, as we mentioned, centralized social media systems. One of the key insights is to take this bridging idea, but instead of applying it just after the fact, like community notes, which only takes place after something divisive goes viral, and apply it to the main feed. The great thing about applying it to the main feed is that you can see for each post whether it's popular with a community or with multiple communities that you're associated with, and also to represent fairly the two sides or three sides of division that your communities were debating on this subject.

And this basically means that instead of looking at one or two viral memes and mistakenly thinking that was the consensus when it probably was not. It's a caricature of a polarized camp, and you can see a very clear representative of what is actually divisive and what actually is the underlying infrastructure of common knowledge that everybody now can know that everybody else knows that informs this debate in a way that's not polarizing but rather focusing. So, think about the bridging system we just mentioned, but applying it to the main social feed instead of just the community notes or the fact-checking that was the main part of this paper, Pro-Social Media.

Justin Hendrix:

Beth, in the paper, the question for inquiry regarding community moderation is how we can empower community leaders to better guide the norms of discourse to support healthy, inclusive online communities. What do community leaders mean to you when it comes to community-driven moderation?

Beth Goldberg:

So these are often the folks who are already forming and shaping and then shepherding their online communities. Imagine a subreddit, Discord server, even Wikipedia communities, they actually have a lot of these community leaders who are moderating community spaces online already where there is some centralized moderation. But a lot of the work of shaping a healthy conversation space, shaping a healthy community is being done by these volunteer moderators. And so right now, these folks are spending a lot of time and energy working with frankly pretty clunky tools for the most part. There are often pretty brittle collections of banned words or tools that are given as binary. This crosses a line. This doesn't cross the line for each comment. And so it ends up being a lot of manual labor for these folks to be moderating and keeping their spaces safe.

When my team did some deep ethnographic research with a couple of dozen of these community moderators, they told us they really don't like this job of having to shield their communities from the trolls and the disinformation and the other sorts of hate speech. What they really enjoy is shepherding what they describe as more like the pro-social elements of trying to foster healthy conversation. Doing the stuff that's uplifting the good. And what they really wanted was more bespoke tooling. They wanted large language model-type tools that allowed them to say, "Let's automatically filter out this type of content that isn't welcome in our space, and then let's uprank this type of content that we want to reward and see more of in our space." And so we're just getting to the technology that's allowing them to have that level of nuance to really shape and steer the types of communities that they want to have online.

Justin Hendrix:

I want to get to the last area of query here, which I think is maybe the most forward-looking one in a way. This idea of proof of humanity where the question is with advances in AI making it increasingly difficult to tell humans from machines online, how do we balance the competing interests of privacy, free speech, and authenticity? So I guess the flip side of we can use LLMs perhaps to make better collective dialogue systems and platforms. How do we tell if it's humans that are engaged in that dialogue?

This reminds me of a group of students a couple of years ago in my tech media and democracy class. We came up with various prototypes for ways to think about how to build a more democratic technology environment. And these students came up with a project which is basically the internet, but only for people. And their whole idea was that eventually, we're going to be in a place where it's going to be quite hard to tell sometimes who you're interacting with. But I don't know, how do we do that? How do we know that we are engaging with humans but maybe also preserve some of the things that we know empower expression, particularly in difficult circumstances or in authoritarian circumstances, things like anonymity or the ability to at least obscure your identity?

Audrey Tang:

I think there's a range between pure anonymity, which makes it very difficult to tell a bot from a human, or pure real name systems where everybody has to digitally sign everything, including video and live stream and things like that, which creates its own problems in terms of coercion and also power asymmetry makes whistleblowing much harder and so on. I think in the paper, we develop a few ideas that are in the middle of this spectrum. So you can think of as meronymic systems, that is to say, it reveals a partial identity without revealing the full identity. A classic example is actually the age signals that many jurisdictions, including Australia, are now developing. The idea is that you need to show that you're over a certain age in order to, I don't know, drive or purchase addictive substances, including social media. But if you overly disclose the real name, then that creates a hoard of problems, a lot of which will reside in state surveillance basically.

And so the Australians, like the Taiwanese, invested in the infrastructure of what's called selective disclosure so that you can only, for example, prove that you're over a certain age but not your birthday or where you live, or you can prove that you have interacted previously as a human with this community without exactly specifying what was the piece of content that you have posted and so on and so forth. So the idea is that everybody will then be able to establish some sort of proof of humanity credentials without overly disclosing really anything else besides what is required to show to the group of people that you're participating as a community member. So a lot of zero-knowledge cryptography and so on is going into this.

Justin Hendrix:

One of the things that I definitely want to make sure we address and that I did find quite interesting in the conclusion of this paper, the paper seems to recognize that try as you might, as much effort as you might put into collective dialogue, or content moderation, or bridging, there are certain situations where the only answer is to send a pretty strong signal that someone is not engaged in meaningful or productive conversation. You have this paragraph where you say, "Some discussions involve actors who attempt to inflame tensions, scapegoat others, engage in intimidation, or amplify known misinformation. This is especially true with discussing issues germane to historically marginalized groups who may not wish to engage in communities where such tactics undermine their humanity and dignity. Part of cultivating healthy public squares is deescalating such conflicts and removing bad-faith actors."

How do we draw the line there? Because we were talking earlier about the idea of sending signals, of trying to avoid, of course, that circumstance where you end up having to take such drastic action as to essentially tell someone, "You're no longer a part of this particular collective dialogue." How do we think about designing systems that perhaps limit the number of times that we have to do that? And I don't know, Ravi, is this maybe a good question to put to you?

Ravi Iyer:

I think the first thing we have to do is stop rewarding people for being bad-faith actors. So right now there is some amount of monetary reward in the form of advertising that you can get by being a bad-faith actor and saying the most outrageous thing you want regardless of political persuasion. It could be about health instead; it doesn't have to be political. And so we just need to stop that incentive in order to expect that there will be less of that to have to clean up afterward.

The second thing is I think we need to have feedback or some way to hold people accountable. So it's not, most people, most people in the world do not want... They're not trying to dominate a conversation. They are good-faith actors, they care about other people, they're willing to listen. If we just can do something... And we all know how to deal with that small group of people who are bad-faith actors. We know how to deal with that in everyday life. We send them bad signals; we stop inviting them to our parties. How do we do that online? How do we stop inviting those people to our parties online so that the rest of us who are of good faith and who want to bridge divisions, who want to understand people on both sides can have a meaningful conversation? And so if we can stop incentivizing the bad actors, stop letting them dominate, and then maybe stop inviting them to our parties, maybe we can start to have a better conversation.

Justin Hendrix:

Changing those incentives might require a little more of a revolution than you're letting on. But Audrey, how do you think about this in some of the systems you've built about removing bad-faith actors?

Audrey Tang:

I was involved in the prototyping of vTaiwan, so that was more than 10 years ago now, a way to build pro-social systems in which the bad faith acts are de-escalated, but the good face parts are amplified even though it comes from the same person. So you can see trolls gradually getting reformed. Actually, I practiced this on a personal level, too. My hobby is what's called troll hugging. So some people hug trees, but I hug trolls. And the way it goes is that during my cabinet tenure of seven and a half years as digital minister, sometimes I just see thousands of words from a person on social media making ad hominem attacks, toxic attacks, and so on, about me and my identity and my policy, so on and so forth. But I can construe a constructive meaning out of maybe five words out of this 3,000-word rant.

So, I would just focus on that and reply either through this group chat or through quotes that are a quote that is only constructive parts and then engage very meaningfully with it. And so it shows the bystanders that the flaming words do not have any reach. Indeed, in the original design of vTaiwan, we simply limit the reach of those parts, and then only the parts that can be constructive become the topic, becoming agenda-setting. And so for many people, after a while, they learned that only constructive acts of performance can get the genuine attention that they crave. And that is what most trolls are coming up with because they really seek some sort of response, and the social media, the antisocial corner of it anyway, rewards the kind of behavior that is more fringe, more polarized, and so on. So, as soon as you can detect that, I do use language models to help to find out those parts that are constructive and also construe a response, it reduces my emotional labor and then makes it easier to engage in a good faith way.

Beth Goldberg:

The design of our spaces can set the norms for these actors. And so we can very intentionally redesign these spaces to either reward the bad actors or quote, unquote, "help incentivize them to become bad actors," or we can disincentivize that. It's really, it's a design choice that we make. For example, we know from a bunch of studies now that the comment that gets pinned to the top of a comment section is often the de facto norm setting for all the comments that are to follow. And so if that's a really trolly comment, you're going to see a whole bunch of trolly comments to follow because that's what people think is expected there and is welcome in that space. That's the type of party that they're trying to have in that space, to use Ravi's metaphor there.

But if the party host or the facilitator of that comment space, maybe the creator who's moderating that says, "Hey, that's not the type of party we're going to have. I'm going to pin this much more constructive comment," that completely changes the conversation. Another good design example is removing the reply button, and I know for a lot of these collective dialogue systems, they let you post comments and then uprank or downrank other folks' comments, but they don't let you reply to them. And that's a really intentional design choice. Mastodon does this as well, and what it does is it encourages users to post their own individual ideas. So you actually get a wider range of unique contributions and it's discouraging people from just rebutting and trolling one another. And so I actually think that reply button is a much more powerful design choice than maybe we often think about.

Justin Hendrix:

My last question comes to this question around recommendations for future work. So you do make some recommendations at the end of this paper, almost more suggestions that this is somewhat uncharted space. We're talking about using artificial intelligence to shape human discourse. There are a lot of questions here about trust, about the extent to which people will trust these systems, and will trust AI. I'm sure there's someone listening to this right now who's thinking, "These people are talking about social engineering. They're talking about using artificial intelligence to change politics or change the way we engage in politics." You call for, of course, transparency. You call for diverse perspectives on this work, but I don't know, how do you think about the real-world circumstances where these types of ideas are being deployed? How do you avoid it simply being seen as another form of technological meddling in human politics?

Audrey Tang:

Social engineering has a meaning in cybersecurity, meaning to use psychological manipulation to trick people into making security mistakes and giving away sensitive information. And that is indeed a failure mode for this kind of work, right? Because if you make an opaque language model and then you make opaque applications of it, and then you detect people's affiliations, the groups they associate with the most, and then you strip mine it only to sell individualized advertisement, that really is a kind of social engineering, which is by the way also a business model. And what we are working on now is not just transparency in system design but also making sure that people have the components of those tools so that they can deploy it themselves. What I like about, for example, Jigsaw's work on sense-making is that it's not opinionated about which language models they use or which input modalities they prefer. It's just out there as open source so that when people want to analyze a particular conversation, they get to determine exactly how it's done.

And I work with The Collective Intelligence Project who basically look at the Jigsaw sense-making, re-implemented a large part of it. But because it's open source, we get to keep, for example, the topic, subtopic detection, and things like that. And so, basically, the idea here is that it's like Lego bricks, and instead of designing one central monolithic system that all the community have to align to the logic of AI, this is very lowercase technology, civic tech, that people can then take as individual Lego bricks to build systems to their particular community that hopefully federate together as we argue in the pro-social media paper. But even if they don't, that is still something that helps the community moderators, the people who shepherd the conversation, to reduce their cognitive labor.

Ravi Iyer:

Any design choice you make is going to change a system. So there is some amount of engineering that you are doing that is going to change the system. And there is some amount that has already happened that has created the systems we have today, and there's some amount that you would be changing with some of the recommendations we're making. I think the big question is, who are you engineering for? And so my hope is that a lot of systems are engineered for businesses to make money, and they reward tiny groups of people who enjoy arguing online. And that's not most people, right? Most people stay out of it because it's not worth it to them.

And can we engineer something that's not for us, but for the user, right? Can we engineer something... And there's a lot of people who feel manipulated by these systems, they're turned off by these systems. And so regardless of whatever point of view ends up winning, just having something that people feel proud of, they're posting things they're proud of, they're learning new things, they're connecting with other people. To some degree, I'm agnostic about what they're actually connecting about or what they're actually learning, but just helping them with their aspirations rather than engineering for a business or the tiny group of people that wins the arguments online.

Beth Goldberg:

A third line that I'm hearing in what both Audrey and Ravi are saying is getting people more agency. And if I want to think about some of the subcomponents of what it looks like to give people agency, especially if they're not developers, they're not doing what Audrey did and refactoring some of these open-source tools to their own liking, they're adopting the tech that's out there. What does it actually mean to give them agency? And I'll quote Zoe Weinberg here, who talks about three different types of agency that people can have. One is the choice over the tech that they use so that they're not locked into one specific type of tech that they can actually switch between them and understand their option sets.

Next, having context to being able to make more informed decisions, not just having to take the tool as it's coming to you. And you can actually go in and understand how to change the settings yourself. And lastly is control. We talk a lot about control over data but actually control over the whole experience that you're a part of, being able to opt in and opt-out. So it's basically making sure that yes, people are still excited by maybe the AI magic or the platform that they're a part of, the conversation they're a part of, but it shouldn't feel like they're having a black box that's happening to them, that they're actually able to choose and have control over the tools that they're working with.

One way that we can think about getting to these three Cs is actually through co-design. This is something that my team is thinking a lot about. How can we make sure that these large language models are being built with the affected communities that are ultimately going to use them? And I think that's where we need to start if we're ultimately going to be building tools that get to a place that feels transparent, explainable, accessible, et cetera.

Justin Hendrix:

I look forward to seeing how this work plays out in the future, both from a research perspective and how it's put into practice, both experiments that Jigsaw might run, things you might get up to, Audrey, prototypes you might build, Ravi. I appreciate the three of you spending some time taking us through this. If folks want to find this paper, they can google AI and the Future of Digital Public Squares. I'm sure they'll find it in our archive or certainly in the show notes for this episode. So I appreciate the three of you taking the time. Thank you so much.