The Sunday Show: A Conversation with evelyn douek

Audio of this conversation is available via your favorite podcast service.

Lawmakers around the world want to do something about social media, and in particular content moderation. But what if the interventions they are developing are based on a flawed conceptual framework about how content moderation works, or how it should work?

This week I had a chance to talk to one of the smartest legal minds on questions related to content moderation to explore some fresh thinking on the subject: evelyn douek, a Doctoral Candidate at Harvard Law School and Senior Research Fellow at the Knight First Amendment Institute at Columbia University. evelyn is the author of “Content Moderation as Administration,” forthcoming in Harvard Law Review, a new paper that serves as the basis for our discussion.

(I’d be remiss if I didn’t also point out that evelyn has an excellent podcast with Quinta Jurecic at Lawfare called Arbiters of Truth. Check it out.)

Justin Hendrix:

We're going to talk today about your new opus, "Content Moderation as Administration", which is forthcoming, I believe, in Harvard Law Review Volume 136, for folks that are going to bookmark that and keep an eye on it.

evelyn douek:

Yeah, I think academic publishing timelines mean that it's still a while away. I think November is the publication date. This is a nice and early sneak peek.

Justin Hendrix:

Excellent. This is 82 pages of material.

evelyn douek:

Apologies.

Justin Hendrix:

I didn't add up the number of footnotes, but how long have you been working on this?

evelyn douek:

Oh, boy. I think this is about a year in progress. This is about the fifth draft or something like that. Normally, maybe I would have taken a little less time, but the year was what the year was. I think all of us have maybe taken a little more time to turn things around at the moment. It was also my job market paper for the academic job market. It took a little bit of extra loving care.

Justin Hendrix:

I want to get into your goals and what might happen to you after you finish your program, but want to get right into the meat of this. Now, you set the scene for this paper by describing how you think of the mental model of content moderation that most people are using and how that works. You have this idea of, "The rough online analog of offline judicial adjudication of speech rights." In this model, we're thinking about a set of rules, or policies, or laws, that are applied over and over again to individual bits of content that we post up that are handled by a bureaucracy of moderators. You talk a little bit about how this model came to be.

evelyn douek:

I think that when we think about content moderation these days, or when a lot of people talk about content moderation, they do use that model where it's like you've got a set group of rules that are written by an internal platform policy team that might resemble a legislative branch of a state or a country. Then they sort of send those rules out to this vast army of content moderators dispersed around the globe and program them into AI tools. Then those rules are just applied again and again to individual pieces of content. If a user is not satisfied with the decision that those frontline moderators make, you could appeal it up a hierarchy of further moderators and try to get the original decision overturned. I think this will look very familiar to a lot of lawyers because that's how we think in the judicial system, in the offline justice system, of speech cases working, like First Amendment cases.

You've got your iconic cases where there's an individual plaintiff who said something or wore something on their t-shirt and the state tried to shut them up. They went to court, and they go all the way to the Supreme Court. We think about it in those terms. I think that lawyers have dominated this debate for a very long time. I think that it's been thought about both inside and outside platforms in academia by lawyers because it's sort of, again, it looks like regulation. I think that for better or for worse, and my argument in the paper is maybe a little bit for worse, that means that when you put speech in front of lawyers, if you put that kind of thing in front of lawyers, they immediately think ‘First Amendment’ and ‘offline justice system.’

Justin Hendrix:

I'm reminded of this apparatus that Facebook's established. You've got policies developed inside the company. Those policies are enforced based on guidance the company ships off to its outsourced content moderators. The decisions can get appealed. They've got now this quasi-judicial entity, the Oversight Board, which can also make suggestions on policy. I mean, it really does look like a kind of constitutional system in some ways.

evelyn douek:

Yeah, absolutely. The argument of the paper is that it's inaccurate. That's not actually what content moderation involves, but platforms perpetuate that image nonetheless, even though they know that some of the most important decisions about content moderation happen outside that bureaucracy. They're about the affordances you give users or the amplification or de-amplification. They easily could correct the record, but it suits them to perpetuate this image of content moderation as that really limited style picture, which is why when Facebook released the Oversight Board, it talked about it in terms of the Supreme Court. That was the model that it was intending to sort of have this board be in the image of.

It would take individual cases, look at individual pieces of content and look at the rules and think are these quasi-constitutional in accordance with Facebook's values. That's the kind of model that they sort of present to regulators and legislators as well when they're saying, "Regulate us, please regulate us, but do it according to this picture." I think that that's because it's much more limited and doesn't reach far into the platform and all those other places where platforms are making really important decisions about the speech that we see when we log into their services.

Justin Hendrix:

You say this leads to an endless and irresolvable set of arguments, that this mental model just doesn't work.

evelyn douek:

I think that this paper is mainly born of a frustration of reading endless stories or tweet threads about individual decisions that platforms made to leave such-and-such post up or have such-and-such a rule about whether it's some sort of hate speech or a certain politician's tweet, for example, which are really, really important decisions. I don't want to minimize that, but it's when we're talking about individual posts that I think we're getting a little bit sidetracked in thinking about it. Particularly because when you're moderating at scale, this is now a common refrain, content moderation at scale is impossible. You're never going to get every individual decision right. We're kind of wasting our time by talking about individual decisions. What we need to be talking about is the level of the system.

The other reason why it's sort of misleading is because when you're talking about individual decisions, we're not going to resolve that. That's a fundamental question of values. We have different points of view. We have been arguing for centuries about the best way to make speech rules substantively. It also misleads us a little bit because most governmental regulators won't be able to regulate those substantive questions. That is always going to be somewhat in the hands of platforms. If we really want to make progress, we need to sort of look beyond those substantive individual frontline questions and look at the conditions of the platform and the systems that lead to those. They're all upstream of those downstream posts. They are all made before a single post is even posted or flagged for review by a content moderator.

Justin Hendrix:

You write that you're advocating for a systemic approach to content moderation regulation, which acknowledges that individual errors may be the canary in the coal mine, a systemic failure, but are not by themselves evidence of an inadequate content moderation system, that rectifying such errors will not bring overall accountability to regulators or the public. You also know that this idea that content moderation decisions are not even really, in some cases, based on content and that we have to take that into consideration.

evelyn douek:

I think we're seeing this more and more for exactly the reasons that we just talked about, which is the substantive content questions are kind of irresolvable, intractable. We're just going to keep arguing about that forever. As long as there is speech, we're going to have societal-wide conversations about 'what's the best way to deal with harmful speech?' Is it more speech? Is it censorship? I mean, those are really important conversations. That's how we constitute ourselves, but they're kind of... we're not going to reach a place, a kumbaya moment where we all come together and agree that those are the right kind of rules.

Platforms that have been caught in this endless cycle, have started looking for other signals to moderate content. They're starting to do, what I've talked about is behavioral content moderation. Where they say, "We're not taking this down based on the content of the post. We're taking it down because of the behavior of an individual or a set of accounts." Camille François and I wrote about this in terms of it grew out of information operations. The first time that platforms really started thinking about this was in the aftermath of the 2016 election because Russian information operations, most of that content wasn't particularly objectionable. It was Bernie Sanders coloring books in fluro. That's not the kind of thing that you necessarily have to take down, but what was problematic about it was the way that the actors were working together to amplify certain posts, misrepresenting who they were, trying to game the algorithms. It was sort of seen as a content-agnostic way of moving forward.

We've seen that spread from simply information operations to all sorts of things about how groups behave. I think that's swarming. These kinds of rationales were used when platforms started taking down QAnon content as well. I think that's an increasing area of content moderation that you're not going to see if you look at an individual piece of content. If you look at an individual take down, you might go, "Well, that piece of content wasn't problematic." Or if you look at an individual person, you're not going to see, "Well, yes, they were caught up in a swarm of people that were harassing a particular user." You need to look much more broadly to see that individual post in context.

Justin Hendrix:

That includes also these cross-platform, or government cooperation membranes, that are increasingly at play here. Everything from, I guess, GIFCT, on through to the way that Facebook or Twitter or the rest of the platforms may work with law enforcement. How does that play in?

evelyn douek:

I've written about this in the context of another paper, as well, called "The Rise of Content Cartels", where I talk about these collaborative structures, where platforms work both together and with government actors to find and remove content. In some ways that's really, really good. We might think about that as completely necessary to effective content moderation. If bad actors, which increasingly they are, operate across platforms, an influence operation doesn't go, "We're going to target Facebook today." They go, "We're going to target all of these attack surfaces and cross-post and things like that to be more effective." But also, governments have more information through intelligence services, or vice versa, platforms might want to alert government actors to things they're seeing on their platforms to raise awareness about threats.

But since the beginning of time, or the beginning of free speech thought, we've always been really concerned about government involvement in speech regulation. To the extent that these kinds of decisions are not transparent or that they're taking place on the basis of a government consideration rather than a platform consideration, we might think that's really problematic. It has indeed been really problematic. It might depend, for example, on what you think of the government in question. That, I think, is something that we need a lot more transparency about, but again, if a certain piece of content is taken down because a government has flagged it to a platform, you're not going to see that if you just look at the piece of content versus a particular rule.

That's a system decision, a systematic decision about this institution that they've designed, such as the Global Internet Forum to Counter Terrorism, the GIFCT that you mentioned, which is a shared database of terrorist images. Or other kinds of things like, for example, the platforms and the government working together in the run up to the 2020 election to share intelligence about information operations. You're not going to see that if you just look at an individual. A frontline content moderator, or the appeal system, is not going to understand the real considerations that went into that decision that might have informed whether it was correct or not.

Justin Hendrix:

I guess there are other forms of this delegated decision making. There's fact-checking and all the relationships with the news media?

evelyn douek:

Yeah. This, again, is increasingly a tool that platforms are using because they for so long said, "We don't want to be arbiters of truth," but then they did find themselves in a position because of societal or lawmaker pressure that they needed to take down content based on falsity. Rather than making those decisions themselves, they increasingly delegated them out. The most notable example of this is in the context of the pandemic where they delegated decision making about what they would classify as misinformation to the World Health Organization.

They're trying to wrap themselves in the legitimacy of these authoritative sources in saying, "Hey, it's not us. Get angry at the authorities." But in many cases, we don't have any transparency into how those relationships work, how often they're taking down content based on those judgments. TikTok will tell you, 'We work with fact checkers.' But I could not tell you now, and I've looked hard, how often they take down content based on fact check determinations, or how often what they send to fact checkers or necessarily who those fact checkers are all the time or what languages they use fact checking in. All of these really important institutional design questions that are upstream of those final decisions about take downs, but are going to, of course, fundamentally influence what pool of decisions there even are in terms of leave up or take downs that moderate us might be faced with.

Justin Hendrix:

We've talked a little bit about this standard model that has evolved over, I guess, the last couple of decades. We've talked about some of its weaknesses. You say all this is important because there are various legislative proposals, regulatory proposals that are premised on the idea of making platforms accountable for their moderation decisions, but that most of these reforms are based on this inaccurate understanding so they end up being accountability theater rather than accountability itself. Are there proposed reforms that you think of as emblematic here?

evelyn douek:

I think that when you think about speech decisions as an analog of the offline justice system, the way that we think about often protecting speech interests in those contexts is through due process rights. Which means, afford users more opportunity to be heard before a court, give them reasons for why a decision was taken against them, give them more notice of what the rules are, and if they're unhappy with the decision, give them more opportunity to appeal and present their case again. That's how we think about it in the offline justice system. That's the solution that the standard picture of content moderation leads to. We see a lot of this due process discourse, again, because we can't talk about substantive rules necessarily because they're intractable, but also the governments can't regulate that.

They turn to regulating procedure, which is the right move, but they do it based on this false model of what content moderation is or what will work. They focus on due process rights, but for individual users. We're seeing a slew of civil society proposals that really focus on give users notice of what the rules are, give them an opportunity to present their case, give them reasons why any decision was made against them and give them an opportunity to appeal, always to a human reviewer, if they're unhappy with their decision. That is found in many, many regulatory proposals as well.

Many of the regulatory proposals based in the United States, both in the states, for example, are Florida and Texas bills or the PACT Act or many of the proposals on the Hill, but even the Digital Services Act in the EU, which does adopt a number of sort of systemic considerations, like risk assessments and things like that, still has in it these provisions for individual due process rights. I just think that is going to be very illusory. I think that there's no evidence that giving users all of this extra information and time to present their case is necessarily going to result in more accurate decisions.

I mean, one of the drafts of the DSA says you have to have this appeal system open for six months. Six months after a Facebook post has been posted, no offense, but is it really a big value-add to be like, 'Are we really sure that this decision was right?' The world's moved on. Let's move on. I think that idea of protecting individual user rights is a very limited model of thinking about the influence that these platforms have on society because you're not reflecting a whole bunch of other interests.

If I can give one more example here. I think one of the most useful statistics from one of the Oversight Board's quarterly reports was a breakdown of the number of appeals that they received from people whose own content was taken down versus the number of appeals that they received from people who wanted someone else's content taken down. The ratio was 20 to one. 20 more times people appealed because their content was taken down then because they wanted someone else's. You can easily see the incentive there. You're much more invested in your own content than what happens to someone else's post. You are going to be much more willing to go through that elaborate process, but there's no reason to think that Facebook is 20 times less accurate at taking down posts than leaving up posts.

If you focus on individual rights, you're only going to be protecting certain kinds of interests rather than other kinds of interests. When the board, in many of its decisions, really focuses on telling Facebook, "Tell people more about which hate speech rule you took this down under. Give them more examples. Give them an appeal to a human reviewer." That's going to cost a lot. It's going to take a lot more time. It's going to slow down content moderation. I'm just not sure that it's going to achieve anything.

Justin Hendrix:

With a lot of this coming out of just the way we think about speech, particularly in this country, how we tend to misunderstand the First Amendment, what you call speech squeamishness, is there another way to look at this?

evelyn douek:

Yeah, we do think about trade-offs in many other contexts, in many other rights. The kinds of things that I was just talking about how we might not want to provide users such an extensive process because it's going to slow down the appeal system and the cost benefit analysis just isn't worth it. In due process rights in other contexts, like the administrative state, that's the kind of calculation that we engage in, but that very rarely happens in the context of speech because speech is such a sacred right. The idea that someone's speech right should be compromised for some sort of systemic consideration or that someone should be silenced because somebody else should speak, that's just not how speech rights are thought about, especially in the United States with the First Amendment.

If anything, speech jurisprudence really errs on the side of caution. It says we can't even have rules that suggest chilling effects and things like that. That might be the right calculus in the offline context. I mean, we can argue about that separately and many people do and have for a very long time, but in the online context where the scale is such that we have to talk about error rates, we can't talk about always protecting every individual user because you're just never going to be accurate at that level of scale. I mean, since we've been talking, Facebook has made something like 300,000 take-down decisions. You're just never going to get those right. You have to err on one side or the other. It's not always going to be the most effective to err on the side of caution in leaving speech up just because of individual rights.

But that kind of conversation about saying, "Hey, look. We're going to have more false positives in the context of taking down speech." There are countless examples of this kind of thing. There were examples in the context of the start of the pandemic where Facebook took down people like volunteer mask makers because they were trying to enforce a mask ad ban. They can't get all mask decisions right. We need to have this conversation about what kind of errors do we want to prefer? Do we want to err on the side of more mask ads getting through and possibly counterfeit masks? Or do we err on the side of enforcing more effectively against mask ads, but that might mean that there's some collateral damage? That's an uncomfortable conversation to have, but I think it's just going to be a necessary one because it's illusory to focus again on getting every decision right.

Justin Hendrix:

I'm reminded in what you're describing of the decision that some of the social platforms take in advance of elections or in anticipation of potential civil unrest and important moments where they dial up or dial down certain levers. That's it. It's always this decision between do we prize free expression in this context or do we prize perhaps safety or some other value?

evelyn douek:

In the context of the lead up to certain elections, or for example, in the lead up to the George Floyd trial, Facebook announced that it was taking what it called break-glass measures. It was slowing down amplification on its platform. It was enforcing more rigorously against things that appeared to be inflammatory or false. It was turning down the dial on how much it amplifies things to get tons of reactions or things like that. They announced these measures. The natural reaction to that would be, "Well, why don't you do that all the time? Why don't you just turn down the dial, make things move a little bit more slowly, stop amplifying speech that's getting a lot of exposure and maybe enforce your rules more effectively?"

The response the platforms give in those cases is, "Well, when we do that, we have a lot more false positives. We are turning down the dial on a lot of stuff that you might think is valuable speech, but we think that's the right move in these high-risk situations. We think that if there's a civil unrest or some sort of volatility, maybe it makes more sense to take down things that might be provocative, that might be inflammatory because the risk is just so high." Whether that's true or not, we need data. We need to get independent verification of those claims. I have no idea and that's totally unacceptable, but that's a reasonable conversation to have. That's a reasonable trade-off.

We might say, "Well, when things are pretty stable, maybe we are willing to tolerate more borderline speech because you might also lose some valuable speech, if you are tolerating too many false positive." But when things are in a state of unrest and sort of on the edge, maybe we want to be much more risk-averse and take down things that may be valuable, but in the effort of taking down things that may be inflammatory in such a situation. Our risk tolerance needs to change based on context. It can't just be we'll get it right all the time in any context.

Justin Hendrix:

I'm reminded of the challenge we see right now in this situation in Ukraine and the Russian invasion and the kind of just dreadful decisions that we're having to make about how to manage more propaganda and other violent content out of that region.

evelyn douek:

Right. In a context where there's such a flood of it, it's just happening at volumes that are just so difficult to manage and so hard to distinguish one piece of content from another piece of content. But we are also seeing the real value of a lot of this content, even if it might usually be flagged by tools that are trained to look for gory content or things like that. This content has been extremely valuable, for example, in collecting evidence for future prosecution of war crimes or documenting troop movements or getting the word out about the horrors that are happening in Ukraine at the moment. How platforms should think about their risk tolerance in that context is a really ... I don't envy the decision makers, but just to say to them, "You need to get it perfect," isn't going to advance the conversation. I think it's a really powerful example of how false positives could be really, really costly.

Justin Hendrix:

You're calling for an approach to content moderation that's focused on "ex-ante institutional design choices involved in creating a system of mass administration rather than ex-post individual error correction." You think this opens up new opportunities, mechanisms for regulatory reform because it draws on ideas from the new governance school of administrative law. Now, let's back up a second for those of us who aren't lawyers. What is 'new governance'? Why does it matter here?

evelyn douek:

New governance isn't particularly prescriptive. It's just a label that says we need to involve private actors in the enforcement of public rights. I think that should be a fairly uncontroversial claim in the context of content moderation. I think we should be at a place where we need to accept that platforms will, for the foreseeable future, maybe forever, be the frontline decision makers when it comes to speech. They're the ones that are going to be making the substantive decisions. They're the ones that are going to be applying them, but we're also in a place now where we're not prepared to let them do that with a complete lack of accountability, without any transparency. There needs to be a partnership between the government and these private actors in thinking about how we regulate our public sphere.

New governance comes under a lot of attack in many other forums for being pretty weak. The idea being that we need the government to set more prescriptive norms for industry rather than relying on them to make their own decisions. The problem in the content moderation context is we're talking about speech. The government can't make a lot of really prescriptive norms about what platforms can, should or shouldn't do. That includes things saying to platforms, 'You need your error rate between X,' or, 'You need to err on the side of more false positives than false negatives.' That's going to be beyond power of the government. It's going to be unconstitutional. We're kind of in a place where we just have to accept that platforms are going to be making some of these really important decisions over our public sphere that involve public interest, but finding a way to bring real transparency, and not just transparency or accountability theater, to that kind of activity is really important.

Justin Hendrix:

The contribution to this paper is not just to describe what's wrong with this current mental model, but also to look at what you call a second wave of content moderation institutional design, this framework for making content moderation systems more accountable to regulators and the public. It may seem like a silly question, but why is accountability important in your view to this second wave?

evelyn douek:

I don't think it's a silly question at all. Like I mentioned, this is my job market paper. I have had the joy of presenting it dozens of times in the last few months. I've been grilled by many law faculty on it. There are many people that ask exactly the same questions, many law professors that say, 'Well, why do we care about accountability when they're private companies?' We might often think that they're just private businesses, they're just offering a service. We don't think about accountability necessarily in many other contexts. Although, I think actually we do care about accountability for certain safety norms in many other contexts, but the idea being the accountability comes through the market. They're private actors. To the extent that we care about accountability, that's where we should focus.

I just find that very unsatisfactory. It's true that First Amendment doctrine has often said that the First Amendment only protects against the state, the government's infringements on speech and you have no free speech right against Facebook or Twitter. They can do what they want. But to me, it's true, it's just obvious right now that Facebook and Twitter make far more speech decisions in a minute than the government will make on any given year, perhaps a decade. To cede the public interest in that situation just because they're private businesses feels unsatisfactory to me. I think that it's not enough to say, 'Well, they're private businesses.' I think the fact that when they have such a huge impact on some of our most important speech forums, that there needs to be transparency and accountability for how they're doing that.

Justin Hendrix:

You see this second wave as possibly creating this new opportunity to apply this new model, to think through what we could do differently if we kind of change our mindset. How do you see things working? If everybody kind of read this paper and started, not from scratch, perhaps, but decided to pursue some of the general points you're making here, what does content moderation look like in a decade's time? What does content moderation regulation look like in a decade's time?

evelyn douek:

I mean, the first intervention that I want to make is mostly negative to say, "Whoa, whoa, whoa, stop." All of this regulation is coming down the pipe based on this false model that I think is going to be a waste of time. This individual rights model where we're seeing all of these sort of prescriptions for, as I was saying, individual appeal rights. I want to say that is a false model. That's going to be ineffective. You're going to entrench a model of content moderation that suits the Facebooks and the YouTubes of the world just fine. They are, in fact, advocating for that. It will entrench the dominant platforms and the dominant way of review.

Then the second thing is to say okay, so what? What should we do instead? There is a need to do something, but that doesn't mean we should do something sort of ineffective. It's worth taking a moment to pause. I want to get away from that individual ex-post model of let's look at what happens in the content moderation, the downstream things, and think about upstream, ex-ante decisions the platforms make that create that universe.

I talk about a number of different measures in that, that talk about institutional design, both structural mandates and procedural mandates. For example, separation of functions is one of the things that I talk about. That looks at, for example, I think one of the biggest discontents that people have with content moderation at the moment is the platform say they have these rules, and we're not necessarily sure that they're applying them in practice. We have no idea whether they are actually interested in the neutral application of rules. Indeed, based on a lot of reporting, in particular with respect to Facebook and in the Facebook files, it's come to light that the content moderation team, the content moderators would come to a certain decision with respect to some content and another arm of the company will interfere with the enforcement decision. Whether it's a government lobbyist or growth teams who are interested in their own incentive structure. They're responding to their own incentives, but they're not aligned with the application of content moderation rules.

You could think about correcting that ex-post. You could think, 'Okay, well, they've interfered. There's bias in that enforcement decision. The decision's wrong. Let's give them an appeal. Let's give them more notice and we'll fix it on appeal,' but that just seems both slow and ineffective. You're only going to have an arbitrary pool of the kinds of people that will go through that process of appeal. I think it's more effective to try and prevent that bias from infecting the process upstream ex-ante and say, 'Look, let's just put a wall between those decision makers who are responding to certain kinds of incentives and those decision makers that are responding to the incentive to enforce content moderation rules.' We see that in lots of areas of the law and lots of regulation. We see that in banks. We see that in the administrative state where we say, 'You can't have people who have one kind of incentives interfering in the enforcement decisions of another of the business.'

Justin Hendrix:

You talk about some specific things that you want to have happen. You want annual content moderation plans and compliance reports. I do believe some of those ideas are in some current legislative proposals, quality assurance and auditing, the idea of aggregated claims. Then you lay out some ideas around enforcement.

evelyn douek:

We are definitely seeing regulators moving to these kinds of models much more. Like I mentioned, the Digital Services Act before that focuses on these risk assessments, these sort of plans that say, 'Tell us what you're going to do, what your rules are, how you're going to enforce them and what risk factors there might be to the effective enforcement of those plans?' Now, the DSA ties that to substantive outcomes, like substantive risks to society. I have concerns about that because how we evaluate the risk that certain speech might have to society is a really difficult question and, I think, would be unconstitutional in the United States to pass that kind of legislation.

But to focus on risk assessments to the effective enforcement of a platform's own rules, I think, is much more feasible. This is an incremental measure. It's admittedly sort of a modest proposal, but I think it gives us the tools that can be a midway to maybe future, more prescriptive rules or industry standards, but we're still learning so much about how these platforms work and what works in terms of interventions as well.

Some of my favorite examples are around nudges and friction that platforms are implementing that platforms could say it's part of their risk assessment plans. 'One of the things that we're doing to make sure that we are effectively enforcing our disinformation or misinformation policies is prompting users to be more thoughtful in what they're sharing.' Twitter has some fantastic stats here. My favorite is that when you go to retweet an article that you haven't read, a little pop-up comes up and says, 'Hey, do you want to read this article before you retweet it?' You can totally override that. It's a totally gentle nudge, but Twitter says that 40% more people click through and read that article before they click the retweet button. That's awesome. That's really, really cool. That seems like an effective measure that maybe other platforms should consider.

But I'd think it would be a mistake to regulate, to impose that in legislation right now because every platform is different. Most platforms don't have a retweet button. We don't know if that'll still be the most effective measure in five years. We don't know whether it will be the same effectiveness on different platforms as it is on Twitter. We don't know whether that will be entrenching a kind of model that other platforms, it will be too expensive for them to comply with and will be anti-competitive, all of those sorts of things. I think that kind of ex-ante transparency about the affordances and different experimentations that platforms are taking could increase the pool of knowledge that we have as outside observers and regulators have as an incremental mid-step beyond maybe being more prescriptive in the future.

Justin Hendrix:

You talk about this virtuous cycle of regulatory public industry learning. Now, I'm interested in these legislative proposals that seem to kind of build that in a little bit. Thinking a little bit about the Digital Services Oversight and Safety Act that Lori Trahan and Adam Schiff and Sean Casten, all representatives in Congress, introduced. Are there bits of that legislation, or other legislative proposals in the States, that you regard as positive or optimistic with regard to this framework you're proposing?

evelyn douek:

Yeah, no, definitely. I mean, I think that bill in particular does reflect a lot of the similar kinds of ideas that I talk about in this paper. I was looking at it. It has things like the complaints mechanism that I talk about in my paper. They have whistleblower protection. They do have these risk assessment measures and they are really focusing much more on those ex-ante decisions, which I think are really important and I think reflects so many of the ideas as well. I mean, the name is even close to the Digital Services Act in the EU. I do think that's really important.

I think there's a lot of really encouraging stuff in there, but I do think that the standard picture that we talked about right at the beginning is still kind of inherent to this model. You can see, for example, in section five of that bill, that they're still talking about that appeal system that we mentioned at the beginning. They say for a period of at least six months, they need to have this internal complaints-handling system open to the user that gives them notification or opportunity to appeal, and things like that, but again, I just think is a misdirection would be an ineffective way of holding these systems to account. Including, as well, the transparency reporting that then imposes, which says, 'Tell us how many pieces of content you took down,' and things like that, which those transparency reports are not cheap to produce. Facebook and YouTube love them. They write op-eds in The Washington Post and The Wall Street Journal saying, 'Yes, we really think this is very valuable and you should impose this on all companies.' But I've read so many of these reports right now and I'm none the wiser about the health of the information ecosystem or the effectiveness of Facebook's enforcement of its rules due to these reports of how many individual decisions they took down.

I think there's a lot promising in there and I do think these ideas of ex-ante risk assessment and institutional design, we're starting to see them much more in the legislative models that are coming out now rather than the ones that we had three or four years ago even, the NetzDG in Germany and things like that. But I do think that we still need to move away from this prioritization, or this fundamental protection of individual user rights.

Justin Hendrix:

With just the couple minutes we've got left, I want to just zoom back out. Reading this, I was thinking to myself that clearly this is all premised on being a sort of a body of thought and a body of law within a kind of Western democratic context. There was one paragraph in particular where you wrote, "Constitutional obstacles aside, the sheer scale of speed and technological complexity of content moderation means state actors could not directly commandeer the operations of content moderation. This is a descriptive, not normative observation. The state simply does not have the capacity to assert platforms as the frontline of content moderation." Yes, here. Of course in China, we see almost exactly that, where people are, in fact, even internalizing the interest of the state and doing that on their own.

evelyn douek:

I still think that in many of those cases it's about strong-arming the platform frontline regulators. For example, if we look at what Russia is doing at the moment, it's not commandeering the frontline content moderators and putting their own Russian government enforcement people into making content moderation decisions. It's saying to Facebook, 'We're going to do X to you or your employees if you don't make your own content moderators make decisions in accordance with our own preferences.'

Now, you're right that China has a much broader infrastructure for making those decisions itself. It pays a lot of people, but I think it also still relies on that kind of strong-arming, or jawboning as we talk about it in this context, of the private companies themselves in threatening them with consequences if they don't make decisions in accordance with the government's preferences.

Of course, the other thing about China, and I mean, this is sort of generous in many ... I don't expect the Chinese government to accept my recommendations in this paper. A lot of the things I'm talking about here are not necessarily for every single audience, but content moderation becomes much easier if you don't care at all about false positives. If you, for example, don't care that you're going to take down a lot of innocuous posts or parents organizing child after-school activities if you remove every mention of tennis when you're trying to scrub your social media platforms of evidence of Peng Shuai's disappearance or discussion of that, then content moderation is simple. You just take down, you just sort of obliterate tennis temporarily from the Chinese internet. Your error rate is enormous, but you don't really care. But, if you don't really care about protecting certain speech, it's a much simpler question. That's, obviously, not the kind of calculation that we engage in in most developed Western democracies.

Justin Hendrix:

My last question is really just around a kind of technological aspect of this. These platforms very much would like to automate more and more of this. They are trying to train AI systems. They're trying to minimize the number of human moderators they have to employ. How does the advance of technology change any of this or change the way we should think about content moderation?

evelyn douek:

This is another reason why regulation in this space is so difficult because it's so fluid and changing so quickly. I think it's another area where our conversation needs to be a little more nuanced because there's broadly a lot of negativity about automated moderation as being really infringing on user rights and user speech rights. But, moderation without automated tools would be a disaster. They really are very necessary.

There's other considerations to think about as well, including the human costs of having human moderators look at the worst aspects of the internet all day, every day. We might have different considerations in different contexts. For example, we might tolerate errors in certain contexts where the material is so abhorrent and the value of the speech, or the error rate, is so low that we want to protect human moderators. Child sexual abuse material might be a good example in this context.

But error rates are changing constantly in terms of what automated moderation can do and looking for ways to incentivize platforms to keep developing those tools, rather than entrenching the way that the content moderation systems currently work, I think is a better way of thinking about that. I think we need to have a more sophisticated discussion about how the different error rates and equities as well in terms of appeal rights. This idea of always appealing to a human reviewer will necessarily increase accuracy, I'd like to see some data supporting that because humans are pretty fallible too. Humans often make mistakes. Human error is endemic in content moderation as well.

I think just this fundamental assumption that underlies a lot of proposals and discussion that humans are definitely going to be more effective than AI, or even if they are now, will always be, is one that needs examining or establishing. That's why I think one of the things that we can probably all agree on is opening up these platforms to get more data, to allow independent researchers access to examine these assumptions and examine this information about what's going on in our most important public spheres is really important.

Justin Hendrix:

I know that you, in your podcast ... this is my last question really. In your thinking, you have a similar kind of fascination with January 6th and the investigation of January 6th as a phenomenon. How do you think that relates to this work in your mind or does it?

evelyn douek:

I think that was a systemic problem. I think that thinking about that, I mean, it's a perfect example of how thinking about that in terms of individual posts or individual decisions is not a great way of thinking about this.

I mean, another way that illustrates the argument is a lot of conversations about January 6th, and in the lead up to January 6th, happened in things like private groups or spaces where relying on people to flag the content or complain about the kinds of content is going to be a pretty flawed model because they were quite happy to be sort of engaging in those kind kinds of conversations and breaking the rules. I think that thinking about how systems can be used and abused in advance of those kinds of events, rather than trying to fix it up afterwards when it's probably too late, is really important. Thinking about the risk factors in terms of things that we were talking about before, like collaboration between platforms and government actors and what might work more effectively in future such incidents and what needs to have more transparency around it, I think is really important. It should be a learning opportunity, as well as an example of systemic failure.

Justin Hendrix:

Evelyn, thank you so much.

evelyn douek:

This was a pleasure. Thank you.