On the home turf of the First Amendment,² the most speech-protective national law in the world, private corporations have built and implemented larger systems for censorship than any government ever has.³ Tech companies that own and govern social media platforms now take down millions of pieces of content every day for violations of their substantive rules, determining which content billions of people may transmit or receive.⁴
As those private systems of governance have grown, governments and activists have pressed for changes in the use of that power. They have demanded to know the internal company rules under which content is removed⁵ or suppressed⁶ criticized some of those corporate rules,⁷ and insisted that companies comply with speech laws, both national⁸ and international.⁹
But this is insufficient for making sure that content moderation is just: the rules alone do not reveal their impact on the billions of people who live under them. Rule enforcement is a vital part of the picture, since facially neutral rules or laws can be enforced very differently on different groups of people. Just one striking example, the use of drug laws to imprison Black Americans far more often and for much longer than whites and members of other racial groups, was described in detail by Michelle Alexander in her landmark book, “The New Jim Crow: Mass Incarceration in the Age of Colorblindness”.¹⁰ Ms. Alexander was able to document stunning disparity and injustice, with public records.
In the case of speech governance by tech companies at scale, almost no data is publicly available, and it is difficult to obtain even for scholars.¹¹ Still, information that has been gleaned by multiple investigators indicates that rules are enforced inequitably for people of different groups constituted by ethnicity, nationality, ‘gender, religion or belief, and so on.¹² Inequity may not be a deliberate policy on the part of companies, and some of it is difficult to avoid.¹³ For example, companies typically have better content moderation, and have built more automated tools for detecting prohibited content, in the predominant languages of large, lucrative markets than in languages spoken by fewer people in poorer countries.
Intentional or not, on the vast scale at which tech companies govern speech, the effects of inequitable governance are likely large and significantly hinder the exercise of freedom of expression and other rights of people around the world. The impact is likely to be especially serious for those living in armed conflict, crisis, or oppression, like Palestinians and some Israelis, for whom social media platforms are often a vital communications lifeline.¹⁴ Therefore, content moderation should be studied systematically for disparate impact among other flaws, so that it can be corrected. This requires regular auditing as proposed below.
Most external demands for change in content moderation have come in response to takedown of specific posts, since that is generally the only way outsiders can perceive where companies are drawing lines between permissible and prohibited content.¹⁵ Public critique of corporate content moderation has therefore focused largely on a few (albeit notable) drops in a vast ocean of speech governance.
An illustrative example was Facebook’s 2016 removal, under its rule against nudity, of a 1972 photograph of a Vietnamese girl running as the chemical weapon napalm burned into her bare skin. Norwegian writer Tom Egeland posted the image as one of “seven photographs that changed the history of warfare.” Strongly opposed to its suppression, other Facebook users including Norway’s then-Prime Minister Erna Solberg reposted it, only to have it taken down again. Norway’s biggest newspaper then published a front-page letter to Mark Zuckerberg, Facebook’s¹⁶ CEO, accusing him of abusing his power as “the world’s most powerful editor” of “the world’s most important medium.”¹⁷ After two weeks, Facebook reversed itself and restored the photograph, conceding (as the critics had argued) that its historical importance outweighed the fact that a child was unclothed.¹⁸
In 2018, after years of other public criticism of Facebook decisions, the company decided to give handpicked outsiders a structured opportunity to criticize its rulemaking and content moderation, and even granted them power to overrule some of the company’s decisions. That group, named the Oversight Board, began work in late 2020 and at this writing had twenty-four members and had published thirty-six cases. All of the cases turn on specific pieces of content or accounts, such as a poem in Russian, two cartoons, and a profane reference to the Chinese state. The Board has no authority to get access to data on Meta moderation at scale.¹⁹
The Board has decided one case related to Israel and Palestine, again based on a single post. On May 10, 2021, a Facebook user in Egypt shared a news article about a threat from a terrorist group that Al Jazeera (a news organization) had already posted on Facebook. The Egyptian user added only one word, neither endorsing nor condemning the threat—Arabic for “ooh”—yet Facebook’s moderators took it down under its Dangerous Individuals and Organizations (DIO) policy. (Al Jazeera’s original post was never removed.)
As soon as the Board chose to review Facebook’s decision to delete the post, Facebook reconsidered and restored it. As expected, the Board ruled that the post should not have been removed since it was nearly identical to the Al Jazeera post. It also advised that Facebook revise the DIO policy to better explain the exceptions for neutral discussion, condemnation, and news reporting, to avoid similar mistakes in the future.
The Board also touched on the much bigger underlying issue of disparate content moderation by recommending that Meta commission an independent entity to conduct “a thorough examination to determine whether Facebook’s content moderation in Arabic and Hebrew, including its use of automation, have been applied without bias.”²⁰ This focus on bias made the Board’s request much narrower than what advocates had asked for in public comments on the case. Mnemonic, an NGO that documents human rights violations, urged the Board to recommend a full, independent public audit of content moderation with respect to Palestine.²¹ The NGO Access Now also urged the Board to ask Facebook to conduct a similar full, independent, public audit of content moderation, with respect to both Israel and the Occupied Palestinian Territories.²²
In response Meta commissioned a “human rights due diligence exercise” by the nonprofit consultancy Business for Social Responsibility (BSR) but narrowed the scope of the exercise even further from what the Board recommended, confining it to May 2021, the month of the post at issue in the Oversight Board’s Al Jazeera case. That was a period of increased bloodshed and tension, with a surge of violence between the 10th and 21st, including Palestinian rocket attacks and Israeli airstrikes in which at least 260 Palestinians were killed and 12 people were killed in Israel.²³
BSR found what it described as “various instances of unintentional bias where Meta policy and practice, combined with broader external dynamics, does lead to different human rights impacts on Palestinian and Arabic speaking users.”²⁴ It found that Meta had impinged on Palestinian freedom of expression, among other rights: “Meta’s actions in May 2021 appear to have had an adverse human rights impact. . .on the rights of Palestinian users to freedom of expression, freedom of assembly, political participation, and non-discrimination, and therefore on the ability of Palestinians to share information and insights about their experiences as they occurred.”²⁵
There was more erroneous takedown of Arabic-language content than Hebrew content on a per user basis, BSR reported, and more proactive detection of potentially rule-violating Arabic content than equivalent Hebrew content. Similarly, there was more underenforcement of Hebrew content and Arabic-language content. Meta, BSR noted, had an Arabic hostile speech classifier, but no Hebrew hostile speech classifier. Still some Hebrew content was mistakenly taken down, and some Hebrew-language accounts were erroneously restricted. BSR also expressed concern that “clearly antisemitic content that violated Meta’s policies” was not detected and removed in May 2021, according to organizations that track antisemitism.²⁶
BSR concluded that the bias was not intentional on Meta’s part.²⁷ The effect on people who use Meta’s platforms would be the same whether bias was intentional or not, though. Moreover, as Human Rights Watch pointed out in its comments on the BSR report, critics like HRW itself had long been calling Meta’s attention to the apparently disproportionate impact of its content moderation on Palestinians. Meta apparently failed to investigate and correct the problem adequately so “the unintentional became intentional,” according to HRW.²⁸
Though a useful start, the BSR review was too limited, in many ways. It covered only a few exceptional weeks when Meta was paying especially close attention to Israel and Palestine as a “temporary high-risk location,” so content moderation may have been more accurate and well-informed than usual. Although BSR had access to some internal Meta data, it did not have access to all relevant data, and as the authors noted, even the data they did receive “does not meet the quality, confidence, and assurability requirements for detailed and formal external disclosure.” Finally, the exercise was a one-off effort, which prevented the authors from making important comparisons between Israel and Palestine, and between each of the two areas over time, and between acute crises like May 2021, and more ordinary periods. As BSR itself noted, “there is no established ‘ground truth’ for what absolute or relative rates of content enforcement should be in Israel and Palestine, as no data is available for the prevalence of violating content at the country level.”²⁹
Even without access to data and cooperation from tech companies, other advocates and researchers who have looked for inequities in content moderation have found large ones, in other languages and parts of the world. For example, independent researchers at Equis Labs and Avaaz found, in separate studies, that Spanish-language misinformation about COVID-19 and related vaccines, and about the 2020 U.S. election, was much less likely to be taken down or corrected than misinformation in English.³⁰ And in India, Facebook’s largest market, Equality Labs did an exhaustive study of hate speech against Indian minorities on that platform, and concluded that ninety-three percent of all hate speech posts reported to Facebook were left up, reflecting “a near total failure of the content moderation process”³¹—which is surely not the case regarding hate speech in other markets, and against other groups. Equality Labs called for “an aggressive and thorough human rights audit” including hiring practices, contractor demographics, and slur lists in content moderation³² but did not call for an audit of Facebook’s enforcement of its rules.
Such independent reports only scratch the surface of content moderation by Meta and other companies. The only way to discover and correct inequity in content moderation—not to mention other flaws in moderation that are outside the scope of this paper—is to build a system of regular auditing. In myriad industries whose products and services can harm the public, from air travel to food processing to finance, oversight by external auditors is routine.³³ Tech companies wield enormous power without systematic oversight, largely because their industry is young.
Professional auditors should be granted regular access to relevant data, under strict rules to protect the privacy of social media users and their data. They will need a set of standards to guide their audits; these should be developed in a co-design process, including members of the marginalized groups that seem to be disproportionately affected by content moderation. The auditing system should be international in scope, of course, just as social media is. Auditors would provide regular reports to companies, and to the public, while protecting user privacy and the trade secrets of companies.
Such a major undertaking will have to start with a pilot project in a specific region of the world, to demonstrate value and feasibility. Israel and Palestine would be ideal for such a test, for the following reasons. First, people in both countries produce abundant violating content, even when they are not in the midst of a surge of violence, so there would be plenty of data, alas. Also, there is already evidence of both over- and under-enforcement, so the audit would easily capture both. Finally, civil society is heavily involved in both Israel and Palestine, so the auditors’ reports would not languish unread—activists would push for change based on their findings. Meta should commission such an audit, and if it does not take the initiative to do so, the Oversight Board should recommend a detailed study of the enforcement of rules in content moderation. That effort, and lessons learned from it, can lead to similar audits in other parts of the world and eventually to standard industry practice. Without regular auditing, freedom of expression and other rights cannot be sufficiently protected—just as food processing plants must be inspected to protect public health³⁴, and airplanes are regularly scrutinized (and their failures investigated) by professionals outside the companies that manufacture the planes.
Harm related to online discourse can be much harder to detect than food poisoning or plane crashes.
* Executive Director, Dangerous Speech Project and Faculty Associate, Berkman Klein Center for Internet & Society, Harvard University. I am grateful to Tonei Glavinic for outstanding ideas, research, and editing, to Prof. Omar M. Dajani of the University of the Pacific McGeorge School of Law for organizing the symposium on which this special issue is based; and to the other symposium participants for their insightful contributions.
2. U.S. CONST. amend. I. Most of the biggest social media and messaging platforms with global reach are based in the United States, including Facebook, WhatsApp, Instagram, YouTube, Snapchat, and Twitter.
3. The only country whose censorship system comes close to that of Meta in number of users or volume of content regulated is China, which has fewer than one billion people online. Facebook alone has 2.895 billion monthly active users. Compare Emily Birnbaum, Meta’s Nick Clegg Warns Against ‘Industrial-Scale’ Censorship, BLOOMBERG (Sept. 22, 2022), https://www.bloomberg.com/news/articles/2022-09-22/meta-s-nick- clegg-warns-against-industrial-scale-censorship (on file with the University of the Pacific Law Review) (indicating Meta, which owns Facebook, is capable of the “greatest industrial-scale censor ever in human history”), with FREEDOM HOUSE, CHINA: FREEDOM ON THE NET 2021 COUNTRY REPORT (2021), https://freedomhouse.org/country/china/freedom-net/2021 (on file with the University of the Pacific Law Review) (indicating China’s “Great Firewall” is the “world’s most sophisticated censorship apparatus”).
4. In 2022, Meta removed or restricted more than 601 million pieces of content on Facebook and another 185 million on Instagram for violating substantive platform rules, against hate speech for example. In just the first three quarters of 2022, TikTok removed over 327 million videos, and YouTube removed 429 million (as of this writing, these platforms have not published data for Q4 2022). See Community Guidelines Enforcement Report July 2022 – September 2022, TIKTOK (Dec. 19, 2022), https://www.tiktok.com/transparency/en/community-guidelines-enforcement-2022-3/ (on file with the University of the Pacific Law Review) (reporting that TikTok removed 102,305,516 videos from January to March 2022, 113,809,300 videos from April to June 2022, and 110,954,663 videos from July to September 2022). This does not include millions more posts that companies remove daily for other reasons, such as spam or copyright violations.
5. Over the years, platforms have moved from disclosing only vague general rules, prohibiting nudity or hate speech for example, to explaining how they define such terms. Kate Klonick, The New Governors: The People, Rules, and Processes Governing Online Speech, 131 HARV. L. REV. 1598, 1648–58 (2018). Still, they keep the intricate decision making guides used by human moderators (and fed into machine learning models) secret. Only a few details have been occasionally leaked to journalists, see e.g., Julia Angwin & Hannes Grasseger,Facebook’s Secret Censorship Rules Protect White Men from Hate Speech but Not Black Children, PROPUBLICA (June 28, 2017), https://www.propublica.org/article/facebook-hate-speech-censorship-internal documents-algorithms (on file with the University of the Pacific Law Review).
6. Takedown or removal is only one way in which companies control online discourse. They also make extensive use of several methods to suppress content without actually removing it, such as downranking certain content in users’ feeds, or keeping it out of search results. These tools for speech governance are used largely in secret: companies release relatively little information about takedown, and virtually none about suppression of content.
7. See, e.g., The Terms, CHANGE THE TERMS, https://www.changetheterms.org/the-terms (last visited March 18, 2023) (on file with the University of the Pacific Law Review). In another example, tens of thousands of women protested Facebook’s removal of images showing breastfeeding, until the company changed its rules to permit them. Chris Matyszczyk, Facebook Actually Sorry for Banning Breastfeeding Pic, CNET (Apr. 7, 2013), https://www.cnet.com/culture/facebook-actually-sorry-for-banning-breastfeeding-pic/ (on file with the University of the Pacific Law Review).
8. See, e.g., Network Enforcement Act (Netzdurchsetzunggesetz, NetzDG), GERMAN L. ARCHIVE, https://germanlawarchive.iuscomp.org/?p=1245 (last visited Mar. 18, 2023) (on file with the University of the Pacific Law Review) (requiring social media companies to take down content that is “manifestly unlawful” under German law, within 24 hours).
9. See, e.g., David Kaye (Special Rapporteur), Rep. on the Promotion and Prot. of the Right to Freedom of Opinion and Expression, U.N. Doc. A/HRC/38/35 (Apr. 6, 2018); ARTICLE 19, SIDE-STEPPING RIGHTS: REGULATING SPEECH BY CONTRACT (2018), https://www.article19.org/wp-content/uploads/2018/06/Regulating-speech-by-contract-WEB-v2.pdf (on file with the University of the Pacific Law Review).
10. MICHELLE ALEXANDER, THE NEW JIM CROW: MASS INCARCERATION IN THE AGE OF COLORBLINDNESS (2010).
11. See generally J. Nathan Matias et al., Manifesto: The Coalition for Independent Technology Research, COALITION FOR INDEP. TECH. RSCH. (Oct. 12, 2022), https://independenttechresearch.org/manifesto-the-coalition-for-independent-technology-research/ (on file with the University of the Pacific Law Review).
12. A handful of scholars have begun to explore this issue recently, see e.g., Tarleton Gillespie, Do Not Recommend? Reduction as a Form of Content Moderation, SOCIAL MEDIA + SOC’Y, July–Sept. 2022, at 1; ÁNGEL DÍAZ & LAURA HECHT-FELELLA, BRENNAN CTR. FOR JUST., DOUBLE STANDARDS IN SOCIAL MEDIA CONTENT MODERATION (2021); Oliver L. Haimson et al., Disproportionate Removals and Differing Content Moderation Experiences for Conservative, Transgender, and Black Social Media Users: Marginalization and Moderation in Gray Areas, 5 PROC. OF THE ASS’N FOR COMPUTING MACH. ON HUM.-COMP. INTERACTION, no. 466, Oct. 2021. And NGOs have long criticized tech companies for discriminating against certain groups with their content moderation. See, e.g.,7AMLEH – THE ARAB CTR. FOR THE ADVANCEMENT OF SOC. MEDIA, FACEBOOK AND PALESTINIANS: BIASED OR NEUTRAL CONTENT MODERATION POLICIES? (2018); see also Statement Regarding BSR’s HRA for Meta on Palestine and Israel, HUM. RTS. WATCH (Sept. 27, 2022), https://www.hrw.org/news/2022/09/27/statement-regarding-bsrs-hra-meta-palestine-israel (on file with the University of the Pacific Law Review) (“[W]e have been calling Meta’s attention to the disproportionately negative impact of its content moderation on Palestinians for years.”).
13. The reasons for inequities in speech governance are diverse and complex, including conflicting priorities in companies’ relationships with various states and other external and internal actors. See, e.g., Chinmayi Arun, Facebook’s Faces, 135 HARV. L. REV. F. 236, 236-64 (2022).
14. See, e.g., Public Comment Appendix for 2021-009-FB-UA, OVERSIGHT BD. 22–23 (Sept. 2021), https://www.oversightboard.com/decision/FB-P93JPX02/ (on file with the University of the Pacific Law Review) (listing 7amleh’s Public Comment No. PC-10169); id. at 16–17 (quoting Lara Friedman’s Public Comment No. PC-10159) (“It is impossible to overstate the importance of social media to Palestinians everywhere. It is especially important for Palestinians living under Israeli military occupation in the West Bank, East Jerusalem, and Gaza Strip – locales in which traditional media and major human rights organizations sometimes have difficulty operating . . . .”).
15. There have been some attempts to study takedown systematically, see e.g., JACOB MCHANGAMA & LUKAS CALLESEN, JUSTITIA, THE WILD WEST? ILLEGAL COMMENTS ON FACEBOOK (2022), https://justitia-int.org/en/the-wild-west/ (on file with the University of the Pacific Law Review), who analyze whether content removed on Facebook violated Danish law. However, such efforts have serious limitations due to insufficient data access: for example, the authors of that study could not determine which content had been removed by company moderators, and which was removed by the user who posted it.
16. Facebook changed its corporate name to Meta in October 2021.
17. Espen Egil Hansen, Dear Mark. I Am Writing This to Inform You That I Shall Not Comply with Your Requirement to Remove This Picture., AFTENPOSTEN (Sept. 8, 2016), https://www.aftenposten.no/meninger/kommentar/i/G892Q/dear-mark-i-am-writing-this-to-inform-you-that-i-shall-not-comply-with-your-requirement-to-remove-this-picture (on file with the University of the Pacific Law Review).
18. Facebook’s statement repeated the arguments of protestors: “Because of its status as an iconic image of historical importance, the value of permitting sharing outweighs the value of protecting the community by removal, so we have decided to reinstate the image on Facebook where we are aware it has been removed.” Nick Statt, Facebook Will Let You Post an Iconic Vietnam War Photo After All, VERGE (Sept. 6, 2016), https://www.theverge.com/2016/9/9/12865670/facebook-censorship-napalm-girl-aftenposten-reversal (on file with the University of the Pacific Law Review).
19. Under its current charter the Oversight Board can (1) review moderation decisions on specific pieces of content, at the request of Meta, the poster of the content, or a platform user who reported it to Meta; and (2) issue Policy Advisory Opinions, in relation to a case before the Board or at Meta’s request. Meta is only obligated to provide the Board with data “reasonably required for the board to make a decision” on a case. OVERSIGHT BD., OVERSIGHT BOARD CHARTER (2023).
20. META, META RESPONSE: ISRAEL AND PALESTINE DUE DILIGENCE EXERCISE 3 (2022).
21. Public Comment Appendix for 2021-009-FB-UA, supra note 13, at 19–21 (listing Dia Kayyali’s Public Comment No. PC-10168).
22. Id. at 27–29 (listing Access Now’s Public Comment No. PC-10171).
23. Gaza: Apparent War Crimes During May Fighting, HUM. RTS. WATCH (July 27, 2021), https://www.hrw.org/news/2021/07/27/gaza-apparent-war-crimes-during-may-fighting.
24. BUS. FOR SOCIAL RESP., HUMAN RIGHTS DUE DILIGENCE OF META’S IMPACTS IN ISRAEL AND PALESTINE IN MAY 2021, at 8 (2022), https://www.bsr.org/reports/BSR_Meta_Human_Rights_Israel_Palestine_English.pdf (on file with the University of the Pacific Law Review).
25. Id. at 4.
26. Id. at 6.
27. Id. at 7.
28. Statement Regarding BSR’s HRA for Meta on Palestine and Israel, supra note 11.
29. BUS. FOR SOCIAL RESP., supra note 23, at 2.
30. Stephanie Valencia, Misinformation Online Is Bad in English. But It’s Far Worse in Spanish., WASH. POST (Oct. 28, 2021), https://www.washingtonpost.com/outlook/2021/10/28/misinformation-spanish-facebook-social-media/ (on file with the University of the Pacific Law Review); AVAAZ, HOW FACEBOOK CAN FLATTEN THE CURVE OF THE CORONAVIRUS INFODEMIC (2020), https://secure.avaaz.org/campaign/en/facebook_coronavirus_misinformation/ (2020) (on file with the University of the Pacific Law Review).
31. THENMOZHI SOUNDARARAJAN ET AL., EQUAL. LABS, FACEBOOK INDIA: TOWARDS THE TIPPING POINT OF VIOLENCE: CASTE AND RELIGIOUS HATE SPEECH 7 (2019).
32. Id. at 9.
33. On the urging of civil society activists and some members of the U.S. Congress, Facebook (now Meta) commissioned a civil rights audit of its practices in the United States, which was carried out in 2018. The auditors produced three reports and many recommendations. They did not examine the effects of content moderation on different groups of people, though they noted that civil rights advocates “assert that Facebook unevenly enforces or fails to enforce its own policies against prohibited content.” LAURA MURPHY, FACEBOOK’S CIVIL RIGHTS AUDIT—FINAL REPORT (2021).
34. There is an important literature comparing online governance with public health. J. Nathan Matias first suggested that those who wish to make the internet less toxic study campaigns for food safety. See J. Nathan Matias, A Toxic Web: What the Victorians Can Teach Us About Online Abuse, GUARDIAN (Apr. 16, 2016), https://www.theguardian.com/technology/2016/apr/18/a-toxic-web-what-the-victorians-can-teach-us-about-online-abuse. More recently, Matias has also pointed out that while the field of public health has improved human life, it has conferred those benefits extremely inequitably. See J. Nathan Matias, To Hold Tech Accountable, Look to Public Health, WIRED (Mar. 26, 2023), https://www.wired.com/story/tech-governance-public-health/, Jonathan Zittrain examines public health as a model for making systemic decisions to maximize the public good. Jonathan Zittrain, Three Eras of Digital Governance (Oct. 7, 2019), https://dx.doi.org/10.2139/ssrn.3458435.
Loading...
Reload document 
Open in new tab