Key Findings

For this study we have analysed 135 pieces of misinformation content in five different languages (English, French, Italian, Spanish and Portuguese) that were posted in late 2020 or early 2021 and were rated false and misleading by reputable, independent fact checkers. In the graphs that follow we compare our analysis with the one we performed in 2020 , using exactly the same methodology used in the report we released in the pandemic's first months. For more information please refer to the Methodology and Data section.

Figure 1 shows how many of the posts analysed were either labelled, removed or remained unactioned on the platform during the period of our investigation. From the image, you can see how there is no improvement from last year, with actually a slight decrease in posts detected by Facebook. Another main difference is that twice as many posts were removed, which would be a positive indication that Facebook was acting more robustly, however we have no way to know if the removal was done by Facebook or by the account which had posted the content. It is important to note that Facebook announced on February 8, 2021 , after years of pressure from health experts and civil society organizations, that it would ban misinformation about all vaccines. Nonetheless, Avaaz continued to find such content on the platform.

Moreover, there is also a significant reduction in posts that remain on the platform with a label.


Figure 1 - Comparison between the amount of content removed, labelled and unactioned in our 2020 and in our 2021 analysis

In Figure 2 we break down the same data per the different languages analysed in this report. Italian and French seem to be the most neglected languages. On the opposite side, Spanish is the non-English language with the highest amount of actioned content, potentially as a consequence of the media focus it had during the recent US elections and of the work of civil society groups, including Avaaz, in flagging that Spanish language misinformation was a problem ¹¹ .


Figure 2 - Comparison between the amount of fact-checked misinformation content unactioned in our 2020 and 2021 analysis. Breakdown per language

Figure 3 shows the comparison between English and non-English content. It can be seen that a majority (56%) of the content remains unactioned in major non-English European languages ¹² , compared to only 26% of English-language content debunked by US-based fact checkers ¹³ . This leaves all users in Europe at a greater risk of seeing and interacting with COVID-19-related misinformation without any fact-checking measures.


Figure 3 - Comparison of the rate of fact-checked misinformation content in English (US), English (UK+Ireland) and in non-English that is unactioned on Facebook

The goal of our study was to estimate the amount of misinformation that Facebook was able to detect on its platform. This is why we included misinformation identified by fact checkers who are Facebook partners but also other reputable fact-checking organisations. But in order to identify the percentage of content Facebook could easily have acted on, having been provided fact checks by its partners, in Figure 4 we also look at the percentage of unlabelled content for which a fact check from a Facebook partner was available. Our data shows that for 66% of the misinformation content that remained unactioned on Facebook, the platform had been provided a fact check from a partner organisation.


Figure 4 - Percentage of unlabelled content that is fact checked by a Facebook partner vs percentage of unlabelled content that is fact checked by reputable fact-checking organisations not part of Facebook’s fact-checking network

Next, we focused on the average delay between post publication and Facebook labelling. Unfortunately in 2020 we were not able to record this delay for a significant number of examples, so we only provide it for our 2021 analysis. Avaaz cannot analyse the causes of the delay between when Facebook is notified of a fact check and when it places a label on the content or removes it. Facebook cannot, indeed should not, act before it receives confirmation from a fact-checking partner that a post contains verified misinformation. But the data in Figure 5, though only relating to 26 examples where we were capable of measuring this delay, is indicative of the efficacy of Facebook’s system in relation to its actions after receiving a fact check from one of its partners. The exact points of the delay in the system cannot be interrogated without access to further data, which Facebook does not currently provide. What is clear is that this system needs urgent optimisations.


Figure 5 - Delay between fact-checked post publication and Facebook measure (label or removal) in 2021

In Figure 6 we can see that for post delays, too, there is a difference between English and non-English content, with non-English content on average taking six more days to be labelled or removed. This means non-English speakers on Facebook waited almost one week longer than English speakers to see a label on misinformation or for the content to be removed. As we explain in the recommendations below, Facebook can fix this by being transparent with all users who have been exposed to fact-checked misinformation including through retroactive notifications.


Figure 6 - Delay between publication of post and Facebook labelling for English vs non-English content

Finally, Figure 7 shows the average delay for fact checking to be published after a post’s publication, which in 2021 has been nine days. This may be due to a number of variables, including an increased amount of disinformation to fact check or an attempt by fact checkers to focus on older or more viral disinformation on the platform. Fact-checkers are doing their best to keep up with the scale of the problem, under extremely difficult circumstances, yet our findings highlight the need for more capacity on their end. It must also be noted that while Figure 7 relies on a data set of 135 posts, Figures 5 and 6 rely on a different and smaller data set of 26 posts, as this is the number of posts for which we were able to record the exact moment when a label was applied or the post was removed ¹⁴ . This means that the nine day delay for fact checking a publication and the 30-day delay on average for labelling it cannot be directly compared, although it suggests that labelling a publication takes much longer than fact checking it.


Figure 7 - Average days for fact checking publications after they are posted

Methodology and Data Set

The investigative team analysed misinformation content about the coronavirus posted between December 7, 2020, and February 7, 2021, that met the following criteria:

1. Were fact checked by Facebook’s third-party fact-checking partners or other reputable fact-checking organisations. ^{25 26}
2. Were rated “false” or “misleading,” or any rating falling within these categories, according to the tags used by the fact-checking organisations in their fact-check article.
   • The variation of ratings description is quite broad. Here are some examples (please contact us to see the full list we used):
     • Disinformation - Factually Inaccurate - Cherry-Picking - False - Mostly False - Hoax - Incorrect - Misleading - No Evidence - Not True - Wrong
3. Could cause public harm by undermining public health. Avaaz has included content that impacts public health in the areas of:
   • Preventing disease : e.g., false information on diseases, epidemics and pandemics and anti-vaccination misinformation.
   • Prolonging life and promoting health : e.g., bogus cures and/or encouragement to discontinue recognised medical treatments.
   • Creating distrust in health institutions, health organisations, medical practice and their recommendations : e.g., false information implying that clinicians or governments are creating or hiding health risks.
   • Fearmongering : health-related misinformation that can induce fear and panic, e.g., misinformation stating that the coronavirus is a human-made bio-weapon being used against certain communities or that Chinese products may contain the virus.

Methodology for measuring Facebook labelling and removals (Figures 1, 2, 3, 4, 7)

For the purpose of measuring Facebook’s stated claims about its fact-checking efforts, the investigative team analysed a sample of 135 posts about the coronavirus based on the above criteria that were comparable to the sample analysed in our 2020 study. In order to allow a precise comparison with our 2020 study on coronavirus-related misinformation we selected only content that was published in the five languages previously covered in our 2020 report ²⁷ : English, Italian, Spanish, French and Portuguese.

For each of the false and misleading posts and stories sampled based on the above criteria, Avaaz researchers recorded and analysed, using both direct observation and CrowdTangle ²⁸ :

• The total number of interactions it received;
• The total number of views it received in the case of Facebook videos;
• Whether each had a warning label as false or misleading ²⁹ added to it by Facebook ³⁰ ;
• When misinformation posts would receive a fact-check warning label or be removed ³¹ ; and
• The delay between when the misinformation content was posted and the publication of a fact check by a reputable fact-checking organisation ³² ;

Methodology for measuring labelling and removal delays (Figures 5 and 6)

For the purpose of measuring Facebook’s delay between when the original misinformation content was posted and the date Facebook first applied its moderation policies on the post by either flagging the content as misinformation or removing it from the platform, our team analysed a data set of 26 posts. These were all the posts for which our team, accessing them on a daily basis for 43 days, was able to manually document when a label was first applied, or the post was removed.

In order to collect a significant sample, given the difficulty in recording exactly the day when a measure is applied, we considered posts in English, Italian, Spanish, French and Portuguese, but also German.

In our previous discussions with Facebook, we had requested that the platform provide more access to its systems or provide transparency on both the average time between when misinformation is posted, and when a fact-check label is applied or the content is removed, as well as the number of users that view misinformation content before it is labelled. These data points are important as a means of analysing the platform’s effectiveness in combating misinformation. We continue to urge Facebook to be more transparent about these numbers more broadly, as it is difficult for researchers to continue to manually conduct such an analysis.

Methodology for identifying “clones” and “variants”

During the research process, our investigative team noticed that posts previously documented were spreading in different languages in an exact, or slightly altered, fashion, and were collecting a large amount of interactions. Our team further investigated seven narratives from our sample of 135 posts, to conduct a dedicated research of the spread of such "clones" and "variants".

We used CrowdTangle ³³ to search text from the original post we had documented to identify public shares of the same content - or variations of it - shared by Facebook pages, public groups or verified profiles.

We only included posts when we were able to document at least one occurrence that had been labelled by Facebook but we could also find “clones” or “variants” of the same example that had not been labelled.

With such methodology our team was able to identify a total of 51 posts. The engagement data we estimate for our sample provides some indication of the relative reach of different claims.

General note on methodology

It is important to note that, while we collect data and compute numbers to the best of our ability, this analysis is not exhaustive as we looked only at a sample of fact-checked misinformation posts in five languages. Moreover, this research is made significantly more challenging because Facebook does not provide investigators with access to the data needed to measure the total response rate, moderation speed, number of fact checks and the amount of users who have seen or been targeted with misinformation.

Nonetheless, Facebook is becoming more cooperative with civil society organisations, and we hope the platform continues this positive trend. We also recognise the hard work of Facebook employees across different sub-teams, who have done their best to push the company to fix the platform’s misinformation problem. This report is not an indictment of their personal efforts, but rather highlights the need for much more proactive decisions and solutions implemented by the highest levels of executive power in the company.

This study achieved its purpose by taking a small step towards a better understanding of the scale and scope of the COVID-19 misinformation infodemic on Facebook.

Cooperation across fields, sectors and disciplines is needed more than ever to fight disinformation and misinformation. All social media platforms must become more transparent with their users and with researchers to ensure that the scale of this problem is measured effectively and to help public health officials respond in a more effectual and proportional manner to both the pandemic and the infodemic.

A list of the pieces of misinformation content referenced in this report can be found in the annex.

It is important to note that although fact checks from reputable fact-checking organisations provide a reliable way to identify misinformation content, researchers and fact checkers have a limited window into misinformation spreading in private Facebook groups, on private Facebook profiles and via Facebook messenger.

Similarly, engagement data for Facebook posts analysed in this study are only indicative of wider engagement with, and exposure to, misinformation. Consequently, the findings in this report are likely conservative estimates.