We analysed each of the 7 websites (listed in Table 1 ) and observed how topics were assigned to their claims. We noticed that amongst all these websites, Checkyourfact does not classify claims according to topics. From the remaining 6 websites PolitiFact , 7 Africacheck , 8 FullFact , 9 Snopes , 10 Truthorfiction , 11 and AFP Factcheck . 12 ,we compiled a list of all prevalent topics and created a list of 401 varied topics. After sifting through this list of topics, we performed the following steps:

We selected the most prominent and frequently occurring topics across all these websites, ensuring that each selected topic was found in at least three of the fact-checking websites. For instance, the topic of “Immigration” appeared consistently across all six websites.

Derived a final list of 21 selected topics present on our GitHub page 13 to which all the others could be mapped.

After finalising the list of 21 topics, our task shifted to labelling the claims according to them. During this process, we observed that out of 401 original topics from the websites, 170 of them were semantically similar to our chosen 21 topics or could be mapped as subtopics. For instance, “Legal Issues" from “PolitiFact”, “Law Enforcement,” “Legal”, and “Legal Affairs” from “Snopes” and “Law” from “FullFact” can be mapped under the umbrella of the Legal Affairs topic chosen by us in the final list. Therefore, we streamlined these varied categorisations and devised a unified mapping that aggregated synonymous or closely related topics under a single unified topic. An excerpt from this mapping schema is presented in Table 2 , and the comprehensive scheme is available on our GitHub repository 14 . Remaining 231 topics were irrelevant or ad hoc, those that could not be mapped to any topic. These consisted of words such as Ask PolitiFact , Ad Watch , Viral Phenomena , Uncategorized , etc. These topics were discarded, and the resulting claims that were labelled with only these topics were added to the set of unlabelled claims. Following this mapping process, we relabelled all the claims into the selected 21 topics, which resulted in 36,902 labelled and 28,219 unlabelled claims. The labelling process also involved assigning multiple labels to an individual claim, allowing claims to belong to more than one topic.

Table 2: Sample mapping schema for topic “Immigration"

After creating a collection of labelled claims aligned with our manually crafted set of topics, we proceeded to fine-tune a classifier using these topics to be able to classify the remaining unlabelled claims. Since online claims can be associated with more than one topic, thus the proposed model aims at solving a multi-class multi-label classification problem. In detail, given a claim c and the set of potential topics T = { t 1 , …, t n } we want to create a model ϕ ( c ) → T ′| T ′⊆ T . We performed our experiments on a pre-trained RoBERTa-BASE [ 30 ] model. It is a pretrained model on the English language, trained for a masked language modelling (MLM) [ 36 ] objective. We chose to use Asymmetric Loss ("ASL") [ 7 ] to measure the model's performance. This distinctive loss function operates uniquely on positive and negative samples, dynamically adjusting weights for easy negative samples and applying hard thresholds. This aids in discarding potentially mislabelled samples. We adopted a 5-fold cross-validation method for the fine-tuning. This involved random shuffling of the training dataset into 5 subsets, where 4 subsets were utilised for training while 1 subset was reserved for testing the model's performance. This process was iterated five times, creating five distinct models, each trained over ten epochs. This fine-tuning setting was applied to the total 36,902 labelled claims that consisted of the training data after the mapping process as previously described in section 4.2 . The model that performed the best in terms of macro F1 score was chosen as our classifier.

We observed promising classification results 15 across all topics within the 5-fold dataset. These results represent the average performance of five different models across the fifth subset reserved for testing (as described in Section 4.3 ). Results showed an average Micro F1 score of 0.80 and Macro F1 score of 0.74. For classes such as Covid, Economy, Immigration, and Health, scores were higher than 0.85 as F1 score, having achieved a maximum of 0.91 in some cases. Having achieved satisfactory performance from the classifier, we selected the model with the highest F1 macro score to classify the 28,219 unlabelled claims. In addition to the unlabelled claims, we extended the application of the classifier to the already labelled 36,902 claims sourced from fact-checking websites. This step was motivated by our observation that claims labelled by fact-checkers may not comprehensively capture the content of the claims, resulting in incomplete labels. For example, Figure 1 illustrates a claim from PolitiFact categorised under the issue "Health Care." The claim should ideally cover labels such as Legal Affairs or Politics , considering its references to “health care law” and “bureaucrats”. These nuanced aspects are often overlooked by fact-checkers. Recognising the incompleteness of fact-checker-assigned labels, we deemed it crucial to leverage data labelled and classified by the same classifier for our analysis. In the end, we compiled a list of 65,121 claims that had been assigned labels by the classifier that are used in the analysis presented in the next sections. The complete list of claims classified according to topics is made available. 16

Figure 2: Count of claims from different sources

We plot the distribution of 65,121 claims over the years in Figure 2 and their veracity labels in Figure 3 . We notice that:

During 1995, Snopes reported a very small number of claims, and all of them were False .

Throughout the years, the reported percentage of False claims consistently outweighs that of True claims.

During the peak year in 2021, approximately 65% of the total claims reported were found to be False .

Over time, there has been a notable rise in the proportion of False claims. This is confirmed by studies in social science that suggest False claims have gained a different functional use in mass media serving as a political instrument, e.g., during the 2016 US presidential elections [ 3 ] [ 11 ] and German Parliament [ 26 ]. This evolution has led to a surge in their frequency and relevance in contemporary discourse.

In 2007, there was a notable surge in the number of Mixed claims. However, upon closer examination, it was revealed that the rise was due to the influx of claims from PolitiFact website which came into operation that year. Incidentally most claims from PolitiFact during that time were labelled as “half false" and “mostly true," leading to the observed rise in the category of Mixed claims during that period.

We also observe the ratio of Mixed and Other labels have declined after 2013. This trend suggests that fact-checker assessments have inclined toward a more definitive truth rating rather than maintaining an ambiguous stance. The reason behind this might be purely editorial because conclusive fact-checked claims tend to be shared more than others [ 28 ]. However, the ratio of True claims has remained consistently lower than False claims throughout the entire period. Figure 3: Distribution of claims and their veracity labels over time.

Analysing the allocation of topics to claims, as shown in Figure 4 , reveals a significant trend in veracity distribution. Across various topics, we discovered that False claims dominate, except for Business, Economy, and Education, where there is a noticeable prevalence of claims with Mixed truth values signifying that they are not entirely True or entirely False . COVID-related claims stand out with the highest percentage of falsehoods, approximately 71%. The widespread propagation of false claims about COVID and more in general about Health highlights the significance of studying misinformation and delving into the dynamics of its dissemination on the Internet as already investigated in previous studies [ 12 ]. For instance, misinformation and fact-checking efforts related to COVID have helped reinforce truthfulness in the news media, contributing positively to democracy [ 31 ]. Following closely are topics such as Military (67%), Elections (nearly 63%), and Religion (around 56%) all marked by a significant prevalence of False claims. By examining this distribution, researchers and policymakers can better prioritise resources and interventions to combat misinformation and promote accurate information spread across diverse areas of interest. For example, false claims about the military may arise due to propaganda efforts or attempts to influence public opinion on matters of national security [ 35 ], whereas claims about politics may be used for countervailing political statements and enhance factual knowledge during elections [ 6 ].

Figure 3: Distribution of claims and their veracity labels over time.

Figure 4: Percentage of each Veracity Label for each Topic

The surge of claims from 2019 to 2022 in Figure 2 intrigued us, leading to an exploration of prevalent topics within these claims. During this time frame, we have noticed an increase in the prevalence of certain topics that were not as prominent previously (Figure 5 ). For instance, topics such as COVID, Race&Ethnicity, Sports, Military, and Elections have gained significant dominance and visibility. Certainly, all these occurrences could be correlated with global events during those years. Out of all COVID-related claims, about 43% of them occurred in 2020 and 36% in 2021 (Figure 5 ). Undoubtedly this was due to the global pandemic that broke out in the year 2019 and we noticed a huge number of claims related to COVID during that time(Figure 6 ). We also noticed a small proportion of claims have been labelled as “COVID” in Figure 5 before COVID emerged in 2019. Upon inspecting these claims, we observed that the classifier has associated patterns related to the H1N1 virus, swine flu, and mask-wearing behaviour with COVID. In 2020, Election and Race/Ethnicity gained significance. A closer look reveals that all the websites focused on US presidential Elections and the George Floyd Protests during that time. Sports emerged in 2021, followed by a significant rise in the topic of Military in 2022 predominantly associated with the Russia-Ukraine war. For example, a false claim from Checkyourfact says “A video shared on Facebook claims Russian President Vladimir Putin declared war on Kenya” . However, examining the entire timeline from 1996 to 2022 (as depicted in Figure 6 ), Politics consistently stood as the most prevalent topic across all websites. Other significant subjects like Economy, Technology, Health, and Crime followed suit throughout this extensive period.

Figure 5: Coverage of each topic over the entire time period. For each topic, the reader can observe the percentage of claims occurring per year across all websites. The darker the colour the higher the percentage.

The observation of a rise in sports-related claims in 2021, despite the absence of a particular defining event, is intriguing. The identified surge seems to be attributed to a collective series of events associated with Basketball, the India-Australia cricket series, the UEFA European Championship and the Tokyo Olympics. It is noteworthy that this surge occurred after a two-year pause imposed by the global pandemic. This phenomenon could be indicative of the gradual return to normalcy in the world of sports after the pandemic-induced disruptions. The resumption of major sports events and activities might have stimulated increased public attention and discussion, resulting in a corresponding surge in claims related to these events. This observation not only reflects the close relationship between societal events and claim dynamics but also highlights the potential impact of global circumstances on spread of facts [ 13 , 29 ].

Figure 6: Relative appearance of each topic per year. For each year, the relative proportion of claims adhering to a specific topic across all fact-check sources is shown.

Figure 7: Topic distribution across key fact-check sources

We also analyse the evolving dynamics of topics over time, revealing intriguing patterns aligned with real-world events. We notice a spike in claims associated with Economy in 2008 which continued till years 2011-2012 as seen in Figure 6 . During this period, the discourse predominantly centred around the aftermath of the 2008 Economic Crisis and persisted for several subsequent years. For example, a claim by PolitiFact in 2008 asserted that “On the economic crisis, the biggest problem in this whole process was the deregulation of the financial system.” highlighting the economic crisis. Topics also included discussions on state budgets and US military spending. In 2011 and 2012, claims still referenced issues like inflation, unemployment, and the ramifications of the economic crisis, particularly during President Barack Obama's tenure. For instance, a 2012 PolitiFact claim stated that “RNC Chair Reince Priebus says the U.S. has lost 26 million jobs under Barack Obama,” while another claim in 2012 noted that “The U.S. unemployment rate has remained above 8 percent for 43 consecutive months, the longest stretch since the Great Depression” . Snopes and PolitiFact emerged as the primary fact-checking sources during this period. While Snopes addressed a wider range of claims, PolitiFact predominantly focused on claims related to the economy from 2008 to 2012. We looked at the spike in Terrorism-related claims during 2001 (Figure 6 ). Analysing the surge revealed a prevalent focus on the 9/11 attacks and prominent terrorist group leaders. Some of the example claims during that period are “World Trade Center attack survivor Adam Mayblum gave a first-person account of his harrowing escape from one of the doomed towers.” and “Osama bin Laden owns extensive gum arabic holdings” .

Figure 8: Veracity Label distribution from various sources

Figure 9: Frequently occurring entities across each website

Figure 10: Distribution of six prominent topics across each fact-checking source (a) Factcheck AFP, (b) PolitiFact, (c) FullFact, (d)Snopes, (e) Checkyourfact, (f) Africacheck, (g) TruthorFiction

We performed a deeper analysis of some of the prominent topics and how individual websites focused on them in Figure 10 . We observed that websites like Factcheck AFP, Checkyourfact, PolitiFact, and Africacheck shifted their focus from other topics to verifying more claims related to COVID during 2019. Similarly, during 2022, platforms like Africacheck notably pivoted their attention towards Election (Figure 10 ) due to events such as the Nigerian elections and the economic landscape across the African subcontinent. By examining claims through the lens of various topics, we could uncover patterns, generate insights into global events, and shed light on issues that are not only significant but also warrant further investigation.

Following our examination of claims based on topics and truth values, we aimed to explore the origins of these claims and their distributions across different sources. We observed in Figure 7 that various sources have different focal points in their topics. For instance, Snopes and TruthorFiction tend to emphasise politics, FullFact leaned toward health topics, and Factcheck AFP primarily focused on COVID related subjects in recent years. For the other websites, topics seem to be distributed evenly without any notable emphasis on specific subjects. We also analysed the predominant truth values occurring in each of these sources in Figure 8 . Checkyourfact and FactcheckAFP reports a relatively higher number of False claims followed by Africacheck and Snopes. TruthorFiction is the only website reporting more True claims than False claims. PolitiFact has the highest proportion of Mixed claims and FullFact reported the maximum number of claims that fell into Others category. The varied focus of different fact-checking websites on distinct topics and veracity labels can be attributed to the diverse editorial inclinations of the fact-checkers themselves [ 33 ] [ 27 ]. These inclinations are influenced by their inherent biases toward what they perceive as check worthy [ 9 ] [ 25 ].

We further expanded our analysis by examining the most frequently occurring DBpedia entities within both the claims and their respective reviews. The reader can observe this analysis in Figure 9 . 17 Interestingly, a substantial proportion of these entities were references to social media platforms like Facebook, and Twitter, often indicating the claim's origin. Additionally, we observed a prevalence of country and place names like the United States, Texas, and Florida among these entities. We have graphed the top 100 entities across various sources excluding social media, news channels, and names of places, and observed that most originate from PolitiFact. This prominence on PolitiFact's end likely stems from its high volume of claims and included entities like PresidencyofDonaldTrump, PresidencyofBarackObama AffordableCareAct, ClimateChange, Covid19, and Unemployment to name a few. Apart from PolitiFact, Factcheck AFP emerged as another significant platform that contributed more to COVID-related claims compared to others. As illustrated in Figure 7 , AFP Factcheck notably concentrated on COVID-related claims in recent years. We also looked at some of the similar and repetitive entities that have emerged in our analysis. For example, Covid19 and COVID-19vaccine or DonaldTrump and PresidencyofDonaldTrump. We have noted that despite their apparent similarities, these entities are extracted within distinct contexts. “Covid-19” entity is extracted for only Covid-related claims. For example “No, UK's Boris Johnson didn't say Covid-19 pandemic was a lie – fake BBC headline" whereas “Covid-19vaccine” for claims specific to vaccine safety like “...Regulatory Agency (MHRA) has been shared on social media, and asks people to report any side effects from Covid-19 vaccines”. The same holds for discussions about Donald Trump and his Presidency, where one focuses on Donald Trump as a person with no clear indication of his role as U.S. president, while the other delves into his presidency and elections. These entities provide insight into how seemingly similar claims pertain to different contexts and overlapping topics.

Table 3: Key entities identified within each website from inception until December 2022

5.5.1 Entities and Topics: What insights do they offer?. We delved into exploring the correlation between entities and topics by examining the top 30 entities associated with claims on each website. The objective was to ascertain whether the prevalence of notable named entities within claims could offer insights into the dominant topics. Our findings revealed a connection between entities and prominent topics across various websites.

Figure 11: Overlap Counts of Similar Claims between Fact-Checking Websites

For instance, scrutiny of entities in Table 3 and Figure 7 allows us to infer that Fullfact extensively covers Health and Covid-related topics. This inference is drawn from the presence of entities such as NationalHealthService, Vaccine, and Covid-19. Also it is more focused on UK politics, being an independent fact-checking organisation from the UK. On the other hand, Snopes predominantly focuses on Politics, evident from the high occurrence of entities like DemocraticParty(UnitedStates) and RepublicanParty(UnitedStates). Additionally, it shows a keen interest in Technology, as reflected by entities like Email and Photographs(MestAlbum), predicted as technology by the classifier. The examination of entities across various websites, such as Factcheck AFP emphasizing COVID-related topics, TruthorFiction focusing on Politics, and Politifact delving into the Economy through entities like Unemployment and Tax, offers insightful observations. This analysis establishes a clear correlation between entities and the prevalent topics covered by each fact-checking website.

A certain number of identical claims is present within the websites, published on different dates, and reviews. For instance, the same claim was fact-checked by different websites and assigned a rating by their fact-checkers. We investigated the presence of these claims within the dataset and identified 2,450 instances out of a total of 61,151 claims. We used a sentence transformer model [ 38 ] where the sentences with similar meanings are associated with embeddings that are close in the vector space. We used this model to compute an embedding for each sentence. Then, the semantic textual similarity between two sentences is computed, and we have sentence pairs annotated together with a score indicating the similarity between them. The model uses a Siamese network structure that was trained using CosineSimilarityLoss[ 37 ]. We chose the similarity threshold of 0.85, to ensure that only highly similar items are considered matches, and to reduce the likelihood of false positives. We also did a manual check and found that the sentences below this threshold were dissimilar in meaning. While the subset of 2,450 claims is relatively small to discern any clear pattern, it provides a glimpse into prevalent trends and themes within the fact-checking domain.In our analysis of the 2,450 claims, we found that 6.8% of Snopes fact-checked claims are similar to claims in other fact-check sources while 5.4% of the claims verified by Politifact are similar to claims in other fact-check sources The analysis also pointed out certain pairs of websites that have numerous similar facts. For example, Politifact and Snopes have the highest number of similar co-occurrent claims followed by Politifact and Checkyourfact, and Snopes and Checkyourfact. A breakdown of overlapping claims between specific websites is highlighted in Figure 11 Snopes emerged as the most authoritative fact-checker, often being the first to verify similar claims, and sometimes fact-checking claims even 15 years before other websites. Among the claims that Snopes fact-checked and are similar to those in other fact-check sources, Snopes was the first to fact-check 71% of them before any other website. Politics and COVID emerged as the most prevalent topic within the dataset, followed by Health. Notably, the year 2020 witnessed the highest level of overlap, with 497 claims undergoing fact-checking, primarily revolving around COVID and Health-related issues due to the spread of the pandemic and most websites shifting their focus to COVID related claims as seen in Figure 10 .

In our examination of veracity labels, we observed that 50% of these co-occurrant claims were false. We also found a remarkable consistency among various fact-checking platforms. While differences existed in the terminology used for labelling, such as "True" by Snopes and PolitiFact versus "Correct Attribution" by TruthorFiction, there were no conflicting assessments. We did not encounter any instances where a claim was deemed true by one website while being considered false by another. This agreement in fact-checkers stances reinforces the confidence in the reliability and accuracy of their assessments.