The fairness doctrine was a policy established in the United States in 1949 by the Federal Communication Commission (FCC), to protect the First Amendment free speech rights, during an era in which radio and TV started contributing to an unequal amplification of speech. It was abolished in 1987, and then removed entirely from the federal books in 2011. The doctrine required broadcasters to devote “adequate time to important and controversial issues” and do so by “allowing reasonable opportunity for opposing viewpoints to be expressed on air” [20]. From the beginning the doctrine was controversial, with some arguing that it limited the freedom of press, and others that it contributed to a more informed electorate. Research has shown that the doctrine often led to harassment of the press by groups that believed they were not treated fairly [21]. This is reminiscent of present day, when some conservative politicians blame Google for being biased against them by showing in its results mostly left-leaning sources.[10]

The Fairness Doctrine impacted media in a profound way by establishing the “public interest” standard [9]. This meant that the right of the public to be informed was paramount. Publishers, as public trustees, had an obligation to the public. This imperative to be balanced in their coverage and represent a diversity of viewpoints was embraced by many publishers and continues to be a cornerstone of mainstream journalism. Meanwhile, publishers who wanted to represent only a viewpoint, e.g. religion, aggressively pushed the boundaries of the doctrine challenging FCC and its rules

[8]. The advent of digital technologies that did not require access to the limited wave spectrum (a public good) weakened FCC’s argument for a diversity of viewpoints. Additionally, the political mantra of Reagan’s administration for less intrusion by the Federal Government and broad deregulation, created the context for doctrine’s demise. One of the direct outcomes of this event was the explosion of conservative right-wing media, that created its own alternative reality, whose consequences culminated in the election of Trump in 2016 [12].

In their book “Network Propaganda” [3], the authors consider a set of actors and technological drivers that have been identified as causing the present state of information disorder: “fake news” entrepreneurs, political clickbait fabricators; Russian hackers, bots, and sockpuppets; the Facebook Newsfeed algorithm and online

10https://www.politifact.com/factchecks/2018/aug/29/donald-trump/no-96-googlenews-stories-trump-arent-left-wing-ou/

echochambers; Cambridge Analytica; or white supremacists and alt-right trolls. Ultimately, they settle on the right-wing media ecosystem as “the primary culprit in sowing confusion and distrust in the broader American media ecosystem.” Their focus is Breitbart News, which was instrumental in setting an election agenda centered on immigration issues and Clinton’s scandals, which was also embraced by mainstream media, as detailed analyses of their election coverage showed [15, 25]. Although a few of the most problematic hyperpartisan news sources were discredited (e.g. InfoWars), many continue to operate and are surfaced by news aggregators like Google’s Top stories.

What is found on Google Search, matters. Studies have shown that as more people abandon traditional forms of news exposure such as newspapers and TV, a new way of accessing news is spreading, distributed discovery [22]. “I just google it” has become a new way of getting informed, in the hope of getting unbiased coverage [24]. The most recent survey that investigated how distributed discovery is divided among the various channels: search engines, social media, news apps, etc., indicated that 20% of respondents use search to get news, compared to 24% who use social media [13]. Interestingly, using search media to get access to news is “associated with more diverse and more balanced news consumption”

[6], which conforms with the findings of this paper (in terms of diversity and balance of coverage by Google’s Top stories).

As the amount of search engine referrals to news websites increases, it is worth investigating what news is shown by search engines. Top stories is the first element of the search page for many important events (e.g. elections). Given the immense supply of news stories, by selecting only a few of them at a time, Google’s Top stories is engaging on what is known as “algorithmic news curation” [4]. If algorithms are curating news, how are they choosing, especially with respect to news sources with problematic credibility? Are the right-wing publications identified in [3] as the cause of our current information disorder (pre- and post- 2016 US Election) being promoted by Google Top stories? Does that apply to leftwing publications?

This line of research, which falls under the umbrella of algorithm auditing [18], is important for two reasons: 1) more people use search engines than social media on a daily basis. If they are being exposed to news in this way, we need to understand how algorithms are curating the news; 2) there is evidence that as many as half of all search queries don’t lead to clicks,[11] because a user’s information need is fulfilled by the search page content. Thus, the news headlines that a user reads in Top stories might be all they remember about a news event. Together, these headlines can frame issues in a partisan way and help with media agenda setting. Although there have been already several audits of Google’s Top stories [11, 17, 23] and Apple News [2], this study has the longest duration (a whole year) and a specific focus on candidates for a political election.

It is important to summarize some of the biggest findings from the previous literature on auditing top stories: 1) concentration of

11https://searchengineland.com/49-of-all-google-searches-are-no-click-study-finds318426

sources; and 2) a left-leaning bias of news coverage. With concentration of sources is meant the phenomenon of encountering the same news sources frequently. Our own audit of 2020 Top stories reinforces this finding, since 1/3 of the news articles in the dataset belonged to only eight publishers [10]. The left-leaning bias is connected to the shift of news consumption in the public. What were usually considered main-stream media for decades, now appear left-leaning (or center left) due to the political polarization of the audience.

Throughout 2019, we monitored and captured the Top stories element of Google’s search results pages for queries that are the names of 30 political figures, for example, Joe Biden, Kamala Harris, Donald Trump, etc., who were running (or rumored to run) in the primary season of the 2020 US presidential election. Out of the 30 candidates, 28 are Democrats and 2 are Republicans. There are only two republicans, because one of them is the incumbent president, Donald Trump, and no other republicans decided to challenge a sitting president running for a second presidential term. Our dataset[12]

is publicly available and a detailed overview of the process and data appears in [10]. We refer the reader to that paper to learn about the technical details of the data collection such as use of Selenium to automate the Chrome browser in incognito mode, the capture of HTML web pages of results, and repetition of the data collection in a second location to establish lack of personalization of Top stories based on location.[13]

From December 2018 to June 2019, Top stories were observed every 6 hours, then from the end of June 2019 to December 2019, every 2 hours. For the majority of candidates, the data collection has been successful 95% of the time. An observation is a Top stories panel identified by the query and the time of collection. In general, every panel consists of 10 news stories. From each panel, we extracted the news story, its headline, URL, second-level domain to identify a source, and the position of story on the panel.

Overall, the dataset contains a total of 588,112 observed stories in the panels of the 30 candidates. For popular politicians such as President Trump and Joe Biden, the panels are refreshed frequently, leading to a large overall number of articles. Meanwhile, less familiar candidates are covered infrequently by the media and their Top stories panels are often composed of the same articles over multiple days. Within all these observations, we find 79,903 unique news articles, which were published by 2,168 unique news sources. The distribution of articles is heavily concentrated among a handful of sources. For example, 1/3 of articles originate from only 8 (0.37% of) sources: The Hill, CNN, Fox News, Politico, Washington Post, Washington Examiner, NY Post, NY Times. Another 1/3 of the articles come from 48 news outlets, including very popular publishers such

12Available at https://doi.org/10.7910/DVN/0ZLHOK 13Google in its documentation that explain the Top stories feature has confirmed that Top stories generally are not localized. For more, refer to: https://newsinitiative. withgoogle.com/hownewsworks/approach/presenting-news-in-helpful-ways/.

as NBC News or USA Today. As a stark contrast, 34.18% of news sources in the dataset are represented by only one article, although it’s hard to believe they have not written more often about the candidates during the whole year. For the purposes of this paper, we decided to focus on the 56 news sources (out of a total of 2,168 sources) that together contributed 2/3 of the collected articles. Such a source concentration suggests that for political news, Google’s algorithm has picked its “preferred” sources and samples from them more frequently than from all other sources combined.

In the United States, it is particularly difficult to characterize news sources in terms of political orientation (or partisanship), because most of them declare themselves independent of any political influence. Some of the major newspapers such as New York Times and The Washington Post maintain a separate Opinion section that covers a range spectrum of opinions covering the spectrum.[14] This is different from news outlets in European countries, where newspapers are often openly affiliated with political parties or movements. To overcome this obstacle, researchers have tried to measure the partisan bias indirectly through the bias of their audiences. A well known example was presented by Facebook [1]. A more recent one is [17], which created the Partisan Audience Bias (PAB) dataset[15]

from website links shared by real users on Twitter. Using voter registration records of US citizens with Republican and Democratic affiliations, the study identified 519,000 Twitter accounts matching these citizens. By extracting second-level domains from URLs that appeared in tweets of at least 50 different users, the authors obtained a dataset of 19,022 sites. For each site, they calculated a bias score between -1 (a site shared only by Democratic voters) to +1 (a site shared only by Republican voters). Sites that get a bias

score between -1 and +1 were shared by a mix of Democratic and Republican voters. For example, The Wall Street Journal had a score of 0.0106, signaling that it is a news source shared almost equally by both sides.

By mapping news sources from the two datasets, we obtained the PAB score for 1,508 out of 2,168 sources. PAB scores for 660 sources are missing, and since these sources only contribute a very small percentage (3.3% of stories) to the dataset, they were discarded.

Due to our focus on the Top stories panel, we were interested in learning whether users notice or care about Top stories. Although there are surveys that indicate that users access news from search engines [13], such surveys are not specific to components of the search results page. Therefore, we decided to run our own largescale survey to learn about the extent to which users pay attention to Top stories. The survey was deployed through Amazon Mechanical Turk and successfully completed by 951 participants. Participants provided consent before beginning the survey and were paid

14As an example, recall the controversy about Tom Cotton’s op-ed in New York Times, that led to the resignation of the editor: https://www.washingtonpost.com/lifestyle/ media/new-york-times-editorial-page-editor-resigns-after-uproar-over-cotton-op- ed/2020/06/07/bca09606-a8fd-11ea-9063-e69bd6520940story.html. 15Available at https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/ DVN/QAN5VX

$0.25 for its completion, which on average lasted one minute. No personally-identifying data of any kind were collected.

The survey focused on two types of questions: 1) Demographic questions to characterize participants’ news access habits and factors that can influence news preferences, and 2) Perception-based questions regarding participants’ trust towards sources frequently shown on Top stories. Answers to all questions of the first type are displayed on Table 1.

Table 1: Summary of survey responses from n = 951 partici- pants. A big majority of participants are aware of Top stories and access news via mobile phones.

Data Type Response

Democratic (504, 53.00%)

Registered

Republican (253, 26.60%)

political party

Independent (194, 20.40%) Have you ever noticed Yes (795, 83.60%) the Top stories panel? No (156, 16.40%)

Always (45, 5.66%)

How often do you read

Most of the time (276, 34.72%)

the headlines in the Top

About half the time (183, 23.02%)

Stories panel when it

Sometimes (263, 33.08%)

appears?

Never (28, 3.52%)

Search engine results (403, 42.38%) Direct access to website (238, 25.03%)

What is your most

Links to social feeds (186, 19.56%)

frequently used method

Mobile app (95, 9.99%)

of getting news online?

Email newsletters (18, 1.89%) I don’t read news online (11, 1.16%) Do you use your mobile

Yes (798, 83.91%)

phone to search for

No (153, 16.09%)

news?

For the second set of questions on respondents’ levels of trust towards news sources, we utilized the 56 most frequently shown news sources which were identified by our analysis of the Top stories dataset [10]. From these 56 sources, the survey randomly selected five sources for each participant. For each of these five sources, participants were asked to answer how much they trust the news source (A lot, Somewhat, Not at all) or indicate if they have never heard of them (Never heard of them). Throughout this paper, we refer to calculated trust scores for each source. The scores are a weighted average of participants’ responses where A lot = 1, Somewhat = 0.5, and Not at all = 0.

Surveys with crowdworkers often suffer from the fact that participants develop survey-fatigue and are rushing to answer quickly and move on to the next earning opportunity [7]. We were very conscious of this limitation and took protective measures: the respondents needed to have a high approval rating of their previous work (above 90%). Additionally, we made the survey short by limiting the number of questions for each participant. While it would have been ideal to ask every participant to provide a trust score for each of the 56 news sources, we reasoned that this will lead participants to answer randomly due to fatigue.

Our survey targeted those who voted during the 2016 US presidential elections. Thus, the responses correspond to a demographic that is more likely to regularly consume political news. While this demographic’s survey responses are particularly suitable for understanding the influence of Top stories on political queries, the

results may be skewed by an audience that is more attuned to the political landscape. Questions regarding noticing the Top stories panel, reading news online, or familiarity with news sources may be an upper bound for a more general, less politically-interested audience.

Finally, because the survey was completed by crowdworkers, many demographic factors were contingent on who was available to work. We didn’t ask for any demographic information, except for the political party registration. We notice that the participant pool is skewed toward individuals who identified as registered Democrats. The Democratic to Republican participant ratio is roughly 2:1, the same for Democratic to Independent. In order to account for this imbalance, we report results for each group separately, so that answers don’t suffer from the oversampling of one group.

Our Top stories dataset contains articles from 2,168 news sources, and we mapped them to bias scores from the PAB dataset.[16] The distribution of news sources by bias scores is shown in Figure 2: Distribution of Partisan Audience Bias (PAB) scores for 1,508 sources in the Top stories dataset. This distribution shows the overall sources from which Top stories samples articles. It shows that apriori to article selection, there is somewhat a balanced selection of sources.

Figure 2. We first notice that the overwhelming majority of sources are in the center (between -0.5 and 0.5). Using chunks of 0.5 to divide the

[-1,+1] political bias segment, we find 221 far-left, 587 center-left, 598 center-right, and 102 far-right sources in each corresponding segment. Thus, in the center, Google is equally sampling from center-left and center-right sources. Things look different for the two tails, with far-left sources outnumbering far-right sources 2:1 (221 versus 102). It is this difference that leads to the finding (see related research too) of an overall left-leaning bias in Google’s news products. However, this result might be open to revision, given that the bias scores were calculated based on the audience bias.

16Reminder: Many news sources didn’t have a PAB score, but these sources contributed only 3.3% of articles and were excluded from the analysis.

Differently from the balanced distribution of news sources in Figure 2, the article distribution over almost 600K observations in our dataset present a different picture. Here we see that center-right contributes very little to the overall distribution, while far-right articles seem to make up for the anemic showing of the center-right.

ipfs://bafkreifajjlefl3neulbiq2dnjl2fc57njayjlf2xsw5ek3l3loqvstu5u)

In addition to looking at the overall bias distribution among news sources and news articles from our dataset, we considered the average bias scores of individual Top stories panels, given that users will experience one panel at a time when searching Google. For each observation, identified by a unique (candidate name, time of collection) pair, we computed the average bias score of news sources present in the panel. Sources with missing PAB scores were not included. Figure 3: Distribution of PAB scores for article observations in Top stories panels. The percentage is observations of sources with bias score > 0.5 is higher than expected. That is, despite the small number of far-right sources in the overall distribution of Figure 2, articles from these sources are over represented.

Figure 4 displays box plots of the average panel PAB score for seven candidates (one is President Trump and the six others were still in the race at the end of February 2020. We are focusing on them, because they emerged as front-runners after one year of campaigning). Here, too, the distribution is shifted slightly to the left for most candidates. Joe Biden’s and Michael Bloomberg’s

panels are centered very close to 0. Statistical t-tests for all distributions indicated that all candidates but Biden and Bloomberg differ significantly from a mean=0 bias. When we compared Biden’s panels’ bias distribution with that of Sanders (the other main contender for the Democratic nomination), we noticed that Biden had 8% more coverage from far-right sources (22.1% vs 14.3% is the coverage from far-right sources). This can be explained by the fact that during most of 2019, Biden was regarded as the Democratic candidate with the highest chance of becoming the nominee, and thus far-right sources, particularly Breitbart News were focused on attacking his record. As a comparison point, there are 346 articles by Breitbart News about Biden in our Top stories dataset, but only 6 Breitbart articles for Sanders.

Further, we looked at the distribution of PAB scores along each of the 10 Top Stories panel positions, in order to find out whether bias is evenly distributed across the positions or concentrated in the top positions.

Figure 5 aggregates average PAB scores from all observations for each of the top seven candidates. Values in the heatmap cells range between -0.18 to 0.06. While overall (first row of heatmap) the distribution of bias looks mostly even, there are differences among the candidates. Earlier we focused on Joe Biden, for whom the average bias score was around 0, but with a strong component of the distribution coming from right-leaning sources. In Figure 5 we notice that the most right-leaning coverage is concentrated in the first four positions, meaning that not only did right-leaning sources target Biden more consistently than other candidates, but these sources got a prime spot on Top stories.

Overall, we observe that each individual Top stories panel seems politically balanced, by combining articles that come from a majority of center-left sources with articles coming from a minority of far-right sources.

Although on average the top 56 news sources seem to be known and trusted by the survey participants, a closer look gives us two notable results: (1) A plurality of the surveyed news sources are not well trusted by both Democrats and Republicans; and (2) There

exists a set of sources that are neither well-known nor well-trusted by participants of all political orientations. First, 26 out of the surveyed 56 sources are widely trusted by participants of one of the two main political parties, but not both. On average, Democrats trusted more sources at 20, while Republicans trusted only 6. On the other hand, 21 out of 56 news sources were not widely trusted by either Democrats or Republicans, among which 17 sources were not trusted by all of the three groups. Here, we consider ’not trusted’ sources as sources with trust scores less than 0.5. Next, there are 14 sources that are not well known by Democrats, Republicans, and Independents. Here, we consider sources to be not well known if more than 1/3 of respondents indicated they have not heard of them.

These findings suggest that public’s perception of trustworthy news sources is not aligned with that of the Top stories algorithm. We hypothesize that Google may be prioritizing showing a ’fair and balanced’ set of news sources, even if the sources are not familiar or trusted, as long as they provide political diversity. Using our before-mentioned categorization of sources into far-left, center-left, center-right, and far-right by a 0.5 PAB score unit, we find 7 far-left, 26 center-left, 14 center-right, and 9 far-right in the group of 56 sources in Table 2. Google seems to make up for this imbalance in the number of available sources in two ways: (a) by oversampling from the far-right sources. Concretely, the 9 far-right sources contributed 12.58% of unique stories while 7 far-left sources contributed only 4.89%, a stark contrast. (b) by giving more volume to far-right sources at the top-tier. In the group of 8 most frequent sources that cover together 31% of all stories, two (Fox News and Washington Examiner) are far-right sources whereas none are far-left sources. As a result, the higher volume of these far-right sources compensates for the larger number of center-left mainstream media sources.

Here is one more fact: Fox News has a 0.72 trust score among Republicans, but an overall trust score of 0.36. Meanwhile, Washington Examiner is not that well-known (68.7% familiarity) or trusted by Republicans with a score of 0.39, but it is still the second most frequent conservative news source in Top stories. This provides additional evidence that neither popularity nor perceived trustworthiness are the main criteria for being included in the Top stories.

Looking at the trust based rankings of the 56 sources between participants with different political affiliations, we found significant positive correlations between the ranking of Independent participants and both Democrats r𝜏 = 0.40, p <.001, and Republicans,

Figure 6, we can see that the trust scores of Independents move in the same direction as Democrats and Republicans, and the majority of the trust scores lie near the diagonal line, where they have the same trust score for the news sources. That Democrats and Independents are more aligned than Democrats and Republicans, shows one more time that trust is driven by political polarization. The results also align with the findings from [16], in which the trust rankings of Democrats and Republicans were found to be significantly positively correlated with a political neutral group, fact-checkers. As for the comparison between Google’s algorithm and participants, there is no significant correlation between Google ranking and either composite trust

score based ranking, or categorized rankings from three political parties. It reinforces our claim that Google’s algorithm does not rely on public’s perceived trust as a criterion for preferring a news source.

Meanwhile, Table 2 provides additional evidence that the ranking of sources by the participants and Google’s ranking are not aligned. The top ranked news sources based on Democrats, Republicans, and Independents’ trust scores are The Washington Post, Fox News, and NPR respectively, but the top Google ranked news source, The Hill, is not within the top 10 of any of the participants’ ranking list. To explore this further, we conducted Wilcoxon signed-rank tests to compare the difference between Google’s ranking and the trust score ranking from respondents with different political affiliations. We found that Google’s top 10 ranked news sources have a significantly different ranking order as compared to the Democrats’ trust score based rankings (z = 8.0, p <.05), Republicans’ trust score based rankings (z = 1.0, p <.01), and Independents’ rankings (z = 5.0, p<.05). We also observe such significant differences between participants and Google’s rankings in the top 20 and last third of the 56 news sources.

To conclude, our various tests and visualizations indicate that no group of a certain political orientation in the US can be satisfied with Google’s ranking, because Google is not relying solely on commonly used metrics such as familiarity or trust of news sources to assemble the panel of Top stories.

Google has the difficult task of aggregating news stories from many sources, so that every user can find a multitude of diverse perspectives. However, as our survey indicates, the public disagrees about which news sources are trustworthy, therefore, Google has

to select a few sources that are trusted only by minority groups in the audience. How is Google doing this? According to Google’s public documentation, they use human Search raters[17] to rate news sources across several criteria, and then have algorithms that automatically learn how to rank articles based on such signals. Furthermore, Google explains that it doesn’t try to assess the “ideological or political leaning” of publishers or users, while at the same time, promising a diversity of points of views. There is a tension here that Google’s documentation doesn’t resolve. What kind of diversity are they referring to?

The evidence from our one-year long audit shows that Google is sampling from a diverse set of news publishers (where diversity here is simply the partisan audience bias score), as shown in Figure 2. We consider this as an effort to be “fair and balanced” in the spirit of the old “fairness doctrine” which required broadcasters to strive fulfilling the “public interest”. However, as with the doctrine itself, the meaning of “fair” remains contested. Fair to the audience? Fair to the political candidates? Fair to the news publishers? As everything that is determined algorithmically, such choices are both open to criticism as well as exploitation. To an individual that has a limited number of trusted news sources, as it seems to be the case with Republican voters (they trusted 6 out of 56, compared to 20 out of 56 for Democrats), any Top stories panel configuration will look as heavily left biased, although the very concept of “left” here is doubtful, because it simply means that people registered as Democrats find this source trustworthy. In order to be fair to Republicans, Google has to surface news from the right. But the distorted right-wing news ecosystem, with a few far-right sources dominating the coverage, leads to their disproportionate amplification, often at the expense of trustworthy journalism that is less polarizing. One thing is clear, Top stories do not create an echo chamber, and might be even susceptible to right-wing propaganda.

17https://newsinitiative.withgoogle.com/hownewsworks/approach/surfacing-useful- and-relevant-content/

In anticipation of mobile devices’ usage passing that of desktop devices, which happened around the end of 2016,[18] Google pushed news publishers to adopt its new content-delivery technology, AMP (Accelerated Mobile Pages), and made that the cornerstone of a new user interface element: a scrollable carousel of Top stories explicitly made for mobile devices. Google’s efforts seem to have paid off. In our survey, we learned that most US voters are aware of Google’s Top stories, especially the ones who use mobile devices to access online news. Our survey also indicates a large number of voters using search engines to find news (more than for any other online medium). However, that number might be an over-estimation, due to our sample being entirely composed of crowdworkers whose digital habits might differ from the general voting population. Nevertheless, Top stories seem to have become a very noticeable news aggregator. Given that more people use Google Search daily than any other service, the integration of Top stories on Google Search is an important development that needs to receive more attention from researchers, in terms of understanding how users access and consume news online. This move toward consuming news on mobile devices poses a technical challenge for the auditing community. The audits we cite and our own audit as well were carried out on desktop machines. A recent small-scale test we performed manually on a mobile device indicated that the first row of Top stories on a mobile device conforms with Top stories on a desktop device, but as we can notice in Figure 1, on a mobile device Google Search is showing several rows of Top stories. Thus, we need to develop an auditing infrastructure that can run on mobile devices, something that is currently missing.