Confirmation bias is the tendency to seek information in a way that confirms one’s beliefs to avoid mental discomfort [12, 18, 27]. Confirmation bias sometimes leads people to make inaccurate decisions [16, 26, 40]. For example, people who believe that green tea extract can promote weight loss might seek positive information related to green tea extract even though its effectiveness has been questioned [7, 19]. In terms of social interactions, previous studies have suggested that people tend to ask questions to which a positive answer is considered as strongly confirmatory and a negative answer is considered as weakly disconfirmatory [10, 33, 34, 36]. In addition, typically, people prefer confirmatory answers over disconfirmatory answers [15].

Question answering systems have received significant attention in the information retrieval community. With a question answering system, searchers promptly obtain relevant information rather than going through a large number of web documents [23, 39]. Recently, major search engines have started to present a question answering system called People also ask in response to some queries where questions are selected by the search engine as a function of frequency the questions have been asked [24]. Although previous studies have explored the effect of question answering systems on search performance (i.e., decreased number of queries) [4, 5, 8, 23], such studies did not consider the effect of question answering systems to confirmation bias in a search.

Previous studies have found a relationship between search behaviors and confirmation bias. For example, people’s decisions after searching were more supportive of a medical remedy even though that remedy was not actually helpful [40–43]. Moreover, it is difficult to change people’s pre-existing beliefs. In addition, Pogacar et al. [29] found that people were likely to make inaccurate decisions if the majority of search results were inaccurate.

To mitigate confirmation bias, previous studies have suggested that people should encounter alternative or opposing information that they would have not expected [6, 25, 35]. Consistent with this ideology, previous studies have suggested that recommending information that challenges the user’s predisposition can stimulate divergent thinking and consideration of diverse information [21, 22, 31, 32]. On the other hand, studies in pragmatics found that people tend to select questions that deviate from their hypothesis if such questions are provided [37, 38].

The present study considers and extends these studies in the following ways. First, we investigate whether questions and answers can encourage careful information behaviors in the search context. Second, while previous studies [21, 22, 31, 32] revealed that presenting information that is inconsistent with people’s preference can help mitigate the confirmation bias, this study wants to determine if such an effect persisted when contradictory information is presented as answers.

-----

Table 1: Search tasks translated from Japanese. Candidate remedies and their Cochrane review URLs are shown.

Search Task Remedies

ID Symptom Task Target Remedy Alternative Remedy

1 back pain Is acupuncture effective in relieving back pain? acupuncture (https://bit.ly/2kbOlm0) insoles (https://bit.ly/2m6YPDL) 2 eczema Are moisturizers effective in relieving eczema? moisturizers (https://bit.ly/2kqZyPN) probiotics (https://bit.ly/2kBBCsN) 3 tooth decay Are sealants effective in relieving tooth decay? sealants (https://bit.ly/2kBRQlF) chlorhexidine (https://bit.ly/2kpy9xH) 4 asthma Is caffeine effective in relieving asthma? caffeine (https://bit.ly/2kCcGBB) homeopathy (https://bit.ly/2kBBWYx)

This section introduces the tasks which were used in both experiments and the method used to collect questions and answers.

Four medical symptoms were adopted from a study by Pogacar et al. [29] as these symptoms have more than one possible remedy. For each symptom, two candidate remedies were selected from the Cochrane medical knowledge base as the target and alternative remedy. Medical experts know that the systematic reviews carried out by Cochrane meet the highest standard in evidence-based healthcare [9]. To form the tasks, we created questions related to the effectiveness of the target remedy for a given symptom. Table 1 lists the tasks, symptoms, target remedies, and alternative remedies. Note that in Table 1, Task 4 was used as the training task.

To collect the questions, a query in the format "target remedy + symptom" (e.g., acupuncture back pain) was issued to Google and the top four yes/no questions were collected from People also ask. We used yes/no questions because we could pair the answers with any questions that were collected for the same task. Moreover, yes/no questions make us able to measure participants’ degrees of beliefs easily [40]. In addition, the query in the form of "alternative remedy + symptom" was issued to Google in order to collect a single alternative-related yes/no question from People also ask that would be used in the second experiment.

To collect answers, workers from the Japanese crowdsourcing platform Lancers [2] were recruited. For each symptom and remedy in Table 1, the workers were asked to look for articles on the Internet that support/oppose the remedy. The workers were asked to collect articles that satisfied either of the following conditions: (1) the article explicitly refers to an academic paper or technical report presented at an academic conference; (2) the article is from a medical institute or government agency; and (3) the article explicitly reveals that the author is a medical expert. As Cochrane reviews are considered as the highest standard, it is very likely that their articles satisfy the established criteria resulting in a risk that workers would only submit articles from Cochrane. Thus, to obtain articles from diverse sources, workers were asked not to submit any articles from Cochrane. Finally, they were asked to submit the article and its URL that they think supports/opposes the remedy. Each worker received approximately $4 as compensation. For each remedy, we selected two supporting and opposing paragraphs from the submitted articles as the answers. Note that the selected text was easy even laypersons to understand. Table 2 lists the questions and example answers used in the experiments.

2https://www.lancers.jp/

This section first provides an overview of the method used in both experiments and then describes each experiment in detail. We also introduce the search system that was designed for this study and outline how participants were recruited.

First, participants were informed how their data would be used for research purposes and asked for consent. Only those who agreed to the conditions could participate in the training task (Table 1, Task 4: Is caffeine effective for asthma?). The participants completed this task using our search system. During training, People also ask containing four questions with supporting answers for caffeine was presented. After the training task, participants were asked to complete three search tasks as follows. First, for each search task, the participants were given instructions that comprised the background of the symptom and a target remedy. After reading the instructions, the participants were asked to complete a pre-task questionnaire. They were asked to indicate their beliefs in relation to the target remedy by answering the question Do you think acupuncture is effective in relieving back pain? using a four-point Likert scale (1: No; 2: Lean no; 3: Lean yes; 4: Yes). The scale was structured such participants had to indicate whether they believed the remedy to be ineffective or effective. The scores for this question were considered as their prior beliefs. Then, participants were asked to rate their prior knowledge about the symptom and target remedy using a four-point Likert scale (1: Not at all; 2: A little; 3: Good; 4: Excellent).

Subsequently, participants performed the search task using our search system (Section 4.6). Note that they were not allowed to use other commercial search engines. During the search task, they were allowed to issue queries to the system and click on documents returned by the system without time constraints. Once they felt satisfied that they had completed the task, they could click the finish button at the top right of the search page (Figure 2).

At the end of each search task, participants were asked to complete a post-task questionnaire. They were first asked to provide their beliefs about the target remedy again using the same fourpoint Likert scale. Their scores for this question were regarded as their posterior beliefs. Next, they were asked to provide their satisfaction with the search system on a four-point Likert scale (1: Not satisfied; 2: Somewhat not satisfied; 3: Somewhat satisfied; 4: Satisfied). Finally, they were asked to select the documents that contain evidence that support their posterior beliefs. A list of the documents they clicked on during the search task was provided. They were asked to select one or more documents from the list.

After completing the three search tasks, participants had to complete an exit questionnaire that collected demographic information (gender, education level, and search engine familiarity). Note that participants did not have to answer the demographic questions on

-----

Table 2: Questions and example answers that would be contained in People also ask (translated from Japanese). As the opinion was not manipulated in Experiment 2, the opposing answers were not required for the alternative question.

gender and educational background. Finally, they were asked if anything had bothered them during the search task to ascertain which participants had noticed the goal of the experiments.

For both experiments, two types of dependent variables were analyzed, i.e., search behaviors and beliefs. Regarding search behaviors, Yamamoto et al. [45–47] discussed the behaviors of people who search carefully. Such people are expected to spend more time searching, issue more queries, browse more documents, check a deeper ranked result, spend more time on a search task, and increase the evidence to support their decisions. In this study, it is assumed that people who search carefully are likely to have the search behaviors mentioned above. The detail of each search behavior will be discussed in Section 5.1.

In this study beliefs were based on the answer obtained from the belief-related questionnaires that were asked before and after the search task. The focus was on cases in which participants changed (or did not change) their prior beliefs from one polarity to the opposite polarity after the search task. We considered cases in which the participants changed their prior beliefs from one polar to the opposite polar as they altered their beliefs, and cases in which participants did not change prior beliefs from one polar to the opposite polar as retained beliefs.

Experiment 1: Regarding RQ1, we investigated the effect of People also ask’s components including the question, answer, and answer’s opinion that participants first encountered on search behaviors and beliefs. According to previous studies, people are likely to spend less effort on a SERP when presented by a question answering system [4, 5, 8, 23]. We assumed that People also ask should have

the same effect as other question answering systems. Therefore, we proposed the following hypothesis.

H1-1: Presenting People also ask discourages people from search- ing carefully compared to when it is not presented.

However, there are studies, which suggested that people are likely to search carefully and alter their beliefs after the search task when they encounter information that is inconsistent with their beliefs [30, 32]. Thus, People also ask might help mitigate confirmation bias if information that is inconsistent with the searcher’s beliefs is included. Based on these discussions, we proposed the following hypotheses.

H1-2: People who first encounter a belief-inconsistent answer are likely to search carefully compared to first encountering a belief-consistent answer. H1-3: People who first encounter a belief-consistent answer are likely to retain their beliefs after the search compared to first encountering a belief-inconsistent answer.

Experiment 2: Regarding RQ2, we investigated the effect of the alternative question, i.e., the question contained in People also ask that is related to the solution that can achieve the same goal as a query on the search behaviors and beliefs. According to Sunstein, people should consider information that they might have not chosen in advance to mitigate confirmation bias and make more effective decisions [35]. In this study, we consider such information as an alternative, i.e., a remedy that differs from the one explained in the search task but which allows people to improve the same symptom. An efficient practice to encourage people to consider an alternative is to present alternatives as questions as suggested by Trope and Bassok that people are likely to consider questions related to the alternative hypothesis when such questions are provided [37, 38]. Therefore, we proposed the following hypotheses.

H2-1: Presenting an alternative question encourages people to search carefully.

-----

H2-2: Presenting an alternative question encourages people to alter their beliefs related to the target remedy.

Experimental design: Participants were presented with People also ask containing four questions, all of which were related to the target remedy. The answers were revealed only if the questions were clicked on. The experiment was manipulated with two conditions: belief-consistent first and belief-inconsistent first. For belief-consistent first, answers that were consistent with participants’ beliefs were placed in the first and second ranked questions while inconsistent answers were placed in third and fourth ranked questions. For belief-inconsistent first, answers that were inconsistent with participants’ beliefs were placed in the first and second ranked questions while consistent answers were placed in the third and fourth ranked questions. Therefore, we were able to balance the number of participants for each type of opinion as the higher ranked items were likely to receive more attention than lower ranked items [14, 17]. In addition, we assigned the controlled condition where People also ask was not presented. Based on these settings, there are three experimental conditions: 𝐶1 (controlled condition), 𝐶2 (belief-consistent first), and 𝐶3 (belief-inconsistent first). This experiment was repeated such that each participant was assigned all three conditions. For each condition, the participants were assigned one of the three tasks in Table 1. The tasks were rotated according to the Graeco-Latin square design [20].

Independent variables: In this experiment, question, answer, and opinion were major factors under investigation. Question is the binary variable indicating whether People also ask was presented. Answer is the variable indicating whether participants clicked on the question to see the answer. Opinion is the variable indicating whether participants first encounter a belief-consistent or a belief-inconsistent answer. The observations can be considered as hierarchical whereby factors are nested within each other such that the answer is nested within the question, which made it possible for participants to click on the question to see the answers only if the questions were presented. Opinion is nested within the answer such that participants were affected by the answer’s opinion only if they clicked on the question.

Statistical analysis: To estimate the effect of the nested factors, orthogonal contrasts are required. Orthogonal contrasts are helpful in obtaining estimates of main and nested effects for mean comparisons between groups of data to obtain specific residuals [28]. Apart from the question that was not nested under any factors, orthogonal contrasts were created separately for answer and opinion. For the analyses regarding RQ1, we conducted linear mixed-effects regression analysis, which is applicable for hierarchical and unbalanced data [3, 44]. We constructed a linear mixed-effects regression model where question, answer, and opinion were regarded as fixed effects. The model applied nested random effect structure, i.e., Question/Answer/Opinion to simulate the hierarchical structure of the experiment [3]. Initially, we included random intercepts and random slopes; 1+Question/Answer/Opinion on participants and tasks to maintain the maximal random effect structure to ensure that the Type I error is controlled [2]. However, we found that the model

3The model is expressed in the syntax of lmer, a widely used mixed-effects fitting method contained in lme4 [3]

did not reach convergence due to the size of the observation data. Thus, we removed the by-participants and by-task random slopes from the model. The limitation of removing random slopes will be discussed in Section 6.2. The final model contains a nested random effect, random intercepts on participants, and random intercepts on tasks. The formula for such model is 𝑑 ∼ Question + Answer

Opinion + (1|Question/Answer/Opinion) + (1 | Participant) + (1

considers only random effects. Likelihood ratio test is helpful to validate whether the fixed effects significantly improved the model as suggested by Field et al [13]. The significance level in this study was set to 5%. Note that log transformation was applied to temporal features, such as page dwell time.

Experimental design: In this experiment, the rank of the alterna- tive question was manipulated based on whether the alternative question appeared on the first or fourth rank of People also ask. In addition, there was also controlled condition where all the questions presented in People also ask were related to the target remedy. All questions contained only the supporting answer toward both target and alternative remedies. Based on these settings, three experiment conditions were generated: 𝐶1 (controlled condition), 𝐶2 (alternative question ranked first), and 𝐶3 (alternative question ranked fourth). Similar to Experiment 1, this experiment was repeated and tasks were rotated according to the Graeco-Latin square design.

Independent variables: In this experiment, alternative ques- tion, answer, and alternative answer were investigated. Alternative question indicated whether the alternative question is presented at rank 1, rank 4, or not presented (control). Answer indicates whether participants clicked on the question to see the answer. Alterna- tive answer indicates whether participants clicked on alternative question once.

Statistical analysis: The observations are hierarchical such that participants could encounter an alternative answer only if the alternative question was presented and clicked on. Thus, orthogonal contrasts were created separately for the alternative question and alternative answer. Similar to the first experiment, a linear mixed- effects regression analysis was performed. A linear mixed-effects regression model was constructed, whereby alternative question, answer, and alternative answer were considered fixed effects. The model applied the following nested random effect structure: Alternative question/Answer/Alternative answer. The model also included random intercepts on participants and tasks. The final model we reported contains nested random effects, random intercepts on participants, and random intercepts on tasks. The formula for such a model is 𝑑 ∼ Alternative question + Answer + Alternative answer + (1|Alternative question/Answer/Alternative answer) + (1

Figure 2 shows a screenshot of our search system. To help the participants get familiar with our search system easily, we imitated the user interface of a commercial search engine with task instructions above the search box. The initial screen comprises the initial query

-----

Figure 2: Screenshot of our search system (translated from Japanese). People also ask was presented above organic search results. The answer was revealed when the question was clicked on. "target remedy symptom" in the search box, People also ask located under the search box, and organic search results under People also ask. The questions and answers in People also ask were selected based on the experiment and condition. Each answer contained text, a title, a snippet, and a URL. Fifty organic search results were obtained from the Bing Web Search API[4] based on the initial query. Each search result comprised a title, a snippet, and a URL. For both the answer and organic search result, the Web page was displayed in a separate tab when either the title or URL was clicked. Note that the initial screen was the only screen that presented People also ask. The participants could freely issue any queries and received organic search results generated by Bing Web Search API, but People also ask would not appear afterward.

Participants were recruited via Lancers for both experiments. Since we used a crowdsourcing platform to conduct the experiment, controlling the quality is important. To control the quality, participants were informed (in the instruction page) that their submissions may be rejected if we judged they did not perform the task seriously. Each participant received approximately $4 as compensation.

Experiment 1: 276 participants were recruited from August 21 to August 22, 2019. Out of 276 participants, we removed 10 participants who did not perform the task seriously, and we removed 14 participants who accidentally closed the browser and attempted to resume the task. We then removed the data of four outliers who spent too little/too much time on the task. In total, 744 search tasks from 248 participants were obtained for analysis. The median task completion time was 17 minutes and 59 seconds, and the standard deviation was 15 minutes and 15 seconds. The demographic data for these 248 participants are shown in Table 3(a).

Experiment 2: 301 participants were recruited from September 27 to September 29, 2019. Out of 301 participants, we removed 28 participants who did not perform the task seriously, and we removed 15 participants who accidentally closed the browser and attempted to resume the task. We also removed a single outlier who spent too much time on the task. In total, 771 search tasks from 257 participants were obtained for analysis. The median of the task completion time were 17 minutes and 33 seconds, while the

4https://azure.microsoft.com/en-us/services/cognitive-services/

Table 3: Demographic data of participants.

Experiment 1 (a)

Gender 𝑛 Educational background 𝑛 Search engine familiarity 𝑛

male 151 university educated 160 rarely use 1 female 96 not university educated 68 several times per week 21 N/A 1 N/A 20 once a day 39 more than once a day 187

Experiment 2 (b)

Gender 𝑛 Educational background 𝑛 Search engine familiarity 𝑛

male 127 university educated 159 rarely use 0 female 129 not university educated 69 several times per week 42 N/A 1 N/A 29 once a day 20 more than once a day 195

standard deviation was 17 minutes and 7 seconds. The demographic data for these 257 participants are shown in Table 3(b). Note that people who participated in Experiment 1 were not permitted to participate in Experiment 2.

This section first discusses the effect of question and answer on participants’ search behaviors and beliefs (RQ1). Then, the effect of an alternative question is discussed (RQ2).

The following search behaviors were analyzed for each participant during the search task:

# of Queries: Number of queries issued in a search task.

# of Clicks: Number of documents clicked on in a search task

(excluding clicks from the answers).

# of Evidence: Number of evidence (excluding those from the

answers).

Deepest Document Rank: Lowest rank the participant clicked.

Page Dwell Time: Average time the participant spent on the

documents.

SERP Dwell Time: Average time the participant spent on the

search results page.

Task Time Spent: Amount of time the participant spent on the

search task.

To verify hypotheses H1-1 to H1-3, we performed a linear mixedeffects regression analysis with each participants’ search behaviors obtained in Experiment 1. Table 4 presents the results of the analysis. The “Fixed Effects” column in Table 4 shows the coefficient, standard error, 𝑡-statistic, and 𝑝-value of each fixed effect.

Effect of question: We found a main effect of the question on the number of queries (𝛽 = −0.73, 𝑝 < 0.001) and time spent on SERP (𝛽 = 0.15, 𝑝 < 0.01). Table 5 shows the results of these behaviors according to whether questions were presented. As can be seen, the presence of questions discouraged participants from issuing queries (absent: 𝑀 = 2.37, present: 𝑀 = 1.80), and spending time on the SERP (absent: 𝑀 = 25.30, present: 𝑀 = 24.99).

Effect of answer: We found a main effect of the answer on the number of queries (𝛽 = −0.27, 𝑝 < 0.01), number of document

-----

Table 4: Mean (𝑀) and standard deviation (𝑆𝐷) for each experimental condition in Experiment 1 (𝐶1: control, 𝐶2: belief- consistent first, and 𝐶3: belief-inconsistent first) and statistical testing results of each search behavior of the model with different fixed effects ( : significance level at 0.001, : 0.01, and : 0.05). Coefficient (𝛽), standard error (𝑆𝐸), t-statistic (𝑡), and p-value (𝑝) of each fixed effect are reported. The 𝜒 [2] and 𝑝-values of the model which is statistically significant compared to the null model are also reported.

Condition Fixed Effects

𝑪1 𝑪2 𝑪3 Question Answer Opinion 𝑀 𝑆𝐷 𝑀 𝑆𝐷 𝑀 𝑆𝐷 𝛽 𝑆𝐸 𝑡 𝑝 𝛽 𝑆𝐸 𝑡 𝑝 𝛽 𝑆𝐸 𝑡 𝑝 𝜒 [2] 𝑝

To verify hypotheses H2-1 and H2-2, linear mixed-effects regression analysis was performed with each participants’ search behaviors obtained in Experiment 2. The results are presented in Table 9.

Effect of alternative question: We found a main effect of the alternative question on the number of queries (𝛽𝑎 = −0.10, 𝑝 < 0.01) and number of evidence (𝛽𝑎 = 0.09, 𝑝 < 0.001). Table 10 shows the results of this behavior according to whether the alternative question was presented. The results demonstrate that the presence of an alternative question encouraged participants to issue fewer queries (absent: 𝑀 = 2.37, present: 𝑀 = 2.02) and submit more evidence (absent: 𝑀 = 1.87, present: 𝑀 = 2.15).

Effect of answer: Similarly to Section 5.2, we found a main effect of answer on the number of queries (𝛽 = −0.85, 𝑝 < 0.001), and time spent on the SERP (𝛽 = 0.58, 𝑝 < 0.001). In addition, we found an effect of the answer on the number of evidence submitted by participants (𝛽 = 0.28, 𝑝 < 0.05). This may due the to data size of this experiment which contained more search sessions than Experiment 1. The results shown in Table 11 demonstrate that answers encouraged participants to issue fewer queries (not encounter: 𝑀 = 2.26, encounter: 𝑀 = 1.41), submit more evidence (not encounter: 𝑀 = 2.00, encounter: 𝑀 = 2.36) and spend more time on the SERP (not encounter: 𝑀 = 19.84, encounter: 𝑀 = 42.00).

Effect of alternative answer: We found a main effect of al- ternative answer on the number of queries (𝛽 = −0.33, 𝑝 < 0.05). The results in Table 12 show that participants who encountered an alternative answer once were less likely to issue queries (not encounter: 𝑀 = 1.73, encounter: 𝑀 = 0.53).

Beliefs: We constructed a logistic mixed-effects model in which beliefs 𝑑 was considered a binary dependent variable (1: alter, 0:retain). Similarly to Section 5.2, the model was not statistically significant compared to the null model (𝜒 [2] = 8.11, 𝑝 = 0.09). Therefore, we cannot accept hypothesis H2-2. However, there is an interesting result to discussed. Although Tables 13 (a) and (b) indicate that most participants altered their beliefs toward yes polar after the search task, the participants were less likely to retain their beliefs in the same polar when the alternative question was presented at

the fourth position in People also ask ( 4 ) compared to the first position ( 3 ). One possible explanation for this result is that highly salient verticals are more likely to be noticed at lower ranks [1]. The alternative question could be considered highly salient item because it satisfied the information need with the information that obviously differs from the other three questions. In addition, the result appears to support the suggestion that diverse information can mitigate the effect of confirmation bias [35].

Summary: We found that the alternative question encouraged participants to issue fewer queries and submit more evidence. Similarly to the Section 5.2, we found that answer encourage participants to issue fewer queries, spend more time on SERP, and submit more evidence. In addition, we found that participants were less likely to issue queries when they encountered an alternative answer. Thus, the results partially support hypothesis H2-1. However, we cannot accept hypothesis H2-2 because the model was not statistically significant compared to the null model.

The analysis revealed that question discouraged participants from issuing queries and spending time on the SERP, while the answer and its opinion encouraged participants to spend more time on SERP, click on more documents, check deeper rank documents, and spend more time on a task. We also found that the alternative question encouraged participants to submit more evidence and discouraged them from issuing queries. This section discusses the implications of the results and limitations.

Regarding RQ1, we found that the question discouraged participants from issuing queries and spending time on the SERP. This is consistent with previous studies, which suggested that people tend to spend less effort on a search task when presented by a question answering system [4, 5, 8, 23]. We also found that the answer and belief-consistent answer encouraged participants to spend more time on SERP, click on more documents, check deeper rank documents, and spend more time on the task. However, the answer discouraged the participants from issuing queries. One possible explanation for these results is that participants who clicked on questions to check the answers performed the task seriously and behaved carefully even prior to participating to the experiment.

Although the previous study suggested that presenting information that is inconsistent with people’s beliefs in organic search results is beneficial because it encourages people to perform a careful information search [30], we did not observe such an effect in this study as belief-inconsistent answers discouraged participants from clicking on documents, checking the deeper rank documents, and spending time on the search tasks. This phenomenon may be explained in reference to a previous study, which suggested that questioners are likely to prefer confirmation responses over disconfirmations [15]. Therefore, participants may considered a belief-inconsistent answer as irrelevant information, which results in perceiving the SERP as being low quality. This implies that presenting belief-inconsistent information as an answer in People also ask may not encourage people to conduct careful information search in contrast to presenting it within organic search results.

-----

Table 9: Mean (𝑀) and standard deviation (𝑆𝐷) for each experimental condition in Experiment 2 (𝐶1: control, 𝐶2: rank 1, and 𝐶3: rank 4) and statistical testing results of each search behavior of the model with different fixed effects ( : significance level at 0.001, : 0.01, and : 0.05). Coefficient (𝛽), standard error (𝑆𝐸), t-statistic (𝑡), and p-value (𝑝) of each fixed effect are reported. For the variable alternative question, 𝛽𝑎 represents the effect of the alternative question while 𝛽𝑟 represents the effect of its rank. The 𝜒 [2] and 𝑝-values of the model which is statistically significant compared to the null model are also reported.

Condition Fixed Effects

𝑪1 𝑪2 𝑪3 Alternative question Answer Alternative answer 𝑀 𝑆𝐷 𝑀 𝑆𝐷 𝑀 𝑆𝐷 𝛽𝑎 𝑆𝐸 𝑡 𝑝 𝛽𝑟 𝑆𝐸 𝑡 𝑝 𝛽 𝑆𝐸 𝑡 𝑝 𝛽 𝑆𝐸 𝑡 𝑝 𝜒 [2] 𝑝

This study has several limitations that should be acknowledged. First, it might be questioned whether the findings are applicable to

Table 13: Percentage of beliefs for Experiment 2 grouped by whether the alternative question was placed at rank 1 or rank 4.

the general population of searchers because only a small number of participants clicked on People also ask (∼20%). In addition, it might be questioned that search behaviors were actually affected by the organic search results or participants themselves behaved carefully prior to the task (especially the ones who clicked on the question to see the answer). The analysis presented in Section 5 cannot fully separate these effects and may thus not be generalizable to the general population of searchers.

Relative to search tasks, the results of this study cannot be generalized to other domains of search tasks. The question remains open whether the findings are applicable to other domains, e.g., the political domain, which is related to one’s social identity. People may have a negative impression of an alternative political candidate from a party whose ideology opposes their beliefs. Nevertheless, search tasks tackled in this study (i.e., health-related search tasks that can be answered with yes or no) are also important search tasks in contemporary society.

This study investigated the effects People also ask has on people’s search behaviors and beliefs. The results demonstrate that the question discouraged participants from issuing queries and spending time on the SERP while the answer encouraged participants to issue fewer queries, spend more time on the SERP, and click more documents. However, we observed that participants who first encountered a belief-inconsistent answer were less likely to click

(a) Alternative question placed at rank 1 (n=257)

Posterior Belief

No Lean-No Lean-Yes Yes (n=7) (n=31) (n=139) (n=80)

No (n=23) 13.04% 21.74% 30.43% 34.78%

3

Lean No (n=58) 3.45% 17.24% 48.28% 31.03% Lean Yes (n=146) 0.68% 10.96% 63.01% 25.34%

3

Yes (n=30) 3.33% 0.00% 40.00% 56.67%

(b) Alternative question placed at rank 4 (n=257)

Posterior Belief

No Lean No Lean Yes Yes (n=11) (n=34) (n=138) (n=74)

No (n=25) 16.00% 12.00% 48.00% 24.00%

4

Lean No (n=78) 3.85% 15.38% 62.82% 17.95% Lean Yes (n=134) 2.99% 12.69% 54.48% 29.85% Yes (n=20) 0.00% 4 10.00% 20.00% 70.00%

-----

on documents, check deeper rank documents, and spend time on the search task. In addition, we found that an alternative question encouraged participants to issue fewer queries and submit more evidence. In this study, we could not confirm an effect of People also ask on beliefs. These results imply that presenting People also ask is convenient for searchers because they tend to exert less effort on the search. Nonetheless, the presence of People also ask may not be able to mitigate confirmation bias. In future, we plan to conduct an experiment to validate the effect of People also ask on beliefs.

This work was supported in part by JSPS KAKENHI Grant Numbers JP18H03494, 18KT0097, JP16H01756, 18H03243, and JP16H02906.

[1] Jaime Arguello and Robert Capra. 2014. The effects of vertical rank and bor der on aggregated search coherence and search behavior. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM’14). 539–548.

[2] Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J.Tily. 2011. Random

effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language 68, 3 (2011), 255–278.

[3] Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear

mixed-Effects models using lme4. Journal of Statistical Software 67, 1 (2015), 1–48.

[4] NJ Belkin, A Keller, D Kelly, J Perez-Carballo, C Sikora, and Y Sun. 2000. Support

for Question-Answering in Interactive Information Retrieval: Rutgers’ TREC-9 Interactive Track Experience.. In The Text REtrieval Conference (TREC-9). 352–821.

[5] Michael S Bernstein, Jaime Teevan, Susan Dumais, Daniel Liebling, and Eric

Horvitz. 2012. Direct answers for search queries in the long tail. In Proceedings of the SIGCHI conference on human factors in computing systems (CHI’12). 237–246.

[6] Celine Buchs, Fabrizio Butera, Gabriel Mugny, and C˜ eline Darnon. 2004.˜ Conflict

elaboration and cognitive outcomes. Theory Into Practice 43, 1 (2004), 23–30.

[7] Naga P. Chalasani, Paul H. Hayashi, Herbert L. Bonkovsky, Victor J. Navarro,

William M. Lee, and Robert J. Fontana. 2014. ACG clinical guideline: the diagnosis and management of idiosyncratic drug-induced liver injury. The American Journal of Gastroenterology 109 (2014), 950–966.

[8] Lydia B Chilton and Jaime Teevan. 2011. Addressing people’s information needs

directly in a web search result page. In Proceedings of the 20th international conference on World wide web (WWW’11). 27–36.

[9] A Cipriani, TA Furukawa, and C Barbui. 2011. What is a Cochrane review?

Epidemiology and psychiatric sciences 20, 3 (2011), 231–233.

[10] Benoit Dardenne and Jacques-Philippe Leyens. 1995. Confirmation bias as a

social skill. Personality and Social Psychology Bulletin 21, 11 (1995), 1229–1239.

[11] Kari Edwards and Edward E Smith. 1996. A disconfirmation bias in the evaluation

of arguments. Journal of Personality and Social Psychology 71, 1 (1996), 5.

[12] Leon Festinger. 1957. A theory of cognitive dissonance. Stanford University Press.

[13] Andy Field, Jeremy Miles, and Zoë Field. 2012. Discovering statistics using R.

SAGE Learning.

[14] Zhiwei Guan and Edward Cutrell. 2007. An eye tracking study of the effect

of target rank on web search. In Proceedings of the 25th SIGCHI Conference on Human Factors in Computing Systems (CHI’07). 417–420.

[15] John Heritage. 2013. Garfinkel and ethnomethodology. John Wiley & Sons.

[16] Irving L. Janis. 1982. Groupthink. Cengage Learning.

[17] Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski,

and Geri Gay. 2007. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transaction on Information Systems (TOIS) 25, 2 (2007).

[18] Eva Jonas, Stefan Schulz-Hardt, Dieter Frey, and Norman Thelen. 2001. Con firmation bias in sequential information search after preliminary decisions: an expansion of dissonance theoretical research on selective exposure to information. Journal of Personality and Social Psychology 80 (2001), 557–571.

[19] Tannis M. Jurgens, Anne M. Whelan, Lara Killian, Steve Doucette, Sara Kirk,

and Elizabeth Foy. 2012. Green tea for weight loss and weight maintenance in overweight or obese adults. Cochrane Database of Systematic Reviews 12 (2012).

[20] Diane Kelly. 2009. Methods for evaluating interactive information retrieval

systems with users. Foundations and Trends® in Information Retrieval 3, 1–2 (2009), 1–224.

[21] Vera Q. Liao and Wai-Tat Fu. 2014. Can you hear me now?: mitigating the

echo chamber effect by source position indicators. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW’14). 184–196.

[22] Vera Q. Liao, Wai-Tat Fu, and Sri Shilpa Mamidi. 2015. It is all about perspective: an

exploration of mitigating selective exposure with aspect indicators. In Proceedings of the 33rd SIGCHI Conference on Human Factors in Computing Systems (CHI’15). 1439–1448.

[23] Jimmy Lin, Dennis Quan, Vineet Sinha, Karun Bakshi, David Huynh, Boris Katz,

and David R Karger. 2003. What makes a good answer? The role of context in question answering. In Proceedings of the Ninth IFIP TC13 International Conference on Human-Computer Interaction. 25–32.

[24] Yossi Matias, Dvir Keysar, Gal Chechik, Ziv Bar-Yossef, and Tomer Shmiel. 2017.

Generating related questions for search queries. US Patent 9,679,027.

[25] Charlan Nemeth, Kathleen Mosier, and Cyntia Chiles. 1992. When convergent

thought improves performance: majority versus minority influence. Personality and Social Psychology Bulletin 18, 2 (1992), 139–144.

[26] Charlan Nemeth and John Rogers. 1996. Dissent and the search for information.

British Journal of Social Psychology 35 (1996), 67–76.

[27] Raymond S. Nickerson. 1998. Confirmation bias: a ubiquitous phenomenon in

many guises. Review of General Psychology 2, 2 (1998), 175–220.

[28] Maria Cristina Stolf Nogueira. 2004. Orthogonal contrasts: definitions and con cepts. Scientia Agricola 61, 1 (2004), 118–124.

[29] Frances A. Pogacar, Amira Ghenai, Mark D. Smucker, and Chales L.A. Clarke.

The positive and negative influence of search results on people’s decisions

about the efficacy of medical treatments. In Proceedings of the 3rd ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR’17). 209–216.

[30] Suppanut Pothirattanachaikul, Takehiro Yamamoto, Yusuke Yamamoto, and

Masatoshi Yoshikawa. 2019. Analyzing the Effects of Document’s Opinion and Credibility on Search Behaviors and Belief Dynamics. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM’19). 1653–1662.

[31] Christina Schwind and Jürgen Buder. 2012. Reducing confirmation bias and

evaluation bias: when are preference-inconsistent recommendations effective and when not? Computers in Human Behavior 28, 6 (2012), 2280 – 2290.

[32] Christina Schwind, Jürgen Buder, Ulrike Cress, and Friedrich W. Hesse. 2012.

Preference-inconsistent recommendations: an effective approach for reducing confirmation bias and stimulating divergent thinking? Computers & Education 58 (2012), 787–796.

[33] Richard B Skov and Steven J Sherman. 1986. Information-gathering processes:

Diagnosticity, hypothesis-confirmatory strategies, and perceived hypothesis confirmation. Journal of Experimental Social Psychology 22, 2 (1986), 93–121.

[34] Mark Snyder and William B Swann. 1978. Hypothesis-testing processes in social

interaction. Journal of Personality and Social Psychology 36, 11 (1978), 1202.

[35] Cass R. Sunstein. 2017. #Republic divided democracy in the age of social media.

Princeton Press.

[36] William B Swann, Toni Giuliano, and Daniel M Wegner. 1982. Where leading

questions can lead: The power of conjecture in social interaction. Journal of Personality and Social Psychology 42, 6 (1982), 1025.

[37] Yaacov Trope and Miriam Bassok. 1982. Confirmatory and diagnosing strategies

in social information gathering. Journal of personality and social psychology 43, 1 (1982), 22.

[38] Yaacov Trope and Miriam Bassok. 1983. Information-gathering strategies in

hypothesis-testing. Journal of Experimental Social Psychology 19, 6 (1983), 560– 576.

[39] Ellen M Voorhees and Dawn M Tice. 1999. The TREC-8 question answering track

evaluation. In The Text REtrieval Conference (TREC-8), Vol. 1999. 82.

[40] Ryen W. White. 2013. Beliefs and biases in web search. In Proceedings of the 36th

SIGIR Conference on Research and Development in Information Retrieval (SIGIR’13). 3–12.

[41] Ryen W. White. 2014. Belief dynamics in web search. Journal of the Association

for Information Science and Technology 65, 11 (2014), 2165–2178.

[42] Ryen W. White and Ahmed Hassan. 2014. Content bias in online health search.

ACM Transactions on the Web (TWEB) 8, 4 (2014), 25:1–25:33.

[43] Ryen W. White and Eric Horvitz. 2015. Belief dynamics and biases in web search.

ACM Transactions on Information Systems (TOIS) 33, 4 (2015), 18:1–18:46.

[44] Bodo Winter. 2013. Linear models and linear mixed effects models in R with

linguistic applications. arXiv preprint arXiv:1308.5499 (2013).

[45] Takehiro Yamamoto, Yusuke Yamamoto, and Sumio Fujita. 2018. Exploring

people’s attitudes and behaviors toward careful information seeking in web search. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM’18). 963–972.

[46] Yusuke Yamamoto and Takehiro Yamamoto. 2018. Query priming for promoting

critical thinking in web search. In Proceedings of the 3rd ACM SIGIR Conference on Human Information Interaction & Retrieval (CHIIR’18). 12–21.

[47] Yusuke Yamamoto, Takehiro Yamamoto, Hiroaki Ohshima, and Kawakami Hi roshi. 2018. Web access literacy scale to evaluate how critically users can browse and search for web information. In Proceedings of the 10th ACM Conference on Web Science (WebSci’18). 97–106.

-----