The Impact of Non-Verbalization in Think-Aloud: Understanding Knowledge Gain Indicators Considering Think-Aloud Web Searches

Marcelo Tibau, Sean Wolfgand Matsui Siqueira, Bernardo Pereira Nunes

Published in: HT ’22: Proceedings of the 33rd ACM Conference on Hypertext and Social Media · DOI: 10.1145/3511095.3531272

ACM source attribution: Reconciled from authorized ACM publisher HTML and ACM HyperText 2022 proceedings PDF. Publisher image endpoints returned HTTP 403; the corresponding embedded PDF visual regions are preserved as IPFS assets.

---

Marcelo Tibau, Federal University of the State of Rio de Janeiro, Brazil, marcelo.tibau@uniriotec.br

Sean Wolfgand Matsui Siqueira, Federal University of the State of Rio de Janeiro, Brazil, sean@uniriotec.br

Bernardo Pereira Nunes, Australian National University, Australia, Bernardo.Nunes@anu.edu.au

DOI: https://doi.org/10.1145/3511095.3531272 HT '22: Proceedings of the 33rd ACM Conference on Hypertext and Social Media, Barcelona, Spain, June 2022

Web searching and knowledge gain are intertwined processes that share mental and physical activities at the core of both human cognition and hypertext theory, such as identifying, comparing, linking, and combining different subsets of existing or new information. As a consequence of the improvement of our ability to retrieve information across multiple sources provided by Web search engines, the necessity to understand how a user's knowledge evolves through a Web search session increased. Previous works focused on understanding the knowledge gained in Web searches by using think-aloud protocols. From the user's verbalization of her searching procedures, it is possible to identify her cognitive processing. Notwithstanding, we argue that user's searching and browsing behaviors should be analyzed not only through the verbalization periods, as usually accepted by think-aloud studies, since not all cognitive decisions are made consciously, some are unconscious or subconscious. Hence, it is possible to identify more knowledge gained than it would be attainable focusing solely on what was verbalized. In this sense, we evaluated the statistical significance level derived from the relationship between verbal and non-verbal search periods mapped from online information searching strategy indicators. Then, we identified a positive association regarding non-verbalization and some indicators related to knowledge gain concepts and discovered that the values of non-verbal periods tend to increase as the values of particular indicators related to knowledge gain also increase. The knowledge gain concepts were identified using constructs representing cognitive absorption, comprehension, elaboration, and memory. Concerning the impact of Think-Aloud on knowledge gain processes, we found out that verbalization does affect how participants handle their search tasks. However, our result also showed a predominance of non-verbal periods during metacognitive-based searching activities, which may indicate that Think-Aloud protocols should not only rely on verbalization for indication of knowledge gain. Although verbalization may not disrupt the thought process, it might cut in on the cognitive process as the participant tries to explain her action while performing it. A search engine could use the identified indicators to account for the knowledge gained during search sessions, which would make it more adapted to identify user information needs and promote personalized information-adding.

CCS Concepts: • Information systems;

Keywords: Think-aloud protocols,

online information searching strategy, knowledge gain indicators, hypertext theory, Web search.

ACM Reference Format: Marcelo Tibau, Sean Wolfgand Matsui Siqueira, and Bernardo Pereira Nunes. 2022. The Impact of Non-Verbalization in Think-Aloud: Understanding Knowledge Gain Indicators Considering Think-Aloud Web Searches. In Proceedings of the 33rd ACM Conference on Hypertext and Social Media (HT '22), June 28-July 1, 2022, Barcelona, Spain. ACM, New York, NY, USA 14 Pages. https://doi.org/10.1145/3511095.3531272

1 INTRODUCTION

During online searching, users are subjected to a set of information-seeking subprocesses (e.g., information need recognition, search problem definition, source selection, query formulation and execution, result examination, and information extraction) that demand much time and mental effort. Although Web search engines have rendered some of these subprocesses almost effortless (e.g., query auto-completion), strain and time dedicated to results’ examination are still high. The reason is that there is still “more to learn, more forms of knowledge expression, and more revision and change in what there is to know” [32]. Searching environments continuously shift the way people devote mental and physical resources to information seeking [32]. Therefore, understanding users’ online information searching behavior and acquiring ways to evaluate their knowledge gain from sources to people [40] are important to advance search engine's development on subjects like intent matching [22]; recommender systems, domain and user modeling [23]; and dwell time distribution for implicit feedback [28].

Think-Aloud protocols are used to capture users’ direct reasoning during their task performance [11]. They have been applied to identify knowledge gain in searching processes [19] [15] [14] through the user's verbalization of her searching procedures. The basic principle of the Think-Aloud method is that participants are asked to verbalize what is in their minds as they solve a wide variety of problems [10]. Examining the features that provide knowledge gain in search sessions using a Think-Aloud searching experiment, we noticed a non-verbal predominance when the participants performed cognitive activities related to their examination of the search results. Moreover, we noticed that those examinations could be connected to knowledge gain. Since a Think-Aloud protocol demands verbalization and the ordinary Web search is usually carried out in silence, this paper identified periods of verbalization and non-verbalization (silence, verbal fillers, and writing) during a qualitative searching study. The Think-Aloud protocol was conducted with 26 users from March 2020 to July 2021. Each analysis took an average time of eight hours corresponding to 208 hours in total. We used an online information searching strategy (OISS) framework to compare the identified verbal and non-verbal periods to searching strategy indicators. The goal was to find out whether there is a statistical significance between search activities during periods of verbalization and non-verbalization and indicators related to knowledge gain activities, such as comparing and contrasting the pieces of information retrieved from different websites and keeping an account of the information relevant to the search task found on a website.

Including this introductory first Section, this paper is organized as follows: Section 2 presents the theoretical grounding; Section 3 presents some related works; Section 4 contextualizes the Think-Aloud method and presents the statistical significance method applied as well as the hypotheses evaluated; Section 5 presents the result and its discussion; and, finally, Section 6 concludes the paper.

2 THEORETICAL GROUNDING

2.1 Think-Aloud

Think-Aloud protocol's theoretical grounding is rooted in the concept of inner speech [52, p.237-239] and its interaction with abstract thought [52, p.263-264], as an inner movement through different planes (abstract and verbal). “Inner speech is an autonomous speech function” that can be regarded as a “distinct plane of verbal thought.” It comes to fruition as “a complex, dynamic process involving the transformation of the predicative, idiomatic structure of inner speech into syntactically articulated speech intelligible to others” [52, p.263]. To Vygotsky, “thought is not merely expressed in words; it comes into existence through them” (p.231). Charters [10] noted that the application of inner speech to Think-Aloud protocol makes its predicate dominance clearer because the subject of the Think-Aloud is usually visible and evident to the participant.

Even though Think-Aloud protocols are widely used to gather data in studies involving cognitive psychology, there is still some controversy regarding insufficient knowledge on two issues: its completeness and accuracy and its reactivity [27]. Completeness and accuracy raised questions regarding whether a Think-Aloud report is consistent with one's thinking process and product [27]. About the matter, we argue that Vygotsky's concept of inner speech is a robust indication against this particular controversy.

On the other hand, reactivity raised questions about whether one's task performance under the condition of thinking aloud differs from one's performance in the same task under the condition of not thinking aloud [53]. Regardless of the thought-word connection, the requirement to Think-Aloud while working may affect how participants handle tasks, the time it takes them to carry out tasks, and their eventual success in task completion [48]. Indeed, reactivity can change the sequence of thought processes and interfere with cognitive processing [11], which motivated us to verify the impact of verbalization/non-verbalization on the searching process.

2.2 Hypertext Theory

Hypertext theory spans works from computer science to literary studies and became closely identified with post-structuralist theory [6], especially the concept of deconstruction [24, p.2-3]. Deconstruction considers that a text has not just one but many different meanings and should be seen as an endless stream of signifiers, with words only pointing to other words without any final meaning [13, p.6-26]. In this sense, Landow argues that conceptual systems founded upon ideas of center, margin, hierarchy, and linearity must be abandoned and replaced by ideas of multilinearity, nodes, links, and networks [24, p.2]. Barthes proposes a division to the concept of text, claiming that texts are either readerly or writerly [7, p.4]. Readerly means there is a unilateral relationship between a text and reader (traditional text), and writerly requires the reader to perform a more active role by browsing from a departure node to a destination node by links (hypertext). Hypertext would mark a new paradigm, introducing a mutual process of communication1, establishing a bilateral (or multilateral) relationship between text and reader and requiring an active and attentive reader who has to find out the text's meaning [7, p.142-148].

Hypertext theory adheres to our study through the user's browsing behavior. The notion of author is reconfigured to be conceived not as the author of the text but as the text itself [25, p.126] as well as the reader. They became a centerless network of codes (the author, reader, and text), which serve as nodes within another centerless network (the internet) [25, p.127]. Hence, identifying and relating users’ searching behaviors to the knowledge gained during the Web search session is related to hypertext and information exploration and visualization. It is a step toward improving Web search engines to aid users in satisfying their information retrieval-related objectives.

3 RELATED WORKS

Regarding Think-Aloud studies and cognitive activities related to knowledge gain, Hofer [19] states that thinking aloud “may be the best means of learning about the actual, situated nature of epistemic thinking”. However, Hofer [19] does not indicate the number of students who participated in the study or a method to reach sample saturation. Goldman et al. [17] used a think-aloud protocol methodology to better understand the processing that learners engaged in during a learning task from information sources they find on the Internet. They had 21 participants divided in two categories (better learners, 10 individuals, and poor learners, 11 individuals) and discovered that better learners would spend more time on reliable sites than poorer learners. Also, that better learners were more strategic than the poorer learners in both how and what they read and used more monitoring and evaluation processes to determine not only what they understood from the information provided but also whether it was scientifically credible or task relevant. Besides monitoring and evaluation processes, our study also considered the basic skills required for manipulating and searching the Web, such as how the participants manipulated the browser and search engine and their self-awareness about their searching orientation. In addition, our study also considered procedural approaches such as trial-and-error and problem-solving strategies.

Gider and Hamm's study [16] focused on search and information processing approaches used by 21 participants to find information on ethical production methods. They examined the participants’ navigation behavior on websites and conducted think-aloud protocols to map their information search through preselected websites of food companies to find information on these companies’ corporate social responsibility (CSR) activities. They found that the website's organization influence on the participants’ ability to find the information needed and their navigational approach. For example, the participants were more likely to navigate through the horizontal navigation bar than via a hyperlink in the text. Our study did not preselected sources, instead we set-up search tasks in which the result from one search session was built upon the result from the previous session. Macias et al. [29] employed the think-aloud protocol to gain an in-depth look at how 23 individuals searched for online health information. By examining the participants’ search ability, motivation, depth of search and knowledge gain, they found out that individuals’ search patterns are most likely part habit, part positive/negative reinforcement, and part how their brains work. They observed two main reading/exploring strategies: click, read, go back, and repeat or the seemingly more advanced, but less common, strategy of opening multiple tabs from the main search screen and then slowly working through them and closing the tabs as they went. Our study applied an online information searching framework with 35 strategies indicators, which allowed us to observe strategies related not only to reading/exploring strategies but also to knowledge gain strategies. Additionally, we were able to verify the impact of query reformulation and searching strategies on online behavior and the way users perceive, remember, and assess the retrieved information.

In addition to the differences already exposed, the cited studies focused only on periods of verbalization, unlike our study that also accounted for non-verbalization periods. Also, our study included more participants (a total of 26).

According to some researchers [37] [36, p.156], five to nine participants using a Think-Aloud protocol are sufficient for an effective usability test. Lewis and Mack [26] had six participants to show the evidence of thinking aloud protocols in a learning situation on an information system, the same number of participants of Dias’ work [14]. Van Oostendorp and de Mul [49] considered sixteen university students divided into two groups to analyze learning by exploration. Gerjets et al. [15] had 15 participants asked about the evaluation criteria they applied while selecting search results and assessing Web pages.

4 THE THINK-ALOUD METHOD

4.1 Procedures

26 Master and Ph.D. students were selected from five different Brazilian universities2 to be part of this qualitative study. They were chosen due to their information searching experience, which can be determined based on the students’ daily information searching time and their knowledge of information searching techniques [5] [30]. The profile was determined following Aula and Nordhausen's procedure [5] to ascertain the participants’ background by tracking their years of computer and Web experience; frequency of using search engines, the Web and computers; and their own evaluation of search skills. Their familiarity with the tasks was also evaluated. The data saturation process is described in Appendix 1.

The participant's sessions took place through online chat (i.e., Google Meet), and the chat screens were recorded with the participants’ consent. The role of the researchers in the online sessions consisted mostly in reminding the participants to verbalize their thoughts, as required by the chosen Think-Aloud protocol. The participants were asked to perform three different search tasks (Table 1) based on instructional activities developed by Kelley et al. [21]. The difficulty of the tasks was planned with different levels, with the first two tasks meant to encourage the participant to perform a search with learning intent and the last task proposing a challenging activity using the information retrieved from the previous two searches.

Table 1: Overview of the proposed search tasks.

Search Session	Task Description
Search 1: Introduction to weight, mass, volume and density	Describe weight, mass, volume and density. Compare and contrast weight and mass.
Search 2: Skeletal and muscular systems and movable joints	Identify major bones and muscles in the human body. Identify and locate examples of movable joints in the human body.
Search 3: Prosthetic limb	Explain the features of movable joints in a prosthetic limb.

We asked the participants to open a Web search engine of their choice to initiate the tasks, and we captured the query states, starting with the terms they used to formulate the initial query. The other query states derived from the ensuing reformulation performed by the participants either on the search engine or a website or platform embedded searching system. In addition to the terms, we registered the whole sequence of actions and spoken descriptions based on the Tsai and Tsai's framework [47] as described by Reisoğlu et al. [42] to identify the online information searching strategy (OISS). Reisoğlu et al. [42] thoroughly described codes and indicators for each OISS, which we transcribed in Table 2 to support the reader's understanding. The framework is composed of three main domains3: Behavioral, Procedural and Metacognitive. The Behavioral domain has two sets of strategies, Control (with ten indicators) and Disorientation (with four indicators). The Procedural domain has two sets of strategies, Trial and Error (with three indicators) and Problem-Solving (with two indicators). Finally, the Metacognitive domain has three sets of strategies, Purposeful Thinking (with four indicators), Select Main Ideas (with eight indicators), and Evaluation (with four indicators).

Table 2: Online Information Searching Strategies’ indicators [42].

Behavioral (Behav)

Control

C1: Using the most familiar or known search engine in the first place.

C2: Searching by typing the name of the search engine on the browser.

C3: Entering the name of the website on the search engine.

C4: Entering the name of the website on the address bar.

C5: Using the “home” button to return to the beginning of the search.

C6: Using the “next” and “previous” buttons of the browser.

C7: Using Boolean logic operators for narrowing/widening the search parameters.

C8: Doing a customized search with the help of the images, videos, maps, and other similar features of the search engine.

C9: Utilizing the advanced search options of images, videos, maps, and other similar features of the search engine.

C10: Utilizing the advanced search options of the search engine.

Disorientation

D1: Giving up in the case of failure to find an answer.

D2: Using search terms that are not given in the search task.

D3: Not having any idea about what to do when doing an Internet search.

D4: Feeling bad in the case of failure to retrieve the desired information.

Procedural (Proced)

Trial and Error

TE1: Modifying the keywords.

TE2: Using different search engines.

TE3: Opening different websites.

Problem-Solving

PS1: Doing one's best to resolve any problem that occurs during a search.

PS2: Trying to find out the possible reasons for any problem that occurs during a search.

Metacognitive (Metacog)

Purposeful Thinking

PT1: Narrowing down the searching field (subject).

PT2: Accessing additional websites from a main website.

PT3: Simultaneous information searching from different sources.

PT4: Doing in-site search.

Select Main Ideas

SMI1: Directly opening a website that is known to be relevant to a given search task.

SMI2: Typing specific terms about the search task.

SMI3: Following the search suggestions of the search engine.

SMI4: Following the outputs of in-site search.

SMI5: Looking through the hyperlinks provided on a website.

SMI6: Looking through the titles on a website.

SMI7: Keeping an account of the information relevant to the search task found on a website.

SMI8: Looking for specific words in a website by means of Ctrl + F.

Evaluation

E1: Evaluating the relationships between the retrieved pieces of information.

E2: Comparing and contrasting the pieces of information retrieved from different websites.

E3: Determining whether a piece of information from a website is worth referencing.

E4: Assessing how to combine and present the data gathered from the Web.

The framework presented in Table 2 aided us in establishing a set of codes that could be applied to categorize our data. The coding process was executed by the researchers themselves. We registered the terms used in each search task by each user, the mapped query state4, and their online information searching strategies.

We transcribed the participants’ search sessions to show how keywords and terms evolved throughout the search and identified the searching strategies and domains based on the indicators from the participants’ behaviors (see Table 2 for the codes’ description). The online information searching strategy was determined by the user's attitudes and explanations during the Think-Aloud procedures.

4.2 Knowledge Gain Indicators

Controlled cognitive processing is an information-processing approach defined by cognitive psychology as the required conscious effort from individuals to comprehend, elaborate, and restore into memory the information presented by an information source [2, p.9-10]. Its effects can be measured by different outcome variables such as knowledge, affect, attitude, intention, or actual behavior [53]. Controlled cognitive processing is applied to examine the range of cognitive responses related to knowledge gain. We analyzed all the identified indicators considering their intrinsic concept to determine knowledge gain strategies. For that matter, it is essential to observe the appearance of four specific elements during a cognitive activity, e.g., a Web search, namely the concepts of cognitive absorption, comprehension, elaboration, and memory.

Cognitive absorption is a key prerequisite for controlled cognitive processing related to attention by selectively concentrating on certain objects while ignoring perceivable others [53]. The background of cognitive absorption derives from three closely inter-related domains: the trait of absorption (the propensity of deep awareness and engagement); the state of flow (characterized by a total concentration in an activity); and the notion of cognitive engagement (defined as a significant learning outcome derived from technology-mediated learning) [1]. Cognitive absorption is influenced by the perceived usefulness of an information source [34].

Comprehension determines the extent to which one can understand a piece of information and comprises the content and the structure of the message being communicated [53]. It involves the perception of the encoding used (e.g., the language), the process by which the content is transformed into a mental representation, and the understanding of how to use the mental representation (e.g., if the mental representation is a question, one may answer it) [2, p.313].

Elaboration is built upon comprehension and is a key asset to substantive knowledge gain [53]. It is defined as the degree to which individuals cognitively process the information to scrutinize its argument, weigh the evidence, analyze the logic, and assess the soundness of the claims presented [45]. Elaboration manifests itself as the ability to generate relevant thoughts and fill in additional information about the information retrieved, and, in Web searching, it has often been linked to thoughts users generated in response to website information [53].

Memory has been commonly operationalized as recall and recognition, and it refers to the storage and retrieval of information presented [53]. It confers the ability to judge whether one has encountered an object, idea, or situation in the past [35]. It is another essential cognitive component of knowledge gain in Web searching because users tend to jump back and forth among pages and to skip and attend only to certain parts of a message [53].

To observe the appearance of the four specific elements during the search sessions, we borrowed constructs from the mathematical logic to use in the concepts of cognitive absorption, comprehension, elaboration, and memory. The term “construct” is used in this work as understood by Set Theory's logic. Quine defines it as sets of ordered pairs matching each member of the population to a specific value for the variable [41, p.291]. As constructs, the concepts’ attributes (components or characteristics) and the indicators can be represented as a two-place logical relation5 [33]. It means that the variable (i.e., the concept's attribute) can be treated as a predicate that either applies to a particular subject (i.e., the indicator) or not [33]. For example, if H represents “honesty” as a simple binary, non-vague predicate, then for any person, x, either H(x) or ¬H(x). That is, each individual has the property of honesty or not [33].

Then, the concepts’ attributes were chosen based on the following definitions: cognitive absorption as the construct that better describes the state of full engagement and immersion [31]; comprehension as the construct that better describes the abilities to interpret word meaning in context, to recognize relevant information, and to synthesize scattered information [46]; elaboration as the construct that better describes the abilities to organize knowledge into a coherent structure and integrate new information with existing knowledge structures [20]; memory as the construct that better describes the abilities to encode, store, and retrieve data or information when needed [38].

The concept attributes were used to evaluate the 31 indicators based on their description shown in Table 2. As the attribute is binary, a score of 1 indicates presence and a tagged 0 otherwise. Table 7, in the Appendix 2, presents the knowledge gain concepts’ presence/absence evaluation. We were able to determine sixteen indicators with at least one of the attributes related to knowledge gain: PS2, PT1, PT3, PT4, SMI1, SMI2, SMI3, SMI4, SMI5, SMI6, SMI7, SMI8, E1, E2, E3, and E4. We called them knowledge gain indicators due to their relation to the knowledge gain attributes.

4.3 Hypotheses

On reactivity (see Subsection 2.1), some factors may modulate the potential difference of performance. We argue that articulating thoughts in words during a search task may not account for the normal condition of one's search performance. During our analysis of the recorded searches, we realized that the participant reflected on the content seen in a non-verbal way in many cases. Also, we realized that some of the 31 identified indicators (16 of them) could be related to knowledge gain concepts [53]. The cited knowledge gain require higher-order thinking skills and beliefs about the usefulness of the information or knowledge in question [43, p.317]. Then, we decided to verify if verbalization periods6 discourages knowledge gain during Web searches by analyzing the participants’ behavior regarding the searches’ periods of verbalization or non-verbalization compared to the remainder of the observed indicators.

In this sense, to answer the question “does verbalization discourages knowledge gain during Web searches?”, we sought to verify the ensuing null and alternative hypotheses:

Null-hypothesis (H0): knowledge gain indicators are not related to search activities during periods of verbalization;

Alternative-hypothesis (H1): knowledge gain indicators are related to search activities during periods of verbalization, but not related to search activities during periods of non-verbalization;

Alternative-hypothesis (H2): knowledge gain indicators are related to search activities during periods of both verbalization and non-verbalization.

If the null-hypothesis (H0) is confirmed, it means that there are statistically sufficient grounds to believe there is a negative relationship between verbalization and knowledge gain periods because thinking aloud would affect the participants’ online information searching strategies during the search sessions (i.e., preventing them or interfering with their ability to gain knowledge). If the null-hypothesis (H0) is rejected, it means that verbalization would not influence their online information searching strategies. Nevertheless, their ability to gain knowledge might be positively related to their choice of whether think aloud or not (H1), or yet not be affected at all by it (H2).

4.4 Evaluation method

The traditional approach to evaluating the association between two categorical variables uses the Pearson chi-square test of independence [8] [3]. As the chi-square test of independence is designed under the assumption that all observed frequencies are in the determined form (i.e., are not an approximate estimation) [3], it is a good fit to test whether the two criteria of classification presented in either hypothesis (null and alternative) are independent or not. For completeness, the chi-square test equation is shown in Appendix 3. The calculated chi-square is then compared to a critical value, which is determined by the level of significance (typically, p-value = 0.05) and the degrees of freedom (df), which equation is also shown in the Appendix 3.

We also analyze the nature of the dependence between the row and the column variables to determine the most contributing cells to the total chi-square. We achieve it by calculating the Pearson residuals (PeaRes) for each cell. Once again, its equation is shown in Appendix 3. The Pearson residuals are used to verify the type of association between a particular row and column. If the value is positive, it is said that they have a positive association (also known as an attraction). On the other hand, if the value is negative, it is said that they have a negative association (a.k.a. repulsion).

5 RESULTS AND DISCUSSION

5.1 The identified indicators

We examined the data for patterns based on the constant comparative method (CCM) [9]. Each participant's performance was analyzed to label the searching strategies with the most appropriate codes. Then, we compared the participants’ searches. The purpose of this type of comparison is twofold: firstly, it intends to identify similar strategies which could be grouped together, forming a pattern; secondly, to observe the impact of adding a new participant and determine the overall number of participants to the study. In this case, when new participants did not bring any new information to our analysis and the formula we used to calculate the discovered sample (see Appendix 1) indicated the standard percentage of 80-85% of discovered information, we considered the study saturated.

An amount of 26 participants was needed to reach the discovered sample equivalent to 0.86 (above the 80-85% threshold). The average time of the participants’ search session was one hour and nine minutes (average time=1:09:19) and the standard deviation was twenty-nine minutes and fifty-three seconds (σ=0:29:53), considering the three search activities (see Table 1). We identified 31 out of 33 possible indicators7 from a total of 78 search activities.

We counted the verbalization and non-verbalization periods from the search session videos. The analysis was conducted independently, with each video annotator reviewing separately all the videos. By comparing different parts of the search with the answers provided by the participant, the interview consistency was examined during the coding process and the annotators agreed on the results by comparing their notes from their coded tables. When the participants verbalized what they were doing or their thoughts regarding their searching activities and results checking, we considered it a period of verbalization. When the participants remained silent while performing a searching activity and results checking or some task-relating activities (e.g., reading or writing) or yet used a verbal filler, which is an apparently meaningless word, phrase, or sound that marks a pause or hesitation in verbal speech (e.g., terms like “uh-huh” or “mmhm”) [12], we considered it a period of non-verbalization. The observed frequencies for each indicator identified are shown in Table 3.

Table 3: The observed frequencies for each indicator.

Indicator	Verbalization	Non-verbalization	Indicator	Verbalization	Non-verbalization
C1	26	23	PT1	110	40
C2	20	9	PT3	5	24
C3	1	1	PT4	8	3
C4	7	2	SMI1	21	5
C6	64	95	SMI2	95	24
C7	4	1	SMI3	36	13
C8	45	18	SMI4	37	13
C9	1	2	SMI5	21	13
C10	9	4	SMI6	18	271
D2	3	2	SMI7	54	108
D4	2	0	SMI8	27	35
TE1	101	28	E1	7	33
TE2	1	1	E2	19	36
TE3	56	10	E3	59	75
PS1	9	0	E4	21	82
PS2	1	0	-	-	-

5.2 The calculations

The calculations were done using R. The data were entered as shown in Table 3, with rows representing each indicator, columns accounting for the periods of verbalization or non-verbalization, and values representing the frequencies scored by the indicators for each period type. The expected values (Table 4 - its formula is shown in Appendix 3) and the degrees of freedom (df=30) were calculated.

Table 4: The expected frequencies under the null hypothesis of no association

Table 4: The expected frequencies under the null hypothesis of no association.

The expected values are the values that would be expected under the null hypothesis if there is no association between the variables. The chi-square (χ2) is based on a test that measures the observed data's divergence from the expected values. The calculated chi-square is the following: (χ2=536.24), which indicates a substantial divergence between the observed data and the null hypothesis8. However, the number by itself is not that useful because we cannot certify if the χ2=536.24 is unusually large since there is no previous calculation regarding the instance. Hence, we need to place it into a broader context to determine whether it is an extreme value or not. The observed p-value=4.326807e-94 can provide a broader context. Note that the value is indeed less than the assigned significance level (≤ 0.05). In fact, it is very close to zero, which would indicate a significant result.

To cross-validate the results, we calculated the Pearson residuals. The statistical model underlying the Pearson chi-square test of independence is that the cell counts are independent Poisson random variables [4]. It means that the number of events (periods of verbalization or non-verbalization associated with the indicators) that happen during a fixed time interval (during the search tasks) is a random number with a Poisson distribution (no particular indicator changes the probability of when the next indicator will occur). Since a total row or column count is imposed, the tally's resulting conditional distributions become multinomial [44], allowing calculating the residuals. A residual is a measure of the distance a data point is from a particular line; in other words, it shows how well this particular line fits an individual data point. The calculated Pearson residuals are shown in Table 5.

Table 5: The Pearson residuals

Table 5: The Pearson residuals.

Note that the positive and negative residuals do bear a relation to the frequencies shown in Table 3. Figure 1 provides a better look at the nature of the dependence between the variables. We slightly changed the indicators’ position (8 out of 16 knowledge gain indicators in the first positions) to enhance the effect. Positive residuals are in blue and negative residuals are in red. The knowledge gain indicators positively associate with the non-verbalization column. Besides the eight indicators (PT3, SMI6, SMI7, SMI8, E1, E2, E3, and E4), the C6 and C9 indicators also bear an attraction to the non-verbalization while the remaining indicators show a repulsion. In this sense, we can reject the null hypothesis (H0). Concerning the alternative hypotheses, we argue for the acceptance of the H2 since half of the identified knowledge gain indicators bear an attraction to non-verbalization periods, and half of them bear an attraction to verbalization periods.

Figure 1

Figure 1: Residuals visualization - positive residuals are in blue and negative residuals are in red.

Concerning the impact of Think-Aloud on knowledge gain processes, verbalization does affect how participants handle their search tasks, as the Pearson residuals show, 21 out of 31 indicators positively associate with verbalization. However, a better part of the identified Metacognitive strategies is performed in a non-verbal way (8 out of 15). Eight knowledge gain indicators with a positive association with non-verbalization also have at least two concept attributes associated with knowledge gain (see Table 7 in Appendix 2). These knowledge gain-related activities tend to be performed in quietness despite Think-Aloud's verbalization influence. Their appearing indicates a controlled cognitive processing applied by the participants when performing strategies related to metacognition (i.e., indicators related to Purposeful Thinking, Selection of Main Ideas, and Evaluation). As the result shows a predominance of non-verbal periods during metacognitive-based searching activities, it may indicate that Think-Aloud protocols should not only rely on verbalization periods for indication of knowledge gain, non-verbalization periods should be considered as well. We argue that non-verbalization periods is still relevant when Concurrent Think-Aloud protocol is not used, as there is the case with other Think-Aloud methods in which participants are not required to verbalize their thoughts when performing tasks, such as the Retrospective and the Partial Concurrent Think-Aloud. Although verbalization may not disrupt the thought process, it might cut in on the cognitive process as the participant tries to explain her action while performing it. Also, search engines could stimulate more interactions for exploration and information-adding focused on knowledge gain interactions, regardless they are from verbal or non-verbal behavior. As Marchionini suggests, it is “increasingly apparent that information seeking augments our intellects at various levels of life” [32]. A computationally enhanced search process focused on knowledge gain interactions (e.g., grounding on the concepts of cognitive absorption, comprehension, elaboration, and memory) could facilitate the knowledge gain during online search sessions with impacts on identifying user information needs and adding personalized information throughout the search.

6 CONCLUSION

In this paper, we seek to verify the impact of verbal and non-verbal periods in Think-Aloud protocols. We were able to observe a predominance of periods of non-verbalization when the participants displayed a behavior more likely to be characterized as cognitive. The fact that both the p-value and the Pearson residuals did indicate the association between knowledge gain indicators and periods of non-verbalization seems to provide a window to explore the overlap between hyperlink theory, searching, and knowledge gain by observing the cognitive effects of search interactivity. Furthermore, it might provide features to a search engine to account for the knowledge gained from a search session if it can successfully identify at least the eight OISS indicators related to cognitive tasks captured by non-verbalization. It would provide more opportunities to identify user information needs and promote personalized information-adding.

This qualitative research is subjected to a threat regarding its complex, subjective and time-consuming nature. We tried to mitigate it by defining objective procedures to identify and assess the online information searching strategy indicators and identify the positive association to knowledge gain concepts, and also by conducting a methodical and thorough data saturation and comparative analysis processes. As future work, we intend to expand the diversity of participants in the study to include novices to Web searching and different knowledge domains to observe possible differences in participants’ behavior. To better address reactivity, we would like to explore two scenarios: with think-aloud and without think-aloud, and compare the results regarding the knowledge gain indicators. Also, Think-Aloud protocols could explicitly monitor the identified knowledge gain indicators, paying more attention to periods of non-verbalization.

ACKNOWLEDGMENTS

This study was financed in part by the “National Council for Scientific and Technological Development (CNPq) - Brazil” - Process 315374/2018-7, Project “Searching as Learning: the information search as a tool for learning” and by the “Coordination for the Improvement of Higher Education Personnel” (CAPES) – Brazil – Finance Code 001.

REFERENCES

Ritu Agarwal and Elena Karahanna. 2000. Time Flies When You're Having Fun: Cognitive Absorption and Beliefs about Information Technology Usage. MIS Quarterly 24, 4 (2000), 665–694. http://www.jstor.org/stable/3250951

John R. Anderson. 2015. Cognitive Psychology and Its Implications. Worth Publishers, New York.

Muhammad Aslam and Osama H. Arif. 2020. Test of Association in the Presence of Complex Environment. https://doi.org/10.1155/2020/2935435

A. C. Atkinson. 1979. The Computer Generation of Poisson Random Variables. Journal of the Royal Statistical Society: Series C (Applied Statistics) 28, 1(1979), 29–35. https://doi.org/10.2307/2346807

Anne Aula and Klaus Nordhausen. 2006. Modeling successful performance in Web searching. Journal of the American Society for Information Science and Technology 57, 12 (2006), 1678–1693. https://doi.org/10.1002/asi.20340

Craig Baehr and Susan M. Lang. 2012. Hypertext Theory: Rethinking and Reformulating What We Know, Web 2.0. Journal of Technical Writing and Communication 42, 1(2012), 39–56. https://doi.org/10.2190/TW.42.1.d arXiv:https://doi.org/10.2190/TW.42.1.d

Roland Barthes. 1993. Image, Music, Text. Fontana Press.

Eric Benhamou and Valentin Melot. 2018. Seven proofs of the Pearson Chi-squared independence test and its graphical interpretation. arXiv:1808.09171 [math, stat] (Sept. 2018). http://arxiv.org/abs/1808.09171 arXiv:1808.09171.

Hennie Boeije. 2002. A Purposeful Approach to the Constant Comparative Method in the Analysis of Qualitative Interviews. Quality and Quantity 36 (11 2002), 391–409. https://doi.org/10.1023/A:1020909529486

Elizabeth Charters. 2003. The Use of Think-aloud Methods in Qualitative Research An Introduction to Think-aloud Methods. Brock Education Journal 12, 2 (2003).

L. Cooke. 2010. Assessing Concurrent Think-Aloud Protocol as a Usability Test Method: A Technical Communication Approach. IEEE Transactions on Professional Communication 53, 3(2010), 202–215. https://doi.org/10.1109/TPC.2010.2052859

Ludivine Crible and Elena Pascual. 2020. Combinations of discourse markers with repairs and repetitions in English, French and Spanish. Journal of Pragmatics 156 (Jan. 2020), 54–67. https://doi.org/10.1016/j.pragma.2019.05.002

Jacques Derrida. 1998. Of grammatology. Johns Hopkins University Press, Baltimore.

Marcelo T.V Dias. 2019. Understanding Web Search Patterns Through Exploratory Search as a Knowledge-Intensive Process. Master's thesis. Federal University of the State of Rio de Janeiro - UNIRIO.

Peter Gerjets, Yvonne Kammerer, and Benita Werner. 2011. Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent thinking-aloud protocols and eye-tracking data. Learning and Instruction 21, 2 (2011), 220–231. https://doi.org/10.1016/j.learninstruc.2010.02.005 Special Section I: Solving information-based problems: Evaluating sources and information Special Section II: Stretching the limits in help-seeking research: Theoretical, methodological, and technological advances.

Denise Gider and Ulrich Hamm. 2019. How do consumers search for and process corporate social responsibility information on food companies’ websites?The international food and agribusiness management review 22, 2(2019), 229–246.

Susan R. Goldman, Jason L.G. Braasch, Jennifer Wiley, Arthur C. Graesser, and Kamila Brodowinska. 2012. Comprehending and Learning From Internet Sources: Processing Patterns of Better and Poorer Learners. Reading Research Quarterly 47, 4 (2012), 356–381. https://doi.org/10.1002/RRQ.027 arXiv:https://ila.onlinelibrary.wiley.com/doi/pdf/10.1002/RRQ.027

Morten Hertzum. 2020. Usability Testing: A Practitioner's Guide to Evaluating the User Experience. Synthesis Lectures on Human-Centered Informatics 1, 1 (March 2020), i–105. https://doi.org/10.2200/S00987ED1V01Y202001HCI045

Barbara K. Hofer. 2004. Epistemological Understanding as a Metacognitive Process: Thinking Aloud During Online Searching. Educational Psychologist 39, 1 (2004), 43–55. https://doi.org/10.1207/s15326985ep39015

Slava Kalyuga. 2009. Knowledge elaboration: A cognitive load perspective. Learning and Instruction 19, 5 (2009), 402–410. https://doi.org/10.1016/j.learninstruc.2009.02.003 Cognitive load in interactive knowledge construction.

Todd Kelley, Brenda Capobianco, and Kevin Kaluf. 2015. Concurrent think-aloud protocols to assess elementary design students.International Journal of Technology & Design Education 25, 4 (2015), 521 – 540.

Christoph Kofler, Martha Larson, and Alan Hanjalic. 2016. User Intent in Multimedia Search: A Survey of the State of the Art and Future Challenges. ACM Comput. Surv. 49, 2, Article 36 (Aug. 2016), 37 pages. https://doi.org/10.1145/2954930

Martin Labaj and Mária Bieliková. 2015. Conducting a Web Browsing Behaviour Study – An Educational Scenario. In SOFSEM 2015: Theory and Practice of Computer Science(Lecture Notes in Computer Science), Giuseppe F. Italiano, Tiziana Margaria-Steffen, Jaroslav Pokorný, Jean-Jacques Quisquater, and Roger Wattenhofer (Eds.). Springer, Berlin, Heidelberg, 531–542. https://doi.org/10.1007/978-3-662-46078-844

George P. Landow. 1991. Hypertext: The Convergence of Contemporary Critical Theory and Technology. Johns Hopkins University Press, Baltimore.

George P. Landow. 2005. Hypertext 3.0: Critical Theory and New Media in an Era of Globalization (third edition ed.). Johns Hopkins University Press, Baltimore.

Clayton Lewis and Robert Mack. 1982. Learning to Use a Text Processing System: Evidence from “Thinking Aloud” Protocols. In Proceedings of the 1982 Conference on Human Factors in Computing Systems (Gaithersburg, Maryland, USA) (CHI ’82). Association for Computing Machinery, New York, NY, USA, 387–392. https://doi.org/10.1145/800049.801817

Yinghui Li and Hengbin Yan. 2018. On the Reactivity of Think-Aloud as a Data Collection Tool and the Potential Modulating Effect of Think-Aloud Instruction on the Reactivity of this Tool. International Journal of Social Sciences and English Literature 2 (Dec. 2018), 27–32. https://doi.org/10.20448/journal.527.2018.21.27.32

Chao Liu, Ryen W. White, and Susan Dumais. 2010. Understanding Web Browsing Behaviors through Weibull Analysis of Dwell Time. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Geneva, Switzerland) (SIGIR ’10). Association for Computing Machinery, New York, NY, USA, 379–386. https://doi.org/10.1145/1835449.1835513

Wendy Macias, Mina Lee, and Nicole Cunningham. 2018. Inside the Mind of the Online Health Information Searcher using Think-Aloud Protocol. Health communication 33, 12 (2018), 1482–1493.

Ihadjadene Madjid, Chaudiron Stéphane, and Martins Daniel. 2003. The effect of individual differences on searching the web. Proceedings of the American Society for Information Science and Technology 40, 1 (2003), 240–246. https://doi.org/10.1002/meet.1450400130

Massimo Magni, Chiara Paolino, Rossella Cappetta, and Luigi Proserpio. 2013. Diving Too Deep: How Cognitive Absorption and Group Learning Behavior Affect Individual Learning. Academy of Management Learning & Education 12, 1 (2013), 51–69. https://doi.org/10.5465/amle.2011.0096 arXiv:https://doi.org/10.5465/amle.2011.0096

Gary Marchionini. 2019. Search, sense making and learning: closing gaps. Information and Learning Sciences 120, 1/2 (Jan. 2019), 74–86. https://doi.org/10.1108/ILS-06-2018-0049

Keith A. Markus. 2008. Constructs, Concepts and the Worlds of Possibility: Connecting the Measurement, Manipulation, and Meaning of Variables. Measurement: Interdisciplinary Research and Perspectives 6, 1-2(2008), 54–77. https://doi.org/10.1080/15366360802035513 arXiv:https://doi.org/10.1080/15366360802035513

Mercy Mpinganjira. 2019. Cognitive absorption and behavioural intentions in virtual health communities: A focus on content posters. Journal of Systems and Information Technology 21, 1 (Jan. 2019), 122–145. https://doi.org/10.1108/JSIT-06-2017-0044

E.A. Murray. 2001. Recognition Memory (in Primates), Neural Basis of. In International Encyclopedia of the Social & Behavioral Sciences, Neil J. Smelserand Paul B. Baltes (Eds.). Pergamon, Oxford, 12829–12832. https://doi.org/10.1016/B0-08-043076-7/03511-7

Jakob Nielsen. 1994. Usability Engineering. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

Jakob Nielsen and Thomas K. Landauer. 1993. A Mathematical Model of the Finding of Usability Problems. In Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems (Amsterdam, The Netherlands) (CHI ’93). Association for Computing Machinery, New York, NY, USA, 206–213. https://doi.org/10.1145/169059.169166

Graham Nuthall. 2000. The Role of Memory in the Acquisition and Retention of Knowledge in Science and Social Studies Units. Cognition and Instruction 18, 1 (2000), 83–139. https://doi.org/10.1207/S1532690XCI180104

Athanasios Papoulis. 1984. Bernoulli trials. Probability, random variables, and stochastic processes (1984), 57–63.

Quoc Trung Pham and Minh Chau Huynh. 2018. Learning achievement and knowledge transfer: the impact factor of e-learning system at Bach Khoa University, Vietnam. International Journal of Innovation 6, 3 (June 2018), 194–206. https://doi.org/10.5585/iji.v6i3.235

Willard Van Orman Quine. 1981. Mathematical Logic. Harvard University Press, Cambridge, MA, USA.

Ilknur Reisoğlu, Ayça Çebi, and Tuğba Bahçekapılı. 2019. Online information searching behaviours: examining the impact of task complexity, information searching experience, and cognitive style. Interactive Learning Environments 0, 0 (2019), 1–18. https://doi.org/10.1080/10494820.2019.1662456

D.H. Schunk. 2012. Learning Theories: An Educational Perspective. Pearson.

GAF Seber and SO Nyangoma. 2000. Residuals for multinomial models. Biometrika 87, 1 (03 2000), 183–191. https://doi.org/10.1093/biomet/87.1.183

Lijiang Shen and Su yeun Seung. 2018. On Measures of Message Elaboration in Narrative Communication. Communication Quarterly 66, 1 (2018), 79–95. https://doi.org/10.1080/01463373.2017.1334682 arXiv:https://doi.org/10.1080/01463373.2017.1334682

Dubravka Svetina, Joanna S. Gorin, and Kikumi K. Tatsuoka. 2011. Defining and Comparing the Reading Comprehension Construct: A Cognitive-Psychometric Modeling Approach. International Journal of Testing 11, 1 (2011), 1–23. https://doi.org/10.1080/15305058.2010.518261 arXiv:https://doi.org/10.1080/15305058.2010.518261

Meng-Jung Tsai and Chin-Chung Tsai. 2003. Information searching strategies in web-based science learning: the role of internet self-efficacy. Innovations in Education and Teaching International 40, 1(2003), 43–50. https://doi.org/10.1080/1355800032000038822

Maaike J. van den Haak, Menno D.T. de Jong, and Peter Jan Schellens. 2003. Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behaviour & Information Technology 22, 5 (2003), 339–351.

Herre Van Oostendorp and Sjaak De Mul. 1999. Learning by exploration: Thinking aloud while exploring an information system. Instructional Science 27, 3-4 (July 1999), 269–284. https://doi.org/10.1007/BF00897322

Frank J. van Rijnsoever. 2017. (I Can't Get No) Saturation: A simulation and guidelines for sample sizes in qualitative research. PLOS ONE 12, 7 (07 2017), 1–17. https://doi.org/10.1371/journal.pone.0181689

Robert A. Virzi. 1992. Refining the Test Phase of Usability Evaluation: How Many Subjects is Enough?Hum. Factors 34, 4 (Aug. 1992), 457–468. https://doi.org/10.1177/001872089203400407

Lev Semenovich Vygotsky. 2012. Thought and language. MIT press.

Fan Yang and Fuyuan Shen. 2019. Involvement without Knowledge Gain: A Meta-Analysis of the Cognitive Effects of Website Interactivity. Journal of Broadcasting & Electronic Media 63, 2 (2019), 211–230. https://doi.org/10.1080/08838151.2019.1622341 arXiv:https://doi.org/10.1080/08838151.2019.1622341

APPENDIX 1 - THE DATA SATURATION PROCESS

This method used progressive judgments about data saturation capable of helping us decide whether or not to add more participants. Probability theory indicates that the proportion of information one expects to find at a given sample size is given by the formula:

begin{equation} 1-(1-p)^n end{equation}

(1)

where p is the probability of detecting a given information and n is the sample size [ 51].

Given that a maximum of 35 indicators were depicted by the chosen online information searching strategy framework [42], we were not interested in the probability of detecting a given indicator, but the proportion of indicators found. Curves relating the proportion of information detected to the number of users in an evaluation are approximated by the formula [18, p.35]:

begin{equation} D = 1-(1-phi)^n end{equation}

(2)

where D is the discovered sample, ϕ is the average proportion of information found by a user, and n is the sample size. The presence or absence of an indicator in an information source (i.e., a user) can be seen as a random Bernoulli trial [ 50]. Thus, our study is a random experiment with exactly two possible outcomes (e.g., presence and absence; success and failure), in which the probability of an indicator being present or not is the same every time the study is carried out [ 39]. The number of indicators in the population is denoted by k. The vector of the probabilities of each indicator being present in the population is given by Φk, which has length k:

begin{equation} Phi k = (phi 1, phi 2,..., phi k) end{equation}

(3)

To estimate the ϕ it is necessary to calculate the average proportion. However, we mentioned that the total number of possible indicators is known (35 indicators). For this reason, we have to consider the probability of each indicator being present to define the mean probability of the observing information. In addition, we assumed that two indicators are impossible to be mapped due to the participants’ profile and equipment used. They are the indicators D3 (Not having any idea about what to do when doing an Internet search) and C5 (Using the “home” button to return to the beginning of the search). D3 because all participants have experience with exploratory searches, so they know how to conduct an online search. C5 because the “home” button is either disabled in the default setting or unavailable in most popular browsers (i.e., Chrome, Microsoft Edge, and Opera). Hence, we consider the number of indicators presented in our population as 33 (i.e., k=33). It changes the formula to calculate the discovery sample D because we have to explore the relationship between n, k, and the vector Φk analytically under the conditions of random chance. As a consequence, we considered the following formula to monitor our data saturation process [50]:

begin{equation} D = (1-(1-Phi k)^n)^k end{equation}

(4)

Table 6 shows the number of indicators identified per user and the monitoring process with the Φk refinement and the computing of D.

Table 6: Data Saturation monitoring results.

Number of Users (n)	Indicators Found	Accumulated Indicators	Estimated Φk	D
1	13	13	0.3939393939	0
2	9	22	0.3333333333	0.000000003767101584
3	1	23	0.2323232323	0.000000002338568347
4	0	23	0.2323232323	0.0000007680589693
5	4	27	0.2045454545	0.00000319798258
6	0	27	0.2045454545	0.00006504061798
7	0	27	0.2045454545	0.000595389971
8	0	27	0.2045454545	0.003134467001
9	0	27	0.2045454545	0.01109490882
10	0	27	0.2045454545	0.02932652978
11	0	27	0.2045454545	0.06228625707
12	0	27	0.2045454545	0.1120421067
13	0	27	0.2045454545	0.1774282339
14	0	27	0.2045454545	0.2546045156
15	0	27	0.2045454545	0.3383869586
16	4	31	0.1878787879	0.3002293493
17	0	31	0.1878787879	0.377653205
18	0	31	0.1878787879	0.4544711275
19	0	31	0.1878787879	0.5278142668
20	0	31	0.1878787879	0.5957069417
21	0	31	0.1878787879	0.6570063084
22	0	31	0.1878787879	0.7112495139
23	0	31	0.1878787879	0.7584740518
24	0	31	0.1878787879	0.7990504151
25	0	31	0.1878787879	0.8335449488
26	0	31	0.1878787879	0.8626171793

APPENDIX 2 - KNOWLEDGE GAIN CONCEPTS’ PRESENCE/ABSENCE EVALUATION

Table 7: Evaluation of the knowledge gain concepts’ presence/absence for each indicator.

Indicator	Cognitive Absorption	Comprehension	Elaboration	Memory
C1	0	0	0	0
C2	0	0	0	0
C3	0	0	0	0
C4	0	0	0	0
C6	0	0	0	0
C7	0	0	0	0
C8	0	0	0	0
C9	0	0	0	0
C10	0	0	0	0
D2	0	0	0	0
D4	0	0	0	0
TE1	0	0	0	0
TE2	0	0	0	0
TE3	0	0	0	0
PS1	0	0	0	0
PS2	1	1	0	0
PT1	0	1	0	0
PT3	1	1	0	1
PT4	1	0	0	0
SMI1	0	1	0	1
SMI2	0	1	0	0
SMI3	0	1	0	0
SMI4	0	1	0	0
SMI5	0	1	0	0
SMI6	1	1	0	0
SMI7	1	1	0	1
SMI8	1	1	0	0
E1	1	1	1	1
E2	1	1	1	1
E3	1	1	0	1
E4	1	1	1	1

APPENDIX 3 - EVALUATION OF VERBALIZATION/NON-VERBALIZATION ON THE SEARCHING PROCESS

The chi-square test is calculated as follows:

begin{equation} chi ^2=sum frac{(o - e)^2}{e} end{equation}

(5)

where o is the observed value, and e is the expected value. The expected value is calculated as:

begin{equation} e=frac{row.sum times column.sum }{grand.total} end{equation}

(6)

The degrees of freedom (df), which is calculated by the following equation:

begin{equation} df=(r-1) times (c-1) end{equation}

(7)

where r is the number of rows and c is the number of columns.

The Pearson residuals (PeaRes) for each cell is calculated as follows:

begin{equation} PeaRes=frac{o-e}{sqrt {e}} end{equation}

(8)

FOOTNOTE

1It means that the readers are allowed to make alterations and contributions to the text.

2UNIRIO, UFRJ, UERJ, UFJF, and USP.

3Despite the term “domain”, their role is more axis-like.

4We will not describe the mapped query state because it does not influence the statistical significance level derived from the relationship between verbal and non-verbal search periods.

5A set of ordered pairs.

6A feature taken from the concept of reactivity.

7We describe in Appendix 1 the reasons why we decided to left out two from the 35 indicators.

8Larger values usually indicate divergence.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

HT '22, June 28–July 01, 2022, Barcelona, Spain

ACM ISBN 978-1-4503-9233-4/22/06…$15.00.

DOI: https://doi.org/10.1145/3511095.3531272