Powerful predictive AI systems have demonstrated great potential in augmenting human decision making. Recent empirical work has argued that the vision for optimal human-AI collaboration requires ‘ appropriate reliance ’ on AI systems. However, accurately estimating the trustworthiness of AI advice at the instance level is quite challenging, especially in the absence of performance feedback pertaining to the AI system. In practice, the performance disparity of machine learning models on out-of-distribution data makes the dataset-specific performance feedback unreliable in human-AI collaboration. Inspired by existing literature on critical thinking and mindsets, we propose debugging an AI system as an intervention to foster appropriate reliance. This paper explores whether a critical evaluation of AI performance within a debugging setting can better calibrate users’ assessment of an AI system. Through a quantitative empirical study ( N = 234), we found that our proposed debugging intervention does not work as expected in facilitating appropriate reliance. Instead, we observe a decrease in reliance on the AI system — potentially resulting from an early exposure to the AI system's weakness. Our findings have important implications for designing effective interventions to facilitate appropriate reliance and better human-AI collaboration.

With the rise of deep learning systems over the last decade, there has been widespread adoption of AI systems in supporting human decision makers [33], albeit without always fully understanding the societal impact or downstream consequences of relying on such systems [13, 14]. Due to the opaqueness of some AI systems, users (especially laypeople) have struggled to determine when exactly they are trustworthy. To realize the full potential of complementary team performance [3], human decision makers need to identify when they should rely on AI systems and when they are better off relying on themselves. Such a reliance pattern has been defined as appropriate reliance [26, 33, 43, 51].

In practice, it is common that users need to deal with data from unknown distributions and unseen contexts, meaning that AI systems in the real world need to provide advice on out-of-distribution data [9, 42]. Under such circumstances, the estimated performance of an AI system or the so-called ‘stated accuracy’ [25, 49, 61] of the system (i.e., accuracy on pre-defined test sets) cannot faithfully reflect the trustworthiness of the AI system. To help users assess the trustworthiness of AI systems, a practical solution that has been proposed, is to provide meaningful explanations along with AI advice [50, 56]. Post-hoc explanations have been found to improve user understanding of AI advice in empirical studies exploring human-AI decision making [33, 58]. However, such explanations require relatively high AI literacy [8] and domain expertise [23] to verify. As a result, most existing XAI methods have remained ineffective in helping laypeople assess the trustworthiness of AI advice at the instance level, adversely affecting their degree of appropriate reliance (often inducing over-reliance) on AI systems [7, 58].

To realize the goal of appropriate reliance, human decision makers need to be capable of evaluating AI advice and the trustworthiness of the AI system critically. We argue that such a critical mindset can help users avoid blindly following AI advice (i.e., avoiding over-reliance), and also prevent them from distrusting AI advice when it can be productive (i.e., avoiding under-reliance). Inspired by recent works on explanation-based human debugging of AI systems [1, 37], we propose explanation-based debugging as a training intervention to increase appropriate reliance on AI systems. We posit that such a debugging intervention has the potential to help laypeople understand the limitations of AI systems — that neither explanations of the AI advice nor the advice itself are always reliable. Recognizing these limitations can help users better understand when an AI system is trustworthy and thereby increase appropriate reliance on the system. In this paper, we aim to empirically evaluate the effectiveness of using a debugging intervention as a means to increase appropriate reliance. We address the following research questions — (RQ1) How can a debugging intervention help users to estimate the performance of an AI system, both at the instance and at the global level? and (RQ2) How does a debugging intervention affect the reliance of users on an AI system?

To this end, we propose three hypotheses considering the effect of the debugging intervention on AI performance assessment as well as reliance, and the task ordering effect of debugging intervention on appropriate reliance. We tested these hypotheses in an empirical study (N = 234) of human-AI decision making in a deceptive review detection task (i.e., identifying whether a review excerpt is written based on real experience). However, we found that the proposed debugging intervention neither calibrates user estimation of AI performance nor promotes appropriate reliance.

Our results highlight that when presented with the weakness of the AI system in an early stage of the debugging intervention, users underestimate AI performance and rely less on the AI system. Users’ overestimation of their own competence may further amplify such an effect. Through an analysis exploring relatively less-competent individuals, we found that the underestimation of AI trustworthiness may also play a role in shaping under-reliance, which is potentially relevant to the metacognitive bias called the Dunning-Kruger effect [31]. Our work has important implications for designing effective interventions to promote appropriate reliance in the context of human-AI decision making.

We position our work in the context of (a) studies on human-AI decision making, (b) appropriate reliance on AI systems, and (c) explanation-based debugging of machine learning systems.

Human-AI Decision making. In recent years, deep learning methods have been used in a wide range of applications (like medical image analysis [40], autonomous driving [21]). However, due to the intrinsic uncertainty and opaqueness, it would be undesirable to make such AI systems automate decision making, especially in high-stakes scenarios (e.g., legal judgment, medical diagnosis). Under such circumstances, AI systems are expected to play a supporting role for human decision makers. According to GDPR, users have the right to obtain meaningful explanations to work with such AI systems [53]. Motivated by this, a series of work has proposed constructing human-centered explainable AI systems [11, 12, 24, 39] for better human-AI collaboration. Existing work has widely explored how different user factors (e.g., expertise [10, 45], risk perception [20], machine learning literacy [8]) and interaction designs (e.g., performance feedback [2, 49, 60], explanation [58], user tutorial [34, 44]) will affect user trust in and reliance on AI systems.

Appropriate Reliance on AI Systems. One important goal of human-AI decision making is complementary team performance [3, 42], which requires appropriate reliance [36]. In practice, however, humans always misuse (i.e., over-reliance [48]) or disuse (i.e., under-reliance [5, 58, 59]) AI systems. Such inappropriate reliance results in sub-optimal team performance, which is even worse than AI alone [3, 42]. To mitigate such issues, existing work has proposed different interventions including user tutorials [8, 34], cognitive force functions [4], and improving AI literacy [9]. Another stream of work proposed to improve the transparency of AI systems with effective explanations [35, 58], performance feedback [43], and global model properties [6]. In summary, these works presented users with extra information about AI systems (more than advice) or changed users’ mindset and knowledge of AI systems.

Explanation-based Debugging. Explanation-based debugging was found to be helpful for improving human understanding of machine learning system [32]. Recent works [1, 37] have explored how to leverage explanations for model debugging across a wide range of tasks. The core idea of explanation-based debugging is to check whether the explanations from AI systems misalign with human knowledge. From human feedback, it would be possible to improve machine learning models’ robustness, e.g., by reducing spurious reasoning patterns [41, 54] and bias in dataset [27]. Debugging in programming is the process by which programmers can determine the potential errors in the source code and resolve these errors [57]. Inspired by this, we proposed debugging as an intervention to help participants understand the limitations of both explanations and advice of AI systems. In such an error finding and resolution process, users may learn when the AI system is trustworthy as a result of engaging directly with their pitfalls.

3.1 Deceptive Review Detection Task

In this paper, we base our experiment within a challenging task – deceptive review detection. In each task, based on a hotel review, participants are asked to identify whether it is genuine (i.e., written by real customers) or deceptive (i.e., written by people who did not stay at the hotel). An example of this task is shown in Figure 1(a) . This task has been used in prior work exploring Human-AI decision making [34, 35]. We also used the same public dataset [35]. 1

Figure 1

Figure 1: Screenshots of deceptive review detection task in our study.

Using Text Highlights as Explanations. In our study, we consider a real-world scenario where the performance of an AI system is not provided or available. To help participants assess the trustworthiness of advice from the AI system in each instance of decision making, we provide local explanations for each prediction. Following Lai et al.  [34], we adopted BERT-LIME to generate text highlights as local explanations for each AI advice. We first finetuned the BERT [29] (bert-base-uncased) on the deceptive review detection dataset, and then generated the top-10 highlighted features from post-hoc XAI method LIME [50] as explanations.

Selection of Tasks. To measure the effect of the debugging intervention in our study, two batches of tasks with compatible difficulty levels are required. For that purpose, we conducted a pilot study on human performance over 20 tasks randomly sampled from evaluation and test set of the deceptive review detection dataset. We divided the trial cases into two sets of 10 tasks with equal human performance in a pilot study (10 participants). In each task batch, the AI system achieved $80%$ accuracy.

Two-stage Decision Making. Following existing empirical study design of human-AI decision making [18, 19, 26], all participants in our study work on each trial case with two stages of decision making. In the first stage, only task input is provided; participants make an initial decision by themselves. After that, the same task input along with a local explanation and AI advice are provided. Participants make their final decision based on all information.

3.2 Hypotheses

3.3 Debugging Intervention

Experimental Conditions. In all conditions, the top-10 most important features obtained from BERT-LIME are highlighted as an explanation for AI advice to help participants identify the trustworthiness of AI advice. The differences between conditions are whether debugging intervention is adopted and the order of debugging tasks. To comprehensively study the effect of debugging intervention, we considered four experimental conditions in our study: (1) no debugging intervention (Control), (2) with debugging intervention, debugging tasks in random order (Debugging-R), (3) with debugging intervention, debugging tasks in decreasing impression order (Debugging-D), (4) with debugging intervention, debugging tasks in increasing impression order (Debugging-I).

Measures And Variables. To verify H1, we assessed participants’ global estimation of AI system's performance with two questions: “From the previous 10 tasks, on how many tasks do you estimate the AI advice to be correct?” and “From the previous 10 tasks, how many questions do you estimate to have been answered correctly? (after receiving AI advice)”. The answers to the two questions correspond to participants’ estimation of AI performance and team performance respectively. We can refer to the estimated trustworthiness as estimated AI performance (EAP) and estimated team performance (ETP). Comparing that performance estimation with actual performance in abstract difference, we can calculate the degree of miscalibration of AI performance (MAP) and team performance (MTP). For the AI performance estimation at instance level, we calculated the number of tasks they made the correct final decision with an indication of “Very Confident” (CCD).

To verify H2 and H3, we measured both reliance and appropriate reliance of participants on the AI system. Details of calculating these measures can be found in [26]. The reliance is measured with two widely adopted metrics: the Agreement Fraction and the Switch Fraction. These look at the degree to which participants are in agreement with AI advice, and how often they adopt AI advice in cases of initial disagreement. As for the appropriate reliance, we followed Max et al.  [51] to calculate the appropriate reliance with relative positive AI reliance (RAIR) and relative positive self-reliance (RSR). The two measures are calculated when an initial disagreement between the human initial decision and AI advice exists where the correct answer occurs in one of them. In addition, we consider the accuracy in batches to measure participants’ performance with AI assistance.

For a deeper analysis of our results, a number of additional measures were considered based on observations from existing literature [38, 52, 55]: Trust in Automation (TiA) questionnaire [30], a validated instrument to measure (subjective) trust [55] consisting of 6 subscales. Affinity for Technology Interaction Scale (ATI) [16], administered in the pre-task questionnaire. Thus, we account for the effect of participants’ affinity with technology on their reliance on systems  [55]. NASA-TLX questionnaire [22] for the working load assessment of the debugging intervention.

Participants. Following previous work [25, 26], we computed the required sample size in a power analysis for a Between-Subjects ANOVA using G*Power [15]. This resulted in a required sample size of 230 participants. We thereby recruited 324 participants from the crowdsourcing platform Prolific, in order to accommodate potential exclusion. All participants were rewarded with £3.8, amounting to an hourly wage of £7.6 (estimated completion time was 30 minutes). We rewarded participants with extra bonuses of £0.05 for every correct decision in the 20 trial cases. Such an extra bonus for correct decisions provides an additional monetary incentive for crowd workers to try their best on each task, a widely adopted design in existing work to increase participant reliability [9, 34].

Filter Criteria. All participants were proficient English speakers above the age of 18. For a high-quality study, we require participants to have an approval rate of at least 90% and more than 80 successful submissions on the Prolific platform. After reading the basic introduction and guidelines about the deceptive review detection task, participants who failed any qualification test (about understanding the task) were removed from our study. After data collection, we excluded participants from our analysis if they failed any attention check (90 participants). The resulting sample of 234 participants had an average age of 39 (SD = 13) and a gender distribution ($48.7%$ female, $49.6%$ male, $ 1.7%$ other).

Procedure. The complete procedure of our study is illustrated in Figure 2. All participants are first presented with an introduction of the deceptive review detection task. According to Lai et al.  [34], guidelines about identifying deceptive reviews are highly useful in improving user performance on this task. Thus, we also follow them to provide the guidelines in the introduction. Then, participants will be checked with two qualification questions to ensure they carefully read the instruction and understand this task. Any failure at the qualification test will result in removal from our study. All reserved participants will then be asked to answer a pre-task questionnaire.

Figure 2: Illustration of the procedure that participants followed within our study.

For all conditions, participants will work on the first batch of tasks and go through a post-task questionnaire to assess AI performance and subjective trust in AI system (i.e., with TiA subscales). The main difference between conditions (shown in the dashed box of Figure 2) is the 8 tasks presented after the post-task questionnaire. In condition Control, participants will work on the 8 tasks as normal trial cases. No debugging intervention and result feedback will be provided. In conditions with debugging intervention, the participants will go through the debugging tasks with different task orders and be asked about the task working load resulting from the debugging intervention, using the NASA-TLX [22] questionnaire. Then, participants in all conditions will continue to work on another batch of tasks and answer the same post-task questionnaire as the one after the first task batch. The AI accuracy on each task batch is $80%$. To eliminate the potential ordering effect of trial cases, we randomly assigned one batch of selected tasks (see section 3.1) as the first batch and shuffled the task order within each batch.

In our analysis, we only consider participants who passed all attention checks as a control of participant reliability [17]. Participants were distributed in a balanced fashion over experimental conditions: 57 (Control), 59 (Debugging-R), 60 (Debugging-D), 58 (Debugging-I). On average, participants spend around 51 minutes. Due to the page limit, more results can be found in the Appendix.

Performance Overview. On average across all conditions, participants achieved an accuracy of 0.64 over the two batches of tasks, still lower than the aforementioned AI accuracy of 0.8. The agreement fraction is 0.66 while the switching fraction is 0.31. With these measures, we confirm that participants in our study did not blindly rely on the AI system when disagreement appeared. In the two batches of tasks, the average estimated AI performance is 5.81 and 5.79 respectively; the average estimated team performance is 6.64 and 6.44 respectively. Overall, participants underestimated the performance of the AI system and believed they could outperform the AI system on this task after receiving AI advice.

5.1 Hypothesis Tests

5.2 Exploratory Analyses

Key Findings. To promote appropriate reliance on AI systems by calibrating user estimation of AI performance, we proposed a debugging intervention to educate participants that AI systems are not always reliable and that the explanations may not always be informative. As opposed to our hypotheses backed by existing work, we found that such a debugging intervention fails to calibrate participants’ estimation of AI performance at both the global and local levels. Participants tended to rely less on the AI system after receiving the debugging intervention. Through an exploratory analysis based on performance quartiles, we found that participants who performed worse in our study tended to underestimate AI performance. Thus, their suboptimal team performance can be largely explained by the under-reliance on the AI system. The plausibility of the XAI intervention [28] can alternatively explain our findings. The debugging intervention may have potentially made the XAI (i.e., text highlights in our study) seem less plausible to users, resulting in a tendency to underestimate AI performance.

In our study, no significant difference was found between the different ordering of debugging tasks across experimental conditions. However, participants who were exposed to the weakness of the AI system at the beginning of the debugging intervention showed a more obvious tendency to disuse the AI system. Such under-reliance was found to result in sub-optimal team performance. This finding is in line with recent work that has uncovered similar ordering effects and cognitive biases influencing outcomes in human interaction with intelligent systems [46, 55]: a bad first impression of an AI system can lead to an underestimation of AI competence and reduced reliance on the system.

Implications. Our findings suggest that the debugging intervention and similar interventions with training purposes (e.g., user tutorials) may suffer from the cognitive bias brought by the ordering effect within such interventions. While using such interventions to demonstrate the strength and weakness of AI systems, we should be careful not to leave users with a bad first impression, highlighting the weakness of the AI system. In our study, participants tended to be optimistic about the team performance while underestimating the AI performance. This is possibly caused by the meta-cognitive bias of the Dunning-Kruger effect [26, 31]. In our study, we found that less-competent individuals showed a greater tendency to underestimate the AI performance and make fewer correct decisions with confidence (see Table 2). This indicates that underestimating AI systems can also contribute to under-reliance in the context of human-AI decision making. According to He et al.  [26], an overestimation of self-competence can result in under-reliance on the AI system. Both the overestimation of self-competence and the underestimation of AI competence can contribute to an illusion of superior competence over the AI system. As a result, users with such an illusion tend to disuse the AI system.

In an age where laypeople have democratized access to various AI-based decision support systems on the web, promoting appropriate reliance on AI systems in decision-making contexts is an important goal. In this paper, we present an empirical study to understand the impact of a debugging intervention on the estimation of AI performance and user reliance on the AI system. Our results suggest that we should be careful in presenting the weakness of the AI system to users to avoid anchoring effects which may result in under-reliance. While our experimental results do not provide support to our original hypotheses informed by prior work, more work is required to understand the potential of debugging interventions in facilitating appropriate reliance on AI systems. Future work may explore how to mitigate potential bias brought by the users’ overestimation of themselves and the underestimation of AI performance.