Data quality is a foundational research construct [11], including personas derived predominantly from surveys. If survey data is unverifiable, the resulting personas are also unverifiable, undermining design and decision-making processes [15] as personas themselves are difficult to verify. Prior work has demonstrated that inconsistencies in survey responses can hinder decision making efficiency [5]. It is documented that fraudulent inputs, including careless responding, falsified answers, and bot-generated data, are significant issues in survey research [28]. Although survey design strategies have been proposed [25], alongside algorithmic techniques aimed at detecting or mitigating low-quality data [22], none resolve the core methodological challenge: invalid survey data often cannot be reliably distinguished from valid data using internal criteria alone. These concerns are compounded by the growing presence of AI-generated survey responses [3] that can appear coherent and plausible, yet be substantively meaningless. This difficulty is exacerbated by LLMs, which can generate substantial volumes of responses that mimic valid human input while containing factual inaccuracies [24]. Accordingly, our research proceeds from the premise that, even with meticulous cleaning and preprocessing, internal survey-based validation is insufficient for establishing the validity of large-scale survey data or of the personas derived from it, positioning persona creation as a revealing exemplar of this broader limitation.

Although personas are frequently constructed from survey data, the problem of invalid survey responses has received surprisingly little direct attention in HCI persona research. Prior work addresses data quality concerns in principle but offers no practical guidance on how to determine whether data is low-quality. McGinn and Kotamraju [31] cautioned that poor data quality undermines persona accuracy and user experience (UX) yet offered no empirical analysis of how invalid data affects persona construction. Similarly, Matthews et al. [29] emphasized the risk of misleading personas, but focused on how practitioners perceive and use personas rather than on how invalid survey data propagates into persona artifacts. As a result, systematic empirical studies examining the impact of survey data on persona generation are rare, if not absent. The dominant implication of the “garbage in, garbage out” (GIGO) principle [12] is that low-quality input will visibly degrade outputs, discouraging closer scrutiny of cases in which artifacts remain plausible despite ‘garbage’ data. In contrast, our study investigates whether it is even possible to determine the validity of survey data for persona creation.

Advances in ML, AI, and data analytics (DA) are increasingly used to generate personas. Nevertheless, they do not resolve the fundamental problem of determining whether the underlying survey data is valid. Prior work has focused primarily on evaluating personas as artifacts rather than on validating the data from which they are derived. APG researchers typically focus on algorithms that enhance persona generation [56]. However, to our knowledge, these approaches have not examined how survey data quality itself influences persona outcomes or offered techniques for validating survey responses. Prior work in APG offers objectives for persona generation, including fairness, diversity, and consistency [41]. However, these objectives presuppose that the underlying survey data is valid, overlooking the possibility that personas may be grounded in unreliable or invalid responses. As a result, personas can appear reliable, yet they do not accurately represent the users they are intended to model. In our study, we examine how progressively higher levels of degraded survey data alter personas relative to supposedly valid data. As such, we demonstrate that personas built on invalidatable data appear trustworthy yet unverifiable, highlighting a critical challenge in HCI: user representation.

Related work proposes probabilistic approaches for assessing data source credibility [55], emphasizing trustworthiness through confidence, such as confidence modeling [4]. However, these approaches do not address the unresolved problem that, when survey data cannot be validated, neither personas nor other findings derived from it can be validated. This lack of empirical evidence on unverifiable survey data is the central focus of the present research, using persona generation as an example of the broader challenge of determining the validity of survey data in HCI. Chapman et al. [7] proposed a model for analyzing persona attributes, observing that persona descriptiveness diminishes as the number of attributes increases. However, even these critics of personas presume reliable input data, failing to address the effects of invalid survey responses. Similarly, Shenton et al. [16] advocate rigor in qualitative research, emphasizing credibility, transferability, dependability, and confirmability. However, these standards assume that the collected data is trustworthy and offer no mechanisms to verify validity. Salminen et al. [49] introduced evaluative dimensions, such as credibility, completeness, clarity, and consistency, for assessing personas. However, these metrics also operate at the level of the persona representation and remain agnostic to the quality of the survey data that generated them. These approaches reflect efforts to evaluate persona outputs, but they leave unresolved the central methodological challenge: when survey data quality is uncertain, personas still fail to accurately represent users.

Although prior work has acknowledged the importance of data quality, we are not aware of any study that directly examines the trustworthiness of survey data for persona creation or for other HCI applications aimed at understanding users. The validation of survey data quality is largely absent from prior HCI work, leaving a critical knowledge gap in understanding how data quality affects persona generation and other areas of research. Our study examines this challenge of distinguishing between valid and invalid survey data, highlighting a foundational risk in research and practices that rely on surveys.

The dataset for this study was from a large-scale survey designed to capture user behaviors, preferences, and demographic characteristics relevant to the creation of personas for social media users in the Middle East and North Africa (MENA). The survey collected over 20,000 responses from 16 MENA countries through an international, reputable survey provider, in accordance with their company policy governing user data. Following the survey company's internal quality control procedures and two embedded attention-check items, 8,140 responses were classified as ‘valid’. The remaining 11,860 responses were categorized as ‘invalid’ data, representing participants who failed attention checks [32]. We use this valid-invalid separation as the foundation for systematically examining the validity of survey data and the implications for persona generation.

The survey incorporated two commonly used attention check methods to identify inattentive responding. The first was an instructional manipulation check embedded in the questionnaire that required respondents to select a specific response option to demonstrate that they had read the question carefully (i.e., “To show that you are actually reading the statements, select ‘Disagree’”). The second attention check evaluated whether respondents followed simple instructions (i.e., “Select ‘Strongly Disagree’ to prove that you are actually reading the statements”). Both are standard methods employed in survey research. Online surveys frequently use these methods to detect invalid responses and represent standard internal survey quality controls [1]. To account for the possibility that earlier survey questions are less affected by attention, we split the demographic questions at the beginning (i.e., age) and end (i.e., gender, nationality) of the survey. Behavioral and preference items that were the primary basis for persona generation appeared throughout the survey. Demographic variables accounted for only a small subset of the 115 variables used in persona construction, with the majority derived from these behavioral and attitudinal items.

Our use of attention checks to separate the data does not imply that survey responses can be definitively classified as ‘valid’ or ‘invalid’ (our premise is that they do not). Instead, attention checks operationalize the dominant and widely accepted mechanism by which HCI and other survey researchers attempt to identify problematic responses (i.e., the invalid ones only). This framing of attention checks as an internal survey control check is intentional. If attention checks meaningfully distinguish between low-quality and high-quality survey data, then introducing responses that fail these attention checks should result in changes in downstream analyses, such as persona creation. However, if the commonly accepted internal validity survey control, relied upon by researchers, fails to produce detectable differences in survey data, this would raise concerns about its effectiveness as a validity check. The absence of change would challenge the practical effectiveness of these controls, raising concerns for artifact creation downstream.

To address this examination, this study analyzed data collected from a survey instrument comprising multiple items, supplemented with two attention-check questions. Several items were structured as matrix questions (e.g., “In the past week, approximately how much time per day have you spent on the following on average”), covering eleven major social media platforms such as Facebook, Twitter, YouTube, Snapchat, and Instagram. For analysis, each response within the matrix was treated as a distinct survey variable, aligning with the persona generation algorithm and standard survey research practices. This procedure produced 115 items used in the persona generation process. Before analysis, all responses, both ‘valid’ and ‘invalid’, based on passing or failing the attention checks, underwent the same pre-processing phase [8] involving standardization of categorical responses and Likert scale data. We make both datasets publicly available.1

We first constructed a baseline dataset of ‘valid’ survey responses, comprising 8,140 respondents who passed the attention checks. We designated it as the gold standard for the analysis. In survey-based studies, survey datasets that meet these criteria are assumed to be valid and suitable for downstream analytical tasks, including persona generation [57]. This gold-standard dataset serves as the reference condition for examining whether the accepted indicator of survey validity or invalidity yields discernible differences in persona outputs. We then construct ten additional datasets by introducing responses classified as supposedly ‘invalid’ into the baseline dataset in increments of 10%; each 10% increment replaced an equivalent proportion of reportedly valid responses. This process yielded eleven datasets in total: the gold standard dataset (GS0), which consisted exclusively of responses that met the validity criteria, and datasets D1-D10, which contained between 10% and 100% of reportedly invalid responses from respondents who failed the attention check (Figure 2). This controlled change of the survey data reflects empirically realistic conditions in which invalid responses are intermingled with ostensibly valid ones, remaining undetected at the time of analysis. Our experimental design provides a sensitivity analysis of whether and at what point supposed changes in survey data quality affect persona generation.

Figure 2: Process of building test datasets from the baseline dataset. From the 8,140 good responses (green) baseline, we progressively replaced valid responses with invalid responses in increments of 814 (red).

We generated personas from all eleven datasets using Survey2Persona (S2P) [23], a publicly available APG system. S2P is designed to transform survey responses into structured persona profiles. S2P supports multiple survey question types, including Likert-scale, binary, categorical, open-ended, and demographic data, which enables researchers and developers to construct personas directly from survey datasets. Via its online workflow, end users select one or more survey items as the cognitive lens for S2P to process using clustering, an established method in data-driven persona research [42], to identify patterns and produce representative persona archetypes.

S2P uses demographic variables (e.g., age, gender, nationality) along with a substantially larger set of behavioral and attitudinal survey items to generate personas, thereby contextualizing clusters derived primarily from behavioral patterns. The S2P process uses the full set of survey variables (115), reducing the likelihood that demographic variables will overly dominate persona creation. The number of personas (16 in this research) aligns with prior quantitative persona research, which recommends larger persona sets for heterogeneous populations [14]. Importantly, our experimental objective is to examine whether commonly used internal survey validity checks, specifically attention checks, produce detectable changes in downstream research artifacts. If these validity checks can distinguish low-quality from high-quality responses, then the proportion of responses that fail these checks would be expected to produce observable changes in persona outputs.

We used an identical persona-generation procedure with S2P across all datasets (GS0, D01–D10) to ensure that any differences in persona were due solely to changes in the survey data. By holding the persona generation process constant, the design isolates survey data as the experimental variable to examine whether changes produce detectable effects at the persona level. As S2P is publicly accessible and the data are released as supplementary material, the study is replicable, supporting transparency in research and enabling independent verification [44].

The persona generation workflow applied across all eleven datasets is illustrated in Figure 3(a). First, we formatted the survey responses as CSV files and loaded them into S2P for processing. We then mapped survey items to corresponding persona profile fields to preserve alignment between survey constructs and persona attributes. We performed data preprocessing, including data cleaning, handling missing values, and converting the data into compatible formats. We generate hierarchical clustering, a common technique in persona research [38]. We employed three input variables for persona construction: (a) demographic attributes of age, gender, and nationality, (b) behavioral variables describing platform usage patterns, and (c) attitudinal or preference variables derived from Likert-scale survey items. This approach aligns with methodologies of identifying patterns of user behavior and preferences [14], with the demographics humanizing the data. S2P then groups and summarizes the survey items, producing persona profiles with descriptive headlines, narratives, and standard persona attributes.

Figure 3: (a) The persona creation approach of Survey2Persona. The core algorithmic method is hierarchical clustering, the most common approach for creating quantitative personas. (b) A persona profile generated from S2P showing the three levels of analysis: (a) demographic variables, (b) the persona profile attributes, and (c) the persona attribute value diversity.

We generated sixteen personas for each dataset. Sixteen is consistent with recommendations for large-scale datasets, as a larger persona set improves representation of heterogeneous populations [46] and supports more inclusive persona-based design practices [14]. Sixteen personas provide greater granularity in representing heterogeneous user populations [46] and mitigate limitations associated with smaller persona sets [47]. Our use of a large persona set strengthens the study's ability to detect potential effects of survey data degradation. If changes in data quality meaningfully influenced persona outputs, such effects would be more likely to surface across a richer set of profiles. To address potential ordering effects, we observed no differences in persona generation. Nonetheless, we randomly shuffled responses across all datasets to ensure an even distribution across conditions. We executed each experimental condition three times on S2P, producing identical persona sets across iterations, confirming the stability of the algorithmic process itself.

We conducted multilevel comparative analysis across the persona sets from each dataset. First, we examined demographic overlap by comparing distributions of gender, age, and nationality across persona sets. Second, we analyzed attribute overlap by identifying persona attributes shared across sets [30]. Although some prior research suggests that a small number of core attributes typically drive most interpretation and design, we believe that analyzing all attributes preserves the breadth of persona attributes. Finally, we operationalized the actual diversity of survey values [18]. to enable an evaluation of whether changes in the survey data [20] increase variability in persona representations, as the survey values underpin the attributes in the persona profiles. Each of these levels of analysis is illustrated in Figure 3(b).

We employed the Overlap Coefficient (a.k.a., Szymkiewicz–Simpson coefficient) [54] to compare demographic attributes and persona profile items across persona sets, as it is well-suited for assessing whether changes in survey data quality produce detectable differences in persona outputs. The Overlap Coefficient (OC) quantifies similarity between two finite sets by relating the size of their intersection to the size of the smaller set. Formally, for two sets $A $ and $B$ , it is defined as: ${rm{Overlap}}( {A,B} ) = frac{{mid A cap Bmid }}{{{rm{min}}( {mid Amid ,mid Bmid } )}}.$

Because the denominator is the smaller of the two sets, if the sets are of different sizes, the coefficient ranges from 0 to 1, where 1 indicates complete containment of the smaller set within the larger or identical set, and 0 indicates no shared elements. The OC is appropriate for our study, as it allows comparisons across persona sets and provides a measure for detecting whether variations in survey data translate into differences among personas. As OC is a measure of similarity, 1-OC is a measure of difference. We then employed ANOVA for comparisons across sets. ANOVA is a conventional approach for comparing group means across multiple experimental conditions in HCI and persona research [13], and for comparisons involved a small, fixed number of conditions (GS0 versus D1–D10) with comparable sample sizes across sets, satisfying the test's structural requirements. Also, ANOVA is quite robust to non-normalized data [19].

As shown in Table 1, the introduction of invalid survey data to the supposedly valid data did not yield a systematic change in persona demographics. A reminder that the demographic survey questions were split: one at the beginning and two towards the end, to mitigate the risk of response fatigue affecting/not affecting demographics. Across six of the ten experimental comparisons, the persona demographics of the sixteen personas in each set remained unchanged relative to the gold standard set GS0. In the remaining four comparisons, only a single persona differed from the sixteen, and in each instance, the change was limited to an age category, with no variation observed in other demographic attributes. For the six comparisons exhibiting identical values, we could not perform an ANOVA without producing a degenerate p-value (i.e., the sets were identical). In the four remaining comparisons, one-way ANOVA revealed no statistically significant differences between the gold standard and the experimental datasets (F(2, 30) = 0, p = 1). Accordingly, H01 is not supported: increases in invalid survey data did not produce corresponding increases in demographic instability.

Table 1: Overlap results for demographics (similarity and difference) compared to the GS0 dataset. Datasets with identical overlap are shaded.

Table 2 presents the results of the analysis of persona profile item consistency. Unlike demographic attributes, the changes in the survey data did produce some changes in persona profile items across datasets. As shown in Table 2, differences emerged between persona sets as the proportion of invalid data increased. However, the magnitude of these shifts remained modest, suggesting that differences in the underlying datasets persisted. The resulting changes in persona profiles are limited, with about 1 attribute in 10 changed from the gold-standard baseline. GS0.

Table 2: Results for attribute (similarity and difference) compared to the GS0 dataset.

As shown in Table 2, we observed statistically significant differences between each experimental condition (D1–D10) and the baseline dataset (D1) when comparing the values of the persona profile items. Accordingly, H02 is supported: increases in the proportion of invalid survey data are associated with increased changes in persona profile attributes. However, readers should interpret this finding cautiously. Although the effect is statistically detectable, the changes are practically limited across all persona attributes. While persona profile values exhibit sensitivity to changes in survey data, the magnitudes do not provide a reliable basis for determining whether the underlying survey data is valid. Of the 115 possible persona attributes in a persona profile, about 14 items (12.2%) changed per persona set, with the change nearly constant across the ten datasets, regardless of the percentage of invalid survey data.

As shown in Table 3, changes in the survey data did not significantly affect the diversity of persona attribute values. Across datasets, changes in the range of attribute values were minimal: two datasets exhibited a single novel response, two datasets exhibited two novel responses, and six datasets exhibited three novel responses. These did not result in statistically significant differences when compared to the gold standard dataset, GS0, as confirmed by one-way ANOVA (F(1, 196) = 0, p > 0.99). Accordingly, H03 is not supported; survey value diversity does not increase as the proportion of invalid survey data increases.

Table 3: Results for survey value diversity (similarity and difference) compared to the GS0 dataset.

We also conducted an assessment to determine whether the results would be similar or different when using another persona-generation algorithm (though the analysis for RQ3 was independent of any persona creation algorithm). We selected the k-prototypes clustering algorithm, as it is a common approach in persona generation. For this check, we compared the GS0 gold standard dataset to D10, the dataset with all invalid data for the demographic overlap. We used all the survey questions and set the number of personas to 15. Comparing demographics, the OC was 0.40 (i.e., 6 of the 15 personas were the same), lower than with hierarchical clustering but still quite high. Also, the remaining personas typically varied by only one demographic attribute (generally age or nationality). This finding shows that, as one would expect, the algorithmic choice does influence the number of overlapping personas, but overlapping personas still exist, and, with at least these two algorithms, it is quite high.

Large samples are often used to capture the segments within heterogeneous user populations [26]. However, our findings show that as the survey sample size increases, the number of plausible response combinations of bounded-choice items (e.g., Likert and binary items) grows, increasing the likelihood that invalid responses become statistically indistinguishable from valid responses [10]. This study demonstrates that the validity of survey data cannot be reliably inferred from the survey data itself or from artifacts derived from it, such as personas. The findings challenge a straightforward application of the GIGO assumption. With survey data, relying solely on internal measures such as attention checks does not allow one to determine what is “garbage” and what is not. Based on our findings, large-scale surveys (i.e., surveys with many respondents and questionnaire items) can mask both valid and invalid responses, producing personas that appear stable yet are grounded in unreliable data—a paradox of data stability. Therefore, survey-based personas and related practices are placed into question, as using survey data may create internally coherent personas that still misrepresent users. Importantly, these results suggest that commonly used internal validity controls (e.g., attention checks, completion time thresholds) are insufficient, particularly for lengthy surveys administered at scale. Large survey samples make it practically impossible to determine valid from invalid responses.

Lengthy survey instruments may encourage satisficing or careless responding, while large sample sizes amplify response heterogeneity, further obscuring the presence of invalid data (i.e., with more respondents, the probability increases that each response to a question is chosen). Survey data (e.g., Boolean, Likert, and Multiple-Choice items) can obscure the validity of responses due to the constrained nature of survey response formats. We refer to this as the Validity Masking Effect. Employing finite response categories (Boolean, Likert, multiple-choice) means that any response is already plausible. Given this pigeonhole principle setup, the Validity Masking Effect states determining validity is practically impossible from the survey data itself for finite response categories, as the probability is that all options are selected by someone as the sample size grows. Many survey items have a finite set of predefined options, so large-scale surveys inherently exhibit significant response variance. For instance, binary questions force all responses into “yes” or “no” categories, producing seemingly stable patterns regardless of the respondent's intent or accuracy. Likert scales allow respondents to choose from a handful of options, rendering responses statistically indistinguishable in large samples where all options are likely to be selected by at least some respondents. These structural constraints make it impossible to distinguish valid from invalid survey responses using internal survey methods, such as attention checks, and, by extension, to distinguish between valid and invalid personas derived from that data. The Validity Masking Effect is a critical limitation of relying on survey data for research, which often includes Boolean, multiple-choice, or Likert-scale questions. These factors increase the difficulty of determining survey data validity and point to the need for alternative approaches, such as shorter, more targeted surveys with constrained sampling frames, and the incorporation of validation techniques derived from established methods in computer science and statistical analysis.

Our findings challenge the assumption that attention checks (and perhaps other internal survey measures) ensure the collection of higher-quality survey data. From our large-scale survey, responses from participants who passed the attention checks were nearly indistinguishable from those who failed, indicating that such internal measures have a limited effect on survey data validity. This finding is supported by some prior research indicating that attention checks are commonly used [39], but their effectiveness is questionable [51]. Attention checks may create a false sense of methodological rigor while masking broader reliability problems in survey data used for persona generation and other uses. Failing attention checks have been used to flag invalidity [33, 36]. However, our findings show that responses from those failing attention checks were mostly statistically indistinguishable from those of those passing attention checks [52]. Therefore, attention checks are not a means for certifying validity, and, at least for large scale surveys, for ensuring validity.

Now what to do about it? First, this finding requires the HCI research community to reconsider lengthy surveys in favor of shorter, targeted surveys that make data validation more manageable, especially for tasks such as persona creation. Shorter surveys have the advantage of avoiding respondent fatigue and a greater focus on a limited number of research constructs. Second, findings point to the need to cap the number of respondents per survey, as response variability with bounded choice questions increases with the number of respondents, making it more difficult to assess data validity (the Validity Masking Effect). In situations where large survey samples are needed, HCI and other researchers need to leverage validation mechanisms, such as stratified data-quality audits, response-pattern analysis, or qualitative verification. However, even these methods need to be evaluated, along with pursuing other novel approaches [6]. With personas created from survey data, independent data sources are needed for verification once the personas have been created.

The strength of this research for the community is that it systematically introduced invalid survey responses into valid data, enabling a controlled examination of how survey data degradation affects persona generation and exposing the practical difficulty of detecting invalid survey data. While the large-scale dataset is representative of many research surveys, future studies should investigate whether the invalidatable effect also occurs in smaller surveys or in surveys conducted with smaller samples. It would be interesting to determine whether the effectiveness of attention checks and other internal mechanisms is inversely proportional to the sample size, for example. Although the professional survey provider implemented the attention checks employed as part of their standard data-quality protocol, future research could investigate the effect of other types of attention checks and different validation mechanisms. Finally, an exhaustive investigation of algorithmic selection for persona creation would be quite beneficial for determining which algorithms are most responsive to minuscule changes in survey responses and which are not.