Survey Data Quality & Scrubbing Lab

Data Wrangling Qualtrics Hygiene

Audit raw 1-, 2-, and 3-row Qualtrics exports for preview/partial rows, speeders, contextual matrix straightlining, embedded attention-check traps, reverse-worded contradictions, and demographic attrition drift.

👨‍🏫 Professor Mode: Guided Learning Experience

Want a step-by-step walkthrough of how to audit Qualtrics survey exports, spot embedded traps, and verify demographic composition?

OVERVIEW & WHAT THIS TOOL SOLVES

Before you trust a survey's scale reliability, brand-lift test, or regression slope, you have to ask a blunt question: how many rows in your raw Qualtrics export came from preview test runs, 25-second speeders, matrix flatliners, or sweepstakes bots? Leaving careless responders in your dataset can destroy internal consistency and bury real treatment effects—while over-zealously deleting respondents who genuinely rated a short 4-item scale 5 out of 5 can bias your sample demographics.

🎯 Questions It Answers

  • Which columns in my 2-row or 3-row Qualtrics export are scale blocks, reverse-worded items, embedded attention-check traps, or demographics?
  • Is a matrix flatline (zero within-person variance) a dire carelessness signal (8+ items or crossing a reverse item) or an ambiguous short-block rating (4 unidirectional items)?
  • Does scrubbing low-quality responses restore reliability and signal without shifting sample demographics (attrition drift)?

📂 What Data You Need

  • A raw Qualtrics CSV/TSV export (3-row header with Row 2 question text and Row 3 ImportId JSON, 2-row header, or 1-row standard CSV).
  • Supports both "Use numeric values" (1..5) and "Use choice text" ("Strongly Agree") exports, plus shifted duration column names.
  • Ideally includes Duration (in seconds), 1+ multi-item scale blocks, an attention check, and 1+ demographic/quota columns.

💡 What You'll Discover

  • How to separate Stage 1 Structural Hygiene (previews & incomplete dropouts) from Stage 2 Behavioral Carelessness so partials never distort your median completion time.
  • How to spot hidden reverse-worded items and matrix-embedded attention checks using live item-rest correlation coaching.
  • How to export an analysis-ready 1-row-header CSV (with reverse-aligned items and composite scale means) plus an APA CONSORT cleaning statement.
🔍 Key Nuances & Common Traps to Watch For
  • Trap 1 — Leaving a Matrix Attention Check Inside Your Scale Block: Students often embed an attention check as Row 6 of a Likert matrix (e.g., Q4_6: "For quality control, select Strongly Disagree (1)"). If you leave Q4_6 classified as a regular scale item, attentive respondents who followed instructions look inconsistent with the positive items and tank Cronbach's \(\alpha\)! Reclassify Q4_6 as an Attention Check (IMC) in the Column Role Workbench.
  • Trap 2 — Treating All Straightlines Equally ("Short Block Over-Scrubbing"): Flatlining 4 unidirectional items (5, 5, 5, 5 on 4 positive satisfaction questions) frequently reflects a genuine promoter! Deleting respondents solely on a 3–4 item unidirectional flatline truncates your happiest customers. Our Contextual Straightlining Severity weights short unidirectional flatlines at 0.5 (soft corroborating only) while weighting 8+ item grids or reverse-item flatlines at 2.0 (dire).
  • Trap 3 — Differential Demographic Attrition Drift: In viral social or giveaway surveys, careless clicking often clusters in mobile or younger segments. Always check the Demographic Drift \(\chi^2\) Test to verify whether scrubbing shifted your sample composition.
  • Trap 4 — Scrubbing-Based \(p\)-Hacking: Never tweak quality cutoffs one row at a time just to push a \(p\)-value below \(.05\). Use the 4-Tier Robustness Matrix to verify your finding under the pre-registered Conservative Standard.
📐 Under the Hood: Model Equations & Variable Definitions
Speeder Index & Intra-Individual Response Variability (IRV) Stage 2 Behavioral Metrics
\[ \text{SI}_i = \frac{T_i}{\text{Median}_{k \in \mathcal{E}}(T_k)}, \qquad \text{IRV}_{i,b} = \sqrt{\frac{1}{m_b - 1}\sum_{j=1}^{m_b}\bigl(x_{i,b,j} - \bar{x}_{i,b}\bigr)^2}, \qquad P_{\text{random}}(\text{Flatline}_b) = K_b^{-(m_b - 1)} \]

💬 Read it in plain English: Relative Speeder Ratio \(\text{SI}_i\) compares respondent \(i\)'s completion time \(T_i\) against the median duration of Stage-1 eligible completers \(\mathcal{E}\). Intra-Individual Response Variability \(\text{IRV}_{i,b}\) measures how much respondent \(i\)'s ratings vary across the \(m_b\) items of matrix block \(b\) (\(\text{IRV}_{i,b} = 0\) means a 100% flatline, which occurs by chance with probability \(K_b^{-(m_b - 1)}\)).

Cronbach's Alpha & Sample Composition Drift Reliability & Attrition Drift
\[ \alpha_b = \frac{m_b}{m_b - 1}\left(1 - \frac{\sum_{j=1}^{m_b} s_j^2}{s_{\text{total}}^2}\right), \qquad \text{TVD}(p_{\text{clean}}, p_{\text{pool}}) = \frac{1}{2}\sum_{c=1}^{C}\bigl|p_{\text{clean},c} - p_{\text{pool},c}\bigr|, \qquad V = \sqrt{\frac{\chi^2}{N_{\text{pool}}\cdot \min(1, C-1)}} \]

💬 Read it in plain English: Cronbach's \(\alpha_b\) measures internal consistency after reverse-worded items are polarity-aligned. Total Variation Distance (\(\text{TVD}\)), Pearson's \(\chi^2\), and Cramér's \(V\) test whether the demographic shares of the retained sample \(p_{\text{clean},c}\) drift away from the eligible pool \(p_{\text{pool},c}\) across \(C\) categories.

Variable & Symbol Legend

Ti, ℰ
Completion Time & Eligible Pool YOUR DATA ⓘ

Respondent i's total survey duration in seconds (Ti) and the set of Stage-1 eligible survey completers (ℰ) used to anchor the sample median duration.

xi,b,j, x̄i,b
Matrix Item Rating & Row Mean YOUR DATA ⓘ

Respondent i's raw numeric rating on item j of matrix block b (xi,b,j) and respondent i's within-block average rating across those items (x̄i,b).

mb, Kb
Block Length & Scale Points YOUR DATA ⓘ

Number of scale items (mb) and response options (Kb, e.g., 5-point Likert) in block b, governing random flatline baseline probability Prandom.

SIi, IRVi,b
Speed Ratio & Within-Person SD OUTPUT ⓘ

Relative completion speed ratio compared to the eligible median (SIi) and intra-individual response variability across block b (IRVi,b = 0 indicates a flatline).

αb, sj2, stotal2
Cronbach's Alpha & Item/Scale Variances ESTIMATED ⓘ

Internal consistency reliability (αb) computed from individual item variances (sj2) and total composite sum variance (stotal2).

pclean,c, ppool,c, C
Subgroup Shares & Category Count YOUR DATA ⓘ

Proportion of respondents in demographic category c (out of C levels) in the retained clean sample (pclean,c) versus the Stage-1 eligible pool (ppool,c).

TVD, Npool
Total Variation Drift & Eligible N OUTPUT ⓘ

Total Variation Distance between retained and eligible demographic distributions across Npool eligible completers (0 indicates identical subgroup shares).

χ2, V
Attrition Chi-Square & Cramér's V INFERENCE ⓘ

Pearson's χ2 test of independence (retained vs. scrubbed across subgroups) and Cramér's V effect size testing whether scrubbing biases sample representativeness.

DATA SOURCE

Use a Case Study

Select a Qualtrics marketing case study above to inspect 2-row and 3-row headers, embedded matrix attention checks, contextual straightlining severity, and demographic attrition drift—or upload your own export on the right.

Upload Your Data

Drag & drop your Qualtrics .csv or .tsv export here

Automatically detects 3-row Qualtrics CSVs (Row 2 prose + Row 3 ImportId JSON), 2-row headers, and 1-row flat CSVs

No file uploaded.

INPUTS & SETTINGS: COLUMN ROLE STEERING & QUESTION INSPECTOR

Qualtrics often names columns with generic codes like Q4_5, Q4_6, or Q9_Trap—and shifts duration headers between Duration (in seconds) and Duration__in_seconds_. Inspect Row 2 question wording below and steer any column's role at any time.

Student Coaching Guide: How to Spot Hidden Traps, Reverse Items, & Scale Blocks

1. Matrix-Embedded Trap Questions (e.g., Q4_6) Look at the Row 2 Question Wording column below for phrases like "please select" or "quality control". If an attention check sits inside a matrix block, mark it as Attention Check / Trap (IMC) so it doesn't corrupt Cronbach's \(\alpha\)!
2. Unmarked Reverse-Worded Items (Negative \(r_{\text{item-rest}}\)) When a matrix item is phrased negatively (e.g., "bitter aftertaste", "overpriced", "breaks workflows"), attentive fans rate it 1 or 2. Marking it Scale Item — Reverse-Worded (−) aligns its polarity and activates the Reverse-Item Contradiction detector.
3. Demographics & Outcome Sensitivity Tag your demographic or quota columns as Demographic / Quota / Condition to test whether scrubbing biases your sample composition (\(\chi^2\) drift check), and tag 1 primary dependent variable as Outcome / Target Variable.
Column ID Qualtrics Row 2 Question Wording & Cues Answer Distribution Preview Assigned Role (Steerable) Block / Passing Target / Item \(r\)

SCRUBBING RULES & CONTEXTUAL STRAIGHTLINING SEVERITY

🧱 Matrix Scale Blocks: Contextual Straightlining Danger & Reliability

Not all matrix flatlines (\(\text{IRV}_i = 0\)) are equally damning! Compare each block's item count \(m_b\), reverse-worded item presence, random flatline baseline \(P_{\text{random}} = K_b^{-(m_b-1)}\), and assigned scrub weight below:

VISUAL OUTPUT: DIAGNOSTICS, 4-TIER ROBUSTNESS & DEMOGRAPHIC DRIFT

⏱️ Completion-Time Distribution & Speeder Cliff

Computed strictly on Stage-1 eligible completers so preview/partial rows never drag down the median \(M_T\).

🔍 Reading Guide: How to Spot the Speeder Cliff

Survey completion times are naturally right-skewed (some attentive readers pause or leave a tab open), which is why this lab anchors on the median \(M_T\) (green dashed line) rather than the mean. Look for a bimodal spike to the left of the orange dotted Speeder Cliff: when a cluster of respondents finishes a 3-minute questionnaire in under 35 seconds, they physically could not read the stems.

🚩 Two-Stage Attrition & Careless Flag Breakdown

Counts of structural Stage-1 exclusions vs. behavioral Stage-2 quality flags across the sample.

🔍 Reading Guide: Overlapping Signals vs. Soft Flags

A single careless respondent often triggers multiple behavioral bars at once (for instance, speeding, failing the IMC trap, and flatlining an 8-item matrix). Notice that Short-Block Flatliners (Wt 0.5 Soft) are tracked separately from Dire Matrix Straightliners (Wt 2.0): under the Conservative Standard, a 0.5 soft flag only drops a respondent if corroborated by speeding, bot telemetry, or another suspicious signal.

🧬 Respondent × Item "Straightliner Barcode" Heatmap

Respondents sorted from lowest within-person variability (\(\text{IRV}_i = 0.00\) flatliners at top) to normal variability below.

🔍 Reading Guide: Reading the Straightliner Barcode

Each horizontal row represents one respondent, and each column is a raw Likert item before reverse-coding. Solid mono-color horizontal bands at the top indicate \(\text{IRV}_i = 0.00\) flatliners. On blocks containing a (REV) column, attentive fans produce a distinct vertical color stripe on the reverse item (rating positive items blue 4–5 and the negative item orange 1–2), whereas careless flatliners paint straight across it.

⚖️ "Why Scrubbing Matters": 4-Tier Downstream Sensitivity & Anti-\(p\)-Hacking Matrix

Compare how your primary construct reliability (Cronbach's \(\alpha\)), scale-to-outcome correlation (\(r\)), and treatment/group difference (\(t\)-test) behave across all four canonical scrubbing tiers simultaneously:

🔍 Reading Guide: Guarding Against Scrubbing-Based p-Hacking

Random noise from speeders and bots attenuates both Cronbach's \(\alpha\) and substantive correlations toward zero. Comparing all four tiers side-by-side verifies whether your conclusion holds under the pre-specified Stage 2A Conservative Standard rather than cherry-picking an aggressive cutoff to force \(p < .05\).

Scrubbing Rule Tier Retained \(N\) Primary Block Cronbach's \(\alpha\) Scale Mean (SD) Outcome Mean (SD) Scale–Outcome \(r\) (\(p\)) Two-Group Difference \(\Delta\) (\(t\)-test \(p\))

👥 Sample Composition & Demographic Attrition Drift Check (Full Pool vs. Clean vs. Scrubbed)

Tests whether scrubbing removes careless responders evenly across subgroups (Composition Holds) or introduces selection bias (Differential Attrition Drift).

🔍 Reading Guide: Interpreting Composition Holds vs. Differential Drift

Compare the sky-blue Eligible Pool % bar against the green Clean Retained % bar for each subgroup. When the two bars stay within ~2–3 percentage points (\(\chi^2\; p \ge .05\)), scrubbing removed noise evenly. When the striped orange Scrubbed Out Group % towers over one subgroup (such as Mobile or 18–24 giveaway entrants), cleaning alters sample representativeness and should be disclosed or weighted.

Demographic / Quota Variable Subgroup Level Eligible Pool % (\(n\)) Clean Retained % (\(n\)) Shift \(\Delta\text{pp}\) Subgroup Scrub Rate % Attrition \(\chi^2\) (\(p\)) & Cramér's \(V\) Representativeness Status

TEST RESULTS & EXPORT: AUDIT TRAIL, APA STATEMENT & CLEAN CSV

📝 Copy-Ready APA / Methods Section Data-Cleaning & Attrition Statement

🔎 Respondent-Level Audit Inspector

Response ID Stage 1 Gate Duration (\(\text{SI}_i\)) Min Block \(\text{IRV}_i\) IMC Trap Open-Text Verbatim Preview Weighted Score & Flag Reasons Status Under Active Rule