HITL and Dataset Curation Interview Prep

Label Studio vs Argilla vs Prodigy — Annotation Tool Trade-offs

Sample interview questions

1. Which of the following best describes the five production HITL workflow patterns and their primary differentiator?
   • A. All five patterns route every output to a human reviewer; the only difference is the reviewer's seniority level.
   • B. The five patterns are pre-label-and-review, full-label, confidence-thresholded escalation, adjudication, and audit/spot-check. They are differentiated by stakes, throughput, and cost-per-output: pre-label-and-review is the default for medium-stakes high-volume systems; full-label is reserved for regulated or initial-golden-set construction; confidence-thresholded escalation (the active learning workhorse) auto-ships high-confidence outputs and queues borderline ones; adjudication uses multiple independent reviewers and a third-reviewer tie-break for golden-set construction; audit/spot-check reviews a retrospective statistical sample to catch drift. ✓
   • C. HITL patterns apply only during model development, not in production systems where automation must be complete.
   • D. The difference between patterns is solely the annotation tool used (Label Studio vs Argilla vs Prodigy).

   Option B correctly describes all five patterns and their decision axes. Pre-label-and-review is the common production default: a model pre-labels everything, humans review a sample (typically 1–10%). Cost-effective, but depends on a reliable confidence signal to route the right 1–10%. Full-label means humans review every output — slow and expensive but used for regulated domains (medical, legal) and for building the initial golden / regression set before any model can pre-label. Confidence-thresholded escalation is the active learning deployment pattern: high-confidence outputs ship without review, low-confidence outputs go to a human queue. As the golden set grows and the model improves, confidence on previously-borderline outputs rises, and the queue shrinks — the system gets cheaper over time. Adjudication uses two or more reviewers labeling the same output independently; a third reviewer (or rule) resolves disagreements. The adjudicated label is the golden truth. Disagreements are useful signal — high-disagreement cases are often edge cases worth promoting to the validation split. Audit / spot-check reviews a statistical sample of already-shipped outputs retrospectively to detect drift and validate that production quality is still acceptable. Mandatory in regulated domains; valuable everywhere. Production reality: most systems run several of these in parallel — full-label for the golden set, confidence-thresholded escalation for live inference, periodic audit on shipped outputs.

2. An active learning system claims to reduce annotation cost by 5–10x versus random sampling. What mechanism produces this reduction?
   • A. Active learning uses cheaper crowdsourced annotators instead of domain experts, reducing the per-label cost.
   • B. Instead of randomly sampling from the unlabeled pool, active learning routes only the most uncertain examples — those near the decision boundary with highest prediction entropy — to human annotators. Random sampling wastes budget on easy examples the model already handles correctly; these provide minimal gradient signal. Focusing the budget on uncertain, boundary-straddling examples provides maximal information gain per label. On most realistic distributions this routing achieves the same model accuracy as random sampling with 5–10x fewer labeled examples. ✓
   • C. Active learning removes the need for human review entirely by using LLM self-labeling for all uncertain examples.
   • D. The savings come from batching annotation requests, not from any smarter selection of which examples to label.

   Option B correctly describes the mechanism underlying the active learning cost reduction. The inefficiency of random sampling: a model trained on randomly selected examples includes many it already predicts correctly with high confidence. These easy examples provide little gradient signal during retraining — the model has already learned the pattern they represent. Annotating them is wasted budget. The efficiency of active learning: the model identifies examples where it is most uncertain — predictions with highest entropy over the output distribution, smallest margin between the top-2 predicted class probabilities, or highest disagreement among ensemble members. Annotating these uncertain, boundary-straddling examples shifts the decision boundary the most per label. The loop property from WP-12: as the golden set grows and the model improves, confidence on previously-borderline outputs rises and the queue shrinks. A well-designed active learning system therefore gets progressively cheaper over time — the more the model gets right, the less human time it costs. OSS libraries modAL and small-text implement uncertainty sampling, query-by-committee, and information-density strategies for this pattern. Options A, C, and D mischaracterize the source of savings: the improvement is structural (which examples get labeled) not staffing (who does the labeling), and human judgment remains central. Production reality: uncertainty-only sampling over-samples the model's known weak clusters. Production teams mix uncertainty + diversity (embedding-space coverage) + recent-failures to avoid repeatedly labeling the same kind of hard case.

3. How do you choose between Label Studio, Argilla, and Prodigy for a production annotation project?
   • A. They are functionally equivalent; choose the one with the best UI regardless of workflow.
   • B. Label Studio (Apache 2.0) supports 20+ task types — text, image, audio, video, time series — with a JSON-based labeling configuration language and strong team features (RBAC, adjudication UI, project management). Best for diverse data types and enterprise annotation teams. Argilla (Apache 2.0) specializes in NLP and LLM tasks with built-in Distilabel integration for LLM pre-labeling and native uncertainty-based queuing. Best for text annotation at scale and LLM fine-tuning data curation. Prodigy (commercial) is a scriptable minimal-overhead tool with native active learning and tight spaCy integration, optimized for NLP researchers who need rapid iteration. ✓
   • C. Prodigy is always the superior choice because it is the only commercial tool and therefore the most validated.
   • D. All three tools require dedicated infrastructure teams and are unsuitable for projects with fewer than 50,000 examples.

   Option B correctly characterizes the trade-offs for the three main OSS-and-near-OSS annotation platforms. Label Studio: Apache 2.0, widely deployed, the common starting point. Supports all data modalities via a JSON labeling configuration language. Role-based access control, per-project annotator assignment, and an adjudication UI make it suitable for large annotation teams with diverse task types. Active learning requires a custom ML backend plugin. Best for: enterprise teams with heterogeneous data and adjudication workflows. Argilla: Apache 2.0, purpose-built for NLP and LLM workflows. Suggestion-based pre-labeling via Distilabel (the Argilla-maintained synthetic-data and pre-labeling framework) means an LLM pre-labels examples and humans verify — 3–5x faster than labeling from scratch. Native uncertainty-based queuing integrates the active learning loop without custom code. Best for: LLM fine-tuning data curation, RLHF preference labeling, text annotation at scale with LLM-assisted workflows. Prodigy: commercial ($490/seat, one-time). Scriptable Python recipe engine — you write a recipe to define annotation tasks. Terminal-based, minimal overhead. Native active learning with entropy sampling out of the box. Tight spaCy integration for NLP pipelines. Weakness: no team collaboration features in the base product. Best for: NLP researchers and small teams who need rapid iteration cycles and know Python. Also worth knowing: Doccano (MIT) is a lighter-weight text-classification / NER / translation tool. LangSmith, Phoenix, and Langfuse each embed an annotation queue directly in their observability platform — convenient for teams already using those for tracing. Production reality: the annotation tool matters less than whether labels from it are wired back into the versioned golden set and retraining pipeline.

4. What are the five stages of the dataset lifecycle in a production ML system?
   • A. The lifecycle has only two stages: collect data and train the model. After that, the dataset is static.
   • B. The five stages are: (1) Bootstrap — hand-label a small seed set (<500 examples) sufficient to train a first classifier and validate a first rubric. (2) Feedback-driven expansion — mine production traces, low-confidence outputs, user reports, drift signals, and reviewer disagreements for new examples; production datasets typically grow 10–100x in their first 12 months this way. (3) Synthetic augmentation — generate examples for rare classes using Distilabel or Snorkel; verify every synthetic example with a critic agent or human spot-check before adding. (4) Hardening — correct stale labels, promote validation examples to regression, prune near-duplicates and low-quality items. (5) Versioning — record every change with DVC or LakeFS so downstream eval scores can be tagged with the dataset version they were computed against. ✓
   • C. The lifecycle is manage by the annotation tool automatically; engineers do not need to actively curate datasets.
   • D. Versioning is only needed for the final production dataset, not during development iterations.

   Option B correctly describes all five stages and their roles in a production dataset curation workflow. Bootstrap: the starting state. A small team hand-labels 100–500 examples — enough to train a first classifier and validate a first rubric. Most production datasets begin here. A bootstrap set without active learning guidance will become quickly stale because it was collected before the system knew which cases were hard. Feedback-driven expansion: the main growth engine. Production traces, low-confidence outputs (from confidence-thresholded routing), user-reported errors, drift signals, and reviewer disagreements are mined for new labeled examples. This is how a 500-example bootstrap dataset grows to 5,000–50,000 examples in the first year. This stage IS the active learning loop in production. Synthetic augmentation: for rare classes or expensive-to-collect cases, generate synthetic examples using Distilabel (LLM-driven pipeline) or Snorkel (weak supervision / labeling functions). Critical: every synthetic example must be verified — by a critic agent (adversarial critique pattern) or a human spot-check sample — before joining the golden set. Tag the source field: human | synthetic | production-pumped. Hardening: periodic audit to maintain quality, not just grow size. Correct labels that have drifted due to policy changes. Move examples from validation to regression when the model has learned to handle them reliably. Prune near-duplicates (Lilac) and examples flagged as likely mislabeled (Cleanlab). Bigger is not always better. Versioning: every change to the dataset is committed with DVC or LakeFS. Downstream eval results are annotated with the dataset version they were computed against. This is what enables dataset bisection — checking out an old dataset version and comparing it to the current one to trace a model regression. Production reality: teams that skip versioning lose the ability to debug regressions after a data refresh. "We evaluated against the golden set" is undefined as soon as the set changes without a version tag.

5. How do you measure and improve inter-rater reliability (IRR) in your annotation team?
   • A. IRR is the percentage of labels that match across annotators with no statistical adjustment for chance agreement.
   • B. IRR for two annotators is measured with Cohen's kappa: κ = (P_o − P_e) / (1 − P_e), where P_o is observed agreement and P_e is chance agreement. For 3+ annotators or ordinal scales, use Krippendorff alpha. Threshold: 0.6 is the minimum bar for most ML tasks; 0.8+ is strong. To improve IRR: run pre-annotation calibration sessions with 20–30 borderline examples; add concrete edge-case decisions to the annotation guidelines; track per-annotator kappa on a rolling basis; use adjudication (third-reviewer resolution) for high-stakes label classes rather than majority vote. ✓
   • C. IRR above 50% is acceptable for any annotation task regardless of label importance.
   • D. IRR measurement is only required for image annotation workflows, not for text or structured data labeling.

   Option B correctly describes the statistical approach to IRR measurement and the standard improvement techniques. Cohen's kappa (for 2 annotators): corrects for chance agreement — a metric that would be 1.0 even if both annotators labeled randomly at the same rate gets a kappa of 0.0. Formula: κ = (P_o − P_e) / (1 − P_e). Interpretation from the WP-12 and GLOSSARY thresholds: below 0.40 is poor (guidelines are too ambiguous), 0.41–0.60 is moderate, 0.61–0.80 is substantial, 0.80+ is near-perfect. For ML annotation, 0.6 is the minimum acceptable threshold — below this, labels are too inconsistent to be reliable training signal. Krippendorff alpha: generalizes to any number of annotators and any data scale (nominal, ordinal, interval, ratio). Use when you have 3+ annotators or ordinal severity/quality ratings. IRR improvement techniques: Calibration sessions before annotation: all annotators independently label the same 20–30 borderline examples, then discuss disagreements as a group. This surfaces implicit interpretations of the guidelines and aligns them before labeling at scale. Guideline quality: the most common cause of low IRR is ambiguous guidelines. Add concrete examples of borderline decisions — "label this A not B because..." — for every edge case that emerged in calibration. Rolling per-annotator monitoring: track kappa per annotator pair per batch. A sudden drop signals guideline drift or a new annotator interpreting differently. Argilla and Label Studio expose per-annotator agreement metrics. Adjudication vs majority vote: majority vote is statistically efficient but discards the disagreement signal. For high-stakes label classes (safety, medical severity, legal compliance), use adjudication — a third, more experienced reviewer resolves conflicts and documents the reasoning. The disagreement itself is often an edge case worth promoting to the validation split. Production reality: if human-vs-human kappa is below 0.6, the labeling guidelines are too ambiguous — fix the guidelines before scaling.

6. A model regresses after a data refresh — validation accuracy drops from 0.91 to 0.84. Walk through the dataset bisection methodology.
   • A. Assume the model architecture is the cause; retrain from scratch with a different architecture before examining data.
   • B. Run a systematic dataset bisection: (1) use DVC diff to identify which data slices changed between the previous and current dataset versions; (2) evaluate the old model (checkpoint before retraining) on the new data to determine whether the regression is in the data or the training procedure; (3) run Cleanlab on the new slice to detect likely mislabeled examples; (4) check for class imbalance shift in the new batch; (5) compare per-batch IRR — if kappa dropped on the new batch, the annotation quality is the root cause, not the model. ✓
   • C. Roll back the dataset to the previous version and permanently accept lower accuracy on the new data.
   • D. Dataset regressions are always caused by model hyperparameter drift, not data changes; investigate learning rate and batch size first.

   Option B describes the principled dataset bisection workflow from WP-12. Step 1 — DVC diff: DVC tracks dataset versions in git-like snapshots. Run `dvc diff HEAD~1 HEAD` to see exactly which data files changed, how many examples were added or removed, and which splits were affected. This tells you where to look before touching any model. Step 2 — Data-model isolation: Evaluate the old model checkpoint on the new dataset. If the old model also scores 0.84 on the new data, the regression is in the data (distribution shift or label noise introduced in the refresh). If the old model scores 0.91 on the new data, the regression is in the training procedure, not the data source. Step 3 — Cleanlab label-error detection: Cleanlab uses confident learning — cross-validating the model on its training set — to find examples the model consistently misclassifies despite high annotator confidence. These are likely mislabeled. Run Cleanlab on the new data batch to identify labeling errors introduced in the refresh. Step 4 — Class imbalance check: compare class distributions between the old and new datasets. If the new batch inadvertently added 3x as many examples of one class (a data source change, a filter change), the model may have overfit to that class during retraining at the expense of others. Step 5 — IRR audit: Pull per-batch kappa from Label Studio or Argilla. If IRR dropped from 0.82 to 0.65 on the new batch, annotations are lower quality — possibly a new annotator, rushed labeling, or unclear guideline updates. This is the labeling quality signal, and it is often the culprit after rapid data expansion. Production reality: dataset regressions are often more subtle than model regressions and require version-aware tooling to diagnose. DVC + Cleanlab + IRR monitoring form the standard toolkit for this class of debugging.

7. When is synthetic data safe to add to a golden set, and what verification steps are non-negotiable?
   • A. Synthetic data is always as good as real data because a capable LLM generates realistic, unbiased examples.
   • B. Synthetic data is appropriate for augmenting rare-class examples, generating paraphrases to increase diversity, and bootstrapping before real data is available. Before adding to the golden set, every synthetic example must pass: (1) critic-agent or classifier verification to catch model-generated artifacts and hallucinated entities; (2) a human spot-check sample (typically 5–10%) to validate that the synthetic distribution matches real data; (3) source tagging — every golden-set example carries source: human | synthetic | production-pumped so downstream consumers can filter. When the synthetic-to-real ratio exceeds ~50/50, audit periodically for divergence from real-data behavior. ✓
   • C. Synthetic data should never be used; only real human-labeled data produces reliable models.
   • D. Verification is only needed for image synthetic data; LLM-generated text synthetic data is inherently self-verifying.

   Option B correctly describes the conditions and verification discipline for synthetic data in production. Where synthetic data helps: Class imbalance augmentation: if 95% of training examples are class A and 5% are class B, the model is biased toward class A. Generating synthetic class B examples rebalances the training distribution. Well-established in fraud detection, rare medical conditions, and rare NLP phenomena. Paraphrase augmentation: generating paraphrases of existing training examples increases data diversity and reduces overfitting to surface form. Distilabel makes this straightforward. Bootstrapping: generating 200–500 synthetic examples to train an initial model before real data is collected accelerates the cold-start phase. Why verification is non-negotiable: Distribution shift: LLM-generated text tends to be more formal, grammatically correct, and longer than real user messages. A model trained on LLM-generated support tickets may fail on actual support tickets with abbreviations, typos, and informal register. Verification catches this before it enters the golden set. Bias amplification: the generator LLM carries its own biases. Synthetic NER data from a biased generator will encode those biases. The trained model will inherit and amplify them. Cleanlab can detect synthetic examples that are systematically mislabeled. Adversarial critic pattern (WP-07): running an adversarial critic over synthetic examples before golden-set admission catches model-generated artifacts — hallucinated entities, off-distribution phrasing — that a human reviewer might miss at scale. Source tagging: every example in the golden set should carry a source field. This lets you run controlled experiments — evaluate the model on human-only examples to check if synthetic augmentation is hurting real-data performance. Production reality: synthetic data without verification teaches the model the generator's biases at scale. With verification and source tagging, it is a common, effective scaling technique.

8. What is reviewer drift and what three mitigations does WP-12 recommend?
   • A. Reviewer drift describes annotators who quit mid-project; the mitigation is automated labeling to replace them.
   • B. Reviewer drift is gradual change in a reviewer's labeling behavior over time, independent of guideline changes. A reviewer who labels 1,000 examples in a week starts to develop labeling patterns that diverge from the original guidelines — a form of fatigue-induced inconsistency. The three mitigations are: (1) rotate reviewers across batches so no single reviewer labels the same domain continuously; (2) run periodic re-tests where every N batches the reviewer labels a small set of previously-labeled examples and the system tracks intra-reviewer consistency over time; (3) enforce daily review caps (200–500 labels per reviewer per day) to avoid fatigue-driven degradation. ✓
   • C. Reviewer drift only occurs when annotation guidelines change; stable guidelines prevent drift entirely.
   • D. Reviewer drift is a model evaluation problem, not a HITL workflow problem, and should be addressed by improving the LLM-judge.

   Option B correctly defines reviewer drift and the three standard mitigations from WP-12. Reviewer drift (canonical definition from GLOSSARY): gradual change in a reviewer's labeling behavior over time, independent of guideline changes. It manifests as slow label-distribution shift — a reviewer starts classifying ambiguous cases more liberally or conservatively over time, often without realizing it. The IRR between that reviewer and others drops; the model trained on their labels degrades. Mitigation 1 — Rotate reviewers across batches: ensure that no single reviewer labels the same domain continuously. Rotation breaks the fatigue cycle and prevents one reviewer's drift from dominating an entire slice of the golden set. Mitigation 2 — Periodic re-test against past labels: every N batches (or on a fixed calendar cadence), a reviewer labels a small set of examples they (or others) labeled previously. The system computes intra-reviewer consistency over time. A drop in self-consistency is an early warning of drift before it corrupts new batches. Mitigation 3 — Daily review caps: most annotation platforms recommend 200–500 labels per reviewer per day as a soft cap. Fatigue-driven errors increase sharply after prolonged annotation sessions. This is a staffing policy decision, not a technical one, but it is named explicitly in production HITL frameworks. One additional signal WP-12 raises: if intra-reviewer kappa against past labels drops while inter-reviewer kappa stays constant, the reviewer is drifting relative to their own past decisions. This is distinct from guideline drift (which would show up as inter-reviewer kappa drop for all reviewers simultaneously). Production reality: reviewer drift is the silent quality killer in long-running annotation projects. IRR monitoring is necessary but not sufficient — you also need intra-reviewer re-test to catch single-reviewer drift.

9. What is the golden / regression set and how does it differ from the test split?
   • A. The golden set and test split are the same thing; using two names is a naming inconsistency with no functional difference.
   • B. The test split is a statistically sized held-out partition (typically 10–20% of the full dataset) used for final model evaluation with confidence intervals. The golden / regression set is a separately curated, smaller collection (50–500 hand-picked examples) of canonical "must-pass" cases that runs in CI on every deploy. Key differences: test is statistical (large enough for confidence intervals), golden is canonical (smaller, every example is hand-curated and must pass). Golden set examples are treated like code — versioned in DVC, reviewed by PR, never exposed in eval prompts (golden-set leak makes every downstream metric unreliable). ✓
   • C. The golden set is simply the training set renamed; it includes all examples seen during model training.
   • D. The test split runs in CI on every deploy; the golden set is only evaluated once per quarter during a dedicated eval sprint.

   Option B correctly distinguishes the two constructs from the WP-12 dataset lifecycle. Test split: statistically sized (10–20% of the full dataset). Purpose: measure model quality with sufficient statistical power to compute confidence intervals and detect meaningful accuracy differences. Used for final evaluation and for release-decision comparisons. Touched as rarely as possible to prevent benchmark overfitting. Golden / regression set (GLOSSARY canonical definition): a hand-curated dataset of canonical "must-pass" examples that runs in CI on every deploy. Typically 50–500 examples. Every item is chosen deliberately — hard cases, edge cases, regression cases for known past failures. Small enough to run fast; carefully enough curated to be meaningful. Key differences in practice: Size and selection: the test split is large and randomly sampled; the golden set is small and deliberately curated. A test split with 5,000 examples catches average performance degradation; a golden set with 200 examples catches specific regressions. CI gating: the golden set runs on every deploy and blocks a release if any example fails. The test split is evaluated at release-decision points. Privacy: the golden set must never appear in eval prompts or system prompts where an LLM could memorize it across runs — golden-set leak produces flattering scores that don't generalize. Keep it in a holdout repo that production never reads from. Versioning discipline: the golden set is treated like code — every addition, correction, or removal is reviewed by PR and committed with DVC. Eval results from last month are comparable to this month's only because the golden set version is tagged alongside the result. Production reality: most teams start with only a test split and add a golden / regression set once they have been bitten by a regression that the test split caught too late.

10. What is LLM-assisted labeling and what is the anchoring bias risk it introduces?
    • A. LLM-assisted labeling means having the LLM label all examples with no human review, reducing annotation cost to zero.
    • B. LLM-assisted labeling is the pattern where an LLM pre-labels examples and human reviewers verify rather than label from scratch — 3–5x faster per example than pure human labeling. The anchoring bias risk: humans tend to accept the LLM's suggestion rather than evaluate independently, so accept rates drift toward the LLM's label distribution. Mitigation: periodically have a sample of reviewers label examples without seeing the LLM's pre-label (a blind arm) and compare their independent labels to the LLM-assisted labels to recalibrate. ✓
    • C. LLM-assisted labeling eliminates the need for inter-rater reliability measurement because the LLM is always consistent.
    • D. The anchoring bias in LLM-assisted labeling is benign because LLMs are more accurate than human annotators on all tasks.

    Option B correctly describes both the efficiency mechanism and the anchoring bias risk from WP-12. LLM-assisted labeling mechanism: an LLM (via Distilabel, Argilla's pre-labeling feature, or a custom pipeline) generates a suggested label for each example. Human reviewers see the suggestion and either accept it or override it. Because evaluating a suggested answer is faster than generating one from scratch, reviewer throughput increases 3–5x per example. Argilla and Prodigy both support this pattern natively. Anchoring bias (GLOSSARY canonical definition): in LLM-assisted labeling, human reviewers tend to accept the LLM's pre-label rather than evaluate from scratch. This is a well-studied cognitive bias — humans anchor on the first piece of information presented and insufficiently adjust from it. Consequences: if the LLM has systematic errors (a bias toward certain labels, a blind spot for certain phrasings), anchoring bias propagates those errors into the golden set. The model trained on those labels inherits the generator's biases. Mitigation: run a periodic "blind arm" — have a sample of reviewers label a subset of examples WITHOUT seeing the LLM's pre-label. Compare the blind-arm labels to the LLM-assisted labels. If the distributions diverge, the LLM pre-label is anchoring the reviewers away from their independent judgment. Use the divergence as a signal to recalibrate the pre-labeling pipeline or increase the blind-arm fraction. Production reality: LLM-assisted labeling is a genuine efficiency gain, but the anchoring bias is real. Teams that skip the blind-arm recalibration step often discover — only after a model regression — that their golden set has silently drifted toward the generator LLM's label distribution.

Frequently asked questions

What are the five HITL workflow patterns and when do you use each?

The five patterns are: (1) Pre-label-and-review — model labels everything, humans review a sample (1–10%); the common default for medium-stakes high-volume systems. (2) Full-label — humans review every output; used for regulated domains and initial golden-set construction. (3) Confidence-thresholded escalation — high-confidence outputs ship automatically, low-confidence outputs queue for human review; the active learning workhorse. (4) Adjudication — two or more reviewers label independently, a third resolves disagreements; used for golden-set construction where label quality matters more than throughput. (5) Audit / spot-check — a statistical sample of already-shipped outputs is reviewed retrospectively to detect drift. Most production systems run several of these in parallel.

What is active learning and how does it reduce annotation cost by 5–10x?

Active learning routes only the most uncertain examples — those near the decision boundary with highest prediction entropy — to human annotators rather than sampling randomly. Random sampling wastes annotation budget on easy examples the model already handles correctly. Focusing the budget on uncertain, boundary-straddling examples provides maximal information gain per label, achieving the same model accuracy with 5–10x fewer labeled examples. A well-designed active learning system also improves over time: as the golden set grows and the model improves, the queue of uncertain examples shrinks. OSS: modAL, small-text.

What is the dataset lifecycle for ML systems?

Five stages: (1) Bootstrap — hand-label 100–500 seed examples to train a first classifier. (2) Feedback-driven expansion — mine production traces, low-confidence outputs, and reviewer disagreements for new labeled examples; datasets typically grow 10–100x in the first 12 months. (3) Synthetic augmentation — generate examples for rare classes using Distilabel or Snorkel; verify every synthetic example before adding. (4) Hardening — correct stale labels, promote validation to regression, prune near-duplicates and mislabeled examples. (5) Versioning — commit every change to DVC or LakeFS so downstream eval results can be tagged with the dataset version they were computed against.

What is confidence-thresholded escalation and what threshold should I use?

Confidence-thresholded escalation routes model predictions to a decision point: if the model confidence exceeds a threshold (e.g., >0.85), the prediction ships without human review; below the threshold, the example queues for annotation. This is the core active learning deployment pattern. The threshold is tuned on a validation set by trading off human review cost against prediction accuracy — there is no universal value. Caveat: LLM self-reported confidence scores are noisy. Production systems add a trained calibration layer so that a model reporting 85% confidence is empirically right 85% of the time.

How do you measure inter-rater reliability (IRR)?

For two annotators: Cohen's kappa κ = (P_o − P_e) / (1 − P_e), where P_o is observed agreement and P_e is chance agreement. For 3+ annotators or ordinal scales: Krippendorff alpha. Thresholds (from GLOSSARY): 0.6 is the minimum acceptable bar for ML annotation; 0.8+ is strong. If IRR is below 0.6, the labeling guidelines are too ambiguous — fix the guidelines before scaling. Improve IRR with: pre-annotation calibration sessions, concrete edge-case examples in guidelines, and rolling per-annotator kappa monitoring.

What is the golden / regression set and how does it differ from the test split?

The golden / regression set is a small (50–500 examples) hand-curated collection of canonical "must-pass" cases that runs in CI on every deploy. It blocks a release if any example fails. The test split is statistically sized (10–20% of the full dataset) and used for measuring average model quality with confidence intervals. Key distinction: test is statistical (large, random sample); golden is canonical (small, deliberately curated). Golden set examples are versioned like code, reviewed by PR, and never exposed in eval prompts — golden-set leak makes every downstream metric unreliable.

When is synthetic data acceptable and when does it harm quality?

Use synthetic data for: augmenting rare-class examples, generating paraphrases to increase diversity, and bootstrapping models before real data is available. Avoid or verify carefully when: the generator LLM introduces distribution shift (synthetic text tends to be more formal than real user text), when it amplifies generator biases into the golden set, or when authenticity matters (medical records, legal documents). Non-negotiable verification steps: critic-agent verification of every synthetic example, a human spot-check sample (5–10%), and source tagging (human | synthetic | production-pumped) on every golden-set example.

How do you version datasets with DVC?

DVC (Data Version Control) provides Git-style versioning for datasets: `dvc add data/training.csv` creates a .dvc pointer committed to git; the actual data is pushed to a DVC remote (S3, GCS, Azure Blob). `dvc diff HEAD~1 HEAD` shows changes between dataset versions. This enables dataset bisection: check out an old git commit, run `dvc pull`, and you have the exact dataset version used to train a previous model. Every eval result should be annotated with the dataset version it was computed against — "we evaluated against the golden set" is undefined without this.

What is reviewer drift and how do you detect it?

Reviewer drift is gradual change in a reviewer's labeling behavior over time, independent of guideline changes. A reviewer who labels 1,000 examples in a week develops patterns that diverge from the original guidelines due to fatigue. Detect it with: intra-reviewer re-tests (have the reviewer label a set of previously-labeled examples every N batches and track self-consistency over time) and rolling per-batch kappa monitoring. Mitigate with: reviewer rotation across batches, daily review caps (200–500 labels/day), and the re-test cadence. Reviewer drift is distinct from guideline drift (which shows up as inter-reviewer kappa drop for all reviewers simultaneously).

What are the most common pitfalls in production HITL and dataset curation?

Ten pitfalls from WP-12: (1) Random sampling instead of active — wastes 80% of annotation budget. (2) Single-reviewer labels with no IRR check — undetectable label noise. (3) Reviewer burnout — drifty low-quality labels from fatigue. (4) No dataset versioning — regressions after data refresh cannot be traced. (5) Treating synthetic data as gold without verification — teaches model the generator's biases. (6) Golden set leak into eval prompts — model memorizes; scores don't generalize. (7) HITL without feedback — labels sit in the annotation tool and never improve the model. (8) Treating dataset growth as monotone — bigger is not better if examples are redundant or stale. (9) One reviewer for everything — SME leaves and institutional memory goes with them. (10) Optimizing for IRR at the expense of validity — two reviewers agree on the wrong rule.

Related topics