Adversarial Critique for AI Evals Interview Prep

Critic Types — Prompt Flavor, Output Format, Cost, and Production Role

Sample interview questions

1. What is the structural difference between rubric scoring and adversarial critique, and why does a production eval pipeline need both?
   • A. Rubric scoring and adversarial critique are synonyms — both assign a numeric score to each output dimension
   • B. Rubric scoring is quantitative: it assigns a numeric value per dimension and determines accept/review/reject. Adversarial critique is diagnostic: it produces a structured findings list — each finding has a location in the output, a description of what is wrong, a severity level (blocker/major/minor), a one-sentence explanation, and an optional suggested fix. A rubric score tells you something is wrong; the adversarial critic tells you what to fix and where. The two are paired by design: the score is the gate, the critic findings are the fix signal. ✓
   • C. Adversarial critique replaces rubric scoring because it is more informative; rubric scores are deprecated in modern AI eval pipelines
   • D. Rubric scoring is only used during model training; adversarial critique is only used at inference time

   Option B correctly captures the quantitative-vs-diagnostic distinction from WP-07. Rubric scoring (WP-05): assign a numeric score per quality dimension. The composite score determines the routing decision — accept (above threshold), review (borderline), reject (below floor gate). This is the gate. It tells you "the clarity dimension scored 3.2 out of 5." Adversarial critique (WP-07): produce a structured findings list. Each finding is: - Where: the specific location in the output (paragraph 2, sentence 1; the second bullet; the closing recommendation). - What: the specific problem (hallucinated company claim, logical gap, off-brand tone). - Severity: blocker (blocks acceptance), major (should fix), minor (nice to fix). - Why: one-sentence explanation a revisor or human can act on. - Suggested fix (optional): what a corrected version might look like. The critic is the bridge between scoring and improvement. An output that scored 3.2 on "clarity" is not actionable without a finding that says "paragraph 2, sentence 1 assumes the reader knows what 'TLS handshake' means — no definition or context is provided." With that finding, a revisor can fix the right thing. The two are paired by design: run the rubric scorer first to gate, run the adversarial critic on borderline and failed outputs to generate fix signals. Running both on every output is cost-inefficient; cascading them (score first, critique on failure or borderline) is the production pattern. Options A, C, and D are incorrect: they are complementary not synonymous, critique does not replace scoring, and both are used at inference time in production eval pipelines. Production reality: the scorer and critic share the same rubric definition — the critic's dimensions mirror the rubric's dimensions, so critic findings are directly traceable to which scored dimension failed.

2. What are the three flavors of adversarial critic prompt, and when does each earn its added cost?
   • A. There is only one critic prompt type: "Rate this output on a scale of 1 to 10 and explain your score"
   • B. The three critic prompt flavors are: (1) Generic ("find anything wrong") — fastest and most flexible, but tends to surface only obvious issues and misses subtle ones; useful for early-stage triage before a rubric exists. (2) Rubric-driven ("attack each dimension") — the critic receives the rubric and produces findings per dimension; more structured and thorough but more expensive (one critic call per dimension or per cluster); preferred for production quality gates. (3) Constitutional/principled ("here are the rules to enforce") — the critic walks a fixed list of principles and reports violations; optimized for safety and policy compliance (Llama Guard implements this as a classifier). The flavor choice is: cheap+broad for exploration, rubric-driven for regression tracking, constitutional for safety gates. ✓
   • C. Prompt flavor only matters during model training; at inference time all critic prompts behave identically
   • D. Rubric-driven prompts are always the correct choice; the other flavors are deprecated patterns

   Option B correctly describes the three flavors and their deployment contexts from WP-07 §2. Generic critic ("find anything wrong"): the simplest framing. "Read this output and list everything that seems off." Cheap, broad, unstructured. Tends to surface obvious issues; misses subtle ones a focused critic would catch. The critic's output is prose, not a structured JSON findings list — hard to aggregate into dashboards. Best for: rapid triage, early-stage evaluation before a rubric exists. Cost: one LLM call. Rubric-driven critic ("attack each dimension"): the critic receives the rubric from WP-05 and produces findings per dimension. Returns a structured JSON array of findings, each tied to a rubric dimension. More thorough — it has a mandate to attack each dimension, not just the obvious issues. More expensive — one critic call per dimension (or per cluster of dimensions). Best for: production quality gates where findings must be traceable to scored dimensions, and where regression tracking across dimensions is required. DeepEval G-Eval implements this pattern. Constitutional/principled critic ("here are the rules to enforce"): the critic walks a fixed list of principles (a "constitution") and reports violations. Each principle is a rule: "refuse harmful requests," "cite sources for factual claims," "do not generate code without sandboxing." Anthropic Constitutional AI is the canonical design. Llama Guard 3 implements this as a fine-tuned classifier against the MLCommons Hazard Taxonomy. Best for: safety guardrails, policy compliance screening. Cost: fast (classifier) or moderate (LLM walking a list). Framing matters more than most teams expect: "Is this good?" produces praise. "Find every way this could be wrong, assuming someone wrote it carelessly" produces useful findings. Production critic prompts almost always include the phrase "you are a hostile reviewer" or "your job is to fail this output." Production reality: cascade the flavors. Specialist classifiers (constitutional) run first — cheap and narrow. Rubric-driven critic runs on outputs that pass the constitutional gate. Generic critic is reserved for exploratory audits of new failure categories.

3. What is sycophancy in LLM-as-judge evaluation and how does cross-model critique prevent it?
   • A. Sycophancy is when an LLM refuses to answer difficult questions; cross-model critique fixes this by using a larger model
   • B. Sycophancy in LLM-as-judge evaluation is the tendency of a model to give higher scores to outputs from the same model family — because the judge and generator share training data, pretraining biases, and hallucination patterns, so the judge cannot detect errors it would make itself. Cross-model critique prevents sycophancy by routing evaluation to a judge from a different provider or training lineage: a GPT-4o generator critiqued by a Claude judge, or a Gemini generator critiqued by a Llama-based judge. Cross-family judging is the cheapest sycophancy mitigation available; three same-family critics with majority vote do not fix it because the shared priors produce one signal repeated three times. ✓
   • C. Sycophancy only affects RLHF training and has no effect on inference-time evaluation
   • D. Cross-model critique requires two identical models from different checkpoints; a judge from a different provider introduces too much variance

   Option B correctly defines sycophancy in the LLM-as-judge context and the cross-model mitigation from WP-07 §6. The sycophancy failure mode: when Model A generates an output and Model A evaluates that output, A tends to give itself higher scores than an independent human reviewer would. The mechanism: the judge and generator share training data, pretraining biases, and hallucination patterns. The judge cannot detect errors it would make itself. It also tends to validate outputs that match its own stylistic preferences regardless of accuracy. Documented evidence: Zheng et al. (2023) "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" found that models significantly favor outputs from their own family when controlling for human preference scores. Cross-model critique as prevention: use a judge from a different training lineage. WP-07 §6 recommends explicitly: "generate with one family, critique with another." Example pairings used in production: - OpenAI generator + Anthropic judge (different RLHF, different pretraining data) - Google generator + Meta/Llama judge (different architecture, different instruction tuning) - Proprietary generator + open-weights judge (different commercial constraints) A common production deployment: generate with the cheap model (Haiku, gpt-4o-mini), critique with the more capable one (Sonnet, GPT-4), or invert if the cheap model is the better judge for the specific task. Test both directions. Why three same-family critics don't fix it: same-family critics share priors. Majority vote among them is not three independent signals — it is one signal repeated three times. One cross-family critic adds genuine independence that three same-family critics cannot. Additional mitigations that compound with cross-model critique: position randomization (swap A/B order in pairwise comparison), structured rubrics (reduce surface area for bias by requiring per-dimension scores rather than holistic ratings), explicit adversarial prompt framing. Production reality: cross-model critique is the cheapest sycophancy fix available. It requires only changing which model handles the judge call — no architecture change, no additional infrastructure.

4. What are the four classic adversarial critic failure modes defined in WP-07, and how do you fix each?
   • A. Adversarial critics have only one failure mode: they are too expensive to run in production
   • B. The four classic failure modes are: (1) Sycophancy — the critic defaults to praise, especially when it is the same model family as the generator. Fix: cross-model critique + adversarial prompt framing + explicit anti-praise instructions. (2) Hallucinated findings — the critic reports a problem that does not exist (a missing citation that is present, a factual error in a correct claim). Fix: require findings to quote the problematic span; validate periodically against humans. (3) Mirror effect — the critic returns one finding per rubric dimension regardless of whether each dimension has a real issue. Fix: prompt the critic to return an empty array when a dimension is fine; reward terse output. (4) Drift — a critic prompt that worked six months ago now misses obvious issues because the model or the world changed. Fix: re-validate the critic against humans quarterly; pin the critic model version. ✓
   • C. The four failure modes are cost, latency, hallucination, and bias — all fixed by using a smaller model
   • D. Adversarial critics have no failure modes if they use a constitutional AI prompt

   Option B correctly lists and fixes the four classic failure modes from WP-07 §7. Failure mode 1 — Sycophancy: the most common failure. The critic — especially when it is the same model that generated the output — defaults to "this looks good, minor suggestions: ..." The framing matters enormously: "Is this good?" produces praise. "Find every way this could be wrong, assuming someone wrote it carelessly" produces useful findings. Fix: cross-model critique (different provider/family), adversarial prompt framing ("you are a hostile reviewer; your job is to fail this output"), explicit anti-praise instructions ("do not look for things to praise"). Failure mode 2 — Hallucinated findings: the critic confidently reports a factual error that is not there, or invents a missing citation when one is present. This is the inverse of output hallucination — the critic hallucinates a problem. Fix: require findings to quote the problematic span from the output (a finding without a quote is a hallucination); sample 5–10% of findings for human verification on an ongoing basis; if hallucination rate is above a few percent, tighten the prompt or move to a more capable model. Failure mode 3 — Mirror effect: the critic returns exactly one finding per rubric dimension regardless of whether each dimension has a real issue — it mirrors the rubric structure instead of attacking the output. Fix: explicitly prompt "return an empty array for any dimension that is genuinely fine"; reward terse output over thorough output; treat an empty array on a dimension as a valid and expected outcome. Failure mode 4 — Drift: a critic prompt that worked great six months ago now misses obvious issues because the underlying model changed or the distribution of inputs shifted. The critic still runs and returns findings, but its precision against human judgment has quietly degraded. Fix: re-validate the critic against humans on a quarterly cadence (WP-15 covers drift detection); pin the critic model version explicitly; track critic-finding precision (true findings divided by all findings) over time — the curve is your earliest drift signal. A fifth failure mode noted in WP-07: severity inflation — everything becomes a "blocker." Fix: calibrate severity rates against your actual tolerance and treat severity definitions as code that gets reviewed and tested. Production reality: hallucinated findings and mirror effect are the failure modes teams discover latest, typically only after a human review audit reveals that the critic is noisy or redundant.

5. What is the advocate / adversary / gap-finder / orchestrator pattern for round-table critics, and when does it outperform a single adversarial critic?
   • A. The advocate/adversary/gap-finder/orchestrator pattern is a training technique; it is not used in production evaluation
   • B. In the role-based round-table pattern from WP-08, four agents read the same output in parallel from distinct perspectives: the Advocate argues for the output's strengths, the Adversary attacks every weakness (the single adversarial critic from WP-07 in parallel), the Gap-finder identifies what the output did not address but should have, and the Auditor verifies the other three roles' findings for hallucinated or exaggerated claims. An Orchestrator then reads all four verdicts and synthesizes a final judgment with a written audit trail. It outperforms a single critic when: single-critic blind spots are costing money; the output is contested (different framings produce different findings); or compliance requires a documented review trail. ✓
   • C. The pattern requires all four roles to use the same model to ensure consistency
   • D. The orchestrator role makes the pattern equivalent to a single critic because it collapses all findings to one verdict

   Option B correctly describes the role-based round-table pattern from WP-08 §2. The four roles and their mandates: - Advocate: "defend the strengths of this output; identify what works and why; be specific; cite quotes." Prevents the system from ignoring genuine strengths when deciding whether to revise or reject. - Adversary: "find every weakness, error, or unsupported claim; do not look for things to praise; cite quotes." This is WP-07's single adversarial critic in parallel at the round-table. - Gap-finder: "what did this output NOT address that it should have? list specific missing topics, claims, or context." Catches omission errors that the adversary (which focuses on what is wrong with what IS there) misses. - Auditor: reviews the other three roles' findings and flags any that are themselves hallucinated, exaggerated, or contradict the output text. The meta-critic from WP-07 §6 built into the pattern. - Orchestrator: reads all four verdicts, weighs them, and synthesizes a final verdict — a single severity (blocker/major/minor/clean), a 2–3 sentence rationale, and a deduplicated list of validated findings (excluding any the auditor flagged). The pattern produces a written audit trail (each role's verdict + reasoning), which is far more interpretable than a single composite score and is required for compliance-sensitive workflows where decisions must be defensible. When it outperforms a single critic (from WP-07 §8): 1. Single-critic blind spots are costing money — one critic misses a class of issue another framing would catch. 2. The output is genuinely contested — different framings (advocate/adversary/gap) produce different findings on the same output; synthesizing them is the round-table's purpose. 3. Compliance requires a documented review trail — multiple critic perspectives recorded per output is the audit-trail-first approach. Cost: 3–5 parallel critic calls + 1 auditor call + 1 orchestrator call = 4–6× a single-critic pass. WP-08 §6 recommends tiering the spend: run the round-table only when the single-critic score is borderline, the user requests high quality, or the output goes to a high-stakes downstream. Production reality: cross-family model assignment across roles (Claude for advocate, GPT-4 for adversary, Gemini or open-weights for gap-finder) prevents same-family bias from cascading through every role into the orchestrator's verdict.

6. How do PyRIT and Garak differ in adversarial red-teaming for LLMs, and when do you use each?
   • A. PyRIT and Garak are identical tools from Microsoft; only one needs to be installed
   • B. PyRIT (Python Risk Identification Toolkit, Microsoft) is an orchestration framework that automates multi-turn adversarial conversations: an attacker LLM iteratively refines jailbreak prompts against a target LLM until a harmful response is elicited or a turn budget is exhausted. Garak (Generative AI Red-teaming and Assessment Kit, NVIDIA) is a probe-and-detector scanner that runs a library of 100+ static probes (prompt injection, hallucination, toxicity, bias, DAN variants) against an LLM endpoint and reports which probes the model fails. Use PyRIT for dynamic, adaptive red-teaming of multi-turn systems; use Garak for broad static coverage of known vulnerability categories. Practical workflow: run Garak first to identify weak categories, then use PyRIT to confirm those weaknesses are dynamically exploitable. ✓
   • C. Red-teaming tools like PyRIT and Garak are only for security researchers; product teams do not need them
   • D. Both tools require white-box model access (weights and internals); they cannot be used against API-only models

   Option B correctly contrasts PyRIT and Garak and identifies their respective deployment contexts from WP-07 §3 and WP-08. PyRIT (Microsoft, Apache 2.0): an orchestration framework for automated adversarial attack campaigns. Core loop: (1) an "attacker" LLM generates a jailbreak prompt, (2) the prompt is sent to the "target" LLM, (3) a "scorer" LLM judges whether the response contains harmful content, (4) if not, the attacker refines the prompt for the next turn. The loop continues until a harmful response is found or a turn budget (typically 5–20 turns) is exhausted. PyRIT is designed for: multi-turn chatbot systems where context accumulation enables attacks that single-turn probes miss; agentic systems where tool-calling can be exploited across turns; iterative red-team campaigns where the attack strategy adapts to the target model's responses. Garak (NVIDIA, Apache 2.0): a static probe library with 100+ probes organized by category — prompt injection, hallucination, toxicity generation, jailbreak, encoding attacks (base64, leetspeak), bias probing, DAN variants, package hallucination. Each probe runs against the target endpoint; detectors score whether the response indicates a failure. Garak is designed for: broad vulnerability scanning of a new model or API before deployment; regression testing (did a fine-tuned model break safety behavior the base model had?); compliance coverage reports. Practical red-team workflow from WP-07: run Garak first (broad static scan, 2–4 hours) to identify vulnerable categories, then use PyRIT to confirm dynamic exploitability of the discovered vulnerabilities with multi-turn attacks. Both work against black-box API endpoints. Both tools are classified in WP-07 as "specialist critics in the red-team/adversarial-probing category" — closer to "automated red team" than per-output critics, but playing the critic role in the safety dimension of a production eval pipeline. Production reality: Garak is the right pre-deployment coverage check. PyRIT is the right tool for confirming that a specific vulnerability found by Garak is dynamically exploitable by an adaptive attacker — not just by a static probe.

7. What does HHEM-2.1 (Vectara) measure, and why is it preferred over an LLM-judge for hallucination detection in RAG pipelines?
   • A. HHEM-2.1 uses keyword matching to detect hallucinations; it does not use a learned model
   • B. HHEM-2.1 (Vectara HHEM-2.1-Open, Apache 2.0) is a fine-tuned cross-encoder that scores how well each claim in a RAG answer is grounded in the retrieved context. It takes a (context, claim) pair and outputs a score between 0 (hallucinated) and 1 (grounded). It is preferred over LLM-as-judge for hallucination detection because: (1) it is orders of magnitude faster and cheaper than an LLM generation call, (2) it is trained independently of popular generator models, eliminating same-family sycophancy bias, and (3) it produces per-sentence scores identifying exactly which sentences are hallucinated rather than a holistic pass/fail verdict. ✓
   • C. HHEM-2.1 only evaluates open-weights model outputs; it cannot evaluate closed-source API outputs like GPT-4o or Claude
   • D. HHEM-2.1 requires the full retrieval pipeline to be rebuilt for each evaluation; it cannot be used with pre-generated answers

   Option B correctly describes HHEM-2.1 architecture, design intent, and production advantages from WP-07 §3 and §8. Architecture: HHEM-2.1 is a fine-tuned cross-encoder built on FLAN-T5 (an encoder-decoder model, not a generative chat LLM). Input: the (context, claim) pair scored jointly. Output: probability score 0–1 indicating how well the claim is grounded in the context. The cross-encoder setup allows HHEM-2.1 to attend to both context and claim jointly — a stronger faithfulness signal than embedding similarity, which encodes each independently. Why it is listed in WP-07 as a "specialist critic" (Tier-2 from WP-02): a specialist critic is a trained classifier used in place of an LLM-judge for measurable dimensions. HHEM-2.1 handles the faithfulness dimension — which is measurable and where a classifier outperforms an LLM-judge — leaving the LLM-judge for nuanced dimensions a classifier cannot measure. Three advantages over LLM-as-judge for hallucination: 1. Speed and cost: cross-encoder inference is orders of magnitude faster than LLM generation. At production scale, one HHEM-2.1 call per sentence costs a fraction of one LLM-judge call per output. 2. Cross-model independence (no sycophancy): HHEM-2.1 is trained by Vectara, independent of OpenAI, Anthropic, Google, or Meta. When evaluating a GPT-4o or Claude output, it has no self-evaluation bias. 3. Per-sentence granularity: HHEM-2.1 produces a score per sentence, identifying exactly which sentences are hallucinated. An LLM-judge typically returns a holistic verdict, making targeted correction harder. RAG evaluation workflow: (1) retrieve context chunks, (2) generate the answer, (3) split the answer into sentences, (4) run HHEM-2.1(context_chunks, sentence) for each sentence, (5) flag sentences below threshold (default 0.5), (6) aggregate into an answer-level hallucination rate. Patronus Lynx is an alternative hallucination judge (CC-BY-NC-4.0, non-commercial license) with stronger benchmark metrics in some comparisons. HHEM-2.1-Open is the Apache-2.0 choice for commercial deployments. Production reality: the specialist-critic cascade from WP-07 §6 is: run HHEM-2.1 first for faithfulness (cheap, deterministic for a given threshold); escalate to an LLM-judge only for outputs HHEM-2.1 flags as borderline (0.4–0.6 range).

8. How do you validate an adversarial critic against human judgment, and what agreement threshold signals the critic is trustworthy?
   • A. Critic validation is unnecessary if you are using a state-of-the-art model as the judge
   • B. Validation requires a held-out set of 50 outputs rated by two expert human annotators per rubric dimension. Run the critic on the same set and compute Spearman or Pearson correlation between critic scores and human scores per dimension. A well-calibrated critic should show correlation above 0.7 per dimension. For critic findings (not just scores), compute precision (fraction of critic findings that humans confirm as real issues) and recall (fraction of human-identified issues that the critic surfaced). A critic with below-70% inter-rater agreement between its findings and human expert findings is not ready for a production gate — iterate the prompt before scaling. ✓
   • C. Critic validation only applies to constitutional classifiers; LLM-judge critics are inherently unbiased
   • D. Validation requires a dataset of at least 10,000 examples; smaller held-out sets produce unreliable agreement estimates

   Option B correctly describes the critic validation workflow from WP-07 §6, which applies the same validation discipline as WP-05 rubric scorer validation. The validation workflow: 1. Collect a held-out set of 50 outputs — enough to measure agreement without breaking the budget, per the WP-07 production checklist: "validate critic findings against 50 human-labeled outputs before trusting them in any production gate." 2. Have two expert human annotators independently flag issues on each output per rubric dimension. Compute inter-rater agreement between the two humans (the ceiling for what a critic can achieve). If humans agree on fewer than 60% of issues, the rubric criteria are ambiguous — fix the rubric before evaluating the critic. 3. Run the critic on the same 50 outputs. 4. Compare critic findings to human findings: - For scored dimensions: Spearman or Pearson correlation between critic scores and human scores per dimension. Threshold: 0.7+ per dimension. - For findings: precision (what fraction of critic findings are confirmed by humans as real issues) and recall (what fraction of human-identified issues did the critic surface). A critic that disagrees with humans is worse than no critic: it introduces noise into the downstream revisor (WP-06) and the CI gate, causing false blocks and false approvals. The validation workflow from WP-05 §6 applies here per WP-07 §6: "a critic that disagrees with humans is worse than no critic." What low agreement diagnoses (from WP-07 §6): if critic-vs-human agreement is materially below human-vs-human agreement, the critic prompt is the bottleneck — not the model. Common prompt issues: rubric criteria are ambiguous, anti-praise framing is missing, findings are not required to cite specific spans (enabling hallucinated findings). Per-severity calibration: check the distribution of severity ratings. If the critic fires "blocker" at >10–15% of outputs but human annotators rate only 1–3% as blockers, severity inflation is present. Adjust the severity definitions in the prompt until rates match. Production reality: track critic-finding precision over time as an ongoing drift signal. The curve decays as the model changes or the input distribution shifts — re-validate quarterly (WP-07 production checklist).

9. When should you cascade specialist classifiers in front of a generative adversarial critic, and what does the cascade save?
   • A. Specialist classifiers should always run after the generative critic because the generative critic is more accurate
   • B. Specialist classifiers (Tier-2: HHEM-2.1 for faithfulness, Llama Guard for safety, Detoxify for toxicity) should run before the generative adversarial critic for dimensions they were trained to measure. A toxicity classifier is cheaper and faster than asking an LLM "is this toxic?" An HHEM-2.1 faithfulness score is cheaper than asking an LLM "is this answer supported by the context?" The cascade saves cost: run classifiers first for measurable dimensions; invoke the generative LLM critic only for dimensions a classifier cannot address (tone nuance, logical coherence, audience-appropriateness). This is the same hybrid tier pattern from WP-02 applied to critique. ✓
   • C. Specialist classifiers are only suitable for training data curation, not production critique pipelines
   • D. Cascading classifiers in front of a generative critic requires white-box access to both models

   Option B correctly describes the specialist-classifier-first cascade from WP-07 §6 and §8, which applies WP-02's hybrid tier pattern to critique. The WP-02 hybrid pattern applied to critics: choose the cheapest, most reproducible critic first. Layer an LLM-judge on top only for dimensions a classifier cannot measure. Classifier-eligible dimensions (measurable, deterministic critics exist): - Safety / toxicity: Llama Guard 3 (MLCommons Hazard Taxonomy), Detoxify, LlamaPromptGuard. Sub-100ms at production throughput. More reliable than a prompted LLM for these specific categories. - Faithfulness / hallucination: HHEM-2.1 (Vectara). Per-sentence grounding scores against retrieved context. Faster and cheaper than an LLM-judge. - PII detection: Microsoft Presidio. Rule-based + ML for detecting emails, SSNs, phone numbers. - NLI / entailment: HuggingFace NLI classifiers for claim-entailment from context. LLM-judge-only dimensions (no classifier matches this nuance): - Tone and audience-appropriateness. - Logical coherence and argument structure. - Creative quality and brand voice. - Domain-specific reasoning quality (legal, medical, engineering). Cost rationale from WP-07 §8: "a toxicity classifier is cheaper than asking an LLM 'is this toxic.' Cascade: specialist classifiers first for measurable dimensions, generative critic only for dimensions a classifier can't address." The critic budget ceiling from WP-07 §7: "running a strong LLM critic on every single inference is expensive. Most teams critique only when (a) the rubric score is borderline, (b) the user explicitly requested high quality, or (c) the output is going to a high-stakes downstream." The classifier cascade and the budget tier work together: classifiers gate cheaply, generative critic activates only on outputs that pass classifiers but are borderline on rubric score. For the round-table pattern (WP-08), the same logic applies: "production round-tables often include non-LLM judges alongside LLM critics. The classifiers handle the dimensions they were trained on; the LLM critics handle the nuanced dimensions; the orchestrator synthesizes." Production reality: HHEM-2.1 in front of an LLM-judge for faithfulness is the most common specialist-classifier-first deployment in RAG pipelines. It routes only borderline-faithfulness outputs to the expensive LLM-judge, cutting judge invocations by 60–80% in most production RAG systems.

10. What are the three families of round-table critic patterns from WP-08, and which has the strongest production fit for high-stakes output scoring?
    • A. There is only one round-table pattern: all critics debate until they reach a consensus score
    • B. The three families are: (1) Debate — N agents discuss until they converge on a verdict (ChatEval, Multi-Agent Debate); right when you genuinely need the system to find an answer through deliberation. (2) Role-based ensemble — each agent has a distinct role (advocate, adversary, gap-finder, auditor) running in parallel; an orchestrator synthesizes; produces the strongest audit trail; the strongest production fit for high-stakes scoring. (3) Layered refinement — agents organized in stages, each layer refining the previous (Mixture-of-Agents, MetaGPT); right for staged generation problems, not concurrent critique. The role-based ensemble is favored in production because it runs parallel (lower latency than debate), distributes reasoning across specialized perspectives, and produces a per-role audit trail required for compliance. ✓
    • C. The role-based ensemble is weakest because having an advocate role introduces bias toward accepting bad outputs
    • D. Debate is always the correct choice because it produces the highest-quality verdict through iterative refinement

    Option B correctly describes the three round-table families and their production fitness from WP-08 §1 and §2. Family 1 — Debate: N agents read the same output, each independently produces a critique, then they discuss across rounds until converging. ChatEval (Chan et al., 2023): agents with distinct personas debate across rounds until the orchestrator declares consensus. Multi-Agent Debate (MAD, Liang et al., 2023): divergent phase (each agent scores independently, no cross-exposure), then convergent phase (they see each other's positions and discuss). Peer Rank and Discussion (PRD, Li et al., 2024): rank-discuss-rerank. Right for: problems where the answer is genuinely unknown and deliberation is expected to find it. Failure mode: groupthink — agents anchor on the first-presented verdict and produce false consensus; cap debate rounds at 2–3. Family 2 — Role-based ensemble (WP-08's primary focus): each agent has a distinct role (advocate, adversary, gap-finder, auditor), runs in parallel, produces its own findings, and an orchestrator synthesizes. Canonical implementation: AutoGen GroupChat (the GroupChatManager is the orchestrator). Why it is the strongest production fit: (1) parallel execution — lower wall-clock latency than serial debate, (2) specialized perspective — each role sees different things, catching issues a single framing misses, (3) audit trail — the transcript of each role's findings is interpretable by human reviewers, required for compliance-sensitive workflows, (4) the advocate role is not bias toward acceptance — it identifies strengths so the orchestrator can weigh them appropriately, preventing over-rejection. Family 3 — Layered refinement: agents in staged layers, each refining the previous layer's output. Mixture-of-Agents (Together AI, 2024): Layer 1 produces N candidates, Layer 2 refines them, Layer 3 aggregates. MetaGPT: PM → Architect → Engineer → QA roles in a waterfall. Right for: staged generation where each layer adds a distinct refinement (not concurrent critique of the same output). WP-08 §7 self-check answer: role-based ensemble > debate for high-stakes content moderation because debate is expensive and prone to convergence on the wrong answer via groupthink, while layered refinement bakes in the previous layer's biases. Production reality: the OSS landscape has the runtime substrate (AutoGen, CrewAI, LangGraph) and the research patterns (ChatEval, MAD, MoA) but lacks a packaged end-to-end role-based scoring component with calibrated orchestration. This is the single largest gap in the OSS scoring landscape as of mid-2026.

Frequently asked questions

What is an adversarial critic and how is it different from a rubric scorer?

An adversarial critic is an agent whose only job is to find what is wrong with another agent's output — not to score it on a scale, not to confirm it is acceptable, but to attack it and produce a structured findings list. Each finding has a location in the output, a description of the problem, a severity (blocker/major/minor), a one-sentence explanation, and an optional suggested fix. A rubric scorer is quantitative: it assigns a number per dimension and determines accept/review/reject. The two are paired by design — the score is the gate, the critic findings are the fix signal. A rubric score tells you something is wrong; the adversarial critic tells you what to fix and where.

What are the three critic prompt flavors and when does each apply?

Generic ("find anything wrong"): fastest, most flexible, unstructured output; good for early-stage triage before a rubric exists. Rubric-driven ("attack each dimension"): the critic receives the rubric and produces structured JSON findings per dimension; preferred for production quality gates where regression tracking across dimensions is required. Constitutional/principled ("here are the rules to enforce"): the critic walks a fixed list of principles and reports violations; optimized for safety and policy compliance — Llama Guard implements this as a classifier against the MLCommons Hazard Taxonomy. Framing matters: "Is this good?" produces praise; "Find every way this could be wrong, assuming someone wrote it carelessly" produces useful findings.

Why is cross-model critique necessary to prevent sycophancy?

When a model evaluates its own outputs, it cannot detect errors it would make itself — the judge and generator share training data, pretraining biases, and hallucination patterns. Cross-model critique routes evaluation to a judge from a different provider or training lineage: a GPT-4o generator evaluated by a Claude judge, or a Gemini generator evaluated by a Llama-based judge. Three same-family critics with majority vote does not fix sycophancy because they share priors — it is one signal repeated three times. One cross-family critic adds genuine independence that three same-family critics cannot.

What are the four classic adversarial critic failure modes?

Sycophancy: the critic defaults to praise, especially when using the same model family as the generator. Fix: cross-model critique and adversarial prompt framing ("your job is to fail this output"). Hallucinated findings: the critic reports a problem that does not exist. Fix: require findings to quote the problematic span from the output. Mirror effect: the critic returns one finding per rubric dimension regardless of whether each dimension has a real issue. Fix: explicitly prompt it to return an empty array when a dimension is fine. Drift: a critic prompt that worked six months ago now misses obvious issues because the model or input distribution changed. Fix: re-validate against humans quarterly and pin the critic model version.

What is the advocate / adversary / gap-finder / orchestrator pattern and when does it outperform a single critic?

This role-based round-table pattern from WP-08 runs four agents in parallel: the Advocate argues for the output's strengths, the Adversary attacks every weakness, the Gap-finder identifies what the output did not address but should have, and the Auditor flags any of the other three roles' findings that are themselves hallucinated or exaggerated. An Orchestrator then synthesizes a final verdict with a written audit trail. It outperforms a single critic when: a single critic's blind spots are costing money; the output is genuinely contested; or compliance requires a documented review trail per output. Cost: 4–6× a single-critic pass — tier the spend by running it only on borderline cases or high-stakes outputs.

How do PyRIT and Garak differ, and how do they fit in a production eval pipeline?

Garak (NVIDIA, Apache 2.0) is a static probe scanner: it runs 100+ probes across known vulnerability categories (prompt injection, jailbreak, toxicity, DAN variants) against an LLM endpoint and reports which probes the model fails. Use it for broad pre-deployment coverage scanning and regression testing. PyRIT (Microsoft, Apache 2.0) is an adaptive red-teaming orchestrator: an attacker LLM iteratively refines jailbreak prompts across multiple turns until it elicits a harmful response or exhausts a turn budget. Use it to confirm that specific vulnerabilities found by Garak are dynamically exploitable. Practical workflow: run Garak first (2–4 hours) to identify weak categories, then use PyRIT to confirm those weaknesses are exploitable by an adaptive multi-turn attacker.

Why is HHEM-2.1 preferred over LLM-as-judge for hallucination detection?

HHEM-2.1 (Vectara, Apache 2.0) is a fine-tuned cross-encoder that scores per-sentence faithfulness: given (context, claim) it outputs a 0–1 grounding score. It is preferred over LLM-as-judge for hallucination because it is orders of magnitude faster and cheaper (cross-encoder inference vs generative LLM call), it is a cross-model judge trained independently of popular generators (no sycophancy bias), and it produces per-sentence scores identifying exactly which sentences are hallucinated rather than a holistic verdict. This makes it the correct specialist critic for the faithfulness dimension in a cascaded critic pipeline.

How do you validate an adversarial critic before using it as a production gate?

Collect a held-out set of 50 outputs rated by two expert human annotators per rubric dimension. Compute inter-rater agreement between the humans — if it is below 0.6, the rubric criteria are ambiguous (fix the rubric first). Run the critic on the same set. For scored dimensions, compute Spearman or Pearson correlation between critic scores and human scores — threshold is 0.7+ per dimension. For findings, compute precision (fraction of critic findings humans confirm as real) and recall (fraction of human-identified issues the critic surfaced). A critic with below-70% agreement against human judgment is not ready for a production gate — iterate the prompt before scaling. Re-validate quarterly to catch critic drift.

What is the specialist-classifier cascade and what does it save?

The cascade runs cheap, narrow specialist classifiers (Tier-2) before the generative LLM critic for dimensions they were trained to measure: HHEM-2.1 for faithfulness, Llama Guard for safety, Detoxify for toxicity, Microsoft Presidio for PII. A toxicity classifier is faster and cheaper than asking an LLM "is this toxic?" The generative critic is invoked only for dimensions a classifier cannot address (tone nuance, logical coherence, domain-specific reasoning). This is the hybrid tier pattern from WP-02 applied to critique. In practice it cuts generative LLM-critic invocations by 60–80% in RAG pipelines, since HHEM-2.1 screens faithfulness cheaply and only borderline cases escalate.

What are the three round-table families and which is preferred for high-stakes scoring?

Debate (ChatEval, Multi-Agent Debate): N agents discuss until converging on a verdict; right when the answer is genuinely unknown but prone to groupthink if agents anchor on the first-presented position. Role-based ensemble (advocate/adversary/gap-finder/orchestrator): agents run in parallel with distinct roles; an orchestrator synthesizes; produces the strongest audit trail and is the standard production pattern for high-stakes output scoring. Layered refinement (Mixture-of-Agents, MetaGPT): agents staged in layers each refining the previous; right for staged generation, not concurrent critique. The role-based ensemble is preferred because it runs in parallel (lower latency than debate), distributes reasoning across specialized perspectives, and produces a per-role transcript required for compliance-sensitive workflows.

Related topics