AI Engineer Interview Questions and Strong Answers

How do you reduce hallucination in an LLM application?

Hallucination is the model producing fluent, confident output unsupported by fact — it is intrinsic to how LLMs generate, so you contain it rather than eliminate it. The highest-leverage move is grounding: retrieval-augmented generation supplies the relevant source text in context and you instruct the model to answer only from it, then surface citations so a wrong answer is traceable. Layer on constraints: lower temperature for factual tasks, allow "I don't know," and validate output against the retrieved context with a faithfulness scorer — RAGAS faithfulness (an LLM judge that decomposes the answer into atomic claims and checks each against the retrieved text) or a purpose-trained detector like Vectara's HHEM-2.1, gating below a calibrated threshold (often ~0.5–0.7). For high stakes, add a verification pass or human review. The interview tell is treating hallucination as a bug to be patched versus a property to be bounded with grounding, constraints, and evaluation.

When do you use RAG, fine-tuning, or just better prompting?

Match the method to the gap. Prompting (including few-shot examples and clearer instructions) is the first lever — cheap, instant, and often enough. Retrieval (see /topics/rag-retrieval-augmented-generation) fixes a knowledge gap: facts that are private, fresh, or too large to fit in context; it stays updatable without retraining and grounds answers in citable sources. Fine-tuning fixes a behavior gap: a consistent format, tone, or narrow skill the base model will not follow reliably through prompting alone — and modern fine-tuning is parameter-efficient (e.g. LoRA), not full retraining. They compose: fine-tune for form, retrieve for facts. The red flag interviewers listen for is fine-tuning to inject knowledge that changes weekly — that bakes in stale facts and still hallucinates; it is a retrieval problem.

How do you design reliable tool calling for an agent?

Treat each tool call as an unreliable external request. Define tools with strict, typed schemas so the model returns validated arguments, and validate before executing. Constrain the loop: a step budget to prevent runaway iteration, explicit error handling so a failed call returns a message the model can recover from rather than crashing the chain, and idempotency or confirmation for destructive actions. Keep the action space small and observable — log every call, argument, and result — because multi-step agents compound errors, so a 90%-reliable step is far worse over ten steps than it sounds. Interleaving reasoning with acting (the ReAct pattern) helps the model decide and recover, but the reliability comes from the schema, the limits, and the observability around it, not from the prompt alone.

How do you get reliable structured (JSON) output from an LLM?

Do not parse free text with brittle string logic. Use the strongest constraint the platform offers: a schema-enforced mode like OpenAI Structured Outputs (which constrains decoding to a supplied JSON Schema), function calling with a typed schema, or a constrained-decoding library such as Outlines for open models — so the model can only emit schema-conforming tokens. Define the schema explicitly (see /topics/llm-structured-output-extraction), validate the result against it, and handle the failure path — retry with the validation error fed back, or fall back to a stricter mode. Keep the schema as simple as the task allows, since deeply nested or ambiguous schemas raise the failure rate. The principle: make malformed output structurally impossible where you can, and validate-and-retry where you cannot.

How do you evaluate an LLM system where there is no single correct answer?

Replace exact-match with a layered evaluation (see /topics/llm-evaluation-best-practices). Build a fixed reference set of representative inputs with rubric-based expectations. Use task-appropriate automatic signals — unit tests or exact match where output is checkable, retrieval metrics (did the right source get retrieved), and faithfulness/groundedness scorers (RAGAS, HHEM-2.1) for RAG. Add an LLM-as-judge with a clear rubric for fluent open-ended output, but validate the judge against human labels and watch its biases (position, verbosity, self-preference). Keep a human slice for the highest-stakes cases. The discipline interviewers reward: define "good" before building, track regressions on a fixed set, and never ship on a handful of cherry-picked prompts.

How do you defend against prompt injection?

Prompt injection — OWASP's LLM01, the top-ranked LLM-application risk — is untrusted text (a user message, a retrieved document, a web page) carrying instructions the model then follows, because the LLM cannot reliably tell data from command. There is no single fix, so you defend in depth (see /topics/llm-guardrails-and-safety): treat all external content as untrusted, separate it structurally from system instructions, and never let the model's raw output trigger a privileged action without validation. Apply least privilege to tools (scoped, read-mostly, human confirmation for destructive or exfiltrating actions), sanitize and constrain outputs, and add input/output filters for known attack and data-leak patterns. The senior framing: assume injection will happen and limit the blast radius, rather than trying to prompt your way to a model that always refuses.

How do you control latency and cost in an LLM application?

Both scale with tokens, so manage context first — retrieve only what is needed, trim history, and cache. Measure the tail (p95/p99) and stream tokens so perceived latency drops even when total time is fixed. Route by difficulty: a smaller, cheaper model handles the common case with escalation to a larger one only when needed (a cascade), and prompt/response caching (see /topics/llm-routing-and-caching) removes repeat work. Batch where throughput matters. For agents, the step budget is also a cost budget. The framing interviewers want: latency, cost, and quality are one joint budget you trade explicitly — "use the biggest model everywhere" is almost never the right point on that curve.

What is chain-of-thought prompting, and when does it help or hurt?

Chain-of-thought asks the model to produce intermediate reasoning steps before its answer, which improves accuracy on multi-step reasoning (math, logic, planning) by giving the model space to work. It helps when the task genuinely needs decomposition. It hurts when it does not: it adds latency and token cost, can rationalize a wrong answer into a confident one, and exposes reasoning you may not want in a user-facing response. Practical handling: reserve it for hard reasoning steps, keep the visible reasoning internal when only the answer should ship, and remember that a reasoning-tuned model may already do this internally, so forcing it can be redundant. Match the technique to task difficulty rather than applying it everywhere.

How does context-window management affect RAG quality?

More context is not automatically better. Chunking dominates retrieval quality — chunks too large dilute the relevant signal and waste tokens, too small lose the context needed to answer. Even with a large window, models attend unevenly across long context (relevant content buried in the middle is often used worse than content at the edges), so packing in everything degrades answers and raises cost. Better: retrieve precisely, re-rank to put the most relevant passages first, and pass a tight, ordered set rather than a large dump. Hybrid retrieval (dense plus keyword) and a re-ranking stage usually beat raw vector recall (see /topics/vector-databases-and-embeddings). The point is relevance density, not volume.

Why do instruction-following and RLHF matter for how you prompt a model?

Base language models predict text; they do not inherently follow instructions. Instruction tuning plus reinforcement learning from human feedback (RLHF) is what aligns a model to follow directions and prefer helpful, honest, harmless responses — it is why a modern assistant responds to "summarize this" rather than continuing the text. For an AI engineer this shapes prompting: clear, explicit instructions and examples work because the model was optimized to follow them, but the same alignment introduces tendencies (over-refusal, verbosity, sycophancy toward the user's framing) you design around. Understanding that the model's helpfulness is a trained behavior, not a guarantee, is what separates prompt engineering from wishful prompting.

How do you decide between a larger model and a smaller, cheaper one?

Start from the task and the metric, not the model. Establish the quality bar on your evaluation set, then find the cheapest model that clears it for each subtask — many production paths use a small model for routing, extraction, and classification and reserve a frontier model for the genuinely hard generation. A cascade (cheap model first, escalate on low confidence or failed validation) captures most of the cost savings while protecting quality. Factor in latency and rate limits, not just per-token price. The interview signal is treating model choice as a measured cost-quality-latency tradeoff per task, rather than defaulting to the largest model everywhere.