Machine Learning Engineer Interview Questions and Strong Answers

Explain the bias-variance tradeoff and how it guides model choice.

Expected error decomposes into bias (systematic error from a model too simple to capture the signal), variance (sensitivity to the particular training sample), and irreducible noise. High bias shows up as underfitting — train and validation error both high and close. High variance shows up as overfitting — low train error, much higher validation error. The tradeoff guides you operationally: a large validation gap means add regularization, more data, or a simpler model; train and validation both poor means add capacity or better features. Modern over-parameterized networks complicate the textbook U-curve (double descent), but the diagnostic discipline — compare train versus validation, then act on the gap — still holds.

How do you detect and prevent data leakage?

Leakage is any information in training that will not be available — honestly — at prediction time, and it produces offline metrics that collapse in production. Classic sources: target leakage (a feature that is a proxy for the label, like "account_closed_date" predicting churn), temporal leakage (using future information for a past prediction), and preprocessing leakage (fitting a scaler, imputer, or vectorizer on the full dataset before splitting, so test statistics bleed into training). Prevent it by splitting first and fitting every transform inside a pipeline on the training fold only; using time-based splits for any temporal problem; and treating a suspiciously high score as a leakage hunt, not a success. The tell is a feature with implausibly high importance — audit it before you trust it.

Which evaluation metric do you choose, and why is accuracy often misleading?

Choose the metric from the cost of each error type, not by default. Accuracy is misleading under class imbalance — a 99%-negative problem scores 99% by always predicting negative while catching nothing. For imbalanced or ranking problems, precision and recall expose the real tradeoff, and PR-AUC reflects positive-class performance better than ROC-AUC because ROC-AUC can look strong even when precision is poor (the false-positive rate has a huge negative denominator). Pick the operating point from the asymmetry: optimize recall when a miss is expensive (fraud, disease screening), precision when a false alarm is expensive. Above all, the offline metric should correlate with the business or product outcome — otherwise you are optimizing the wrong thing precisely.

Your training loss keeps dropping but validation loss starts rising. What is happening and what do you do?

That divergence is the signature of overfitting: the model is memorizing training-set specifics that do not generalize. The first move is regularization and early stopping — stop at the validation minimum, add weight decay or dropout, or reduce capacity. Then attack the data: more or more-diverse training data, and augmentation where it is valid. Re-check for leakage in reverse — a too-good train fit with poor validation can also signal that the validation set is genuinely different (distribution shift), so confirm the split is representative. Only after those do you reconsider architecture. Order matters: cheap regularization and data fixes before expensive model surgery.

How do you handle class imbalance?

First decide whether it even needs handling — if the metric and threshold are chosen correctly, many models handle moderate imbalance fine. When it does bite, the levers are: resampling (oversample the minority, e.g. SMOTE, or undersample the majority), class weights in the loss to penalize minority errors more, and threshold tuning on the probability output rather than defaulting to 0.5. Evaluate with precision/recall and PR-AUC, never raw accuracy. The subtle trap: resampling changes the base rate, so predicted probabilities are no longer calibrated to the real-world prior — recalibrate (e.g. Platt scaling or isotonic regression) if you need probabilities, not just rankings.

Why use batch normalization, and how does layer normalization differ?

Batch norm normalizes activations across the batch dimension per feature, which stabilizes and speeds training by reducing internal covariate shift and smoothing the loss surface. Its weakness is the batch dependency: it behaves differently at train versus inference (it switches to running statistics), degrades with very small batches, and is awkward for sequence models. Layer norm normalizes across the feature dimension within a single example, so it is batch-independent — which is why transformers use it: it works with variable-length sequences and tiny or size-one batches at inference. The general rule interviewers want: batch norm for CNN-style fixed-feature vision workloads with healthy batch sizes, layer norm for sequence and transformer architectures.

When do you choose retrieval-augmented generation over fine-tuning an LLM?

Match the method to the gap. Retrieval (see /topics/rag-retrieval-augmented-generation) fixes a knowledge gap: the model lacks facts that are private, fresh, or too large to memorize. It keeps the knowledge base updatable without retraining, grounds answers in citable sources, and reduces hallucination on factual queries. Fine-tuning fixes a behavior gap: you need a consistent format, tone, domain style, or a narrow task the base model does not follow reliably — things prompting alone cannot stabilize. They are not exclusive; a common production pattern is fine-tune for form plus retrieve for facts. The interview red flag is proposing fine-tuning to inject knowledge that changes weekly — that is a retrieval problem, and fine-tuning would bake in stale facts and still hallucinate.

How do embeddings and vector search work, and what breaks at scale?

An embedding maps text (or images, audio) into a dense vector where semantic similarity is geometric proximity, usually measured by cosine similarity. Retrieval embeds the query and finds nearest neighbors in the corpus (see /topics/vector-databases-and-embeddings). Exact nearest-neighbor search is linear in corpus size, so at scale you use approximate nearest neighbor indexes (HNSW, IVF) that trade a little recall for large speedups. What breaks: embedding-model mismatch between indexing and querying silently destroys relevance; chunking strategy dominates quality (too large dilutes the signal, too small loses context); and pure vector recall misses exact-match and keyword cases, which is why hybrid search (dense plus BM25/keyword) and a re-ranking stage usually beat vectors alone. Also, embeddings drift — re-embedding the whole corpus on a model upgrade is a real operational cost.

How do you evaluate an LLM system where there is no single correct answer?

You replace exact-match with a layered evaluation (see /topics/llm-evaluation-best-practices). Build a reference set of representative inputs with rubric-based expectations rather than gold strings. Use task-appropriate automatic signals — exact match or unit tests where output is checkable, retrieval metrics (is the right source retrieved), and faithfulness/groundedness checks for RAG. Add an LLM-as-judge with a clear rubric for fluent, open-ended output, but validate the judge against human labels and watch its biases (position, verbosity, self-preference). Keep a human-eval slice for the highest-stakes cases. The discipline interviewers reward: define what "good" means before you build, track regressions on a fixed set, and never ship on vibes from a few hand-picked prompts.

How do you serve a model under a strict latency budget?

Start by measuring the tail, not the mean — p95/p99 is what users feel and what SLAs are written against. Then optimize the model: quantization (FP16/INT8) and distillation to a smaller student, plus an export/runtime path (ONNX, TensorRT) that fuses ops for the target hardware. At the serving layer, dynamic batching trades a little latency for large throughput gains, and caching helps for repeated inputs. Architecturally, decide what runs where — on-device or edge for the strictest latency, server for the heaviest models — and whether a cheaper model can handle the common case with escalation to a larger one (a cascade). The framing interviewers want: latency, throughput, accuracy, and cost are a joint budget; you trade explicitly among them, you do not maximize one in isolation.

How do you detect and respond to model drift in production?

Distinguish the two. Data (covariate) drift is the input distribution moving — detectable without labels by monitoring feature distributions (e.g. population stability index, KS tests) and prediction distributions. Concept drift is the input-output relationship changing — only confirmable once labels arrive, by tracking the live metric against its baseline. Build the monitoring before you need it: log inputs, predictions, and (when available) outcomes; alert on distribution shifts and metric degradation. Respond proportionally — investigate the cause first (a broken upstream feature pipeline mimics drift and is far more common), then retrain on recent data, adjust thresholds, or roll back. Scheduled retraining is a backstop, not a substitute for monitoring; you want a trigger tied to measured degradation, plus a canary or shadow deployment to validate the new model before it takes full traffic.