ML EngineerInterview Questions & Practice

Q1. How do you tell whether a model is underfitting or overfitting, and what do you change first?

Strong answer: Compare training error against validation error. Both high means high bias (underfitting) — add capacity or features, train longer, or relax regularization. A low training error with a large validation gap means high variance (overfitting) — more data, stronger regularization (weight decay, dropout), or early stopping. A strong answer reaches for the cheapest lever first (early stopping, regularization) before adding data or parameters, and never tunes on the test set.

Weak answer: A weak answer reflexively reaches for a bigger model, or reports a number tuned on the test set so the gap is hidden rather than diagnosed.

Q2. When do you reach for layer normalization instead of batch normalization?

Strong answer: Batch norm normalizes each feature using statistics computed over the mini-batch, so it depends on batch statistics — it degrades with small or variable batches and is awkward for variable-length sequences and autoregressive inference, where the running statistics no longer match. Layer norm normalizes across features within each example, independent of batch size, which is why transformers (see /topics/transformer-architecture) use it. Reach for layer norm on sequence and attention models and small/variable batches; batch norm still wins in CNNs with large batches.

Weak answer: A weak answer treats the two as interchangeable normalization tricks and cannot say which dimension each normalizes over or why sequence models avoid batch norm.

Q3. Training loss spikes to NaN a few hundred steps in. How do you isolate the cause?

Strong answer: The usual causes are a learning rate too high for the post-warmup scale, numerical overflow in mixed precision, or bad data. Be forensic: locate the first non-finite tensor with finite checks (or an anomaly-detection hook) on the inputs, loss, activations, and gradients — that localizes overflow versus corrupt data versus an unstable op before touching hyperparameters. Then treat the cause: extend warmup and clip gradients for optimization instability, check for fp16 overflow to Inf (the dynamic loss scaler should catch it and skip the step) in mixed precision, and look for exploding activations from bad initialization or a missing normalization layer. Lowering the learning rate confirms instability but does not substitute for finding the first NaN.

Weak answer: A weak answer randomly lowers the learning rate without separating overflow from instability from corrupt data, so the same divergence returns under a different config.

Q4. When do you move beyond plain data-parallel training, and to what?

Strong answer: Data parallel replicates the whole model on every device and all-reduces gradients — it holds until the model plus optimizer states no longer fit in one device. Then shard in ZeRO stages (optimizer state, then gradients, then parameters); if a single layer still does not fit, tensor/model parallelism splits that layer across devices; if the stack of layers does not fit or device utilization is poor, pipeline parallelism splits layers across stages — each trades communication overhead for memory. A strong answer first buys activation-memory headroom with gradient checkpointing and mixed precision before paying the parallelism communication cost.

Weak answer: A weak answer jumps to model parallelism while the model still fits in memory, paying communication overhead that optimizer sharding would have avoided.

Q5. Design the serving path for a real-time recommender under a tight latency budget. Where does the budget go?

Strong answer: Split the problem across stages: precompute item embeddings and build the vector index offline (user embeddings too, but refresh them online when recent behavior matters), then run approximate-nearest-neighbor retrieval (see /topics/vector-databases-and-embeddings) and a light ranking model online per request. The budget goes to feature fetch (feature store), online retrieval, and ranker inference — so cache and batch aggressively, quantize the ranker, and precompute what you can. A strong answer names which stages are offline versus online, states where the milliseconds actually go, and includes monitoring plus a retraining trigger.

Weak answer: A weak answer runs one large model synchronously for every request and ignores feature-fetch latency, so the design cannot meet the budget at production scale.

Q6. How do you know a deployed model has degraded, and what should trigger a retrain?

Strong answer: Instrument two things: input drift (feature distributions shifting away from training) and performance drift (calibration now, and label-based metrics once ground truth arrives). A strong answer distinguishes covariate shift (inputs move) from concept drift (the input-to-label mapping moves) because they demand different responses, and makes the retrain trigger threshold-based on those signals rather than a purely fixed calendar (while still allowing scheduled retrains for seasonality or compliance) — and accounts for label latency that hides concept drift until it is expensive.

Weak answer: A weak answer retrains on a fixed schedule with no drift signal, or watches only accuracy and misses that labels arrive late so degradation is caught weeks after it started.

Q7. A stakeholder wants the model to "know" a set of internal company documents. RAG or fine-tuning?

Strong answer: Default to retrieval (see /topics/rag-retrieval-augmented-generation) when the knowledge is factual, changes often, or needs citations: index the documents and retrieve at inference, so an update is a re-index rather than a re-train. Fine-tune to change behavior, format, or a skill the base model lacks — not to inject volatile facts. A strong answer maps the requirement (facts vs behavior, freshness, attribution) to the mechanism and often uses both — see /topics/llm-fine-tuning-vs-rag-decision-guide.

Weak answer: A weak answer fine-tunes to add facts, then cannot update them without another training run and produces no citations, which is exactly what retrieval was for.

Q8. How do you evaluate an LLM feature that has no single correct answer?

Strong answer: Treat it as a rubric-and-dataset problem, not exact match: define task-specific criteria, build a labeled reference set, and combine graded human review with an LLM-as-judge calibrated against those human labels — reporting held-out scores, not in-sample ones. Add guardrail checks such as faithfulness or groundedness for retrieval features, and track regressions across versions. A strong answer refuses to trust a single accuracy number or an uncalibrated judge — see /topics/llm-evaluation-best-practices.

Weak answer: A weak answer eyeballs a handful of outputs or treats an uncalibrated judge model as ground truth, so the metric drifts with the judge and cannot catch regressions.