Numerical Methods Interview Prep
Numerical methods interview prep — floating-point, condition number, linear systems, Newton, quadrature, ODE/PDE solvers, and the truncation-vs-roundoff tradeoff.
Quick answer
Numerical Methods is the study of algorithms for solving continuous mathematical problems on finite-precision computers.
Numerical methods questions test whether candidates understand the gap between mathematical exactness and computational reality.
Editorial review
Written by
CompoundLearn editorial team
Wireless / RF / hardware engineering
Reviewed by
CompoundLearn editorial team
Wireless / RF / hardware engineering
Last reviewed
Built from curated topic maps, editorial validation, and subject-matter review so the page stays aligned with the interview intent and the current content pipeline.
What it is
Numerical Methods is the study of algorithms for solving continuous mathematical problems on finite-precision computers. Standard topics include floating-point representation (IEEE 754, machine epsilon, denormals, NaN/Inf), error analysis (truncation vs round-off, forward and backward stability, condition number), root-finding (bisection, secant, Newton, Brent, with convergence rates), unconstrained optimization (gradient descent, Newton, quasi-Newton like BFGS, line search and trust region), constrained optimization (Lagrangian, KKT, interior-point methods, SQP), linear-system solvers (LU with partial pivoting, Cholesky, QR, SVD), iterative methods (Jacobi, Gauss-Seidel, conjugate gradient, GMRES, preconditioning), eigenvalue algorithms (power iteration, QR algorithm, Lanczos, Arnoldi), polynomial interpolation and the Runge phenomenon, splines, numerical integration (Newton-Cotes, Gauss quadrature, adaptive quadrature, Romberg), numerical solution of ODEs (Euler, RK4, adaptive methods, stiff solvers like BDF), and an introduction to numerical PDE methods (finite difference, finite element, spectral methods). For engineering candidates, numerical methods is the layer between mathematical models and code: every controller, every signal-processing pipeline, every ML training loop, every simulation calls into numerical algorithms whose properties (stability, convergence rate, condition-number sensitivity) determine whether the system actually works.
Why interviewers ask
Numerical methods questions test whether candidates understand the gap between mathematical exactness and computational reality. Many engineers can write down the normal equations for least-squares but do not know that solving them produces half the accuracy of QR — and that gap shows up as a real bug at scale. ML interviews probe gradient-descent convergence rates, why Adam works on non-stationary objectives, when stochastic gradient outperforms batch (and vice versa), and the role of conditioning in optimization. Controls and simulation interviews test ODE solver selection (explicit vs implicit, stiff vs non-stiff, fixed-step vs adaptive) and CFL conditions for PDE-based simulations. Numerical-software interviews probe linear-system solver selection (LU vs Cholesky vs QR vs SVD), iterative methods and their preconditioning requirements, and condition-number arguments. The strongest candidates know which algorithm is appropriate for each problem, articulate the tradeoffs, and quote convergence rates and per-iteration costs. Interviewers especially probe candidates who claim ML or systems experience but cannot explain mixed-precision training, why Newton diverges on a non-convex loss, or what happens when a Cholesky factorization encounters a near-zero pivot. Numerical maturity is rare and senior interviewers reward it disproportionately.
Common mistakes
The most common mistake is treating floating-point arithmetic as if it were exact — candidates compare floats with == instead of with a tolerance, or accumulate sums in the wrong order and lose precision (Kahan summation exists exactly because naive summation accumulates O(n·eps) error per term). A second mistake is solving least-squares via the normal equations (x = (A^T A)^(-1) A^T b) without realizing this squares the condition number; QR or SVD is preferred numerically. A third mistake is using explicit time integration on stiff ODEs and reporting that the simulation "explodes"; the right answer is an implicit method or BDF. A fourth mistake is assuming Newton's method converges from any starting point — it can diverge, oscillate, or jump to a wrong root, and globalization (line search, trust region) is mandatory for robustness. A fifth mistake is using high-degree polynomial interpolation on equispaced points (Runge phenomenon — error grows at the interval ends); Chebyshev nodes or splines avoid this. A sixth mistake is conflating forward and backward stability — backward stable algorithms produce the exact answer to a slightly perturbed problem, which is the right notion for ill-conditioned problems where forward error is unavoidably large. A seventh mistake is preconditioning iterative solvers as an afterthought — for ill-conditioned systems, preconditioning is the entire game — conjugate gradient without preconditioning needs O(√κ) iterations (κ = condition number), and ill-conditioned nonsymmetric Krylov methods (GMRES) can stagnate entirely. An eighth mistake is mishandling step-size selection: too-large h gives big truncation error, too-small h gives round-off-dominated noise, and the optimum is at the crossover, which depends on machine precision and the algorithm order. Finally, candidates underestimate the role of structure — sparse matrices, banded matrices, Toeplitz matrices, and circulant matrices each admit specialized algorithms with much lower complexity than dense methods, and exploiting structure is often the difference between a feasible and infeasible computation.
Frequently asked questions
- What is the difference between truncation error and round-off error?
- Truncation error comes from approximating a mathematical operation with a finite-step algorithm — Taylor series remainder when truncated to k terms, or step-size discretization in ODE solvers. It scales with the algorithm parameter (h^k for an order-k method). Round-off error comes from finite-precision floating point — every operation can lose bits, and these errors accumulate. Truncation shrinks as you take more steps; round-off grows. The optimal step size minimizes the sum, and below that the round-off term dominates.
- What is the condition number, and why does it matter?
- The condition number of a matrix κ(A) = ||A|| · ||A^(-1)|| measures how much the solution of Ax = b can change when b is perturbed: relative output error ≤ κ(A) · relative input error. A condition number of 10^6 means up to 6 digits of accuracy can be lost. Ill-conditioned systems (κ near 1/machine epsilon) can produce nonsense even with exact arithmetic on the problem. The condition number of A^T A is κ(A)^2 — which is exactly why solving least-squares via the normal equations is half as accurate as via QR or SVD.
- How do you choose between LU, Cholesky, QR, and SVD for solving linear systems?
- LU (Gaussian elimination with partial pivoting) is the general-purpose square-system solver — O(n^3) flops, works for any non-singular matrix. Cholesky factorizes symmetric positive-definite matrices as L·L^T at half the cost — preferred for normal equations and Newton systems. QR is the numerically-stable choice for least-squares — preserves condition number, unlike normal equations. SVD is the most robust but costliest — handles rank-deficient matrices via the pseudo-inverse and gives explicit information about singular values for regularization decisions.
- When does Newton's method converge, and what is the rate?
- For a single equation f(x) = 0, Newton's iteration x_{n+1} = x_n - f(x_n)/f'(x_n) converges quadratically (error roughly squares each step) when started close enough to a simple root and f is smooth. For systems, Newton uses the Jacobian: x_{n+1} = x_n - J^(-1) · f(x_n), again quadratic locally. The cost per step is high (Jacobian solve), so quasi-Newton methods (Broyden, BFGS) approximate the Jacobian update from secant information. Far from the root, Newton can diverge or oscillate — globalization (line search, trust region) is required for robustness.
- How do you choose an integration rule (trapezoid, Simpson, Gauss)?
- Trapezoid: O(h^2) error, integrates linear functions exactly. Simpson: O(h^4), exact for cubics, requires uniform spacing in the simple form. Gaussian quadrature: O(h^(2n)) for n-point rule, exact for polynomials of degree 2n - 1, and chooses both nodes and weights to maximize precision — but the nodes are non-uniform (zeros of orthogonal polynomials). Adaptive quadrature (QUADPACK, GSL) recursively subdivides intervals where the integrand is rough and is the right tool for production use. For oscillatory integrands, specialized methods (Filon, Levin) avoid the catastrophic cancellation of standard quadrature.
- What is iterative refinement and when is it useful?
- After computing an approximate solution x̃ to A·x = b via direct factorization, compute the residual r = b - A·x̃ in higher precision, solve A·d = r for the correction, and update x̃ = x̃ + d. The corrected solution can recover much of the accuracy lost to round-off, especially when the factorization was performed in lower precision (mixed-precision iterative refinement is the foundation of modern GPU-accelerated linear solvers using FP16 factor + FP32 refinement to deliver FP32-level accuracy at FP16 speed).
Related topics
Essential AI-Native Skills for Numerical Methods
Modern engineering work increasingly uses AI tools for design and code review, debugging, documentation, test and testbench generation, and workflow automation. The goal is not to let AI replace engineering judgment — it is to move faster while keeping verification discipline.
- Use AI to explain unfamiliar code, logs, waveforms, datasheets, or test failures.
- Break large problems into small, reviewable steps you can verify independently.
- Ask AI for hypotheses, then validate them against tests, measurements, simulations, or lab data.
- Version-control your analysis scripts, testbenches, and configs — keep changes small and reviewable.
- Document your assumptions, design tradeoffs, and debugging decisions.
- Verify AI output before trusting it: run the checks that fit the domain — unit tests, linters, simulations, or bench/lab measurements.
- Review AI output for correctness, edge cases, and real-world consequences.
Numerical Methods — coming to the question bank
The adaptive practice engine is already live for core wireless, RF, and ML systems. Numerical Methods isn't covered in the question bank yet — get notified when it's added.