Edge AI & Model Quantization: Running ML on Constrained Devices Interview Prep
Edge AI and model quantization explained for embedded engineers — INT8 vs FP32, PTQ vs QAT, per-channel and static quantization, pruning and distillation, and the memory/latency/accuracy tradeoff on MCUs and NPUs.
Quick answer
Edge AI is the practice of running machine-learning inference on resource-constrained devices — microcontrollers, DSPs, NPUs, and edge SoCs — rather than in the cloud, and model quantization is the technique that most often makes it feasible.
Edge-AI questions separate candidates who only know how to train a model from those who can deploy one under hard constraints.
Editorial review
Written by
CompoundLearn editorial team
Wireless / RF / hardware engineering
Reviewed by
CompoundLearn editorial team
Wireless / RF / hardware engineering
Last reviewed
Built from curated topic maps, editorial validation, and subject-matter review so the page stays aligned with the interview intent and the current content pipeline.
Key points
- Edge AI runs inference on-device (MCU, NPU, SoC) for latency, privacy, offline operation, and cost — trading a datacenter GPU for a tight memory, latency, and energy budget.
- Quantization cuts numerical precision (FP32 -> INT8): ~4x smaller and faster, more energy-efficient integer math, at the cost of quantization error — the central edge-ML technique.
- PTQ quantizes a trained model with a calibration set (fast, some accuracy loss); QAT simulates quantization during training (better low-precision accuracy, needs retraining). Try PTQ first, escalate to QAT if accuracy drops too far.
- Per-channel weight quantization recovers accuracy for conv layers; static activation quantization (calibrated) gives fully-integer inference that edge accelerators want.
- Pruning, knowledge distillation, and efficient architectures (depthwise-separable convs) are complementary levers — combine them, measuring accuracy against the budget at each step.
- Memory is usually the binding constraint, especially on microcontrollers (weights in flash, activation arena in SRAM): a model that does not fit cannot run, so footprint matters more than raw FLOPs.
- The interview signal is reasoning about the accuracy-versus-footprint tradeoff with real numbers, not naming a framework.
What it is
Edge AI is the practice of running machine-learning inference on resource-constrained devices — microcontrollers, DSPs, NPUs, and edge SoCs — rather than in the cloud, and model quantization is the technique that most often makes it feasible. The motivation for moving inference to the edge is concrete: lower latency (no network round trip), privacy (raw data stays on the device), offline operation, and avoiding bandwidth and cloud-compute cost. The challenge is that the target has orders of magnitude less memory, compute, and power than the GPU the model was trained on, so a model that is accurate in the lab may simply not fit or not run fast enough. Quantization addresses this by lowering numerical precision — typically from 32-bit floating point to 8-bit integers — which shrinks the model roughly fourfold, reduces memory bandwidth, and replaces costly floating-point operations with efficient integer math that microcontrollers and NPUs execute well (see /topics/machine-learning-fundamentals-for-engineers for the model side). The unavoidable tradeoff is accuracy: representing a wide range of values with 256 integer levels introduces error, so the work is keeping that error inside the application's budget. Post-training quantization (PTQ) does this with a calibration dataset and no retraining; quantization-aware training (QAT) bakes the rounding into training for better low-precision accuracy. Quantization sits alongside pruning, knowledge distillation, and efficient architectures as the toolkit for fitting a model to the edge — and the binding constraint, especially on microcontrollers, is usually memory: weights in flash and the activation arena in SRAM.
Why interviewers ask
Edge-AI questions separate candidates who only know how to train a model from those who can deploy one under hard constraints. Training on a GPU is forgiving; getting a model onto a microcontroller with tens of kilobytes of SRAM, a fixed latency target, and a power budget is where real engineering judgment shows. Interviewers for embedded-ML, TinyML, and edge-deployment roles probe whether you reason about the deployment budget — memory for weights and activations, latency per inference, energy — and whether you understand quantization beyond the buzzword: why INT8, what a calibration set does, when PTQ is enough and when you must move to QAT, and what per-channel and static quantization buy you. The topic is also a systems-integration probe, because it sits exactly between two disciplines (see /topics/embedded-engineer-interview-signals). It pulls in model concepts on one side and embedded realities on the other: memory maps, real-time scheduling, datasheet reading for an accelerator's supported operators and data types. Strong candidates quantify the tradeoff — "INT8 PTQ cut the model 4x and cost two points of accuracy, which was within budget; if it had not, QAT was the next step" — and know that footprint, not FLOPs, is what usually decides whether a model runs at all. Weak candidates treat edge deployment as just exporting the model, with no model of where it breaks.
Common mistakes
The most common mistake is treating edge deployment as cloud ML with a smaller model — ignoring that memory, not compute, is usually the binding constraint, and that a model which does not fit in SRAM cannot run no matter how fast the core is. A second mistake is conflating PTQ and QAT, or not knowing that post-training quantization needs a representative calibration dataset to set activation ranges; skipping or mis-choosing calibration data is a frequent cause of unexpected accuracy loss. A third is assuming quantization is free — INT8 introduces real error, and aggressive formats like INT4 can degrade accuracy sharply without quantization-aware training to compensate. A fourth is forgetting per-channel weight quantization and static activation quantization, then being surprised that a naive per-tensor dynamic scheme is both less accurate and slower than the hardware could deliver. A fifth is over-indexing on FLOPs while ignoring the activation memory (the tensor arena) and operator support on the target accelerator — a model can be small in parameters but still overflow SRAM at its widest layer, or use an operator the runtime cannot accelerate. Finally, candidates often reach for a specific framework as the answer when the real skill is reasoning about the accuracy-versus-footprint-versus-latency tradeoff for the specific target.
Frequently asked questions
- What is edge AI, and how is it different from cloud ML?
- Edge AI runs model inference on or near the device that captures the data — a microcontroller, an NPU-equipped SoC, a phone, a sensor — instead of sending data to a server. The motivations are latency (no round trip), privacy (data never leaves the device), offline operation, and cost (no bandwidth or cloud-compute bill). The constraint is that you trade a datacenter GPU for a target with kilobytes-to-megabytes of memory, a modest clock, and a tight power budget. So edge AI is less about training bigger models and more about making a trained model small and fast enough to run within a hard memory, latency, and energy envelope — which is why quantization and related compression techniques are central.
- What is model quantization and why does it help on edge devices?
- Quantization reduces the numerical precision of a model's weights and activations — most commonly from 32-bit floating point (FP32) to 8-bit integers (INT8). That gives roughly a 4x reduction in model size and memory bandwidth, and integer math is faster and far more energy-efficient than floating point on most embedded targets (many microcontrollers have weak or no FPU but efficient integer units, and NPUs are built around INT8). The cost is precision: mapping a wide float range onto 256 integer levels introduces quantization error, so the engineering question is how much accuracy you lose and whether it stays within budget. More aggressive formats (INT4, or binary/ternary) shrink further at greater accuracy risk.
- What is the difference between post-training quantization (PTQ) and quantization-aware training (QAT)?
- PTQ quantizes an already-trained FP32 model without retraining. It needs a small representative calibration dataset to measure the range of activations so it can choose scale factors; it is fast and requires no training pipeline, but accuracy can drop, especially for INT8 on sensitive models or for sub-8-bit formats. QAT inserts simulated ("fake") quantization into the training graph so the model learns weights that are robust to the rounding it will face at inference; it recovers most of the lost accuracy and is usually necessary below INT8, but it requires the full training setup and extra training time. The practical rule: try PTQ first, and move to QAT only if the accuracy drop exceeds your budget.
- What is the difference between per-tensor and per-channel quantization, and static vs dynamic?
- Per-tensor quantization uses one scale/zero-point for an entire weight tensor; per-channel uses a separate scale per output channel, which fits the differing weight ranges across channels and usually recovers meaningful accuracy for convolutional layers at little cost — it is the common default for weights. Static vs dynamic refers to activations: static quantization fixes activation scales ahead of time using calibration data (fully integer inference, best for fixed hardware like an NPU), while dynamic quantization quantizes weights offline but computes activation ranges on the fly at runtime (simpler, no calibration set, but less of the speedup). Edge accelerators almost always want fully static, per-channel-weight INT8.
- What techniques besides quantization shrink a model for the edge?
- Quantization is one of three complementary levers. Pruning removes weights or whole channels that contribute little, creating sparsity that can cut size and compute — structured pruning (whole channels/filters) maps to real speedups on general hardware, while unstructured pruning needs sparse-aware kernels to pay off. Knowledge distillation trains a small "student" model to mimic a large "teacher," often reaching better accuracy at a given size than training the small model directly. And efficient architectures — depthwise-separable convolutions and mobile-class backbones — are designed for a high accuracy-per-FLOP ratio from the start. In practice you combine them: pick an efficient architecture, distill or prune, then quantize, measuring accuracy at each step against the deployment budget.
- What hardware actually runs models at the edge, and what limits it?
- The spectrum runs from microcontrollers (Cortex-M-class, where on-chip SRAM and flash measured in tens-to-hundreds of kilobytes are the binding constraint, often without an FPU) up through application processors, DSPs, and dedicated NPUs / ML accelerators that provide efficient INT8 (and increasingly INT4) matrix engines. The limiting resource is usually memory — both storing the weights and holding intermediate activations (tensor arena) — followed by latency and energy per inference. This is why model footprint matters more than raw FLOPs on small targets: a model that does not fit in SRAM cannot run regardless of how fast the core is. Matching the model to the memory and the available integer/accelerator path is the core deployment skill (see /topics/computer-architecture).
- How should an embedded engineer start with edge AI in interviews and on the job?
- Anchor everything to the deployment budget: memory (flash for weights, SRAM for the activation arena), latency per inference, and energy. Be able to walk a model from FP32 to INT8 with PTQ, explain what a calibration set does, and say when you would escalate to QAT. Know that on a microcontroller the model often must fit in SRAM, so footprint dominates. Connect it to the embedded skills you already have (see /topics/embedded-engineer-interview-signals) — memory maps, real-time constraints (see /topics/rtos), and reading a datasheet for the accelerator's supported operators and data types. Refresh the underlying model concepts at the depth your role needs (see /topics/machine-learning-fundamentals-for-engineers, /topics/neural-networks-explained-for-engineers). The differentiator is reasoning about the accuracy-versus-footprint tradeoff with real numbers, not naming a framework.
Related topics
Siblings
Practice
Essential AI-Native Skills for Edge AI & Model Quantization: Running ML on Constrained Devices
Modern engineering work increasingly uses AI tools for design and code review, debugging, documentation, test and testbench generation, and workflow automation. The goal is not to let AI replace engineering judgment — it is to move faster while keeping verification discipline.
- Use AI to explain unfamiliar code, logs, waveforms, datasheets, or test failures.
- Break large problems into small, reviewable steps you can verify independently.
- Ask AI for hypotheses, then validate them against tests, measurements, simulations, or lab data.
- Version-control your analysis scripts, testbenches, and configs — keep changes small and reviewable.
- Document your assumptions, design tradeoffs, and debugging decisions.
- Verify AI output before trusting it: run the checks that fit the domain — unit tests, linters, simulations, or bench/lab measurements.
- Review AI output for correctness, edge cases, and real-world consequences.
Edge AI & Model Quantization: Running ML on Constrained Devices — coming to the question bank
The adaptive practice engine is already live for core wireless, RF, and ML systems. Edge AI & Model Quantization: Running ML on Constrained Devices isn't covered in the question bank yet — get notified when it's added.