CUDA & GPU Programming for Engineers: Interview Guide Interview Prep

Frequently asked questions

What is the difference between a thread, a warp, and a block in CUDA?

These are the three levels of CUDA’s execution hierarchy. A thread is the smallest unit of work and runs one instance of the kernel. Threads are grouped into a block, and threads within a block can cooperate through fast shared memory and barrier synchronization (__syncthreads). Blocks are organized into a grid, which covers the whole problem. The hardware schedules execution in warps — a fixed group of 32 threads on NVIDIA GPUs that execute the same instruction together (SIMT). A block is assigned to one streaming multiprocessor and split into warps; the warp, not the block, is what the scheduler actually issues. Interviewers expect you to know the warp size is 32, that blocks map to an SM, and that cross-block communication is limited — it usually means a separate kernel launch (see /topics/computer-architecture).

What is SIMT and how does warp divergence hurt performance?

SIMT — Single Instruction, Multiple Threads — is the execution model where the 32 threads of a warp share one program counter and execute the same instruction in lockstep, each on its own data. It looks like independent threads to the programmer but runs like a wide vector unit underneath. Warp divergence happens when threads in the same warp take different branches of an if/else: the hardware must serialize the paths, executing the taken branch with the other threads masked off, then the other branch. A fully divergent warp can lose much of its throughput. The fix is to structure data and conditionals so threads in a warp follow the same path — sorting work, masking arithmetic instead of branching, or aligning branch boundaries to warp size. This connects to how vector and parallel hardware actually executes (see /topics/computer-architecture).

Why does memory coalescing matter so much for GPU kernels?

Global memory bandwidth, not compute, limits most real kernels, so how a warp accesses memory dominates performance. Coalescing means the 32 threads of a warp access consecutive, aligned addresses so the hardware merges them into a few wide memory transactions instead of many small ones. A coalesced load can be an order of magnitude faster than a scattered one. The classic mistake is a strided or transposed access pattern — for example, threads walking down a column of a row-major matrix — which forces one transaction per thread and wastes most of the fetched bytes. Common fixes are restructuring the data layout, having each thread read adjacent elements, or staging a tile through shared memory so the global access is coalesced and the awkward pattern happens in fast on-chip memory. Recognizing a non-coalesced pattern is a core GPU interview signal.

What is the GPU memory hierarchy and how do you use shared memory?

A CUDA kernel sees several memory spaces with very different speed and scope. Registers are per-thread and fastest but limited; spilling to local memory (actually in global) is slow. Shared memory is on-chip, per-block, and roughly as fast as an L1 cache — threads in a block use it as a software-managed scratchpad to share data and avoid repeated global loads. Global memory (device DRAM) is large but high-latency and bandwidth-bound. There are also read-only constant and texture caches. The standard optimization pattern is tiling: cooperatively load a block of data from global memory into shared memory once (coalesced), synchronize with __syncthreads, then have all threads reuse it from shared memory — as in tiled matrix multiply. Knowing this hierarchy and the tiling pattern is what separates someone who has written real kernels from someone who has only read about them (see /topics/computer-architecture).

What is occupancy and is higher always better?

Occupancy is the ratio of active warps on a streaming multiprocessor to the hardware maximum. The GPU hides memory and instruction latency by switching among many resident warps, so enough occupancy is needed to keep the SM busy while some warps wait on memory. Occupancy is capped by per-thread register usage, per-block shared-memory usage, and block size — a kernel that uses many registers or lots of shared memory fits fewer warps. But higher occupancy is not always better: beyond the point where latency is hidden, more occupancy gives nothing, and pushing it can force register spills or smaller tiles that hurt overall throughput. Strong candidates treat occupancy as one lever among many — enough to hide latency — rather than a metric to maximize blindly. This is a frequent trap in GPU optimization interviews.

When does GPU acceleration actually help, and when does it not?

GPUs win when the work is massively data-parallel, arithmetically heavy relative to the data it moves, and large enough to amortize launch and transfer overhead — dense linear algebra, deep-learning training and inference, large FFTs and convolutions, and Monte Carlo. They are a poor fit for serial, branch-heavy, or pointer-chasing work, for small problems where kernel-launch and PCIe transfer dominate, and for memory-latency-bound tasks with irregular access. A decisive factor is host–device transfer: moving data over PCIe between CPU and GPU is slow, so a kernel that is fast in isolation can lose to the CPU once you count the copy in and out. The right framing is arithmetic intensity (FLOPs per byte) and the roofline model: compute-bound, parallel workloads benefit; latency-bound or transfer-dominated ones often do not. This judgment matters for AI and edge deployment (see /topics/edge-ai-and-model-quantization, /topics/machine-learning-fundamentals-for-engineers).

How should an engineer prepare CUDA and GPU topics for interviews?

Anchor on the execution and memory models, not API trivia. Be able to explain the thread/warp/block hierarchy, that a warp is 32 threads executing SIMT, and why divergence and non-coalesced access destroy throughput. Know the memory hierarchy — registers, shared memory, global — and the tiling pattern that stages data through shared memory. Understand occupancy as latency-hiding, not a number to maximize, and be ready to reason about a kernel’s bottleneck with arithmetic intensity and the roofline model. Above all, be able to argue when a GPU is the wrong tool: small, serial, branchy, or transfer-dominated work. Tie GPU concepts to the ML workloads that drive the demand — training, inference, and quantized deployment — and to the underlying hardware (see /topics/neural-networks-explained-for-engineers, /topics/edge-ai-and-model-quantization, /topics/semiconductor-devices). Reasoning about the machine beats reciting CUDA syntax.

Related topics