Decoding the Jargon: An Essential Guide to GPU Terminology for Developers

As GPUs evolve from graphics processors to general-purpose computing powerhouses, developers face a growing lexicon of proprietary terminology. NVIDIA's CUDA ecosystem alone contains dozens of specialized terms spanning hardware architecture, memory systems, and parallel programming concepts. Understanding this language isn't academic—it's essential for optimizing AI training, scientific computing, and real-time analytics workloads.

The Hardware Lexicon: Inside NVIDIA's Streaming Multiprocessors

At the silicon level, Streaming Multiprocessors (SMs) serve as the fundamental building blocks. Each SM contains:
- CUDA Cores: Basic parallel processing units handling integer and floating-point operations
- Tensor Cores: Dedicated hardware for matrix operations (crucial for deep learning)
- Warp Schedulers: Units managing groups of 32 threads (warps) for instruction dispatch

"The SM's hierarchical design—from cores to Texture Processing Clusters (TPCs) up to GPU Processing Clusters (GPCs)—creates a nested parallelism structure that developers must understand to avoid bottlenecks," notes GPU architect Dr. Elena Rodriguez.

Memory architecture introduces another layer: Registers (fastest), Shared Memory (SM-local), L1 Caches, and Global Memory (GPU RAM). The new Tensor Memory Accelerator (TMA) exemplifies how hardware evolves to address specific workloads like transformer models.

Programming Model: Warps, Blocks, and Kernels

NVIDIA's execution model operates on multiple abstraction levels:

// Simplified CUDA kernel launch
__global__ void vectorAdd(float* A, float* B, float* C) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    C[i] = A[i] + B[i];
}
// Launch configuration: 256 threads per block, 16 blocks
vectorAdd<<<16, 256>>>(d_A, d_B, d_C);

Key concepts include:
- Kernels: Functions executing across thousands of threads
- Thread Hierarchy: Organized into blocks and grids
- Cooperative Thread Arrays (CTAs): Synchronized thread groups sharing memory

Performance hinges on managing warp divergence (threads taking different paths) and bank conflicts in shared memory—issues directly tied to terminology comprehension.

Software Stack: From Drivers to Domain Libraries

The host-side ecosystem includes:
- CUDA Driver API (libcuda.so) for low-level control
- nvcc compiler translating CUDA C++ to PTX intermediate code
- cuBLAS/cuDNN: Optimized libraries for linear algebra and deep learning

Tools like NVIDIA Nsight Systems and CUPTI leverage this vocabulary to expose metrics like occupancy (active warps/SM) and arithmetic intensity (compute-to-memory ratio).

Performance Optimization: Speaking the Language of Efficiency

Understanding terms like memory-bound vs compute-bound workloads enables targeted optimization using the Roofline Model. Key metrics include:

Developers who master this vocabulary can diagnose issues like branch efficiency penalties and optimize pipe utilization—transforming abstract terms into tangible performance gains.

As GPU architectures grow increasingly complex, fluency in this specialized language separates effective parallel programmers from those struggling with black-box optimization. The terminology provides the conceptual scaffolding needed to harness teraflops of compute potential—whether training billion-parameter models or simulating fluid dynamics.

Source: Modal GPU Glossary