NVIDIA’s cuDNN documentation is specific. Routines are designed for the same bit-wise results across runs on GPUs with the same architecture. Across architectures — Volta to Turing, Turing to Ampere, Ampere to Hopper — no cuDNN routine guarantees bit-wise reproducibility.
Each generation implements floating-point differently, schedules threads differently, and hits memory differently.
Architecture differences
Tensor Cores. Multiply-accumulate pipelines differ by generation. cuDNN notes Tensor Core results are “very close, but not always identical” to scalar floating-point; the deviation pattern changes with hardware.
Thread scheduling. SM counts, warp schedulers, and register files differ. NVIDIA’s floating-point precision notes explain why parallel reductions are sensitive to execution order — and scheduling is architecture-dependent.
Memory hierarchy. L2 size, shared memory layout, and bandwidth change what data is available when, which can change accumulation order.
cuBLAS. Matrix routines can behave nondeterministically depending on buffer heuristics. CUBLAS_WORKSPACE_CONFIG can force deterministic behavior — only within a single architecture.
Bit-exact reproducibility requirements
NVIDIA’s GTC 2019 determinism presentation lists the conditions for bit-exact reproducibility:
- Same GPU architecture
- Same driver version
- Same CUDA version
- Same cuDNN version
- Same framework version
- Same number of GPUs
- Same distribution setup
Drop any one and the guarantee is void. A model validated on an A100 cannot be assumed to produce identical outputs on an H100 — even with identical weights, inputs, and framework code.
Hardware refresh under change control
Hardware upgrades improve throughput and cost. They also invalidate bit-identity baselines from the previous generation.
That creates a real tension: newer silicon is better to run; validation evidence was produced on older silicon; re-validation will not produce bit-identical results even when the model is behaving correctly.
Governance has to treat hardware variation as a declared property, not an anomaly. Tolerance-based comparison — equivalence within a defined numerical bound — replaces exact match. That bound needs definition, justification, and a home in the deployment record.
Tensor Cores and CUDA cores
One path to better cross-run reproducibility is restricting computation to CUDA cores and avoiding Tensor Cores. Ingonyama’s engineering write-up reported identical outputs across three architectures that way, with careful control of operation order.
The cost is real. Tensor Cores exist because they are much faster for matrix work. Disabling them can cut inference throughput by roughly 2–10× depending on model and batch. For most production loads, that tradeoff does not hold.
Where MoE routing adds another variation source, hardware and algorithmic nondeterminism stack.
Deployment record
Treat hardware-related reproducibility limits as part of the deployment specification:
- GPU architecture, driver, and CUDA/cuDNN versions in use
- Whether Tensor Cores are enabled and which precision modes are active
- Defined tolerance for output comparison across runs
- Re-validation procedure when any hardware or driver component changes
- Whether deterministic mode is enabled and what performance cost is accepted
If someone will ask “would this system produce the same result if you ran it again?” — and in CMMC-scoped environments they will — the answers need to exist before the question does.