NVIDIA’s framework determinism guidance is direct: the same prompt through the same model on the same GPU can still produce different outputs because parallel execution order varies. Where someone must reconstruct what happened and why, that is a material property of the system — not a quirk to ignore.
Sources of variation
GPU inference pipelines are not deterministic by default. The core mechanism is floating-point non-associativity: the order of floating-point additions changes the result, and parallel hardware does not guarantee order.
Three concrete contributors:
Parallel reduction order. When thousands of partial products sum across cores, thread scheduling sets the order. Different schedules produce different rounding sequences that propagate through later layers.
Atomic operations in kernels. cuDNN documentation lists routines that use atomics in ways that introduce random floating-point rounding errors, including some paths used in certain inference configurations.
Tensor Core precision. Tensor Core pipelines differ from scalar CUDA cores. Hardware-level variation matters more when deployments span GPU architectures.
These are properties of parallel floating-point hardware, not application bugs.
Reconstruction and change validation
Regulated audit frameworks often assume a basic question is answerable: same inputs, same system, same output? If the honest answer is “usually, not always, and we cannot predict when,” several assumptions break.
When an output contributed to a decision and the system cannot reliably reproduce it, reviewers cannot cleanly separate “the model behaved correctly on those inputs” from “something changed.”
Teams that compare outputs across environments cannot trust exact match. A mismatch may be a real regression or FPNA noise. Without a defined tolerance, every mismatch becomes manual review.
Programs that require documented system behavior face a definitional problem: behavior includes a nondeterministic component that cannot be fully specified in advance.
Deterministic mode and its cost
PyTorch’s deterministic mode forces more reproducible operation. The docs are clear: deterministic ops are often slower, and some ops have no deterministic implementation and error instead.
NVIDIA’s GTC 2019 guidance is stricter: bit-exact reproducibility requires locking the full software and hardware stack at once. Change any layer and the guarantee is gone.
Mixture-of-Experts routing adds another variation layer on top of numerical nondeterminism when those architectures are in play.
Tolerances, logs, and declared variation
There is no free fix that keeps both peak performance and perfect bit-identity. The choices are explicit.
Bit-exact reproduction means version-controlling and pinning the entire inference stack — see the requirements list. If exact reproduction is not required, define acceptable deviation and document the rationale as operational policy.
When outputs may differ, the run record needs enough context to explain expected variation: software versions, hardware identifiers, inference configuration — not only prompt and response.
Document nondeterminism as a system property and design review around bounded, expected variation. An auditor who finds unexplained drift is in a worse position than one who finds documented bounds.
The hardware dimensions go further when the fleet spans architectures.