Date: November 20, 2026
Speaker: Raja Gond, Cornell Bowers
Title: LLM-42: Enabling Determinism in LLM Inference
Host: Rachee Singh
Abstract: In LLM inference, the same prompt often returns different outputs across different runs. At the system level, this non-determinism arises from floating-point non-associativity interacting with dynamic batching and GPU kernels whose reduction orders vary with batch size. A straightforward way to eliminate nondeterminism is to disable dynamic batching, but doing so severely degrades throughput. Another approach is to make the kernels batch-invariant; however, this tightly couples determinism to kernel design, requires new
implementations, and imposes fixed overheads regardless of how much of the workload actually requires determinism.
Inspired by self-speculative decoding, we present LLM-42 – a scheduling-based approach to enable determinism in LLM inference. Our key observation is that a given sequence of tokens is likely to emit the same next token across multiple runs, even with dynamic batching. Moreover, most GPU kernels use the same reduction schedules for all inputs of a given shape. Leveraging these insights, LLM-42 decodes tokens using a nondeterministic fast path and enforces determinism through a verify-rollback protocol.
Link: https://dl.acm.org/doi/epdf/10.1145/3830418.3843901
Bio: Raja is a first-year CS Ph.D. student at Cornell. He is broadly interested in systems for ML and low-level GPU optimizations.