A new technical investigation into large language model inference has produced a troubling finding: the same model weights, loaded on the same GPU, can generate different tokens depending on which attention backend, quantization format, or CUDA kernel the software stack uses. The divergence is not cosmetic. In controlled experiments, token-level flips led to incorrect tool calls, botched Cisco router commands, and corrupted API arguments. The work challenges a core assumption held by many local LLM users: that identical weights produce identical results across setups.
The experiments, run by a researcher at Turnstone lab, focused on the Qwen3.6-27B model, a dense hybrid architecture with 64 layers arranged in a repeating pattern of three Gated DeltaNet linear-attention layers followed by one full-attention layer, for 16 full-attention layers total. The official BF16 checkpoint was tested on an RTX PRO 6000 Blackwell GPU with tensor parallelism set to 1, BF16 KV cache, no weight or activation quantization, a pinned nightly vLLM build, eager execution, CUDA graphs disabled, prefix caching disabled, MTP disabled, and 2k-token chunked prefill. The workload, called Prompt 2, was a real Turnstone lab workstream of roughly 100k tokens with multiple tool calls.
The investigation unfolded in two parts. Part 1 captured only 3% of logits for a high-level statistical view. Part 2, the current focus, captures 100% of logits during tool calls and uses a forking multiverse visualizer to show parallel streams of output tokens from two to five comparable runtimes. When the streams differ, the visualizer branches and follows both paths. The author acknowledged that the 3% sampling in Part 1 was not a correct overall methodology, but said it was needed for the big picture.
Attention Backends Produce Divergent Tokens
The first experiment compared the three full attention backends available in vLLM: FlashAttention 2, Flash Inference, and Triton Attention. The only change between executions was the attention backend; the rest of the software stack remained stable. The researcher captured full-vocabulary logits in BF16 every 32 prompt tokens, with KLD calculated in FP64. That works out to 250 sampled positions per 8k-token window.
For the first several thousand tokens, all backends agreed. Then disagreements appeared in clusters, varying with prompt content rather than increasing smoothly with context length. The graph, using Triton as the baseline, shows the percentage of sampled logits that resulted in top-1 flips. A top-1 flip means a backend would have chosen a different greedy next token at that position. The forced token history was kept constant, so the choice did not alter the remaining history.
To rule out random noise, the researcher ran the same test multiple times. Logits were bit-for-bit identical across runs with the same backend. That isolation points to a specific source: divergence comes exclusively from matrix multiplication and addition operations during prefill inside Triton, FlashAttention 2, or Flash Inference. Attention backends impact prefill speed and precision, and each requires different CUDA kernels for each GPU family and SM compute capability.
The practical consequences surfaced quickly. In one example, a single token flip caused FlashAttention 2 to target GigabitEthernet0/1/4 instead of GigabitEthernet0/0/1.201. The model then ran "show run" instead of "show mac address table". In another case, a FlashAttention 2 token flip failed to configure an interface description. These errors are repeatable with bit-identical logit captures.
The researcher also compared tensor parallelism configurations. TP1 produced an acceptable tool call. TP2 failed. TP4 succeeded. The usual culprit, the author noted, is NCCL, NVIDIA's Collective Communications Library.
KV-Cache Quantization and the Five-Way Quant Bakeoff
The second experiment quantized only the KV cache, using int8 and int4, while weights and activations stayed in BF16. The int8 KV cache eventually recovered from tool-call errors. The int4 KV cache did not.
The third experiment was a five-way quant bakeoff: BF16, FP8, INT8, W4A16 via AWQ, and NVFP4, with all KV caches in BF16. CUDA kernel, GEMM, and MMA instructions differ for each quantization format, which is precisely where the divergence originates.
The results were stark. TheDude, a W8A16 weight quantization method, performed best, beating first-party FP8 and NVFP4. First-party FP8, W8A8, came in second. NVFP4, NVIDIA's FP4 quantization, finished dead last with roughly 50% token flips at 88k context. Both NVFP4 and AWQ failed to close tool calls and botched Cisco command syntax, executing "show run" instead of "show arp". FP8 and INT8 completed correct calls. BF16 was the baseline.
The author noted that NVIDIA released FP4 quantization, referred to as "FP4-is-a-Lie," and it performed worst in the bakeoff. Future experiments are planned on fused GEMMs for the same weights.
The author also ran a quick RTX 5090 quant buyers guide titled "Qwen 3.8 Quant Selection Guide for RTX 5090," addressing the consumer GPU crowd.
The Fine-Tune Problem: Heretic-ARA, Huihui, Blackfrost, and AEON
With the release of Qwen3.8-27B, a newer model from Qwen, many fine-tunes labeled "HERETIC! UNCENSORED! ABLITERATED!" have become popular. The researcher compared four popular fine-tunes of Qwen3.8-27B, all in full BF16, not quantized, against reference BF16 logits captured on SM120 for the stock Qwen/Qwen3.8-27B. A limited W4A16 side experiment was also run.
The four derivatives were Heretic-ARA, Huihui, Blackfrost, and AEON. The results split cleanly into two tiers. Heretic-ARA and Huihui appeared remarkably functional. Blackfrost and AEON, the author wrote, should be on the do-not-use list.
The analysis used two probe sets. SP04 covers 1,535 assistant-output tokens in six natural ranges. SP06 covers 4,339 tokens in seven ranges, including prose, CLI/SQL/code, multi-tool calls, recovery actions, and architecture recommendations. The term "invalid branch futures" refers to structurally invalid output among selected top-1 divergence roots, not a general tool-call failure rate.
On SP06, 57 of 58 flips occurred where stock Qwen was already uncertain. Zero strongly preferred stock tokens were overturned. When Qwen was confident, faithful derivatives did not flip. Only the truly heretical fine-tunes overruled Qwen's strong next-token signal, and that is where errors appeared.
Huihui's method is the most reproducible. The author called Huihui the strongest example of an ablation that changed target behavior without broadly destabilizing ordinary technical output. Huihui explicitly labels its process a crude proof of concept, yet it landed in almost the same conservative tier as Heretic-ARA.
AEON combines multiple techniques, and its model card says the selected trial prioritized coherence rather than minimum KL. AEON at SP06 token position 42,950 is a nearly perfect visual explainer of the problem. A second AEON case corrupted a known hostname. Another AEON example changed psycopg's page_size=100 into size=100, likely an invalid API argument.
All four derivatives preserved vision and MTP tensors byte-for-byte, independently hashed. That means the text results come from language-weight changes, not hidden vision or MTP modifications. The vision path was not activated in the panel, so exact weight preservation was established but not identical vision behavior.
Heretic-ARA and Huihui preserved stock behavior far better than Blackfrost and AEON on long-context technical workloads. AEON produced the clearest reproducible operational damage, but its bundled recipe prevents attributing the damage solely to abliteration. None of the measurements establishes which derivative is most successfully uncensored. Refusal benchmarks and model-card KLD numbers do not characterize collateral changes to tool use, exact literals, code, or long-context agentic work.
Logits, KL Divergence, and the Limits of Measurement
The investigation leans heavily on logits, the model's scores for each possible next token. Logits are normalized into probabilities, passed through a sampler, and converted to text by a detokenizer. Sampler settings and chat templates are specified on the Hugging Face model card, such as temperature 1.0 and top-p 0.95. Setting temperature too low can cause models like Qwen to loop in THINK output.
KL Divergence, or KLD, measures how far a probability distribution has moved from a baseline. Lower KLD means closer to baseline, not automatically "smarter." KLD is directional; the order of distributions matters. Interpreting KLD requires disclosure of reference checkpoints, runtime environment, evaluation text, calibration data, context lengths, sampled positions, KL direction, vocabulary truncation, and aggregation method. The author's harness normalizes logits to float64, applies log_softmax, calculates directional KLD with P_BF16 as the reference, reverse KL, and Jensen-Shannon divergence. Per-token values are retained and summarized within output ranges, not as a global average.
BF16 is a numerical-fidelity reference, not an oracle or correctness label. Top-1 disagreement is calculated as changed winner positions divided by evaluated positions. Top-1 and KL describe different things: KL measures whole distribution movement, while top-1 is a discontinuous winner test. A teacher-forced top-1 flip is a counterfactual root, not automatically a different complete answer. Branching is used to inspect what actually happens.
The Software Stack Is a Minefield
The vLLM nightly container image used in the experiments had 734 packages, including 252 uv/pip Python packages, each with bugs and idiosyncrasies. That is the environment where attention backends are selected and where quantization formats are applied. The reference implementation, the lab that published the model and offers first-party hosting, runs on different hardware and software than home users. Home lab users may mix GPU generations with different instruction sets. Standard benchmarks like terminal bench, hle, SWEthis, HELLAthat, and MMLU are used to measure performance, but zero-shot tests are not good analogs for agentic tasks. Long-context tool-calling and domain-specific evaluations are needed.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
The author is working on packaging the testing tools and dataset for distribution. Preliminary runs on H200 and B200 GPUs finished last night and this morning, respectively, and the results are "very interesting." A larger readout on configs, capabilities, and costs is planned.
The takeaway, the author wrote, is simple: don't use small dumb models for important tasks, or more precisely, don't use small dumb models full stop. The author thanked readers for their engagement and noted being featured in Wendell's latest video. Wendell, referenced in the article, is quoted about logits. The author's P.S. mentions the appearance.
The broader lesson is that local LLM users who download quantized models and perceive them as "dumber" than benchmarks suggest may be right, but for the wrong reasons. The model is not dumber. The runtime is different. The CUDA kernels are different. The quantization is different. And those differences produce measurable, repeatable, and sometimes operationally damaging token-level divergence. "If a simple runtime difference in cuda kernels" can change which interface a model targets or which command it runs on a router, then the assumption of identical behavior across setups is not just wrong. It is dangerous.
The investigation does not claim that every divergence leads to failure. Many flips are harmless. But the ones that matter, the ones that break tool calls, corrupt hostnames, and invalidate API arguments, are reproducible. They are not random noise. They are the direct result of implementation-specific choices in the inference stack.
The author's methodology is careful about hedges. Top-1 flips are counterfactual roots, not automatically different complete answers. BF16 is a numerical-fidelity reference, not an oracle. The 3% token distribution methodology in Part 1 is not correct overall, but was needed for the big picture. The measurements do not establish which derivative is most successfully uncensored. Refusal benchmarks and model-card KLD numbers do not characterize collateral changes to tool use, exact literals, code, or long-context agentic work.
What is established is that attention backends, KV-cache quantization, weight quantization, and CUDA kernels all cause token-level divergence. Same-backend runs are bit-identical, isolating divergence to prefill matrix operations. NVFP4 and AWQ quants failed tool calls, while FP8 and INT8 succeeded. TP2 failed a tool call while TP1 and TP4 succeeded, usually due to NCCL. Heretic-ARA and Huihui preserved stock behavior better than Blackfrost and AEON. AEON produced the clearest reproducible operational damage, but its bundled recipe prevents attributing the damage solely to abliteration.
The fine-tune comparison is particularly relevant for the community that flocks to "uncensored" models. The four derivatives tested, Heretic-ARA, Huihui, Blackfrost, and AEON, all come from Qwen3.8-27B. The first two appear remarkably functional. The latter two should be avoided. The distinction is not about censorship. It is about whether the fine-tuning process destabilized the model's ability to produce correct technical output. Huihui, which explicitly labels its process a crude proof of concept, landed in almost the same conservative tier as Heretic-ARA. Blackfrost and AEON did not.
The AEON case at token position 42,950 in SP06 is the clearest illustration. The corrupted hostname and the psycopg page_size error are concrete, reproducible failures. They are not theoretical. They happened in a real workstream with real tool calls.
The investigation also touches on the philosophy of determinism. The author notes that models are "grown" not programmed, and emergent properties come from gaussian noise. A truly deterministic model might lose something. But the current state of floating-point inference means users accept error for speed. The int64 accumulator path is computationally prohibitive, and there is no int64 accumulator in GPU hardware. Int16 models are experimental or abandoned. Int32 accumulators would still be lossy.
The practical advice for users is to be aware of the stack. The same weights on different backends are not the same model in practice. The same weights with different quantization are not the same model in practice. The same weights with different tensor parallelism settings are not the same model in practice. The divergence is measurable, and it matters most for agentic tasks where a single wrong token can trigger a wrong command.
The author's future plans include a larger readout on configs, capabilities, and costs, plus packaging the tools and dataset for distribution. That will allow the community to test their own stacks and see where their own divergence lies.
The H200 and B200 preliminary results are described as "very interesting," which suggests the findings extend beyond the RTX PRO 6000 Blackwell GPU used in the main experiments. The author ran a quick 5090 quant buyers guide, indicating the consumer GPU crowd is not exempt.
The investigation is a reminder that the software stack is part of the model. The 734 packages in the vLLM nightly container are not neutral infrastructure. They are active participants in every token generated. The attention backend, the quantization format, the CUDA kernel, the NCCL collective, all of them shape the output. When a tool call fails, the model may not be at fault. The runtime may be.
The author's response to the commenter about INT calculations is instructive. The idea of reproducible builds via integer math is appealing, but the hardware does not support it at scale. Int64 accumulators would require more registers, more power, wider add paths, more routing, more bandwidth, and more output-tile storage. The cost is prohibitive. Speed is why most people accept the error.
The investigation does not offer a simple fix. It offers a diagnosis. The same model weights do not produce identical results across setups. The divergence is real, measurable, and sometimes damaging. The takeaway is to test your own stack, know your own divergence, and do not assume that a model card tells you what the model will do on your hardware.
The author's final advice is blunt: don't use small dumb models for important tasks. The fuller version is that any model, large or small, can diverge from its reference behavior depending on the inference stack. The smaller and dumber the model, the less margin for error. The larger and smarter the model, the more likely it is to recover from a token flip. But recovery is not guaranteed.
The investigation is a significant contribution to the local LLM community, which has long debated why quantized models feel different from their BF16 counterparts. The answer, it turns out, is not just the quantization. It is the entire stack. The attention backend, the KV cache format, the CUDA kernels, the tensor parallelism settings, all of them contribute to token-level divergence. And that divergence can break tool calls.
The author's methodology is thorough. The same-backend cross-GPU repeatability control captured full-vocabulary logits in BF16 every 32 prompt tokens, with KLD calculated in FP64. The forced token history was kept constant. The random noise was accounted for by running the same test multiple times. The logits were bit-for-bit identical across runs with the same backend. That is a strong control.
The divergence clusters are also notable. They do not increase smoothly with context length. They vary with prompt content. That means the problem is not simply about long contexts. It is about specific content that triggers different matrix operations in different backends.
The quant bakeoff results are clear. TheDude, W8A16, performed best. First-party FP8 came second. NVFP4 was dead last with roughly 50% token flips at 88k context. AWQ failed to close tool calls. FP8 and INT8 completed correct calls. The pattern is consistent: the more aggressive the quantization, the more divergence, and the more likely a tool call fails.
The fine-tune comparison adds another layer. Heretic-ARA and Huihui preserved stock behavior far better than Blackfrost and AEON. The difference is not about censorship. It is about whether the fine-tuning process destabilized the model's technical output. Huihui, despite being a crude proof of concept, landed in almost the same conservative tier as Heretic-ARA. Blackfrost and AEON did not.
The AEON case is the most damaging. The corrupted hostname and the invalid psycopg argument are reproducible operational failures. The token position 42,950 in SP06 is a nearly perfect visual explainer of the problem. The bundled recipe prevents attributing the damage solely to abliteration, but the damage is real.
The investigation also critiques the use of KLD numbers on model cards. Refusal benchmarks and model-card KLD numbers do not characterize collateral changes to tool use, exact literals, code, or long-context agentic work. A model can look good on a refusal benchmark and still break tool calls.
The author's harness is designed to capture these collateral changes. It normalizes logits to float64, applies log_softmax, calculates directional KLD, reverse KL, and Jensen-Shannon divergence. Per-token values are retained and summarized within output ranges, not as a global average. That is a more granular view than a single KLD number.
The investigation is not finished. The H200 and B200 runs are done, and the results are "very interesting." A larger readout on configs, capabilities, and costs is planned. The tools and dataset will be packaged for distribution. The community will be able to test their own stacks.
The author's P.S. mentions being featured in Wendell's latest video. Wendell is quoted about logits. The connection suggests the findings are reaching a wider audience.
The investigation is a reminder that local LLM inference is not a solved problem. The same weights are not the same model. The stack matters. The divergence is real. And it can break tool calls.
The author's final takeaway is practical: don't use small dumb models for important tasks. The fuller version is that any model can diverge, and the smaller and dumber the model, the less margin for error. Test your stack. Know your divergence. Do not assume.