What Is LLM Temperature? Why It Does Not Fix Random Choice

In short: LLM temperature is the setting that controls how much variety a language model shows when it picks the next token. A low value makes the output more consistent, a high value more varied. But temperature does not fix random choice: according to a 2026 paper by Google DeepMind researchers, models asked to "pick at random" mostly put the correct answer at C and lean toward 7 and 42 when picking numbers. The fix is to generate the randomness in code and give the result to the model.

It is easy to ask an AI agent to shuffle the options of a quiz, split users into groups for an A/B test or pick records at random for an audit. The model describes the requested distribution correctly, but it does not do what it describes. The paper The Illusion of Stochasticity in LLMs by Xiangming Gu and colleagues at Google DeepMind, published at COLM 2026, shows in detail why the temperature setting does not fix this.

Temperature is one of the settings that decide how the model picks the next token at each step. At every step the model gives each possible token a probability and picks one of them. With a low temperature the choice concentrates on the most likely tokens; with a high one, less likely tokens can be picked too. We explain what a token is in What is a token in AI?

According to OpenAI's API reference, temperature takes a value between 0 and 2. Higher values like 0.8 make the output more random, lower values like 0.2 make it more focused and deterministic. OpenAI recommends changing this setting or top_p, but not both. Top-p and top-k are decoding settings that limit the same choice. The paper tests all three.

That is why raising the temperature looks like the way to make a model choose at random. The experiments in the paper show that it does not work.

The researchers asked Gemini models to write a four-option question and state the correct answer. According to the paper, the position of the correct answer should be random to prevent cheating, so it should appear equally often at each option. Gemini-2.5-Flash, Gemini-2.5-Pro and Gemini-3.0-Pro instead put it clearly more often at C. That bias makes the result predictable.

Numbers go the same way. Qwen3 and Gemini models were asked for samples from uniform, Gaussian and other distributions in 1,024 independent calls. In the statistical tests the p-values came out very close to zero, which means the numbers did not follow the requested distribution. The models favor numbers like 7 and 42 and values in the middle of the range. Every model tested failed, from Qwen3-8B to Qwen3-32B and from Gemini-2.5-Pro to Gemini-3.0-Pro. The authors also note that the probabilities of the base (pre-trained) models are much closer to uniform, so post-training may amplify the bias.

It does not. The researchers tried temperature values from 0 to 2 and different top-p and top-k settings. For the uniform distribution the p-values stayed close to zero in every setting. For the Gaussian distribution the best result came at a temperature of 1.8, and even that did not pass the test. Very high values between 2.5 and 10 failed too, and the model also struggled to give the answer in the requested format and produced extreme numbers.

Asking for a direct answer without step-by-step reasoning (chain-of-thought turned off) does not help either. According to the paper the bias grows in Qwen3 in that case: asked for a uniform pick between 1 and 100, Qwen3-8B produces only 42.

Having the model write its own code is not enough on its own. Models that can use tools write code that calls random.randint(), but that code usually sets no random seed. If the environment that runs the code starts with the same seed every time, the outputs repeat. And when the model is asked to come up with the seed itself, the same bias shows up in the choice of seed.

The clearest example in the paper is a security game. An agent inspects one of four servers on each request, and an attacker watches the agent and attacks the least inspected server. If the agent picks the four servers with equal probability, the attack succeeds 75% of the time, which is the best possible defense. Over 1,024 calls the attacker reaches 96.6% with Gemini-3 Flash and 99.9% with Gemini-3.1 Pro and Gemini-3.5 Flash. The models can say that equal choice is the best strategy, but their behavior is still biased.

The same problem appears in everyday work: the order of quiz and survey options, group assignment in an A/B test, audits based on random samples, and simulations. Such decisions are often left to an AI agent. We cover architectures where several agents work together in One agent or many?

Order matters not only where the model makes a choice but also where it acts as a judge. When a language model decides which of two answers is better, it tends to favor the one shown first. In the MT-Bench paper's measurement, Claude-v1 picked the first answer in 75% of cases. We explain how to correct for it in our LLM as a judge post.

The approach the paper proposes is simple. Generate the random number with a real random number generator such as Python's and add this number between 0 and 1 to the prompt. Models can reliably convert that number to the requested distribution. The conversion works for uniform and Gaussian distributions and also for more complex examples such as a weighted choice and a two-peaked distribution. In the test with the Qwen3 family, uniform choice needs models larger than 1.7 billion parameters and the Gaussian needs models larger than 4 billion.

If the choice is simple, the more robust way is to leave the decision entirely to code:

import random

options = ["A", "B", "C", "D"]

# 1) Most robust: the code makes the choice, the model only uses the result.
correct_option = random.choice(options)
prompt = (
    "Write a four-option multiple-choice question. "
    f"Put the correct answer at option {correct_option}."
)

# 2) The paper's method: give the model a number in [0, 1)
#    and let it do the conversion. Generate a new number for every call.
r = random.random()
prompt2 = (
    f"A random number sampled from [0, 1]: {r}. "
    f"Use this number to pick one item from {options} "
    "with a uniform distribution."
)

According to the Python documentation, the random module uses the operating system's randomness source when no seed is given, and the system time if that is not available. The same page says the module should not be used for security purposes and points to the secrets module instead. If you are new to code, our Python Programming course covers the basics. Designing which tool an agent uses for decisions like random choice is one of the topics of the AI Agent Fundamentals course.

Temperature is the setting that controls how much variety a language model shows when it picks the next token. On OpenAI the value is between 0 and 2: low values make the output more focused and deterministic, high values more random.

The answers vary more but do not follow the requested distribution. In the paper, none of the temperature values between 0 and 10 was enough to pass the statistical tests.

Not for the Gemini and Qwen models tested in the paper. The models favor numbers like 7 and 42 and the middle of the range. The paper does not test ChatGPT itself, but the same problem appears in every model tested, from small to large.

Make the choice in code and give the result to the agent. If the agent itself has to choose, add a number between 0 and 1 generated with Python to every call. In the paper's experiments the models converted this number to the requested distribution correctly.

• Xiangming Gu, Soham De, Michalis Titsias, Larisa Markeeva, Petar Veličković and Razvan Pascanu, The Illusion of Stochasticity in LLMs, COLM 2026: Figure 1 (option C), Table 1 (security game), sections 3.2 and 3.3 (decoding settings, chain-of-thought), Table 5 (model size).
• OpenAI, Chat Completions API reference: the temperature parameter.
• Python Software Foundation, random: Generate pseudo-random numbers: seeding and the security warning.
• Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Table 2: position bias.

• LLM as a judge and position bias
• What is an AI agent?
• One agent or many?
• What is a token in AI?
• Courses: Python Programming, AI Agent Fundamentals