What Is LLM as a Judge? AI Judges and Position Bias

In short: LLM as a judge means letting a language model decide which of two answers is better. These judges often favor whichever answer they see first. The fix is simple: ask the same comparison twice, with the answers swapped. If both verdicts agree, accept the winner, and if the verdict flips with the order, count the result as a tie.

The fastest way to measure AI output is to let one model grade another. You put two versions of a prompt, two models or two drafts side by side and ask which one is better, which is far cheaper and faster than human review. The catch is that AI judges do not pick on content alone: part of what decides the verdict is the position in which each answer is shown. This is called position bias. With LLM as a judge, the order of the options can matter as much as the answers themselves.

LLM as a judge means using a large language model (LLM) to score or compare the answers another model produces. The term spread with the 2023 paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena by Lianmin Zheng and colleagues from UC Berkeley and other universities (NeurIPS 2023 Datasets and Benchmarks). According to the paper, GPT-4 as a judge agrees with human preferences more than 80% of the time. That is the same level at which humans agree with each other.

The same paper also lists the weak spots of these judges: position bias, a preference for longer answers, a preference for their own answers and limited reasoning on tasks such as math.

Position bias is when a judge picks an answer because of where it sits in the list, not because of what it says. In the paper’s example, GPT-4 is given answers from GPT-3.5 and Vicuna-13B. When GPT-3.5’s answer comes first, GPT-4 finds it more detailed and better. When the order is swapped, its verdict flips too, and this time it picks Vicuna.

To measure the effect, the authors prepared two similar answers for each MT-Bench question and showed the judges both orders. With the default prompt, the results were:

This is a hard test, because the answers were chosen to be very close on purpose. According to the authors, even GPT-4, the most consistent judge, gives the same verdict in only just over 60% of cases.

Yes. In the position_bias test that independent researcher Lech Mazur published on GitHub, 36 current models judge the same two story edits in both orders. On average, the models pick the text shown first 64.3% of the time. The median model changes its verdict in 41.3% of cases when the order is swapped. It is not a peer-reviewed study and it covers only story edits, but the direction matches the MT-Bench results.

Models trained to make decisions come with the same warning. TypeSafe writes in its jev-1.13 documentation that the order of the options can change the model’s answer and that the model leans toward the first option. Its advice is the same: swap the order of the options and check whether the answer stays consistent.

We ran a small trial of our own. In the notebook prepared for RuneLab’s decision-models workshop, an open model with 1.5 billion parameters (Qwen2.5-1.5B-Instruct) routed 24 fictional Turkish support tickets that we wrote to one of four teams. When we reversed the order of the options, 11 of the 24 decisions changed. In other words, the same model can pick a different team for the same ticket when only the position of the options changes.

The MT-Bench paper proposes two fixes. The conservative one works like this: call the judge twice and swap the positions of the answers in the second call. Count an answer as the winner only if it was chosen in both orders. If the two calls disagree, call it a tie. The authors use this method in their own experiments.

def judge(question, first, second):
    """Calls the judge model and returns "A" (first is better), "B" (second is better)
    or "tie". Fill in your own API call."""
    raise NotImplementedError


def compare_both_orders(question, x, y):
    round1 = judge(question, x, y)  # x is shown first
    round2 = judge(question, y, x)  # y is shown first
    if round1 == "A" and round2 == "B":
        return "x"
    if round1 == "B" and round2 == "A":
        return "y"
    return "tie"  # the verdict flipped with the order, or the judge called a tie

This method doubles the cost of every comparison. In return, the verdicts the judge reached because of the order drop out of your list of winners. The tie rate also gives you a measure of how much the judge depends on the order.

The paper’s second option is bolder: randomize the positions of the answers in every comparison. According to the authors, this can work across a large number of comparisons when used with the right expectations. The bias then stays in individual verdicts, but it is spread evenly over both sides. Generate the random order in code rather than leaving it to the model, for example with random.shuffle in Python. We explain why models stay biased when they choose at random in What is LLM temperature?

In the paper, showing GPT-4 three example judgments (few-shot) raised the share of verdicts that stay the same after a swap from 65% to 77.5%. The authors add three caveats: high consistency may not mean high accuracy, the examples may bring in new biases, and the longer prompts make the API cost four times higher. A short routine for building your own judge:

• Compare the judge against a small sample set that people have labeled.
• Ask every comparison in both orders and record the tie rate.
• Run the same test again whenever you change the judge model or the prompt.

We explain how to build a human-labeled test set like this for RAG systems in our Turkish RAG evaluation guide. Setting up the judge together with rule-based checks is the hands-on subject of our Rule-Based Assertions and Model-Graded (LLM-as-a-Judge) Evaluators course.

It means using a large language model as a judge. The model scores another model’s answers or picks the better of two answers. It is also written LLM-as-a-judge.

Zheng and colleagues suggest the cause may lie in the training data or in the left-to-right way these models read text, but they do not prove it. What is certain is that the effect can be measured: in the 2023 test, Claude-v1 picked the first answer in 75% of cases.

Count that comparison as a tie. This is the conservative method the MT-Bench paper recommends: an answer wins only if it is chosen in both orders. A high number of ties shows that the judge is not reliable enough for that task.

It can work for aggregate results across many comparisons, because it spreads the bias evenly over both sides. Individual verdicts are still swayed by order. Where a single verdict matters, ask twice.

• Lianmin Zheng and colleagues, the MT-Bench and Chatbot Arena paper (arXiv 2306.05685, NeurIPS 2023 Datasets and Benchmarks), Section 3.3, Table 2 and Section 3.4
• Lech Mazur, LLM Position Bias Benchmark, GitHub (independent test, not peer reviewed)
• TypeSafe, Jev 1.13 jaggedness: Choice option order
• RuneLab, decision-models workshop notebook (internal trial, Qwen2.5-1.5B-Instruct, float32, 24 fictional tickets)

• One agent or many?
• How to evaluate Turkish RAG systems
• Courses: LLM Evaluations (Evals) Fundamentals: A Practical Introduction, AI Agent Evaluation, Benchmarking and Quality Assurance