Position bias, verbosity bias, and self-preference bias can silently distort agent eval scores — here is how to design a judge pipeline that resists them, plus why trajectory evaluation catches what outcome scoring misses.
Most teams building agent evals reach for the same pattern: generate outputs, hand them to a strong model with a grading prompt, average the scores, ship. It works well enough to feel trustworthy, which is exactly the problem. LLM-as-judge is not a neutral measuring instrument. It's a language model doing the same kind of pattern-matching it does everywhere else, and that pattern-matching has documented, reproducible biases that will silently distort your eval numbers if you don't design around them.
This isn't a reason to abandon LLM-as-judge — for most teams there's no practical alternative to grading thousands of agent transcripts by hand. It's a reason to know exactly where it lies to you.
Agent evaluation has a harder ground-truth problem than most ML evaluation. A classifier has a label. An agent trajectory — a sequence of tool calls, intermediate reasoning, and a final answer — usually doesn't have one "correct" path, and exact-match scoring against a reference answer breaks the moment an agent solves a task in a different but equally valid order. So teams turn to a second LLM as an approximate human grader: give it the task, the transcript, a rubric, and ask for a score or a pairwise preference.
This is genuinely useful. Well-designed LLM judges correlate with human preference at levels that make them viable for iteration loops (research from groups building the MT-Bench and Chatbot Arena methodologies put agreement in the 80-85% range against human raters for pairwise comparisons, roughly on par with inter-human agreement). The failure mode isn't that judges are useless — it's that teams treat judge output as ground truth instead of as a noisy, biased signal that needs the same skepticism you'd apply to any measurement instrument.
Ask a judge to compare response A and response B, and the verdict changes depending on which one is listed first — even when A and B are held constant and only the order swaps. This is the best-documented LLM-as-judge bias and it's large: swapping order alone can flip the verdict in a meaningful fraction of comparisons, sometimes reported north of 10-20% depending on the judge model and how close the two candidates are in quality.
The mechanism is straightforward: autoregressive models weight earlier context more heavily in some tasks, and grading prompts are no exception. A judge primed with "Response A: ... Response B: ..." isn't immune to the same recency/primacy effects that show up in human list-ranking studies.
Fix: never trust a single-order pairwise comparison. Run each comparison twice with the order swapped, and treat a verdict as valid only if it's consistent across both orders. Ties (or flips) get thrown out or escalated to a third judge. This roughly doubles judge cost per comparison but it's the single highest-leverage fix available.
Judges systematically prefer longer responses, independent of whether the extra length adds correct information. This has been measured directly by having a model pad an otherwise-identical response with restated points or hedging language, and watching the judge's preference shift toward the padded version.
For agent evals specifically, this shows up as judges preferring agents that narrate their reasoning verbosely over agents that act efficiently and correctly with minimal explanation — the opposite of what you usually want in a production agent, where excess narration means more tokens, more latency, and more surface area for the user to get lost in.
Fix: put an explicit length-invariance instruction in the rubric ("do not reward length; a correct, concise answer should score at least as well as a correct, verbose one") and spot-check with a length-controlled ablation — same content, two lengths, confirm the judge doesn't reward the longer one. Some public leaderboards (AlpacaEval's length-controlled variant is the most visible example) now report a length-corrected score for exactly this reason.
Judges tend to rate outputs from their own model family more favorably than outputs from a different family, even when a blinded human panel rates them as equivalent. This has been shown across multiple judge/generator pairings and isn't fully understood mechanistically — the leading explanation is that a model's own stylistic conventions (how it structures explanations, hedges uncertainty, formats lists) read as "higher quality" to that same model, a bit like an editor unconsciously favoring their own house style.
This matters most when you're evaluating your own agent against a competitor's, or when you're doing model selection ("should we switch from model X to model Y for our agent's tool-calling step") using model X as the judge. The comparison is quietly rigged before you run a single trial.
Fix: use a judge model from a different family than any of the systems under evaluation. If you're comparing agents built on Claude, Gemini, and GPT-family models, don't judge with any of the three — or if you must, run the same eval with at least two different judge families and check whether the ranking is stable. If the ranking flips depending on which model is judging, you don't have a result yet.
Judges are disproportionately swayed by formatting: markdown structure, bullet points, bold text, and confident phrasing all shift scores upward independent of the underlying correctness of the content. A well-formatted wrong answer regularly beats a correctly-reasoned but plainly-formatted one in blind pairwise tests.
For agent evals this compounds with tool-use transcripts, where a judge can be swayed by an agent that produces clean, well-labeled intermediate output even if a tool call actually failed or returned wrong data three steps earlier — the polish of the final summary papers over an incorrect trajectory.
Fix: grade the trajectory, not just the final answer (more on this below), and where possible strip formatting before grading final-answer-only content, or add an explicit rubric line telling the judge that formatting quality is not a scoring dimension unless the task specifically asked for it.
On a 1-10 or "rate the quality" scale, judges cluster scores in a narrow high band — most responses get 7s, 8s, and 9s regardless of real quality spread, which compresses your ability to detect regressions. A prompt change that measurably degrades agent behavior can show up as a 0.2-point drop on a 10-point scale that's statistically indistinguishable from noise, even though a human reviewer would immediately notice the difference.
Fix: prefer pairwise comparison over absolute (pointwise) scoring wherever you can. "Is trajectory A better than trajectory B, or are they equivalent?" produces a much sharper signal than "rate this trajectory 1-10," because it sidesteps the judge's tendency to anchor near the top of an absolute scale. When you do need an absolute score (e.g., for a dashboard tracking quality over time), calibrate it against a small set of human-labeled anchor examples spanning the full quality range, and recalibrate periodically — judge leniency also drifts as you update rubric wording or swap judge model versions.
Everything above applies to grading any LLM output, but agent evals have an extra dimension that plain response-grading doesn't: the trajectory itself — which tools got called, in what order, with what arguments, and whether intermediate steps succeeded — carries information that the final answer alone discards.
Outcome evaluation ("did the agent get the right answer") is cheap, easy to automate when there's a checkable ground truth, and blind to how the agent got there. An agent that stumbled into the right answer through a failed tool call and a lucky retry scores identically to one that executed cleanly — but only one of those is a system you can trust on the next 10,000 inputs where the lucky retry doesn't land.
Trajectory evaluation grades the full sequence: did it call the right tools, in a sensible order, with valid arguments, recovering appropriately from errors? This catches reliability risk that outcome scoring misses entirely, but it's more expensive to run (the judge has to read the whole transcript, not just the final answer) and it's more exposed to the surface-feature and verbosity biases above, because transcripts are longer and messier than clean final answers.
In practice, the two are complementary, not substitutes: outcome scoring as a cheap, high-volume filter to catch obvious regressions, and trajectory scoring on a sampled subset — especially failures and near-misses — to understand why something went wrong. Teams that only track outcome accuracy tend to get blindsided by reliability regressions (more retries, more token spend, more fragile error handling) that never show up in the top-line pass rate.
You are evaluating an AI agent's transcript against a task.
Task: {task_description}
Transcript: {full_trajectory_including_tool_calls}
Score each dimension independently on a 1-5 scale. Do NOT let response
length, formatting quality, or writing style influence any score below —
grade only the dimension named.
1. Task completion: did the agent's final answer satisfy the task?
2. Tool-use correctness: were tool calls valid, well-targeted, and in a
sensible order? Penalize unnecessary or redundant calls.
3. Error recovery: if a tool call failed, did the agent recover
appropriately, or did it proceed on bad data / give up silently?
4. Efficiency: did the agent reach the answer in a reasonable number of
steps for this task's difficulty?
Return each score with a one-sentence justification citing a specific
step in the transcript, not the final answer alone.
The "cite a specific step" instruction is doing real work here — judges asked to justify a score against the transcript are harder to fool with a polished final answer that papers over a broken middle, because the justification forces at least partial attention back onto the trajectory.
None of this means LLM-as-judge is unusable — it means treating a judge score the way you'd treat any noisy instrument: run comparisons in both orders, use a judge from a different model family than what you're evaluating, prefer pairwise over absolute scoring, grade trajectories for anything where reliability matters more than the happy-path answer, and calibrate periodically against a small human-labeled set so you notice when the instrument itself drifts. An eval pipeline that skips all five checks isn't measuring your agent — it's measuring how well your agent's outputs happen to match the judge's stylistic priors, which is a different (and much less useful) thing to optimize for.
Agents that call tools over MCP produce exactly the kind of multi-step transcripts this piece is about — if you're building or evaluating one, Utilix exposes its calculators and format converters as both a REST API and an MCP server, so an agent under evaluation can be graded on real tool-call trajectories rather than mocked ones.