pass@k vs pass^k: The Metric That Tells You Whether an Agent Is Reliable

pass@k and pass^k look like the same metric with a typo between them, but they answer opposite questions: pass@k asks whether an agent can solve a task in any of k tries, while pass^k asks whether it solves the task on every one of k tries. If you are shipping an agent that acts once per user request, pass^k is the number that predicts your support tickets, and a strong pass@1 can hide a pass^k that falls off a cliff.

This post works through both metrics with real numbers, shows the unbiased estimators you should actually compute, and explains why two agents with an identical pass@1 can need completely different engineering fixes.

Where the two metrics come from

pass@k: the code-generation metric

pass@k was popularized by OpenAI's Codex paper, which introduced the HumanEval benchmark. The setup: generate k candidate programs for a problem, and count the problem as solved if at least one candidate passes the unit tests. That framing made sense for code generation because unit tests are a cheap, reliable verifier. If you can sample ten programs and run the tests on all of them, you only care that one of them works.

For a single task where the model succeeds with probability p on each independent attempt:

• pass@k = 1 - (1 - p)^k

Each extra attempt is another lottery ticket, so pass@k only goes up as k grows.

pass^k: the agent reliability metric

pass^k (read "pass hat k") came from Sierra's τ-bench paper, which evaluates agents that hold multi-turn conversations with a simulated user while calling tools against a database, in retail and airline customer-service domains. There is no verifier to pick the best of k attempts in that setting: a customer gets one conversation, and a refund issued wrongly is issued. So τ-bench asked the inverse question. If you run the same task k times, does the agent succeed on all k?

• pass^k = p^k

Now each extra attempt is another chance to fail, so pass^k only goes down as k grows. The paper's headline finding was exactly this gap: even strong function-calling agents solved under half of the tasks, and gpt-4o's pass^8 in the retail domain was below 25%.

The same agent, two very different curves

Here is what both metrics look like for a single task at three per-attempt success rates.

The row that matters is k = 8. An agent that succeeds 90% of the time looks nearly perfect under pass@8 and fails at least once in more than half of eight-run sequences under pass^8. The 70% agent is a near-certain success under pass@8 and a near-certain failure under pass^8. Same model, same tasks, same transcripts.

This is the same compounding math behind multi-step agent failures, covered in why benchmark scores don't predict production reliability, applied one level up: instead of steps inside one run, it is runs of the same task.

Two agents with identical pass@1

The single most useful thing pass^k reveals is something pass@1 structurally cannot: whether failures are spread evenly or concentrated.

Consider a 100-task benchmark and two agents, both scoring pass@1 = 80%.

• Agent A is uniformly flaky. Every task has an 80% per-attempt success rate.
• Agent B is consistent but limited. It solves 80 tasks every single time and never solves the other 20.

Agent A looks better under pass@8 and dramatically worse under pass^8. Neither number is wrong. They are telling you different things about the failure distribution, and they call for different fixes.

Why the distinction drives engineering decisions

Agent A has a variance problem. It can do every task, so the work is about making it do so reliably: tightening prompts and tool descriptions, lowering sampling temperature where it is not buying anything, adding a verification step before irreversible actions, and constraining output formats. If you have a trustworthy checker, it is also the agent that benefits most from best-of-k sampling.

Agent B has a capability problem on a specific slice. Retrying is pointless because it will fail the same 20 tasks forever. The fix is to find those tasks, see what they share (a tool it misuses, a policy it misreads, a multi-hop lookup it cannot plan), and either improve that capability or route that class of request to a human. On the plus side, Agent B is predictable, which is often what a product team wants most.

This is also why averaging hides information. Benchmark-level pass^k is the mean of per-task pass^k, which is not the same as the benchmark-level pass@1 raised to the k-th power. For Agent A those happen to coincide (0.8^8 = 16.8%); for Agent B they are 80% versus 16.8%. Always compute pass^k per task first, then average.

Computing them correctly

You do not observe p directly. You run each task n times, count c successes, and estimate. Two mistakes are common here.

The first is computing pass@k by literally sampling exactly k attempts once. That estimate is extremely noisy. The Codex paper's approach is to draw n ≥ k samples per task and use an unbiased combinatorial estimator.

The second is plugging the observed success rate into a power, as in (c/n)^k for pass^k. Because the power function is convex, that plug-in estimate is biased upward on average: it systematically overstates reliability, and the bias is worst exactly where you care, on tasks with few runs and middling success rates. The unbiased versions use combinations instead:

from math import comb

def pass_at_k(n: int, c: int, k: int) -> float:
    """P(at least one success in k draws without replacement from n runs, c of them successful)."""
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n: int, c: int, k: int) -> float:
    """P(all k draws are successes): unbiased estimate of p**k."""
    if c < k:
        return 0.0
    return comb(c, k) / comb(n, k)

def benchmark_scores(results: dict[str, list[bool]], k: int) -> tuple[float, float]:
    """results maps task_id -> list of per-run outcomes; average per task, never pool runs."""
    at, hat = [], []
    for runs in results.values():
        n, c = len(runs), sum(runs)
        if n < k:
            raise ValueError("need at least k runs per task")
        at.append(pass_at_k(n, c, k))
        hat.append(pass_hat_k(n, c, k))
    return sum(at) / len(at), sum(hat) / len(hat)

For example, a task that succeeded on 7 of 10 runs gets pass^4 = C(7,4)/C(10,4) = 35/210 ≈ 16.7%, while the naive (0.7)^4 gives 24.0%. The naive number flatters the agent by about seven points on a single task. The Codex paper notes that its version uses a numerically stable product for large n; with Python's arbitrary-precision integers, math.comb is fine for the run counts agent evals actually use.

How many runs and tasks you need

Two sources of noise stack up, and repeated runs only fix one of them.

Running each task more times (bigger n) sharpens your estimate of each task's p. It does nothing about the fact that your 50 tasks are a sample of the tasks your users will send. With 50 tasks and an 80% pass@1, the standard error from task sampling alone is about 5.7 points, so a 95% interval spans roughly plus or minus 11 points. Two agents within that band are not distinguishable on that suite, no matter how many runs you do. At 500 tasks the interval tightens to about plus or minus 3.5 points.

A practical setup for an internal agent eval is n = 8 to 16 runs per task, reporting pass@1, pass^k at the k that matches your use case, and the task-level confidence interval. That multiplies eval cost by n, which is the real reason most public leaderboards report a single run.

Which metric matches your product

The right k and the right metric come from how the agent is deployed, not from what a leaderboard prints.

The pass@k row has a catch: it assumes the verifier is trustworthy. Best-of-k with a weak verifier, such as an LLM judge with known biases (see the five documented biases in LLM-as-judge evals), selects the most convincing wrong answer as often as the right one. Your effective success rate is bounded by the verifier's precision, not by pass@k. That is the same constraint that limits how far test-time compute scaling can go on tasks without a crisp check.

Reading leaderboards with this in mind

Most headline agent numbers are pass@1 from a single run per task. That is a reasonable capability signal, but it says nothing about consistency. When you read a score:

• Check how many runs per task were used. One run means the score blends model capability with sampling luck, task by task.
• Check whether the benchmark reports pass^k at all. τ-bench and its successors do. Most coding benchmarks report pass@1; the SWE-bench leaderboard is a resolved rate, typically from one submitted attempt per issue.
• Remember that "50% success" thresholds, like the one behind METR's task-length chart, are explicitly a coin-flip reliability level. A task an agent completes half the time has a pass^4 of about 6%.

Takeaway

Report pass@1, then add pass^k with k set to how many times the same kind of task hits your agent before anyone looks at the results, computed per task with the unbiased estimator. If pass^k falls much faster than pass@1 would suggest, your agent is flaky across the board and needs variance reduction. If it holds steady near pass@1, your failures are concentrated on specific tasks, and you need to find those tasks, not retry them. And if your eval suite is too small to tell those two cases apart, add tasks before you add runs.