Blog

Deep-dive guides on AI agents, agent orchestration, MCP, and developer tooling.

2 posts found

Evals & Benchmarks

pass@k vs pass^k: The Metric That Tells You Whether an Agent Is Reliable

pass@k asks if an agent can solve a task in any of k tries; pass^k asks if it solves it every time. Two agents with the same pass@1 can need opposite fixes, and only pass^k shows which.

November 14, 2026 · 10 min read

Evals & Benchmarks

Why Your Agent Benchmark Score Doesn't Predict Production Reliability

Benchmark leaderboards measure task completion under lab conditions, not the compounding step failures that sink agents in production. Here is the math, and a blueprint for an eval harness that actually predicts reliability.

July 1, 2026 · 7 min read

New posts by email

One email when a new post goes up. No other mail; unsubscribe in one click.

We store your email only to send new posts. See the privacy policy.