Blog
Deep-dive guides on AI agents, agent orchestration, MCP, and developer tooling.
5 posts found
pass@k vs pass^k: The Metric That Tells You Whether an Agent Is Reliable
pass@k asks if an agent can solve a task in any of k tries; pass^k asks if it solves it every time. Two agents with the same pass@1 can need opposite fixes, and only pass^k shows which.
METR's Task-Length Benchmark: What "AI Can Now Work for Hours" Actually Means
METR's finding that AI task-completion time horizons double roughly every seven months is the most-cited capability chart in AI. Here's what the 50%-reliability metric behind it actually measures, and what it quietly leaves out.
LLM-as-Judge Evaluations Have Five Known Biases: Here's How to Design Around Them
Position bias, verbosity bias, and self-preference bias can silently distort agent eval scores. Here is how to design a judge pipeline that resists them, plus why trajectory evaluation catches what outcome scoring misses.
What SWE-bench Actually Measures vs. What the Leaderboard Tells You
SWE-bench is the most credible agentic coding benchmark available, but its leaderboard number answers a narrower question than most headlines imply. Here is exactly what it does and does not test.
Why Your Agent Benchmark Score Doesn't Predict Production Reliability
Benchmark leaderboards measure task completion under lab conditions, not the compounding step failures that sink agents in production. Here is the math, and a blueprint for an eval harness that actually predicts reliability.
New posts by email
One email when a new post goes up. No other mail; unsubscribe in one click.
We store your email only to send new posts. See the privacy policy.