Blog

Deep-dive guides on AI agents, agent orchestration, MCP, and developer tooling.

4 posts found

Evals & Benchmarks

pass@k vs pass^k: The Metric That Tells You Whether an Agent Is Reliable

pass@k asks if an agent can solve a task in any of k tries; pass^k asks if it solves it every time. Two agents with the same pass@1 can need opposite fixes, and only pass^k shows which.

November 14, 2026 · 10 min read

Evals & Benchmarks

METR's Task-Length Benchmark: What "AI Can Now Work for Hours" Actually Means

METR's finding that AI task-completion time horizons double roughly every seven months is the most-cited capability chart in AI. Here's what the 50%-reliability metric behind it actually measures, and what it quietly leaves out.

October 10, 2026 · 7 min read

Evals & Benchmarks

LLM-as-Judge Evaluations Have Five Known Biases: Here's How to Design Around Them

Position bias, verbosity bias, and self-preference bias can silently distort agent eval scores. Here is how to design a judge pipeline that resists them, plus why trajectory evaluation catches what outcome scoring misses.

August 25, 2026 · 10 min read

Evals & Benchmarks

Why Your Agent Benchmark Score Doesn't Predict Production Reliability

Benchmark leaderboards measure task completion under lab conditions, not the compounding step failures that sink agents in production. Here is the math, and a blueprint for an eval harness that actually predicts reliability.

July 1, 2026 · 7 min read

New posts by email

One email when a new post goes up. No other mail; unsubscribe in one click.

We store your email only to send new posts. See the privacy policy.