Blog
Deep-dive guides on AI agents, agent orchestration, MCP, and developer tooling.
pass@k vs pass^k: The Metric That Tells You Whether an Agent Is Reliable
pass@k asks if an agent can solve a task in any of k tries; pass^k asks if it solves it every time. Two agents with the same pass@1 can need opposite fixes, and only pass^k shows which.
Most read this week
- 1AI News & PolicyThe Memory Wall: Why On-Device LLM Inference Is Bottlenecked by Bandwidth, Not Compute
- 2AI News & PolicyDeepSeek, Profiled: How a Quant Hedge Fund's AI Lab Rewrote the Scaling Debate
- 3AI News & PolicyRubin, Ironwood, and the Groq LPU Aren't Competing for the Same Job
- 4AI AgentsHow Background Coding Agents Actually Work: Sandboxes, Diffs, and the PR Review Bottleneck
LiteLLM vs. OpenRouter vs. Portkey: What an LLM Gateway Actually Does Between Your App and the Model
LiteLLM, OpenRouter, and Portkey all promise one API in front of every model provider. Here is how their routing, caching, and failover architectures actually differ once you look past the pitch.
DeepSeek, Profiled: How a Quant Hedge Fund's AI Lab Rewrote the Scaling Debate
DeepSeek's R1 release wiped roughly $589 billion off Nvidia's market cap in a single trading day by suggesting frontier-level reasoning didn't require frontier-level spend. Here's what the company actually built, what its cost claims do and don't mean, and why its chip workarounds matter more than its benchmark scores.
Prompt Versioning and Regression Testing: Why Prompts Need a CI Pipeline Too
A one-line system prompt tweak that 'fixes' one failure case can silently break three others. Here's how teams are starting to treat prompts as versioned, tested artifacts instead of strings in a config file.
Context Engineering: What Changed When Prompt Engineering Outgrew the Prompt
Prompt engineering asked what to say to the model. Context engineering asks what the model actually needs to see, in what order, and what gets thrown away to make room for it: the discipline agents forced into existence.
METR's Task-Length Benchmark: What "AI Can Now Work for Hours" Actually Means
METR's finding that AI task-completion time horizons double roughly every seven months is the most-cited capability chart in AI. Here's what the 50%-reliability metric behind it actually measures, and what it quietly leaves out.
vLLM, SGLang, or TensorRT-LLM: What Actually Differs When You Self-Host an LLM
vLLM, SGLang, and TensorRT-LLM all promise fast self-hosted inference, but they make different bets on KV cache management, structured output, and hardware lock-in. Here's what actually changes depending on your traffic pattern.
Claude Haiku 4.5 vs. GPT-5.4 Mini vs. Gemini 3.8 Flash: The Economics of Cheap Inference
The three labs' cheap-tier models now sit within a quarter-dollar of each other per million input tokens, so the real differences are in release cadence, cache economics, and how much of your "cheap" budget goes to invisible reasoning tokens.
Fixed-Size, Semantic, or Late Chunking: Where RAG Retrieval Actually Breaks
Most RAG recall failures are decided at the chunking step, long before retrieval or reranking. Here is how fixed-size, semantic, and late chunking actually differ, and when each one silently loses the fact you need.
Rubin, Ironwood, and the Groq LPU Aren't Competing for the Same Job
The 2026 AI chip race looks like one contest on a spec sheet, but Nvidia's Rubin, Google's Ironwood, and Groq's LPU are actually built for three different jobs. Comparing their PFLOPS numbers head-to-head misses why each one wins where it does.
Test-Time Compute Scaling: Why Letting a Model "Think Longer" Sometimes Beats Making It Bigger
A 2024 DeepMind study found that spending more compute at inference time, not training time, can let a small model match one many times its size on math problems. Here's the mechanism, and why it only works in a narrow band of difficulty.
Colorado, California, Texas: Mapping the 2026 US State AI Law Patchwork
Texas bans specific AI uses, California regulates frontier model developers, and Colorado rewrote its flagship AI law under litigation pressure. Here's what each actually requires, and why Trump's preemption executive order hasn't changed any of it yet.
Where AI Actually Works in Telecom: Self-Organizing Networks, Robocall Detection, and the RAN Latency Problem
From antennas that retune themselves in milliseconds to the scam-call filter that silently screens your phone, telecom runs some of the largest and least forgiving ML deployments in production. Almost none of it looks like a chatbot.
Perplexity's Browser Bet: Why It Doubled Down on Comet the Same Summer OpenAI Killed Atlas
When OpenAI shut down its ChatGPT Atlas browser in August 2026, Perplexity was raising money at a $30B+ valuation to build Comet into the opposite bet: that the AI browser, not the chat window, is where the agent economy actually lives.
Compound vs. Simple Interest: Why the Same Rate Can Mean Different Money
Two loans with the identical 6% rate can differ by thousands of dollars in total interest. The gap comes entirely from how (and how often) that interest compounds, worked through with real numbers.
Designing Sane Pagination: Why Offset Breaks Under Writes and Keyset Fixes It
LIMIT/OFFSET pagination silently duplicates and skips rows the moment writes happen concurrently, and gets slower the deeper a client pages. Here is why, and how keyset (cursor) pagination avoids both problems.
Where AI Actually Works in Gaming: Voice Moderation, NPC Dialogue, and the Procedural Generation Reality Check
Call of Duty already runs AI on live voice chat to catch harassment in real time, and Microsoft trained a world model on years of Xbox gameplay. But the 'AI will write your NPCs and build your levels' pitch is running years ahead of what's actually shipping.
How Background Coding Agents Actually Work: Sandboxes, Diffs, and the PR Review Bottleneck
Codex, Copilot's cloud agent, Cursor's cloud agents, Jules, Devin, and cloud Claude Code sessions all follow the same five-stage pipeline. The differences that matter are in sandbox scope, repo access, and where the real trust boundary sits.
Where AI Actually Works in Marketing: Programmatic Bidding, Generative Creative, and the Attribution Crisis
From Performance Max's black-box bidding to Coca-Cola's uncanny AI holiday ad, marketing bet on AI years before it had reliable measurement to prove any of it works. Here's where it actually delivers, and where attribution is still the unsolved problem.
Where AI Actually Works in Government: Citizen Chatbots, Fraud Detection, and the Due-Process Problem
Government AI splits into two very different tracks: citizen-facing chatbots that have to be boring and accurate, and back-office risk-scoring systems that have already produced real due-process failures. Here's how each actually works today.
Where AI Actually Works in Real Estate: The Zestimate Reckoning, AI Leasing Agents, and the New Disclosure Laws
Zillow lost over $500 million learning that a pricing model is not a trading strategy. Five years later, AI has quietly taken over apartment leasing and lease abstraction, while a new wave of state laws forces agents to label every AI-touched photo.
Langfuse vs. Braintrust vs. Arize Phoenix: Picking an LLM Observability Stack in 2026
Three tools dominate LLM/agent observability right now, each built around a different bet: open self-hosting, eval-driven CI/CD, or OpenTelemetry portability. Here is what the pricing table does not tell you.
Where AI Actually Works in Energy: Forecasting, Digital Twins, and the Datacenter That Learned to Negotiate With the Grid
AI is already reshaping the power sector, but the clearest wins are boring forecasting and maintenance systems, not the AI-controls-the-grid story. The newest twist is AI datacenters themselves being turned into flexible grid assets.
MCP Sampling: How a Server Asks the Client's Model to Think for It
Sampling is the MCP primitive that inverts the relationship, letting a server without its own model borrow the client's LLM through a two-gate approval flow. Here is how it works, and why the 2026-07-28 spec revision deprecated it.
LangGraph, CrewAI, OpenAI Agents SDK, Claude Agent SDK: Four Different Bets on How Agents Should Talk to Each Other
Four multi-agent frameworks, four incompatible answers to the same question: when one agent hands work to another, what happens to the context? The choice determines what breaks in production.
Where AI Actually Works in Logistics: Route Optimization, Driverless Freight, and the Data-Quality Wall
Driverless trucks are now billing customers on real Texas highways, not just demoing for reporters. Here's what's actually shipped in logistics AI versus what's still a slide deck.
Letta, Mem0, and LangGraph Stores: Three Different Answers to Where Agent Memory Lives
Letta, Mem0, and LangGraph checkpointers all get called "agent memory," but they put the decision to remember or forget in three different places: an agent runtime, an external service, or your own application code.
Where AI Actually Works in Hiring: Resume Screening, Interview Scoring, and the Bias Audit Laws Catching Up to It
From Workday's algorithmic screening lawsuit to NYC's mandatory bias audits, AI hiring tools are now mature enough to be regulated, and the regulations are starting to bite.
LLM-as-Judge Evaluations Have Five Known Biases: Here's How to Design Around Them
Position bias, verbosity bias, and self-preference bias can silently distort agent eval scores. Here is how to design a judge pipeline that resists them, plus why trajectory evaluation catches what outcome scoring misses.
Where AI Actually Works in Retail: Cashierless Checkout's Walkback, Demand Forecasting, and the Personalization Ceiling
Amazon quietly pulled Just Walk Out from most of its Fresh stores in 2024, which makes a good starting point for separating retail AI that actually works from the version in the marketing copy.
Where AI Actually Works in Insurance: Claims Photos, Underwriting Models, and the NAIC Compliance Gap
AI in insurance is furthest along in the parts of the workflow farthest from a payout decision, like damage photos and fraud flags. It stalls hard exactly where a model output decides who gets paid or covered.
YAML's Whitespace Footguns: Why the Norway Problem and Tab Indentation Keep Breaking Configs
YAML's indentation-as-syntax and implicit type inference make it readable to write and dangerous to get subtly wrong. Here's why 'NO' becomes false, why tabs are banned, and why 1.10 silently becomes 1.1.
Webhook Signature Verification: How Stripe, GitHub, and Svix Actually Stop Forged Events
A webhook endpoint is just a public URL that runs code when it receives a POST, so anyone who finds it can send you fake events unless you verify the signature correctly, including the two checks most implementations skip.
JSON Schema in Practice: What allOf, oneOf, and additionalProperties Actually Do
JSON Schema looks like static typing for JSON, but its composition keywords follow evaluation rules that trip up most people writing their first real-world schema. Here's the mental model that actually holds up.
AI Copyright Litigation, Mapped: Where Fair Use Actually Stands After the Anthropic Settlement
A field guide to the AI training-data lawsuits that have actually produced rulings (Thomson Reuters v. Ross, Bartz v. Anthropic, Kadrey v. Meta, Getty v. Stability) and what they do and don't establish for anyone building on top of a foundation model.
ROI vs. IRR vs. Payback Period: Why the "Best" Investment Depends on Which One You Ask
Three investments, three different "winners" depending on which metric you compute. Here is the arithmetic behind ROI, IRR, and payback period, and exactly when each one lies to you.
What SWE-bench Actually Measures vs. What the Leaderboard Tells You
SWE-bench is the most credible agentic coding benchmark available, but its leaderboard number answers a narrower question than most headlines imply. Here is exactly what it does and does not test.
Where AI Actually Works in Scientific Research: Protein Design, Materials Discovery, and the GNoME Lesson
AlphaFold's Nobel Prize and GNoME's 2.2 million materials both got real headlines, but the honest story is about candidate generation, not discovery, and the gap between those two claims is where the hype lives.
Claude Sonnet 5 vs. GPT-5.6 vs. Gemini 3.1 Pro: What the Pricing Table Doesn't Tell You
A side-by-side look at pricing, context windows, and tiering across the three frontier model families as of August 2026, and why the spec sheet undersells how differently they actually behave in agent loops.
Margin vs. Markup: The Pricing Mistake That Quietly Eats Small-Business Profit
A 25% markup and a 25% margin are not the same number, and mixing them up systematically underprices every product you sell. Here's the actual math, a worked example, and why the confusion never seems to die.
Why Your Email Regex Is a Denial-of-Service Waiting to Happen
Catastrophic backtracking turned a routine validation regex into Cloudflare's 2019 global outage. Here is the exact pattern shape that causes it, why email regexes are especially prone to it, and which engines make the bug impossible.
Mistral AI, Profiled: The Product Isn't the Model, It's the Deployment Terms
ASML led a ~€1.7B round in Mistral AI in 2025: an industrial policy trade dressed as a venture round. A grounded look at what the Paris lab actually ships, why sparse MoE was the right asymmetric bet, and where the open-weight-plus-sovereignty strategy is fragile.
Your Model's Reasoning Trace Is Not a Log: What Chain-of-Thought Faithfulness Research Actually Shows
Reasoning models write confident step-by-step explanations that often omit the thing that actually changed their answer. Here's the experimental evidence, the mechanism behind it, and which of your uses of chain-of-thought survive it.
JWT vs Session Cookies: The Trade Isn't Statelessness, It's Revocation
"JWTs are stateless, so they scale better" is the sentence that leads teams straight into the logout-doesn't-work bug. Here's the real trade-off, the revocation problem it creates, and when each auth model is actually the right call.
Base64 vs Base32 vs Base58: Why Bitcoin Doesn't Use the Encoding Your API Does
Base64, Base32, and Base58 all turn bytes into printable text, but they solve different problems. Two chop the bitstream; one converts the whole thing as a big number, and that single distinction decides which belongs in front of a human.
Where AI Actually Works in Education: Tutoring, Grading, and the Detection Problem That Won't Die
AI tutoring, autograding, and language apps are shipping to millions of students, while AI-cheating detectors keep getting disabled. A grounded look at what works, what doesn't, and the boundary that explains both.
How Amortization Actually Works: Why Two Loans at the Same Rate Cost Different Amounts
Two borrowers with identical 6% mortgages can pay $156k or $348k in interest. The rate isn't the culprit; the term and the amortization schedule are, and here's the math with real numbers.
The Memory Wall: Why On-Device LLM Inference Is Bottlenecked by Bandwidth, Not Compute
Your phone's NPU isn't the reason local LLMs feel slow; the memory bus is. Here's why token generation is bandwidth-bound, and why quantization, KV-cache tricks, and unified memory are all attacks on the same wall.
Idempotency Keys: Why 'Just Retry the Request' Breaks in Production
A dropped connection makes a client retry a POST that already succeeded, and without an idempotency key that retry becomes a duplicate charge, not a safety net. Here's how the pattern actually works and where naive implementations fail.
NDJSON vs JSON Arrays for Streaming: Why the Trailing Bracket Is the Problem
A JSON array requires seeing the closing bracket before any of it is valid, which is exactly what breaks when you stream millions of records. Newline-delimited JSON fixes that by making every line a complete, independent document.
How AI Coding Assistants Actually Work: Context, Diffs, and Permission Boundaries
AI coding assistants aren't one architecture. They're three separable design decisions (how context gets assembled, how edits get applied, how execution gets contained) that every tool from Claude Code to Cursor answers differently.
Prompt Engineering for Agents Is a Different Discipline Than Prompt Engineering for Chat
In agent systems, instructions live across four surfaces (system prompt, tool schemas, tool results, and few-shot text), not one. Most prompt debugging still only looks at the first.
Naive RAG, Agentic RAG, and GraphRAG: What Actually Changes Architecturally
RAG is not one architecture; it is three structurally different systems with different costs and failure modes. Here is what actually changes between naive, agentic, and graph-based retrieval, and how to pick without over-building.
The EU AI Act's High-Risk Rules Moved to December 2027: What Actually Changes for Builders
The EU AI Act high-risk obligations were due on August 2, 2026, until the AI Omnibus moved Annex III to December 2, 2027. Here is what is required, of whom, and how it compares with the US, UK, and China.
Why Your Agent Benchmark Score Doesn't Predict Production Reliability
Benchmark leaderboards measure task completion under lab conditions, not the compounding step failures that sink agents in production. Here is the math, and a blueprint for an eval harness that actually predicts reliability.
Agent Memory Isn't RAG: Why Vector Retrieval Falls Apart for Stateful Agents
Vector similarity search answers what text is topically related, but long-running agents need to know what is true right now. Conflating the two is why agents keep resurrecting overturned decisions.
New posts by email
One email when a new post goes up. No other mail; unsubscribe in one click.
We store your email only to send new posts. See the privacy policy.