Blog

Deep-dive guides on AI agents, agent orchestration, MCP, and developer tooling.

4 posts found

LLM Engineering

vLLM, SGLang, or TensorRT-LLM: What Actually Differs When You Self-Host an LLM

vLLM, SGLang, and TensorRT-LLM all promise fast self-hosted inference, but they make different bets on KV cache management, structured output, and hardware lock-in. Here's what actually changes depending on your traffic pattern.

October 8, 2026 · 9 min read

LLM Engineering

How Constrained Decoding Actually Forces an LLM to Emit Valid JSON

Structured outputs and strict tool use aren't prompting tricks. They mask invalid tokens at the decoding layer so malformed JSON becomes mechanically impossible, not just less likely.

August 24, 2026 · 10 min read

AI News & Policy

The Memory Wall: Why On-Device LLM Inference Is Bottlenecked by Bandwidth, Not Compute

Your phone's NPU isn't the reason local LLMs feel slow; the memory bus is. Here's why token generation is bandwidth-bound, and why quantization, KV-cache tricks, and unified memory are all attacks on the same wall.

July 18, 2026 · 9 min read

LLM Engineering

KV Cache Reuse and the Hidden Latency Budget of Agent Loops

Agent latency is usually a prompt-construction problem in disguise: prefix caching, continuous batching, and tool-call structure determine how much of your context gets reprocessed on every turn.

July 2, 2026 · 8 min read

New posts by email

One email when a new post goes up. No other mail; unsubscribe in one click.

We store your email only to send new posts. See the privacy policy.