A one-line system prompt tweak that 'fixes' one failure case can silently break three others — here's how teams are starting to treat prompts as versioned, tested artifacts instead of strings in a config file.
A support bot's system prompt said, roughly, "always confirm the order number before issuing a refund." Someone noticed it was also refusing to issue refunds when a customer didn't have an order number handy — a reasonable edge case, like a gift purchase. So they added a clause: "if the customer cannot provide an order number, look it up by email instead." Shipped it, closed the ticket, moved on.
Two weeks later, refund approvals for a completely unrelated flow — store credit, which had never mentioned order numbers — dropped by a third. The new clause had shifted the model's weighting of when to ask clarifying questions at all, and it started asking for emails in places it previously didn't. Nobody connected the two changes for four days, because the prompt diff was three lines and nobody tested anything outside the original edge case.
This is not a model problem. It's a process problem, and it's an extremely common one: teams edit prompts like they edit a README, when a system prompt that gates refund decisions is closer to a pricing engine. It has a public interface (the conversations it ever handles), hidden state (the model's probability landscape, which shifts in model-version-dependent and frequently non-local ways), and measurable failure modes. Code with that profile gets a test suite before it merges. Prompts, in most shops, still don't.
The reflexive objection is correct as far as it goes: you can't unit-test a prompt the way you unit-test a function, because there's no fixed output to assert against. generate_refund_response(order_id) doesn't return the same string twice. Three properties make prompt regressions structurally different from code regressions:
These properties don't mean testing is impossible — they mean testing has to be probabilistic and dataset-driven instead of assertion-driven, which is a different design than most engineers default to.
The teams doing this well converge on roughly the same four-stage shape, whether they're using an off-the-shelf tool like promptfoo, Braintrust, or LangSmith, or a homegrown harness around their model provider's API.
The box at the bottom matters as much as the top row: the same four-stage pipeline needs to fire when your model provider ships a new version, not just when a human edits the prompt text. A prompt that scored 94% on the regression set against one model can silently drop to 81% against its successor with zero lines changed — this is one of the most common ways "it was working fine yesterday" incidents happen in production LLM apps.
Three kinds of cases belong in the suite, and conflating them is the most common mistake:
A golden set with 40 cases and a regression set with 200 accumulated-bug cases is a perfectly reasonable starting point. The regression set tends to become more valuable than the golden set within a couple of months, because it encodes failures you've actually paid for once already.
Not every test case needs the same grading strategy, and picking the wrong one either produces false confidence or so much noise nobody trusts the suite:
| Grader type | Good for | Weak point |
|---|---|---|
| Exact / regex match | Structured output, tool-call arguments, refusal boundaries | Useless for open-ended prose |
| Rubric-based LLM judge | Tone, helpfulness, policy adherence in free text | Inherits judge biases — position, verbosity, self-preference |
| Human review | High-stakes or ambiguous cases, judge-calibration spot checks | Doesn't scale; reserve for a sampled subset, not every run |
The practical pattern is layered: exact-match and structural checks run on every commit because they're fast and deterministic; an LLM judge scores the prose-heavy cases against a written rubric (not a vibe); and a human reviews a random 5–10% sample weekly to catch judge drift before it compounds.
The part that actually prevents incidents like the refund example isn't the eval suite running — it's what happens to the scores. A single run tells you almost nothing in isolation; the useful signal is the delta against the last known-good baseline. Concretely:
refund-flow, tone, tool-call-format) so a regression surfaces as "refund-flow dropped 12 points," not as an unlocated three-point aggregate dip.A minimal promptfoo-style config for the refund scenario looks like this — note that it's checking structured fields (refund_type, the tool call shape) deterministically, and reserving the LLM judge for tone:
# refund-flow.promptfooconfig.yaml
prompts:
- file://prompts/refund_agent_system.txt
providers:
- id: anthropic:claude-sonnet-5
tests:
- description: "store credit refund, no order number mentioned"
vars:
user_message: "Can I get store credit for my last order? I don't have the order number."
assert:
- type: contains-json
value: { refund_type: "store_credit" }
- type: not-contains
value: "order number"
metric: no_spurious_order_number_request
- description: "refund request with valid order id"
vars:
user_message: "Refund order #A48291, it arrived damaged."
assert:
- type: contains-json
value: { refund_approved: true, order_id: "A48291" }
- description: "tone check — frustrated customer"
vars:
user_message: "This is the third time I've had to ask. Just refund it."
assert:
- type: llm-rubric
value: "Response acknowledges frustration without being obsequious, and does not repeat a policy explanation the customer has already heard."
That no_spurious_order_number_request assertion is the one that would have caught the regression from the opening story — it's a regression-set case added after the incident, which is exactly how these suites tend to grow: reactively at first, then increasingly proactively as the team learns which classes of change cause trouble.
The mechanics are less interesting than the discipline, but they matter: prompts should live in version control as their own files (not inlined as string literals scattered through application code), reviewed via pull request, and tagged with a version identifier that gets logged alongside every production completion. When something goes wrong in production, "which prompt version generated this" needs to be a join, not an archaeology project.
Two details trip people up:
A two-person team building on top of a model API doesn't need Braintrust or LangSmith — a promptfoo eval step in GitHub Actions, a regression YAML file that grows every time something breaks, and a Slack notification on score drops covers most of the risk for a fraction of the setup cost. The payoff usually shows up the first time it blocks a bad merge, which for an actively iterated prompt is typically a matter of weeks, not months.
At larger scale — multiple prompts, multiple models, a product surface where a regression affects real money or real users — dedicated observability and eval platforms earn their cost mainly through score history and category tagging at scale, not through anything a team couldn't build itself. The underlying four-stage shape is identical either way, and an agent iterating on prompts programmatically can drive the same eval loop through a REST API or MCP tool call just as a human would through a CLI.
Treat a production system prompt as a versioned artifact with a test suite, not as a string someone edits in place. The specific tooling matters far less than two habits: every bug becomes a permanent regression-test case, and every score is diffed by category against a baseline instead of read as a single aggregate number. Teams that adopt just those two habits catch the "fixed one thing, broke three others" class of incident before it reaches production, instead of four days after a customer notices.