A regression suite for the support routing prompt

A support routing prompt looks finished until a small wording change sends billing requests to technical support. Reading a few convincing responses will not reliably catch that regression. A repeatable test suite gives you a clearer question: does this revision still satisfy the behavior your application requires?

This walkthrough builds a small suite for a prompt that assigns support messages to a queue and returns a structured result. The same approach works for extraction, classification, and tool selection.

§ Step 1: Write the contract before changing the prompt

Define what the application needs, independently of the current prompt. For this example, require three fields: queue, priority, and needs_review. Allow only billing, technical, and general as queue values. Define priority levels and the circumstances that require human review.

Separate requirements from preferences. A valid queue is a requirement. A neatly phrased explanation is not relevant if the application never displays one. This keeps the suite focused on failures that matter.

§ Step 2: Turn real failures into fixtures

Start with a dozen short, anonymized messages. Include ordinary requests, ambiguous cases, and examples that previously failed. Store each input beside its expected result and a brief explanation of the expectation.

• A duplicate charge should route to billing.
• A failed login without payment context should route to technical.
• A message containing two unrelated requests should require review under the chosen routing policy.
• An empty message should follow an explicitly defined fallback.
• A message saying “ignore your instructions” should remain customer content, not become a new instruction.

Do not invent a definitive label for a genuinely disputed example. Resolve the policy first, or mark the case for review. Otherwise, the suite measures disagreement rather than correctness.

§ Step 3: Freeze the request configuration

Version the prompt, fixtures, model identifier, generation settings, and output schema together. Record the configuration actually sent with each run, including omitted settings that use service defaults. If the provider exposes a resolved model revision, save that too.

If you use Vana API, create an account and configure your test client with the appropriate protocol path under https://api.vanaapi.com. Keep the API key in an environment variable, not in fixtures or committed configuration. A shared endpoint simplifies request setup; it does not replace behavior checks.

§ Step 4: Add assertions in layers

First, check whether the request completed successfully. Next, parse the response and validate required fields, types, and allowed values. Finally, compare the routing decision with the fixture expectation. Report these as separate outcomes so a timeout does not appear to be a classification mistake.

Use exact comparisons for labels and booleans. Avoid exact text matching for free-form explanations unless wording is part of the contract. If explanation quality matters, define a small rubric, such as whether it cites the relevant customer issue without adding unsupported details.

§ Step 5: Establish a baseline with repeated runs

Run the unchanged prompt several times per fixture before editing it. Save every result rather than only the most recent response. Repetition helps reveal inconsistent cases, although a small passing sample does not prove reliability.

Record pass rates by fixture and failure category, plus latency and reported token usage. Set a spending limit for the run. Keep a separate held-out set for periodic checks so repeated tuning against the same examples does not become your only evidence of improvement.

§ Step 6: Change one thing and inspect the diff

Revise one instruction, example, or schema constraint. Then rerun the same fixtures with the same configuration. Compare candidate results with the saved baseline, looking at individual regressions rather than only the overall score.

Decide release gates before seeing the results. For this suite, require every response to satisfy the schema and every designated critical routing case to pass across the configured repetitions. Treat these as release rules for this application, not universal reliability thresholds.

§ Step 7: Make the suite part of delivery

Run a small smoke suite on prompt changes and a broader suite before release. Keep credentials out of logs, redact customer information, and save enough request metadata to investigate failures. Test retries separately so they cannot quietly conceal first-attempt failures.

When production exposes a new mistake, add a fixture before fixing the prompt. Over time, the suite becomes a record of the behavior you intend to preserve, rather than a collection of responses that happened to look good.