A replacement model in the staging queue

A model change starts with a small question: will the replacement handle the work this application actually receives? A convincing demonstration is not enough. Before switching production traffic, put the current model and the candidate through the same cases, with the same acceptance rules.

For example, if you are considering Claude Opus 5.5 for a workflow currently using Claude Sonnet 5.5, treat the replacement as a hypothesis, not an upgrade already established. The useful comparison is between two configured systems doing your work, not two names on a model list.

§ Put the two runs on equal footing

Build a small evaluation set from representative requests, with sensitive information removed. Include routine tasks, long inputs, ambiguous instructions, malformed data, and cases where the correct response is to ask a question or decline an action. Keep a separate holdout set for the final check.

• Current model: Save the production prompt, request settings, tool definitions, and outputs as the baseline.
• Candidate model: Start with the same inputs and compatible settings. Record any unsupported parameters rather than silently dropping them.

Vana API provides one endpoint at https://api.vanaapi.com with OpenAI-compatible access, alongside Anthropic Messages and Gemini native protocols. That can reduce integration changes, but shared access does not make model behavior identical. Keep the protocol stable where possible so the first comparison isolates the model change.

§ Compare correctness before style

Write acceptance criteria before reading the results. For a support reply, these might include citing the supplied policy, avoiding invented refund terms, and requesting missing order information. For extraction, check required fields and exact values. A polished answer can still fail the task.

• Current model: Count accepted outputs, factual errors, missing requirements, and unnecessary refusals.
• Candidate model: Apply the same rubric, preferably with reviewers unaware of which model produced each answer.

Review disagreements individually. A single aggregate score can hide a regression in a rare but consequential task. Repeat important cases several times to expose inconsistent behavior; one successful response is not evidence of reliability.

§ Exercise the contract around the answer

The application consumes more than prose. It may depend on structured output, tool arguments, streaming events, or predictable handling of an empty result. Test those boundaries explicitly.

• Current model: Record schema validation failures, invalid tool arguments, repeated tool calls, and recovery after a tool error.
• Candidate model: Run the same checks, including interrupted streams and requests near the application's input limits.

Use a sandbox for actions with side effects. Replaying production requests must not send duplicate emails, charge customers, or modify live records. If the candidate needs a different prompt, preserve that as a separate experiment so its benefit is not confused with a drop-in replacement.

§ Measure time and cost on completed work

Measure from the application's perspective. For interactive work, time to first output matters; for background jobs, completion time may matter more. Track median and tail latency under representative concurrency, not just one quiet request.

• Current model: Record input and output tokens, retries, total elapsed time, and the cost of accepted results.
• Candidate model: Include the same measurements, including extra attempts needed to produce a usable answer.

Vana API bills per real token with no monthly fee. Compare actual usage at the applicable rates rather than assuming a shorter answer or lower advertised rate means a cheaper completed task. Keep failed attempts in the accounting.

§ Make the switch reversible

Decide the release gate in advance: no critical regressions, an acceptable task pass rate, and latency and cost within your budget. Run the holdout set only after tuning is finished.

Then move a small share of eligible traffic, monitor the same measures, and retain an explicit route back to the current model. Define who can roll back and what triggers that decision.

Create a Vana API account when you are ready to run the comparison. The useful outcome is not necessarily a replacement: keeping the current model, routing only certain tasks to the candidate, or postponing the change are all valid results of a careful test.