A smaller model for the overnight queue

An overnight queue often contains work that matters without needing a large model: sorting support messages, labeling product descriptions, extracting dates from documents, or drafting short internal summaries. Each task is small. Repeated across thousands of records, though, the token bill becomes worth examining.

A cheaper model can be a sensible starting point for this work. The goal is not to find the lowest advertised price and send everything there. It is to find a model that completes a clearly defined task reliably enough that corrections and retries do not erase the savings.

§ Start with a narrow job

A background job runs outside the immediate conversation with a person. Your application puts work in a queue, and a worker processes it later. Because nobody is waiting for each answer on screen, you usually have more room to test, retry, or send uncertain results for review.

Start with something easy to check. Assigning a support ticket to one of six categories is easier to evaluate than asking for an open-ended business recommendation. Extracting an invoice date is easier to validate than producing a polished customer response.

A useful first task has clear inputs, a limited output, and an obvious definition of success. If you cannot explain what a correct result looks like, changing models will be difficult to evaluate.

§ Compare the cost of finished work

Model pricing is commonly based on input and output tokens: the pieces of text sent to the model and returned by it. A long instruction repeated for every record can become a meaningful part of the bill. So can an answer that includes several paragraphs when your application only needs a category label.

Keep instructions concise, provide only relevant context, and request the smallest useful response. For structured extraction, specify the required fields and explain what to return when information is missing.

When evaluating candidates such as Gemini 2.5 Flash or DeepSeek V4 Flash, check current rates rather than assuming a name guarantees a particular cost. Then measure the cost per usable result, including any extra attempts.

• Record input and output token usage.
• Count results that pass your quality checks.
• Include retries, review time, and fallback calls in the comparison.
• Check whether processing speed keeps up with the queue.

§ Test before moving the whole queue

Choose a small sample of real records, including messy examples. Include missing fields, unusually long inputs, ambiguous wording, and any languages your application needs to handle. Remove sensitive information where possible.

Write down the expected result for each record. Run the same sample through your candidate model and inspect the answers. A response can be valid JSON and still contain the wrong category or an invented date, so format checks alone are not enough.

Decide what level of error the job can tolerate. Internal search labels may allow occasional mistakes. Financial records or messages sent directly to customers need stricter checks and may require human review.

§ Give failures a separate path

Do not retry every bad answer indefinitely. Set a retry limit, validate required fields, and send unresolved records to a review queue or a more capable model. For temporary connection errors or rate limits, wait before retrying rather than immediately repeating the request.

Keep generation separate from irreversible actions. A worker can propose a label or draft a message without automatically deleting a record or sending an email. This makes early experiments easier to inspect and undo.

§ Keep the connection simple

Vana API provides one OpenAI-compatible endpoint, allowing a worker built with an OpenAI SDK to try different supported models without building a separate integration for each provider. It also supports the Anthropic Messages and Gemini native protocols. Usage is billed per real token, with no monthly fee.

Create a Vana API account when you are ready to test a small sample. Keep your existing workflow intact, compare usable results, and move more of the queue only after the measurements support it. A cheaper background job is one that finishes correctly at a lower total cost.