How to Measure Coding Agent Cost per Task

A coding agent rarely finishes a task in one model call. It reads files, proposes changes, runs tools, inspects failures, and tries again. Measuring only the first response misses much of the bill. This walkthrough shows how to measure the API cost of a complete task and compare configurations without confusing cheap attempts with useful results.

§ Step 1: Define a task and its finish line

Choose a concrete unit of work: fix one bug, add one endpoint, or migrate one component. Write down the acceptance criteria before starting. For example: the regression test passes, existing tests remain green, and the change passes review.

Give each task a unique identifier. Keep it attached to every attempt, including abandoned runs and manual restarts. Otherwise, a failed first attempt and a successful second attempt can look like two unrelated jobs.

§ Step 2: Set the measurement boundary

Start the clock when the agent receives the task. Stop when the result meets your acceptance criteria or reaches a predefined limit. Set limits for elapsed time, spend, or retries so that unsuccessful tasks have a consistent endpoint.

Separate model API spend from other costs. Shell commands and tests may add no model tokens themselves, but their output can become input to the next request. Hosted execution, external services, and human review may add separate charges or labor costs.

• API cost: charges for all model requests associated with the task.
• Execution cost: billable compute and external tools.
• Human effort: time spent steering, reviewing, and repairing the result.

§ Step 3: Capture every model request

Record a task ID, attempt ID, request timestamp, actual model identifier, token usage, and request outcome. Include planning calls, delegated work, summarization, retries, and final verification. If the agent uses several models, record each separately.

Vana API offers one OpenAI-compatible endpoint and also supports Anthropic Messages and Gemini native protocols. It works with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients. This can keep provider access consistent, but it does not automatically make every client expose identical usage data. Check which usage fields your client preserves, especially with streaming.

Before a larger benchmark, create a Vana API account and run one small task to verify your usage capture. Vana API bills per real token with no monthly fee, so you can begin with a bounded experiment.

§ Step 4: Convert usage into money

Use the rates applicable to the actual model and request at the time of the run. Store those rates with your measurements; do not recalculate historical runs using today's prices.

For a simple rate structure, request cost = input tokens × input rate + output tokens × output rate. If prices are quoted per million tokens, divide each token count by 1,000,000 first.

Where billing distinguishes cached input or other token categories, calculate each category separately. Avoid double-counting cached tokens if they are already included in an input total. Use billing records to reconcile your calculation where available, rather than assuming every attempted request was charged.

§ Step 5: Aggregate by task, not by response

Add the cost of every billable request under the same task ID. Keep failed attempts in the total. If a task costs $0.18 on its first attempt and $0.27 on a successful retry, its API cost is $0.45.

Track unresolved tasks too. Across a benchmark, API cost per successful task = total API spend across all tasks ÷ number of successful tasks. This includes money spent on failures. Also report success rate and median task cost so the aggregate does not hide inconsistent behavior.

§ Step 6: Compare controlled configurations

Run the same representative tasks from the same repository state with the same acceptance checks, permissions, and retry limits. Change one variable at a time, such as model, context policy, or agent instructions. Repeat runs where practical because agent behavior varies.

A lower token price does not guarantee a lower task cost. Extra turns, repeated file reads, or unsuccessful patches can outweigh the rate difference.

§ Step 7: Make cost a quality-aware metric

Keep a compact report containing task outcome, API spend, elapsed time, retries, and review effort. Investigate expensive outliers before changing defaults. The useful target is not the cheapest response: it is an accepted result at a predictable cost.