The same prompt in two Cherry Studio panes

Two answers beside each other are easier to inspect than two answers remembered from separate conversations. Cherry Studio is useful for this kind of comparison: send the same task to different models, keep their responses visible, and read for specific differences. The useful result is not a universal winner. It is a clearer choice for a recurring piece of work.

§ Start with one connection and two models

Vana API provides one OpenAI-compatible endpoint at https://api.vanaapi.com. In Cherry Studio, configure an OpenAI-compatible provider with your Vana API key and the API address format that your installed version expects. Check whether the address field expects a versioned base URL before appending a path; duplicated version segments can cause requests to fail.

Add the model identifiers shown in your Vana API account rather than guessing them from display names. For a first comparison, you might choose Claude Sonnet 5.5 and GPT-6.1 Sol. Those are candidates to evaluate, not a prediction about which will perform better.

Use Cherry Studio’s multi-model conversation controls to select the pair. Labels and response layouts can vary by release. If your version does not offer the arrangement you need, use two fresh conversations and submit identical prompts. The important property is matched input, not a particular screen layout.

§ Make the two sides comparable

A comparison becomes difficult to interpret when one side has extra instructions, previous messages, or access to tools that the other lacks. Start with fresh histories and match the settings that both models support. A matching temperature value does not guarantee identical sampling behavior, but recording it still makes the setup easier to revisit.

• Left side: the task, source material, system instructions, and output requirements.
• Right side: exactly the same material, without an extra hint added after reading the first answer.
• Both sides: equivalent tool access, a sufficient output limit, and no unrelated conversation history.

If you revise the prompt, rerun both sides. Otherwise, you are comparing a model change and a prompt change at the same time.

§ Read paired evidence, not overall impressions

Choose a task with an answer you can inspect. For example, provide a short incident report and request a summary, unresolved questions, and a customer-facing update. Include one important uncertainty and one explicit formatting constraint. This exposes differences that polished prose can hide.

• Accuracy, left and right: does each response preserve the source facts and distinguish an observation from an inference?
• Instruction following, left and right: does each include every requested section and respect the stated length?
• Uncertainty, left and right: does either turn a possible cause into a confirmed explanation?
• Editing effort, left and right: what would you actually change before sending the result?

Write down concrete examples. “Added an unsupported recovery time” is more useful than “felt less reliable.” Do not reward a longer response simply because it contains more detail. Extra text can also introduce extra claims to check.

§ Keep speed and cost in separate columns

Perceived speed and total completion time are different measurements. A response that starts quickly may finish later. Network conditions, provider load, and output length can also affect a single run, so repeat a few representative tasks before drawing conclusions.

Vana API bills per real token with no monthly fee. Comparing two models creates two sets of usage, and repeated trials add to that total. Check recorded usage and the applicable rates rather than estimating cost from answer length alone. A cheaper response may still require more editing; an expensive one may offer no practical improvement for a simple task.

§ Keep a small comparison notebook

Save the prompt, model identifiers, date, settings, usage, and a short verdict for each side. Repeat the exercise across several tasks you genuinely perform. Prefer a narrow conclusion, such as “needs fewer factual corrections on these incident summaries,” over a general ranking.

To try this setup, create a Vana API account and connect it to Cherry Studio. Start with two models and one well-defined task, then keep the comparison only as elaborate as the decision requires.