Compare Models Side by Side in Cherry Studio

Cherry Studio is a desktop client for working with multiple AI models. It is useful for comparing how different models handle the same task without writing a separate integration for each provider. This walkthrough shows how to connect Vana API, prepare a controlled comparison, and evaluate the results beyond which answer sounds better.

§ 1. Prepare your API access

Create a Vana API account, then obtain an API key from your dashboard. Keep the key private: enter it in the client’s provider settings, not in a conversation or a shared screenshot.

Vana API provides one OpenAI-compatible endpoint at https://api.vanaapi.com, so you can compare models through one provider configuration. It also supports Anthropic Messages and Gemini native protocols, but the OpenAI-compatible connection is the straightforward route for this setup.

§ 2. Add Vana API to Cherry Studio

Open Cherry Studio’s settings and find the model provider section. Add a custom provider using the OpenAI-compatible provider type. Exact labels can differ between Cherry Studio versions.

• Provider name: Use Vana API so the connection is easy to identify.
• API key: Paste the key from your dashboard.
• API address: Use https://api.vanaapi.com as the service endpoint. If the field expects a versioned OpenAI base URL, use https://api.vanaapi.com/v1. Check whether Cherry Studio appends /v1 automatically before adding it yourself.

Save the configuration and use the connection check if your version provides one. If a request fails, verify the key, account balance, and address format before changing other settings. A duplicated /v1 segment can cause a routing error.

§ 3. Add two models to compare

Use the provider’s model discovery option if available. Otherwise, add models manually using the exact model identifiers shown in Vana API’s current model list. A display name is not necessarily the identifier required by the API.

For a focused first comparison, choose Claude Sonnet 5.5 and GPT-6.1 Sol. Enable both under the Vana API provider and send each a short test message. Confirm that both return a response before starting your evaluation.

§ 4. Build one repeatable test prompt

Choose a task that resembles your actual work. A practical comparison needs a clear objective, shared input, and explicit constraints. Avoid starting with a vague request such as “write something useful.”

Example prompt: “Draft a customer email explaining that a scheduled data export is delayed by two hours. Use a calm, accountable tone. Include the revised delivery time, one practical workaround, and a brief apology. Keep the email under 140 words. Do not invent a cause for the delay. Original delivery time: 14:00 UTC.”

Use identical instructions and attachments for both models. Start fresh conversations, disable tools or web access unless they are part of the test, and keep generation settings aligned where supported. Matching settings reduces avoidable differences; it does not make different models behave identically.

§ 5. Run the comparison in Cherry Studio

Use Cherry Studio’s multi-model conversation controls to address the same prompt to both models. Depending on your version, model selection may be available through the model picker or an @ mention menu. Check that both selected entries belong to your Vana API provider.

If your version does not provide the comparison layout you need, use two fresh conversations and paste the identical prompt into each. Arrange the results side by side on your desktop. Keep follow-up instructions identical rather than helping one model more than the other.

§ 6. Evaluate quality, speed, and cost

• Accuracy: Is the revised delivery time correct, with no invented explanation?
• Instruction following: Does the answer include every requested element and stay within the word limit?
• Usefulness: Could you send it with minimal editing?
• Responsiveness: How long does a complete, usable answer take?
• Cost: Review token usage and charges in Vana API rather than judging cost by response length alone.

Vana API bills per real token with no monthly fee. Repeat the comparison across several representative tasks before choosing a default. Keep the prompts and a simple scorecard: the useful result is a model that reliably fits your workload, not a winner from one unusually good answer.