Skip to content

Run one eval across many models

Loop the same prompts over a list of catalog ids with one key and one client. Compare answers, latency, and cost side by side.

Set it up

Eval loop
import { createAnyRouter } from "@anyr/ai-sdk-provider"
import { generateText } from "ai"

const anyrouter = createAnyRouter()
const models = ["z-ai/glm-4.7-flash", "deepseek/deepseek-v4-flash", "google/gemma-4-31b"]

for (const id of models) {
  const started = Date.now()
  const { text, usage } = await generateText({
    model: anyrouter(id),
    prompt: "Summarize the CAP theorem in two sentences.",
  })
  console.log(id, Date.now() - started, "ms", usage.totalTokens, "tokens\n", text)
}
  1. 1.Run anyr models (or open /models) and pick the ids you want to compare.
  2. 2.Send the same prompt to each id through one AI SDK provider. Models on your own provider keys (OpenAI, Anthropic) work the same way.
  3. 3.Compare outputs and token usage, then check cost per request in Dashboard → Logs.
Why use this

Evaluating models from different vendors means one SDK and one key per provider, and cost lands on separate invoices you have to add up.

Other ways to set it up
anyr CLI
anyr models
Endpoints and key
OpenAI-compatible
https://anyrouter.dev/api/v1
Anthropic-compatible
https://anyrouter.dev/api
Key
ANYROUTER_API_KEY (sk-ar-…)
Watch the walkthrough