EvalsSDKs
Run one eval across many models
Loop the same prompts over a list of catalog ids with one key and one client. Compare answers, latency, and cost side by side.
Set it up
Eval loop
import { createAnyRouter } from "@anyr/ai-sdk-provider"
import { generateText } from "ai"
const anyrouter = createAnyRouter()
const models = ["z-ai/glm-4.7-flash", "deepseek/deepseek-v4-flash", "google/gemma-4-31b"]
for (const id of models) {
const started = Date.now()
const { text, usage } = await generateText({
model: anyrouter(id),
prompt: "Summarize the CAP theorem in two sentences.",
})
console.log(id, Date.now() - started, "ms", usage.totalTokens, "tokens\n", text)
}- 1.Run anyr models (or open /models) and pick the ids you want to compare.
- 2.Send the same prompt to each id through one AI SDK provider. Models on your own provider keys (OpenAI, Anthropic) work the same way.
- 3.Compare outputs and token usage, then check cost per request in Dashboard → Logs.
Why use this
Evaluating models from different vendors means one SDK and one key per provider, and cost lands on separate invoices you have to add up.
Other ways to set it up
anyr CLI
anyr modelsEndpoints and key
- OpenAI-compatible
- https://anyrouter.dev/api/v1
- Anthropic-compatible
- https://anyrouter.dev/api
- Key
- ANYROUTER_API_KEY (sk-ar-…)