Evals

Run one eval across many models

Loop the same prompts over a list of catalog ids with one key and one client. Compare answers, latency, and cost side by side.

Model catalog Compare AI SDK guide

The problem

What this is for

Evaluating models from different vendors means one SDK and one key per provider, and cost lands on separate invoices you have to add up.

What to do

Three steps, then copy

1.Run anyr models (or open /models) and pick the ids you want to compare.
2.Send the same prompt to each id through one AI SDK provider.
3.Compare outputs and token usage, then check cost per request in Dashboard → Logs.

Eval loop

import { createAnyRouter } from "@anyr/ai-sdk-provider"
import { generateText } from "ai"

const anyrouter = createAnyRouter()
const models = ["openai/gpt-5.4-mini", "anthropic/claude-sonnet-4.6", "z-ai/glm-4.7-flash"]

for (const id of models) {
  const started = Date.now()
  const { text, usage } = await generateText({
    model: anyrouter(id),
    prompt: "Summarize the CAP theorem in two sentences.",
  })
  console.log(id, Date.now() - started, "ms", usage.totalTokens, "tokens\n", text)
}

anyr CLI

bash

anyr models

Ready to try it

The guide has the full walkthrough. Sign up if you still need a key.

Model catalog Start free

One AI SDK provider for the whole catalog

Install @anyr/ai-sdk-provider. generateText and streamText take a catalog id — including anyrouter/free.

Open

Shop for the cheapest model without lock-in

Compare list prices in the catalog, then route through one URL so switching models is a string change.

Open

One API in front of every provider dialect

OpenAI /chat/completions, Anthropic /messages, and a Responses API — same key, same catalog.

Open