Methodology

What is measured, how, and what is and is not comparable.

Subjects

A score belongs to a subject: model/checkpoint/quant/engine. The runner resolves every part from what actually ran (the Hugging Face revision, the quant producer and revision, the engine version), and nothing is typed by hand. Hardware is recorded on every run and is part of the key only for speed.

Recipe subjects come from inference-engines: the launcher brings a pinned recipe up, bench measures it, and the launcher takes it down. The recipe and its commit are on every run page.
Provider-opaque subjects are hosted APIs. Their checkpoint is the provider's model ID plus the month of the run, because providers change what serves a model ID.
Private engine, not reproducible: the scores are published, the engine's code is not. Such a subject is never a reference.

One harness

bench builds every request, records every byte and extracts answers identically for every subject. Scoring is offline, by pinned scorers reading the captured outputs, never by calling a model. Every run publishes run.json (full identity and settings), summary.json and raw.jsonl, and the summary can be recomputed from raw data with bench summarize.

Leaderboards and uncertainty

One table is one suite, one tier and one settings profile. There is no global table.
Proportions get Wilson 95% intervals. Ranks are bands: a row's range spans every position its interval allows, and overlapping rows tie.
A metric a subject cannot produce is absent, never 0. Partial runs are published and never ranked.

The home page's Index

The home page has one table per kind of model: Language, Decision, Embedding and Image. Their scores do not mean the same thing (a decision model's accuracy, an embedding model's retrieval score), so they are never averaged together. A suite's category decides its table, so a model appears in every table it has results in: Qwen answering decisions through the gateway is in Decision, and its coding runs will be in Language. Only ranked suites count: pass/fail suites such as decision-conformance are listed, not scored.

Language: Taste, Coding, Math, Tool calling, Knowledge, Instruction following, Long context. Speed: prefill, decode, TTFT, latency.
Decision: Typed decisions (accuracy), Calibration ((1 − Brier) × 100 on typed-decisions), Scenes, Classic, Sealed. Speed and cost: latency per decision request, $ per 1,000 decisions.
Embedding: Text (MTEB), Code, Multilingual, Image, Audio. Speed: prefill only.
Image: Text-to-image, Image editing. Speed: time per image.

Category score: the mean of the subject's suite headlines in that category, each on a 0–100 scale (decision accuracy, mean MTEB score). For each suite, every subject is compared on the same board: the tier most subjects ran.
Index: in each category the subject ran, its score as a share of the best subject's score there (best = 100), averaged over those categories. It is computed within one table only.
Coverage (e.g. 1/2) is shown next to every Index: how many of the categories with results this subject has. The Index says how close a subject is to the best at what it runs, and nothing about what it does not run. Read it with its coverage, never alone.
Taste counts only once the owner's ranking exists. Speed columns come from the subject's latest run on its stated hardware and are not part of the Index: speed only compares on the same machine. Prefill (input tokens per second) and decode (output tokens per second) are separate columns, because they are different work; embedding models only prefill. Decode and TTFT are not measured yet.

This is a starting rule and will change as categories fill in. Every change will be listed here.

Embeddings

Prompts are the model card's own, read from the checkpoint at its pinned revision and applied with MTEB's rules. The server adds none.
Scores are MTEB's own main scores, computed offline from the vectors bench collected. The headline is the mean over the tier's tasks.
Dimensions 768, 512, 256 and 128 come from one run: the vectors are truncated and re-normalized (Matryoshka), which is exactly what a user of a smaller dimension gets.
Δ vs reference is paired task by task against the checkpoint's reference subject at the same dimension, with a bootstrap interval over tasks. The score's own interval also resamples tasks, so it mostly reflects task choice; that is why ranks overlap and the Δ column is the one to read.
Fidelity compares each run's vectors with the reference's on the same texts: mean, p1 and min cosine, and top-10 neighbor overlap for up to 1,000 queries against 20,000 documents. Verdicts: lossless (cosine ≥ 0.999 and overlap ≥ 0.98), faithful, degraded (cosine < 0.98 or overlap < 0.80), broken (< 0.90 or < 0.50). The thresholds are provisional.
Speed tables are per hardware class and rank only faithful or lossless subjects. Engines are run stock and untuned.

Decisions

The typed-decisions suite sends structured decision questions (Choice, Score, Noul) in the System One /v1/systemone shape. Accuracy counts every dispatched decision, so an invalid output is a wrong answer; KL, Brier and ECE use valid decisions only. Models without a native decision API are reached through the gateway, and the gateway mode is part of the run's mode.