d1-omni-600m
Checkpoint d1-omni-600m/hf-02b55d7. Reference: fp32-liquidai/decide-server-d1-torch-cuda-tf5.19.0-torch2.14.1-cu126. Scored on typed-decisions, core tier: 400 cases × 5 questions = 2,000 decisions per run. Δ is accuracy minus the reference's, in points.
Accuracy by variant and machine
| Variant \ Hardware | NVIDIA P100 · 16 GB | Apple M4 Pro · 24 GB |
|---|---|---|
| fp32-liquidaidecide-server-d1 | 57.4%referenceBrier 0.173 · CI 55.2%–59.6%28.5 decisions/sdecide-server-d1-torch-cuda tf5.19.0-torch2.14.1+cu126 | 57.4%+0.0 ptsBrier 0.173 · CI 55.2%–59.6%20.7 decisions/sdecide-server-d1-torch-mps tf5.19.0-torch2.14.1 |
| fp16-liquidaidecide-server-d1 | 57.4%−0.0 ptsBrier 0.173 · CI 55.2%–59.5%29.7 decisions/sdecide-server-d1-torch-cuda tf5.19.0-torch2.14.1+cu126 | 57.4%−0.0 ptsBrier 0.173 · CI 55.2%–59.5%24.5 decisions/sdecide-server-d1-torch-mps tf5.19.0-torch2.14.1 |
Accuracy counts every dispatched decision. The interval (CI) is Wilson 95%; differences smaller than the intervals' overlap are not resolved by 2,000 decisions.
All runs
| Variant · machine | Accuracy | KL ↓ | Brier ↓ | ECE ↓ | choice | noul | score | Decisions/s | Input tok/s* | p50 / case | Conformance |
|---|---|---|---|---|---|---|---|---|---|---|---|
| fp32-liquidai · NVIDIA P100 · 16 GBreference | 0.35 | 0.173 | 0.063 | 60.5% | 71.0% | 44.9% | 28.5 | 8,732 | 355 ms | 48/49 | |
| fp32-liquidai · Apple M4 Pro · 24 GB | 0.35 | 0.173 | 0.063 | 60.5% | 71.0% | 44.9% | 20.7 | 6,332 | 520 ms | 48/49 | |
| fp16-liquidai · NVIDIA P100 · 16 GB | 0.35 | 0.173 | 0.062 | 60.7% | 71.0% | 44.6% | 29.7 | 9,103 | 352 ms | 48/49 | |
| fp16-liquidai · Apple M4 Pro · 24 GB | 0.35 | 0.173 | 0.062 | 60.5% | 70.8% | 44.9% | 24.5 | 7,513 | 432 ms | 48/49 |
Decisions/s is over the run's wall-clock time, client to server, so a remote machine includes the network. Engine-only figures are on each run's page. *Input tok/s counts tokens with each model's own tokenizer and prompt layout (d1 counts the state once per question), so it compares one model across machines, not models with each other. p50 is per case (all five questions); runs kept two requests in flight against servers that answer one at a time, so it includes waiting for the other.