jev-27b-vl
Checkpoint jev-27b-vl/hf-ba3f0d5. Reference: none set. Scored on typed-decisions, core tier: 400 cases × 5 questions = 2,000 decisions per run. Δ is accuracy minus the reference's, in points.
Accuracy by variant and machine
| Variant \ Hardware | NVIDIA P100 · 16 GB |
|---|---|
| gguf-q3_k_m-iammrduncanllamacpp | 69.7%no referenceBrier 0.161 · CI 67.6%–71.7%0.4 decisions/sllamacpp-cuda b11370+jev |
Accuracy counts every dispatched decision. The interval (CI) is Wilson 95%; differences smaller than the intervals' overlap are not resolved by 2,000 decisions.
All runs
| Variant · machine | Accuracy | KL ↓ | Brier ↓ | ECE ↓ | choice | noul | score | Decisions/s | Input tok/s* | p50 / case | Conformance |
|---|---|---|---|---|---|---|---|---|---|---|---|
| gguf-q3_k_m-iammrduncan · NVIDIA P100 · 16 GB | 0.47 | 0.161 | 0.070 | 64.7% | 80.3% | 65.5% | 0.4 | 134 | 22.48 s | 49/49 |
Decisions/s is over the run's wall-clock time, client to server, so a remote machine includes the network. Engine-only figures are on each run's page. *Input tok/s counts tokens with each model's own tokenizer and prompt layout (d1 counts the state once per question), so it compares one model across machines, not models with each other. p50 is per case (all five questions); runs kept two requests in flight against servers that answer one at a time, so it includes waiting for the other.