embeddings / embeddinggemma-2/hf-914f7f8/q4-onnx-community/onnxruntime-cpu-ort1.30.0 /

2026-10-07-quick-card-prompts-50b823

By output dimension

Dims	Score (95% interval over tasks)	Δ vs reference	Fidelity
768	66.1954.52–77.36	−0.71 [−1.01, −0.43]	degradedcos 0.9805 · top-10 0.72
512	66.1654.45–77.53	−0.27 [−0.93, +0.28]	degradedcos 0.9812 · top-10 0.72
256	64.5352.53–75.99	−0.45 [−1.23, +0.55]	degradedcos 0.9832 · top-10 0.69
128	60.3748.38–72.05	−0.24 [−1.06, +0.50]	degradedcos 0.9879 · top-10 0.63

Reference: embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126/embeddings/2026-10-07-quick-card-prompts-d0777f

Per task

MTEB main score ×100

Task	768	512	256	128
NanoArguAnaRetrieval	65.01	64.78	65.27	59.49
NanoClimateFeverRetrieval	27.28	28.36	26.81	26.29
NanoDBPediaRetrieval	64.40	62.33	61.19	55.47
NanoFEVERRetrieval	90.47	92.36	89.03	85.97
NanoFiQA2018Retrieval	44.45	44.09	39.67	33.58
NanoHotpotQARetrieval	88.79	89.10	87.52	79.34
NanoMSMARCORetrieval	56.46	56.38	55.00	51.56
NanoNFCorpusRetrieval	38.27	36.33	34.20	28.92
NanoNQRetrieval	76.70	75.96	71.22	70.66
NanoQuoraRetrieval	95.27	95.32	94.90	92.20
NanoSCIDOCSRetrieval	36.67	38.10	35.77	30.70
NanoSciFactRetrieval	86.77	86.08	85.17	74.45
NanoTouche2020Retrieval	43.42	44.29	43.39	39.80
STSBenchmark	87.24	87.29	87.24	86.83
Banking77Classification	91.61	91.62	91.57	90.34

Throughput

Prefill tokens/s (engine): 1,391 (embedding has no decode)
Prefill tokens/s (wall): 1,390 (includes HTTP and base64)
Texts: 66,939 texts · 11,589,994 tokens · 8.0 texts/s
Engine: backend=onnxruntime · onnx_file=model_q4.onnx · requested_provider=CPUExecutionProvider · active_providers=CPUExecutionProvider · onnxruntime=1.30.0 · transformers=5.19.0 · max_seq_length=8192 · token_budget_per_forward=16384 · python=3.12.12 · platform=macOS-26.5.2-arm64-arm-64bit · machine=arm64 · numpy=2.5.3

Run

Subject: embeddinggemma-2/hf-914f7f8/q4-onnx-community/onnxruntime-cpu-ort1.30.0
Suite: embeddings · quick tier · profile card-prompts · mode native
Items: 2,092 succeeded of 2,092 attempted (66,939 planned)
When: 2026-10-07 07:34:33 UTC · 2 h 22 min
Hardware: Apple M4 Pro · 24 GB
Harness: bench @ 753c0c2 · scorer mteb 2.22.5
Suite hash: d499aab0d7091e3ba5b211e1cbaedf79ed4c9c86a7214156946d2ee4d21299ec
Recipe: embeddinggemma-2/apple-m/st-ort · profile ort-q4 · inference-engines @ 0ddc734
Latency: p50 2.16 s · p95 11.81 s · p99 34.49 s (per request)
Usage: 11,589,994 input tokens · 0 output tokens · $0.00

dimensions below 768 are the same vectors truncated and re-normalized (Matryoshka); fidelity compares each dimension with the reference at the same dimension

Reproduce

npm run bench -- run --recipe embeddinggemma-2/apple-m/st-ort \
    --recipe-profile ort-q4 \
    --on <host with a apple-m:m4-pro> \
    --suite embeddings --tier quick --profile card-prompts \
    --reference results/embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126/embeddings/2026-10-07-quick-card-prompts-d0777f

Files

exactly as committed under results/