embeddings / embeddinggemma-2/hf-914f7f8/q8-onnx-community/onnxruntime-cpu-ort1.30.0 /

2026-10-07-quick-card-prompts-2fa048

By output dimension

Dims	Score (95% interval over tasks)	Δ vs reference	Fidelity
768	66.8054.95–78.12	−0.10 [−0.38, +0.09]	faithfulcos 0.9999 · top-10 0.98
512	66.4854.66–77.88	+0.05 [−0.05, +0.18]	faithfulcos 0.9999 · top-10 0.98
256	65.0653.22–76.64	+0.08 [−0.05, +0.22]	faithfulcos 0.9999 · top-10 0.97
128	60.6648.53–72.62	+0.05 [−0.07, +0.19]	faithfulcos 1.0000 · top-10 0.96

Reference: embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126/embeddings/2026-10-07-quick-card-prompts-d0777f

Per task

MTEB main score ×100

Task	768	512	256	128
NanoArguAnaRetrieval	64.91	62.85	59.93	56.64
NanoClimateFeverRetrieval	27.97	28.56	27.62	26.75
NanoDBPediaRetrieval	65.17	64.24	63.78	57.72
NanoFEVERRetrieval	92.34	93.22	91.06	86.85
NanoFiQA2018Retrieval	44.75	44.02	42.76	33.42
NanoHotpotQARetrieval	89.56	88.29	87.53	78.56
NanoMSMARCORetrieval	55.50	55.13	54.78	50.24
NanoNFCorpusRetrieval	38.10	37.11	35.15	30.48
NanoNQRetrieval	78.49	76.70	73.28	71.47
NanoQuoraRetrieval	95.46	95.45	94.81	92.88
NanoSCIDOCSRetrieval	37.86	37.66	35.62	30.89
NanoSciFactRetrieval	88.08	89.61	87.24	77.83
NanoTouche2020Retrieval	44.07	44.58	42.86	38.01
STSBenchmark	87.48	87.54	87.37	86.90
Banking77Classification	92.28	92.26	92.07	91.27

Throughput

Prefill tokens/s (engine): 2,756 (embedding has no decode)
Prefill tokens/s (wall): 2,750 (includes HTTP and base64)
Texts: 66,939 texts · 11,589,994 tokens · 15.9 texts/s
Engine: backend=onnxruntime · onnx_file=model_quantized.onnx · requested_provider=CPUExecutionProvider · active_providers=CPUExecutionProvider · onnxruntime=1.30.0 · transformers=5.19.0 · max_seq_length=8192 · token_budget_per_forward=16384 · python=3.12.12 · platform=macOS-26.5.2-arm64-arm-64bit · machine=arm64 · numpy=2.5.3

Run

Subject: embeddinggemma-2/hf-914f7f8/q8-onnx-community/onnxruntime-cpu-ort1.30.0
Suite: embeddings · quick tier · profile card-prompts · mode native
Items: 2,092 succeeded of 2,092 attempted (66,939 planned)
When: 2026-10-07 06:20:43 UTC · 1 h 14 min
Hardware: Apple M4 Pro · 24 GB
Harness: bench @ 5f5fc24 · scorer mteb 2.22.5
Suite hash: d499aab0d7091e3ba5b211e1cbaedf79ed4c9c86a7214156946d2ee4d21299ec
Recipe: embeddinggemma-2/apple-m/st-ort · profile ort-q8 · inference-engines @ 0ddc734
Latency: p50 969 ms · p95 6.34 s · p99 17.17 s (per request)
Usage: 11,589,994 input tokens · 0 output tokens · $0.00

dimensions below 768 are the same vectors truncated and re-normalized (Matryoshka); fidelity compares each dimension with the reference at the same dimension

Reproduce

npm run bench -- run --recipe embeddinggemma-2/apple-m/st-ort \
    --recipe-profile ort-q8 \
    --on <host with a apple-m:m4-pro> \
    --suite embeddings --tier quick --profile card-prompts \
    --reference results/embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126/embeddings/2026-10-07-quick-card-prompts-d0777f

Files

exactly as committed under results/