embeddings / embeddinggemma-2/hf-914f7f8/q8-onnx-community/onnxruntime-cuda-ort1.29.0-cuda12 /

2026-10-07-quick-card-prompts-b17c83

By output dimension

Dims	Score (95% interval over tasks)	Δ vs reference	Fidelity
768	66.8054.95–78.12	−0.10 [−0.38, +0.09]	faithfulcos 0.9999 · top-10 0.98
512	66.4854.66–77.88	+0.05 [−0.05, +0.18]	faithfulcos 0.9999 · top-10 0.98
256	65.0653.22–76.64	+0.08 [−0.05, +0.22]	faithfulcos 0.9999 · top-10 0.97
128	60.6648.53–72.62	+0.05 [−0.07, +0.19]	faithfulcos 1.0000 · top-10 0.96

Reference: embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126/embeddings/2026-10-07-quick-card-prompts-d0777f

Per task

MTEB main score ×100

Task	768	512	256	128
NanoArguAnaRetrieval	64.91	62.85	59.93	56.64
NanoClimateFeverRetrieval	27.97	28.56	27.62	26.75
NanoDBPediaRetrieval	65.17	64.24	63.78	57.72
NanoFEVERRetrieval	92.34	93.22	91.06	86.85
NanoFiQA2018Retrieval	44.75	44.02	42.76	33.42
NanoHotpotQARetrieval	89.56	88.29	87.53	78.56
NanoMSMARCORetrieval	55.50	55.13	54.78	50.24
NanoNFCorpusRetrieval	38.10	37.11	35.15	30.48
NanoNQRetrieval	78.49	76.70	73.28	71.47
NanoQuoraRetrieval	95.46	95.45	94.81	92.88
NanoSCIDOCSRetrieval	37.86	37.66	35.62	30.89
NanoSciFactRetrieval	88.08	89.61	87.24	77.83
NanoTouche2020Retrieval	44.07	44.58	42.86	38.01
STSBenchmark	87.48	87.54	87.37	86.90
Banking77Classification	92.28	92.26	92.07	91.27

Throughput

Prefill tokens/s (engine): 12,620 (embedding has no decode)
Prefill tokens/s (wall): 7,689 (includes HTTP and base64)
Texts: 66,939 texts · 11,589,994 tokens · 44.4 texts/s
Engine: backend=onnxruntime · onnx_file=model_quantized.onnx · requested_provider=CUDAExecutionProvider · active_providers=CUDAExecutionProvider,CPUExecutionProvider · onnxruntime=1.29.0 · transformers=5.19.0 · max_seq_length=8192 · token_budget_per_forward=16384 · python=3.12.15 · platform=Linux-7.0.0-34-generic-x86_64-with-glibc2.36 · machine=x86_64 · numpy=2.5.3

Run

Subject: embeddinggemma-2/hf-914f7f8/q8-onnx-community/onnxruntime-cuda-ort1.29.0-cuda12
Suite: embeddings · quick tier · profile card-prompts · mode native
Items: 2,092 succeeded of 2,092 attempted (66,939 planned)
When: 2026-10-07 04:25:02 UTC · 29 min
Hardware: NVIDIA P100 · 16 GB
2026-10-07: device facts copied from the operator's inventory after the run; the launcher did not record them yet
Harness: bench @ d7cc285 · scorer mteb 2.22.5
Suite hash: d499aab0d7091e3ba5b211e1cbaedf79ed4c9c86a7214156946d2ee4d21299ec
Recipe: embeddinggemma-2/p100/st-ort · profile ort-q8 · inference-engines @ 98e8454
Latency: p50 519 ms · p95 1.63 s · p99 4.07 s (per request)
Usage: 11,589,994 input tokens · 0 output tokens · $0.00

dimensions below 768 are the same vectors truncated and re-normalized (Matryoshka); fidelity compares each dimension with the reference at the same dimension

Reproduce

npm run bench -- run --recipe embeddinggemma-2/p100/st-ort \
    --recipe-profile ort-q8 \
    --on <host with a p100:p100> \
    --suite embeddings --tier quick --profile card-prompts \
    --reference results/embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126/embeddings/2026-10-07-quick-card-prompts-d0777f

Files

exactly as committed under results/