embeddings / embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-mps-st6.1.0-torch2.14.1 /

2026-10-07-quick-card-prompts-96892c

harness had local changes

By output dimension

Dims	Score (95% interval over tasks)	Δ vs reference	Fidelity
768	66.9055.02–78.13	+0.00 [+0.00, +0.00]	losslesscos 1.0000 · top-10 1.00
512	66.4354.55–77.84	+0.00 [+0.00, +0.00]	losslesscos 1.0000 · top-10 1.00
256	64.9853.11–76.57	+0.00 [+0.00, +0.00]	losslesscos 1.0000 · top-10 1.00
128	60.6148.58–72.51	+0.00 [+0.00, +0.00]	losslesscos 1.0000 · top-10 1.00

Reference: embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126/embeddings/2026-10-07-quick-card-prompts-d0777f

Per task

MTEB main score ×100

Task	768	512	256	128
NanoArguAnaRetrieval	65.05	63.24	60.19	56.78
NanoClimateFeverRetrieval	28.01	28.42	27.57	27.11
NanoDBPediaRetrieval	65.29	64.04	63.44	57.99
NanoFEVERRetrieval	92.43	93.22	91.10	86.85
NanoFiQA2018Retrieval	44.59	43.72	42.54	32.73
NanoHotpotQARetrieval	89.56	88.38	87.62	78.12
NanoMSMARCORetrieval	57.39	55.13	54.54	50.24
NanoNFCorpusRetrieval	38.02	37.08	35.19	30.34
NanoNQRetrieval	78.21	76.67	72.88	71.42
NanoQuoraRetrieval	95.46	95.45	94.81	92.88
NanoSCIDOCSRetrieval	37.60	37.70	35.46	30.75
NanoSciFactRetrieval	88.22	89.70	86.55	77.69
NanoTouche2020Retrieval	43.88	43.88	43.31	38.12
STSBenchmark	87.50	87.55	87.38	86.90
Banking77Classification	92.26	92.27	92.08	91.25

Throughput

Prefill tokens/s (engine): 4,793 (embedding has no decode)
Prefill tokens/s (wall): 4,763 (includes HTTP and base64)
Texts: 66,939 texts · 11,589,994 tokens · 27.5 texts/s
Engine: backend=sentence-transformers · device=mps · dtype=float32 · torch=2.14.1 · cuda_arch_list=null · device_name=mps · sentence_transformers=6.1.0 · transformers=5.19.0 · max_seq_length=8192 · token_budget_per_forward=16384 · python=3.12.12 · platform=macOS-26.5.2-arm64-arm-64bit · machine=arm64 · numpy=2.5.3

Run

Subject: embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-mps-st6.1.0-torch2.14.1
Suite: embeddings · quick tier · profile card-prompts · mode native
Items: 2,092 succeeded of 2,092 attempted (66,939 planned)
When: 2026-10-07 03:43:49 UTC · 44 min
Hardware: Apple M4 Pro · 24 GB
2026-10-07: device facts copied from the operator's inventory after the run; the launcher did not record them yet
Harness: bench @ c62a98f (with local changes) · scorer mteb 2.22.5
Suite hash: d499aab0d7091e3ba5b211e1cbaedf79ed4c9c86a7214156946d2ee4d21299ec
Recipe: embeddinggemma-2/apple-m/st-ort · profile st-fp32 · inference-engines @ 0ddc734
Latency: p50 578 ms · p95 3.84 s · p99 11.01 s (per request)
Usage: 11,589,994 input tokens · 0 output tokens · $0.00

dimensions below 768 are the same vectors truncated and re-normalized (Matryoshka); fidelity compares each dimension with the reference at the same dimension

Reproduce

npm run bench -- run --recipe embeddinggemma-2/apple-m/st-ort \
    --recipe-profile st-fp32 \
    --on <host with a apple-m:m4-pro> \
    --suite embeddings --tier quick --profile card-prompts \
    --reference results/embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126/embeddings/2026-10-07-quick-card-prompts-d0777f

Files

exactly as committed under results/