embeddings / lfm2.5-embedding-350m/hf-f35ae2c/bf16-liquidai/sentence-transformers-torch-mps-st5.1.1-tf4.56.2-torch2.14.1 /

2026-10-09-quick-card-prompts-249a86

By output dimension

Dims	Score (95% interval over tasks)	Δ vs reference	Fidelity
1024	66.0757.22–74.98	+0.02 [−0.03, +0.07]	losslesscos 0.9999 · top-10 0.98

Reference: lfm2.5-embedding-350m/hf-f35ae2c/fp32-liquidai/sentence-transformers-torch-cuda-st5.1.1-tf4.56.2-torch2.14.1-cu126/embeddings/2026-10-08-quick-card-prompts-efe68a

Per task

MTEB main score ×100

Task	1024
NanoArguAnaRetrieval	50.40
NanoClimateFeverRetrieval	45.07
NanoDBPediaRetrieval	65.01
NanoFEVERRetrieval	95.31
NanoFiQA2018Retrieval	55.73
NanoHotpotQARetrieval	85.23
NanoMSMARCORetrieval	64.11
NanoNFCorpusRetrieval	36.93
NanoNQRetrieval	72.41
NanoQuoraRetrieval	89.56
NanoSCIDOCSRetrieval	39.12
NanoSciFactRetrieval	74.99
NanoTouche2020Retrieval	54.21
STSBenchmark	80.11
Banking77Classification	82.91

Throughput

Prefill tokens/s (engine): 6,203 (embedding has no decode)
Prefill tokens/s (wall): 6,181 (includes HTTP and base64)
Texts: 66,939 texts · 9,883,483 tokens · 41.9 texts/s
Engine: backend=sentence-transformers · device=mps · dtype=bfloat16 · torch=2.14.1 · cuda_arch_list=null · device_name=mps · sentence_transformers=5.1.1 · transformers=4.56.2 · max_seq_length=512 · dimension=1024 · trust_remote_code=true · unpadded=true · triton_overrides=null · token_budget_per_forward=16384 · python=3.12.12 · platform=macOS-26.5.2-arm64-arm-64bit · machine=arm64 · numpy=2.5.3

Run

Subject: lfm2.5-embedding-350m/hf-f35ae2c/bf16-liquidai/sentence-transformers-torch-mps-st5.1.1-tf4.56.2-torch2.14.1
Suite: embeddings · quick tier · profile card-prompts · mode native
Items: 2,092 succeeded of 2,092 attempted (66,939 planned)
When: 2026-10-09 00:36:51 UTC · 28 min
Hardware: Apple M4 Pro · 24 GB
Harness: bench @ 2298f0b · scorer mteb 2.22.5
Suite hash: 4c216f567e7755fcba7cd61c6ffe3e6ea2c399c52c3d2a2a6aedefb20066cf05
Recipe: lfm2.5-embedding-350m/apple-m/st · profile st-bf16 · inference-engines @ 1ef8974
Latency: p50 593 ms · p95 1.89 s · p99 1.94 s (per request)
Usage: 9,883,483 input tokens · 0 output tokens · $0.00

dimensions below 768 are the same vectors truncated and re-normalized (Matryoshka); fidelity compares each dimension with the reference at the same dimension

Reproduce

npm run bench -- run --recipe lfm2.5-embedding-350m/apple-m/st \
    --recipe-profile st-bf16 \
    --on <host with a apple-m:m4-pro> \
    --suite embeddings --tier quick --profile card-prompts \
    --reference results/lfm2.5-embedding-350m/hf-f35ae2c/fp32-liquidai/sentence-transformers-torch-cuda-st5.1.1-tf4.56.2-torch2.14.1-cu126/embeddings/2026-10-08-quick-card-prompts-efe68a

Files

exactly as committed under results/