MTEB Quick Start¶

Quick Answer: Timing¶

TL;DR: - Quick test (2 tasks): ~10-25 minutes for all 5 models - Comprehensive (5 tasks): ~100-150 minutes (1.7-2.5 hours) for all 5 models

See MTEB Timing Guide for detailed timing information.

Having issues? Check the MTEB Troubleshooting Guide.

Container image requirements¶

MTEB benchmarks require the vllm-mteb container with MTEB 2.19.0+ (upstream OpenAIAPIEncodeWrapper). The default image is:

quay.io/vllm-cpu-perf-eval/vllm-mteb:latest

After upgrading the benchmark runner in this repo, rebuild and push the image before running sweeps against :latest:

cd container-images/vllm-mteb
./build.sh
# or trigger .github/workflows/build-mteb-container.yml on merge to main

To pin a known-good image, set MTEB_CONTAINER_IMAGE or pass --extra mteb_container_image=quay.io/vllm-cpu-perf-eval/vllm-mteb:latest-<git-sha>.

Quick Start¶

# Quick smoke test on all models (default task preset: quick, 2 tasks per model)
automation/test-execution/scripts/bash/run-mteb-model-sweep.sh

# Recommended: run via cpueval (skips interactive prompt, same defaults)
./cpueval --suite mteb

# Full quality sweep (override task preset only when you need more coverage)
./cpueval --suite mteb --extra task_preset=full --cores 32 --vllm-cpus 0-31 \
  --extra vllm_mode=dut-only --continue-on-error

See Execution modes for managed, dut-only, and external setups.

Default task preset: `quick`¶

You only need --extra task_preset=... (or --task-preset on the bash script) when you want more than the default smoke test. By default everything runs the quick preset (2 classification tasks per model, ~10–25 minutes for all 5 models). Use comprehensive or full when you deliberately want longer coverage — see Task Presets below.

Run All Models (bash script)¶

automation/test-execution/scripts/bash/run-mteb-model-sweep.sh

This runs the "quick" preset (2 tasks) on all 5 models (~10-25 minutes).

Run via cpueval (recommended)¶

# Default: all models, quick preset, 32 vLLM cores, managed mode
./cpueval --suite mteb

# DUT-only on a single host with explicit CPU pinning
export VLLM_MODE=dut-only
./cpueval --suite mteb --cores 32 --vllm-cpus 0-31

# Full task coverage (14 tasks per model — several hours)
./cpueval --suite mteb --extra task_preset=full --continue-on-error

# Preview commands without running
./cpueval --suite mteb --extra task_preset=full --dry-run

See cpueval CLI for all flags.

Execution modes¶

MTEB uses the same vLLM deployment modes as the embedding performance suite. The Ansible playbook (mteb-benchmark.yml) supports all three; cpueval --suite mteb forwards mode via env vars or flags (it no longer forces managed by default).

Mode	vLLM	MTEB runner	Hosts	When to use
managed	Started on DUT	Load generator	2 (`DUT_HOSTNAME` ≠ `LOADGEN_HOSTNAME`)	Production-like split: CPU server + client
dut-only	Started on DUT	Same DUT	1 (set both hostnames to the same EC2)	Single-node EC2, lab laptop, all-in-one
external	Pre-existing endpoint	Load generator	1+ (any reachable vLLM URL)	K8s LB, RHAIIS cluster, prod endpoint

Managed (2-host, default)¶

vLLM on the DUT; MTEB container runs on the load generator and calls the DUT over the network.

export DUT_HOSTNAME=dut.example.com
export LOADGEN_HOSTNAME=loadgen.example.com
# VLLM_MODE defaults to managed

./cpueval --suite mteb --cores 32 --vllm-cpus 0-31

DUT-only (single-host)¶

Both vLLM and MTEB on one machine. Point both inventory hostnames at the same host (or only the DUT is used for vLLM+MTEB).

export DUT_HOSTNAME=ec2-xxx.compute.amazonaws.com
export LOADGEN_HOSTNAME=ec2-xxx.compute.amazonaws.com   # same host OK

./cpueval --suite mteb \
  --cores 32 \
  --vllm-cpus 0-31 \
  --extra vllm_mode=dut-only

Or: export VLLM_MODE=dut-only (no --extra needed after the cpueval fix).

External endpoint¶

Skip vLLM startup; run MTEB against an existing /v1/embeddings URL. MTEB runs on the load generator host.

export LOADGEN_HOSTNAME=bench-runner.example.com

./cpueval --suite mteb \
  --endpoint-url http://vllm-lb.example.com:8000 \
  --model RedHatAI/granite-embedding-english-r2

Equivalent:

export VLLM_MODE=external
export VLLM_ENDPOINT_URL=http://vllm-lb.example.com:8000
./cpueval --suite mteb --extra vllm_mode=external --extra endpoint_url=http://vllm-lb.example.com:8000

Note: --endpoint-url on cpueval automatically sets external mode. CPU pinning flags (--cores, --vllm-cpus) are ignored in external mode.

Run Comprehensive Tests¶

automation/test-execution/scripts/bash/run-mteb-model-sweep.sh --task-preset comprehensive

This runs 5 tasks on all 5 models (~1.7-2.5 hours).

Run Specific Models Only¶

automation/test-execution/scripts/bash/run-mteb-model-sweep.sh \
  --models "RedHatAI/all-MiniLM-L6-v2,RedHatAI/granite-embedding-english-r2"

Fixing Directory Structure¶

Option 1: Dashboard Auto-Handles It (Recommended)¶

The dashboard already supports both directory structures: - no_model_name_available/no_revision_available/*.json (MTEB default) - TaskName/test.json (clean format)

Just view results in the dashboard - no action needed.

Option 2: Reorganize Results (Optional)¶

If you prefer clean directories:

cd scripts
./reorganize-mteb-results.sh

This converts:

FROM: results/mteb/MODEL/TIMESTAMP/no_model_name_available/no_revision_available/Banking77Classification.json
TO:   results/mteb/MODEL/TIMESTAMP/Banking77Classification/test.json

Viewing Results¶

After tests complete:

# Start dashboard
cd automation/test-execution/dashboard-examples/vllm_dashboard
streamlit run Home.py

# Navigate to: 📊 Embedding Metrics → 🎯 MTEB Quality tab

All Available Options¶

automation/test-execution/scripts/bash/run-mteb-model-sweep.sh --help

Common Options¶

Option	Description	Example
`--task-preset`	Task set to run	`--task-preset comprehensive`
`--models`	Specific models	`--models "RedHatAI/all-MiniLM-L6-v2"`
`--skip-models`	Skip large models	`--skip-models "RedHatAI/Qwen3-Embedding-8B"`
`--cores`	CPU cores (default: 32)	`--cores 16`
`--dry-run`	Preview without running	`--dry-run`

Task Presets¶

Preset	Tasks	Categories	Time (all 5 models)	Use Case
`quick`	2	Classification	10-25 min	Fast smoke test
`retrieval`	3	Retrieval	50-75 min	IR performance
`classification`	3	Classification	25-50 min	Text categorization
`sts`	3	STS	15-40 min	Similarity
`reranking`	3	Reranking	40-60 min	Document reranking
`pair_classification`	2	Pair Classification	20-30 min	Text pair classification
`comprehensive`	5	3 categories	100-150 min	Balanced evaluation
`full`	14	5 categories	250-400 min	Maximum coverage

Categories Covered: - quick/classification: Classification only - retrieval: Retrieval only - sts: Semantic Textual Similarity only - reranking: Reranking only - pair_classification: Pair Classification only - comprehensive: Classification, Retrieval, STS (3 categories) - full: Classification, Retrieval, STS, Reranking, Pair Classification (5 categories)

Note: Clustering tasks are currently disabled due to segmentation faults with certain models.

Models Tested¶

RedHatAI/all-MiniLM-L6-v2 (22.7M) - Fastest
RedHatAI/granite-embedding-english-r2 (109M)
RedHatAI/nomic-embed-text-v1.5 (137M)
RedHatAI/embeddinggemma-300m (300M)
RedHatAI/Qwen3-Embedding-8B (8B) - Most thorough, slowest

Examples¶

Skip slowest models during development¶

automation/test-execution/scripts/bash/run-mteb-model-sweep.sh \
  --skip-models "RedHatAI/Qwen3-Embedding-8B,RedHatAI/embeddinggemma-300m"

Time: ~6-15 minutes (3 models, quick preset)

Test only small fast model¶

automation/test-execution/scripts/bash/run-mteb-model-sweep.sh \
  --models "RedHatAI/all-MiniLM-L6-v2" \
  --task-preset quick

Time: ~2-5 minutes (1 model, 2 tasks)

Full benchmark for report¶

automation/test-execution/scripts/bash/run-mteb-model-sweep.sh --task-preset comprehensive

Time: ~1.7-2.5 hours (5 models, 5 tasks)

Troubleshooting¶

Results not showing in dashboard¶

Check results exist:
```
ls -la results/mteb/
```
Reorganize if needed:
```
./scripts/reorganize-mteb-results.sh
```
Reload dashboard:
Click "🔄 Reload Data" in dashboard sidebar
Or restart Streamlit

Tests taking too long¶

Use --skip-models to exclude large models
Use --task-preset quick for faster tests
Test fewer models with --models

Out of memory¶

Large models (especially Qwen3-8B) require significant RAM: - 16GB minimum for small models - 32GB+ recommended for 8B models - Use --skip-models to exclude large models if needed

Next Steps¶

Run quick test to validate setup
View results in dashboard
Run comprehensive for final benchmarking
Generate report from dashboard data

Files Created¶

File	Purpose
`run-mteb-model-sweep.sh`	Main sweep script (also used by `cpueval --suite mteb`)
`automation/cli/src/cpueval/suites/mteb.yaml`	cpueval suite definition
`docs/mteb-sweep-guide.md`	This guide
`docs/mteb-timing-guide.md`	Detailed timing info
`docs/mteb-troubleshooting.md`	Common issues and fixes