MTEB Quick Start¶

Quick Answer: Timing¶

TL;DR: - Quick test (2 tasks): ~10-25 minutes for all 5 models - Comprehensive (5 tasks): ~100-150 minutes (1.7-2.5 hours) for all 5 models

See MTEB Timing Guide for detailed timing information.

Having issues? Check the MTEB Troubleshooting Guide.

Quick Start¶

Run All Models (Quick Test)¶

cd automation/test-execution/scripts
./run-mteb-model-sweep.sh

This runs the "quick" preset (2 tasks) on all 5 models (~10-25 minutes).

Run Comprehensive Tests¶

./run-mteb-model-sweep.sh --task-preset comprehensive

This runs 5 tasks on all 5 models (~1.7-2.5 hours).

Run Specific Models Only¶

./run-mteb-model-sweep.sh \
  --models "RedHatAI/all-MiniLM-L6-v2,RedHatAI/granite-embedding-english-r2"

Fixing Directory Structure¶

Option 1: Dashboard Auto-Handles It (Recommended)¶

The dashboard already supports both directory structures: - no_model_name_available/no_revision_available/*.json (MTEB default) - TaskName/test.json (clean format)

Just view results in the dashboard - no action needed.

Option 2: Reorganize Results (Optional)¶

If you prefer clean directories:

cd scripts
./reorganize-mteb-results.sh

This converts:

FROM: results/mteb/MODEL/TIMESTAMP/no_model_name_available/no_revision_available/Banking77Classification.json
TO:   results/mteb/MODEL/TIMESTAMP/Banking77Classification/test.json

Viewing Results¶

After tests complete:

# Start dashboard
cd automation/test-execution/dashboard-examples/vllm_dashboard
streamlit run Home.py

# Navigate to: 📊 Embedding Metrics → 🎯 MTEB Quality tab

All Available Options¶

./run-mteb-model-sweep.sh --help

Common Options¶

Option	Description	Example
`--task-preset`	Task set to run	`--task-preset comprehensive`
`--models`	Specific models	`--models "RedHatAI/all-MiniLM-L6-v2"`
`--skip-models`	Skip large models	`--skip-models "RedHatAI/Qwen3-Embedding-8B"`
`--cores`	CPU cores (default: 4)	`--cores 16`
`--dry-run`	Preview without running	`--dry-run`

Task Presets¶

Preset	Tasks	Categories	Time (all 5 models)	Use Case
`quick`	2	Classification	10-25 min	Fast smoke test
`retrieval`	3	Retrieval	50-75 min	IR performance
`classification`	3	Classification	25-50 min	Text categorization
`sts`	3	STS	15-40 min	Similarity
`reranking`	3	Reranking	40-60 min	Document reranking
`pair_classification`	2	Pair Classification	20-30 min	Text pair classification
`comprehensive`	5	3 categories	100-150 min	Balanced evaluation
`full`	14	5 categories	250-400 min	Maximum coverage

Categories Covered: - quick/classification: Classification only - retrieval: Retrieval only - sts: Semantic Textual Similarity only - reranking: Reranking only - pair_classification: Pair Classification only - comprehensive: Classification, Retrieval, STS (3 categories) - full: Classification, Retrieval, STS, Reranking, Pair Classification (5 categories)

Note: Clustering tasks are currently disabled due to segmentation faults with certain models.

Models Tested¶

RedHatAI/all-MiniLM-L6-v2 (22.7M) - Fastest
RedHatAI/granite-embedding-english-r2 (109M)
RedHatAI/nomic-embed-text-v1.5 (137M)
RedHatAI/embeddinggemma-300m (300M)
RedHatAI/Qwen3-Embedding-8B (8B) - Most thorough, slowest

Examples¶

Skip slowest models during development¶

./run-mteb-model-sweep.sh \
  --skip-models "RedHatAI/Qwen3-Embedding-8B,RedHatAI/embeddinggemma-300m"

Time: ~6-15 minutes (3 models, quick preset)

Test only small fast model¶

./run-mteb-model-sweep.sh \
  --models "RedHatAI/all-MiniLM-L6-v2" \
  --task-preset quick

Time: ~2-5 minutes (1 model, 2 tasks)

Full benchmark for report¶

./run-mteb-model-sweep.sh --task-preset comprehensive

Time: ~1.7-2.5 hours (5 models, 5 tasks)

Troubleshooting¶

Results not showing in dashboard¶

Check results exist:
```
ls -la results/mteb/
```
Reorganize if needed:
```
./scripts/reorganize-mteb-results.sh
```
Reload dashboard:
Click "🔄 Reload Data" in dashboard sidebar
Or restart Streamlit

Tests taking too long¶

Use --skip-models to exclude large models
Use --task-preset quick for faster tests
Test fewer models with --models

Out of memory¶

Large models (especially Qwen3-8B) require significant RAM: - 16GB minimum for small models - 32GB+ recommended for 8B models - Use --skip-models to exclude large models if needed

Next Steps¶

Run quick test to validate setup
View results in dashboard
Run comprehensive for final benchmarking
Generate report from dashboard data

Files Created¶

File	Purpose
`run-mteb-model-sweep.sh`	Main sweep script
`docs/mteb-sweep-guide.md`	This guide
`docs/mteb-timing-guide.md`	Detailed timing info
`docs/mteb-troubleshooting.md`	Common issues and fixes