Audio Model Testing Suite

Comprehensive test suite for evaluating audio model performance on vLLM CPU deployments.

⚠️ Known Issues

vLLM v0.25.1 CPU Compatibility

vLLM v0.25.1 V1 engine has CPU regressions for encoder-decoder models (Whisper). Workaround: Disable V1 engine using environment variable:

# When using v0.25.1, add to vllm_env_vars in playbook:
VLLM_USE_V2_MODEL_RUNNER: "0"

# Or via Ansible extra vars:
-e '{"vllm_env_vars": {"VLLM_USE_V2_MODEL_RUNNER": "0"}}'

# Alternative: Use v0.20.0 (works but older)
export VLLM_CONTAINER_IMAGE=docker.io/vllm/vllm-openai-cpu:v0.20.0

Issues in v0.25.1 V1 engine: 1. pin_memory=True crash in encoder_decoder.py (fix submitted upstream) 2. Triton GPU kernel compilation failures on CPU in sampling path

Status: Workaround verified working. Reported to Intel vLLM team.

TODO: Update once Intel fixes V1 engine CPU support. Test whisper-tiny before removing workaround.

Overview

This test suite provides automated benchmarking for: - Audio Transcription (ASR) - Speech-to-text conversion

Key Questions Answered

This test suite covers two distinct usage patterns:

1. Offline Batch Processing: - "I have N audio files on disk - how long to transcribe them all?" - Focus: Total completion time, maximum throughput - Use cases: Post-call transcription, media archive processing, batch ETL

2. Online Serving: - "How many concurrent users can submit audio for real-time transcription?" - Focus: Per-request latency (P50, P95, P99), concurrent user experience - Use cases: Live transcription API, voice assistant backend, streaming services

Plus comprehensive coverage of scalability, format impact, and sustained load behavior.

Test Scenarios

Scenario	Primary Focus	Serving Pattern	Use Case
transcription-throughput	Total time to process N files	Both (Offline + Online)	Batch processing + concurrent user simulation
transcription-latency	Per-request latency under load	Online Serving	Real-time transcription, SLA validation
transcription-quality	WER/CER accuracy	Quality	Accuracy measurement vs ground truth
audio-duration-scaling	Performance vs audio length	Offline Batch	Capacity planning, workload optimization
constant-rate-stress	Sustained load stability	Online Serving	Production readiness, resource planning
format-comparison	Audio format impact	Offline Batch	Bandwidth optimization, quality tradeoffs

Understanding Test Profiles

GuideLLM Profiles Explained:

• synchronous - Sequential processing (one request at a time)
• Serving pattern: Offline batch baseline
• Measures: Total time for N files processed serially

• Analogy: Single-threaded batch job

• concurrent with rate: N - Maintains N concurrent requests

• Serving pattern: Online serving simulation
• Measures: Latency under N concurrent users
• Analogy: N users simultaneously using a web API

• Important: This is NOT parallel batch processing - it simulates continuous concurrent load

• throughput - Maximum request rate the server can sustain

• Serving pattern: Capacity test (applies to both)
• Measures: Maximum files/sec, maximum audio_seconds/sec
• Analogy: Stress test to find breaking point

Which Test Should I Run?

Use Case: "I have 1000 audio files from call recordings, need to transcribe them all overnight" → Offline batch processing → Run: transcription-throughput (focus on sequential + max-throughput stages) → Key metric: Total wall-clock time, files/second

Use Case: "Building a real-time transcription API for customer calls" → Online serving → Run: transcription-latency + constant-rate-stress → Key metrics: P95 latency, concurrent user capacity, sustained load stability

Use Case: "Need to support both batch exports and live API" → Both patterns → Run: transcription-throughput (all stages) + transcription-latency → Analyze offline (sequential, max-throughput) and online (concurrent-N) results separately

Quick Start

Prerequisites

1. Install Ansible and collections (from the repository root):

   ./cpueval install
   

2. Verify system access:

   ./cpueval doctor
   

3. Container runtime:

4. Podman installed on load generator host
5. GuideLLM runs in container (ghcr.io/vllm-project/guidellm:v0.7.2)
6. No need to install guidellm on host

Run Your First Test

Option 1: Managed Mode (vLLM started by Ansible)

cd automation/test-execution/ansible

# Transcription throughput test (answers: how long for N files?)
ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=transcription-throughput" \
  -e "requested_cores=32"

# Quick test with custom file count
ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-tiny" \
  -e "test_scenario=transcription-throughput" \
  -e "requested_cores=32" \
  -e "audio_num_files=10"  # Override to 10 files per stage

Option 2: External Endpoint (vLLM already running)

# Point to existing vLLM server
export VLLM_ENDPOINT_URL=http://your-vllm-host:8000

ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_scenario=transcription-throughput" \
  -e vllm_endpoint.mode=external \
  -e vllm_endpoint.external.url=$VLLM_ENDPOINT_URL \
  -e "audio_num_files=50"  # Optional: override file count

Detailed Usage

Test Scenario: Transcription Throughput

Answers: How long to transcribe 100 audio files? (Offline batch + Online serving patterns)

ansible-playbook audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=transcription-throughput" \
  -e "requested_cores=32" \
  -e "requested_tensor_parallel=1"

What it measures (5 stages):

1. Sequential (profile: synchronous) - OFFLINE BATCH BASELINE
2. Process 100 files one at a time, serially
3. Answers: "How long if I process files one-by-one?"

4. Metric focus: Total wall-clock time

5. Concurrent-2/4/8 (profile: concurrent, rate: 2/4/8) - ONLINE SERVING

6. Simulate 2/4/8 concurrent users continuously submitting audio
7. Answers: "What's the latency when N users are using the API simultaneously?"
8. Metric focus: Per-request latency (P50, P95, P99), concurrent user experience

9. NOT parallel batch processing - this maintains continuous concurrent load

10. Max-throughput (profile: throughput, rate: 50) - CAPACITY TEST

11. Maintains 50 concurrent request streams to saturate server
12. Answers: "What's the maximum files/second this server can sustain?"
13. Metric focus: Maximum files/sec, maximum audio_seconds/sec
14. Use for: Capacity planning, finding server limits
15. Note: rate: 50 is configurable in scenario YAML - increase for load balancer/multi-instance setups

Key Distinction: - Offline batch: "I have 100 files on disk, transcribe them all ASAP" → Use sequential (baseline) and max-throughput results - Online serving: "How many users can use my transcription API concurrently?" → Use concurrent-N results

Results location:

results/audio-models/openai__whisper-small/transcription-throughput-20260422-110932/
├── sequential/               # Offline batch baseline
│   ├── benchmarks.json
│   └── benchmarks.csv
├── concurrent-2/             # Online: 2 concurrent users
│   ├── benchmarks.json
│   └── benchmarks.csv
├── concurrent-4/             # Online: 4 concurrent users
│   ├── benchmarks.json
│   └── benchmarks.csv
├── concurrent-8/             # Online: 8 concurrent users
│   ├── benchmarks.json
│   └── benchmarks.csv
├── max-throughput/           # Maximum capacity test
│   ├── benchmarks.json
│   └── benchmarks.csv
├── vllm-metrics.json
└── test-metadata.json

Results are organized by timestamp ({scenario}-{YYYYMMDD-HHMMSS}) to prevent overwriting on subsequent runs.

Test Scenario: Transcription Latency

Serving Pattern: ONLINE SERVING (Real-time API use case)

Answers: What is latency under different concurrent loads?

ansible-playbook audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=transcription-latency" \
  -e "requested_cores=32"

What it measures: - Baseline latency (single user, no contention) - Latency under light/medium/heavy concurrent load (2/5/10 users) - P50/P95/P99 latency percentiles - Latency degradation vs concurrent user count

Use case: SLA validation for real-time transcription APIs (e.g., "Can we guarantee <200ms P95 latency for up to 10 concurrent users?")

Test Scenario: Audio Duration Scaling

Serving Pattern: OFFLINE BATCH (Sequential processing)

Answers: How does processing time scale with audio length?

ansible-playbook audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=audio-duration-scaling" \
  -e "requested_cores=32"

What it measures: - Processing time for short (≤5s) clips - Medium (≤15s) clips - Long (≤30s) clips - Full-length (no truncation) - Linear vs non-linear scaling (is it O(n) with audio duration?)

Note: Stages use truncation-based buckets (max_duration), not exclusive ranges. GuideLLM's encode_media preprocessor does not support min_duration. For exclusive duration analysis, filter per-request audio_seconds from benchmarks.json in post-processing.

Use case: Capacity planning - "If our average audio file is 30 seconds, how many can we process per hour?"

Test Scenario: Constant Rate Stress

Serving Pattern: ONLINE SERVING (Sustained production load)

Answers: Can the system handle sustained load? Any memory leaks or degradation?

ansible-playbook audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=constant-rate-stress" \
  -e "requested_cores=64" \
  -e "requested_tensor_parallel=2"

What it measures: - Sustained load at 2, 5, 10 req/s (5 min each) - Extended duration test (15 min) - Latency stability over time (does P95 degrade after 10 minutes?) - Memory growth/leaks (resource exhaustion detection) - Error rates under sustained load

Use case: Production readiness validation - "Can this deployment handle 10 req/s continuously for hours without degradation?"

Test Scenario: Format Comparison

Serving Pattern: OFFLINE BATCH (Sequential processing)

Answers: Which audio format provides best performance/bandwidth tradeoff?

ansible-playbook audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=format-comparison" \
  -e "requested_cores=32"

What it measures: - MP3 (64kbps, 128kbps) - lossy compression - WAV (uncompressed, 8kHz/16kHz/48kHz) - raw audio - FLAC (lossless) - lossless compression - Stereo vs Mono channel impact - Decoding overhead vs network transfer time

Use case: Bandwidth optimization - "Should we send 64kbps MP3 or 16kHz WAV? Does higher quality improve accuracy or just waste bandwidth?"

Advanced Configuration

CPU Optimization

Auto-calculated (recommended):

# Tensor parallel and OMP threads auto-calculated
ansible-playbook audio-benchmark.yml \
  -e "test_model=openai/whisper-medium" \
  -e "test_scenario=transcription-throughput" \
  -e "requested_cores=64"

Manual configuration:

# Fine-tune CPU parameters
ansible-playbook audio-benchmark.yml \
  -e "test_model=openai/whisper-medium" \
  -e "test_scenario=transcription-throughput" \
  -e "requested_cores=64" \
  -e "requested_tensor_parallel=2" \
  -e "omp_num_threads=32" \
  -e "vllm_dtype=bfloat16" \
  -e "vllm_max_model_len=448" \
  -e "vllm_kvcache_space=4GiB"

Parameters Explained

Parameter	Description	Default	Impact
requested_cores	Total CPU cores to allocate	Required	Higher = more throughput
audio_num_files	Number of audio files per stage	From scenario YAML	Override file count for all stages
requested_tensor_parallel	Tensor parallelism degree (1,2)	Auto (1 or 2)	Parallel model execution
omp_num_threads	OpenMP threads per TP rank	Auto	CPU thread utilization
vllm_dtype	Model precision	auto (vLLM chooses)	Memory & speed tradeoff
vllm_max_model_len	Max sequence length	448 (Whisper)	Memory usage
vllm_kvcache_space	KV cache memory	2GiB	Batch size capacity

🔧 CPU Allocation Breakdown (Auto-Calculated)

The playbook automatically reserves 2 cores for vLLM's internal control plane. When you specify requested_cores=32:

Container Allocation:  cores 0-31 (32 total)
├── Control Plane:     2 cores (scheduler, KV cache, async ops)
└── Worker Threads:    30 cores (OMP_NUM_THREADS=30)
    └── Per TP rank:   30/TP cores

Formula:

OMP_NUM_THREADS = (requested_cores - 2) / tensor_parallel

Examples:
  requested_cores=32, TP=1 → OMP=30 (reserve 2 for control plane)
  requested_cores=64, TP=2 → OMP=31 (reserve 2 for control plane)
  requested_cores=8,  TP=1 → OMP=6  (reserve 2 for control plane)

Why reserve 2 cores from OMP? - vLLM control plane threads need CPU time for: - Main Python thread (GIL, request dispatcher, scheduler) - KV cache manager (memory allocation, cache eviction) - Async operations (network I/O, tokenization, preprocessing) - Without reservation: OMP workers saturate all cores, starving control plane - With reservation: Control plane gets dedicated CPU capacity, stable performance

Container and Execution Mode

Default behavior (containerized): - vLLM runs in container (podman) - GuideLLM runs in container (ghcr.io/vllm-project/guidellm:v0.6.0) - Requires sudo access for container management

Override for non-root execution:

# Run without containers and without sudo
ansible-playbook audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=transcription-throughput" \
  -e "requested_cores=32" \
  -e "guidellm_use_container=false" \
  -e "ansible_become=false"

# Prerequisites for host mode:
#   pip install guidellm[audio,recommended]
#   vLLM must be running separately

Use cases for non-container mode: - Development/testing environments - Systems without podman/docker - Running as non-root user without sudo - Custom vLLM installations

Using Different Models

Whisper models:

# Tiny (fastest, 39M params)
-e "test_model=openai/whisper-tiny"

# Small (balanced, 244M params)
-e "test_model=openai/whisper-small"

# Medium (high quality, 769M params)
-e "test_model=openai/whisper-medium"

Using Different Datasets

LibriSpeech (default): - Clean English speech - Good for baseline benchmarks

Common Voice (more diversity):

Edit the test scenario YAML file:

dataset:
  name: "mozilla-foundation/common_voice_11_0"
  config: "en"
  split: "test"
  audio_column: "audio"

Custom dataset: - Must be HuggingFace dataset or local JSON/CSV - Must have audio column with supported format

Results Analysis

Understanding Metrics

Audio-specific metrics: - audio_seconds: Total duration of audio processed - audio_samples: Raw sample count - audio_bytes: Payload size - audio_tokens: Model tokens (if reported)

Performance metrics: - Request throughput (req/s): Files per second - Audio throughput (audio_seconds/s): Audio seconds processed per wall-clock second - Real-time factor: processing_time / audio_duration - < 1.0 = Faster than real-time ✅ - = 1.0 = Real-time processing - > 1.0 = Slower than real-time ⚠️

Latency metrics: - Mean E2E latency: Average request time - P95/P99 latency: Tail latency (SLA validation) - TTFT: Time to first token (if supported)

Example Results Interpretation

Transcription Throughput Results:

Sequential:     10 files/sec,  80 audio_seconds/sec
Concurrent (4): 35 files/sec, 280 audio_seconds/sec
Speedup:        3.5x (near-linear scaling)
Real-time factor: 0.125 (8x faster than real-time)

Interpretation: - Sequential baseline: 10 files/sec - Concurrency helps significantly (3.5x speedup) - Processing 8x faster than real-time - Can handle 35 simultaneous users or batch 35 files

Troubleshooting

Common Issues

Issue: GuideLLM container image pull fails

# Solution: Check container registry access
podman pull ghcr.io/vllm-project/guidellm:v0.6.0

# Or set custom image
export GUIDELLM_CONTAINER_IMAGE=your-registry/guidellm:tag

Issue: Audio encoding errors

# Solution: Check dataset audio column format
# Supported: HF Audio feature, WAV/MP3/FLAC files, URLs, numpy/torch arrays

Issue: vLLM server won't start

# Check container logs
podman logs vllm-audio-<test_run_id>

# Check journald logs
journalctl -t vllm-audio-<test_run_id> -f

# Common causes:
# - Insufficient memory
# - Model download failed
# - Port already in use

Issue: Low throughput

# Try increasing cores or tensor parallel
-e "requested_cores=64" \
-e "requested_tensor_parallel=2"

Issue: Host-mode execution (non-container)

# Set use_container to false in group_vars
# Then install guidellm on load generator host:
pip install guidellm[audio,recommended]

Integration with Existing Tests

The audio test suite follows the same patterns as LLM and embedding tests:

vllm-cpu-perf-eval/
├── models/
│   ├── llm-models/
│   ├── embedding-models/
│   └── audio-models/          # ← New
│       └── model-matrix.yaml
├── tests/
│   ├── concurrent-load/
│   ├── embedding-models/
│   └── audio-models/          # ← New
│       ├── transcription-throughput.yaml
│       ├── transcription-latency.yaml
│       └── ...
└── automation/test-execution/ansible/
    ├── llm-benchmark-auto.yml
    ├── embedding-benchmark.yml
    └── audio-benchmark.yml    # ← New

Enterprise Pack (Quick Reference)

The enterprise pack is a minimal set of scenarios that answers the most common operator questions:

# 1. Throughput + concurrency
ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=transcription-throughput" \
  -e "requested_cores=32"

# 2. Latency SLA
ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=transcription-latency" \
  -e "requested_cores=32"

# 3. Accuracy (requires jiwer + datasets + soundfile + requests on the controller)
#    pip install jiwer datasets soundfile requests
ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=transcription-quality" \
  -e "requested_cores=32"

The transcription-quality scenario runs GuideLLM for RTF context, then automatically calls evaluate_audio_quality.py to compute WER/CER and write quality-results.json into the run directory.

A terminal report prints automatically after each playbook run. See docs/audio-benchmarking.md for full details.

Next Steps

1. Run baseline test to establish performance baseline
2. Optimize CPU settings based on results
3. Run comprehensive suite for production validation
4. Compare results across different models/configurations
5. Integrate into CI/CD for regression testing

Reference

• GuideLLM Audio Docs: guidellm/docs/guides/multimodal/audio.md
• Supported Models: See models/audio-models/model-matrix.yaml
• Test Scenarios: All YAML files in this directory
• Ansible Playbook: automation/test-execution/ansible/audio-benchmark.yml