Audio Model Benchmarking Guide¶

Comprehensive guide for benchmarking audio models (ASR, translation, chat) on vLLM CPU deployments.

Table of Contents¶

Overview
Quick Start
Test Scenarios
Understanding Results
Advanced Configuration
Troubleshooting
Best Practices

Overview¶

What Audio Models Are Supported?¶

Audio Transcription (ASR) - Speech-to-text (Whisper family)

Key Questions Answered¶

Offline Batch Processing: - "I have 1000 audio files - how long to transcribe them all?" - "What's my maximum throughput (files/second)?" - "How does audio duration affect processing time?"

Online Serving: - "How many concurrent users can my transcription API support?" - "What's the P95 latency under 10 concurrent users?" - "Can the system handle sustained load without degradation?"

Prerequisites¶

System Requirements: - vLLM 0.19.0+ with audio support - Audio dependencies: librosa, soundfile, ffmpeg-python - Podman (containerized mode) or direct Python install (host mode) - Network connectivity between load generator and vLLM server

AWS/Cloud Users: - Security groups must allow TCP port 8000 between instances - For VPC deployments, use private IPs for better performance - See Troubleshooting: Network Connectivity below

Quick Start¶

From the repository root, install Ansible and Galaxy collections once:

./cpueval install

1. Configure Hosts¶

# Set connection variables
export DUT_HOSTNAME=your-vllm-server-hostname
export LOADGEN_HOSTNAME=your-load-generator-hostname
export ANSIBLE_SSH_USER=ec2-user
export ANSIBLE_SSH_KEY=~/.ssh/your-key.pem
export HF_TOKEN=$(cat ~/hf_token)  # If using gated models

# For AWS VPC (optional but recommended):
export DUT_PRIVATE_IP=172.31.x.x  # Private IP for faster benchmarking

2. Run First Test¶

cd automation/test-execution/ansible

# Quick test (5 files per stage, ~2 minutes)
ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-tiny" \
  -e "test_scenario=transcription-throughput" \
  -e "requested_cores=32" \
  -e "audio_num_files=5"

# Production test (100 files per stage)
ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=transcription-throughput" \
  -e "requested_cores=32"

3. View Results¶

# Results location
ls results/audio-models/openai__whisper-tiny/transcription-throughput-*/

# Launch Streamlit dashboard
cd automation/test-execution/dashboard-examples/vllm_dashboard
./launch-dashboard.sh
# Open http://localhost:8501 and set results path to audio-models

Test Scenarios¶

Transcription Throughput¶

Purpose: Measure total time and throughput for processing N audio files

Use Cases: - Batch processing: "How long for 100 call recordings?" - API capacity: "How many concurrent users can we support?"

Stages: 1. Sequential - Process files one-by-one (offline batch baseline) 2. Concurrent-2/4/8 - Simulate 2/4/8 concurrent users (online serving) 3. Max-throughput - Find maximum files/second capacity

Example:

ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=transcription-throughput" \
  -e "requested_cores=64" \
  -e "audio_num_files=100"

Key Metrics: - Sequential stage: Total time for batch processing - Concurrent stages: Per-request latency (P50, P95, P99) - Max-throughput: Files/second, audio_seconds/second

Transcription Latency¶

Purpose: Measure latency under different concurrent loads (online serving focus)

Use Cases: - SLA validation: "Can we guarantee <200ms P95 latency?" - Capacity planning: "How many concurrent users before latency degrades?"

Example:

ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=transcription-latency" \
  -e "requested_cores=32"

Audio Duration Scaling¶

Purpose: Understand how processing time scales with audio length

Use Cases: - Capacity planning: "If average call is 5 minutes, how many/hour?" - Performance optimization: "Is processing time linear with audio duration?"

Example:

ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=audio-duration-scaling" \
  -e "requested_cores=32"

Constant Rate Stress¶

Purpose: Validate sustained load stability (production readiness)

Use Cases: - Production validation: "Can we handle 10 req/s continuously?" - Memory leak detection: "Does performance degrade over time?"

Example:

ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=constant-rate-stress" \
  -e "requested_cores=64"

Format Comparison¶

Purpose: Compare performance across different audio formats

Use Cases: - Bandwidth optimization: "MP3 vs WAV - which is faster?" - Quality tradeoffs: "Does higher quality improve accuracy?"

Example:

ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
  -e "test_model=openai/whisper-small" \
  -e "test_scenario=format-comparison" \
  -e "requested_cores=32"

Understanding Results¶

Result Directory Structure¶

Base path: results/audio-models/openai__whisper-small/transcription-throughput-20260423-103307/

Path	Description
`sequential/benchmarks.json`	Full GuideLLM results (offline batch baseline)
`sequential/benchmarks.csv`	Summary CSV
`sequential/guidellm.log`	Benchmark logs
`concurrent-2/benchmarks.json`	2 concurrent users
`concurrent-2/benchmarks.csv`	Summary CSV
`concurrent-4/`	4 concurrent users
`concurrent-8/`	8 concurrent users
`max-throughput/`	Maximum capacity test
`vllm-metrics.json`	vLLM server metrics
`test-metadata.json`	Test configuration

Key Metrics Explained¶

Throughput Metrics: - requests_per_second.successful.mean: Average files/second - requests_per_second.successful.p50/p95/p99: Percentile throughput

Latency Metrics: - end_to_end_latency.mean: Average request time - end_to_end_latency.p95/p99: Tail latency (SLA validation)

Success Metrics: - request_totals.successful: Completed requests - request_totals.errored: Failed requests - Success rate = successful / (successful + errored)

Audio-Specific Metrics (if available): - audio_seconds: Total audio duration processed - audio_throughput: Audio seconds per wall-clock second - Real-time factor = processing_time / audio_duration

Example Interpretation¶

Stage: sequential
  Duration: 2.34s
  Successful: 10 requests
  Throughput: 4.26 req/s

Stage: concurrent-8
  Duration: 0.91s
  Successful: 10 requests
  Throughput: 11.02 req/s
  Speedup: 2.6x vs sequential

Interpretation: - Sequential: Baseline 4.26 files/sec - Concurrent-8: 2.6x speedup with 8 concurrent requests - Measured throughput: ~11 files/sec under sustained concurrent load

Advanced Configuration¶

CPU Optimization¶

Auto-calculated (recommended):

ansible-playbook audio-benchmark.yml \
  -e "test_model=openai/whisper-medium" \
  -e "test_scenario=transcription-throughput" \
  -e "requested_cores=64"
# TP and OMP auto-calculated based on cores

Manual tuning:

ansible-playbook audio-benchmark.yml \
  -e "test_model=openai/whisper-medium" \
  -e "requested_cores=64" \
  -e "requested_tensor_parallel=2" \
  -e "omp_num_threads=31" \
  -e "vllm_dtype=bfloat16"

CPU Allocation Formula¶

Component	Cores	Formula
Container total	`requested_cores`	e.g. 32
Control plane	2	Scheduler, KV cache, async ops
Worker threads	remaining	`(requested_cores - 2) / tensor_parallel`

Examples:

`requested_cores`	`tensor_parallel`	OMP threads
32	1	30
64	2	31 per rank
16	1	14

Using Different Models¶

Whisper Models:

# Tiny (fastest, 39M params)
-e "test_model=openai/whisper-tiny"

# Small (balanced, 244M params)
-e "test_model=openai/whisper-small"

# Medium (best quality, 769M params)
-e "test_model=openai/whisper-medium"

Non-Container Mode¶

Requirements:

# Install GuideLLM with audio support
pip install guidellm[audio,recommended]

# Ensure vLLM is running separately

Run without containers (external endpoint):

ansible-playbook audio-benchmark.yml \
  -e "test_scenario=transcription-throughput" \
  -e '{"vllm_endpoint": {"mode": "external", "external": {"url": "http://your-vllm-host:8000"}}}' \
  -e "guidellm_use_container=false" \
  -e "ansible_become=false"

Troubleshooting¶

Network Connectivity¶

Symptom: Load generator cannot reach vLLM server

FAILED: Verify load generator can reach vLLM server
Status: -1 (timeout)

Root Cause: Firewall/security group blocking port 8000

Solution 1: Fix Security Group (AWS/Cloud)

# Add inbound rule to vLLM server security group:
# - Protocol: TCP
# - Port: 8000
# - Source: Load generator security group or IP

Solution 2: Use Private IP (AWS VPC)

# Get DUT private IP (from vLLM server):
hostname -I

# Set environment variable:
export DUT_PRIVATE_IP=172.31.x.x

# Update inventory/hosts.yml:
bench_config:
  vllm_host: "{{ lookup('env', 'DUT_PRIVATE_IP') | default(hostvars['vllm-server']['ansible_host'], true) }}"

Verification:

# From load generator:
ssh ec2-user@$LOADGEN_HOSTNAME
curl -v http://$DUT_HOSTNAME:8000/health
curl -v http://$DUT_PRIVATE_IP:8000/health

Missing Audio Dependencies¶

Symptom: vLLM server fails to start with audio models

ModuleNotFoundError: No module named 'librosa'
ModuleNotFoundError: No module named 'soundfile'

Root Cause: vLLM v0.19.0 image doesn't include audio dependencies

Solution: Playbook automatically builds custom image with audio deps

The playbook detects this and builds localhost/vllm-audio-cpu:v0.19.0 with: - librosa - soundfile - ffmpeg-python

Manual verification:

# Check if audio image exists
podman images | grep vllm-audio-cpu

# Verify dependencies
podman run --rm --entrypoint python3 \
  localhost/vllm-audio-cpu:v0.19.0 \
  -c 'import librosa; import soundfile; print("OK")'

Force rebuild:

# Remove existing image
podman rmi localhost/vllm-audio-cpu:v0.19.0

# Re-run playbook (will rebuild)

Multiple vLLM Containers Running¶

Symptom: Port 8000 already in use, or multiple containers running

Root Cause: Previous test didn't clean up containers

Solution: Playbook now automatically stops all vllm-* containers

Manual cleanup:

# On vLLM server (DUT):
podman ps -a | grep vllm-
podman stop $(podman ps -a --filter "name=vllm-" --format "{{.Names}}")
podman rm $(podman ps -a --filter "name=vllm-" --format "{{.Names}}")

Dataset Download Issues¶

Symptom: Benchmark hangs at "Downloading data"

Downloading data: 65% ... (very slow)

Root Cause: Downloading entire LibriSpeech dataset (2620 samples, 48 tar files)

Solution: Playbook now limits dataset samples to 2 × max_requests

For max_requests=100, only 200 samples are loaded (avoids full download).

Manual override:

# Test with fewer files
-e "audio_num_files=10"  # Loads only 20 samples

GuideLLM Backend Validation Failed¶

Symptom: GuideLLM exits during startup

RuntimeError: Backend validation request failed.
Could not connect to the server or validate the backend configuration.

Possible Causes: 1. vLLM server not fully started (wait longer) 2. Network connectivity (see Network Connectivity) 3. Audio endpoint not available (missing audio support)

Diagnostics:

# Check vLLM is running
podman ps | grep vllm-audio

# Check vLLM logs
sudo journalctl -t vllm-audio-openai__whisper-tiny -f

# Test endpoints manually
curl http://$DUT_HOSTNAME:8000/health
curl http://$DUT_HOSTNAME:8000/v1/models
curl -X OPTIONS http://$DUT_HOSTNAME:8000/v1/audio/transcriptions

Pre-flight checks (automatically run): - Load generator → vLLM health check - Audio endpoint accessibility test

Low Throughput¶

Symptom: Much lower throughput than expected

Possible Causes: 1. Insufficient CPU cores 2. Tensor parallelism not optimal 3. vLLM server throttled by control plane

Solutions:

# Increase cores
-e "requested_cores=64"

# Try different TP settings
-e "requested_tensor_parallel=1"  # vs 2

# Check if OMP threads auto-calculated correctly
# Should be: (cores - 2) / TP

Diagnostics:

# Check vLLM CPU usage
ssh $DUT_HOSTNAME
top -H -p $(pgrep -f vllm-audio)

# Check container resource limits
podman inspect vllm-audio-* | jq '.[].HostConfig.CpusetCpus'

Results Not Fetched¶

Symptom: Results directory empty on controller (local machine)

Root Cause: Ansible fetch tasks failed

Check:

# On load generator:
ssh $LOADGEN_HOSTNAME
ls -la /path/to/results/audio-models/.../*/benchmarks.json

Solution:

# Fetch manually
scp -r $LOADGEN_HOSTNAME:/path/to/results/audio-models ./results/

Best Practices¶

1. Start with Quick Tests¶

# Quick validation (5 files, 2 minutes)
-e "audio_num_files=5"

# Then scale up for production
-e "audio_num_files=100"

2. Use Private IPs in Cloud/VPC¶

# AWS VPC: Use private IPs for 10x lower latency
export DUT_PRIVATE_IP=172.31.x.x

3. Verify Pre-flight Checks¶

Watch for these in playbook output:

✓ Load Generator → vLLM Connectivity: PASS
✓ Audio Endpoint Pre-flight Check: PASS

4. Monitor vLLM Logs¶

# In separate terminal during benchmark:
ssh $DUT_HOSTNAME
sudo journalctl -t vllm-audio-* -f

5. Clean Up Between Tests¶

# Playbook now auto-cleans, but verify:
podman ps -a | grep vllm-
# Should show only current test container

6. Organize Results¶

Results are auto-organized by timestamp under results/audio-models/<model>/<scenario>-<timestamp>/. Each run directory contains subdirectories for each concurrency level (sequential/, concurrent-2/, concurrent-4/, etc.) plus top-level vllm-metrics.json and test-metadata.json.

# Export to CSV for analysis
cp results/audio-models/.../*/benchmarks.csv /analysis/

7. Document Your Configuration¶

# Save test metadata
cat results/audio-models/.../test-metadata.json

{
  "model": "openai/whisper-small",
  "scenario": "transcription-throughput",
  "cores": 32,
  "tensor_parallel": 1,
  "timestamp": "20260423-103307"
}

Default Enterprise Pack¶

The enterprise pack is a recommended run list that answers the most common operator questions with minimal wall-clock time.

Models: whisper-small (primary), whisper-tiny (fast baseline)

Default scenarios:

Scenario	Purpose	~Runtime (100 files, 32 cores)
transcription-throughput	Batch capacity & online concurrency	~5 min
transcription-latency	SLA validation (P95/P99)	~5 min
transcription-quality	WER/CER accuracy (50 clips)	~3 min

Deep-dive only (run when needed): - format-comparison — audio format impact on throughput - audio-duration-scaling — processing time vs clip length - constant-rate-stress — sustained load stability - whisper-medium — higher-accuracy model comparison

Run the pack:

cd automation/test-execution/ansible

for scenario in transcription-throughput transcription-latency transcription-quality; do
  ansible-playbook -i inventory/hosts.yml audio-benchmark.yml \
    -e "test_model=openai/whisper-small" \
    -e "test_scenario=$scenario" \
    -e "requested_cores=32"
done

Controller prerequisites for quality evaluation:

pip install jiwer datasets soundfile requests

View results:

# Terminal report (prints automatically after each run)
python3 automation/test-execution/scripts/ansible/audio_enterprise_report.py \
  results/audio-models/

# Dashboard
cd automation/test-execution/dashboard-examples/vllm_dashboard
./launch-dashboard.sh
# Open Audio Metrics → tabs: Performance, Quality, Capacity / Sizing

Terminal Report¶

The CLI report summarises enterprise metrics without requiring the Streamlit dashboard.

# All runs in the results tree
python3 scripts/ansible/audio_enterprise_report.py results/audio-models/

# Filter to one run
python3 scripts/ansible/audio_enterprise_report.py results/audio-models/ \
  --run-id 20260723-103307

# Custom P95 target + batch ETA
python3 scripts/ansible/audio_enterprise_report.py results/audio-models/ \
  --p95-target 1.5 --eta-files 100000 --eta-audio-hours 500

# JSON output for scripts
python3 scripts/ansible/audio_enterprise_report.py results/audio-models/ --json

Example output:

==================================================
  Audio Enterprise Report
  Model: openai/whisper-small | Cores: 32
  Scenario: transcription-throughput
  Run: 20260723-103307
==================================================

Quality
  WER: 4.2% (n=50)

Offline Batch
  Audio hours/hour:  18.40
  Files/hour:        12,400

Online (P95 ≤ 2.0s)
  Max concurrency: 8
  P95 @ max:       1.70s

Efficiency
  Throughput/core:        0.58
  Core-hours/audio-hour:  1.72

Warmup
  Ready time:       12.30s
  First RTF:        1.10
  Steady RTF:       0.062
==================================================

The report prints automatically at the end of each playbook run. Suppress with -e audio_print_enterprise_report=false.

Quality Evaluation (WER)¶

Transcription accuracy is measured with evaluate_audio_quality.py, which sends audio clips to a running vLLM endpoint and computes WER/CER with jiwer.

Automated (recommended): The playbook runs the evaluator automatically when test_scenario=transcription-quality. It writes quality-results.json into the run directory (next to test-metadata.json) so the dashboard and CLI report can join WER to the correct run.

Prerequisites (on the Ansible controller):

pip install jiwer datasets soundfile requests

If these are missing the playbook prints a warning but does not fail.

Manual standalone (against a live endpoint):

# Point --output-dir at a run directory that has test-metadata.json
python3 automation/test-execution/scripts/ansible/evaluate_audio_quality.py \
  --endpoint http://dut:8000 \
  --output-dir results/audio-models/openai__whisper-small/transcription-quality-<run-id>/ \
  --model openai/whisper-small \
  --num-clips 50

# Or pass --test-run-id and --cores explicitly when no test-metadata.json exists
python3 automation/test-execution/scripts/ansible/evaluate_audio_quality.py \
  --endpoint http://dut:8000 \
  --output-dir /tmp/quality-eval/ \
  --model openai/whisper-small \
  --test-run-id my-run-001 --cores 32

Use local audio files instead of HuggingFace:

python3 evaluate_audio_quality.py \
  --endpoint http://dut:8000 \
  --output-dir results/ \
  --audio-dir /path/to/clips/ \
  --model openai/whisper-small

The --audio-dir must contain audio files and a references.json mapping filenames to ground-truth text. Note: --audio-format applies only to local files; HuggingFace clips are always uploaded as WAV.

Results are written to quality-results.json and automatically picked up by both the dashboard (Quality tab) and the terminal report.

Future Work¶

The following areas are planned but not yet implemented:

Streaming / chunked audio — measure latency for real-time streaming ASR
Distil-Whisper / large-v3 — additional model sizes for accuracy-speed trade-offs
Backend comparison — whisper.cpp, OpenVINO, CTranslate2 vs vLLM
Poisson arrivals — realistic request arrival patterns
VAD → ASR pipeline — voice activity detection feeding into transcription
CI integration — automated regression testing with quick-test + WER gate

Reference¶

Audio Test Suite: tests/audio-models/README.md
Model Matrix: models/audio-models/model-matrix.yaml
Ansible Playbook: automation/test-execution/ansible/audio-benchmark.yml
Terminal Report: scripts/ansible/audio_enterprise_report.py
Quality Evaluator: scripts/ansible/evaluate_audio_quality.py
GuideLLM Audio Docs: GuideLLM Audio Guide
vLLM Audio Support: vLLM Audio Documentation