Ansible Test Execution Guide¶

Automated testing framework for vLLM performance evaluation with NUMA-aware CPU optimization.

📚 Complete Reference: For full details on all playbooks, roles, and configuration options, see the Ansible automation guide in the repository.

Quick Start¶

Prerequisites¶

Control Machine (where you run Ansible):

From the repository root:

# Install Ansible, Galaxy collections, and tab completion
./cpueval install

See Getting Started and the cpueval CLI install command for platform notes (--skip-system-deps, UBI 9, macOS/Ubuntu).

Then run playbooks from automation/test-execution/ansible/.

Test Hosts (DUT and Load Generator):

OS: Ubuntu 22.04+, RHEL 9+, or Fedora 38+
SSH access with sudo privileges
Python 3.8+

Run Your First Test¶

# Simple LLM benchmark
ansible-playbook llm-benchmark-auto.yml \
  -e "test_model=meta-llama/Llama-3.2-1B-Instruct" \
  -e "workload_type=chat" \
  -e "requested_cores=16"

Available Playbooks¶

LLM Testing¶

Playbook	Purpose	Configuration
`llm-benchmark-auto.yml`	Single LLM test	Auto (core count)
`llm-benchmark-concurrent-load.yml`	3-phase concurrent load test	Auto
`llm-core-sweep-auto.yml`	Multi-core sweep	Auto
`start-vllm-server.yml`	Start vLLM server without benchmark	Manual

Embedding Model Testing¶

Playbook	Purpose	Configuration
`embedding-benchmark.yml`	Embedding model performance test	Manual
`mteb-benchmark.yml`	MTEB quality benchmarks	Manual

Results Management¶

Playbook	Purpose
`collect-sweep-results.yml`	Consolidate core sweep results
`log-to-mlflow.yml`	Log benchmark results to MLflow (single or batch)
`publish-existing-results.yml`	Republish GuideLLM results to Prometheus

Monitoring & Visualization¶

Playbook	Purpose
`start-grafana.yml`	Start Grafana dashboard with Prometheus
`stop-grafana.yml`	Stop Grafana and Prometheus services

Platform Setup¶

Playbook	Purpose
`setup-platform.yml`	Install packages, configure performance settings
`health-check.yml`	Verify system readiness

Test Configuration¶

Basic Parameters¶

Parameter	Description	Example
`test_model`	HuggingFace model ID	`meta-llama/Llama-3.2-1B-Instruct`
`workload_type`	Test workload	`chat`, `code`, `summarization`
`requested_cores`	CPU cores for vLLM	`16`, `32`, `64`

Optional Parameters¶

Parameter	Description	Default
`requested_tensor_parallel`	Tensor parallelism	Auto-calculated
`guidellm_profile`	Benchmark profile	`concurrent`
`guidellm_rate`	Concurrency levels	`[1,2,4,8,16,32]`
`guidellm_max_seconds`	Test duration (seconds)	`600`

Advanced Features¶

Socket Pinning (NUMA Affinity)¶

Socket pinning isolates vLLM server and GuideLLM load generator to different CPU sockets/NUMA nodes, minimizing performance interference.

Supported Playbooks:

✅ llm-benchmark-auto.yml
✅ llm-benchmark-concurrent-load.yml

Parameters:

Parameter	Description	Example
`vllm_cpu_start`	CPU ID to start vLLM allocation	`64` (socket 1)
`vllm_numa_node`	NUMA node for vLLM	`1`
`guidellm_cpus`	CPU range for load generator	`"0-31"`
`guidellm_numa_node`	NUMA node for load generator	`0`

Determine System Socket Layout:

# Show NUMA topology
lscpu | grep NUMA

# Show cores per NUMA node
numactl --hardware

# Example output:
# node 0 cpus: 0 1 2 ... 31
# node 1 cpus: 64 65 66 ... 95

Example: vLLM on Socket 1, GuideLLM on Socket 0

Assuming 2-socket system:

Socket 0: Cores 0-31 (NUMA node 0)
Socket 1: Cores 64-95 (NUMA node 1)

# Single test with socket separation
ansible-playbook llm-benchmark-auto.yml \
  -e "test_model=meta-llama/Llama-3.2-1B-Instruct" \
  -e "workload_type=chat" \
  -e "requested_cores=32" \
  -e "vllm_cpu_start=64" \
  -e "vllm_numa_node=1" \
  -e "guidellm_cpus=0-31" \
  -e "guidellm_numa_node=0"

# Concurrent load test with socket separation
ansible-playbook llm-benchmark-concurrent-load.yml \
  -e "test_model=meta-llama/Llama-3.2-1B-Instruct" \
  -e "base_workload=chat" \
  -e "requested_cores=32" \
  -e "vllm_cpu_start=64" \
  -e "vllm_numa_node=1" \
  -e "guidellm_cpus=0-31" \
  -e "guidellm_numa_node=0"

Validation:

# Verify vLLM container pinning
podman ps | grep vllm
podman inspect <container-id> | grep -A 2 cpuset
# Expected: "CpusetCpus": "64-95", "CpusetMems": "1"

# Verify GuideLLM container pinning
podman ps | grep guidellm
podman inspect <container-id> | grep -A 2 cpuset
# Expected: "CpusetCpus": "0-31", "CpusetMems": "0"

Use Cases:

Minimize Interference: Eliminate CPU contention between server and load generator
Test Cross-NUMA Performance: Measure impact of cross-socket memory access
Multi-Tenant Systems: Isolate benchmarks from other workloads

Notes:

Socket pinning requires vllm_numa_node to be set
When pinning to single NUMA node, tensor_parallel must equal 1
The automation validates requested cores fit within specified socket

Tensor Parallel (TP) for Model Sharding¶

Tensor parallelism splits model layers across multiple NUMA nodes to improve performance on high-latency workloads.

When to Use: - Large input contexts: RAG (8K tokens), Summarization (2K tokens) - 2+ socket systems: Distribute model across NUMA nodes - Memory-bound workloads: Split large models to utilize more memory bandwidth

Supported Values: 1 (disabled), 2, 4, or 8

Requirements: - Core count must be evenly divisible by TP value - Example: 32 cores with TP=2 → 16 cores per NUMA node

Examples:

# Test with TP=2 on 32 cores (auto-distributes across 2 NUMA nodes)
ansible-playbook llm-benchmark-auto.yml \
  -e "test_model=meta-llama/Llama-3.2-3B-Instruct" \
  -e "workload_type=rag" \
  -e "requested_cores=32" \
  -e "requested_tensor_parallel=2"

# Concurrent load test with TP=2
ansible-playbook llm-benchmark-concurrent-load.yml \
  -e "test_model=meta-llama/Llama-3.2-3B-Instruct" \
  -e "base_workload=summarization" \
  -e "requested_cores=64" \
  -e "requested_tensor_parallel=2"

# TP=4 for very large models on 4-socket systems
ansible-playbook llm-benchmark-auto.yml \
  -e "test_model=meta-llama/Llama-3.2-3B-Instruct" \
  -e "workload_type=rag" \
  -e "requested_cores=64" \
  -e "requested_tensor_parallel=4"

Auto-Calculation: If requested_tensor_parallel is not specified, the automation calculates an optimal value based on: - Number of available NUMA nodes - Requested core count - System topology

Dashboard Filtering: Results from TP tests can be filtered in the Client Metrics dashboard using the "Tensor Parallel" dropdown.

Performance Impact: - ✅ Improves: Large context processing (RAG, summarization) - ✅ Improves: Memory bandwidth utilization on multi-socket systems - ⚠️ May degrade: Small context workloads (chat, code) due to inter-node communication overhead

Compatibility: - Cannot combine with single-NUMA socket pinning (use multi-NUMA or auto) - Best results on systems with fast inter-node links (UPI on Intel, Infinity Fabric on AMD)

3-Phase Concurrent Load Testing¶

The concurrent load test runs three phases to evaluate different scenarios:

Phase 1: Baseline (Fixed Tokens, No Caching)

Fixed input/output token counts
Prefix caching disabled
Establishes baseline performance

Phase 2: Realistic (Variable Tokens, No Caching)

Variable token counts (realistic traffic)
Prefix caching disabled
Tests production-like variability

Phase 3: Production (Variable Tokens, With Caching)

Variable token counts
Prefix caching enabled
Tests production configuration

Example:

# Run all 3 phases
ansible-playbook llm-benchmark-concurrent-load.yml \
  -e "test_model=meta-llama/Llama-3.2-1B-Instruct" \
  -e "base_workload=chat" \
  -e "requested_cores=32"

# Run only Phase 1 (baseline)
ansible-playbook llm-benchmark-concurrent-load.yml \
  -e "test_model=meta-llama/Llama-3.2-1B-Instruct" \
  -e "base_workload=chat" \
  -e "requested_cores=16" \
  -e "skip_phase_2=true" \
  -e "skip_phase_3=true"

Platform-Specific Configurations¶

AMD EPYC with ZenDNN¶

AMD provides optimized vLLM containers with ZenDNN (Zen Deep Neural Network) and ZenTorch for improved performance on AMD EPYC processors.

Why custom configuration is needed:

AMD ZenDNN containers require an explicit entrypoint (vllm serve) because the vLLM executable is not in the default PATH.

Setup:

# AMD ZenDNN optimized container
export VLLM_CONTAINER_IMAGE=docker.io/amdih/zendnn_zentorch:vllm_v0.18.0_zentorch_v5.2.1_rhel9.5_r5.2.1
export VLLM_CONTAINER_ENTRYPOINT='["vllm", "serve"]'
export VLLM_HEALTH_TIMEOUT=1200  # Longer startup time for ZenDNN

# Run benchmark
ansible-playbook llm-benchmark-auto.yml \
  -e "test_model=meta-llama/Llama-3.2-1B-Instruct" \
  -e "workload_type=chat" \
  -e "requested_cores=64"

Available versions:

RHEL 9.5: docker.io/amdih/zendnn_zentorch:vllm_v0.18.0_zentorch_v5.2.1_rhel9.5_r5.2.1
Ubuntu 22.04: docker.io/amdih/zendnn_zentorch:vllm_v0.18.0_zentorch_v5.2.1_ubuntu22.04_r5.2.1

What gets optimized:

ZenDNN: AMD's optimized deep learning library
ZenTorch: PyTorch integration with ZenDNN
NUMA-aware execution for AMD EPYC multi-die architecture
AVX-512 & AVX2 CPU instruction optimizations

NUMA Pinning:

AMD ZenDNN containers use CPU pinning with special thread affinity configuration: - CPU pinning via cpuset_cpus - Thread affinity via VLLM_CPU_OMP_THREADS_BIND environment variable - Memory binding skipped (ZenDNN libnuma requires access to all nodes)

Socket pinning parameters work normally - the automation detects AMD containers and applies the correct configuration automatically.

Verify ZenDNN is active:

# Check logs for ZenDNN initialization
ssh <dut-hostname> "podman logs vllm-server 2>&1 | grep -i zendnn"

Switch back to standard vLLM:

export VLLM_CONTAINER_IMAGE=docker.io/vllm/vllm-openai-cpu:v0.18.0
unset VLLM_CONTAINER_ENTRYPOINT

For complete details, see the AMD ZenDNN section in the full Ansible guide.

Results¶

Test results are saved to results/llm/<model>/<test-run-id>/:

File	Description
`benchmarks.json`	GuideLLM benchmark results
`benchmarks.csv`	CSV format
`test-metadata.json`	Test configuration
`vllm-metrics.log`	vLLM server metrics
`guidellm.log`	Load generator logs

Example path: results/llm/meta-llama__Llama-3.2-1B-Instruct/20240428-120000/

Workload Types¶

Baseline Workloads (Fixed Tokens)¶

Workload	Input Tokens	Output Tokens	Use Case
`chat`	1024	256	Conversational AI
`code`	1024	512	Code generation
`summarization`	2048	256	Document summarization
`rag`	2048	512	RAG applications

Variable Workloads (Realistic Traffic)¶

Workload	Token Range	Use Case
`chat_var`	ISL: 256-2048, OSL: 64-512	Production chat
`code_var`	ISL: 512-2048, OSL: 128-1024	Production code gen
`summarization_var`	ISL: 1024-4096, OSL: 64-512	Production summarization

Troubleshooting¶

Connection Issues¶

# Test SSH connectivity
ansible -i inventory/hosts.yml all -m ping

# Verbose output
ansible -i inventory/hosts.yml all -m ping -vvv

vLLM Won't Start¶

# Check container logs
podman logs <container-id>

# Check if port is in use
ss -tlnp | grep 8000

HuggingFace Token¶

For gated models (Llama, Mistral):

# Set token as environment variable
export HF_TOKEN=your_token_here

# Or configure in inventory
vim inventory/group_vars/all/credentials.yml

Next Steps¶

Understanding Metrics - Metrics definitions
Platform Setup - Intel Xeon tuning
Complete Ansible Guide - Full reference