Skip to content

vLLM Performance Evaluation Guide - CPU Inferencing

Introduction

This guide outlines the methodology, tools, and procedures for evaluating the performance of the vLLM framework (CPU Mode) when running Small Language Models (SLMs), Tiny Models, and Embedding models. This evaluation is structured into multiple phases, with each phase incorporating specific models, test scenarios, and performance goals.

Primary Goals

The primary goals of this performance evaluation effort are to:

  • Establish Baseline Performance for CPU inferencing in vLLM: Determine the fundamental performance characteristics (latency and throughput) of vLLM's CPU inferencing capabilities for various SLMs
  • Identify Optimal Operating Points: Find the sweet spot between throughput and latency for different workload types
  • Characterize Architectural Differences: Understand performance characteristics across different model architectures

Future Stages

Future stages will incorporate the following goals:

  • Establish which models work on CPU
  • Establish guidance on coexistence of CPU inference loads and enterprise compute/service loads
  • Assess the performance impact of using a subset of cores and/or NUMA nodes
  • Measure configuration impact of vLLM CPU tuning and quantization parameters

Performance Evaluation Utility

The chosen Performance Evaluation tool is GuideLLM (specifically v0.5.0). GuideLLM is a platform for evaluating and optimizing the deployment of Large Language Models (LLMs). By simulating real-world inference workloads, GuideLLM enables users to assess the performance, resource requirements, and cost implications of deploying LLMs on various hardware configurations.

GuideLLM serves a dual role: it functions as the benchmark utility, meaning it provides the framework and metrics for measuring performance, and it also acts as the load generator, meaning it simulates the user requests or workload necessary to test the system under realistic or stress conditions.

GuideLLM Parameters

Test Type Guidellm CLI Parameter Example Guidellm Podman Parameter Example
concurrent guidellm benchmark --target "http://localhost:8000" --profile concurrent --warmup 0.1 --rate 1,2,4,8,16,32 --max-seconds 600 --request-timeout 600 --data "prompt_tokens=512,output_tokens=256" sudo podman run --rm -it --network=host --cpuset-cpus=17-31 -v "/tmp/results:/results:z" -e GUIDELLM_TARGET=http://localhost:8000 -e GUIDELLM_PROFILE=concurrent -e GUIDELLM_RATE=1,2,4,8,16,32 -e GUIDELLM_MAX_SECONDS=600 -e GUIDELLM_REQUEST_TIMEOUT=600 -e GUIDELLM_WARMUP=0.1 -e GUIDELLM_DATA="prompt_tokens=512,output_tokens=256" -e HF_TOKEN=$HF_TOKEN ghcr.io/vllm-project/guidellm:v0.7.2
sweep guidellm benchmark --target "http://localhost:8000" --profile sweep --warmup 30 --rampup 15.0 --max-requests 130 --data "prompt_tokens=512,output_tokens=256" sudo podman run --rm -it --network=host --cpuset-cpus=17-31 -v "/tmp/results:/results:z" -e GUIDELLM_TARGET=http://localhost:8000 -e GUIDELLM_PROFILE=sweep -e GUIDELLM_RAMPUP=15.0 -e GUIDELLM_WARMUP=30 -e GUIDELLM_MAX_REQUESTS=130 -e GUIDELLM_DATA="prompt_tokens=512,output_tokens=256" -e HF_TOKEN=$HF_TOKEN ghcr.io/vllm-project/guidellm:v0.7.2
synchronous guidellm benchmark --target "http://localhost:8000" --profile synchronous --warmup 30 --max-requests 100 --data "prompt_tokens=512,output_tokens=256" sudo podman run --rm -it --network=host --cpuset-cpus=17-31 -v "/tmp/results:/results:z" -e GUIDELLM_TARGET=http://localhost:8000 -e GUIDELLM_PROFILE=synchronous -e GUIDELLM_WARMUP=30 -e GUIDELLM_MAX_REQUESTS=100 -e GUIDELLM_DATA="prompt_tokens=512,output_tokens=256" -e HF_TOKEN=$HF_TOKEN ghcr.io/vllm-project/guidellm:v0.7.2
poisson guidellm benchmark --target "http://localhost:8000" --profile poisson --warmup 30 --rate 32 --max-requests 280 --data "prompt_tokens=512,output_tokens=256" sudo podman run --rm -it --network=host --cpuset-cpus=17-31 -v "/tmp/results:/results:z" -e GUIDELLM_TARGET=http://localhost:8000 -e GUIDELLM_PROFILE=poisson -e GUIDELLM_RATE=32 -e GUIDELLM_MAX_REQUESTS=280 -e GUIDELLM_WARMUP=30 -e GUIDELLM_DATA="prompt_tokens=512,output_tokens=256" -e HF_TOKEN=$HF_TOKEN ghcr.io/vllm-project/guidellm:v0.7.2

Parameter Clarifications

Warmup (--warmup):

  • Excludes initial requests/time from metrics to allow the system to stabilize
  • With --max-requests: Value represents number of warmup requests (e.g., --warmup 30 = first 30 requests excluded)
  • With time-based tests: Use fraction for percentage (e.g., --warmup 0.1 = 10% of duration)
  • Warmup requests are executed but not included in final metrics

Rampup (--rampup):

  • Gradually increases request rate over specified seconds to avoid sudden load spikes
  • Specified in seconds (e.g., --rampup 15.0 = 15 seconds to reach target rate)
  • Applicable to: throughput, concurrent, and constant profiles
  • Not applicable to: synchronous and poisson profiles
  • Helps identify system behavior during load transitions

Note: GuideLLM does not support embedding endpoints. For embedding model testing, use vllm bench serve with the --backend openai-embeddings flag instead.

Testbed Configuration

Component Description
Device Under Test (DUT) Server Platform (e.g. Intel Xeon 6, AMD EPYC 9005)
Load Generator Separate node (≥ 16 cores), same network segment as DUT (≥ 10 GbE), running GuideLLM
SUT Boundary Model Engine (vLLM inference runtime only, per IETF profiles)

Guidellm Simulated Workload Rate Types

GuideLLM supports various mechanisms for driving inference load against the vLLM server, allowing for comprehensive testing across different traffic patterns. The following table details the simulated workload rate types, which define how user requests are submitted during the benchmarking process.

Type Description Primary Purpose / What It Measures
Sweep Runs synchronous (min), throughput (max) benchmarks, and then runs a series of benchmarks equally spaced between the two rates. The number of benchmarks is set by --rate (default is 10). • Full load–latency curve (end-to-end scalability profile)
• Determines optimal operating point before saturation
Synchronous Runs a single stream of requests one at a time • Baseline model response latency (no concurrency)
• Measures raw end-to-end response time per request
• Establishes performance floor before parallel load
Throughput Issues all requests concurrently to fully saturate the model/server. Equivalent to a "max load" test • Maximum throughput capacity of the serving system
• Identifies saturation point where latency begins to increase sharply
• Useful for sizing instances or cores
Constant Sends requests at a fixed, steady rate (RPS), regardless of completion time. • Stability of serving performance under steady, controlled load
• Queueing effects when request arrival exceeds processing rate
Poisson Sends requests according to a Poisson process (random intervals, same average rate). Simulates unpredictable real-world traffic • System responsiveness to bursty or stochastic traffic
• Latency jitter and recovery time after spikes
Concurrent Runs a fixed number of streams of requests in parallel • Scalability with concurrency
• Measures how latency and throughput scale as more users or parallel sessions connect

Test Suites

Testing is organized into independent test suites, each with specific goals and test scenarios. Test suites can be run in any order based on your evaluation needs. For detailed test case specifications, models under test, and execution instructions, see the README in each test suite directory.

3-Phase Testing Methodology

Note: 3-phase testing is currently only implemented for concurrent load tests. Other test suites (scalability, embedding models) use baseline testing approaches.

The Concurrent Load Test Suite implements a structured 3-phase testing approach:

  1. Phase 1: Baseline Tests - Fixed tokens, no caching
  2. Establishes reproducible performance baselines
  3. Enables cross-architecture comparison

  4. Provides pure performance characteristics

  5. Phase 2: Realistic Tests - Variable tokens, no caching

  6. Simulates real-world traffic variability
  7. Quantifies impact of token distribution variance

  8. Tests performance stability under realistic load

  9. Phase 3: Production Tests - Realistic datasets, with caching

  10. Simulates true production conditions
  11. Measures performance with optimizations enabled
  12. Validates production configurations

For detailed 3-phase methodology and how it applies to concurrent load testing, see Testing Phases.

Test Suite 1: Concurrent Load

Goal: Measure P95 latency and throughput scaling under concurrent load

  • Tests 7 LLM models + 2 embedding models
  • Concurrency levels: {1, 2, 4, 8, 16, 32} for LLMs
  • Concurrency levels: {4, 8, 16, 32, 64} for embeddings
  • Workloads: Chat, RAG, CodeGen, Summarization, Embedding

📚 See Concurrent Load Test Suite for complete test specifications

Test Suite 2: Scalability

Goal: Characterize maximum throughput and generate load-latency curves

  • Sweep tests for full performance curves
  • Synchronous baseline tests
  • Poisson tests for bursty traffic
  • Focus on max throughput (OTPS/TTPS), TTFT scaling, and KV cache efficiency

📚 See Scalability Test Suite for complete test specifications

Test Suite 3: Resource Contention (Planned)

Goal: Test platform stability with resource sharing and noisy neighbors

  • Fractional core allocation
  • NUMA node isolation
  • Co-located workloads
  • Multi-tenant scenarios

📚 See Resource Contention Test Suite for status and planned tests

Test Suite 4: Configuration Tuning (Future)

Goal: Measure configuration impact of vLLM CPU tuning and quantization

  • KV cache size variations
  • Quantization methods (AWQ, GPTQ)
  • Thread binding strategies
  • Batch size optimization

References