vLLM Performance Tuning: The Ultimate Guide to xPU Inference Configuration

Additional contributors include Hossein Sarshar, Ashish Narasimham, and Chenyang Li.  

Large Language Models (LLMs) are revolutionizing how we interact with technology, but serving these powerful models efficiently can be a challenge. vLLM has rapidly become the primary choice for serving open source large language models at scale, but using vLLM is not a silver bullet. Teams that are serving LLMs for downstream applications have stringent latency and throughput requirements that necessitate a thorough analysis of which accelerator to run on and what configuration offers the best possible performance.

This guide provides a bottoms-up approach to determining the best accelerator for your use case and optimizing your vLLM’s configuration to achieve the best and most cost effective results possible.

Note: This guide assumes that you are familiar with xPUs, vLLM, and the underlying features that make it such an effective serving framework.

Prerequisites

Before we begin, ensure you have:

Gathering Information on Your Use Case

Choosing the right accelerator can feel like an intimidating process because each inference use case is unique. There is no a priori ideal set up from a cost/performance perspective; we can’t say model X should always be run on accelerator Y. 

The following considerations need to be taken into account to best determine how to proceed:

What model are you using?

Our example model is google/gemma-3-27b-it. This is a 27-billion parameter instruction-tuned model from Google’s Gemma 3 family.

What is the precision of the model you’re using?

We will use bfloat16 (BF16).

Note: Model precision determines the number of bytes used to store each model weight. Common options are float32 (4 bytes), float16 (2 bytes), and bfloat16 (2 bytes). Many models are now also available in quantized formats like 8-bit, 4-bit (e.g., GPTQ, AWQ), or even lower. Lower precision reduces memory requirements and can increase speed, but may come with a slight trade-off in accuracy.

Workload characteristics: How many requests/second are you expecting?

We are targeting support for 100 requests/second.

What is the average sequence length per request?

• Input Length: 1500 tokens
• Output Length: 200 tokens
• The total sequence length per request is therefore 1500 + 200 = 1700 tokens on average.

What is the maximum total sequence length we will need to be able to handle?

Let’s say in this case it is 2000 total tokens

What is the GPU Utilization you’ll be using?

The gpu_memory_utilization parameter in vLLM controls how much of the xPU’s VRAM is pre-allocated for the KV cache (given the allocated memory for the model weights). By default, this is 90% in vLLM, but we generally want to set this as high as possible to optimize performance without causing OOM issues – which is how our auto_tune.sh script works (as described in the “Benchmarking, Tuning and Finalizing Your vLLM Configuration” section of this post).

What is your prefix cache rate?

This will be determined from application logs, but we’ll estimate 50% for our calculations.

Note: Prefix caching is a powerful vLLM optimization that reuses the computed KV cache for shared prefixes across different requests. For example, if many requests share the same lengthy system prompt, the KV cache for that prompt is calculated once and shared, saving significant computation and memory. The hit rate is highly application-specific. You can estimate it by analyzing your request logs for common instruction patterns or system prompts.

What is your latency requirement?

The end-to-end latency from request to final token should not exceed 10 seconds (P99 E2E). This is our primary performance constraint.

Selecting Accelerators (xPU)

We live in a world of resource scarcity! What does this mean for your use case? It means that of course you could probably get the best possible latency and throughput by using the most up to date hardware – but as an engineer it makes no sense to do this when you can achieve your requirements at a better price/performance point.

Identifying Candidate Accelerators

We can refer to our Accelerator-Optimized Machine Family of Google Cloud Instances to determine which accelerator-optimized instances are viable candidates.

We can refer to our Cloud TPU offerings to determine which TPUs are viable candidates.

The following are examples of accelerators that can be used for our workloads, as we will see in the “Calculate Memory Requirements” section.

The following options have different Tensor Parallelism (TP) configurations required depending on the total VRAM. Please see the next section for an explanation of Tensor Parallelism.

Accelerator-optimized Options

• g2-standard-48

• a2-ultragpu-1g

• a3-highgpu-1g

TPU Options

Calculate Memory Requirements

We must estimate the total minimum VRAM needed. This will tell us if the model can fit on a single accelerator or if we need to use parallelism. Memory utilization can be broken down into two main components: static memory from our model weights, activations, and overhead plus the KV Cache memory.

The following tool was created to answer this question: Colab: HBM Calculator. 

You can enter the information we determined above to estimate the minimum required VRAM to run our model.

• Hugging Face API Key

• Model Name from Hugging Face

• Number of Active Parameters (billions)

• The average input and output length (in tokens) for your workload.

• A batch size of 1