Optimizing LLM Serving on AMD GPUs with Advanced Methodology

With the rise of large language models (LLMs) in enterprise settings, the quest for optimized serving configurations has become increasingly critical. OpenAI's recent insights indicate that effective LLM serving involves balancing latency and responsiveness against infrastructure costs, rather than simply maximizing throughput. A structured approach to tuning prefill-decode disaggregated serving using the llm-d1 framework is essential to manage this complexity.

The methodology starts with benchmarking the main phases of LLM generation: decode, prefill, and aggregated serving. By isolating these stages, practitioners can gain valuable insights into each phase's performance capabilities. This detailed understanding enables teams to select configurations that align with specific model compute profiles, removing uncertainty about cluster shapes and resource ratios.

After the initial benchmarking, a Pareto sweep is performed across various candidate setups to evaluate trade-offs between latency, concurrency, and efficiency. This analysis helps identify optimal configurations tailored to different load levels and service-level objectives (SLOs). Instead of settling on a single 'best' setup, teams can create a decision framework that adapts to the concurrency requirements and SLOs of their applications.

Practical Implementation on AMD Infrastructure

To validate this methodology, a scale-out run was executed, illustrating the workflow's transition from microbenchmarking to practical distributed deployments. The testing involved a 72-GPU cluster spread across nine Kubernetes nodes on Oracle Cloud Infrastructure (OCI). Each node was equipped with eight MI300X GPUs and RoCEv2-enabled ConnectX-7 NICs, providing the necessary infrastructure for high-performance LLM serving.

Two model architectures were examined: an MoE model and a dense model. The selected models were OpenAI's gpt-oss-120b and RedHatAI's Llama-3.3-70B-Instruct-FP8-dynamic, both popular choices for enterprise applications. Their size and complexity make them ideal candidates for parallel configurations, which are essential for demonstrating the speed advantages of disaggregated deployments over aggregated ones.

In addition to the architectural considerations, the implementation benefited from two AMD-specific optimizations for the vLLM framework. These included the AITER unified intention implementation and the quick reduce quantization2 optimization, both contributing to improved performance.

Benchmarking and Evaluation

For evaluation, the team used the recommended gpt_oss evaluation scripts for the gpt-oss-120b model and the lm-evals for Llama-3.3-70B-Instruct-FP8-dynamic. To simplifies the process, justfile recipes were provided for easy access to the evaluation metrics. All decode benchmarks were conducted using vLLM’s DecodeBenchConnector, which optimizes the KV cache filling by bypassing the prefill stage and accurately measuring the attention cost during pure decode workloads.

This rigorous and repeatable workflow enables teams to effectively use llm-d, making sure that inference SLOs are met with the most efficient prefill-decode configurations. As AI workloads continue to evolve, the insights gained from this approach will be invaluable for organizations looking to optimize their LLM serving strategies on AMD's advanced GPU infrastructure.