The LMSYS team recently released inference optimization results for the GLM-5.2 NVFP4 model. Built on the SGLang inference framework, the team pushed the performance of multi-turn agent coding scenarios to a new level in just two weeks of iteration.

Under the test condition of 8 B300 GPUs and a batch size of 1, SGLang ultimately achieved a throughput of over 500 TPS. The test was based on the OpenHands multi-turn agent coding scenario: each conversation has an average context of around 80,000 tokens, outputs 220 tokens per turn, has 13 turns in total, and the overall prefix cache hit rate is around 92%.

Results show that compared to the day-0 version that just added initial support for GLM-5.2, this optimization improved single-user interaction speed by 18% to 34%; peak throughput in high-concurrency scenarios also saw no compromise, increasing by 6% to 11%. The performance comparison is shown below:


This optimization does not rely on any mysterious black magic – all performance gains come from layer-by-layer granular tuning. The main optimizations fall into four parts:

### 1. Eliminating Bubbles in Spec V2 Scheduling

Spec V2 is the overlapping speculative decoding scheduling scheme previously launched by SGLang. When the GPU is running the forward pass of the current model, the CPU prepares KV allocation and metadata for the next step in advance on a separate stream, hiding CPU overhead within the GPU forward pass process. This optimization added further adaptations:

- Enable CUDA Graph support for DSA's draft-extend path

- Remove unnecessary device-side synchronization operations

- Fuse small metadata operations

The comparison of decoding traces before and after optimization shows a very intuitive change: there are almost no blank bubbles between two consecutive iterations after optimization:



This step directly delivered an 11% end-to-end TPS improvement.

### 2. Adding Support for IndexShare MTP Speculative Decoding

GLM-5.2 natively includes a well-performing MTP speculative decoding head, which achieves an average acceptance length of over 5, bringing significant improvements to low-latency scenarios. SGLang added two key adaptations:

- Reuse the top-k results calculated by the DSA indexer in the first step, eliminating the need for recalculation in subsequent draft steps. This reduces the cost of draft steps by up to 1.9x in long-context scenarios

- Adapt to the asynchronous scheduling of Spec V2, passing top-k seeds through the scheduling buffer to avoid losing data between iterations

### 3. Accelerating TopK by 2.3x to 10x

DSA sparse attention requires scoring historical KV positions and selecting TopK. The original TopK implementation used a sorting approach, which was upgraded to a selection-based approach called TopK-V2 in this optimization.

The core design is clustered radix selection: 8 CTAs each build a local 10-bin histogram, perform a global reduction to find the bin containing the k-th largest score, directly output candidates above that bin, and perform precise FP32 radix selection only for boundary candidates. The entire selection process and page table conversion are also fused into a single kernel to reduce extra overhead.


Practical tests show that at an input length of 80,000 tokens, the latency of the TopK kernel dropped from 40.7 microseconds to 17.5 microseconds, a speedup of 2.33x; at an input length of 1 million tokens, latency dropped from 372.1 microseconds to 36.6 microseconds, a speedup of 10.17x.


This optimization directly solves the problem of plummeting interaction speed in long-context scenarios: in the original day-0 version, indexing cost grows as input length increases, causing single-user interaction speed to drop rapidly; TopK-V2 removes this bottleneck, allowing interaction speed to remain basically stable even at 1 million input tokens.


### 4. Fusing Indexer Prologue and Optimizing BF16 GEMM

Originally, the preparation work for the DSA indexer prologue was split into 12 small kernels, which lengthened the critical path. This optimization fused the dependency chain twice, ultimately reducing the number of kernels from 12 to 4 and shortening the critical path. At batch size 1, decoding throughput increased directly by 8%, and it still delivers a stable 5% improvement at batch size 128.


Last is the optimization for BF16 GEMM: To maintain accuracy, GLM-5.2 retains BF16 precision for attention projection and shared expert MLP layers. SGLang added a new CuTe DSL-based BF16 GEMM backend. Through task-based pipelined loading, it overlaps the execution of loading, computation, and writeback, and preloads data into GPU shared memory as much as possible, making it faster than the generic cuBLAS implementation.

In tests on two common BF16 layers in practice, fused QKV projection is on average 8% faster than cuBLAS, with a peak speedup of 13%; output projection is on average 5% faster, with a peak speedup of 8%, and end-to-end decoding gets an additional 4% improvement at batch size 1.


### Final Overall Performance

The final Pareto curve comparison leads to three conclusions:

1. The GLM-5.2 architecture is inherently more efficient than GLM-5.1. On the same SGLang version, it improves both single-user interaction speed and GPU throughput by 30% to 40%

2. This two-week optimization pushed single-user interaction speed up by 18% to 34%, achieving over 500 TPS for single-user scenarios on 8 B300 GPUs

3. High-concurrency throughput also saw improvements, with peak throughput increasing 6% to 11% – no performance trade-off was made


All optimizations in this work target the low-concurrency, high-cache-hit-rate scenarios common in agent use cases. The LMSYS team noted that they will continue optimizing for higher-concurrency scenarios next, including better prefill TopK kernels, PD disaggregation and expert parallelism optimizations, and adaptation of existing cache optimization techniques.

After seeing the 8 B300 GPUs for single-user workload configuration, some netizens joked about being unable to afford such hardware, while others pointed out the title clearly stated the configuration upfront. Of course, in real production, this performance refers to total throughput under concurrent scenarios, not that the setup only serves one single user.

All optimization details, reproduction code, and launch commands are already public in the official blog. Interested readers can check the original post here:

https://www.lmsys.org/blog/2026-07-13-glm52-optimization

SGLang itself is an open-source high-performance large language model inference framework. It has been deployed on over 400,000 GPUs worldwide and produces trillions of tokens per day. The project repository is here:

https://github.com/sgl-project/sglang