SGLang has now added DFlash2 recipes to the Qwen3.8-27B cookbook. Link: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B

DFlash2 is a speculative decoding technique released by Inco AI, originally developed at MIT Z Lab. When it was first announced, Qwen3.8-27B achieved 70 tok/s on an M5 Max MacBook Pro, 4.6x faster than standard autoregressive decoding while maintaining identical output, accepting one extra token per pass. Zhijian Liu shared a demo of it running on OpenCode:

A community member summarized the key technical points of DFlash2, calling it "next-generation DFlash", which originated from Z Lab and was upgraded by Inco AI:


The recommended combination in these recipes is NVFP4 + DFlash2. NVFP4-quantized weights take up around 16.5GB, which runs comfortably on 32GB cards like the RTX 5090. It has received good feedback from the community, but there is one catch when getting started: DFlash2 needs to be installed from source, as its selector support is newer than the latest official release. Specifically, you need to merge PR #35371 (for DFlash2 itself) and PR #35496 (for the quantized lm_head path of NVFP4), otherwise NVFP4 will throw an error on startup.


The documentation provides a command generator that can produce a startup command once you select your GPU and precision. The only value that needs manual calculation is `--mamba-full-memory-ratio`. This parameter controls memory allocation between the GDN state pool and the attention KV pool. The default value of 0.9 over-allocates memory to the KV pool, which silently limits the maximum concurrent requests. The documentation includes a calculator that gives a recommended value when you input your average request length.

Key configuration points:

- `--chunked-prefill-size 2048`: On hybrid GDN models, an 8192-token prefill chunk will cause around 600ms of decode stutter. Changing this to 2048 keeps decoding smooth.

- `--mamba-ssm-dtype bfloat16`: This reduces the state buffer size from 153.9MB with fp32 to 78.4MB, but does not always improve speed. On RTX 5090, NVFP4 + EAGLE is faster with fp32, while FP8 + EAGLE is faster with bf16. It is recommended to test both.

- Hardware compatibility: H200 does not have FP4 tensor cores, so NVFP4 is not supported. On RTX 5090, DFlash2 is recommended to run with bf16 + mem-fraction 0.90. Measured TPOT is 6.1ms, compared to 8.3ms with fp32.

A community member tested official FP8 + DFlash2 on an RTX 6000, and got good decode performance but slower prefill:


Another community member also reminded that it is best to put the draft model on the same GPU as the main model, otherwise copy overhead will eat up all the speed gains. One user asked if it can run on RTX 3090, and the official team has not responded so far.

Overall, these recipes provide a great starting point. DFlash2 is not a silver bullet — prefill performance, memory usage, and draft model placement all need to be verified on your own setup. But at least now you don't have to tune parameters from scratch.

More updates for Qwen3.8-27B are on the way.