Qwen3.8-Flash-Next on a single DGX Spark

Official NVIDIA NVFP4 · staged PLE · MTP3 · FP8 KV · decode CUDA graphs

Field notes from serving NVIDIA's Qwen3.8-Flash-Next NVFP4 checkpoint on one NVIDIA DGX Spark (GB10, 128 GB unified memory) with the single-Spark TP1 recipe.

Card: the earlier September 6 benchmark. Current measurements follow.

Default NVIDIA TP1 recipe (2026-09-10)

scripts/serve.sh runs the official NVIDIA checkpoint with pinned vLLM nightly 8a728663, staged disk PLE gather, MiaAI's 47,149-token MTP3 draft vocabulary, BF16 recurrent state, FP8 KV, six sequence slots, 4,096-token prefill chunks, 262,144-token context, and decode CUDA graphs. Prefix caching and DFlash are disabled. The default 30 GiB host reserve derives GMU=0.7535 on this host.

Measured September 10 at 11:21–11:23 UTC, thinking off and temperature 0. Single-request decode excludes first-token latency: medians of three 384-token samples per workload after warmup. The four-request result is one batch of 256-token outputs and includes prefill. Long-prompt figures are medians of two fresh documents per size and include request overhead.

All 16 benchmark requests succeeded, with no other completed generation requests observed. Minimum available host memory was 19.94 GiB, MTP draft acceptance was 65.7%, and the container remained healthy with zero restarts. Its 7.44 GiB KV budget holds 507,810 tokens: six scheduler slots do not imply six full 262K requests fit. This is a small snapshot, not a matched speedup comparison or a quality benchmark.

Raw samples and measurement method · Optimization validation

Earlier NVIDIA TP1 benchmark (2026-09-06)

This different 40-prompt workload used the 65,536-token draft selection, FP32 recurrent state, and GMU=0.80. Its aggregate median is not directly comparable with the two-workload snapshot above.

The server uses Docker's unless-stopped policy. The full raw prompt-by-prompt result is available in the NVIDIA TP1 validation report.

Legacy RadixArk profiles

How it fits in 128 GB

The official checkpoint holds a 51B n-gram embedding table that each token only reads 16 rows of. The patched vLLM image serves it from NVMe via staged disk reads. The FP8 PLE table accounts for 47.68 GiB of checkpoint data. Model loading uses 76.48 GiB; caches, activations, runtime overhead, and the host share the remaining memory. PLE is part of the target model and remains necessary with any speculative method. NVFP4 PLE storage is not supported by the current loader.

Serving config

scripts/serve.sh          # NVIDIA TP1 · native 262,144 · 6 seqs
# MTP=3 · 47,149-token draft vocabulary · BF16 recurrent state
# FP8 KV · 4,096-token chunks · staged PLE · HOST_RESERVE_GIB=30
# decode CUDA graph widths: [4,8,12,16,20,24]
scripts/serve-500k.sh     # legacy YaRN 524,288 · hybrid · 1 seq

Serving it to other machines

scripts/serve.sh publishes an unauthenticated API on all interfaces. The older scripts/serve-public.sh keeps the legacy RadixArk profile on loopback and fronts it with a small gateway: bearer auth on /v1/* and /metrics, 404 on everything else, streaming passed through untouched. Keys live in the gateway, so rotating one takes effect on the next request instead of restarting a container that loads ~76 GiB of weights.

scripts/serve.sh          # default NVIDIA TP1 service on :18300
scripts/serve-public.sh   # legacy container on 127.0.0.1:18300, gateway on :8080

The admin dashboard it serves at / shows upstream health and the served model, edits the advertised API base URL, and creates, reveals, rotates and revokes keys with per-key request counts.

Full write-up — setup, benchmarks, and memory-reduction options — is in the repository README.