There Are Three Ways to Enhance Mixture-of-Experts Inference on NVIDIA DGX Spark

Developers running Ling-3.0-flash on a single NVIDIA DGX Spark can now decode tokens substantially faster by combining three targeted optimizations, provided they wire them together correctly. Across all 1,319 GSM8K test questions at batch size 1, the stack raised decode throughput from 37.81 to 53.89 tokens per second (+42.5%), cut mean end-to-end latency from 15.21 s to 11.49 s (−24.5%), and left accuracy statistically unchanged (96.89% to 96.82%). The catch: one integration detail, if missed, silently produces a configuration slower than the untouched baseline.

Why decoding was slow in the first place

Profiling Ling-3.0-flash from Inclusion AI at batch size 1 showed decoding was memory-bound: reading model weights, not attention math, drove execution time. Two operations dominated—the mixture-of-experts (MoE) computation and the output-layer projection (LM head) that scores every token in the vocabulary.

The evaluated model, Ling-3.0-flash-fp4, is a MoE model with 124 billion total parameters and 5.1 billion active per token. It has 42 layers: 35 Kimi Delta Attention (KDA) linear-attention layers and seven multi-head latent attention (MLA) layers. Each MoE layer routes a token to eight of 512 routed experts plus one shared expert. The model offers a native 256K-token context window, expandable to 1M tokens, and improves on the previous Ling flash model in long-horizon task stability, tool-calling accuracy, and harness compatibility. Reasoning can be toggled on for complex agent tasks or off for latency-sensitive interactions.

The baseline followed the official cookbook's W4A8 configuration: FlashInfer MXFP4 MoE kernels, NVIDIA CUTLASS dense GEMM, and the `--enable-fp32-lm-head` option. The LM head projects a hidden dimension of 2,560 to a vocabulary of 157,184 tokens.

Those findings suggested two levers: reduce weight traffic per decode step, and generate more tokens per verification round. The three techniques below map to those levers—two attack step cost, one attacks step count.

Optimization 1: swapping in the Humming MXFP4 MoE backend

The first change replaces the `flashinfer_mxfp4` MoE backend with `humming`, an independent kernel library shipped as the `humming-kernels` pip package. Unlike the FlashInfer CUTLASS path, Humming uses NVIDIA CUDA Runtime Compilation (NVRTC) to generate and compile CUBINs at runtime for the actual M/N/K dimensions, data types, and GPU architecture, caching binaries in `~/.humming/cache`.

That specialization matters for Ling-3.0-flash's relatively narrow `moe_intermediate_size` of 768. Compilation granularity also differs sharply: a single Humming kernel finishes in seconds, while the FlashInfer CUTLASS path uses ninja to compile 97 targets and takes on the order of tens of minutes.

On a synthetic workload with 32K input tokens, 2K output tokens, and batch size 1, the swap alone moved time per output token (TPOT) from 27.71 ms to 25.35 ms and decode throughput from 36.09 to 39.45 tokens per second—a 9.3% gain.

Optimization 2: online FP8 quantization of the LM head

In the checkpoint, the LM head is stored in BF16 and listed in `modules_to_not_convert`, so it loads unquantized. Its vocabulary GEMM is 157,184 × 2,560, and the entire weight block must be read for every generated token—a high-value compression target under a bandwidth bottleneck.

The fix applies a dynamic FP8 quantization configuration only when constructing the LM head; the rest of the model keeps the checkpoint's FP8/MXFP4 settings. Moving those weights from BF16 to FP8 halves the layer's read volume. It is enabled with `SGLANG_ENABLE_FP8_LM_HEAD=1` and is mutually exclusive with the baseline `--enable-fp32-lm-head` option, which outputs logits in FP32 through a slower vocabulary-GEMM path.

Stacked on Humming, TPOT fell to 22.76 ms and throughput rose to 43.94 tokens per second—21.7% over baseline, and 11.4% over Humming alone.

Optimization 3: MTP speculative decoding with NEXTN

The first two optimizations lower the cost of each decode step. Speculative decoding takes the other route: fewer steps. Ling-3.0-flash ships a NEXTN prediction layer (`num_nextn_predict_layers: 1`) usable directly as a draft model with no extra training or downloads. The configuration uses three draft steps and a four-token verification window: each round, the draft predicts three tokens in sequence and the target verifies four positions at once, accepting the correct prefix.

The payoff depends on accept length (average accepted tokens per round) and the verification-cost ratio (cost of one verification step versus a normal decode step), measured at roughly 2.26. Notably, the verification cost came not from attention—at batch size 1, KV reads were small relative to weight reads—but from MoE expert activation: a normal decode activates eight experts for one token, while verifying four tokens can activate up to 32.

With all three enabled, the synthetic workload reached 9.72 ms TPOT and 102.88 tokens per second, +185.1% over baseline, with accept length at 3.90 tokens per round.

The integration trap: share the whole LM head, not just its weights

Enabling online FP8 LM-head quantization and MTP together requires the draft model to share the entire LM-head module with the target, not merely its weight tensor. The original Ling NEXTN implementation did not expose this interface. Without it, the framework falls back silently: the draft receives FP8 weights but interprets them as BF16, accept length collapses to 1.00, and throughput ends up lower than a configuration with no optimizations at all. Sharing the full module through that interface restores normal acceptance. The Ling cookbook documents the combined setup.

What the GSM8K run actually showed

Synthetic workloads isolate individual gains but overestimate speculative decoding, because repetitive outputs are easier to draft. So the team also ran a strict end-to-end comparison over all 1,319 GSM8K questions: five-shot prompting with training-set demonstrations, temperature 0.6, top_p 0.95, top_k 29, max_tokens 32,768, chain-of-thought enabled. Sampling parameters, parser, five-shot protocol, random seed, and memory allocation were aligned between arms.

Mean output lengths were 567 and 607 tokens for baseline and optimized runs; median output was about 310 tokens, with P95 output lengths around 1,900–2,060 tokens.

Three differences explain the gap between the 185.1% synthetic figure and the 42.5% real-world one. Synthetic outputs are more repetitive, so accept length hit 3.90 versus 2.68 on GSM8K. Sampling differed: greedy decoding synthetically versus temperature 0.6, top_p 0.95, top_k 29 in the real run. And output length was fixed at 2K tokens synthetically, while the GSM8K median is roughly 310—spreading the fixed per-step overhead reductions from the first two optimizations over a much shorter output.

What this means for your deployment

The mechanisms don't overlap: Humming and the FP8 LM head cut per-step cost, while MTP cuts the number of steps, which is why they stack. But stacking is not automatic. Each optimization worked independently, yet enabling all three demanded the full LM-head sharing interface; otherwise the system degrades silently to something slower than the unoptimized baseline.

The practical takeaway: validate at startup that every optimization is actually active rather than inferring configuration state from later performance numbers. And benchmark on a workload that resembles your intended deployment—the 185.1% synthetic gain does not establish the gain for other workloads. Code and deployment instructions are in the Ling cookbook, with standard MXFP4 deployment steps in the MXFP4 deployment notebook.