NVIDIA Shows 3 Ways to Speed Up MoE Inference on DGX Spark, Lifting Ling-3.0-flash Throughput 42.5%

NVIDIA’s latest technical blog shows how to speed up mixture-of-experts (MoE) inference on one DGX Spark. The test model is Ant Group inclusionAI’s Ling-3.0-flash, a 124B-parameter MoE with 5.1B active parameters. NVIDIA stacked three optimizations: a new MoE kernel backend, an FP8 output layer and built-in speculative decoding. On all 1,319 GSM8K questions, decode throughput rose 42.5% with no statistically detectable accuracy change.

So why is it relevant for Devs? DGX Spark is a desktop system, not a data center node. NVIDIA’s results show a 124B MoE serving single-user requests faster on that desk-side box. The technical post also shows where MoE decode time goes, and how to cut it.

TL;DR

- Size: Ling-3.0-flash-fp4: 124B total, 5.1B active per token; 256K native context.

- Runs on: One NVIDIA DGX Spark (GB10, 128 GB unified memory, 273 GB/s), batch size one.

- Performance: Three stacked optimizations lift GSM8K decode throughput from 37.81 to 53.89 tokens/s.

- Best: +42.5% throughput and -24.5% mean latency, with accuracy flat (96.89% vs 96.82%).

- Lagging: Synthetic gains (+185.1%) shrink on GSM8K as MTP acceptance drops from 3.90 to 2.68.

Why is MoE decoding on DGX Spark memory-bound?

DGX Spark is NVIDIA’s desktop system built on the GB10 Grace Blackwell Superchip. It offers 128 GB of unified memory and 273 GB/s of memory bandwidth. Peak compute is up to 1 petaFLOP of sparse FP4. NVIDIA rates it for inference on models up to 200B parameters.

In the NVIDIA technical post, the NVIDIA team profiled Ling-3.0-flash-fp4 at batch size 1. Decoding was memory-bound: reading model weights drove execution time. Two operations stood out, the MoE computation and the LM head that scores the vocabulary. That points to two levers: move fewer bytes per decode step, and generate more tokens per verification round.

The baseline followed the official W4A8 recipe. It used FlashInfer MXFP4 MoE kernels, CUTLASS dense GEMM and an FP32 LM-head path. On a synthetic 32K-input, 2K-output test, it ran at 27.71 ms per token (36.09 tokens per second).

What are the 3 optimizations?

1. Humming MXFP4 MoE kernels: NVIDIA swapped FlashInfer’s MoE backend for Humming, an independent kernel library. Humming uses NVRTC to compile kernels at runtime for the actual matrix shapes, data types and GPU. That includes the narrow 768-wide expert layer. Each kernel compiles in seconds. The FlashInfer CUTLASS path compiles 97 targets with ninja, on the order of tens of minutes. Throughput rose 9.3% to 39.45 tokens per second.

2. Online FP8 LM head: The LM head is a 157,184 x 2,560 matrix stored in BF16. Every generated token reads all of it. Quantizing only this layer to FP8 at load time halves those reads. Setting SGLANG_ENABLE_FP8_LM_HEAD=1 enables it. It cannot be combined with the baseline –enable-fp32-lm-head option. Combined with Humming, throughput reached 43.94 tokens per second. That is 21.7% over baseline and 11.4% over Humming alone.

3. MTP speculative decoding with NEXTN: Ling-3.0-flash ships with a built-in NEXTN prediction layer, so no extra draft model is needed. It drafts three tokens, and the model verifies four positions at once. Verification costs about 2.26x a normal decode step. The main cost is MoE, not attention. Verifying four tokens can activate up to 32 experts instead of 8. On the synthetic test, an average of 3.90 tokens were accepted per round. Throughput jumped to 102.88 tokens per second, up 185.1%.

The three techniques do not overlap. Humming and the FP8 LM head cut the cost of each step. MTP cuts the number of steps.

The integration trap: Stacking these is not automatic. With FP8 LM head and MTP both on, the draft must share the full LM-head module. Otherwise the framework silently falls back, and the draft interprets FP8 weights as BF16. Accept length drops to 1.00, and throughput falls below the unoptimized baseline. The NVIDIA team advises checking at startup that each optimization is active. NVIDIA points to the Ling cookbook notebook for enabling both together.

What happens on a real workload?

Synthetic tests flatter speculative decoding, so the NVIDIA team also ran all 1,319 GSM8K questions. It used 5-shot prompting with chain-of-thought, temperature 0.6, top_p 0.95 and top_k 29. With all three optimizations, time per output token fell from 26.45 ms to 18.56 ms. Decode throughput rose 42.5%, from 37.81 to 53.89 tokens per second. Mean latency dropped 24.5%, from 15.21 to 11.49 seconds. Accuracy moved from 96.89% to 96.82%, a change NVIDIA calls statistically undetectable.

Accept length here was 2.68, versus 3.90 on synthetic data. The NVIDIA team gives three reasons for the gap. Synthetic outputs are more repetitive, so drafts are accepted more often. The synthetic test used greedy decoding, while GSM8K used sampling. GSM8K answers are also short, with a median near 310 tokens versus a fixed 2K.

What is Ling 3.0, and why does it suit DGX Spark?

Ling-3.0-flash belongs to Ant Group inclusionAI’s Ling 3.0 family, released under the MIT license. The open-weight lineup also includes Ling-3.0-tiny, a 7.9B MoE with 1.3B active parameters. Both come with base checkpoints and FP8, INT4 and GGUF builds; flash also has FP4. OpenRouter also lists inclusionAI’s vision-language, finance and health variants built on Ling 3.0 Flash.

Ling-3.0-flash is built to keep per-token cost low:

- Sparse MoE: Each MoE layer routes a token to 8 of 512 routed experts, along with 1 shared expert. About 4.1% of the weights are active per token.

- Hybrid linear attention: 35 Kimi Delta Attention (KDA) layers and 7 Gated MLA layers alternate 5:1. KDA, from Moonshot AI’s Kimi Linear paper, keeps a fixed-size state instead of a growing KV cache. The layout is native from the start of pretraining.

- Built-in MTP: The model was trained with multi-token prediction. That NEXTN layer is the draft model NVIDIA used.

- Agent focus: inclusionAI team reports 10,000+ interactive training environments for coding, general and deep-research agents. NVIDIA notes the native 256K context expands to 1M tokens.

How strong and efficient is Ling-3.0-flash?

On its model card, inclusionAI reports 56.6 on SWE-Bench Pro and 72.4 on SWE-Bench Multilingual. It also reports 93.2 on MathArena AIME 2026 and 87.0 on HMMT Feb 2026. Artificial Analysis results shown on OpenRouter list 85.5% on GPQA Diamond and 55.4% on Terminal-Bench 2.1. The hardest reasoning tests remain tough: Humanity’s Last Exam scores 22.7 self-reported and 23.7% independently.

API pricing: Artificial Analysis lists InclusionAI’s API at $0.075 per 1M input tokens and $0.22 per 1M output tokens. That blends to about $0.05 per 1M tokens, using its 7:2:1 cache-hit, input and output mix. It measured output speed at 231.7 tokens per second, third of 26 models in its class. On OpenRouter, Novita and DeepInfra list $0.06 input and $0.18 output.

Against peers: Artificial Analysis lists higher blended prices for comparable models. Step 3.7 Flash is $0.18, Gemini 3.5 Flash-Lite $0.33 and GPT-5 (medium) $1.34. These figures compare price only; they are not quality-adjusted.

Quantization costs little quality: inclusionAI’s quantization table compares BF16, FP8, INT4 and FP4. FP4 scores 83.68 on GPQA-diamond versus 84.97 in BF16. IFBench moves from 74.49 to 73.87, and ArcPrize from 68.75 to 68.06. SciCode shows the largest drop, from 41.24 to 38.43. The FP4 checkpoint is the one NVIDIA ran on DGX Spark.

Key Takeaways

- NVIDIA lifted Ling-3.0-flash GSM8K decode throughput 42.5% on 1 DGX Spark.

- Humming kernels and an FP8 LM head cut the cost of each decode step.

- Built-in MTP speculative decoding cuts the number of decode steps.

- Stacking FP8 LM head and MTP requires a shared LM-head module.

- Ling-3.0-flash activates 5.1B of 124B parameters per token, under MIT.

NVIDIA’s full analysis is on the NVIDIA Technical Blog. Model weights and recipes are on Hugging Face, the SGLang cookbook and the Ling cookbook.

Asif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.