Three Parameter Counts, and Only One Tells You Whether the Model Fits
The figure on the model card describes compute per token, not what your GPUs must hold — and cutting experts moves bytes across the wire before it moves any FLOPs.
Resident parameters are the count that decides whether you fit
MoE serving needs three parameter numbers, and they answer non-overlapping questions. Total is checkpoint size and sets the weight-storage floor: 30.5B parameters at two bytes each is 61 GB decimal, 56.8 GiB. Activated is one token's executed path, 3.3B, eight of 128 routed experts at each of 48 layers, and it is valid for per-token FLOPs and throughput modeling only. Resident is what the serving process must keep available. Resident memory is what you size the GPU against.
Model cards publish total and activated. Resident is a property of the deployment, so it has to be computed per configuration. Expert replication raises it: putting a hot logical expert on several GPUs buys tail latency with memory that would otherwise hold KV blocks. At 56.8 GiB of weights already resident, each replica subtracts directly from max concurrency and context budget. Expert offloading to CPU memory lowers it, and pays only when routing is predictable enough to prefetch. Expert parallelism shards it across ranks. The KV headroom left after all three is a calculation someone on the team has to actually do.
Halving top-k is a network change first
The dispatch payload per MoE layer is bounded by T × K × H × b: tokens, experts per token, hidden width, bytes per activation. The bound is linear in K, so cutting eight experts to four halves all-to-all bytes at every layer, not only routed-expert compute. The same formula separates prefill and decode into two network problems. At large T the step is bandwidth-bound. At decode-scale T it is launch-latency-bound. That is why DeepEP ships a throughput path and a separate low-latency path.
Layout couples this in a way that catches teams out. In vLLM with expert parallelism enabled, EP group size = TP × DP. The documented TP=2, DP=4 example produces four two-rank attention groups and an eight-rank EP group for the expert layers. Raising DP to chase throughput widens the all-to-all domain silently, and can push expert traffic off NVLink onto InfiniBand or Ethernet. The penalty repeats at all 48 layers. DeepSeek-V3's node-limited routing bounds exactly this, and it is part of the trained architecture, fixed before anyone writes a serving config.
Which optimizations need a task-level gate
| Bucket | Examples | Preserves expert selection? | Release gate |
|---|---|---|---|
| Execution-only | Kernel fusion, grouped GEMM, expert placement, comms overlap | Yes | Latency and throughput benchmark alone |
| Value-changing | Weight or activation quantization, top-k reduction | No | Multi-seed task-level eval on your workload |
Grouped GEMM sits in the first bucket. It runs several expert matrices in one launch and cuts launch overhead even when row counts differ. The honest caveat is that it cannot create work for an expert that received no token. Grouped GEMM reduces the launch overhead of imbalance without changing the row counts that cause it.
Top-k reduction sits in the second, and the evidence behind it is thin. One benchmark, two checkpoints, no confidence intervals, no seed counts, and the authors report that perplexity and task accuracy preferred different settings. A 10→5 cut on Qwen3.5-397B-A17B lost 0.55 points. A 0.35-point MMLU delta is plausibly inside single-run variance, which cuts both ways: the data cannot separate it from free, and it cannot establish that it is. Gate the change on the task metric you ship against.
What the utilization dashboard smooths away
In expert-parallel MoE, the slowest participating GPU sets step latency. The layer cannot complete until every required contribution returns. An expert holding half the assignments runs a larger GEMM while its peers idle, and average-utilization panels average that signal out by construction. The instrumentation that catches it:
- Per-expert assignment counts per step. Assignments, not unique tokens; top-2 routing means every token contributes two rows.
- Skew ratio at p95: max expert load divided by mean expert load. It predicts the tail better than anything else on the panel.
- Per-stage timers for routing, permutation, grouped GEMM, combine and unpermutation, plus bytes split by network domain.
Diagnose memory first. It is a static check that needs no load test, and the signature is legible: high memory use with low expert compute, confirmed by little room left for KV blocks. Expert compute and imbalance come second, at step level. Communication comes last, because interconnect asymmetry will mask every other effect if introduced early. And the sparsity applies to the routed experts only: attention processes every token, embeddings stay resident, and DeepSeekMoE-style shared experts run regardless of router score.
What to do
Recompute every MoE capacity model this week as total parameters × bytes stored (30.5e9 × 2 = 61 GB for Qwen3-30B-A3B), and assert resident weights plus remaining KV capacity at process startup.
Instrument per-expert assignment counts per step, a p95 skew ratio, and per-stage timers for routing, permutation, grouped GEMM, combine and unpermutation before your next serving-layout change.
Reproduce the normalization-mass-corrected 8→4 reduction as a gated offline experiment across at least three seeds on your production eval suite, scoring task accuracy rather than perplexity.