LibertAIDAI/GLM-5.3-Flash-NVFP4

GLM-5.3-Flash · NVFP4

320B total · 18B active · natively multimodal · 1M context

598.5 GiB → 181 GiB  ·  −70%  ·  round-trip cosine 0.99665

Weight-only NVFP4 quantization of Z.ai's GLM-5.3-Flash — the first GLM to combine sparse and linear attention.

Quantized by LibertAI · not affiliated with Z.ai / Zhipu

---

✨ What this is

A weight-only NVFP4 (NVFP4-A16) checkpoint. The routed-expert FFN tensors — 97% of the model's parameters — are quantized to NVFP4 (E2M1, with FP8-E4M3 per-16-block scales and an FP32 per-tensor global scale). Everything outlier-sensitive stays in BF16:

both attention flavours (all 34 KDA linear-attention layers and the 11 DeepSeek-sparse layers, including the sparse indexer) · the entire vision tower · shared experts · MoE routers · dense-MLP layers · the MTP head · the Manifold-Constrained Hyper-Connection (mHC) tensors · token embeddings · lm_head · all norms

Activations stay BF16 — there is no activation quantization.

Because the experts dominate the footprint, quantizing only them buys the full 70% while protecting quality where it matters. Leaving the vision tower untouched keeps multimodal behaviour bit-identical to the source.

Produced with NVIDIA ModelOpt 0.45.0 through a memory-frugal shard-streaming pass — CPU-only, never more than one shard resident, no calibration data (weight-only NVFP4 derives its scales from the weights themselves).

---

🚦 Engine support — read this first

glm5_next is a brand-new architecture (released 2026-08-26) and is not yet in vLLM main (vllm#53906, sglang#36507). Support ships in per-model images.

Hardware	Engine	Status
H100 / B200 / GB200	vLLM	✅ vendor-verified
GB10 / DGX Spark (sm_121)	SGLang	✅ verified by us — recipe below
GB10 / DGX Spark (sm_121)	vLLM	✅ verified by us — needs the recipe

vllm/vllm-openai:glm53-flash-x86_64-cu130     # x86_64, CUDA 13
vllm/vllm-openai:glm53-flash-arm64-cu130      # arm64 (GH200/GB200/GB10)
lmsysorg/sglang:glm-5.3-flash-arm64           # SGLang, arm64

✅ vLLM on sm_121 (GB10) now works, via Libertai/glm53-flash-vllm-gb10. Two independent faults had to be fixed and either one alone leaves the model degenerate, which is why it long looked like a single unexplainable bug. First, no vLLM MLA backend accepted this model's NoPE dimensions on sm_121, since the sparse decode path asserts pe_dim == 64 and GLM-5.3-Flash has qk_rope_head_dim = 0; that is fixed by a hand-written sparse-MLA CUDA kernel. Second, and not specific to GB10, see the vLLM note below. The checkpoint was never at fault.

llama.cpp has no glm5_next support, so there is no GGUF.

---

🚀 Usage

⚠️ Point your engine at a local directory, not the repo id. vLLM's glm5next.py opens os.path.join(model_path, "processor_config.json") directly instead of resolving it through the hub cache, so a repo-id launch fails with a missing-file error even though the file is right there (discussion #2). Run hf download LibertAIDAI/GLM-5.3-Flash-NVFP4 --local-dir ./glm53 first and pass ./glm53.

SGLang — verified on 2× GB10 (sm_121), TP=2

python3 -m sglang.launch_server \
  --model-path LibertAIDAI/GLM-5.3-Flash-NVFP4 \
  --trust-remote-code --tp-size 2 \
  --attention-backend dsa \
  --dsa-prefill-backend tilelang --dsa-decode-backend tilelang \
  --moe-runner-backend flashinfer_cutlass \
  --kv-cache-dtype bfloat16 \
  --disable-shared-experts-fusion \
  --reasoning-parser glm45 --tool-call-parser glm47 \
  --mem-fraction-static 0.84 \
  --context-length 65536 --max-running-requests 2

Every flag above is load-bearing on sm_121:

Flag	Why it is required
--disable-shared-experts-fusion	The shared expert is BF16 in this checkpoint (only routed experts are quantized). Shared-expert fusion would pack it into the NVFP4 buffer and the load fails on a shape mismatch.
--dsa-*-backend tilelang	The only DSA backend with a NoPE (tail_dim == 0) kernel. flashinfer_sparse_mla hardcodes a rope-bearing 448+64 page layout; all four flashmla_* are excluded by index_kpool=4.
--kv-cache-dtype bfloat16	TileLang on CUDA is bf16-KV-only. Merely omitting an fp8 flag is not enough — the DSA default re-selects fp8.
--moe-runner-backend flashinfer_cutlass	The NVFP4 MoE runner auto-selects a datacenter-Blackwell backend on (12, 1). marlin also works but its repack costs ~19 GiB/rank.
--reasoning-parser glm45 (SGLang only — vLLM needs deepseek_r1)	Without it reasoning_parser is None and the entire thinking trace is emitted inside content, ending in a bare </think> with no opening tag, while reasoning_content stays null. See below.
--tool-call-parser glm47	Enables tool calling. ⚠️ Not glm — see the warning below. Note the two parser flags legitimately take different spellings: glm45 for reasoning, glm47 for tools.

🧠 Thinking / reasoning output

This model starts inside its reasoning block without emitting an opening <think> — the chat template opens it — and closes with </think>. With no reasoning parser configured you therefore get:

// ✗ no --reasoning-parser
"content": "The user is asking ... so the answer is 391.</think>391",
"reasoning_content": null

With --reasoning-parser glm45 it splits correctly:

// ✓
"content": "391",
"reasoning_content": "17 × 23 = 17 × 20 + 17 × 3 = 340 + 51 = 391",
"usage": { "reasoning_tokens": 43 }

deepseek-r1 also splits this shape (it is force_reasoning=True on </think>). On SGLang, prefer glm45: it additionally excludes <tool_call>, </tool_call>, <eop> and <|user|> from the reasoning span, which matters once tool calling is enabled. On vLLM the choice inverts — you must use deepseek_r1, because vLLM's glm45 is an alias for the GLM-4.7-MoE parser engine and discards the reply entirely against this model's prompt-side <think>. See the vLLM section below.

🛠️ Tool calling — use glm47, and beware the silent failure

⚠️ --tool-call-parser glm fails silently on this model. It is the GLM-4.5 format; GLM-5.x emits a different one. The symptom is not an error — the request succeeds and you get:

// ✗ --tool-call-parser glm
"finish_reason": "stop",
"tool_calls": null,
"content": ""          // <- the parser consumed the tool call and dropped it

An empty content alongside tool_calls: null is the tell: the parser matched enough to swallow the output but not enough to emit a call. Use glm47, which is also what the vendor vLLM recipe specifies for GLM-5.x.

📐 Sizing: the KDA state cache is the real constraint, not KV

GLM-5.3-Flash has 34 KDA linear-attention layers, each needing a per-request recurrent state. That cost scales with concurrency, and on a memory-tight box it — not the KV cache — is what limits you:

RuntimeError: Hybrid (mamba/linear-attention) state cache is too small to serve any requests.
max_mamba_cache_size=1, mamba_ratio=5, resulting max_num_reqs=0.

So context length and concurrency compete directly. Measured on 2× GB10 (TP=2) at --mem-fraction-static 0.84:

--max-running-requests	--context-length	result
8	131072	❌ mamba cache → max_num_reqs=0
2	65536	✅ KV 212,864 tok (2.52 GB), mamba cache 31 slots (2.19 GB)

If you hit that error, lower --max-running-requests before lowering context — the state cache is per-request, so concurrency is usually the cheaper thing to give up.

--enable-radix-cache (prefix reuse) and CUDA graphs both work; SGLang reports cuda graph: True in its decode lines and initializes a UnifiedTreeCore radix cache with a MAMBA component.

⚠️ GB10 also needs a small TileLang shared-memory patch

SGLang's stock TileLang tile requests 169,984 B of dynamic shared memory. Consumer/workstation Blackwell allows far less — GB10 measures:

shared_memory_per_block_optin = 101,376 B

so the kernel fails to launch. Working tile on GB10: block_I=32, num_stages=1, threads=128. Measured coupling worth knowing: threads must drop to 128 before block_I can drop to 32, or the m_i/alpha fragment layouts become unsatisfiable; block_I=16 hits an MMA assert.

Retuned kernel verified against a float32 gathered-attention reference: rel err 2.4e-3 (bf16 rounding noise) — correct, not merely non-crashing.

Datacenter parts (H100/B200/GB200) have the shared-memory headroom and need no patch.

If you add a speculative drafter, you need this tile even if plain decode worked. Reported by @randomllama running the incoai DFlash2 drafter on 2× GB10 (SGLang TP=2): the DSA verify shape (num_tokens_per_req=8) requests the same 169,984 B and blows the same 101,376 B limit, while plain decode fits at stock tiles — so a drafter-less boot never surfaces it, and the same block_I=32, num_stages=1, threads=128 tile is the fix.

⚠️ A drafter also breaks the mem-fraction envelope below

The --mem-fraction-static 0.84 figures in the table above are drafter-less. With a drafter at D=8 the hybrid state pool needs per_req * (1 + D) (~633 MB) on top of the ~`ratio=5per-request amplification, and you getmax_mamba_cache_size <= 0`. Working configuration reported in discussion #6:

  --mem-fraction-static 0.88 \
  --max-total-tokens 131072 \
  --mamba-full-memory-ratio 2

measured at 27.6 tok/s code decode, 1.88× over the same stack without speculation.

⚠️ vLLM + NVFP4 MoE: check your output before trusting it

This checkpoint is weight-only NVFP4: activations are not statically quantized.

Updated 2026-08-30. main originally shipped no input_scale at all, which is what triggers the bug below. It now carries placeholder input_scale = 1.0 tensors (commit 357b45cc) so the alpha fold is no longer a multiply-by-zero. Be aware that 1.0 is a placeholder, not a calibrated value — for reference, RedHat's calibrated activation scale for the same projection is ~`6.6e-4` in this convention. We have not numerically verified the placeholder, so still check your output.

A cleaner fix is on the compressed-tensors branch (--revision compressed-tensors): same 4-bit expert tensors, re-laid out in the compressed-tensors format, which never enters ModelOptNvFp4FusedMoE and so has no scale to fold — and whose MTP layer is FP8 rather than FP4. It has not been loaded by an engine yet; if you try it, please report back in the discussions. SGLang users should stay on main — the SGLang path is known to work with the ModelOpt layout here and has not been tested with the other one.

vLLM's ModelOptNvFp4FusedMoE registers the activation scale as an uninitialized parameter and expects the checkpoint to fill it. With a checkpoint that carries no input_scale it stays at zero, so the dequantization alpha for every expert becomes weight_scale_2 * 0, the whole mixture of experts is multiplied by zero, and the model emits one token repeatedly. There is no error and no warning. vLLM's NVFP4 linear methods refuse such a checkpoint explicitly; the MoE method does not.

If you see degenerate output, either:

  --moe-backend marlin          # dequantizes weights, never reads an activation scale

or install the plugin from the recipe repo, which supplies the missing scale and lets you keep flashinfer_cutlass (marlin repacks at roughly 19 GiB per rank, which does not fit at TP=2 on 2× GB10).

We observed this on sm_121. The trigger is the checkpoint rather than the architecture, so we expect it to reproduce on datacenter Blackwell as well; we have not yet confirmed that.

---

vLLM — H100 / B200 / GB200

docker run --gpus all --ipc=host -p 8000:8000 \
  vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  --model LibertAIDAI/GLM-5.3-Flash-NVFP4 \
  --tensor-parallel-size 4 \
  --tool-call-parser glm47 --enable-auto-tool-choice \
  --reasoning-parser deepseek_r1

⚠️ On vLLM use deepseek_r1, NOT glm45. The glm45 recommendation above is SGLang-specific. On vLLM that name resolves to Glm47MoeParserReasoningAdapter, whose state machine expects an opening <think> in the output — but this model's template emits <think> as the last prompt token, so the parser never opens a valid span and silently discards the whole reply: completion_tokens: 400 with content and reasoning_content both empty, on /v1/chat/completions and /v1/messages alike. It presents as a broken or hanging model rather than a parser fault. deepseek_r1 terminates on the bare </think> and splits correctly. Note also that vLLM returns the trace under reasoning, not reasoning_content.

The MTP layer is kept in BF16, so speculative decoding works as in the vendor recipe:

  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'

💡 Engine init on a 320B MoE is slow. Set VLLM_ENGINE_READY_TIMEOUT_S=3600 or the server is killed mid-warmup.

---

📦 What's quantized

Tensor group	Precision	Count
Routed-expert FFN (language_model.layers.*.mlp.experts.*.{gate,up,down}_proj)	NVFP4 (g16, weight-only)	37,152
KDA linear attention (34 layers) · DeepSeek sparse attention + indexer (11 layers)	BF16	—
Vision tower (model.visual.*) · shared experts · routers · dense/MTP MLP · mHC · embeddings · lm_head · norms	BF16	1,618

38,770 tensors total. Every non-expert Linear is listed in config.json's ignore.

📝 That ignore list names modules as they appear in the checkpoint (unfused, e.g. *.self_attn.q_proj). Engines fuse at load time and each picks its own name for the fused module, so the list also carries the fused spellings (qkv_proj, fused_qkvbfg_a_proj, fused_fg_b_proj, qkv_conv1d, fused_qkv_a_proj_with_mqa). Without them SGLang treats BF16 attention as quantized and the load asserts.

💾 Size

	BF16 source	This checkpoint
Routed experts (311.65B params)	623 GB	175 GB (NVFP4)
Everything else (9.67B params)	19 GB	19 GB (BF16)
Total	598.5 GiB	≈181 GiB

That is the difference between "needs a GB200 tray" and "fits on two GB10 desktops."

---

🔬 Provenance & verification

• Base: zai-org/GLM-5.3-Flash (BF16, 120 shards, 598.5 GiB)
• Method: shard-streaming ModelOpt NVFP4QTensor weight-only pass, CPU-only. Expert FFN → NVFP4; everything else copied through in BF16.
• Partition check: every Linear weight is provably quantized XOR ignored — no tensor both, none uncovered.
• Round-trip: per-expert cosine ≈ 0.99665, relative error ≈ 0.0925 vs the BF16 source.
• End-to-end: generates correct, coherent text on 2× GB10 under SGLang (TP=2).

⬜ No throughput numbers are published. We have not benchmarked this checkpoint on any hardware, and we would rather publish nothing than publish a number we did not measure.

---

🧠 Why NVFP4?

On NVIDIA Blackwell GPUs (RTX 50-series, B100/B200, GB10/DGX Spark) NVFP4 weights run on native FP4 tensor-core kernels. For an MoE, whose decode is dominated by memory bandwidth, 4-bit expert weights cut both VRAM and the bytes moved per token. This gives format parity with the NVFP4 path used by vLLM, SGLang and TensorRT-LLM.

🌐 About LibertAI

LibertAI is a decentralized AI platform — private inference, an OpenAI-compatible API, and a chat UI, all running on community GPUs over Aleph Cloud instead of a single company's servers. No accounts required to chat, no logs sent home, and the same models you'd self-host are available behind a sovereign endpoint.

Want to put this model to work as an autonomous agent without running your own infrastructure? LiberClaw hosts Hermes-style agents on Aleph Cloud with LibertAI inference. Free tier: 2 agents, no credit card, 5 minutes to deploy. Open source.

---

MIT licensed, as is the base model · Quantized by LibertAI