aj9o9/Qwen3.8-27B-Escha-W2-GGUF

Escha Qwen3.8-27B W2 GGUF (dense)

I ported EschaLabs/Qwen3.8-27B-Escha-W2 to llama.cpp. Weights are theirs. This is not a requant.

I decode their native 2-bit escha code in-kernel, through a new op, GGML_OP_ESCHA_MUL_MAT. The quantized payload is the same bytes as the safetensors. Nothing is unpacked to a dense tensor on disk or in VRAM.

This is the dense 27B. My earlier port of their MoE 35B is a separate repo and a separate branch.

Stock llama.cpp cannot load these files. You need my fork, branch escha-w2-dense:

https://github.com/Ajay9o9/llama.cpp-escha/tree/escha-w2-dense

If you only download the GGUF, it will not run.

Files

Two builds. Same weights, same kernel. They differ only in how the token embedding and the output head are stored.

File	Size	Embed + head
Escha-Qwen3.8-27B-W2-Q8E.gguf	10.31 GB	Q8_0
Escha-Qwen3.8-27B-W2.gguf	12.69 GB	F16

Both are 2054 tensors, 2.469 bits per weight on the quantized tensors, K=2 and K=3 mixed. 64 layers, 48 linear attention and 16 full attention (full_attention_interval 4), 5120 hidden, 24 heads over 4 KV heads, head dim 256, vocab 248,320.

The Q8_0 build is a repack of the source int8, not a requantization. The row's f16 scale goes into every block and the payload is copied verbatim. I dequantized 64 rows out of the GGUF and compared against int8 * scale from the safetensors: scales identical per row, payload identical, max abs diff 0.0. 2052 of the 2054 tensors are byte-identical between the two files.

Which one to take

Take Q8_0. It is 2.38 GB smaller and 3.4% faster to generate.

The Q8_0 file has exact weights where F16 rounds them by about 0.02%, so I expected it to land closer to Escha's runtime. It lands slightly further: 318/320 against 319/320 on top-1. llama.cpp's Q8_0 matmul (vec_dot_q8_0_q8_1) quantizes the activations to 8 bits for an integer dot product, while the F16 path leaves them in fp32. So the trade is "weights rounded, activations exact" against "weights exact, activations quantized", and the second error is larger. That is read off the kernel and the measurement. I did not instrument it.

One position in 320 is inside the noise. Not a reason to carry 2.38 GB more.

Quality vs Escha SGLang

Same GPU, same day. I send token ids, not text, so both stacks see the same context and tokenizer differences cannot leak in. Teacher-forced: the prefix is fixed at every position, so a miss at i-1 does not poison i.

Top-1 agreement, 5 mixed prompts, 320 positions
F16 head	319 / 320 = 99.7%
Q8_0 head	318 / 320 = 99.4%

Growing context, 24 positions from depth 64 to 8215
F16 head	24 / 24 = 100%
Q8_0 head	24 / 24 = 100%

No drift as context grows. Mean JS divergence over the top-20 was 0.0000 bits. Median absolute logprob difference 0.006 (F16) and 0.013 (Q8_0).

My fp32 build and my fp16 tensor-core prefill build agree with each other on all 320 positions. That is what let me ship the tensor-core path. It separates "is the kernel correct" from "is fp16 good enough", which perplexity alone cannot do.

Op test against a numpy dense-fold reference, CPU and CUDA scored separately: rel RMS around 1e-6 at K=2 and K=3.

Perplexity: 7.4016 from a 512-context back-half estimator. EschaLabs report 7.43 from the same kind of estimator, so I am in the right place. I could not reproduce their 7.2652, no wikitext-2 on this machine. The logit agreement above is the evidence here, not this number.

Speed

One RTX 3090, power limit 250 W (stock is 350 W), single stream, batch 1, full GPU offload. This card is about 43% of a 4090 on FP32 and power capped on top of that.

llama-bench
pp512	700.4 tok/s
tg128, Q8_0 head	24.03 tok/s
tg128, F16 head	23.17 tok/s

Served, through llama-server at -c 70000, measured with llama-benchy:

depth	prefill tok/s	decode tok/s
8k	634.5	22.07
16k	612.8	22.11
32k	574.6	19.65
64k	547.2	20.24

EschaLabs' own SGLang runtime on this same 3090 gives 981 / 987 / 933 prefill and 31.17 / 29.05 / 28.37 decode at 8k / 16k / 32k. Their runtime is still faster at generation.

Prefill went from 212.4 to 700.4 tok/s over the port, 3.3x. Most of it was tensor cores (mma.m16n8k16, fp32 accumulate), cp.async staging, and hoisting integer division out of the tile loops. Decode went from 16.6 to 24.03.

Build the fork

git clone -b escha-w2-dense https://github.com/Ajay9o9/llama.cpp-escha.git
cd llama.cpp-escha
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build -j 12

86 is an RTX 3090. Change it for your GPU. -j 12 is a 12-core CPU.

Tensor cores need compute capability 7.5 or newer, Turing and up. On older cards the prefill falls back to an fp32 kernel on its own. ESCHA_NO_MMA=1 forces that fallback if you want to compare.

Download

hf download aj9o9/Qwen3.8-27B-Escha-W2-GGUF Escha-Qwen3.8-27B-W2-Q8E.gguf --local-dir .

Run it

./build/bin/llama-server \
  -m Escha-Qwen3.8-27B-W2-Q8E.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 -fa on --jinja \
  -np 1 -t 12 \
  -c 131072 \
  --cache-type-k q8_0 --cache-type-v q8_0

The Q8_0 build runs the full 262144 context, with a q8_0 KV cache, in under 20 GB. Measured on a 3090, not estimated. So you can use -c 262144 instead of the 131072 above.

The reason it fits: only 16 of the 64 layers are full attention, with 4 KV heads at head dim 256, so the KV cache is about 34 KB per token at q8_0. Full context is roughly 8.5 GiB of that, on top of 9.6 GiB of weights.

With the 12.69 GB F16 build, drop to -c 98304.

Smaller cards

The weights are 9.60 GiB and the compute buffer is about 0.59 GiB, so the card size mostly decides how much context you get:

context	total VRAM
32768	11.25 GiB
65536	12.31 GiB
131072	14.44 GiB
262144	18.69 GiB

A 16 GB card should run it at up to about 128k context. Only the 262144 row is measured; the rest is arithmetic from the same weights and compute-buffer numbers, and I have not tested any card other than a 3090.

Generation speed will not carry over to a smaller card. Decode here is memory-sensitive, and a 16 GB card with 288 GB/s of bandwidth has under a third of a 3090's, so expect well below 24 tok/s on one.

It is a reasoning model. Give it room to think or you get an empty answer.

What this is not

• Not a Q4_K / Q8_0 requant of a dense reconstruction. The 2-bit code is decoded in the CUDA kernel
• Not upstream llama.cpp
• Not vision. The source config is Qwen3_5ForConditionalGeneration. I converted the text model only
• No MTP. The Escha 27B checkpoint ships no MTP head, 0 draft tensors out of 3253, so there is nothing to port. Their MoE 35B had one, this one does not
• No DFlash2 speculative decoding. Qwen3.8-27B-DFlash2-Q4_K_M.gguf is 81 tensors, 58 plus 23 for the conv and selector blocks, and those 23 are implemented neither in my fork nor in upstream llama.cpp. The fork does carry the older 58-tensor dflash path, which I have not tested against this model
• No MMLU / GSM8K / task-suite eval. No long-context eval past the depth table above
• CPU inference is not the point here. Everything above is -ngl 99

License

Apache-2.0, same as the Escha weights.

Source: EschaLabs/Qwen3.8-27B-Escha-W2 Runtime I compared against: EschaLabs' escha build on SGLang