Instructions to use OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M

Use Docker

docker model run hf.co/OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M

Ollama
How to use OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF with Ollama:
```
ollama run hf.co/OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M
```
Unsloth Desktop

How to use OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF with Docker Model Runner:
```
docker model run hf.co/OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M
```

Lemonade

How to use OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwopus-KAT-Coder-35B-Merged-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Self-speculative Multi-Token Prediction, native vision input, and MoE Adaptive Precision (APEX) quantization — see capabilities and quantization details below.

Contents — At a Glance · Executive Summary · MTP & Vision · Fusion Architecture · Intended Use & Limitations · Quantization & Hardware Sizing · Quickstart · Sampling · Prompt & Chat Templates · Lineage · Citation · License

At a Glance


Total parameters	~35B
Active parameters / token	~3B (8 of 256 routed experts + 1 shared expert)
Layers	40
Attention	Hybrid GatedDeltaNet linear attention with periodic full attention
Context window	32,768+ tokens
Flagship Quantization	APEX Quality (~~21.3 GB) · `Q4_K_M` (~~20.4 GB) · `Q8_0` (~35.2 GB)
Recommended Template	Improved Chat Template for Qwen 3.x
Ideal hardware	Single 24 GB GPU (RTX 3090/4090), or 32 GB+ unified memory (Apple Silicon)
Languages	English, Chinese
License	Apache 2.0

↑ back to top

Executive Summary

Qwopus-KAT-Coder-35B is a Mixture-of-Experts (MoE) coding model built by merging two top-tier 35B fine-tunes:

Jackrong/Qwopus3.6-35B-A3B-Coder — deep algorithmic logic synthesis, complex mathematical reasoning, and MTP speculative decoding.
Kwaipilot/KAT-Coder-V2.5-Dev — state-of-the-art SWE-agent repository execution, native tool-calling, and automated bug resolution.

The result pairs multi-step reasoning depth with autonomous environment tool execution: a private, self-hosted coding agent that fits on a single 24 GB consumer GPU.

Quantization Highlight: This repository features APEX Quality quantization — an adaptive mixed-precision MoE layout that preserves critical shared experts and attention blocks at high precision while applying layer-gradient compression across the 256 routed experts, delivering near-Q8 accuracy at a 21.3 GB footprint.

↑ back to top

MTP & Vision Capabilities

Both parent lines trace back to Qwen3.6-35B-A3B (Alibaba Cloud), inheriting access to native vision encoding and a multi-token prediction (MTP) speculative head.

Multi-token prediction (MTP)

Self-speculative decoding allows the model to draft multiple tokens per forward pass using its own MTP head without requiring an external draft model. Enable it in llama.cpp via --spec-type draft-mtp.

Realistic speedup: Community testing on Qwen3.6-35B-A3B MTP builds reports 1.4–2.2× faster generation with no accuracy loss, depending on prompt predictability and acceptance rates.
Merge behavior note: The Qwopus3.6 parent ships an MTP head, while KAT-Coder-V2.5-Dev does not. The merged weights preserve the MTP projection layers; benchmark --spec-type draft-mtp against standard autoregressive decoding on your specific workload.

Vision / multimodal input

The underlying architecture supports image input for tasks like UI-screenshot-to-code, architecture diagram scaffolding, and GUI bug root-cause analysis.

Requirement: Vision processing requires a matching multimodal projector (mmproj) file passed via --mmproj <path> (e.g. llama-mtmd-cli or llama-server). The text GGUF alone processes text and code tokens.

↑ back to top

The Fusion Architecture

Highlight	Description
Active-parameter efficiency (A3B)	35B total parameters with ~3B active per token (8 of 256 routed experts + 1 shared expert). Achieves 70B-tier coding performance at 3B-tier inference latency.
Hybrid GatedDeltaNet linear attention	Linear recurrence layers interleaved with periodic standard self-attention, keeping KV cache growth and memory scaling flat over long context (32k+ tokens).
Dual-format expert consolidation	Unified blending across Qwopus's fused 3D expert tensors and KAT-Coder's discrete expert matrices, combining logic reasoning and tool use into single cohesive expert representations.

↑ back to top

Intended Use & Limitations

Intended use

Local, self-hosted coding agent: multi-file repository refactoring, bug fixing, test generation, and autonomous terminal tool execution.
High-performance private code assistant running entirely on local consumer hardware (24 GB VRAM or 32 GB+ Apple Silicon).
Multimodal UI-to-code workflows (when paired with a compatible mmproj vision file).

Out of scope

Unreviewed execution in safety-critical production environments without automated validation/sandboxing.
General-purpose casual chat (the model is strongly optimized for coding, reasoning, and agentic JSON/XML tool calling).

↑ back to top

Quantization & Hardware Sizing

Available Quantizations

Quant Type	File Size	Memory Required (VRAM / RAM)	Precision Strategy	Recommended Setup
`APEX-Quality`	~21.3 GB	24 GB VRAM / 32 GB RAM	Adaptive MoE Gradient (Q8/Q6/Q5/Q4)	⭐ Recommended Flagship (RTX 3090/4090, Mac 32GB+)
`Q4_K_M`	~20.4 GB	24 GB VRAM / 32 GB RAM	Standard uniform K-quantization	24 GB VRAM / Apple Silicon
`Q8_0`	~35.2 GB	40 GB+ VRAM / 64 GB RAM	Full 8-bit uniform quantization	A100 / A6000 / Mac Studio (64GB+)

Deep Dive: APEX Quality MoE Architecture

Standard uniform quantization treats all weights equally, which can degrade sparse MoE networks. APEX (Adaptive Precision for EXpert Models) applies mixed-precision quantization mapped to the structural importance of each layer group:

Shared Experts (shexp) $\rightarrow$ Q8_0: Always active across every token; kept at 8-bit precision to eliminate cumulative outlier degradation.
Attention & SSM Projections (attn_qkv, ssm_*) $\rightarrow$ Q6_K: Preserves state-space recurrence and linear attention dynamics over long repositories.
Embeddings & Output (token_embd, output) $\rightarrow$ Q8_0 / Q6_K: Maintains vocabulary coordinate stability and clean token sampling.
256 Routed Experts (ffn_*_exps) $\rightarrow$ 3-Tier Layer Gradient:
- Edge Layers (L0–L4 & L35–L39): Q6_K (preserves initial feature extraction and final token routing).
- Near-Edge Layers (L5–L9 & L30–L34): Q5_K (smooth transition gradient).
- Middle Layers (L10–L29): Q4_K / IQ4_XS (aggressive compression across redundant sparse experts).

Result: APEX Quality matches uncompressed Q8_0 coding benchmarks while fitting entirely within 24 GB VRAM at 30–60+ tokens/sec.

↑ back to top

Quickstart Guide

1. Launch with `llama.cpp`

Interactive CLI (APEX Quality + MTP speculative decoding)

llama-cli \
  -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Qwopus-KAT-Coder-35B-Merged-APEX-Quality.gguf \
  -ngl 99 \
  -c 32768 \
  --spec-type draft-mtp \
  --temp 0.7 --top-p 0.95 --top-k 20 \
  -p "You are an expert autonomous software engineer. Write a production-grade Async Task Queue in Python with Redis backend."

Local OpenAI-Compatible Server

llama-server \
  -hf OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF:Qwopus-KAT-Coder-35B-Merged-APEX-Quality.gguf \
  -ngl 99 \
  -c 32768 \
  --spec-type draft-mtp \
  --temp 0.7 --top-p 0.95 --top-k 20 \
  --port 8080 \
  --host 0.0.0.0

Vision Support: To enable multimodal image input, append --mmproj <path-to-mmproj.gguf> to your llama-server command.

2. Ollama Integration (with Custom Modelfile)

Create a Modelfile:

FROM ./Qwopus-KAT-Coder-35B-Merged-APEX-Quality.gguf

TEMPLATE """{{- if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{- range .Messages }}<|im_start|>{{ .Role }}
{{ .Content }}<|im_end|>
{{ end }}<|im_start|>assistant
"""

PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
PARAMETER temperature 0.7
PARAMETER top_p 0.95
PARAMETER top_k 20
PARAMETER presence_penalty 1.5
PARAMETER repeat_penalty 1.0
PARAMETER num_ctx 32768

Build and run:

ollama create qwopus-kat-35b -f Modelfile
ollama run qwopus-kat-35b

3. Agentic IDE Setup (Cline / Roo-Code / Continue.dev)

Point your IDE agent extension directly to your local llama-server:

{
  "apiProvider": "openai",
  "apiBaseUrl": "http://localhost:8080/v1",
  "apiKey": "local",
  "modelId": "Qwopus-KAT-Coder-35B-Merged-APEX-Quality.gguf",
  "contextWindow": 32768,
  "maxTokens": 4096
}

↑ back to top

Recommended Sampling Hyperparameters

Important: Do not use generic ultra-low temperatures (0.1–0.2) or high repetition penalties (>1.05) on Qwen3.6-35B-A3B MoE models. Doing so causes expert activation collapse and repetitive looping. Use the tested settings below:

temperature: 0.7           # Optimal range is 0.6–0.9 for reasoning + code synthesis
top_p: 0.95
top_k: 20
min_p: 0.00                # Keep at 0.00 to avoid over-filtering sparse expert activations
presence_penalty: 1.5       # Keeps multi-step logic focused and avoids reasoning drift
repeat_penalty: 1.0         # Stay at 1.0 (do not exceed 1.05)
context_window: 32768

↑ back to top

Prompt Template & Chat Template Options

⭐ Recommended: Improved Chat Template for Qwen 3.x

For optimal tool-calling accuracy, strict schema adherence, and multi-turn stability, use the standalone Improved Chat Template for Qwen 3.x:

Repository: OliviaRossi/Improved-Chat-Template-for-Qwen-3.x
Key Enhancements:
- Fixes tool-call XML/JSON schema formatting bugs in complex multi-file coding workflows.
- Eliminates reasoning/thinking tag leakage into assistant responses.
- Properly formats tool execution returns and error traces across long multi-turn sessions.

To use it with llama-server or vLLM:

# Download template
curl -LO https://huggingface.co/OliviaRossi/Improved-Chat-Template-for-Qwen-3.x/raw/main/chat_template.jinja

# Pass directly to llama-server
llama-server --chat-template-file chat_template.jinja ...

Default Template Structure (ChatML)

<|im_start|>system
You are an autonomous AI coding assistant. You analyze complex codebases, write clean and efficient code, debug issues systematically, and produce complete, working implementations.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

↑ back to top

Acknowledgments & Lineage

Role	Model / Resource	Link
Ultimate base architecture	Qwen3.6-35B-A3B (Alibaba Cloud)	—
Qwopus base fine-tune	`Jackrong/Qwopus3.6-35B-A3B-Coder`	huggingface.co
KAT-Coder base fine-tune	`Kwaipilot/KAT-Coder-V2.5-Dev`	huggingface.co
Merged base weights	`OliviaRossi/Qwopus-KAT-Coder-35B-Merged`	huggingface.co
GGUF Quantizations	`OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF`	huggingface.co
Recommended Chat Template	`OliviaRossi/Improved-Chat-Template-for-Qwen-3.x`	huggingface.co
Quantization Toolchain	`llama.cpp` & `apex-quant`	llama.cpp · apex-quant

↑ back to top

Citation

@misc{qwopus-kat-coder-35b-merged,
  title  = {Qwopus-KAT-Coder-35B-Merged},
  author = {OliviaRossi},
  year   = {2026},
  note   = {SLERP merge of Jackrong/Qwopus3.6-35B-A3B-Coder and Kwaipilot/KAT-Coder-V2.5-Dev with APEX-Quality MoE quantization},
  url    = {https://huggingface.co/OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF}
}

↑ back to top

License

Released under the Apache 2.0 license. Usage is also subject to the licenses of the underlying parent models.

Built for developers who want a private, autonomous coding agent running locally on consumer hardware.

Downloads last month: 5,793

GGUF

Model size

36B params

Architecture

qwen35moe

Hardware compatibility

4-bit

8-bit

View +1 variant

Model tree for OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF

Base model

Qwen/Qwen3.6-35B-A3B

Finetuned

unsloth/Qwen3.6-35B-A3B

Adapter

Jackrong/Qwopus3.6-35B-A3B-v1

Adapter

Jackrong/Qwopus3.6-35B-A3B-Coder

Quantized

(21)

this model

At a Glance

Executive Summary

MTP & Vision Capabilities

Multi-token prediction (MTP)

Vision / multimodal input

The Fusion Architecture

Intended Use & Limitations

Quantization & Hardware Sizing

Available Quantizations

Deep Dive: APEX Quality MoE Architecture

Quickstart Guide

1. Launch with llama.cpp

Recommended Sampling Hyperparameters

Prompt Template & Chat Template Options

⭐ Recommended: Improved Chat Template for Qwen 3.x

Default Template Structure (ChatML)

Acknowledgments & Lineage

Citation

License

Model tree for OliviaRossi/Qwopus-KAT-Coder-35B-Merged-GGUF

1. Launch with `llama.cpp`