Qwen3-4B-Feiyue-v1-Q8_0 ⭐

Overview

Q8_0 quantized GGUF — the recommended deployment format for Feiyue-v1. Near-lossless quality in a compact 8-bit package, optimized for RTX 5060 (8 GB VRAM) with full 8K context.

Property	Value
Base Model	Qwen/Qwen3-4B-Instruct-2507
Quant Method	Q8_0 (8-bit, GGUF)
File Size	~4.3 GB
VRAM (8K ctx)	~5.5 GB
License	Apache 2.0

Core Capabilities

1. PRD Generation — Full fidelity, hierarchical task decomposition preserved
2. Code Review — All quality retained from bf16
3. Tool-calling — Near-lossless, production-ready

Quantization Quality

Capability	Q8_0	Q4_K_M	Notes
PRD Generation	✅ 98%+	⚠️ 50%	Q8 retains full logical structure
Code Review	✅ 98%+	✅ 70%	Both functional; Q8 more nuanced
Tool-calling	✅ 99%	⚠️ 70%	Q8 strongly recommended for agents

✅ Production recommendation: Q8_0 on RTX 5060 (8 GB) with 8K context.

System Requirements

GPU VRAM	Max Context	Feasible
8 GB	8,192	✅ Recommended
6 GB	4,096	✅ Works
4 GB	2,048	⚠️ Tight

🦙 Ollama Deployment (Recommended)

Ollama is the simplest way to run Feiyue-v1 on Windows, macOS, or Linux.

Quick Start

# Pull and run directly from HuggingFace
ollama run hf.co/sinonchum/Qwen3-4B-Feiyue-v1-Q8_0

# Or with explicit quant tag
ollama run hf.co/sinonchum/Qwen3-4B-Feiyue-v1-Q8_0:Q8_0

Custom Modelfile (Persistent Setup)

Create a Modelfile for full control over system prompt, temperature, and context:

FROM Qwen3-4B-Feiyue-v1-Q8_0.gguf

PARAMETER temperature 0.7
PARAMETER num_ctx 8192

TEMPLATE """{ if .System }<|im_start|>system
{ .System }<|im_end|>
{ end }{ if .Prompt }<|im_start|>user
{ .Prompt }<|im_end|>
{ end }<|im_start|>assistant
{ .Response }<|im_end|>"""

SYSTEM """You are Feiyue, an AI agent specialized in PRD generation, code review, and tool calling."""

Build and run:

# Save Modelfile, then:
ollama create feiyue-q8 -f Modelfile
ollama run feiyue-q8

Windows Notes

• Install Ollama from ollama.com/download/windows
• Place the .gguf file in C:\Users\<yourname>\.ollama\models\ for local use
• For GPU acceleration on Windows, ensure NVIDIA drivers ≥ 526.x and CUDA ≥ 11.8
• RTX 5060 (8 GB VRAM) with 5.5 GB GGUF: comfortable with 8192 context
• To set model directory: set OLLAMA_MODELS=C:\path\to\models (CMD) or $env:OLLAMA_MODELS="C:\path\to\models" (PowerShell)

Ollama API (Programmatic Access)

import requests

response = requests.post("http://localhost:11434/api/chat", json={
    "model": "feiyue-q8",
    "messages": [
        {"role": "system", "content": "You are Feiyue, an AI agent."},
        {"role": "user", "content": "Review this code for SQL injection vulnerabilities."},
    ],
    "stream": False,
    "options": {"temperature": 0.7, "num_ctx": 8192},
})
print(response.json()["message"]["content"])

llama.cpp Usage

huggingface-cli download sinonchum/Qwen3-4B-Feiyue-v1-Q8_0 Qwen3-4B-Feiyue-v1-Q8_0.gguf --local-dir .

./llama-cli \
  -m Qwen3-4B-Feiyue-v1-Q8_0.gguf \
  -c 8192 -ngl 99 --temp 0.7 \
  --chat-template chatml \
  -p "You are Feiyue, an AI agent."

Python (llama-cpp-python)

from llama_cpp import Llama
llm = Llama.from_pretrained(
    repo_id="sinonchum/Qwen3-4B-Feiyue-v1-Q8_0",
    filename="Qwen3-4B-Feiyue-v1-Q8_0.gguf",
)
llm.create_chat_completion(
    messages=[{"role": "user", "content": "Review this code for security issues."}]
)

Training Summary

See sinonchum/Qwen3-4B-Feiyue-v1-bf16 for full training details.

Related Models

• bf16 Merged — Full precision
• Adapter — LoRA weights
• Q4_K_M — 4-bit compact variant