Instructions to use sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS
# Run inference directly in the terminal:
llama cli -hf sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS
# Run inference directly in the terminal:
llama cli -hf sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS
# Run inference directly in the terminal:
./llama-cli -hf sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS

Use Docker

docker model run hf.co/sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS

LM Studio
Jan

vLLM

How to use sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS

Ollama
How to use sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF with Ollama:
```
ollama run hf.co/sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS
```
Unsloth Desktop

How to use sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF with Docker Model Runner:
```
docker model run hf.co/sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS
```

Lemonade

How to use sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS

Run and chat with the model

lemonade run user.Ornith-1.5-397B-IQ3_XXS-GGUF-IQ3_XXS

List all available models

lemonade list

Hermes Agent

How to use sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF:IQ3_XXS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Ornith-1.5-397B — IQ3_XXS GGUF

An IQ3_XXS quantization of ornith-ai/Ornith-1.5-397B.

The official Ornith-1.5-397B-GGUF repository stops at Q4_K_M (224.08 GiB). That does not fit in 192 GB of VRAM. This one does.

	size	BPW
official Q4_K_M	224.08 GiB	4.86
this IQ3_XXS	142.68 GiB	3.09

Split into 4 files of ≤45 GB. Point llama.cpp at -00001-of-00004 and it loads all four.

Files

file	bytes
`Ornith-1.5-397B-IQ3_XXS-00001-of-00004.gguf`	44,460,551,168
`Ornith-1.5-397B-IQ3_XXS-00002-of-00004.gguf`	44,665,664,288
`Ornith-1.5-397B-IQ3_XXS-00003-of-00004.gguf`	44,695,059,584
`Ornith-1.5-397B-IQ3_XXS-00004-of-00004.gguf`	19,373,124,448

Total 1098 tensors, 153,194,399,488 bytes across 4 files (142.67 GiB; the unsplit file is 153,194,399,008 B — the difference is per-split headers). See SHA256SUMS.txt.

For vision, use mmproj-Ornith-1.5-397B-BF16.gguf from the official GGUF repo (not mirrored here).

How it was made

Source was the official Q8_0 GGUF (392.56 GiB, 8.51 BPW), not the BF16 checkpoint. That means --allow-requantize was used — this is a re-quantization of an already-quantized tensor set. Q8_0 is close to lossless, but this is stated plainly so you can weigh it.

llama-quantize --allow-requantize \
  --imatrix <imatrix.gguf> \
  --token-embedding-type q5_K \
  Ornith-1.5-397B-Q8_0.gguf Ornith-1.5-397B-IQ3_XXS.gguf IQ3_XXS 64

llama-gguf-split --split --split-max-size 45G \
  Ornith-1.5-397B-IQ3_XXS.gguf Ornith-1.5-397B-IQ3_XXS

--token-embedding-type q5_K overrides the IQ3_XXS default (iq3_s) for token_embd. With a 248,320-token vocabulary carrying CJK, the extra ~250 MiB is worth it.

Quantization took 31m40s on a Threadripper PRO 9985WX (64 cores, 64 threads).

About the importance matrix

The imatrix is not ours and is not mirrored here. We used unsloth/Qwen3.5-397B-A17B-GGUF's imatrix_unsloth.gguf_file (80 chunks × 11264 tokens).

This works because Ornith-1.5-397B is a light fine-tune of Qwen/Qwen3.5-397B-A17B:

the 1371 non-MTP tensor names are identical sets (set difference is empty)
the vision tower is bit-identical (frozen), as are linear_attn.A_log and dt_bias
the language trunk has cosine similarity 0.9993–0.99999 (relative L2 of 1–4%)
the safetensors total_size differs by exactly 13,191,153,536 B — precisely the MTP head

We verified name compatibility before quantizing: 765 of 765 imatrix entries match tensors in the Ornith Q8_0 (100%). The 180 quantizable tensors without imatrix coverage are norms and ssm_conv1d, which are not quantized anyway.

Notably, 765 is the same quantize.imatrix.entries_count recorded in the official Ornith GGUF headers — the official build used the same number of entries.

If you want a purpose-built imatrix, compute one against this model directly. We did not, and we say so rather than implying otherwise.

Measured

Pure CPU, Threadripper PRO 9985WX, 64 threads, -dev none:

	prefill	decode
Q8_0 (reference)	41.0–41.8 t/s	9.7–9.8 t/s
IQ3_XXS	33.9–34.6 t/s	13.0–13.1 t/s

Same prompt (three summer haiku, different kigo, one line each), temp 0.8, thinking off:

Q8_0 —

金魚売り通り過ぎていく水の音
青トマトかじれば夏の朝の味
夕立やアスファルト跳ねる子らの声

IQ3_XXS —

夏日や池の鯉ゆく水草かげ
夏炉や炉の灰に眠る火の粉かな
夏空や雲の切れ間より富士の山

Both hold 5-7-5 and use three distinct summer kigo. Q8_0 reaches for more modern imagery, IQ3_XXS sits closer to classical form. Neither is broken.

Perplexity has not been measured. Stated as missing rather than guessed at.

Why not IQ2

We also baked IQ2_XXS (97.65 GiB, 2.12 BPW) and do not recommend it. It answers factual questions correctly ("日本の首都は東京です") but cannot carry out multi-step generation — asked for haiku it emits bullet-point glossaries of season words, and at temp 0.8 it degenerates into repetition with stray tokens. At 2.12 BPW this model does not survive. It is not published here.

IQ3_XXS is, in our measurements, the floor.

Usage

llama-server -m Ornith-1.5-397B-IQ3_XXS-00001-of-00004.gguf \
  -c 32768 --threads 64

Ornith is a reasoning model and it thinks at length. With -n 1500 it had not finished deliberating. For direct answers:

--chat-template-kwargs '{"enable_thinking":false}'

If you keep thinking on, budget generously (the 35B sibling needed ≥6500 tokens) and strip everything before </think> before parsing code out of a response — otherwise you will grade the model's scratch work instead of its answer.

⚠️ GPU offload does not work yet on SM 12.0

On 12× RTX PRO 2000 Blackwell (SM 12.0, CUDA 13.2) this model crashes on GPU:

ggml_cuda_compute_forward: SOFT_MAX failed
CUDA error: invalid argument

Isolated by bisecting -ngl:

-ngl 1 (layer 59, a full_attention layer) → runs
-ngl 2 (adds layer 58, a linear_attention layer) → crashes

So it is the linear-attention (gated delta net) path. -fa on does not help (flash_attn = enabled is logged and SOFT_MAX is still reached), nor does --no-warmup, nor -ub 1 -b 1. Reproduced on both a 2026-08-10 build and on master at d59d455 (174 commits newer). CPU inference is unaffected.

Separately, llama.cpp misclassifies Blackwell as an integrated GPU because cudaDeviceProp.integrated is non-zero (the driver API correctly reports 0 for the same device). Only the first "iGPU" is kept, so -sm/-ts silently do nothing and everything piles onto device 0. Upstream #26901, open since 2026-08-11. Work around it by naming devices explicitly:

-dev CUDA0,CUDA1,CUDA2,CUDA3,CUDA4,CUDA5,CUDA6,CUDA7,CUDA8,CUDA9,CUDA10,CUDA11
-ts 4.5,5,5,5,5,5,5,5,5,5,5,6.5

That does distribute the layers correctly (verified in the load log) — the SOFT_MAX crash is a separate, unresolved problem.

Note for anyone re-converting from safetensors

config.json declares mtp_num_hidden_layers=1, but there is not a single MTP tensor in the checkpoint (1371 tensors, 0 MTP) or in the official GGUF (1098 tensors, 0 nextn). The 35B-A3B sibling does ship 785 of them; the 397B does not, in either 1.0 or 1.5.

Convert with --no-mtp. Without it you get a GGUF declaring block_count=61 with an empty blk.60, and llama.cpp fails at load with a missing-tensor error.

Attribution

Base model: ornith-ai/Ornith-1.5-397B — MIT. All credit for the model belongs to its authors.
Importance matrix: unsloth/Qwen3.5-397B-A17B-GGUF.
Tooling: llama.cpp.

This repository contributes quantized weights and the measurements above. Nothing else.

Downloads last month: 649

GGUF

Model size

396B params

Architecture

qwen35moe

Hardware compatibility

3-bit

Model tree for sakamakismile/Ornith-1.5-397B-IQ3_XXS-GGUF

Base model

ornith-ai/Ornith-1.5-397B

Quantized

(9)

this model