okayuji/gemma-4-12b-it-coreml-128k

Gemma 4 12B IT — Core ML, 128K Context Ladder

Google's Gemma 4 12B IT, converted to a native Core ML graph and shipped as a single int4 bundle. It decodes at the fast 32K speed for normal conversations and steps up to a full 131,072-token context only once a conversation actually grows past 32K — with no reload and no KV copy, and with bit-identical output before and after the step-up.

Built with Gemma. A Core ML conversion of google/gemma-4-12B-it — see License & attribution.

Who this is for

• Yes — you have a 24 GB+ Apple Silicon Mac and want to try a 12B model at very long context.
• Yes — you are building a Core ML LLM runtime and want a working multifunction + MLState bundle to study.
• No — you need iPhone/iPad or the Neural Engine: impossible here, the KV cache is a GPU-resident MLState.
• No — you need sampling (temperature / top-p), or you need 128K to be fast (it is ~3.3 tok/s).

Requires: Apple Silicon Mac · macOS 26 (Tahoe)+ · ~11 GB disk · ~12 GB RAM minimum, 24 GB+ recommended.

Quick start

hf download okayuji/gemma-4-12b-it-coreml-128k --local-dir ./gemma-4-12b-it-coreml-128k

The one thing that makes this bundle different is that a single weight set exposes two Core ML functions driven by one shared MLState. That is the whole ladder, in Swift:

import CoreML

// 1. Compile once. Each .mlpackage becomes one .mlmodelc containing BOTH functions.
//    Measured: ~1.2 s for the whole bundle on an M4 Max (~0.3 s per chunk). Cache and reuse it.
let compiled = try await MLModel.compileModel(at: bundleURL.appending(path: "chunk_0_12.mlpackage"))

// 2. Open the same .mlmodelc twice, once per function. No recompile, weights are deduplicated.
func open(_ function: String) async throws -> MLModel {
    let cfg = MLModelConfiguration()
    cfg.computeUnits = .cpuAndGPU        // required: the KV MLState is GPU-resident
    cfg.functionName = function          // "ctx32k" | "ctx128k"
    return try await MLModel.load(contentsOf: compiled, configuration: cfg)
}
let chunk32  = try await open("ctx32k")
let chunk128 = try await open("ctx128k")

// 3. ONE state, made from either model — it is physically [1, 131072, 512] in both cases.
let state = chunk32.makeState()

// 4. Decode through ctx32k; once the write position reaches 32768, decode through ctx128k
//    instead. Same `state` object, no read_state/write_state — that is the copy-zero promotion.
let model = (position < 32768) ? chunk32 : chunk128
let out = try model.prediction(from: inputs, using: state)

inputs above is elided on purpose: the host-side features (one-hot write slots, RoPE tables, sliding/full attention masks) are built by the companion package, and this card does not print code that has not been run. The snippet through step 3 is the verified part — compile, both functionName loads, and the shared state.

A complete runnable CLI (tokenizer, prompt template, host inputs, speculative decoding) lives in a companion Swift package at https://github.com/oka-yuji/coreml-llm-samples.

Sample output

M4 Max (128 GB) / macOS 26.5.2, CPU_AND_GPU, greedy decode, 32 prompt tokens (BOS + template), one process per run, via the companion CLI.

Prompt (Japanese — "What is the capital of Japan? Answer concisely, with one sentence of reasoning."):

日本の首都はどこですか?理由も一文添えて簡潔に答えてください。

Output:

日本の首都は東京です。政治、経済、文化の中心地として機能しているためです。

("The capital of Japan is Tokyo. It serves as the country's political, economic and cultural center.")

Run	TTFT	Decode	Draft acceptance	Peak memory
Speculative decoding ON	3.83 s	62.3 ms/tok (16.0 tok/s)	0.75	20.50 GB
Speculative decoding OFF	0.26 s	98.3 ms/tok (10.2 tok/s)	—	18.00 GB

The two runs produced byte-identical output — that is what "lossless speculation" means here: only the speed changes, never the text. The speculative run's larger TTFT is a one-time cost inside its first prefill, not a per-token cost.

Key numbers

Measured on an M4 Max (128 GB) / macOS 26.5.2, coremltools 9.0, compute units CPU_AND_GPU, single-token decode (S=1), one process per condition.

Metric	Value	Notes
Decode — ctx32k mode	90.51 ms/tok (~11 tok/s)	ladder function, ≤32K context
Decode — ctx128k mode	300.96 ms/tok (~3.3 tok/s)	ladder function, >32K context
Context length	131,072 tokens	full-attention layers; sliding layers ring at 1024
KV cache size (full)	~2.48 GB	full 8 layers 2.147 GB + sliding 40 layers 0.336 GB; fill-independent (same at 1 token or 128K)
MTP speculative decode	×1.05 – ×1.47, lossless	draft_len=4, regime-switch drafter; acceptance rates bit-identical to non-speculative
Copy-zero promotion (32K → 128K)	bit-exact, ~0.64 s	shared MLState; no re-load, no KV copy
Quantization	int4 matmul (AWQ p999) + int8 lm_head	block size 32
Core ML model bundle	6.71 GB	4 chunks + lm_head; two functions share one weight set

The speculative-decode multipliers span the range ×1.05 (verbatim) → ×1.27 (quoting) → ×1.47 (enumeration), all lossless, and match the standalone 32K bundle to within ±0.03.

How the Context Ladder works

The bundle is an MLProgram + MLState graph that co-hosts two functions — ctx32k and ctx128k — in one weight set. The full-attention KV state is declared once at the maximum width [1, 131072, 512] and shared by both functions, so moving from the 32K to the 128K regime is a copy-zero promotion: the same MLState buffer is simply read by a wider function. Promotion is bit-for-bit lossless (the zero-padded KV slots are masked with -inf, contributing exactly 0 to the softmax), so decode output is identical before and after promotion.

The result: short and medium conversations (≤32K tokens) decode at the fast 32K speed, and the model transparently steps up to 128K only when the context actually exceeds 32K — without a re-load and without a KV copy.

Prompt template

From manifest.json (promptPrefix / promptSuffix). These are not the stock Gemma <start_of_turn> markers — a loader that assumes them will silently produce a different token sequence. A single turn is BOS + promptPrefix + user text + promptSuffix:

<bos><|turn>user
{user text}<turn|>
<|turn>model
<|channel>thought
<channel|>

The model continues from there. For multi-turn, close the assistant's text with <turn|>\n and append the next user turn with the same prefix/suffix:

<bos><|turn>user
{turn 1 user}<turn|>
<|turn>model
<|channel>thought
<channel|>{turn 1 model}<turn|>
<|turn>user
{turn 2 user}<turn|>
<|turn>model
<|channel>thought
<channel|>

The exact strings, verbatim from manifest.json (no trailing newline after <channel|>):

"promptPrefix": "<|turn>user\n",
"promptSuffix": "<turn|>\n<|turn>model\n<|channel>thought\n<channel|>"

Use tokenizer.json from this repository, and prepend BOS if the tokenizer does not. With BOS, the sample prompt shown earlier encodes to 32 tokens.

Files

Total download ~10.2 GB. The four chunk_*.mlpackage folders plus lmhead.mlpackage are the 6.71 GB Core ML model bundle; the drafters, embedding table, and tokenizer are the remaining ~3.5 GB.

File	Role	Size
chunk_0_12.mlpackage	Transformer layers 0–11 (int4, MLProgram + MLState)	~1.4 GB
chunk_12_24.mlpackage	Transformer layers 12–23	~1.4 GB
chunk_24_36.mlpackage	Transformer layers 24–35	~1.4 GB
chunk_36_48.mlpackage	Transformer layers 36–47	~1.4 GB
lmhead.mlpackage	LM head / vocabulary projection (int8)	~1.0 GB
drafter_ring.mlmodelc	MTP draft model, 131072-width (used in the ctx128k regime)	~807 MB
drafter_ring32k.mlmodelc	MTP draft model, 32768-width (used in the ctx32k regime)	~807 MB
embed_fp16.bin	Token embedding table, fp16, shape [262144, 3840]	~1.9 GB
tokenizer.json	Tokenizer	~31 MB
tokenizer_config.json	Tokenizer configuration	~2 KB
manifest.json	Bundle manifest (architecture, context length, prompt template)	<1 KB
convert_config_ladder.json	Chain config (tensor shapes, quant recipe, ladder function windows)	~18 KB

The chunks and lm_head are shipped as .mlpackage only. The Swift loader compiles them to .mlmodelc on first load and caches the result, so no pre-compiled .mlmodelc copies are included. The two drafters are shipped as .mlmodelc (the loader detects those directly).

Requirements

• Apple Silicon Mac (M-series). This bundle is effectively Mac-only: the KV cache lives in an MLState that is GPU-resident, so the Apple Neural Engine (ANE) cannot be used and compute units must be CPU_AND_GPU.
• macOS 26 (Tahoe) or later (multifunction models + MLState function-specialized load).
• Memory: 12 GB at inference; 24 GB+ recommended. The full-attention KV state (2.1 GB) is resident from the very first token, and process RSS reaches ~10 GB. Note that the two sample runs above reported a peak memory figure of 18.0 GB (no speculation) / 20.5 GB (speculation on, which keeps a drafter resident) on a 128 GB machine — the peak is well above the steady-state RSS, so treat 24 GB as a practical floor rather than a comfortable one.
• Disk: ~11 GB for the downloaded bundle.
• GPU is required. There is no CPU-only or ANE fallback for this configuration.

Operational notes & troubleshooting

• First load compiles the model once. Each .mlpackage is compiled to .mlmodelc on first use and cached to disk; subsequent loads are warm. This step is cheap: ~1.2 s for the whole bundle on an M4 Max (about 0.3 s per chunk).
• "It hangs on the first token." The lm_head's GPU specialization runs before the first inference. A 2026-07-05 measurement put this at ~40 s per process; a 2026-07-21 re-check on macOS 26.5.x instead found it cached across processes — consistent with the Sample-output table above, where fresh processes reached the first token in 0.26 s (drafter off) / 3.83 s (drafter on). Budget the full ~40 s only for a first-ever run (no cache, or after clearing the E5RT cache); later runs are fast. Pre-warm at launch if you embed this in an app.
• Disk fills up, or loads get flaky after many runs. Core ML's E5RT cache (~/Library/Caches/com.apple.e5rt.e5bundlecache) grows by several GB per run. Deleting it is safe and recovers the space; it regenerates, and the next load is just slower.
• Out of memory / heavy swapping. Close other large apps. See the memory note in Requirements — below 24 GB this configuration is not realistic.
• "bundle has no drafter — running without speculation." drafter_ring.mlmodelc / drafter_ring32k.mlmodelc were not found next to manifest.json. Re-run the download.
• Loading your own way: point the loader at the bundle directory; manifest.json and convert_config_ladder.json describe the chain, prompt template, and ladder functions.

Limitations

• 128K mode is slow by design. 300.96 ms/tok (~3.3 tok/s) versus 90.51 ms/tok in 32K mode — a fixed wide-context tax of 300.96 / 90.51 ≈ 3.3× at int4. The ladder exists to avoid paying it below 32K; it does not make a genuinely 128K-deep context fast.
• Greedy argmax only. The lm_head emits an argmax token id, not logits. There is no temperature / top-k / top-p sampling — output is deterministic. Sampling would require re-converting with a logits head.
• Mac only. The stateful chain keeps its KV in GPU-resident MLState; CPU-only and ANE configurations are not usable with multiple MLState instances. There is no iOS build.
• Speculation's edge shrinks with context. The fixed-width verify tax means the largest wins are on short, structured prompts; free-form prose is near break-even (×1.05).
• First-ever run pays ~40 s of one-time GPU specialization (measured 2026-07-05). On macOS 26.5.x this is cached across processes (re-checked 2026-07-21) — later launches reach the first token in seconds — so budget ~40 s only for the first run or after clearing the E5RT cache.

Verification

These gates are listed because "it runs" and "it computes the same thing as the reference" are different claims, and this bundle makes the second one.

Correctness was gated against the PyTorch (Hugging Face) reference at the bit level. The 128K conversion passed all 13 correctness gates (RoPE range at pos 131071, schema/runtime, self-consistency, cross-generation int4/fp16 exact matches, ring-boundary, deep-feed to 40960); the Context Ladder passed all four ladder gates exact, including the copy-zero promotion gate (feed 2000 tokens in ctx32k, switch functions on the same state with zero read/write, then match the ctx128k reference greedy output with #diff=0). The rare near-tie cases (fp16 summation-order differences near the 32768 boundary) were attributed by an arbitration panel and never produced a hard failure — every near-tie had the oracle token within the chain's top-2.

License & attribution

• Base model: google/gemma-4-12B-it. The artifacts here are derived from Gemma 4 12B IT: the original files have been modified — the weights converted to a Core ML graph and quantized (int4 AWQ matmul / int8 lm_head).
• License: Apache 2.0. Google releases Gemma 4 under the Apache License 2.0, and these derivative artifacts are redistributed under the same license. The full text is in the bundled LICENSE.
• No source code here. This repository contains model weights and configuration only — no source code. The companion Swift package on GitHub is separately licensed under MIT.
• Built with Gemma.
• "Core ML", "MLState", and "Apple Neural Engine" are trademarks of Apple Inc.; "Gemma" is a trademark of Google LLC. Provided as-is, without warranty. Not affiliated with, sponsored by, or endorsed by Google or Apple.