FP8 KV-Cache on Intel® Arc™ Pro B70: 2× Capacity with strong Long Context Throughput Gains

FP8 KV-cache quantization transforms Intel® Arc™ Pro B70 into a significantly more capable long-context and high-concurrency inference platform by doubling effective KV-cache capacity while preserving accuracy for most tested models.

• Capacity: FP8 KV delivers a deterministic 2.0× KV-cache token-capacity increase over BF16 across all ten evaluated models (1B–72B) and TP configurations (TP=1, 2, 4)

• Concurrency: This directly translates into about 2.0× higher raw 4K session capacity at the same max_model_len. For example, Qwen2.5-14B-Instruct (TP=2) scales from 48.52 → 97.05 sessions, and Mistral-Small-24B-Instruct-2501 (TP=2) from 43.27 → 86.53 sessions, making FP8 KV a strong enabler for capacity-constrained deployments.

• Throughput: The benefit is strongest in long-context workloads (16K–32K), where serving becomes KV-bandwidth bound. The peak gain reaches +42.3% for Qwen2.5-14B-Instruct (TP=1, 16K), with strong gains for Qwen3-8B and Llama-3.1-8B, and material improvements for 70B-class TP=4 models at 32K

• Accuracy: FP8 KV is near-lossless for 8 of 10 models, with composite RULER deltas within ±0.013 at 4K; most remain stable across 16K and 32K as well. The two exceptions are important: DeepSeek-R1-Distill-Qwen-7B shows severe accuracy collapse and should run with BF16 KV, while Gemma-3-1B-IT shows significant degradation and should use FP8 KV only when capacity is the overriding constraint.

• SLA Concurrency: At the 5 s p99 TTFT / 200 ms p99 TPOT SLA gate, FP8 KV does not expand the SLA-passing window for short-context serving because the bottleneck is prefill latency, not KV eviction.

• FP8 vs TurboQuant positioning: FP8 KV should be treated as the default, low-friction built-in vLLM path for Intel® Arc™ Pro B70 deployments. It provides deterministic 2× KV capacity scaling through a built-in vLLM runtime option. TurboQuant should be positioned as the advanced, maximum-compression path for deployments that remain KV-memory-bound after FP8 or require more aggressive concurrency scaling.

1. Introduction

LLM serving on workstation-class GPUs is increasingly constrained by KV-cache memory, especially as customers move toward longer context windows and higher user concurrency. On the Intel® Arc™ Pro B70, the 32 GB memory budget must hold model weights, runtime overhead, and active KV-cache state. As a result, deployments can become memory-limited even when compute resources remain available.

Within this 32 GB memory envelope, KV-cache memory frequently becomes the dominant scalability limiter as context lengths and concurrent sessions increase. Unlike model weights, which remain fixed after model selection, KV-cache requirements grow directly with active tokens and user sessions. Consequently, KV-cache compression provides substantially higher practical value on workstation-class accelerators such as B70 than on larger-memory datacenter GPUs, where more memory headroom is available before KV residency becomes the limiting factor.

FP8 KV-cache quantization in vLLM directly addresses this bottleneck by storing KV tensors in FP8 rather than BF16 using --kv-cache-dtype fp8. Because FP8 uses half the storage per KV element, it doubles effective KV-token capacity within the same physical KV-cache memory allocation. This requires no model retraining, no architecture change, and no larger GPU memory footprint.

This blog evaluates FP8 KV across 10 models from 1B to 72B parameters and TP=1/2/4 deployments, using 4 customer-relevant axes: capacity, concurrency, throughput, and accuracy/SLA behavior. Dense GQA models show the best balance of capacity, long-context throughput gain, and near-lossless accuracy, while Gemma-family models and DeepSeek-R1-Distill-Qwen-7B require model-specific validation before production rollout.

Enable FP8 KV when memory capacity or long-context throughput is the bottleneck, validate model accuracy before production rollout, and treat FP8 KV as a selective optimization rather than a universal toggle.

2. FP8 KV-Cache Configuration in vLLM

FP8 KV-cache quantization is enabled in vLLM with a single runtime flag, --kv-cache-dtype fp8. The BF16 baseline uses the default KV-cache behavior, while the FP8 configuration stores key/value tensors in FP8 format, reducing the per-token KV footprint and doubling effective KV-token capacity within the same physical KV-cache memory budget. This requires no model retraining, no architecture change, and no additional GPU memory.

A representative Intel XPU launch configuration is:

VLLM_TARGET_DEVICE=xpu \
VLLM_MLA_DISABLE=1 \
VLLM_USE_V1=1 \
VLLM_ENGINE_READY_TIMEOUT_S=600 \
HF_HOME=/llm/models \
vllm serve Qwen/Qwen2.5-14B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.9 \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --enforce-eager \
  --block-size 64 \
  --port 8000

The key XPU-specific settings used in this study are VLLM_TARGET_DEVICE=xpu, VLLM_MLA_DISABLE=1, VLLM_USE_V1=1, --enforce-eager, and --block-size 64. The comparison isolates BF16 versus FP8 KV-cache behavior under otherwise matched serving conditions. Gemma 4 31B IT was benchmarked with the explicit vLLM flag --attention-backend FLASH_ATTN

FP8 KV is a simple one-flag optimization for improving B70 serving density, but it should be deployed as a validated model-specific setting—not as an unconditional global default.

3. Experimental Setup

3.1 Platform

Table 1. Hardware and software configuration.

3.2 Model Coverage

Models were selected to represent common customer deployments across Llama, Qwen, Gemma, Mistral, and DeepSeek families.

Table 2. Ten models spanning 1B–72B parameters.

3.3 Benchmark Suite

The benchmark suite measures five deployment-relevant outcomes: KV capacity, raw 4K concurrency, six-scenario throughput, long-context performance at 16K and 32K, and RULER accuracy/SLA behavior. The six-scenario throughput suite covers short decode, decode-heavy, mixed, high-load, long-prefill, and very-long-prefill workloads. The SLA sweep applies a strict online-serving gate of p99 TTFT ≤ 5 seconds and p99 TPOT ≤ 200 ms, while RULER evaluates long-context accuracy at 4K, 16K, and 32K.

4. GPU Memory Profile

vLLM allocates the KV-cache block pool at startup based on --gpu-memory-utilization and the memory remaining after model weights and runtime overhead. FP8 KV-cache quantization does not increase the physical KV-cache allocation in GiB; instead, it changes the packing density inside that allocation. BF16 KV stores each KV element in 2 bytes, while FP8 KV stores it in 1 byte, so the same physical KV-cache budget holds approximately double KV tokens. So the green segment in the chart — KV Cache (physical) — is the same GiB value whether you launch with --kv-cache-dtype auto (BF16) or --kv-cache-dtype fp8. The 2× token capacity multiplier is a packing gain within the same physical allocation, not additional memory.

Figure 1. GPU memory allocation at TP=1 and TP=2 with gpu_memory_utilization set to 0.90.

For smaller TP=1 models, the larger remaining KV budget translates into high raw session capacity. For 70B-class TP=4 deployments, each card has a much smaller per-card KV budget after weight sharding, making the 2× KV-token multiplier especially valuable. In the measured TP=4 memory profiles, the 70B-class models leave only about 7.78–8.55 GiB per card for KV cache, so doubling token density materially improves long-context and concurrency headroom. FP8 KV makes the existing 32 GB B70 memory envelope more productive. Its best understood as a KV packing optimization

Figure 2. GPU memory allocation at TP=4

5. Key Results

5.1 KV Cache Capacity and Raw Concurrency

FP8 KV-cache quantization delivers a deterministic doubling of KV-token capacity across the evaluated models and tensor-parallel configurations. This follows directly from the storage change: BF16 KV uses 2 bytes per KV element, while FP8 KV uses 1 byte.

Figure 3. BF16 vs FP8 KV Token Capacity, all 12 model×TP combinations sorted by capacity

A second important finding is that tensor parallelism and FP8 KV are complementary. Tensor parallelism reduces per-card weight residency and increases the memory available for KV cache, while FP8 doubles the number of tokens that fit within that budget. This makes FP8 KV a particularly effective systems optimization because it compounds cleanly with TP-based scaling without requiring larger GPUs or model changes.

At a fixed 4K context window, FP8 KV-cache quantization delivers an almost perfectly proportional increase in maximum concurrent sessions across all tested models on Intel® Arc™ Pro B70, mirroring the 2× KV-token capacity gain. FP8 KV converts limited B70 memory into higher serving density. At a fixed max_model_len=4096, the 2× KV-capacity gain translates directly into approximately 2× higher raw 4K session capacity. For example, Qwen2.5-14B-Instruct at TP=2 increases from 48.52 to 97.05 concurrent 4K sessions, while Mistral-Small-24B-Instruct-2501 at TP=2 increases from 43.27 to 86.53 sessions.

Figure 4. Maximum concurrent sessions at max_model_len=4096.

An important interpretation of this section is that these values represent the raw concurrency ceiling imposed by KV capacity, not the latency-qualified operating point. The paper separately shows that under strict SLA gates, some deployments become limited by TTFT or compute, not KV residency, which means the full 2× concurrency potential is most directly realized in capacity-bound serving rather than SLA-bound short-context serving.

5.2 Accuracy: RULER Benchmark Results

RULER results show that FP8 KV is near-lossless for 8 of 10 evaluated models. For the stable model group, composite score deltas remain within ±0.013, with six models staying within ±0.007. In the absence of repeated-run variance analysis, these small absolute changes should be interpreted as operationally negligible rather than formally tested for statistical significance. Models such as Llama-3.1-8B-Instruct, Qwen3-8B, Qwen2.5-14B-Instruct, Mistral-Small-24B-Instruct-2501, Llama-3.3-70B-Instruct, Qwen2.5-72B-Instruct, Gemma-4-31B-IT, and DeepSeek-R1-Distill-Llama-70B remain within the reported accuracy tolerance across evaluated contexts.

Two exceptions drive deployment guidance. DeepSeek-R1-Distill-Qwen-7B at TP=1 shows severe accuracy collapse, with RULER dropping from 0.5291 to 0.0891 at 4K and near-zero behavior at longer contexts - a cliff we attribute to its Qwen2-family base lacking QK-norm (so key-cache outlier channels overwhelm the single per-tensor scale) compounded by long chain-of-thought decoding, which propagates one early KV-induced error into a wrong final answer. Gemma-3-1B-IT also degrades materially, dropping from 0.2993 to 0.2082 at 4K and worsening at 32K; because it does have QK-norm, we attribute this milder loss not to key outliers but to its 5:1 local:global architecture, in which only the ~1-in-6 global layers carry long-range retrieval and thus have no redundant pathway to absorb the injected noise.

Figure 5. RULER composite accuracy deltas for BF16 versus FP8 KV across evaluated models and context lengths.

📌 Key Takeaway

FP8 KV should be treated as a validated model-specific optimization, not a universal global switch. It is production-suitable for the stable dense GQA class, but accuracy-sensitive models must be gated by evaluation before deployment.

5.3 Throughput: Modest at Short/Medium Context, Stronger at Long Context

In the six-scenario throughput suite, FP8 KV behaves primarily as a capacity optimization with modest throughput upside at short-to-medium context lengths. Single-card dense GQA models such as Llama-3.1-8B-Instruct and Qwen3-8B show positive throughput behavior, while larger TP=2 and TP=4 models are often closer to throughput-neutral in short/medium scenarios.

Figure 6. FP8 KV throughput impact across short-, medium-, and long-context serving scenarios.

The stronger performance story appears in long-context serving. At 16K–32K context lengths, results suggest workloads become increasingly KV-bandwidth bound, and FP8 KV delivers meaningful peak throughput gains for most dense non-Gemma models. The largest observed gain is Qwen2.5-14B-Instruct TP=1 at 16K, where peak output throughput increases by +42.3%. Qwen3-8B improves by +32.6% at 16K and +29.2% at 32K, while Llama-3.1-8B-Instruct improves by +32.2% at 16K and +25.4% at 32K. The 70B TP=4 models also show material gains at 32K

Figure 7. Long-context FP8 KV throughput gains at 16K and 32K context lengths across evaluated dense GQA models.

FP8 KV should be positioned not only as a memory-capacity feature, but also as a long-context throughput optimization for validated dense GQA models. Its performance value is strongest when KV movement and memory bandwidth become the bottleneck. Reducing KV precision halves KV storage requirements and reduces memory traffic associated with attention operations.

5.4 SLA Concurrency Sweep

The SLA sweep applies a strict online-serving gate of p99 TTFT ≤ 5 seconds and p99 TPOT ≤ 200 ms The test is intentionally stricter than the raw KV-capacity benchmark: every request uses 1024 input tokens and 512 output tokens with max_model_len = 4096, and concurrency is increased stepwise until the model violates the latency gate. This makes the sweep a practical complement to the raw capacity results in Sections 5.1 and 5.2, because it distinguishes between what fits in memory and what still meets an online serving SLA. Concurrency levels tested: 1 → 2 → 4 → 8 → 16 → 32 → 64 → 128 → 256.

The sweep halted after two consecutive failures. Both BF16 KV and FP8 KV configurations were measured for every model.

Figure 8. SLA-passing concurrency under p99 TTFT ≤ 5 s and p99 TPOT ≤ 200 ms gates across BF16 and FP8 KV configurations.

📌 Capacity Gains ≠ SLA Gains

The central finding is that FP8 KV almost never increases the SLA-passing concurrency ceiling, even though it doubles raw KV capacity and raw 4K concurrency elsewhere in the paper. In the summary results, 10 of 12 evaluated model × TP configurations show no change at all in SLA-passing maximum concurrency between BF16 and FP8.

6. Discussion

6.1 The Unified Picture: Accuracy + Throughput + Capacity

FP8 KV-cache quantization should be understood first as a memory optimization that changes system behavior depending on the serving regime, not as a universal performance accelerator. The most consistent result is the doubling of KV-capacity and raw concurrency gain, which directly increases the amount of active request state that can reside on a B70. This benefit is deterministic and architecture-independent.

However, the results show that capacity alone does not define deployment value. In short-context serving, most deployments are limited by prefill latency (TTFT) rather than KV residency, so the additional KV headroom does not translate into higher SLA-qualified concurrency. In contrast, in long-context workloads, where KV movement dominates, FP8 KV can produce meaningful throughput improvements.

FP8 KV is most valuable when 3 conditions align:

• KV memory is the bottleneck,
• The model tolerates FP8 precision,
• The workload is not dominated by short-context TTFT constraints.

When these conditions hold, FP8 KV converts directly into real serving efficiency gains; otherwise, its impact may be limited to raw capacity only.

6.2 FP8 KV vs TurboQuant: Complementary, Not Competing

Taken together with the FP8 results in this paper, the most defensible discussion framing is that FP8 KV and TurboQuant should be treated as complementary points on the same design spectrum, not as mutually exclusive alternatives. TurboQuant, offers much larger compression and concurrency gains than BF16 and explicitly targets memory-bound long-context serving; that makes it the more aggressive option when capacity headroom is the dominant objective. FP8 KV, by contrast, is the lighter-weight end of the spectrum in this paper: it is less aggressive, but it is also a simpler change in representation and, on the strong GQA class, still produces meaningful long-context throughput gains. The practical takeaway is that FP8 KV is the lower-risk, lower-compression step, while TurboQuant is the higher-compression, more transformative step for deployments that are decisively blocked by the KV memory wall.

Table 3. FP8 KV vs TurboQuant*

• Note: TurboQuant results are reproduced from prior published work and were not re-measured in this study.

6.3 Evaluating Native FP8 vs Dynamic FP8 KV on Llama 3.1 8B

The comparison between runtime-quantized FP8 KV (meta-llama/Llama-3.1-8B-Instruct) and a checkpoint-native FP8 model (nvidia/Llama-3.1-8B-Instruct-FP8) shows that the two approaches produce nearly identical serving behavior, despite fundamentally different implementation paths. In the runtime-quantized configuration, the standard meta-llama/Llama-3.1-8B-Instruct Hugging Face BF16 checkpoint is used, and FP8 is applied dynamically at serving time through --quantization fp8 --kv-cache-dtype fp8. In contrast, the nvidia/Llama 3.1 8B Instruct FP8 checkpoint is distributed in a pre-quantized FP8 format, with FP8 already baked into the weights and only --kv-cache-dtype fp8 required at runtime.

Table 4. Runtime-quantized FP8 KV versus checkpoint-native FP8 on Llama-3.1-8B.

Figure 9. Runtime-quantized FP8 KV versus checkpoint-native FP8 serving behavior on Llama-3.1-8B.

7. Deployment Recommendations

The results across capacity, throughput, SLA-constrained concurrency, and long-context behavior show that FP8 KV-cache quantization is not a one-size-fits-all toggle, but a highly effective optimization when applied in the correct deployment regime and model class. This section provides actionable guidance for deployment on Intel® Arc™ Pro B70.

Table 5. Deployment recommendations for FP8 KV-cache quantization on Intel® Arc™ Pro B70.

7.1 Enabling FP8 KV is Suitable For

FP8 KV-cache quantization can be enabled by default for dense GQA-based models including Llama 3.1 8B, Qwen3 8B, Llama 3.3 70B, DeepSeek R1 70B, Qwen2.5 14B, and Mistral Small 24B. Across these models, FP8 KV delivers a deterministic 2× increase in KV capacity and concurrency, along with consistent throughput improvements at long context (16K–32K) and stable latency characteristics under load.

Crucially, this model class maintains near-lossless accuracy under FP8 KV, making the optimization safe for production use. The benefits are most pronounced in multi-session and long-context serving, where KV memory pressure is the dominant bottleneck, but FP8 KV also provides efficiency gains in short-context workloads without introducing regressions.

7.2 Enable FP8 KV With Caveats

For Gemma 3 1B IT, FP8 KV delivers the expected 2× KV capacity and raw concurrency, but long context throughput regresses slightly (≈−4–6% at 16K–32K) and RULER accuracy drops materially, especially at 32K. FP8 KV remains a useful capacity enabler for memory limited deployments, but it should be gated by accuracy evaluation and is best reserved for short-context or strictly memory bound scenarios.

8. Conclusion

FP8 KV-cache quantization provides a deterministic 2.0× increase in KV capacity and concurrency across all evaluated models, establishing it as a low-friction, production-ready optimization for LLM serving.

The practical value of FP8 KV is determined by the deployment bottleneck, not by the capacity gain alone. When long contexts or high memory residency drive the workload, FP8 KV turns the fixed 32 GB B70 memory envelope into a more usable serving headroom by reducing KV-cache pressure and improving long-context efficiency. But when the workload is short-context and SLA-bound, the limiting factor shifts to TTFT or compute, so additional KV capacity may not translate into higher latency-qualified concurrency. The deployment implication is clear: FP8 KV is most valuable when memory pressure is the constraint, and less decisive when prefill latency defines the operating point. The accuracy results further show that FP8 KV should be deployed as a validated optimization, not as a blanket runtime default.

For workstation-class AI inference deployments, FP8 KV-cache quantization significantly extends the practical usefulness of the 32 GB Intel® Arc™ Pro B70 by enabling larger context windows, higher memory-resident concurrency, and more scalable local LLM serving without requiring additional GPU memory.

References

1. vLLM Documentation. KV Cache Quantization. Available online: docs.vllm.ai/en/stable/features/quantization/kv_cache.html
2. Intel Corporation. Intel® Arc™ Pro B70 Product Page. Available online: intel.com/content/www/us/en/products/sku/245797
3. Zandieh, A., Daliri, M., Hadian, M., & Mirrokni, V. (2026). "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate." ICLR 2026. arXiv: 2504.19874
4. Tao, C., et al. (2026). "TurboQuant KV Cache Compression for vLLM on Intel® Arc™ Pro B70." Intel Corporation. Enhancing long-context, high-concurrency LLM serving on a 32 GB workstation GPU: TurboQuant KV Cache
5. NVIDIA. NVIDIA ModelOpt FP8 Checkpoint: nvidia/Llama-3.1-8B-Instruct-FP8. Available on Hugging Face: huggingface.co/nvidia/Llama-3.1-8B-Instruct-FP8
6. Hsieh, C.-P., et al. (2024). "RULER: What's the Real Context Size of Your Long-Context Language Models?" COLM 2024. arXiv: 2404.06654
7. vLLM Project. vLLM Benchmark Serving Script. GitHub Repository: github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_serving.py
8. Micikevicius, P., et al. (2022). "FP8 Formats for Deep Learning." arXiv: 2209.05433
9. Yang, A., et al. (Qwen Team). (2025). "Qwen3 Technical Report." arXiv: 2505.09388
10. Gemma Team. (2025). "Gemma 3 Technical Report." arXiv: 2503.19786
11. Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., & Gholami, A. (2024). "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization." NeurIPS 2024. arXiv: 2401.18079
12. DeepSeek-AI. (2025). "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." Nature, 645, 633–638. arXiv: 2501.12948
13. Sun, M., Chen, X., Kolter, J. Z., & Liu, Z. (2024). "Massive Activations in Large Language Models." COLM 2024. arXiv: 2402.17762