Community Radar.

“2.5x faster inference with Qwen 3.6 27B using MTP - Finally a viable option for local agentic coding - 262k context on 48GB - Fixed chat template - Drop-in OpenAI and Anthropic API endpoints”

> _2026-05-07 edit: I have updated the hardware based recommendations with more focus on quality. I do not recommend q4_0 KV cache anymore beyond 64k context. After multiple rounds of testing with the different size quants, it appears **3 is the optimal number for draft speculative decoding.** The fastest and best quality quant is q8_0-mtp. F16, which I have also uploaded is actually better but ultra slow (6x slower than q8_0). Many keep saying 8bit is virtually lossless compared to 16bit, and 6bit almost as good as 8bit, but this is simply not true: time and time again I have noticed huge differences in quality and correctness between 8bit and 16bit versions of various models._ The recent PR to llama.cpp bring MTP support to Qwen 3.6 27B. This uses the built-in tensor layers for speculative decoding. None of the existing GGUF have it, as they need to be converted with this PR. I have tested it locally on my mac M2 Max 96GB, and the results are amazing: 2.5x speed increase, bringing it to 28 tok/s! I have converted the most useful quants and uploaded them to HF. Even if you are using apple silicon, you should use those instead of MLX. You can download them here: [https://huggingface.co/froggeric/Qwen3.6-27B-MTP-GGUF](https://huggingface.co/froggeric/Qwen3.6-27B-MTP-GGUF) This also includes 7 fixes I made to the original jinja chat template, due to vLLM specificity which broke in other tools: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates For now, you will need to compile your own version of llama.cpp to use them. It is fairly simple to do: ```bash git clone --depth 1 https://github.com/ggml-org/llama.cpp.git cd llama.cpp git fetch origin pull/22673/head:mtp-pr && git checkout mtp-pr cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release cmake --build build --target llama-cli llama-server ``` Then to start serving with the API endpoint, use a command similar to: ```bash llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \ --spec-type mtp --spec-draft-n-max 3 \ --cache-type-k q8_0 --cache-type-v q8_0 \ -np 1 -c 262144 --temp 0.7 --top-k 20 -ngl 99 --port 8081 ``` > **Vision currently crashes llama.cpp when used alongside MTP.** Reported 2026-05-06 in the current PR. That's it. Three optimizations in one command: | Flag | What it does | Impact | |---|---|---| | `--spec-type mtp --spec-draft-n-max 3` | Multi-Token Prediction (built into the model) | **2.5x faster** generation | | `--cache-type-k q8_0 --cache-type-v q8_0` | 8-bit KV cache (instead of 16-bit) | **Half the KV memory**, negligible quality loss | | `-c 262144` | 262K context window | Full native context on **48 GB Mac** with q8_0 KV | Adjust `-m`, `-c`, and `--cache-type-k/v` for your hardware, according to the tables below. Here are my recommendations based on your hardware: ### Apple Silicon Qwen3.6-27B is a hybrid model — only **16 of 65 layers** use KV cache (verified). The other 48 are linear attention (fixed 898 MiB recurrent state). KV memory is **~4× less** than a standard dense model. Runtimes that don't handle this (e.g. vllm) allocate KV for all 65 layers and show much higher memory usage. Numbers below are total memory used (model + KV cache + 0.9 GB recurrent state). Must leave **≥ 8 GB for macOS** (16 GB Macs excepted). | RAM | Quant | KV cache | Max context | Total used | Vision | |---|---|---|---|---:|---| | 16 GB | **`IQ2_M`** | `q8_0` | **42K** | **12.0 GB** | ✗ | | 24 GB | **`IQ3_M`** | | **46K** | **16.0 GB** | ✗ | | 24 GB | `IQ3_M` | `q8_0` | 91K | 16.0 GB | ✗ | | 32 GB | **`Q5_K_M`** | | **74K** | **24.0 GB** | ✗ | | 32 GB | `Q5_K_M` | `q8_0` | 147K | 24.0 GB | ✗ | | 32 GB | `Q4_K_M` | | 99K | 24.0 GB | ✓ | | 48 GB | **`Q6_K`** | | **262K** | **39.7 GB** | ✓ | | 48 GB | `Q8_0` | | 173K | 40.0 GB | ✓ | | 48 GB | `Q8_0` | `q8_0` | 262K | 37.3 GB | ✓ | | 64 GB | **`Q8_0`** | | **262K** | **45.8 GB** | ✓ | | 96 GB | **`Q8_0`** | | **262K** | **45.8 GB** | ✓ | ### NVIDIA GPU Same model memory as Apple Silicon, plus ~1 GB CUDA overhead. | VRAM | Quant | KV cache | Max context | Total VRAM used | Vision | |---|---|---|---|---:|---| | 12 GB | **`IQ2_M`** | `q8_0` | **11K** | **12.0 GB** | ✗ | | 16 GB | **`IQ3_M`** | | **30K** | **16.0 GB** | ✗ | | 16 GB | `IQ3_M` | `q8_0` | 60K | 16.0 GB | ✗ | | 24 GB | **`Q4_K_M`** | | **83K** | **24.0 GB** | ✓ | | 24 GB | `Q4_K_M` | `q8_0` | 167K | 24.0 GB | ✓ | | 24 GB | `Q5_K_M` | | 58K | 24.0 GB | ✗ | | 48 GB | **`Q6_K`** | | **262K** | **40.7 GB** | ✓ | | 48 GB | `Q8_0` | | 262K | 46.8 GB | ✓ | | 80 GB | **`Q8_0`** | | **262K** | **46.8 GB** | ✓ | > **16 GB Mac:** `IQ2_M`/q8_0 — 42K text-only. No vision. > > **24 GB Mac:** `IQ3_M` — 46K (f16 KV) or 91K (q8_0). Vision at 32–65K. > > **32 GB Mac:** `Q5_K_M` — 74K text-only (f16 KV), 147K (q8_0). `Q4_K_M` for vision at 99K. > > **48 GB Mac:** `Q6_K`/f16 KV — 262K with vision. `Q8_0`/q8_0 KV for 262K at higher model quality. > > **64 GB+ Mac:** `Q8_0`/f16 KV — 262K with vision. Maximum quality at practical speed. > > **12 GB GPU:** `IQ2_M`/q8_0 — 11K. Very limited, no vision. > > **16 GB GPU:** `IQ3_M` — 30K (f16 KV) or 60K (q8_0). No vision. > > **24 GB GPU:** `Q4_K_M` — 83K with vision (f16 KV). `Q5_K_M` — 58K text-only (f16 KV), 116K (q8_0). > > **48 GB+ GPU:** `Q6_K`/f16 KV — 262K with vision. `Q8_0` for max quality. Leave KV cache at f16 (blank column) for best quality. Use `q8_0` KV only when f16 doesn't give enough context. `q4_0` KV should not exceed 64K context. Vision adds ~0.9 GB for mmproj. macOS needs **≥ 8 GB** for itself (16 GB Macs excepted — use ~4 GB). You can increase available memory by raising the wired memory limit, e.g. for a 96 GB Mac: `sudo sysctl iogpu.wired_limit_mb=90112` (88 GB). NVIDIA reserves ~1 GB for CUDA.