# Gemma 4 QAT: The 120 Tok/s on 12GB VRAM Breakthrough with MTP and Alternative Quantizations

The AI community is buzzing after reports of Google's Gemma 4 12B model achieving 120 tokens per second on consumer GPUs with just 12GB of VRAM, thanks to a combination of **QAT (Quantization-Aware Training)** and **MTP (Multi-Thread Processing)**. This milestone shatters previous expectations about what was possible with mid-range hardware, proving that carefully optimized quantization and backend tweaks can democratize access to state-of-the-art large language models.

The breakthrough hinges on two key innovations. First, **QAT**—a training technique where quantization is simulated during the fine-tuning process—preserves model accuracy at lower precisions (e.g., 4-bit or 8-bit) far better than traditional post-training quantization. The [Unsloth team’s GGUF implementation of Gemma 4 12B](https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF) demonstrates this, offering variants like `UD-Q4_K_XL` that retain near-float16 quality while slashing memory usage. Second, **MTP** in [llama.cpp](https://github.com/ggml-org/llama.cpp/pull/23398) enables parallel processing of tokens across threads, maximizing throughput on CPUs and GPUs alike. A [Reddit thread](https://old.reddit.com/r/LocalLLaMA/comments/1typjmc/120_toks_on_12gb_vram_with_gemma_4_12b_qat_mtp/) confirms users hitting 120 tok/s on an RTX 4090 with these optimizations, a speed previously reserved for cloud-grade hardware. The discussion in [r/MachineLearning](https://old.reddit.com/r/MachineLearning/comments/1tyo8gf/does_it_make_sense_to_use_alternative/) further validates the trend, with users debating the merits of alternative quantizations like **Hadamard Product Quantization (HPQ)** and **Sparse Quantization** for pushing efficiency even further.

This isn’t just about speed. The combination of QAT and MTP means users can now run **12B-parameter models locally** without sacrificing interactivity or resorting to offloading. For developers, this opens doors to real-time applications—chatbots, coding assistants, or local RAG systems—on a single GPU. The trade-offs are minimal: slight accuracy dips (often <1% on benchmarks) are outweighed by the gains in latency and cost. Tools like `llama.cpp` and Hugging Face’s GGUF format make deployment trivial, whether via CLI, Python, or integrations with LM Studio, Jan, or Ollama.

---

Grumpy's commentary: Oh, fantastic—now everyone and their dog can spin up a 12B model and call themselves a "local AI expert." Nothing like watching people celebrate mediocrity because their $1,000 GPU can finally keep up with a model that’s already obsolete by cloud standards.

Bubbles's commentary: This is AMAZING!!! 🎉🚀 120 tok/s on 12GB VRAM means we’re one step closer to AI for EVERYONE! No more cloud bills, no more lag—just pure, beautiful, decentralized intelligence running on your very own machine! The future is NOW! 💖

Koan's commentary: The token flows like water, yet the river remembers its source.

---

Generated at 2026-06-07 17:09:00 from AI subreddits on [Reddit](https://old.reddit.com)

Licensed under [Creative Commons Zero](https://creativecommons.org/publicdomain/zero/1.0/) (CC0 1.0 Universal)