Quantization is the technique of converting the weights of an AI model from high-precision floating-point formats (FP16, FP32) to lower-bit numeric formats (INT8, INT4, INT2), thereby significantly reducing the amount of RAM or VRAM needed to run the model. With appropriate quantization, you can run a 70B-parameter model on a GPU with 24GB VRAM instead of requiring ~140GB as in the original FP16 version.
How does Quantization work?
An AI model stores billions of parameters as real numbers (floating point). Each parameter in FP16 occupies 2 bytes, while each parameter in FP32 occupies 4 bytes. Quantization compresses these values into discrete levels of a data type with fewer bits.
For example, Q4_K_M quantization compresses each weight to 4-bit but still retains some important weights in their original FP16 format. The block quantization technique divides weights into blocks (usually 128 parameters), with each block having its own scale factor to reconstruct values more accurately during decompression.
Why does reducing the bit count reduce RAM and VRAM usage?
Each model weight in FP16 requires 16 bit = 2 byte. When switching to Q8, each weight requires only 8 bit = 1 byte. When switching to Q4, each weight averages only ~4.5 bit (with the Q4_K_M format). Here is the simple calculation:
- 7B-parameter model × 2 byte (FP16) = 14 GB
- 7B-parameter model × 1 byte (Q8) = ~7 GB
- 7B-parameter model × 0.5 byte (Q4) = ~3.7 GB
General formula: Required RAM/VRAM ≈ (Number of parameters × Bit-per-parameter) ÷ 8 + 1~2 GB overhead. The 1-2 GB overhead is for activation, KV cache, and temporary memory during inference.
Detailed comparison of FP16, FP8, Q8, Q6, Q5, Q4, Q3, Q2
Bits per parameter and formats
The most common quantization levels today, arranged from high to low:
- FP16 — 16 bits/parameter, uncompressed, original quality, fastest speed on GPU.
- FP8 (Float8) — 8 bits/parameter, NVIDIA's new format (H100/H200), retaining nearly the original FP16 quality.
- Q8 (INT8) — 8 bits/parameter, quality very close to FP16 (difference under 1%), saves 50% memory.
- Q6_K — ~6 bits/parameter, a good balance between quality and memory, with strong performance on both CPU and GPU.
- Q5_K_M — ~5.5 bits/parameter, retains most of the quality, suitable when higher accuracy than Q4 is needed.
- Q4_K_M — ~4.5 bits/parameter, currently the most popular format, offering a good balance of quality and memory.
- Q4_K_S — ~4.3 bits/parameter, a more memory-efficient variant of Q4_K_M, with slightly reduced quality.
- Q4_0 / Q4_1 — ~4 bits/parameter, legacy formats of llama.cpp, rarely used nowadays.
- Q3_K_M — ~3.5 bits/parameter, quality degradation begins to become clearly noticeable on complex tasks.
- Q2_K — ~2.5 bits/parameter, maximizes memory savings but quality degrades significantly, use only when necessary.
Characteristics of suffixes in GGUF format names
The GGUF format uses suffixes to distinguish compression methods:
- Q4_K_M: K-quant, medium — 4-bit compression with block size 128, retaining some weights in FP16, balancing quality–size.
- Q5_K_M: K-quant, medium — 5-bit compression, retaining more detail than Q4, suitable when high accuracy is needed.
- Q8_0: Native 8-bit compression, not using K-quant, nearly FP16, twice the size of Q4.
- Q4_K_S: K-quant, small — more space-efficient than M, with slightly lower quality than M.
- Q6_K: 6-bit K-quant, not compressed below Q5 but still smaller than Q8.
The K suffix means using block quantization (a block of 128 parameters), enabling more accurate weight reconstruction than compressing the entire set individually. The M (medium) suffix retains more metadata than S (small), providing better results at the same bit-level.
How do GGUF, GPTQ, AWQ, and EXL2 differ?
Each quantization format is suitable for different platforms and hardware:
- GGUF: A format from llama.cpp, created to run locally on CPUs and discrete GPUs (Apple Metal, CUDA). Files use the .gguf extension, with weights stored separately (previously .bin). Supports many Q2–Q8 levels and runs on diverse hardware. Suitable for: mainstream users running on a CPU or consumer GPU.
- GPTQ: Native quantization for NVIDIA GPUs, optimized row-wise. Commonly used with inference tools such as text-generation-webui and oobabooga. Offers good GPU performance. Suitable for: running on an NVIDIA GPU with sufficient VRAM.
- AWQ (Activation-aware Weight Quantization): An improvement over GPTQ that reduces quantization error based on activation magnitudes, providing higher quality at the same bit-level. Offers strong discrete GPU support and is gradually replacing GPTQ. Suitable for: those who want the best quality on a GPU with limited VRAM.
- EXL2: A format from EXLlama that supports non-uniform mixed-bit quantization (for example, 4-bit for some parts and 6-bit for others), and is highly optimized for VRAM. Suitable for: users with low-VRAM GPUs who want to maximize quality.
In summary: GGUF for CPU and flexibility, AWQ/GPTQ for optimized NVIDIA GPUs, EXL2 for extremely low VRAM on GPUs.
How does quantization affect model quality and speed?
Quality impact
Model quality gradually decreases at lower quantization levels. Based on practical experience from the community:
- FP16 → Q8: The difference is almost imperceptible in most use cases (less than 1% loss on benchmarks).
- Q8 → Q6: Minor reduction; most users will not notice it, making it suitable for most tasks.
- Q6 → Q5_K_M: Still maintains good quality, suitable for most workloads.
- Q5_K_M → Q4_K_M: The most popular balance. Significant memory reduction, with negligible quality degradation for models with 7B parameters or more.
- Q4_K_M → Q3: Clear degradation begins to appear in tasks requiring precise computation (code generation, mathematics, long reasoning).
- Q3 → Q2: Quality drops noticeably, hallucinations increase, and it should only be used when hardware constraints require it.
Speed implications
Quantization significantly improves inference speed on GPUs because it reduces the amount of data that needs to be processed each time. However, on CPUs, speed also depends heavily on the type of quantization:
- NVIDIA GPU: AWQ/GPTQ deliver the fastest speeds. GGUF running through the llama.cpp CUDA backend also performs well.
- CPU: GGUF is the optimal choice. llama.cpp's ongoing SIMD optimizations (AVX2, AVX512, NEON) make Q4 run significantly faster than Q8 on the same CPU.
- Apple Silicon: GGUF with the Metal backend delivers excellent integrated GPU performance at Q4.
How does Weight Quantization differ from KV Cache Quantization?
These are two compression techniques that operate at different stages of the inference process:
Weight Quantization compresses model weights — the numerical values learned during training. This accounts for most of the memory used when running the model. All of these weights are used during each inference. Weight compression reduces both VRAM and RAM.
KV Cache Quantization compresses Key-Value cache memory—intermediate data generated during token sequence processing. When a model processes a long sequence, KV memory grows linearly with the sequence length and number of layers. KV cache compression does not make the model lighter, but makes long-sequence inference more memory-efficient. Many newer GGUF models already have KV cache quantization built in.
In brief: Weight Quantization compresses the model (reduces file size and required RAM/VRAM). KV Cache Quantization compresses runtime data (reduces memory usage for long sequences without changing file size).
Estimated RAM/VRAM by quantization level
The table below estimates the memory required, including model weights and inference overhead (usually 1–2 GB). Values are calculated in GB, with actual overhead when running included.
| Quantization level | 7B parameters | 14B parameters | 32B parameters | 70B parameters |
|---|---|---|---|---|
| FP16 | ~16 GB | ~30 GB | ~66 GB | ~144 GB |
| FP8 | ~9 GB | ~17 GB | ~38 GB | ~80 GB |
| Q8_0 | ~9 GB | ~17 GB | ~38 GB | ~80 GB |
| Q6_K | ~6.5 GB | ~12 GB | ~26 GB | ~55 GB |
| Q5_K_M | ~5.5 GB | ~10 GB | ~22 GB | ~46 GB |
| Q4_K_M | ~4.5 GB | ~8.5 GB | ~18 GB | ~38 GB |
| Q4_K_S | ~4 GB | ~8 GB | ~17 GB | ~35 GB |
| Q3_K_M | ~3.5 GB | ~6.5 GB | ~14 GB | ~28 GB |
| Q2_K | ~2.8 GB | ~5 GB | ~10 GB | ~20 GB |
Note: This is an estimate based on the GGUF format. The actual figure may differ by 0.5–1 GB depending on the model and framework. If running multi-GPU (tensor parallel), each GPU only needs to handle 1 corresponding portion.
Suggestions for choosing the quantization level based on hardware
Choosing the quantization depends on three main factors: available memory capacity, the task being used, and GPU or CPU.
Based on VRAM/RAM capacity
- Under 6 GB VRAM (GTX 1060, RTX 3050, Intel Arc): Q4_K_M or Q3_K_M for 7B. 14B requires Q2_K. Prefer GGUF running via llama.cpp.
- 6–8 GB VRAM (RTX 3060, RTX 4060, RTX 2060 12GB): Q4_K_M for 7B or Q5_K_M if greater accuracy is needed. 14B requires Q3_K_M or Q4_K_S.
- 8–12 GB VRAM (RTX 3080 10GB, RTX 4070, RTX 4070 Ti): Q4_K_M for 14B, Q5_K_M or Q6_K for 7B. AWQ/GPTQ can be run comfortably.
- 12–24 GB VRAM (RTX 3090, RTX 4090, RTX 4070 Ti Super): Q4_K_M or Q5_K_M for 32B, Q6_K or Q8 for 14B.
- 24+ GB VRAM (RTX 4090 24GB, A100, H100): Q4_K_M or Q5_K_M for 70B, Q8/FP8 for 32B, comfortably use large models.
By task type
- Regular chat, creative writing, summarization: Q4_K_M is sufficient; no need for higher.
- Code generation, programming: Q5_K_M or Q6_K — the model needs to accurately preserve syntax and logic.
- Long reasoning, mathematics, complex analysis: Q6_K or higher, ideally Q8 if VRAM allows.
- Quick testing, prototype: Q2_K to quickly test a new model, then move up to Q4.
By platform
- CPU-only: GGUF Q4_K_M is the sweet spot between speed and quality. Q5 and above will be significantly slower on CPU.
- NVIDIA GPU (CUDA): AWQ or GPTQ Q4_K_M for optimal VRAM usage and the fastest speed.
- Apple Silicon (Mac): GGUF with the Metal backend; Q4_K_M runs very smoothly on M-series chips with unified memory.
Conclusion
Quantization is the foundational technique that makes local AI feasible on mainstream hardware. Q4_K_M is the safest choice for most cases — light enough to run on a GPU with 6–8GB VRAM, and sufficiently high-quality for most everyday tasks. If VRAM allows, Q5_K_M or Q6_K provides an almost lossless experience. Only drop down to Q3 or Q2 when memory is truly limited — and accept the trade-off in output quality.
Frequently Asked Questions
Does quantization make a model less intelligent?
Reducing quantization from Q8 to Q4_K_M has almost no effect on model capabilities for models from 7B upward. From Q3 downward, you will start to see more hallucinations and less accurate responses in tasks requiring complex reasoning.
Is running GGUF on a CPU fast?
Speed depends on the CPU and quantization level. Q4_K_M on a modern 8-core CPU achieves approximately 5–15 tokens/second with a 7B model, which is sufficient for chat. Q5 and above will be significantly slower. If speed is needed, prioritize running on a GPU.
Which format should I use — GGUF, GPTQ, or AWQ?
Choose GGUF if running on CPU or Mac, or if you want high flexibility. Choose AWQ or GPTQ if running on an NVIDIA GPU — AWQ for higher quality at the same bit-level, GPTQ for broader compatibility with frameworks.
Can a 6GB VRAM GPU run a 14B model?
Yes, but Q2_K or Q3_K_M quantization is required. A 14B model at Q3_K_M requires around 6.5 GB VRAM. Quality will decrease noticeably compared to the original, making it suitable for experimentation or use when there are no other options.
Does KV Cache Quantization reduce VRAM usage?
KV Cache Quantization does not make the model file smaller; it only reduces the memory required when performing inference on long sequences. If you want to reduce overall VRAM usage, weight quantization (Q4, Q5…) is the main solution you should consider.

