OH-Tech

How to Calculate RAM/VRAM for Running Local AI: Detailed Formula 2025

OH-TECH9 min read
How to Calculate RAM/VRAM for Running Local AI: Detailed Formula 2025

How much RAM/VRAM do you need to run a fully computable local dip AI model. Total required memory = model weights + KV cache + system overhead, with each component having its own formula. This article provides detailed formulas and practical examples for the 8B, 32B and 70B model series at different context levels.

What RAM/VRAM is used for when running local AI

When running an AI model on a local machine, RAM and VRAM (graphics card memory) are divided into three main parts:

  • Model weights (model weights): all parameters of the AI model need to be loaded into memory to perform inference. This is usually the component that occupies the most memory.
  • KV Cache: cache memory that stores Key-Value from previous inference passes, helping the model continue generating new tokens faster without having to recalculate from scratch.
  • Runtime overhead: memory for intermediate layers, activations, buffers, and the operating system.

Among these, model weights and KV cache are the two components whose sizes can be accurately calculated using mathematical formulas. Runtime overhead typically accounts for around 1-3 GB depending on the model and framework.

How to Calculate Model Weights Size Based on Parameters and Quantization

Model weights size depends on the total number of model parameters and the storage format (quantization).

Formula:

Weights size (GB) = Number of parameters × Bytes per parameter

Bytes per parameter for common quantization formats:

  • FP16 (Float16): 2 bytes/parameter
  • INT8 (Q8): 1 byte/parameter
  • INT4 (Q4_K_M): approximately 0.5-0.6 bytes/parameter (depending on the compression method)
  • INT4 (Q4_0): exactly 0.5 bytes/parameter

Example with a 7-8 billion parameter model (7B/8B):

  • FP16: 7 billion × 2 = 14 GB
  • Q8: 7 billion × 1 = 7 GB
  • Q4_K_M: 7 billion × 0.55 ≈ 3.85 GB

Example with a 70 billion parameter model (70B):

  • FP16: 70 billion × 2 = 140 GB
  • Q4_K_M: 70 billion × 0.55 ≈ 38.5 GB

The lower the quantization format (Q4 < Q8 < FP16), the less storage space required, but the lower the inference quality as well. Q4_K_M is a popular choice because it strikes a good balance between storage capacity and accuracy.

What Is KV Cache and Why Longer Context Uses More RAM

KV Cache is the mechanism for storing intermediate results (Key tensor and Value tensor) from the attention heads during inference. When a model processes a token sequence, it needs to reference all previously processed tokens through the attention mechanism. KV Cache stores these values so they do not have to be recalculated when generating new tokens.

As the context length increases, the amount of data that needs to be stored in the KV Cache grows linearly. A 128K-token sequence requires 16 times more KV Cache than an 8K-token sequence. This explains why models that support long context (32K, 128K) require significantly more VRAM than conventional models (8K).

KV Cache also depends on the model's attention architecture. Models using Grouped Query Attention (GQA) use less KV Cache than traditional Multi-Head Attention (MHA) because multiple query heads share a single pair of KV heads.

KV Cache Calculation Formula and the Impact of MHA/GQA/MQA

KV Cache calculation formula:

KV Cache (bytes) = 2 × num_layers × num_kv_heads × head_dim × context_length × bytes_per_param

Where:

  • 2: coefficient for both Key and Value
  • num_layers: number of attention layers in the model
  • num_kv_heads: number of KV heads (different from query heads)
  • head_dim: size of each head (usually 128)
  • context_length: maximum number of tokens in the context
  • bytes_per_param: 2 bytes for FP16, 1 byte for INT8

Compare attention architectures:

  • MHA (Multi-Head Attention): each head has its own KV. E.g.: 32 query heads, 32 KV heads. Largest KV Cache.
  • GQA (Grouped Query Attention): multiple query heads share 1 pair of KV heads. E.g.: 32 query heads, 8 KV heads. Saves ~4 times more KV Cache than MHA.
  • MQA (Multi-Query Attention): all query heads share 1 pair of KV heads. Offers the greatest savings but lower quality.

Specific example with Llama 3.1 8B:

Architecture: 32 layers, 8 KV heads, head_dim 128, running in FP16 (2 bytes)

KV Cache for 1 token = 2 × 32 × 8 × 128 × 1 × 2 = 131,072 bytes ≈ 0.125 MB

KV Cache for 8K tokens = 0.125 MB × 8,000 = 1,000 MB = 1 GB

KV Cache for 128K tokens = 0.125 MB × 128,000 = 16 GB

With the MHA architecture and 32 KV heads, the KV Cache for 128K will increase to 64 GB (4 times as much).

Summary of the formula for calculating the total RAM/VRAM required

General formula:

Total RAM/VRAM = Weights + KV Cache + Overhead

Where:

  • Weights: Number of parameters × Bytes/parameter (according to quantization)
  • KV Cache: 2 × layers × kv_heads × head_dim × context × 2 (FP16)
  • Overhead: typically 1-3 GB for the operating system, framework, and activation

Important note: KV Cache is stored in the same memory as the weights. If you want to run with a 128K context but do not have enough VRAM, the system will automatically swap the KV Cache out to system RAM or reduce the actual context, significantly slowing down inference speed.

For optimization, many frameworks such as Ollama, LM Studio, or vLLM support KV Cache offloading or dynamic splitting between VRAM and RAM, but this depends on the specific hardware and configuration.

VRAM calculator for models 8B, 32B, 70B at context 8K-128K

General assumptions: Typical GQA architecture (8 KV heads for 8B, corresponding numbers for 32B/70B), head_dim 128, overhead 2 GB, weights running Q4_K_M.

7-8B (32 layers, 8 KV heads):

  • Context 8K: Weights ~4.9 GB + KV ~1 GB + Overhead ~2 GB = ~8 GB
  • Context 32K: Weights ~4.9 GB + KV ~4 GB + Overhead ~2 GB = ~11 GB
  • Context 128K: Weights ~4.9 GB + KV ~16 GB + Overhead ~2 GB = ~23 GB

30-34B (60 layers, 8 KV heads):

  • Context 8K: Weights ~18 GB + KV ~2 GB + Overhead ~2 GB = ~22 GB
  • Context 32K: Weights ~18 GB + KV ~8 GB + Overhead ~2 GB = ~28 GB
  • Context 128K: Weights ~18 GB + KV ~32 GB + Overhead ~2 GB = ~52 GB

70B (80 layers, 8 KV heads, Q4_K_M):

  • Context 8K: Weights ~39 GB + KV ~4 GB + Overhead ~2 GB = ~45 GB
  • Context 32K: Weights ~39 GB + KV ~16 GB + Overhead ~2 GB = ~57 GB
  • Context 128K: Weights ~39 GB + KV ~64 GB + Overhead ~2 GB = ~105 GB

Comment: KV Cache becomes the deciding factor when running long context. With a 70B model at 128K context, the KV Cache (~64 GB) is even larger than the weights (~39 GB) when using Q4_K_M.

Common mistakes and RAM recommendations

Mistake 1: Only counting the weights and forgetting the KV Cache

Many people think that model 7B Q4 only needs 5 GB of VRAM. In reality, if you need to run with an 8K context, you need at least an additional 1 GB for the KV Cache, bringing the total to ~7-8 GB. Forgetting this leads to OOM (Out of Memory) when the model starts generating long token sequences.

Mistake 2: Not accounting for system and framework overhead

CUDA kernels, the PyTorch memory allocator, and intermediate buffers consume an additional 1-3 GB of VRAM. Some frameworks, such as HuggingFace's transformers, also require additional memory for activation tensors during the forward pass.

Mistake 3: Not understanding how quantization affects the KV Cache

Quantization only applies to the model's weights. KV Cache is usually kept in FP16 or FP32 in most implementations to ensure attention accuracy. This means that even if you use Q4 for the weights, KV Cache still needs to be computed in FP16.

RAM/VRAM recommendations based on your needs:

  • General use, short context (4-8K): an 8-12 GB card (RTX 3060, RTX 4060) is sufficient for a 7-8B Q4 model
  • Work, medium context (8-32K): a 16-24 GB card (RTX 3090, RTX 4090, RTX 4070 Ti Super) for a 13-34B Q4 model
  • Research, long context (64-128K): a 24-48 GB card (A100, A6000 Ada) or multiple SLI/NVLink cards for a 70B+ model
  • High-speed inference (large batch, streaming): an additional 20-30% VRAM is required compared with the theoretical calculation

In addition to VRAM, system RAM is also important. If VRAM is insufficient, the system will use RAM as swap, but performance will be much slower. Ensure system RAM is at least twice the VRAM if you plan to run large models.

Conclusion

Calculating RAM/VRAM for local AI follows a specific formula: Total = Weights + KV Cache + Overhead. Here, weights depend on the number of parameters and quantization (Q4/Q8/FP16), while KV Cache depends on the model architecture, number of KV heads, and context length. As context increases, KV Cache can exceed weights in terms of required capacity, especially with models of 70B or larger. Calculate carefully before choosing hardware to avoid unwanted OOM.

Frequently asked questions

How much VRAM does the 7B Q4 model need for an 8K context?

With a 7B model running Q4_K_M (~4.9 GB weights), an 8K context requires an additional ~1 GB KV Cache and ~2 GB overhead, totaling approximately 7-8 GB VRAM.

Does Quantization affect KV Cache?

No. Quantization only compresses the model's weights. KV Cache is usually kept in FP16 in most frameworks to ensure the accuracy of the attention mechanism.

Why does a longer context use more VRAM?

KV Cache increases linearly with context length. The formula is 2 × layers × kv_heads × head_dim × context × 2 bytes. Each additional token requires corresponding KV memory across all layers.

Can an 8GB card run a 13B model?

With a short context (2-4K), a 13B Q4 model can run on 8 GB VRAM (weights ~7 GB + small KV). However, a context longer than 8K will require more than 8 GB.

How much VRAM does GQA save compared with MHA?

GQA reduces KV Cache by the ratio num_kv_heads / num_query_heads. For example, Llama 3.1 8B has 32 query heads and 8 KV heads, using 4 times less KV Cache than full MHA.