What Happens When Your AI Model Doesn't Fit in VRAM?

Before you start

You just pulled a 70B parameter model and your GPU has 24GB of VRAM. You launch it and get a CUDA out of memory error. This article explains what happens under the hood when a model doesn't fit in VRAM, how to measure the actual memory usage, and what you can do about it.

You will need a Linux machine with an NVIDIA GPU, Python 3.10 or later, and the NVIDIA driver installed. We will use PyTorch and the transformers library for examples.

• NVIDIA GPU with at least 8GB VRAM (we will test with 24GB but the commands work on any)
• Python 3.10+ with pip
• CUDA toolkit matching your driver (check with nvidia-smi)

Step 1: Check your GPU memory

First, confirm your GPU and its available memory. Run nvidia-smi to see the total VRAM and current usage.

nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv

• The output should show something like: RTX 3090, 24576 MiB, 24576 MiB
• If you see less free memory than total, other processes are using it. Close them or wait.

Step 2: Load a model that exceeds VRAM

To see what happens, we will try to load a model that is too big. We will use a 7B parameter model in full precision (fp32), which needs about 28GB of VRAM. On a 24GB card, this will fail.

Create a Python script and run it. The error message will tell you exactly what went wrong.

python -c "from transformers import AutoModelForCausalLM; m = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-2-7b-hf')"

• This will likely produce a CUDA out of memory error.
• The error says: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 23.65 GiB total capacity; 21.34 GiB already allocated; ...)
• The model needs more memory than the GPU has, so the allocation fails.

Step 3: Quantization to the rescue

Quantization reduces the precision of the weights, making the model smaller. For example, 4-bit quantization can shrink a 7B model to about 4GB. We can use the bitsandbytes library to load a quantized model.

pip install bitsandbytes transformers accelerate

• Then load the model with load_in_4bit=True.
• This trades a little accuracy for a huge memory saving.

Step 4: Measure memory usage

After loading a quantized model, we want to see how much VRAM it actually uses. We can use torch.cuda.memory_allocated and torch.cuda.max_memory_allocated to get the current and peak usage.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-2-7b-hf', load_in_4bit=True, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')
inputs = tokenizer('Hello', return_tensors='pt').to('cuda')
output = model.generate(**inputs, max_new_tokens=10)
print(f"Allocated: {torch.cuda.memory_allocated()/1024**3:.2f} GB")
print(f"Peak: {torch.cuda.max_memory_allocated()/1024**3:.2f} GB")

• The output will show something like: Allocated: 4.12 GB, Peak: 5.01 GB
• This is far less than the 28GB needed for fp32.

What happens when it doesn't fit: the mechanics

When a model exceeds VRAM, the GPU cannot allocate the required memory. The CUDA driver returns an error, and the program crashes. But there are two other things that can happen: swapping and out-of-core computation.

Swapping means parts of the model are moved to system RAM, which is much slower. The GPU has to fetch data over the PCIe bus, which is orders of magnitude slower than VRAM. This can make inference painfully slow, but it might work.

Out-of-core computation is a technique where you manually manage the movement of tensors between CPU and GPU. Libraries like accelerate and deepspeed offer offloading, but you pay a huge performance penalty.

• CUDA out of memory: the program crashes immediately.
• Swapping: the model runs but is extremely slow (seconds per token).
• Offloading: you keep the model on CPU and copy activations to GPU, still slow.

Step 5: Try offloading with accelerate

Accelerate lets you offload layers to the CPU when they are not in use. This allows you to run a model that is larger than VRAM, but at a much slower speed. Here is how to enable CPU offloading.

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-2-7b-hf', device_map='auto', max_memory={0: '10GiB', 'cpu': '32GiB'})

• The max_memory argument tells accelerate to use at most 10GB of VRAM and 32GB of RAM.
• This model will run, but each token generation might take seconds.

Step 6: Use the right tool for the job

For production, you want to choose a model that fits comfortably in VRAM. A good rule of thumb is to leave 20% headroom for activations and overhead.

Here is a quick reference for different model sizes and the VRAM they need in 4-bit and 8-bit quantization.

• 7B model: 4-bit ~4GB, 8-bit ~7GB, fp16 ~14GB
• 13B model: 4-bit ~7GB, 8-bit ~13GB, fp16 ~26GB
• 70B model: 4-bit ~35GB, 8-bit ~70GB, fp16 ~140GB

Recommended setup: a copy-paste script

Here is a complete script that loads a model in 4-bit, runs a prompt, and prints the memory usage. Copy this into a file and run it.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = 'meta-llama/Llama-2-7b-hf'
model = AutoModelForCausalLM.from_pretrained(model_name, load_in_4bit=True, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = 'The capital of France is'
inputs = tokenizer(prompt, return_tensors='pt').to('cuda')
output = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(output[0], skip_special_tokens=True))

print(f"Allocated VRAM: {torch.cuda.memory_allocated()/1024**3:.2f} GB")
print(f"Peak VRAM: {torch.cuda.max_memory_allocated()/1024**3:.2f} GB")

Verification checklist

Run these checks to confirm your setup is working correctly.

• nvidia-smi shows your GPU with enough free memory.
• The Python script runs without a CUDA error.
• The printed memory usage is less than your GPU's total VRAM.
• The model generates text that makes sense.
• If you get an out of memory error, reduce the model size or use quantization.

Troubleshooting

Common issues and how to fix them.

• Error: 'bitsandbytes' not installed. Run pip install bitsandbytes.
• Error: 'accelerate' not installed. Run pip install accelerate.
• If you get a CUDA out of memory even with 4-bit, your GPU might not have enough RAM. Try a smaller model.
• If the model is slow, check if CPU offloading is happening. Reduce the max_memory to force more offloading or use a smaller model.

FAQ

Quick answers to common questions.

• Q: Can I run a 70B model on a 24GB GPU? A: Yes, with heavy quantization and offloading, but it will be very slow.
• Q: What is the difference between 4-bit and 8-bit quantization? A: 4-bit uses less memory but may lose more accuracy.
• Q: Does quantization affect output quality? A: It can, but for many tasks the difference is negligible.
• Q: What is the best way to run a model that doesn't fit? A: Use the largest model that fits in VRAM with 4-bit quantization, and offload only if necessary.

Next action

Run the full script above with a model that fits your GPU. Then try a larger model with quantization and offloading. See the difference in speed and memory. This hands-on experience will teach you more than any guide.