Why AI Models Keep Getting Bigger While GPUs Don't

The problem: your GPU cannot hold the model

You just downloaded the latest open-weight model. It is 70 billion parameters, and the download page says it needs 140 GB of VRAM. Your GPU has 24 GB. The error is immediate: CUDA out of memory. This is not a bug. It is the result of a trend that has been accelerating for years: model sizes grow faster than GPU memory.

In this article, I will explain the mechanics behind that trend, then show you practical ways to run large models on consumer hardware. You will learn about memory math, quantization, offloading, and a setup that works today.

The memory math: why 70B needs 140 GB

Every parameter in a neural network is a number. In full precision, that number is a 32-bit float, which takes 4 bytes. So a 70B model needs 70 billion times 4 bytes, which is 280 GB just for weights. Nobody runs that. Most people use 16-bit precision, which is 2 bytes per parameter, giving 140 GB. That is still far beyond consumer GPUs.

The trend is simple: model sizes double every few years, but GPU VRAM does not. The NVIDIA RTX 3090 had 24 GB in 2020. The RTX 4090 also has 24 GB in 2022. The RTX 5090, released in 2025, still has 32 GB. Meanwhile, models went from 7B to 70B to 405B. The gap is structural.

Why do models get bigger? Because larger models perform better on complex tasks, and researchers have found that scaling laws hold: more parameters and more data lead to predictable improvements. But hardware vendors have not kept up, partly because of cost and power limits.

Quantization: the practical workaround

Quantization reduces the number of bits used to store each weight. Instead of 16-bit floats, you can use 8-bit integers, 4-bit integers, or even 3-bit. This directly shrinks the memory footprint. A 70B model at 4-bit takes about 35 GB, which fits on a 48 GB workstation GPU or can be split across two 24 GB cards.

The trade-off is a small loss in accuracy. In practice, 4-bit quantization is often good enough for chat and code generation. You can measure the impact yourself with a simple script.

Here is a minimal Python script to compare memory usage of a model at different precisions using the transformers library. Install with pip install transformers torch.

Run this script to see the memory difference:

pip install transformers torch

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-2-7b-hf"

# Load in 16-bit
model_16 = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)
mem_16 = model_16.get_memory_footprint() / (1024**3)
print(f"16-bit memory: {mem_16:.2f} GB")

# Load in 8-bit (requires bitsandbytes)
model_8 = AutoModelForCausalLM.from_pretrained(model_id, load_in_8bit=True)
mem_8 = model_8.get_memory_footprint() / (1024**3)
print(f"8-bit memory: {mem_8:.2f} GB")

del model_16, model_8
torch.cuda.empty_cache()

• Expected output: 16-bit uses about 13.5 GB, 8-bit uses about 7.1 GB.
• If you get a bitsandbytes error, install it with pip install bitsandbytes.
• For 4-bit, use load_in_4bit=True instead.

Step 1: Check your GPU memory

Before you try to run a big model, know your hardware limits. Use nvidia-smi to see total VRAM and current usage.

Run this command:

nvidia-smi --query-gpu=name,memory.total,memory.used,memory.free --format=csv

• Note the memory.total value. That is your ceiling.
• If you have multiple GPUs, you can split a model across them, but each GPU must have enough memory for its shard.

Step 2: Pick a model that fits

Model sizes come in standard configurations. A 7B model at 4-bit fits in 4 GB. A 13B model at 4-bit fits in 8 GB. A 70B model at 4-bit needs 35 GB. Use this rough formula: memory in GB is approximately parameters in billions times 0.5 for 4-bit, times 1 for 8-bit, times 2 for 16-bit.

Here is a table of common models and their approximate memory at 4-bit:

Model          Parameters   ~4-bit VRAM
Llama 3.2 3B   3B           1.5 GB
Llama 3.1 8B   8B           4 GB
Mistral 7B     7B           3.5 GB
Llama 3.1 70B  70B          35 GB
Mixtral 8x7B   47B          24 GB

Step 3: Use Ollama for painless local inference

The easiest way to run a quantized model locally is Ollama. It handles quantization, offloading, and the API for you. Install it, then pull a model, and run it.

Here is the exact command sequence:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama run llama3.1:8b "Explain why GPU memory is a bottleneck for AI in one sentence."

• If you have a smaller GPU, start with llama3.2:3b instead.
• Ollama automatically uses the highest quantization that fits your VRAM.
• To see what is running, use ollama ps.

Step 4: Offload layers to CPU when VRAM runs out

If the model does not fit entirely in VRAM, you can offload some layers to system RAM. This is slower, but it lets you run models that would otherwise crash. Ollama does this automatically, but you can control it with the OLLAMA_NUM_GPU environment variable.

Set it to 0 to run entirely on CPU, or a positive number to use that many layers on GPU. The default is -1, which means auto.

Here is how to force CPU offloading:

OLLAMA_NUM_GPU=0 ollama run llama3.1:8b "Hello"

• Expect a significant slowdown. CPU inference is 10-50x slower than GPU.
• A better approach is to use a smaller model that fits fully in VRAM.

Recommended setup for a 24 GB GPU

If you have a 24 GB GPU, here is what I would do. Use Ollama with a 70B model at 4-bit, but be prepared to offload. Alternatively, use a 32B model at 4-bit, which fits comfortably and runs fast. For most tasks, a 32B model is a sweet spot.

Here is a copy-paste starter:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a 32B model (requires ~18 GB VRAM at 4-bit)
ollama pull qwen2.5:32b

# Run it with a system prompt
ollama run qwen2.5:32b "You are a senior software engineer. Answer concisely."

• Model: qwen2.5:32b is a strong open model.
• If you have 16 GB, use qwen2.5:14b.
• For 8 GB, use qwen2.5:7b.

Verify it worked: measure tokens per second

After you run a model, measure the actual speed to see if it is usable. Ollama has a built-in timing output. Run a prompt and look at the eval rate.

Here is how to get a clean measurement:

ollama run qwen2.5:32b "Write a Python function to compute Fibonacci numbers." --verbose

• Look for the line that says eval rate. It shows tokens per second.
• A rate above 20 tokens per second feels interactive. Below 5 is painful.
• If it is too slow, reduce the model size or increase offloading.

Troubleshooting common errors

CUDA out of memory. This means the model does not fit. Use a smaller model or enable offloading. Check that no other process is using VRAM with nvidia-smi.

bitsandbytes not found. Install it with pip install bitsandbytes, or use Ollama which handles quantization internally.

Model not found. Make sure you pulled the correct name. Use ollama list to see what you have.

Slow inference. This is normal when offloading. Try a smaller model or close other GPU applications.

FAQ

Answers to the questions that come up most often on this topic.

• Q: Is 4-bit quantization safe for production? A: It is often good enough for chat and code, but for critical tasks, test accuracy on your data.
• Q: Can I use multiple GPUs to run a bigger model? A: Yes, with Ollama set OLLAMA_NUM_GPU to the total number of layers, or use tensor parallelism in frameworks like vLLM.
• Q: Why do model cards show 16-bit memory? A: That is the standard for evaluation. Quantized versions are community-made and not always listed.
• Q: Will future GPUs have more VRAM? A: Slowly. NVIDIA's 5090 has 32 GB, but models are growing faster. Quantization will remain essential.

Next action

Run the memory measurement script from earlier on your machine. It takes two minutes and tells you exactly what you can run. Then pick a model from the table and try it with Ollama. That is the fastest way to close the gap between model size and GPU memory.