Can System RAM Replace GPU VRAM for AI?

The VRAM wall

You just tried to load a 13B parameter model on your 8GB GPU and got the dreaded CUDA out of memory error. Your system has 64GB of RAM, so why can't you just use that? The short answer is you can, but it comes at a cost.

This article explains the difference between system RAM and GPU VRAM, when you can substitute one for the other, and how to measure the impact. You'll leave with a concrete decision framework and a script to benchmark your own setup.

• Understand the hardware difference: VRAM is on the GPU, RAM is on the motherboard.
• Learn the three ways to use system RAM for AI: CPU offloading, unified memory, and model quantization.
• Run a benchmark to see the real-world tradeoff between speed and capacity.

Before you start

You need a Linux machine with an NVIDIA GPU (for CUDA) or an Apple Silicon Mac (for unified memory). The commands below use Python 3.10+ and PyTorch. Install the required packages:

This setup assumes you have at least 16GB of system RAM and a GPU with at least 4GB VRAM. If you have less, you can still follow along but the results will be more extreme.

pip install torch transformers psutil

Step 1: Measure your baseline

First, run a simple inference with a small model to establish a baseline. We'll use GPT-2 as a proxy for larger models. The script below loads the model on GPU and measures latency and memory usage.

Save this as baseline.py and run it. Note the output: it will show the model size, the time per token, and the GPU memory used.

import torch
import time
import psutil
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to("cuda")

inputs = tokenizer("Hello, world", return_tensors="pt").to("cuda")

# Warm up
with torch.no_grad():
    model.generate(**inputs, max_new_tokens=10)

# Benchmark
start = time.time()
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)
end = time.time()

print(f"Time: {end - start:.2f}s")
print(f"Tokens/sec: {50 / (end - start):.2f}")
print(f"GPU memory: {torch.cuda.memory_allocated() / 1024**3:.2f} GB")
print(f"System RAM: {psutil.virtual_memory().used / 1024**3:.2f} GB")

• Run this on a machine with at least 8GB VRAM to get a stable baseline.
• If you get an out-of-memory error, reduce max_new_tokens to 10.

Step 2: Offload to CPU

The most common way to use system RAM is to offload part of the model to the CPU. PyTorch's accelerate library makes this easy. The following script loads the model with device_map="auto", which places layers on GPU or CPU based on available memory.

Run this script and compare the tokens/sec with the baseline. You'll see a significant slowdown, but the model will run even if it doesn't fit in VRAM.

import torch
import time
import psutil
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

inputs = tokenizer("Hello, world", return_tensors="pt")

# Warm up
with torch.no_grad():
    model.generate(**inputs, max_new_tokens=10)

# Benchmark
start = time.time()
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)
end = time.time()

print(f"Time: {end - start:.2f}s")
print(f"Tokens/sec: {50 / (end - start):.2f}")
print(f"GPU memory: {torch.cuda.memory_allocated() / 1024**3:.2f} GB")
print(f"System RAM: {psutil.virtual_memory().used / 1024**3:.2f} GB")

• device_map="auto" places as many layers as possible on GPU, the rest on CPU.
• Expect 10-50x slower inference compared to pure GPU.
• Works with any Hugging Face model, but you need the accelerate library installed.

Step 3: Quantize the model

Another way to fit larger models is to reduce the precision of the weights. The bitsandbytes library allows 8-bit and 4-bit quantization. This reduces VRAM usage but adds some overhead.

Run this script with quantization. You'll see that the model uses less VRAM but may still be slower than the baseline.

import torch
import time
import psutil
from transformers import AutoModelForCausalLM, AutoTokenizer
import bitsandbytes as bnb

model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, load_in_8bit=True, device_map="auto")

inputs = tokenizer("Hello, world", return_tensors="pt")

# Warm up
with torch.no_grad():
    model.generate(**inputs, max_new_tokens=10)

# Benchmark
start = time.time()
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)
end = time.time()

print(f"Time: {end - start:.2f}s")
print(f"Tokens/sec: {50 / (end - start):.2f}")
print(f"GPU memory: {torch.cuda.memory_allocated() / 1024**3:.2f} GB")
print(f"System RAM: {psutil.virtual_memory().used / 1024**3:.2f} GB")

• 8-bit quantization reduces VRAM usage by about 50%.
• 4-bit quantization can reduce it further but may require a GPU with at least 8GB VRAM.
• Quantization can slightly degrade model quality, but for many tasks it's acceptable.

Step 4: Compare unified memory (Apple Silicon)

On Apple Silicon, the CPU and GPU share the same memory pool. This means you can allocate more memory to the GPU without separate VRAM. The tradeoff is that the unified memory bandwidth is lower than dedicated VRAM.

If you have an M1 or M2 Mac, you can run the same scripts by replacing .to("cuda") with .to("mps"). The following script shows how to use MPS for inference.

import torch
import time
import psutil
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "mps" if torch.backends.mps.is_available() else "cpu"

model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to(device)

inputs = tokenizer("Hello, world", return_tensors="pt").to(device)

# Warm up
with torch.no_grad():
    model.generate(**inputs, max_new_tokens=10)

# Benchmark
start = time.time()
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)
end = time.time()

print(f"Time: {end - start:.2f}s")
print(f"Tokens/sec: {50 / (end - start):.2f}")
print(f"System RAM: {psutil.virtual_memory().used / 1024**3:.2f} GB")

• MPS is PyTorch's Metal Performance Shaders backend.
• Unified memory allows larger models, but you'll still hit memory limits.
• Check torch.backends.mps.is_available() to confirm your Mac supports it.

Step 5: Run a real-world test with a larger model

To see the real impact, try loading a model that doesn't fit in your VRAM. For example, try the 7B parameter model Llama-2-7B-Chat. The script below will attempt to load it with device_map="auto" and measure the performance.

This is where you'll see the tradeoff clearly. If you have an 8GB GPU, the model will likely use 6GB of VRAM and 10GB of system RAM, and inference will be much slower than a smaller model on pure GPU.

python -c "
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch, time, psutil
model_name = 'meta-llama/Llama-2-7b-chat-hf'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto')
inputs = tokenizer('Hello', return_tensors='pt')
with torch.no_grad():
    model.generate(**inputs, max_new_tokens=50)
print('GPU mem:', torch.cuda.memory_allocated()/1024**3, 'GB')
print('RAM used:', psutil.virtual_memory().used/1024**3, 'GB')
"

• You need to request access to Llama-2 on Hugging Face before downloading.
• Expect to wait several minutes for the download if you haven't cached it.
• If you get an out-of-memory error, try loading in 8-bit: load_in_8bit=True.

Decision framework: when to use system RAM

Use this checklist to decide whether system RAM can replace VRAM for your workload.

If you answer yes to most of these, system RAM is a viable option. If not, invest in a GPU with more VRAM.

• Is your task batch inference where latency is not critical?
• Is your model smaller than 13B parameters?
• Do you have at least 32GB of system RAM?
• Can you tolerate 10-50x slower inference?
• Is your budget tight and you already have a decent CPU?
• Is your workload occasional, not production?

What I would do

For a local AI assistant that I use occasionally, I would use CPU offloading with a 7B model quantized to 4-bit. This gives a good balance between quality and speed.

Here is a copy-paste starter script that loads a 4-bit quantized Llama-2-7B model on any machine with at least 16GB RAM. It's slow but it works.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import BitsAndBytesConfig

model_name = "meta-llama/Llama-2-7b-chat-hf"
quantization_config = BitsAndBytesConfig(load_in_4bit=True)

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=quantization_config,
    device_map="auto"
)

prompt = "What is the capital of France?"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

• Install bitsandbytes and accelerate: pip install bitsandbytes accelerate
• This script uses about 5GB of VRAM and 8GB of system RAM.
• Expect about 2-3 tokens per second on a modern CPU.

Troubleshooting

If you run into issues, here are common problems and fixes.

The most common error is CUDA out of memory. Reduce the batch size or use quantization.

• CUDA out of memory: reduce max_new_tokens, use load_in_8bit=True, or offload more layers to CPU.
• Slow inference: increase batch size if you're doing batch processing, or use a smaller model.
• MPS not available: update PyTorch to the latest version and ensure you're on macOS 12.3+.
• bitsandbytes not supported on Windows: use WSL2 or a Linux machine.

FAQ

Can I use system RAM as VRAM for training?

Training is far more memory-intensive than inference. You can use CPU offloading for training, but it's extremely slow. It's better to use a GPU with enough VRAM or use cloud services.

Is unified memory on Apple Silicon better than CPU offloading?

Yes, because the CPU and GPU share the same memory pool, so you don't have the overhead of copying data between separate memory spaces. But it's still slower than dedicated VRAM.

What is the maximum model size I can run with 64GB RAM?

With 4-bit quantization, you can run models up to 70B parameters, but you'll need a lot of patience. For 7B models, 16GB RAM is enough.

Does system RAM speed matter?

Yes, faster system RAM (DDR5 vs DDR4) can reduce the bottleneck when offloading. But the GPU memory bandwidth is still the limiting factor.

Next action

Run the baseline script and the offload script on your machine. Compare the tokens/sec and memory usage. That will tell you if system RAM is a viable substitute for your workloads.

The command to run both is:

python baseline.py && python offload.py