GPU Offloading Explained: How Local AI Uses Your RAM

The Out-of-Memory Wall

You just downloaded a 7B parameter model, fired up llama.cpp, and hit CUDA out of memory. The error says your GPU has 8GB of VRAM but the model needs 14GB. Your first instinct is to buy a bigger GPU. Before you do, know that local inference tools can split the model across GPU and system RAM. This is called GPU offloading, and it is the difference between running a model and not running it.

This article explains what offloading does, how to control it with llama.cpp and Hugging Face, and how to measure the performance hit. You will learn the exact commands to check your hardware, set the number of layers to offload, and benchmark the result.

• Understand the difference between VRAM and system RAM
• Learn the -ngl flag in llama.cpp and device_map in Transformers
• Measure memory usage and tokens per second with real commands
• Get a recommended setup for a typical 8GB GPU

Before You Start

You need a machine with an NVIDIA GPU (for CUDA) and enough system RAM to hold the model. For a 7B model in 4-bit quantization, plan for about 4GB of system RAM if you offload half the layers. You also need the tools: llama.cpp or Python with the Transformers library.

First, verify your GPU and its VRAM. On Linux, run nvidia-smi. On Windows, use the same command if you have the NVIDIA driver installed.

nvidia-smi --query-gpu=name,memory.total --format=csv

• If nvidia-smi is not found, install the NVIDIA driver and CUDA toolkit.
• For AMD or Intel GPUs, the commands differ; this article focuses on NVIDIA.

Step 1: Understand the Memory Split

A model like Llama 3 8B has about 8 billion parameters. At 4-bit quantization, each parameter takes about 0.5 bytes, so the model file is around 4GB. But inference also needs memory for activations and key-value cache, often adding 1-2GB. If your GPU has 8GB VRAM, the whole model might fit, but if you have a 6GB GPU, you need offloading.

Offloading works by placing some transformer layers in system RAM and the rest in VRAM. During inference, data moves between the two. This is slower than all-GPU, but it lets you run models that would otherwise not fit.

Step 2: Install llama.cpp and Get a Model

The easiest way to experiment is with llama.cpp. Clone the repository, build it, and download a quantized model from Hugging Face. For this example, use the 4-bit quantized Llama 3 8B Instruct.

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j

# Download a model (example: Llama 3 8B Instruct 4-bit)
wget https://huggingface.co/bartowski/Meta-Llama-3-8B-Instruct-GGUF/resolve/main/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf

Step 3: Run with Full GPU Offload

By default, llama.cpp tries to offload all layers to the GPU. The -ngl flag sets the number of layers to offload. To use all layers, set -ngl 999. Run a simple prompt and watch the memory usage with nvidia-smi in another terminal.

./main -m Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -p "Hello, how are you?" -n 64 -ngl 999

• If you get a CUDA out of memory error, reduce -ngl.
• Watch nvidia-smi to see VRAM usage peak.

Step 4: Offload Half the Layers

Now reduce the offload to half the layers. Llama 3 8B has 32 layers, so use -ngl 16. This splits the model roughly evenly between GPU and CPU. Run the same prompt and compare the speed.

./main -m Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -p "Hello, how are you?" -n 64 -ngl 16

Step 5: Measure Performance

llama.cpp prints tokens per second at the end of generation. Record that number for each -ngl value. Also note the VRAM usage from nvidia-smi. This gives you a direct tradeoff: more offload means faster but more VRAM, less offload means slower but less VRAM.

# Run a benchmark with a fixed prompt
./main -m model.gguf -p "The sky is" -n 128 -ngl 999 2>&1 | grep "tokens per second"
./main -m model.gguf -p "The sky is" -n 128 -ngl 16 2>&1 | grep "tokens per second"

Step 6: Offload in Python with Transformers

If you prefer Python, the Hugging Face Transformers library lets you control device mapping. The device_map parameter can be set to "auto" to let the library decide, or you can specify a custom mapping. First install the library and load a model with 4-bit quantization.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    load_in_4bit=True,
    bnb_4bit_compute_dtype="float16"
)

inputs = tokenizer("Hello, how are you?", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0]))

Step 7: Custom Device Map

If you want explicit control, define a device map that places layers on GPU 0 and CPU. The example below puts the first 16 layers on GPU and the rest on CPU. This mirrors the -ngl 16 example.

device_map = {
    "model.embed_tokens": 0,
    "model.layers.0": 0,
    "model.layers.1": 0,
    # ... continue for layers 2-15
    "model.layers.15": 0,
    "model.layers.16": "cpu",
    # ... continue for layers 17-31
    "model.layers.31": "cpu",
    "model.norm": "cpu",
    "lm_head": "cpu"
}

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map=device_map,
    load_in_4bit=True
)

Recommended Setup for an 8GB GPU

For an 8GB GPU and a 7B or 8B model in 4-bit, I recommend offloading all layers if the model fits. If it does not, start with -ngl 20 and reduce in steps of 4 until it runs. You want the highest -ngl that does not cause out-of-memory. For a 6GB GPU, you might need -ngl 16 or lower.

Here is a copy-paste command sequence for llama.cpp that prints the memory usage and speed for a range of -ngl values.

for ngl in 999 24 20 16 12 8; do
  echo "=== ngl $ngl ==="
  nvidia-smi --query-gpu=memory.used --format=csv,noheader &
  ./main -m model.gguf -p "Once upon a time" -n 128 -ngl $ngl 2>&1 | grep "tokens per second"
  wait
  sleep 1
  nvidia-smi --query-gpu=memory.used --format=csv,noheader
  echo
 done

Troubleshooting

If you see CUDA out of memory, lower -ngl. If you see system RAM out of memory, close other applications or reduce the context size. If generation is extremely slow, you are offloading too few layers; increase -ngl if VRAM allows.

Common error: cuDNN not found. Rebuild llama.cpp with the correct CUDA architecture. Use make LLAMA_CUDA=1 after setting your GPU architecture in the Makefile.

• CUDA out of memory: reduce -ngl or use a smaller model.
• System RAM out of memory: reduce context size (-c) or close apps.
• Slow generation: increase -ngl if VRAM has room.
• Build errors: set CUDA_ARCH and rebuild.

FAQ

• Q: Does offloading affect output quality? A: No, the model weights are the same; only speed changes.
• Q: Can I offload to multiple GPUs? A: Yes, llama.cpp supports -ngl per GPU with -split-mode, and Transformers device_map can list multiple GPUs.
• Q: What is the maximum model size I can run? A: It depends on system RAM and CPU speed. A 70B model in 4-bit needs about 35GB RAM, so you need that much system RAM.
• Q: Is there a way to measure VRAM usage programmatically? A: Use nvidia-smi or the pynvml library in Python.

Next Step

Run the benchmark loop from the Recommended Setup section on your machine. Note the tokens per second for each -ngl value and pick the highest that does not crash. That is your sweet spot.

Try it on code.live

If you want to quickly test prompts or compare model outputs without setting up a local environment, you can use the API Mock Data Generator to create sample request payloads, but for actual inference you need your own hardware.