Can You Run a 30B AI Model on a Consumer GPU?

The problem: your GPU has 8GB, the model needs 60GB

You just downloaded a 30B parameter model like Llama 3.1 30B or Mistral Medium. The model file is 60GB in FP16. Your consumer GPU has 8GB or 16GB of VRAM. The error message says out of memory. Can you still run it?

The short answer is yes, but you need to shrink the model and change how it runs. This article walks through the practical steps: quantize the model, use llama.cpp or Ollama, and offload layers to CPU when VRAM runs out. You will get a working setup you can run today.

• Check your GPU VRAM with nvidia-smi.
• Understand model size in different precisions.
• Use quantization to fit the model in memory.
• Run with llama.cpp for CPU+GPU offloading.
• Measure tokens per second to see if it is usable.

Before you start: what you need

You need a Linux or Windows machine with an NVIDIA GPU (CUDA) or an Apple Silicon Mac. For this guide, we assume an NVIDIA GPU with at least 8GB VRAM, but the steps work with less if you offload more layers to CPU.

Install the following tools: llama.cpp (for running GGUF models), and optionally Ollama if you want a one-command setup. You also need Python 3.10+ if you want to quantize your own model, but you can skip that if you download a pre-quantized GGUF file.

nvidia-smi
# Example output:
# NVIDIA GeForce RTX 3080
# Memory Usage: 10240 MiB / 10240 MiB
# (Make sure you have at least 8GB free VRAM)

Step 1: Choose a quantized model (GGUF)

A 30B model in FP16 is about 60GB. That is too big for any consumer GPU. The solution is quantization: store weights in fewer bits. A 4-bit quantized model (Q4_K_M) is around 18GB, which fits in a 24GB GPU, but not in 8GB. For 8GB, you need a 2-bit or 3-bit quant (Q2_K or Q3_K_S), which is around 12GB and 15GB respectively. Even then, you will need to offload layers to CPU.

Download a pre-quantized GGUF file from Hugging Face. For example, TheBloke's quantized Llama 3.1 30B. Look for files with Q4_K_M or Q3_K_S in the name. Use the smallest that still gives acceptable quality for your task.

mkdir -p models
cd models
# Download a Q4_K_M GGUF of Llama 3.1 30B (about 18GB)
wget https://huggingface.co/TheBloke/Llama-3.1-30B-Instruct-GGUF/resolve/main/llama-3.1-30b-instruct.Q4_K_M.gguf
# Or a smaller Q3_K_S (about 15GB)
# wget https://huggingface.co/TheBloke/Llama-3.1-30B-Instruct-GGUF/resolve/main/llama-3.1-30b-instruct.Q3_K_S.gguf

Step 2: Install llama.cpp

llama.cpp is a C++ inference engine that can run GGUF models on CPU and GPU. It supports offloading layers to GPU, so you can fit a large model by using both VRAM and system RAM. Clone the repository and build with CUDA support.

If you have a Mac, you can build with Metal support instead. The commands are similar.

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
# Build with CUDA (NVIDIA)
make LLAMA_CUDA=1
# On Mac, use: make LLAMA_METAL=1
# This produces the main executable and llama-cli

Step 3: Run the model with GPU offloading

Now run the model with llama-cli. The key flag is -ngl (number of layers to offload to GPU). Start with as many layers as your VRAM allows. If you have 8GB, try -ngl 20. If you have 24GB, you can offload all layers (-ngl 99).

The model will load partially into VRAM and the rest into system RAM. The CPU will do the work for the non-offloaded layers, which slows things down. Expect 1-5 tokens per second on an RTX 3080 with 8GB VRAM for a 30B model.

./llama-cli -m models/llama-3.1-30b-instruct.Q4_K_M.gguf -ngl 20 -p "Explain quantum computing in simple terms" -n 256
# Output will show prompt processing and generation speed
# Example: 2.5 tokens per second

• If you get CUDA out of memory, reduce -ngl by 5 and try again.
• If generation is too slow, try a smaller quant (Q3_K_S) or reduce context size with -c 2048.
• Monitor VRAM usage with nvidia-smi in another terminal.

Step 4: Use Ollama for a one-command setup

Ollama is a user-friendly wrapper around llama.cpp. It downloads models automatically and manages offloading. You can pull a 30B model and run it with a single command. Ollama uses Q4_K_M by default, which is a good balance.

Install Ollama, then pull the model. The model will be stored in ~/.ollama/models. You can set the number of GPU layers via environment variable OLLAMA_GPU_LAYERS.

# Install Ollama (Linux)
curl -fsSL https://ollama.com/install.sh | sh
# Pull a 30B model (e.g., llama3.1:30b) - this may take a while
ollama pull llama3.1:30b
# Run it, specifying GPU layers (e.g., 20)
OLLAMA_GPU_LAYERS=20 ollama run llama3.1:30b "Explain the water cycle"

Step 5: Verify it works and measure performance

After running the model, you should see a text response. To measure performance, use the timing output from llama.cpp or a simple script. For a quick check, use the -n 1 flag to generate one token and see the speed.

If the speed is below 1 token per second, the model is likely too large for your hardware. Consider a smaller quant or a 13B model.

./llama-cli -m models/llama-3.1-30b-instruct.Q4_K_M.gguf -ngl 20 -p "Hello" -n 1
# Output includes: 'llama_print_timings: load time = ...'
# 'llama_print_timings: eval time = ... ms'
# 'llama_print_timings: eval time = ... ms per token'

Troubleshooting common issues

If you see CUDA out of memory, reduce -ngl. If the model loads but produces garbage, you may have a corrupt download or wrong prompt format. If the speed is terrible, you are offloading too few layers to GPU.

Here is a checklist for common problems.

• CUDA out of memory: lower -ngl or use a smaller quant.
• Slow generation: increase -ngl if VRAM allows, or reduce context size.
• Model outputs gibberish: verify the GGUF file checksum, try a different quant.
• llama.cpp crashes: update to the latest commit, check your CUDA drivers.
• Ollama runs out of RAM: close other apps, reduce OLLAMA_GPU_LAYERS.

What I would do: recommended setup for an 8GB GPU

If you have an 8GB GPU, here is my recommended setup: use Ollama with a 30B Q4_K_M model, set OLLAMA_GPU_LAYERS to 20, and accept 2-4 tokens per second. This is usable for interactive chats but not for real-time tasks.

For better speed, consider a 13B model (Q4_K_M, ~8GB) which fits entirely in VRAM and runs at 10-20 tokens per second. The quality is still good for many tasks.

# Recommended: use Ollama with 30B model, 20 GPU layers
OLLAMA_GPU_LAYERS=20 ollama run llama3.1:30b

# Alternative: use a 13B model for faster speed
ollama pull llama3.1:13b
ollama run llama3.1:13b

FAQ

Answers to the questions that come up most often on this topic.

• Q: Can I run a 30B model on a 4GB GPU? A: Technically yes with heavy CPU offloading, but speed will be below 1 token per second. Not recommended.
• Q: What is the quality difference between Q4_K_M and Q2_K? A: Q4_K_M is much better. Q2_K may produce noticeable errors. Use Q4 if you can fit it.
• Q: Do I need a GPU at all? A: No, you can run on CPU only, but it will be very slow (0.1-0.5 tokens per second).
• Q: Can I use AMD GPUs? A: Yes, with Vulkan support in llama.cpp, but setup is more complex and performance may vary.