The Best GPU Setup for Running Local AI Agents

Before you start

You have an AI agent that needs to run locally, and you are drowning in VRAM requirements, model quantization options, and conflicting advice about whether you need an RTX 4090 or can get away with a used 3060. This guide walks you through the decision process, the hardware math, and the exact software setup to get a local agent running today.

We focus on the two things that matter most for local LLM inference: GPU VRAM capacity and memory bandwidth. The CPU and RAM matter less, but we cover those too. By the end, you will have a reproducible configuration and a working agent loop.

nvidia-smi

• Confirm your operating system: Linux is assumed, but most steps work on Windows with WSL2.
• Install Python 3.10 or newer and pip.
• Have a GPU with at least 8 GB VRAM for small models, 16 GB for 7B-13B models, 24 GB for 30B+ models.
• Install NVIDIA drivers and CUDA toolkit if you have an NVIDIA GPU.
• Optional but recommended: install Ollama or llama.cpp for model serving.

Step 1: Understand the VRAM math

The first step is to calculate how much VRAM your target model needs. The rule of thumb: take the model size in billions of parameters, multiply by 2 bytes per parameter for FP16, then add about 20% for overhead. For 4-bit quantized models, multiply by 0.5 bytes per parameter.

For example, a 7B model in FP16 needs roughly 14 GB, while a 4-bit quantized version needs about 3.5 GB. A 13B model in 4-bit needs about 6.5 GB. A 70B model in 4-bit needs about 35 GB, which will not fit on a single consumer GPU.

python -c "params=7e9; bytes_per_param=2; overhead=1.2; print(f'VRAM needed: {params*bytes_per_param*overhead/1e9:.1f} GB')"

• List your model's parameter count and precision.
• Use the formula: VRAM = parameters * bytes_per_param * 1.2
• Check your GPU's VRAM with nvidia-smi.
• If it does not fit, consider quantization or offloading to CPU.

Step 2: Choose your GPU tier

Based on your VRAM needs, you can pick a GPU tier. Here is a practical comparison of common options as of mid-2025. Prices are street prices in USD and can vary.

If you are on a budget, a used RTX 3060 12GB is a great entry point. It can run 7B and 13B models in 4-bit. For more headroom, the RTX 3090 24GB is a popular choice for 30B models. The RTX 4090 24GB is faster but costs more. For serious multi-agent work, you might need two GPUs or a used RTX 3090.

nvidia-smi --query-gpu=name,memory.total --format=csv

• RTX 3060 12GB: entry-level, runs 7B-13B quantized, around $200 used.
• RTX 3090 24GB: sweet spot for 30B models, around $700 used.
• RTX 4090 24GB: fastest consumer card, around $1600 new.
• Dual RTX 3090: for 70B models, around $1400 used total.
• Apple M-series: unified memory, but software support varies.

Step 3: Install the software stack

You need a model runtime and an agent framework. For the runtime, Ollama is the easiest to start with. For the agent framework, we will use a simple Python script with the OpenAI-compatible API that Ollama exposes.

Install Ollama by running the install script, then pull a model like Llama 3.1 8B. This model is a good balance of quality and size for a 12GB GPU.

curl -fsSL https://ollama.com/install.sh | sh

ollama pull llama3.1:8b

Step 4: Verify the runtime works

Before building an agent, confirm that the model responds. Run a simple prompt through Ollama's API. The command below sends a chat completion request and prints the response.

If you get a response, your GPU is working. If not, check that the Ollama server is running and that the model fits in VRAM.

curl http://localhost:11434/api/generate -d '{"model": "llama3.1:8b", "prompt": "Hello, are you working?"}'

Step 5: Build a minimal agent loop

Now we create a minimal agent that can call tools. We will use Python and the requests library to talk to Ollama. The agent will have a simple loop: it gets a user message, sends it to the model, checks if the model wants to call a tool, executes the tool, and appends the result.

Below is a complete script that defines a tool to get the current time. Save it as agent.py and run it. The script uses a system prompt that instructs the model to output JSON when it wants to call a tool.

import requests
import json
import datetime

OLLAMA_URL = "http://localhost:11434/api/chat"
MODEL = "llama3.1:8b"

def get_current_time():
    return datetime.datetime.now().isoformat()

tools = {
    "get_current_time": get_current_time
}

messages = [
    {"role": "system", "content": "You are a helpful assistant. When you need to call a tool, respond with a JSON object: {"tool": "tool_name", "args": {}}. Otherwise respond normally."}
]

while True:
    user_input = input("You: ")
    if user_input.lower() in ["quit", "exit"]:
        break
    messages.append({"role": "user", "content": user_input})
    response = requests.post(OLLAMA_URL, json={"model": MODEL, "messages": messages, "stream": False})
    data = response.json()
    assistant_message = data["message"]["content"]
    messages.append({"role": "assistant", "content": assistant_message})
    print("Assistant:", assistant_message)

    # Check if the assistant wants to call a tool
    try:
        tool_call = json.loads(assistant_message)
        if "tool" in tool_call:
            tool_name = tool_call["tool"]
            args = tool_call.get("args", {})
            if tool_name in tools:
                result = tools[tool_name](**args)
                messages.append({"role": "tool", "content": json.dumps({"result": result})})
                print("Tool result:", result)
    except json.JSONDecodeError:
        pass

Step 6: Optimize for your GPU

If you have a GPU with less VRAM, you can adjust Ollama's settings to offload fewer layers to the GPU or use a smaller model. For example, to run a 13B model on a 12GB GPU, you might need to offload some layers to CPU. Ollama does this automatically, but you can control it with the OLLAMA_NUM_GPU environment variable.

Set OLLAMA_NUM_GPU to the number of layers you want to offload to the GPU. A value of 999 offloads all layers, but you can lower it to fit. Also, consider using a quantized model like llama3.1:8b-q4_0.

OLLAMA_NUM_GPU=20 ollama serve

• Use the OLLAMA_NUM_GPU variable to control offloading.
• Try smaller quantized models if you hit VRAM limits.
• Monitor VRAM usage with nvidia-smi while running.
• Close other GPU applications to free memory.

Verify it worked

After running the agent script, test it with a prompt that requires the tool. For example, ask 'What time is it?' The model should respond with a JSON tool call, the script executes the tool, and then the model provides the final answer.

If the model does not call the tool, you may need to adjust the system prompt or use a model that is better at tool calling, such as Llama 3.1 which supports tool use natively.

python agent.py

Troubleshooting

If you run into issues, here are common problems and fixes.

If the model does not fit in VRAM, you will see out-of-memory errors. Reduce the context window, use a smaller model, or offload more layers to CPU.

If the model is slow, it might be because you are running on CPU. Check nvidia-smi to see GPU utilization. If it is low, the model may not be fully offloaded.

ollama stop llama3.1:8b && OLLAMA_CONTEXT_LENGTH=2048 ollama run llama3.1:8b

• OOM error: reduce OLLAMA_CONTEXT_LENGTH or use a smaller model.
• Slow response: check GPU utilization; increase OLLAMA_NUM_GPU if possible.
• API connection refused: ensure Ollama is running (ollama serve).
• Tool calling not working: use a model with native tool support like llama3.1.
• GPU not detected: install NVIDIA drivers and CUDA.

What I would do

If you are starting fresh and want a reliable setup for local AI agents, here is my recommended configuration: a used RTX 3090 24GB, 32GB system RAM, and an 8-core CPU. This gives you enough VRAM for 30B models at 4-bit and can handle multi-agent workloads.

For software, use Ollama as the runtime and a simple Python agent loop like the one above. For more advanced agents, consider using LangChain or LlamaIndex, but start with the minimal loop to understand the mechanics.

curl -fsSL https://ollama.com/install.sh | sh && ollama pull llama3.1:8b && python agent.py

FAQ

Answers to the questions that come up most often on this topic.

• Q: Can I run local agents on a Mac? A: Yes, Apple Silicon with unified memory works well, but you need to use llama.cpp or Ollama with Metal support. M1/M2 with 16GB can run 7B models, M2 Max/Ultra with 64GB can run 70B.
• Q: Is it worth buying a used RTX 3090? A: For local AI, yes. It offers 24GB VRAM at a fraction of the cost of a 4090. Just ensure it has adequate cooling and warranty.
• Q: How do I know if my model is using the GPU? A: Run nvidia-smi while the model is running and check GPU memory usage. If it is near zero, the model is running on CPU.
• Q: What is the best quantization for my GPU? A: For 12GB GPUs, use 4-bit quantization (q4_0) for 7B-13B models. For 24GB, you can use 8-bit or even FP16 for smaller models.

Next action

Your next step is to run the agent script and ask it to use a tool. Start with the time tool, then extend it with more tools like web search or file operations. The key is to get comfortable with the loop before scaling up.

python agent.py