I Ran an AI Model Locally Instead of Paying an API Bill

Before you start

You are tired of watching your API bill climb every time you prototype an AI feature. You have a decent laptop with a GPU, or at least 16 GB of RAM, and you want to know if running a model locally is viable.

This article walks through running an open-source LLM with Ollama, measuring its speed and memory usage, comparing costs against a typical API, and integrating it into a small Python app. You will need about 20 minutes and a machine with either an NVIDIA GPU or enough RAM to run a 7B model.

All commands are tested on Ubuntu 22.04 with an NVIDIA RTX 3060 12GB, but they should work on macOS with Apple Silicon or Windows with WSL2.

curl -fsSL https://ollama.com/install.sh | sh

• Check your GPU with nvidia-smi or use CPU-only mode if you have at least 16 GB RAM.
• Install Ollama from the official script or package manager.
• Pull a model, run inference, and measure performance.
• Compare cost and latency against a cloud API.
• Integrate the model into a Python script using the OpenAI-compatible endpoint.

Step 1: Install Ollama and pull a model

Ollama is a tool that manages local LLMs with a simple CLI. It supports many open-source models like Llama 3, Mistral, and Gemma. For this tutorial, use Llama 3.2 3B, which requires about 6 GB of RAM and runs fast on CPU or GPU.

After installation, start the Ollama service (it runs in the background) and pull the model. The pull downloads the model weights, which takes a few minutes depending on your network.

ollama pull llama3.2:3b

Step 2: Run your first inference

Now you can chat with the model directly from the terminal. Ollama provides an interactive mode, but for scripting you can use one-shot prompts with the run command.

The first run may be slower because the model loads into memory. Subsequent calls are faster.

ollama run llama3.2:3b "Explain the difference between TCP and UDP in one sentence."

• Use the --verbose flag to see timing details: ollama run llama3.2:3b --verbose "..."
• For a chat session, just type ollama run llama3.2:3b and enter prompts interactively.

Step 3: Measure performance and memory usage

To decide if local is viable, you need metrics: tokens per second, latency, and RAM/VRAM usage. Ollama's verbose output gives you these numbers, but you can also script it.

The following Python script sends a prompt to the local model and measures response time and token speed. It also prints the model size and memory footprint from the Ollama API.

import requests
import time
import json

MODEL = "llama3.2:3b"
PROMPT = "Write a haiku about debugging."

start = time.time()
response = requests.post(
    "http://localhost:11434/api/generate",
    json={"model": MODEL, "prompt": PROMPT, "stream": False}
)
elapsed = time.time() - start

data = response.json()
print("Response:", data["response"])
print(f"Total time: {elapsed:.2f}s")
print(f"Tokens generated: {data.get('eval_count', 0)}")
print(f"Tokens per second: {data.get('eval_count', 0) / elapsed:.2f}")

# Show model info
info = requests.get("http://localhost:11434/api/show", params={"name": MODEL}).json()
print("Model size:", info.get("details", {}).get("parameter_size"))
print("Quantization:", info.get("details", {}).get("quantization_level"))

• On a 12GB GPU, expect 50-80 tokens/s for a 3B model.
• On CPU-only, expect 10-20 tokens/s with 16GB RAM.
• Monitor VRAM with nvidia-smi while the model is loaded.

Step 4: Compare costs with an API

Now compare the cost of running this locally versus using a hosted API. As of June 2025, OpenAI's GPT-4o mini costs $0.15 per million input tokens and $0.60 per million output tokens. Anthropic's Claude 3.5 Haiku is $0.25/$1.25. For a 3B open model, a comparable API might be Groq's Llama 3.1 8B, which is free at low volume, but for a realistic comparison, use a paid provider like Together AI or Replicate.

Assume you process 500,000 requests per month, each with 1,000 input and 200 output tokens. That is 500M input tokens and 100M output tokens. With GPT-4o mini pricing, that costs $75 for input and $60 for output, total $135 per month. With a local setup, the hardware cost is a one-time purchase or the electricity cost, which is roughly $5-10 per month for a GPU running 24/7.

The table below summarizes the trade-offs.

# Example: estimate API cost with a simple calculation
# 500k requests/month, 1000 input tokens, 200 output tokens
# GPT-4o mini pricing: $0.15/M input, $0.60/M output

input_tokens = 500_000 * 1000  # 500M
output_tokens = 500_000 * 200  # 100M
cost = (input_tokens / 1e6 * 0.15) + (output_tokens / 1e6 * 0.60)
print(f"Estimated monthly API cost: ${cost:.2f}")

• Local: upfront hardware cost, but no per-token fees.
• API: pay per token, but no hardware maintenance.
• Latency: local can be faster for small models on GPU, but API has higher throughput for large models.
• Privacy: local keeps data on-premise.

Step 5: Integrate with your application

Ollama exposes an OpenAI-compatible API on port 11434. This means you can switch your existing OpenAI client to point to http://localhost:11434/v1 without changing much code. Here is a minimal example using the openai Python package.

Create a virtual environment, install the package, and run the script. This lets you test a local model with the same code you would use for a hosted API.

python -m venv venv
source venv/bin/activate
pip install openai

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
    model="llama3.2:3b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"}
    ]
)

print(response.choices[0].message.content)

Step 6: Optimize for production

If you decide to run a local model in production, you need to consider concurrency, queueing, and GPU memory sharing. Ollama can handle multiple requests, but for high throughput, you might want to use a dedicated inference server like vLLM or TGI.

For a single-user or small-team scenario, Ollama is fine. For a larger deployment, set up a systemd service to keep Ollama running, and consider using a GPU with more VRAM to load a larger model.

Here is a systemd unit file example to run Ollama as a service.

sudo tee /etc/systemd/system/ollama.service <<EOF
[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
Restart=always
RestartSec=3
Environment="OLLAMA_HOST=0.0.0.0:11434"

[Install]
WantedBy=default.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now ollama

• Set OLLAMA_HOST to 0.0.0.0 if you want to access it from other machines.
• Use ollama stop <model> to free GPU memory when not in use.
• Monitor logs with journalctl -u ollama -f.

Troubleshooting

If you run into issues, here are common problems and fixes.

• Model not found: run ollama pull <model> again.
• Out of memory: use a smaller model or quantized version (e.g., llama3.2:1b).
• Slow performance: ensure your GPU drivers are installed and CUDA is available.
• Connection refused: start Ollama with ollama serve or check the service status.

What I would do

For a personal project or a small team, I would definitely run a local model with Ollama. The cost savings are real, and the privacy benefits are huge. For a production SaaS with high concurrency, I would start with a hosted API for reliability, but keep a local fallback for development and internal tools.

My recommended setup is a machine with at least 16GB RAM, an NVIDIA GPU with 8GB VRAM, and Ollama as the runtime. Use the 3B model for quick tasks and a 8B model for more complex ones.

Here is a starter script to install everything and pull two models.

#!/bin/bash
# recommended-setup.sh
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2:3b
ollama pull llama3.1:8b
# Test
ollama run llama3.2:3b "Hello, world!"

FAQ

Can I run a local model on a CPU only? Yes, but expect slower speeds. Use a 3B model and at least 16GB RAM.

How much does it cost to run a local model? Hardware costs vary, but electricity is usually under $10 per month. No per-token fees.

Can I use a local model with LangChain or LlamaIndex? Yes, both support Ollama as a provider.

Is local AI as good as GPT-4? Not for complex reasoning, but for many tasks like summarization and classification, 3B models are surprisingly capable.

What about fine-tuning? You can fine-tune open models locally, but that requires more resources and is beyond this article.

Next step

Now that you have a local model running, try integrating it into a real project. Replace a simple API call in your code with the local endpoint and see if performance meets your needs.

Run the following command to start a chat with your local model and explore its capabilities.

ollama run llama3.2:3b