Why Local AI Is Suddenly Becoming Practical

The problem: cloud AI costs and privacy concerns

You are building an app that needs AI features. The obvious move is to call an API from a cloud provider. But then the bill arrives, or you realize you cannot send customer data to a third party. You start wondering: can I run a model on my own hardware?

For years the answer was no. Models were too large, inference was too slow, and you needed a data center GPU. But that changed. Quantization, better runtimes, and smaller models have made local AI viable on a laptop or a modest workstation. This article walks through the practical steps to get a useful local model running today, with commands you can copy and run.

• Check your hardware: you need at least 8 GB of RAM for a 7B model, 16 GB for a 13B model.
• Understand quantization: it reduces model size and memory usage with minimal quality loss.
• Use a runtime like Ollama or llama.cpp to run models efficiently.
• Measure your actual token throughput before committing to local inference.
• Plan for the trade-offs: latency, quality, and hardware cost.

Before you start: what you need

You need a machine with a CPU, preferably with AVX2 support, and enough RAM. A GPU helps but is not required for smaller models. For this guide, I assume you have a Linux or macOS machine with at least 8 GB of RAM and 10 GB of free disk space.

We will use Ollama as the runtime. It wraps llama.cpp and provides a simple CLI and API. It handles model downloads and quantization automatically.

curl -fsSL https://ollama.com/install.sh | sh

• Windows users can use WSL2 or the native Windows build from Ollama's site.
• Check your CPU supports AVX2: run grep avx2 /proc/cpuinfo on Linux.
• Ensure you have at least 10 GB free disk space for the model files.

Step 1: Install Ollama and pull a model

Once Ollama is installed, you pull a model by name. The default model is llama3.2, but for lower memory you can use phi3 or qwen2.5. For this walkthrough, we use llama3.2:3b, a 3-billion parameter model that runs on 8 GB of RAM.

The command downloads the model and stores it locally. The first pull may take a few minutes.

ollama pull llama3.2:3b

• Model names follow the pattern name:tag, where tag is size or quantization.
• Run ollama list to see what you have downloaded.
• For a larger model, try llama3.2:1b if you are tight on RAM.

Step 2: Run your first inference

Now you can chat with the model directly from the terminal. The ollama run command starts an interactive session. Type a prompt and press Enter. The model streams tokens as it generates.

This is the simplest way to verify that your setup works.

ollama run llama3.2:3b "Explain what a transformer is in one sentence."

• You can exit the interactive session with /bye.
• The model uses CPU by default; if you have an NVIDIA GPU, it will use it automatically.
• If you get a 'not enough memory' error, try a smaller model or close other applications.

Step 3: Use the API for programmatic access

Ollama exposes a REST API on port 11434. You can send a POST request with a JSON payload to get completions. This is how you integrate local AI into your own scripts or applications.

The response includes the generated text and metadata like total duration and tokens per second.

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

• Set stream to false to get the full response at once.
• The response contains a 'response' field with the text.
• Use jq to parse the JSON: curl ... | jq .response

Step 4: Measure performance

Before you rely on local inference, you need to know how fast it is. The API response includes eval_count (number of tokens generated) and eval_duration (time in nanoseconds). From these you can compute tokens per second.

Here is a small script that runs a prompt and prints the throughput.

#!/bin/bash
start=$(date +%s%N)
response=$(curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Write a haiku about code.",
  "stream": false
}')
end=$(date +%s%N)
tokens=$(echo $response | jq '.eval_count')
duration=$(echo $response | jq '.eval_duration')
# duration is in ns, convert to seconds
seconds=$(echo "scale=9; $duration / 1000000000" | bc)
tps=$(echo "scale=2; $tokens / $seconds" | bc)
echo "Generated $tokens tokens in $seconds seconds ($tps tokens/sec)"

• Typical numbers: on a modern CPU, a 3B model runs at 20-40 tokens per second.
• On a laptop with an NVIDIA GPU, you might get 50-100 tokens per second.
• If you need higher throughput, consider a smaller model or more aggressive quantization.

Step 5: Use a custom model with a Modelfile

You can create a custom model that includes a system prompt or custom parameters. This is useful for production-like setups. Create a file called Modelfile with the base model and a system message.

Then build the custom model with ollama create.

FROM llama3.2:3b

SYSTEM """You are a helpful coding assistant. Always provide code examples in your answers."""

PARAMETER temperature 0.7
PARAMETER top_p 0.9

ollama create mycoder -f ./Modelfile

• You can set parameters like temperature and top_p in the Modelfile.
• Use ollama show mycoder to inspect the model.
• Now run ollama run mycoder to use your custom assistant.

Recommended setup for a small project

If you want to add local AI to a small project, here is a minimal setup that works today. Use a 3B model, run Ollama as a service, and call it from your application via HTTP.

I recommend starting with llama3.2:3b for general text generation, and qwen2.5:3b for code generation if you need better code output. Keep your system prompt short to save tokens.

# Start Ollama as a background service (if not already running)
ollama serve &

# Pull the recommended model
ollama pull llama3.2:3b

# Test with a quick prompt
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Write a Python function to reverse a string.",
  "stream": false
}' | jq .response

• Use a reverse proxy like nginx if you expose the API beyond localhost.
• Set OLLAMA_HOST=0.0.0.0 if you need to serve on your LAN.
• Monitor RAM usage with htop; if it swaps, reduce the model size.

Troubleshooting common issues

Local AI is not always smooth. Here are the most common problems and their fixes.

If the model is slow, check your CPU governor and close other heavy processes. If you get a memory error, use a smaller model or enable swap (but that will hurt performance).

• Error: 'could not find a suitable llama.cpp build' - update Ollama to the latest version.
• Error: 'not enough memory' - close other apps, or use a 1B model.
• Slow generation - check CPU usage; if 100%, try a smaller model or reduce context length.
• API not responding - ensure ollama serve is running, and check the port with ss -tlnp | grep 11434.
• GPU not used - install CUDA drivers and ensure Ollama detects it (ollama --version shows GPU info).

FAQ

Answers to the questions that come up most often on this topic.

• Q: Can I run a 70B model on a laptop? A: No, you need at least 48 GB of RAM for a 70B model with 4-bit quantization. Use a 7B or 13B model instead.
• Q: How much does it cost to run local AI? A: The software is free, but you need hardware. A used NVIDIA RTX 3060 costs around $200 and can run 7B models.
• Q: Is local AI as good as cloud AI? A: For many tasks, yes, especially with recent models like Llama 3.2. For complex reasoning, cloud models still lead, but the gap is narrowing.
• Q: Can I use local AI in production? A: Yes, but you need to handle scaling, monitoring, and model updates. Ollama provides a simple API, but you may need a more robust deployment for high traffic.

What to do next

Now you have a working local AI setup. The next step is to integrate it into your own application. Start by wrapping the Ollama API in a simple function, then add caching and error handling.

Here is a minimal Python client to get you started.

import requests
import json

def generate(prompt, model="llama3.2:3b"):
    resp = requests.post(
        "http://localhost:11434/api/generate",
        json={"model": model, "prompt": prompt, "stream": False}
    )
    resp.raise_for_status()
    return resp.json()["response"]

print(generate("Explain the difference between TCP and UDP."))

• Run this script and you should see a response from your local model.
• Add a timeout to avoid hanging if the model is busy.
• Consider using the OpenAI-compatible endpoint at /v1/chat/completions if you want to switch between local and cloud easily.