How to Run Your First AI Model on Your Laptop

Before you start

You have heard about running AI models locally, but every tutorial assumes you have a $4,000 GPU. The truth is you can run a useful model on a laptop with 8 GB of RAM, and even a 4 GB GPU can handle a 7B parameter model with quantization. This guide walks you through the exact steps to get a working local AI chat assistant on your machine, whether you use Windows, macOS, or Linux.

The goal is to get you to a working prompt-response loop as fast as possible, then show you how to inspect performance and troubleshoot common issues. By the end, you will have a local model you can talk to, and the knowledge to pick the right model size for your hardware.

• Check your hardware: CPU, RAM, and GPU.
• Install Ollama (recommended for beginners) or llama.cpp (for more control).
• Pull a model and run your first prompt.
• Measure tokens per second to know your speed.
• Troubleshoot common errors like out-of-memory and slow inference.

Step 1: Check your hardware

Before installing anything, know what you are working with. Open a terminal and run the commands below. On Windows, use PowerShell; on macOS, use Terminal; on Linux, any shell works.

# Windows (PowerShell)
Get-CimInstance Win32_ComputerSystem | Select-Object TotalPhysicalMemory
Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM

# macOS
system_profiler SPHardwareDataType | grep Memory
system_profiler SPDisplaysDataType | grep Chipset

# Linux
free -h
lspci | grep -i vga

• RAM: 8 GB minimum for 7B models, 16 GB recommended for 13B+.
• GPU: Even integrated graphics help, but a dedicated GPU with 4 GB VRAM is ideal for 7B models.
• CPU: Any modern x86 or Apple Silicon CPU works.

Step 2: Install Ollama

Ollama is the simplest way to run models locally. It handles model downloads, quantization, and a local API. Install it with the official script or download from ollama.com. On macOS and Windows, there is a one-click installer; on Linux, use the curl command.

# Linux / macOS
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

• Ollama runs as a background service and provides a REST API on port 11434.
• Model files are stored in ~/.ollama/models by default.
• You can use the command line or the API from any programming language.

Step 3: Pull and run a model

Now pull a small model. For most laptops, Llama 3.2 3B is a good starting point. It is fast and runs on 8 GB RAM. If you have more RAM, try Llama 3.1 8B. The command downloads the model and then starts an interactive chat session.

ollama pull llama3.2:3b
ollama run llama3.2:3b

# In the chat, type a prompt:
>>> Write a Python function to reverse a string.

# Exit with /bye

• Model sizes: 3B, 7B, 8B, 13B, 70B. Larger means smarter but slower and more RAM.
• Use the :q4_0 tag for a quantized version that uses less RAM.
• You can exit the chat with /bye.

Step 4: Use the API programmatically

The interactive chat is fun, but you will want to call the model from code. Ollama exposes a REST API. Here is a Python script that sends a prompt and prints the response. It uses the requests library, so install it first with pip.

import requests
import json

url = "http://localhost:11434/api/generate"
payload = {
    "model": "llama3.2:3b",
    "prompt": "Explain what a transformer is in one paragraph.",
    "stream": False
}

response = requests.post(url, json=payload)
print(response.json()["response"])

• The API endpoint is /api/generate for one-off completions.
• Set stream=False to get the full response at once.
• You can also use the official ollama Python library.

Step 5: Use llama.cpp for more control

If you want to run a model without Ollama, or you need to use a custom GGUF file, llama.cpp is the go-to. It is a C++ library that runs models efficiently on CPU and GPU. Clone the repository, build it, and run the server. This is more involved but gives you full control over inference parameters.

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j

# Download a GGUF model (example: TinyLlama 1.1B)
curl -L -o models/tinyllama.gguf https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf

# Run the server
./server -m models/tinyllama.gguf -c 2048

• llama.cpp supports GPU acceleration via CUDA, Metal, and Vulkan.
• You need to download a GGUF model file from Hugging Face.
• The server provides an OpenAI-compatible API.

Step 6: Measure performance

You need to know how fast your setup is. Tokens per second (t/s) is the standard metric. Ollama prints this after each response. You can also measure it programmatically. Here is a simple Python script that times a generation and calculates t/s.

import requests
import time

url = "http://localhost:11434/api/generate"
payload = {
    "model": "llama3.2:3b",
    "prompt": "Write a haiku about coding.",
    "stream": False
}

start = time.time()
response = requests.post(url, json=payload)
elapsed = time.time() - start

output = response.json()["response"]
num_tokens = len(output.split())  # rough estimate
print(f"Time: {elapsed:.2f} seconds")
print(f"Tokens: {num_tokens}")
print(f"Tokens per second: {num_tokens / elapsed:.2f}")

• On a CPU-only laptop, expect 5-15 t/s for a 3B model.
• With a GPU, you can get 20-50 t/s.
• Lower is fine for occasional use, but for interactive chat you want at least 10 t/s.

Step 7: Recommended setup

For most laptops, I recommend Ollama with a 3B or 7B model. It is the least friction and gives you a clean API. If you need to run a specific GGUF file or want to tweak sampling parameters, use llama.cpp. Here is my recommended copy-paste starter for a laptop with 8 GB RAM.

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a 3B model
ollama pull llama3.2:3b

# Set environment variables (Linux/macOS)
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_KEEP_ALIVE=5m

# Run the model
ollama run llama3.2:3b

• Use a quantized model (Q4_K_M) to reduce RAM usage.
• Set OLLAMA_NUM_PARALLEL=1 to avoid memory spikes.
• Keep the model loaded in memory for faster responses.

Troubleshooting

You will hit issues. Here are the most common ones and how to fix them.

# Check if Ollama is running
curl http://localhost:11434/api/tags

# If not, start it
ollama serve

# For llama.cpp, check GPU support
./llama-server --help | grep -i gpu

• Out of memory: Use a smaller model or a more aggressive quantization (Q4_0 instead of Q8_0).
• Slow responses: Close other applications, or use a smaller model.
• GPU not detected: Make sure you installed the GPU version of llama.cpp, or check Ollama's GPU support.
• API connection refused: Ensure Ollama is running (ollama serve).
• Model not found: Run ollama pull <model> first.

FAQ

Answers to the questions that come up most often on this topic.

• Q: Can I run a 70B model on a laptop? A: Not practically. You need at least 32 GB RAM and a powerful GPU. Stick to 7B or smaller.
• Q: What is quantization? A: It reduces the precision of the model weights to save memory, with a small quality trade-off.
• Q: Is it safe to run models locally? A: Yes, your data stays on your machine. That is a major benefit.
• Q: Do I need a GPU? A: No, CPU works, but it will be slower. A GPU with 4 GB VRAM is enough for 7B models.

Next step

Now that you have a working model, try building a small application. Use the Ollama API to create a command-line chatbot or a simple web interface. The fastest way to get value is to write a script that reads a file and asks the model to summarize it. Here is a starter script.

import sys
import requests

if len(sys.argv) != 2:
    print("Usage: python summarize.py <file>")
    sys.exit(1)

with open(sys.argv[1], "r") as f:
    content = f.read()

url = "http://localhost:11434/api/generate"
payload = {
    "model": "llama3.2:3b",
    "prompt": f"Summarize the following text in 3 sentences:\n\n{content}",
    "stream": False
}

response = requests.post(url, json=payload)
print(response.json()["response"])

• Save the script as summarize.py.
• Run it with python summarize.py yourfile.txt.
• The model will output a summary to the console.