Why Your Computer Runs Out of RAM When Running Local AI

The Out-of-Memory Error You Keep Seeing

You try to load a 7B parameter model with llama.cpp or Ollama, and within seconds you see something like 'Not enough memory' or the process gets killed by the OOM killer. Your machine has 16GB of RAM, and you thought that would be plenty. But the model alone needs more than that, and your OS and other apps are using memory too.

This article explains why local AI models consume so much RAM, how to measure exactly what is happening, and what you can do to run larger models or avoid crashes. You will walk away with concrete commands and configuration changes you can apply today.

Why Models Need So Much Memory

The main reason is the model weights. A model with 7 billion parameters, stored in 16-bit floating point, needs about 14GB just for the weights. If you load it in 32-bit, that doubles to 28GB. Even with 8-bit quantization, you still need around 7GB.

But weights are not the only cost. The runtime also allocates memory for activations, key-value caches, and intermediate tensors during inference. This can add another 20-50% on top of the weights, depending on context length and batch size.

Your operating system and other applications also consume RAM. When you run a model, the total memory usage is the sum of the model footprint plus your usual workload. That is why a 7B model can exhaust a 16GB machine even if the model itself fits.

Step 1: Measure Your Current Memory Usage

Before changing anything, you need a baseline. Use the free command to see total, used, and available memory. The important number is 'available', which accounts for buffers and cache that the kernel can reclaim.

Run this command to get a snapshot of your memory situation.

free -h

• Look at the 'available' column under 'Mem'. That is how much memory you can actually use for a model.
• If available is less than the model size plus overhead, you will likely hit OOM.
• Run top or htop to see what else is consuming memory before you start the model.

Step 2: Calculate Model Memory Requirements

You can estimate the minimum RAM needed for a model using this formula: parameters * bytes per parameter. For 16-bit, that is 2 bytes; for 8-bit, 1 byte; for 4-bit, 0.5 bytes.

Add 20-30% for runtime overhead. Let us calculate for a 7B model in different precisions.

python3 - <<'EOF'
params = 7e9
for bits, name in [(16, 'fp16'), (8, 'int8'), (4, 'int4')]:
    bytes_per_param = bits / 8
    weights_gb = params * bytes_per_param / 1e9
    overhead_gb = weights_gb * 0.25
    total_gb = weights_gb + overhead_gb
    print(f"{name}: weights ~{weights_gb:.1f} GB, total ~{total_gb:.1f} GB")
EOF

• A 7B model in fp16 needs roughly 14GB for weights plus about 3.5GB overhead, so around 17.5GB total.
• In int8, that drops to about 8.75GB total. In int4, around 4.4GB.
• If your available memory is below these numbers, you must quantize or use a smaller model.

Step 3: Reduce Memory with Quantization

Quantization reduces the precision of the weights, trading a small amount of accuracy for a large reduction in memory. Many runtimes support quantized models out of the box.

With Ollama, you can pull a quantized version of a model by using a tag like q4_K_M. With llama.cpp, you can convert and quantize a model yourself using the provided scripts.

Here is how to run a 7B model in 4-bit with Ollama.

ollama pull llama2:7b-q4_K_M
ollama run llama2:7b-q4_K_M

• The q4_K_M tag is a popular 4-bit quantization that balances quality and size.
• The model will use roughly 4-5GB of RAM instead of 14GB.
• Check the exact memory usage with free -h while the model is loaded.

Step 4: Use Memory Mapping and Offloading

If you are using llama.cpp directly, you can use mmap to map the model file into memory, which lets the OS page in only the parts that are used. This can reduce peak RAM usage.

You can also offload some layers to the GPU if you have one. Use the -ngl flag to specify the number of layers to offload. Start with a small number and increase until you hit VRAM limits.

Example command for llama.cpp with mmap and GPU offloading.

./main -m models/llama-2-7b.Q4_K_M.gguf -ngl 20 --mmap --prompt "Hello"

• The --mmap flag is often enabled by default, but you can force it.
• The -ngl value depends on your GPU VRAM. Start at 10 and increase if you do not run out of memory.
• If you have no GPU, set -ngl to 0.

Step 5: Enable Swap as a Safety Net

Swap space on disk can act as an overflow buffer. When RAM is full, the kernel moves less-used pages to swap, preventing immediate OOM crashes. However, swapping is slow and will degrade performance.

You can create a swap file if you do not have one. Here is how to add a 8GB swap file on Linux.

sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

• Swap is not a solution for heavy workloads, but it can prevent crashes when you briefly exceed RAM.
• Monitor swap usage with free -h. If you see heavy swapping, reduce the model size or close other apps.
• To make the swap permanent, add an entry to /etc/fstab.

Recommended Setup for 16GB RAM Machines

Based on the calculations and practical tests, here is a setup that works reliably on a typical 16GB machine: use 4-bit quantized models up to 7B parameters, keep your browser and other heavy apps closed, and use swap as a fallback.

Here is a copy-paste starter for Ollama that sets a sensible context length and keeps memory usage predictable.

ollama run llama2:7b-q4_K_M --num-ctx 2048

• The --num-ctx flag limits the context size, which reduces the key-value cache memory.
• A context of 2048 tokens is enough for most interactive tasks.
• If you still run out of memory, try a 3B model or use a GGUF with a lower bit rate like q2_K.

Troubleshooting Common OOM Scenarios

If you still see out-of-memory errors, check the following: the model is not actually quantized, the context size is too large, or other processes are eating RAM.

Use the command below to see which processes are using the most memory before you start the model.

ps aux --sort=-%mem | head -10

• Kill any unnecessary processes that are using more than 1GB of RAM.
• Verify the model file size on disk. If it is 13GB, it is not a 4-bit quantized version.
• Try reducing the context size or batch size in your inference script.

FAQ

Answers to the questions that come up most often on this topic.

• Q: Can I run a 13B model on 16GB RAM? A: Only with aggressive quantization (4-bit or lower) and a small context. It will be tight and may swap.
• Q: Why does my GPU not help? A: The GPU only helps if you offload layers with -ngl. The CPU still needs to hold the rest of the model in RAM.
• Q: Does more RAM always help? A: Yes, but only up to a point. If you have enough RAM for the model, performance depends on CPU and disk speed.
• Q: What is the best quantization? A: For most tasks, q4_K_M offers a good balance. q8_0 is more accurate but uses twice the memory.
• Q: Can I run models without quantization? A: Yes, but you need a machine with enough RAM. A 7B fp16 model needs about 14GB just for weights.

Try it on code.live

If you need to quickly convert a model file or generate a configuration, try the JSON Formatter or YAML Formatter on code.live to clean up your config files before running your model.