How to Build a Private AI Assistant That Never Sends Your Code to the Cloud

Before you start

You paste a code snippet into a web chat, and the next day your employer's DLP alert fires. Or you work on proprietary algorithms and cannot risk sending a single line to a third-party API. The solution is a local AI assistant that runs entirely on your machine, using open-weight models and a local inference server.

This guide walks you through setting up a private AI assistant with Ollama as the model runner, Continue as the IDE extension, and a few configuration tweaks to ensure no data leaves your computer. By the end, you will have autocomplete, chat, and code editing powered by a model that runs on your GPU or CPU.

You need a machine with at least 8 GB of RAM for small models, or a GPU with 6+ GB VRAM for larger ones. You also need Node.js 18+ and Python 3.10+ for the tooling. All commands are for Ubuntu 22.04 or macOS, but they translate to Windows with WSL.

curl -fsSL https://ollama.com/install.sh | sh

• Install Ollama from ollama.com or via the install script
• Install Continue extension in VS Code or JetBrains
• Pull a model like codellama:7b or deepseek-coder:6.7b
• Verify the model runs locally with a test prompt
• Configure Continue to use the local Ollama endpoint

Step 1: Install and verify Ollama

Ollama is a lightweight runner for open-source models like Llama 3, CodeLlama, and DeepSeek Coder. It exposes a REST API on localhost:11434, which your IDE extension will talk to. The install script sets up a systemd service on Linux and a launchd service on macOS.

After installation, confirm the service is running and pull a model. For code completion, a 7B parameter model is a good balance of quality and speed on consumer hardware.

ollama serve &

ollama pull codellama:7b

curl http://localhost:11434/api/generate -d '{"model": "codellama:7b", "prompt": "def fibonacci(n):"}'

• If curl returns an error, check that the service is running with systemctl status ollama
• On macOS, the service starts automatically after install
• Model files are stored in ~/.ollama/models and can take several GB

Step 2: Install Continue in VS Code

Continue is an open-source AI code assistant that works with any model backend. It supports Ollama out of the box. Install the extension from the VS Code marketplace, then configure it to point to your local Ollama instance.

Open the Continue configuration file by clicking the gear icon in the Continue sidebar, or by running the command 'Continue: Open config'.

{
  "models": [
    {
      "title": "Ollama CodeLlama",
      "provider": "ollama",
      "model": "codellama:7b",
      "apiBase": "http://localhost:11434"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Ollama CodeLlama",
    "provider": "ollama",
    "model": "codellama:7b",
    "apiBase": "http://localhost:11434"
  }
}

• The apiBase must be http://localhost:11434
• Set tabAutocompleteModel to enable inline completions
• If you use a different model, change the model field accordingly

Step 3: Test the assistant with a real task

Open a project folder in VS Code. Create a new file called quick_sort.py. Type the beginning of a function and wait for the autocomplete suggestion. Then open the Continue chat panel and ask it to explain the code or refactor it.

The model runs locally, so the response time depends on your hardware. On an M1 Mac or a GTX 3060, you should see completions in under a second.

def quicksort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[len(arr) // 2]
    left = [x for x in arr if x < pivot]
    middle = [x for x in arr if x == pivot]
    right = [x for x in arr if x > pivot]
    return quicksort(left) + middle + quicksort(right)

• If autocomplete does not appear, check the Continue logs for errors
• In the chat, type 'Explain this function' and press Enter
• The response should appear without any network activity

Step 4: Verify no data leaves your machine

The whole point is privacy. You can verify that no requests go to external servers by monitoring network traffic. Use lsof to see which processes connect to the internet, or use a packet sniffer like tcpdump.

Alternatively, run Ollama in a container with no network access. That is the most airtight approach. The Docker command below runs Ollama with the network disabled, so it can only be reached from the host.

docker run -d --name ollama-private --network none -p 127.0.0.1:11434:11434 ollama/ollama

lsof -i -P | grep ollama

• The --network none flag blocks all external connections
• The -p 127.0.0.1:11434:11434 mapping exposes it only to localhost
• Use this container for sensitive projects

Step 5: Optimize for your hardware

If you have a GPU, Ollama will use it automatically. Check the logs to confirm. You can also configure the context window and thread count to avoid out-of-memory errors.

For CPU-only machines, set OLLAMA_NUM_THREADS to the number of physical cores. For GPU, set OLLAMA_GPU_LAYERS to offload more layers to the GPU if you have VRAM headroom.

export OLLAMA_NUM_THREADS=8
ollama serve

export OLLAMA_GPU_LAYERS=35
ollama serve

• Check VRAM usage with nvidia-smi on Linux
• If you get OOM errors, reduce the context length in Continue config
• The model size should roughly fit in VRAM for best performance

Recommended setup: what I would use

For a balance of quality and speed, I recommend DeepSeek Coder 6.7B for autocomplete and CodeLlama 13B for chat if you have 16 GB RAM or a GPU with 12 GB VRAM. For lower-end machines, stick with CodeLlama 7B for both.

Use the Continue config below as a starting point. It sets a reasonable context length and enables streaming for faster responses.

{
  "models": [
    {
      "title": "DeepSeek Coder",
      "provider": "ollama",
      "model": "deepseek-coder:6.7b",
      "apiBase": "http://localhost:11434",
      "contextLength": 4096
    }
  ],
  "tabAutocompleteModel": {
    "title": "CodeLlama 7B",
    "provider": "ollama",
    "model": "codellama:7b",
    "apiBase": "http://localhost:11434",
    "useStreaming": true
  }
}

• Pull deepseek-coder:6.7b with ollama pull deepseek-coder:6.7b
• Adjust contextLength based on your RAM
• Streaming reduces perceived latency

Troubleshooting

If the assistant is slow, it might be using CPU instead of GPU. Check the Ollama logs for 'offload' messages. If you see 'no GPU' or 'CPU only', your GPU is not detected.

If autocomplete does not trigger, ensure the tabAutocompleteModel is set and the model is pulled. Also check that Continue is enabled for the file type you are editing.

• Run ollama ps to see which models are loaded and their size
• If you get a 404 from the API, you have not pulled the model
• For VS Code, ensure the Continue extension is activated (look for the Continue icon in the sidebar)

FAQ

Answers to the questions that come up most often on this topic.

• Q: Can I use this with any IDE? A: Continue supports VS Code and JetBrains. For other editors, you can use the Ollama API directly.
• Q: Will this work offline? A: Yes, after pulling the model, no internet connection is needed. The Docker network-none setup guarantees it.
• Q: What is the maximum code length I can process? A: It depends on the model's context window. CodeLlama 7B supports 16K tokens, but you can reduce it to fit your RAM.
• Q: How do I switch models? A: Pull another model with ollama pull and update the Continue config.
• Q: Is there a risk of data leakage through telemetry? A: Ollama and Continue are open source and do not send telemetry by default. You can disable telemetry in both.

Next steps

You now have a private AI assistant that runs entirely on your machine. The next step is to integrate it into your daily workflow: use it for code reviews, refactoring, and writing tests. If you want to extend it, consider adding more models or using a custom system prompt.

Run the command below to confirm your setup is working, then open your project and start coding with privacy.

ollama list