How to Run a Local AI Coding Assistant

Before you start

You want AI autocomplete and chat in VS Code without sending your code to a third-party server. Maybe you have a privacy policy that forbids cloud AI, or you work offline, or you just want to avoid subscription fees. Running a local coding assistant is now practical on a consumer laptop, but you need to know which model fits your RAM and GPU.

This guide walks you through installing Ollama, pulling a code-specialized model, and wiring it into VS Code with the Continue extension. By the end, you will have autocomplete and chat working locally, plus a troubleshooting checklist for the common failures.

• A machine with at least 16 GB of RAM (8 GB works for small models but with lag).
• macOS 12+, Windows 10/11, or a Linux distro with a recent kernel.
• VS Code or a compatible editor (Cursor works, but we focus on VS Code).
• About 5 GB of free disk space for the model files.
• Basic comfort with the terminal and editing JSON config files.

Step 1: Install Ollama

Ollama is the simplest way to run LLMs locally. It handles model downloads, quantization, and a local API server. Install it with the official script on macOS and Linux, or download the installer for Windows.

curl -fsSL https://ollama.com/install.sh | sh

• On Windows, download the .exe from ollama.com and run it.
• After install, verify with: ollama --version

Step 2: Pull a code model

Ollama hosts many models. For coding, the Qwen2.5 Coder family is strong and runs well on consumer hardware. The 7B version fits in 8 GB of VRAM or system RAM. The 14B is better but needs 16 GB. Start with the 7B to confirm everything works.

Pull the model with the ollama pull command. This downloads several gigabytes, so grab a coffee.

ollama pull qwen2.5-coder:7b

• The :7b tag is the 7 billion parameter version.
• If you have a 16 GB+ machine, also pull qwen2.5-coder:14b for better completions.
• Check available models with: ollama list

Step 3: Test the model via CLI

Before wiring into VS Code, confirm the model responds. Run a quick prompt in the terminal. The response should be a code snippet or explanation, depending on your prompt.

ollama run qwen2.5-coder:7b "Write a Python function to read a CSV file and return a list of dicts."

• You should see the model stream a response.
• If you get an error about memory, close other apps and try again.
• To exit the interactive session, type /bye

Step 4: Install the Continue extension

Continue is an open-source AI assistant for VS Code and JetBrains. It supports local models via Ollama. Install it from the VS Code marketplace or use the command line.

code --install-extension continue.continue

• Alternatively, search for 'Continue' in the Extensions panel.
• After install, you will see a new Continue icon in the sidebar.

Step 5: Configure Continue to use Ollama

Continue uses a config file to define models. Open the config by clicking the gear icon in the Continue sidebar, then select 'Open Config'. Replace the content with the following to use your local Qwen model for both chat and autocomplete.

{
  "models": [
    {
      "title": "Qwen Coder 7B",
      "provider": "ollama",
      "model": "qwen2.5-coder:7b"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen Coder 7B Autocomplete",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b"
  }
}

• Make sure the model name matches exactly what you pulled.
• If you pulled the 14B, change the model field to qwen2.5-coder:14b.
• Save the file; Continue will reload automatically.

Step 6: Verify it worked

Open a code file in VS Code. Start typing a function name and see if autocomplete suggestions appear. For chat, open the Continue sidebar, type a question about your code, and hit Enter. The response should stream from your local model.

curl http://localhost:11434/api/generate -d '{"model": "qwen2.5-coder:7b", "prompt": "Hello"}'

• If autocomplete does not trigger, press Ctrl+Space to manually invoke suggestions.
• Check the Continue logs for errors: click the gear, then 'View Logs'.
• You can also test the Ollama API directly with curl to ensure the server is running.

Troubleshooting common issues

Local AI setups often hit memory, performance, or config issues. Here are the fixes for the most common problems.

• Model loads but is very slow: Close other heavy apps, or switch to a smaller model like qwen2.5-coder:1.5b.
• Out of memory error: Reduce the context window in the config, or use a quantized model (Ollama does this automatically).
• Autocomplete never fires: Ensure tabAutocompleteModel is set and the model is loaded. Try restarting VS Code.
• Continue cannot connect to Ollama: Verify Ollama is running with ollama serve, and check the API endpoint at http://localhost:11434.
• GPU not used: On Linux, install the CUDA version of Ollama. On Windows, ensure your NVIDIA drivers are current.

What I would do: recommended setup

For a balanced experience on a 16 GB Mac or PC, I recommend the 7B model for autocomplete and the 14B for chat, if your RAM allows. You can configure two models in Continue: a fast one for tab completion and a smarter one for chat. Here is a config that uses both, with the 14B for chat and the 7B for autocomplete.

{
  "models": [
    {
      "title": "Qwen Coder 14B Chat",
      "provider": "ollama",
      "model": "qwen2.5-coder:14b"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen Coder 7B Autocomplete",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b"
  }
}

• Pull both models first: ollama pull qwen2.5-coder:14b
• This setup gives you quality chat responses and quick autocomplete.
• If you have 32 GB RAM, you can run both comfortably.

FAQ

• Q: Do I need a GPU? A: No, but a GPU speeds up inference significantly. On CPU, the 7B model is usable but slower.
• Q: Can I use other models? A: Yes, Ollama has many code models like CodeLlama and DeepSeek Coder. Pull them with ollama pull.
• Q: Is my code sent anywhere? A: No, everything runs locally on your machine.
• Q: How much does this cost? A: The software is free and open source. You only pay for electricity.
• Q: Can I use this with other editors? A: Continue supports JetBrains IDEs as well. The config is similar.

Try it on code.live

If you want to experiment with JSON or YAML configs for your AI setup, code.live offers a JSON Formatter and YAML Formatter to validate your Continue config file before you paste it into VS Code.