How to Turn a Local AI Model Into an Agent
Learn to build a local AI agent with tool calling, memory, and safe execution using Ollama and Python.
Before you start
You have a local model running on your machine, and now you want it to do more than chat. You want it to call your APIs, read your files, and run commands. The problem is that models only return text. The missing piece is an agent loop that turns that text into actions.
This guide builds a minimal but functional local agent using Ollama and Python. You will create a tool that the model can call, wire it into a loop, and add a safety layer so the agent does not run arbitrary code without permission.
You will need Python 3.10 or newer, Ollama installed, and a model that supports tool calling. I use llama3.1:8b in the examples because it handles function calling reliably and fits in 8 GB of VRAM.
ollama pull llama3.1:8bStep 1: Define the tools your agent can use
An agent is only as useful as the tools it can call. Start with a simple tool that returns the current time. This keeps the example focused on the loop, not on complex integrations.
In Ollama's API, tools are defined as JSON Schema objects. The model sees these definitions and decides when to call a tool. You pass the schema in the request, and the response contains a tool call if the model decides one is needed.
import json
import datetime
def get_current_time():
"""Return the current date and time as a string."""
return datetime.datetime.now().isoformat()
TOOLS = [
{
"type": "function",
"function": {
"name": "get_current_time",
"description": "Get the current date and time",
"parameters": {
"type": "object",
"properties": {},
"required": []
}
}
}
]Step 2: Write the agent loop
The core of an agent is a loop: send the conversation and tool definitions to the model, check if the response contains a tool call, execute the tool, append the result, and repeat until the model returns a final answer.
This loop uses the requests library to talk to Ollama's /api/chat endpoint. The model is stateless, so you must keep the full message history and send it every time. The loop runs up to five iterations to prevent infinite cycles.
import requests
def run_agent(user_message):
messages = [{"role": "user", "content": user_message}]
for _ in range(5):
response = requests.post(
"http://localhost:11434/api/chat",
json={
"model": "llama3.1:8b",
"messages": messages,
"tools": TOOLS,
"stream": False
}
).json()
msg = response["message"]
messages.append(msg)
if msg.get("tool_calls"):
for tool_call in msg["tool_calls"]:
name = tool_call["function"]["name"]
if name == "get_current_time":
result = get_current_time()
else:
result = f"Unknown tool: {name}"
messages.append({
"role": "tool",
"content": result,
"tool_call_id": tool_call["function"]["name"]
})
else:
return msg["content"]
return "Agent stopped after too many iterations."Step 3: Run the agent and verify it works
Save the code into a file called agent.py and run it with a prompt that asks for the time. The model should recognize that it has a tool for that and call it instead of guessing.
If everything works, you will see output similar to the sample below. The agent returns the actual current time from your system, not a fabricated response.
python agent.pyCurrent time: 2025-03-14T15:22:31.123456Step 4: Add a second tool for file reading
A single tool is not enough to make an agent useful. Add a read_file tool that accepts a filename and returns its contents. This lets the agent inspect files on your machine.
Be careful: reading arbitrary files can expose sensitive data. In a real agent you would restrict paths to a specific directory. For this example, we keep it simple but note the risk.
def read_file(filename: str) -> str:
try:
with open(filename, 'r', encoding='utf-8') as f:
return f.read()
except Exception as e:
return f"Error reading file: {e}"
TOOLS.append({
"type": "function",
"function": {
"name": "read_file",
"description": "Read the contents of a file on the local system",
"parameters": {
"type": "object",
"properties": {
"filename": {
"type": "string",
"description": "Path to the file to read"
}
},
"required": ["filename"]
}
}
})Step 5: Handle tool arguments properly
The get_current_time tool takes no arguments, but read_file does. Your loop must parse the arguments from the tool call and pass them to the function.
The arguments come as a JSON string in the tool_call object. Use json.loads to convert them to a dictionary, then call the function with the correct keyword arguments.
import json
# Inside the loop, replace the tool execution block with:
for tool_call in msg["tool_calls"]:
name = tool_call["function"]["name"]
args = json.loads(tool_call["function"]["arguments"])
if name == "get_current_time":
result = get_current_time()
elif name == "read_file":
result = read_file(args["filename"])
else:
result = f"Unknown tool: {name}"
messages.append({
"role": "tool",
"content": result,
"tool_call_id": name
})Step 6: Add a permission prompt for safety
Running an agent that can read files and execute commands is dangerous. Before executing any tool, ask the user for confirmation. This is a simple guardrail that prevents the model from doing something harmful without your knowledge.
The confirmation can be as simple as printing the tool name and arguments, then waiting for y or n. In production, you would integrate this with a proper permission system.
def confirm_tool(name, args):
print(f"Tool: {name}")
print(f"Args: {json.dumps(args)}")
return input("Allow? (y/n): ").strip().lower() == 'y'
# Inside the loop, before executing:
if not confirm_tool(name, args):
result = "Permission denied by user."
else:
# execute the toolStep 7: Full agent code and run instructions
Combine all the pieces into a single file. The code below is a complete working agent with two tools and a permission prompt. Save it as agent.py and run it.
This is the recommended setup for a local agent. It is minimal but safe enough for experimenting on your own machine.
import json
import datetime
import requests
def get_current_time():
return datetime.datetime.now().isoformat()
def read_file(filename: str) -> str:
try:
with open(filename, 'r', encoding='utf-8') as f:
return f.read()
except Exception as e:
return f"Error reading file: {e}"
TOOLS = [
{
"type": "function",
"function": {
"name": "get_current_time",
"description": "Get the current date and time",
"parameters": {"type": "object", "properties": {}, "required": []}
}
},
{
"type": "function",
"function": {
"name": "read_file",
"description": "Read the contents of a file",
"parameters": {
"type": "object",
"properties": {"filename": {"type": "string"}},
"required": ["filename"]
}
}
}
]
def confirm_tool(name, args):
print(f"Tool: {name}")
print(f"Args: {json.dumps(args)}")
return input("Allow? (y/n): ").strip().lower() == 'y'
def run_agent(user_message):
messages = [{"role": "user", "content": user_message}]
for _ in range(5):
response = requests.post(
"http://localhost:11434/api/chat",
json={"model": "llama3.1:8b", "messages": messages, "tools": TOOLS, "stream": False}
).json()
msg = response["message"]
messages.append(msg)
if msg.get("tool_calls"):
for tool_call in msg["tool_calls"]:
name = tool_call["function"]["name"]
args = json.loads(tool_call["function"]["arguments"])
if not confirm_tool(name, args):
result = "Permission denied by user."
else:
if name == "get_current_time":
result = get_current_time()
elif name == "read_file":
result = read_file(args["filename"])
else:
result = f"Unknown tool: {name}"
messages.append({"role": "tool", "content": result, "tool_call_id": name})
else:
return msg["content"]
return "Agent stopped after too many iterations."
if __name__ == "__main__":
print(run_agent("What time is it?"))Verify it worked
Run the script and test both tools. First ask for the time, then ask to read a file. Confirm that the permission prompt appears before each tool call.
If the model does not call the tool, try a more explicit prompt like 'Use the get_current_time tool to tell me the time.' Some models need a nudge.
python agent.py
# Output: Tool: get_current_time
# Args: {}
# Allow? (y/n): y
# 2025-03-14T15:22:31.123456python -c "from agent import run_agent; print(run_agent('Read the file notes.txt'))"Troubleshooting
If you get a 404 error, check that Ollama is running and the model name is correct. If the model returns text instead of a tool call, your prompt may be ambiguous or the model does not support tools.
If the permission prompt appears but the agent hangs, check that you are sending the tool result back with the correct role and tool_call_id. The model needs that context to continue.
- Ollama not running: start it with 'ollama serve' or the desktop app.
- Model not found: run 'ollama list' to verify the exact name.
- Tool call not happening: use a direct prompt or try a different model like 'mistral' or 'llama3.2'.
- Loop not ending: increase the max iterations or add a stop condition when the model returns a final answer.
What I would do next
This agent is a starting point. I would extend it with a web search tool, a calculator, or a shell command executor. I would also add logging to trace every tool call and result, which helps debug when the agent misbehaves.
For production, consider using a framework like LangChain or LlamaIndex, but the core loop remains the same. The value of building it from scratch is that you understand every step and can debug it yourself.
def execute_command(command: str) -> str:
import subprocess
try:
result = subprocess.run(command, shell=True, capture_output=True, text=True, timeout=10)
return result.stdout or result.stderr
except subprocess.TimeoutExpired:
return "Command timed out."FAQ
Answers to the questions that come up most often on this topic.
- Q: Can I use a smaller model? A: Yes, try phi3 or qwen2.5:3b, but tool calling may be less reliable.
- Q: How do I add more complex tools? A: Define the function and add its schema to TOOLS. The model will discover it automatically.
- Q: What about memory? A: The message history acts as memory. For long sessions, you can summarize old messages or use a vector store.
- Q: Is this safe to run unattended? A: No. Always keep the permission prompt or restrict tools to a sandbox.
- Q: Can I use OpenAI models instead? A: Yes, but you will need to change the API endpoint and request format.
Key takeaways
- Apply one concrete change from this post before collecting more reading.
- Prefer browser-side tools when the work involves secrets, tokens, or PII.
- Document the why next to the how so the next reviewer inherits context.
FAQ
- Who is this guide on local-ai for?
- Working developers who need a practical take on how to turn a local ai model into an agent — not a marketing overview. Skim the sections, apply one tip, then come back when you hit an edge case.
- Do I need an account to use the related tools?
- No. code.live tools run in your browser with no signup. Nothing you paste is uploaded to a server for the client-side utilities linked from this post.
- How often is this article updated?
- This post was published November 2, 2026. Fundamentals stay stable; check linked tool pages and official docs when version-specific behavior matters.