Build an AI Agent That Watches Your Server Logs

Before you start

Your logs are noisy, but buried in them are the first signs of trouble: a spike in 500s, a database connection pool running dry, or a suspicious login pattern. You could grep for known patterns, but what about the ones you don't know to look for? This article walks you through building a minimal AI agent that watches your server logs, learns normal behavior, and alerts you when something looks off.

We will build a Python-based agent that tails a log file, chunks recent lines, sends them to an LLM for analysis, and decides whether to alert. You will end with a runnable script you can point at your own logs, plus a set of rules for when to trust the AI and when to fall back to deterministic checks.

python3 --version
pip install openai python-dotenv

• Python 3.10 or newer installed on your machine or server
• An OpenAI-compatible API key (we will use a local model via Ollama for the demo, but the code works with any API)
• A sample log file to test with (we will generate one)
• Basic familiarity with the command line and Python virtual environments

Step 1: Generate a sample log file

To test the agent, you need a realistic log stream. Instead of using your production logs right away, generate a sample file that mixes normal entries with a few anomalies. This lets you verify the agent detects the anomalies without waiting for a real incident.

Run the following script to create a log file with 200 normal lines and then append a few suspicious ones. The normal lines follow a typical web server format with occasional 404s and 500s.

python3 - <<'EOF'
import random
import datetime

log_lines = []
statuses = [200, 200, 200, 200, 201, 301, 404, 500]
paths = ['/api/users', '/api/orders', '/health', '/static/js/main.js', '/login', '/checkout']

now = datetime.datetime.now()
for i in range(200):
    ts = (now - datetime.timedelta(seconds=random.randint(1, 3600))).strftime('%Y-%m-%d %H:%M:%S')
    ip = f"10.0.{random.randint(0,255)}.{random.randint(1,254)}"
    status = random.choice(statuses)
    path = random.choice(paths)
    log_lines.append(f"{ts} INFO {ip} GET {path} {status} {random.randint(100, 5000)}")

# Add anomalies: a burst of 500s and a suspicious login pattern
for _ in range(10):
    ts = (now - datetime.timedelta(seconds=random.randint(1, 300))).strftime('%Y-%m-%d %H:%M:%S')
    log_lines.append(f"{ts} ERROR 10.0.0.5 GET /api/users 500 5000")
for _ in range(5):
    ts = (now - datetime.timedelta(seconds=random.randint(1, 300))).strftime('%Y-%m-%d %H:%M:%S')
    log_lines.append(f"{ts} WARN 10.0.0.5 POST /login 401 200")

with open('sample.log', 'w') as f:
    f.write('\n'.join(log_lines) + '\n')
EOF

Step 2: Design the agent loop

A log-watching agent is not a single prompt. It is a loop that reads new lines, batches them, sends them to the LLM with a clear instruction, and acts on the response. The loop needs a memory of recent context so the LLM can spot trends, not just single lines.

We will keep the agent simple: every 10 seconds, read the last 50 lines that are new since the last check, send them to the model with a system prompt that asks for a JSON verdict, and if the verdict is 'alert', print a message and optionally send a notification. The code is structured so you can swap the LLM provider or add a webhook later.

import json
import time
from openai import OpenAI

client = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')

SYSTEM_PROMPT = """You are a log anomaly detector. Analyze the given log lines and return a JSON object with two fields: 'risk' (one of 'low', 'medium', 'high') and 'reason' (a short explanation). Look for patterns like error bursts, unusual status codes, repeated failures, or security-related entries. Be conservative: only flag 'high' if you see clear evidence of a problem."""

def analyze_logs(log_lines):
    user_content = "\n".join(log_lines)
    response = client.chat.completions.create(
        model='llama3.2',
        response_format={'type': 'json_object'},
        messages=[
            {'role': 'system', 'content': SYSTEM_PROMPT},
            {'role': 'user', 'content': user_content}
        ],
        temperature=0
    )
    return json.loads(response.choices[0].message.content)

def alert(risk, reason):
    print(f"ALERT [{risk.upper()}]: {reason}")
    # TODO: send to Slack, PagerDuty, or email

def watch_file(filename, interval=10, window=50):
    with open(filename, 'r') as f:
        f.seek(0, 2)  # start at end of file
        while True:
            lines = f.readlines()
            if lines:
                # keep only the last `window` lines
                recent = lines[-window:]
                result = analyze_logs(recent)
                if result['risk'] == 'high':
                    alert(result['risk'], result['reason'])
            time.sleep(interval)

if __name__ == '__main__':
    watch_file('sample.log')

• The agent uses a sliding window of the last 50 lines to give the model enough context.
• It tracks the file offset so it only processes new lines.
• The model returns a JSON object with a risk level and a reason, which the agent parses.
• If the risk level is 'high', the agent triggers an alert function that you can customize.

Step 3: Run the agent with a local model

For this demo, we will use Ollama to run a local LLM. This keeps your log data on your machine and avoids API costs. Install Ollama, pull a small model, and start the server.

Once Ollama is running, the agent script can connect to it using the OpenAI-compatible endpoint. The script expects the model name 'llama3.2' by default; you can change it to any model you have pulled.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2
ollama serve

Step 4: Verify it worked

Run the agent in a terminal, then in another terminal append a few high-risk lines to the sample log. The agent should detect them within one interval and print an alert.

If you do not see an alert, check that Ollama is running and that the model name matches. The agent prints nothing for low or medium risk, so you can also add a debug print to see every verdict.

python3 log_agent.py
# in another terminal:
echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR 10.0.0.5 GET /api/users 500 5000" >> sample.log
echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR 10.0.0.5 GET /api/users 500 5000" >> sample.log
echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR 10.0.0.5 GET /api/users 500 5000" >> sample.log

Step 5: Add deterministic pre-checks

An AI agent should not be your only filter. Combine it with simple deterministic rules to reduce false positives and catch known issues instantly. For example, if the error rate in the last 10 lines exceeds 30%, alert immediately without waiting for the LLM.

This hybrid approach is what I recommend for production: use regex or counters for known patterns, and use the LLM for novel anomalies. The following code adds a quick check before calling the LLM.

def deterministic_check(lines):
    errors = sum(1 for l in lines if 'ERROR' in l)
    if errors / len(lines) > 0.3:
        return {'risk': 'high', 'reason': f'Error rate {errors}/{len(lines)} exceeds 30%'}
    return None

def watch_file(filename, interval=10, window=50):
    with open(filename, 'r') as f:
        f.seek(0, 2)
        while True:
            lines = f.readlines()
            if lines:
                recent = lines[-window:]
                pre = deterministic_check(recent)
                if pre:
                    alert(pre['risk'], pre['reason'])
                else:
                    result = analyze_logs(recent)
                    if result['risk'] == 'high':
                        alert(result['risk'], result['reason'])
            time.sleep(interval)

What I would do for a production setup

For a real deployment, I would not run this script as a bare Python process. I would wrap it in a systemd service, use a proper logging library instead of print, and send alerts to a webhook. The following systemd unit file assumes you have a virtual environment at /opt/log-agent/venv and the script at /opt/log-agent/log_agent.py.

You also want to persist the file offset across restarts so you do not reprocess the whole log on startup. The script above starts at the end each time, which is fine for a demo but loses context after a restart. For production, store the offset in a small state file.

[Unit]
Description=AI Log Agent
After=network.target

[Service]
Type=simple
WorkingDirectory=/opt/log-agent
ExecStart=/opt/log-agent/venv/bin/python /opt/log-agent/log_agent.py
Restart=always
RestartSec=10
Environment=OLLAMA_HOST=http://localhost:11434

[Install]
WantedBy=multi-user.target

Troubleshooting

If the agent does not produce alerts, check the following common issues.

First, confirm Ollama is reachable: curl http://localhost:11434/v1/models should return a list. If not, start ollama serve. Second, verify the model name in the script matches what you pulled. Third, the response_format json_object may not be supported by all models; if you get an error, remove that parameter and parse the text output manually.

• Ollama not running: run ollama serve in the background.
• Model not found: run ollama list to see available models and update the script.
• JSON parse errors: some models wrap the JSON in markdown; strip backticks before parsing.
• High CPU usage: reduce the check interval or window size.
• Missed anomalies: lower the threshold in deterministic_check or adjust the system prompt to be more sensitive.

FAQ

Can I use this with a cloud LLM like GPT-4o? Yes, change the base_url and api_key to point to OpenAI or another provider, and adjust the model name.

How do I avoid alert fatigue? Start with a high threshold and only alert on 'high' risk. Add a cooldown so the same issue does not alert every interval.

Does this replace a proper APM tool? No, it is a supplement. Use it for log analysis that rule-based tools miss.

Can I watch multiple log files? Yes, run multiple instances with different file paths and systemd services.

What about privacy? Using a local model keeps logs on your machine. If you use a cloud API, ensure your logs do not contain sensitive data.

Next step

Create the log_agent.py file with the code from Step 2, add the deterministic check from Step 5, and run it against your sample.log. Then point it at a real log file and observe what it flags. The agent is a starting point; extend it to send alerts to Slack, store verdicts in a database, or retrain the prompt based on your own log patterns.