Build an AI Agent That Investigates Production Errors
Learn to build a practical AI agent that pulls logs, checks services, and diagnoses production errors with safe tool access.
Before you start
You are on call, and the alert says the checkout service is failing. You open the logs, but there are thousands of lines. You check the metrics, but the dashboard is slow. You need to find the root cause fast, and you are tired of clicking through the same screens. This article shows you how to build an AI agent that does the first pass of investigation for you: it pulls recent logs, checks service health, and summarizes what changed.
The agent we build is a minimal but real system. It uses the OpenAI API for the language model, runs in Node.js with TypeScript, and has three tools: read logs, check service status, and run a diagnostic command. We keep the scope small so you can extend it later. You will need Node.js 18 or newer, an OpenAI API key, and a terminal. We do not use any external agent framework, so you see exactly how the loop works.
node --version
npm init -y
npm install typescript tsx @types/node openai dotenv- Node.js 18 or newer installed on your machine
- An OpenAI API key with access to gpt-4o-mini or similar
- A local test environment with some sample logs and a fake service to inspect
- Basic familiarity with TypeScript and async/await
Step 1: Define the agent's tools
An agent is only as useful as its tools. We define three tools that match what a human on call would use: reading recent error logs, checking if a service is healthy, and running a safe diagnostic command. Each tool has a name, a description for the model, and a function that executes the action.
We use a simple JSON schema for tool definitions. The model sees these schemas and decides which tool to call, with what arguments. Our tool implementations are stubs that read from local files and call a fake health endpoint, so you can run the example without a real production environment.
// tools.ts
export type Tool = {
name: string;
description: string;
parameters: Record<string, unknown>;
execute: (args: any) => Promise<string>;
};
export const tools: Tool[] = [
{
name: 'read_logs',
description: 'Read recent log lines from a service. Returns the last N lines.',
parameters: {
type: 'object',
properties: {
service: { type: 'string', description: 'Service name, e.g. checkout' },
lines: { type: 'number', description: 'Number of lines to read, default 50' }
}
},
execute: async ({ service, lines = 50 }) => {
// In production, fetch from your log aggregator (e.g. CloudWatch, Datadog)
const fs = await import('fs');
const file = `./logs/${service}.log`;
if (!fs.existsSync(file)) return `No log file for ${service}`;
const content = fs.readFileSync(file, 'utf-8').trim().split('\n');
return content.slice(-lines).join('\n');
}
},
{
name: 'check_health',
description: 'Check if a service is healthy by calling its /health endpoint.',
parameters: {
type: 'object',
properties: {
service: { type: 'string', description: 'Service name' }
}
},
execute: async ({ service }) => {
// In production, use the real service URL
const url = `http://localhost:3000/${service}/health`;
try {
const res = await fetch(url);
return `Health check for ${service}: HTTP ${res.status}`;
} catch (e) {
return `Health check for ${service}: failed (${e.message})`;
}
}
},
{
name: 'run_command',
description: 'Run a safe diagnostic command on the host. Only whitelisted commands are allowed.',
parameters: {
type: 'object',
properties: {
command: { type: 'string', description: 'Command to run' }
}
},
execute: async ({ command }) => {
const allowed = ['df -h', 'uptime', 'free -m', 'ps aux | head -20'];
if (!allowed.includes(command)) return 'Command not allowed';
const { execSync } = await import('child_process');
return execSync(command).toString();
}
}
];Step 2: Build the agent loop
The core of the agent is a loop: send the conversation to the model, if the model requests a tool call, execute it and feed the result back, then let the model continue. We keep the loop simple and explicit so you can add logging, timeouts, or human approval later.
We use the OpenAI chat completions API with function calling. The model returns a tool_calls array when it wants to use a tool. We execute each call and append the results as a new message with role tool. Then we call the model again. We stop when the model returns a final answer without tool calls.
// agent.ts
import OpenAI from 'openai';
import { tools } from './tools';
const openai = new OpenAI();
export async function runAgent(problem: string) {
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{ role: 'system', content: 'You are a production incident investigator. Use the available tools to gather evidence and then provide a concise root cause analysis with evidence.' },
{ role: 'user', content: problem }
];
const toolDefs = tools.map(t => ({
type: 'function' as const,
function: {
name: t.name,
description: t.description,
parameters: t.parameters
}
}));
for (let i = 0; i < 10; i++) {
const response = await openai.chat.completions.create({
model: 'gpt-4o-mini',
messages,
tools: toolDefs,
tool_choice: 'auto'
});
const msg = response.choices[0].message;
messages.push(msg);
if (msg.tool_calls) {
for (const call of msg.tool_calls) {
const tool = tools.find(t => t.name === call.function.name);
if (!tool) continue;
const args = JSON.parse(call.function.arguments || '{}');
const result = await tool.execute(args);
messages.push({
role: 'tool',
tool_call_id: call.id,
content: result
});
}
} else {
return msg.content;
}
}
return 'Agent did not converge.';
}Step 3: Create sample logs and a test service
To see the agent work, we need some data. Create a logs directory with a checkout.log file that contains realistic error lines. Also create a minimal HTTP server that simulates a health endpoint. We use Node's built-in http module so you do not need extra dependencies.
The log file should have a mix of normal lines and error lines. The agent will read the last few lines and look for patterns. Our fake health server returns 200 for one service and 500 for another, so the agent can correlate.
mkdir -p logs
cat > logs/checkout.log << 'EOF'
2024-01-15T10:00:01Z INFO Checkout started for order 12345
2024-01-15T10:00:02Z DEBUG Cart loaded
2024-01-15T10:00:03Z ERROR Payment failed: timeout connecting to payment gateway
2024-01-15T10:00:04Z ERROR Retry attempt 1 failed
2024-01-15T10:00:05Z ERROR Checkout aborted for order 12345
EOF
cat > logs/payments.log << 'EOF'
2024-01-15T10:00:03Z ERROR Connection timeout to payment gateway
2024-01-15T10:00:03Z WARN Retrying in 100ms
EOF// server.js
const http = require('http');
const server = http.createServer((req, res) => {
const url = new URL(req.url, `http://${req.headers.host}`);
if (url.pathname === '/checkout/health') {
res.writeHead(200);
res.end('ok');
} else if (url.pathname === '/payments/health') {
res.writeHead(500);
res.end('error');
} else {
res.writeHead(404);
res.end('not found');
}
});
server.listen(3000, () => console.log('Test server on 3000'));Step 4: Run the agent
Now we wire everything together. Create an index.ts that starts the test server, waits a moment, and then calls runAgent with a problem description. The agent should read the checkout log, notice the payment error, check the payments service health, and conclude that the payment gateway is down.
Run the server in one terminal and the agent in another. The agent output will show the tool calls and the final analysis. If you do not have an OpenAI key, the code will fail; set the OPENAI_API_KEY environment variable in a .env file.
// index.ts
import { runAgent } from './agent';
async function main() {
// Start the test server (in a real setup you would run this separately)
const { spawn } = await import('child_process');
const server = spawn('node', ['server.js']);
await new Promise(r => setTimeout(r, 1000));
const problem = 'Checkout service is failing for orders. Investigate the root cause.';
const result = await runAgent(problem);
console.log('\nAgent analysis:\n', result);
server.kill();
}
main().catch(console.error);- Set OPENAI_API_KEY in your .env file before running
- Run with npx tsx index.ts
- The agent should call read_logs for checkout, then check_health for payments, then read payments log
- If the agent does not call tools, increase the number of loop iterations or simplify the system prompt
Verify it worked
After running, you should see output that includes the tool calls. The final analysis should mention that the payments service is unhealthy and that the log shows connection timeouts to the payment gateway. That is the expected root cause.
To verify the agent is actually using tools, add a console.log inside the execute functions to print the tool name and arguments. You will see the sequence: read_logs for checkout, check_health for payments, read_logs for payments, and then the final answer.
npx tsx index.ts
# Expected output includes lines like:
# Calling tool: read_logs with {"service":"checkout","lines":50}
# Calling tool: check_health with {"service":"payments"}
# Agent analysis: The checkout failure is caused by the payment service...Step 5: Add safety and observability
A production agent must not run arbitrary commands or read sensitive files. Our run_command tool already whitelists commands, but you should extend that pattern. Also, log every tool call for audit and debugging. Use a simple logger that writes to stdout or a file.
Add a human-in-the-loop step for destructive actions. For example, if the agent wants to restart a service, require confirmation. In our minimal agent, we only allow read-only tools, so it is safe by design. For real use, you would add a confirmation step in the loop.
- Never allow the agent to execute arbitrary shell commands; always use a whitelist
- Add a timeout for each tool call to prevent hangs
- Log every tool call with timestamp, tool name, arguments, and result
- Consider a permission system where certain tools require an approval step
- Use a separate API key with restricted permissions for the agent
Recommended setup for production
For a real deployment, I would structure the agent as a separate service that listens for alerts. It would pull logs from your existing log aggregator, check metrics from your monitoring system, and post findings to your incident channel. The core loop stays the same; only the tool implementations change.
Here is a minimal Dockerfile and a docker-compose example to run the agent as a service. It reads logs from a mounted volume and has a health endpoint for the orchestrator.
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
EXPOSE 8080
CMD ["node", "dist/index.js"]version: '3'
services:
agent:
build: .
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
volumes:
- ./logs:/app/logs:ro
ports:
- "8080:8080"
restart: unless-stoppedTroubleshooting
If the agent does not call any tools, check that the tool definitions are correct and that the model is gpt-4o-mini or newer. Some models do not support function calling well. Also, make sure the problem statement is specific enough.
If the health check fails because the server is not running, start server.js in a separate terminal or run it in the background. The agent will still work, but it will get a connection error, which is also a valid finding.
- Tool call not happening: verify the model supports function calling and the tool schema is valid
- OpenAI API errors: check your API key and that you have credits
- Log file not found: ensure the logs directory exists and the file name matches the service name
- Health check fails: confirm the test server is running on port 3000
- Agent loop not terminating: increase the max iterations or simplify the task
FAQ
Answers to the questions that come up most often on this topic.
- Q: Can I use a different LLM provider? A: Yes, the loop is provider-agnostic. You would replace the OpenAI call with Anthropic or a local model, but the tool-calling protocol differs.
- Q: How do I add more tools? A: Define a new object in the tools array with a name, description, parameters, and an execute function. The model will discover it automatically.
- Q: Is it safe to run this in production? A: Only if you restrict tools to read-only actions and add human approval for anything that changes state.
- Q: What if the agent makes a wrong diagnosis? A: The agent provides evidence, but you should always verify its conclusions. Consider adding a confidence score or requiring a second opinion for critical actions.
Next step
You now have a working AI agent that investigates production errors. The next step is to connect it to your real log aggregator and monitoring tools. Start by replacing the read_logs implementation with a call to your log API, and the check_health with a real service discovery call.
Run the agent against a real incident and see if it finds the root cause faster than your manual process. Then add more tools, such as reading recent deployments or checking error rates, to expand its capability.
Key takeaways
- Apply one concrete change from this post before collecting more reading.
- Prefer browser-side tools when the work involves secrets, tokens, or PII.
- Document the why next to the how so the next reviewer inherits context.
FAQ
- Who is this guide on ai-agents for?
- Working developers who need a practical take on build an ai agent that investigates production errors — not a marketing overview. Skim the sections, apply one tip, then come back when you hit an edge case.
- Do I need an account to use the related tools?
- No. code.live tools run in your browser with no signup. Nothing you paste is uploaded to a server for the client-side utilities linked from this post.
- How often is this article updated?
- This post was published November 9, 2026. Fundamentals stay stable; check linked tool pages and official docs when version-specific behavior matters.