How to Stop an AI Agent From Going Rogue
Learn practical guardrails for AI agents: sandboxing, permission scopes, tool schemas, and observability. Includes runnable code.
Before you start
You built an AI agent that can browse the web, run shell commands, or call internal APIs. It works in demos. Then, in production, it deletes a database row, exfiltrates an API key, or loops on a costly API call. This article is about preventing that.
We will cover concrete guardrails: sandboxed execution, least-privilege permissions, strict tool schemas, and observability. You will get code you can adapt to your stack. We assume you have basic knowledge of JavaScript and Python, and that you have Node.js 18+ and Python 3.9+ installed.
- Node.js 18+ for the TypeScript examples
- Python 3.9+ for the sandbox example
- A terminal with curl for testing
- An OpenAI API key or similar LLM provider key (optional for some examples)
The core problem: agents are autonomous by design
An agent is a loop: the LLM decides which tool to call, the runtime executes it, and the result feeds back into the loop. The danger is that the LLM may misinterpret instructions or be manipulated by prompt injection. Without guardrails, a single bad decision can cause real damage.
The solution is not to remove autonomy, but to constrain it. You need to enforce boundaries at every layer: what the agent can do, what it can see, and what it can change.
Step 1: Define a strict tool schema
The first line of defense is the tool interface. If your agent can only call a few well-defined functions with strict parameters, the blast radius is small. Avoid giving it a generic 'run_command' tool unless absolutely necessary.
Here is a TypeScript example of a safe tool set for a file-management agent. Notice that the schema limits paths to a specific directory and only allows certain operations.
// tools.ts
export const tools = [
{
name: 'read_file',
description: 'Read a file from the workspace. Only files under /workspace are allowed.',
parameters: {
type: 'object',
properties: {
path: { type: 'string', pattern: '^/workspace/[a-zA-Z0-9_./-]+$' }
},
required: ['path']
}
},
{
name: 'write_file',
description: 'Write a file to the workspace. Only files under /workspace are allowed.',
parameters: {
type: 'object',
properties: {
path: { type: 'string', pattern: '^/workspace/[a-zA-Z0-9_./-]+$' },
content: { type: 'string', maxLength: 10000 }
},
required: ['path', 'content']
}
},
{
name: 'list_files',
description: 'List files in a directory under /workspace.',
parameters: {
type: 'object',
properties: {
dir: { type: 'string', pattern: '^/workspace/[a-zA-Z0-9_./-]*$' }
},
required: ['dir']
}
}
];Step 2: Enforce a permission layer
Even with a strict schema, the agent might try to do something harmful. You need a permission layer that checks every tool call against a policy. This can be a simple allowlist or a more sophisticated policy engine.
Here is a Python decorator that wraps tool functions and checks if the action is allowed. It uses a policy file that defines allowed actions per tool.
# permissions.py
import json
import os
from functools import wraps
POLICY_FILE = 'policy.json'
def load_policy():
with open(POLICY_FILE) as f:
return json.load(f)
def check_permission(tool_name, params):
policy = load_policy()
allowed = policy.get(tool_name, [])
# Example: for 'write_file', check if path is in allowed directories
if tool_name == 'write_file':
for rule in allowed:
if params['path'].startswith(rule['path']):
return True
return False
return True
def require_permission(tool_func):
@wraps(tool_func)
def wrapper(params):
if not check_permission(tool_func.__name__, params):
raise PermissionError(f"Permission denied for {tool_func.__name__}")
return tool_func(params)
return wrapper- Policy file example: a JSON file that maps tool names to allowed parameters.
- For write_file, only allow paths under /workspace, not /etc or /home.
- Always fail closed: if no rule matches, deny the call.
Step 3: Sandbox execution
Run the sandbox with no network and a read-only filesystem except for the workspace directory. This prevents the agent from exfiltrating data or modifying system files.
docker build -f Dockerfile.sandbox -t agent-sandbox .
docker run --rm --network none -v /tmp/agent-workspace:/workspace agent-sandbox- Use --network none to disable network access.
- Mount only a temporary directory as /workspace.
- Run as non-root user to limit file system permissions.
Step 4: Add a human-in-the-loop checkpoint
For high-impact actions, require human approval. This is critical for actions like sending emails, deleting data, or making purchases. Implement a confirmation step in your agent loop.
Here is a Node.js example that pauses the agent and asks for confirmation before executing a destructive tool.
// confirm.js
const readline = require('readline');
async function confirmAction(description) {
const rl = readline.createInterface({
input: process.stdin,
output: process.stdout
});
return new Promise((resolve) => {
rl.question(`Allow action: ${description}? (y/n) `, (answer) => {
rl.close();
resolve(answer.toLowerCase() === 'y');
});
});
}
// Example usage in agent loop
async function executeTool(toolName, params) {
if (toolName === 'delete_file') {
const ok = await confirmAction(`Delete file ${params.path}?`);
if (!ok) {
console.log('Action denied by user.');
return;
}
}
// Actually execute tool
console.log(`Executing ${toolName}`);
}Step 5: Set a budget and rate limits
Agents can burn through tokens or API credits if they loop. Set a budget for each run and enforce it. Also limit the number of tool calls per run.
Here is a Python class that tracks token usage and stops the agent when the budget is exceeded.
# budget.py
class BudgetTracker:
def __init__(self, max_tokens, max_calls):
self.max_tokens = max_tokens
self.max_calls = max_calls
self.tokens_used = 0
self.calls = 0
def add_tokens(self, count):
self.tokens_used += count
if self.tokens_used > self.max_tokens:
raise RuntimeError("Token budget exceeded")
def add_call(self):
self.calls += 1
if self.calls > self.max_calls:
raise RuntimeError("Call limit exceeded")
# Usage
budget = BudgetTracker(max_tokens=10000, max_calls=50)
try:
# In agent loop
budget.add_tokens(response.usage.total_tokens)
budget.add_call()
except RuntimeError as e:
print(f"Agent stopped: {e}")Step 6: Log everything for audit
You cannot fix what you cannot see. Log every tool call, input, output, and decision. Use structured logging so you can replay the agent's behavior.
Here is a simple JSON logger for tool calls.
// logger.js
const fs = require('fs');
function logToolCall(toolName, params, result, error) {
const entry = {
timestamp: new Date().toISOString(),
tool: toolName,
params,
result,
error: error ? error.message : null
};
fs.appendFileSync('agent.log', JSON.stringify(entry) + '\n');
}
// In agent loop
logToolCall('write_file', { path: '/workspace/test.txt' }, { success: true }, null);- Include a run ID to correlate logs from one agent run.
- Log the full LLM response, including the raw prompt if possible.
- Store logs in a central place like ELK or CloudWatch.
Recommended setup: a complete guardrail stack
Here is what I would put in place for a production AI agent. This is a copy-paste starter that combines the pieces above.
Create a policy.json file that defines allowed operations, and use a wrapper that enforces it. Then run the agent in a Docker sandbox with a budget.
{
"read_file": [{"path": "/workspace"}],
"write_file": [{"path": "/workspace"}],
"list_files": [{"path": "/workspace"}],
"delete_file": [{"path": "/workspace"}]
}# start_sandbox.sh
#!/bin/bash
# Build and run the sandbox with limits
docker build -f Dockerfile.sandbox -t agent-sandbox .
docker run --rm \
--network none \
-v $(pwd)/workspace:/workspace \
-e MAX_TOKENS=10000 \
-e MAX_CALLS=50 \
agent-sandboxTroubleshooting
If the agent still goes rogue, check these common issues.
Is the agent calling tools outside the allowed set? Verify the tool schema is enforced at runtime, not just in documentation.
Are you using a sandbox? If not, start there. It is the most effective single measure.
Are your logs complete? Without logs, you are blind.
Did you set a budget? A runaway loop can be stopped by a budget.
- Test with a malicious prompt to see if your guardrails hold.
- Use a canary token to detect exfiltration.
- Review your permission rules periodically.
FAQ
Answers to the questions that come up most often on this topic.
- Q: Can prompt injection bypass my tool schema? A: Yes, if the schema is too permissive. That is why you need permission checks and sandboxing.
- Q: Should I use a local LLM to reduce risk? A: Local models can help with data privacy, but they do not solve the autonomy problem. The same guardrails apply.
- Q: How do I handle multi-step tasks that require many tool calls? A: Use a budget and rate limits, and consider breaking the task into sub-agents with narrower scopes.
- Q: What is the most important guardrail? A: Sandboxing. It limits the blast radius even if everything else fails.
Next action
Pick one guardrail and implement it today. I recommend starting with a strict tool schema and a permission decorator, because they are quick and high-impact. Then add a sandbox before you deploy to production.
If you need to test your tool schemas or generate mock data for your agent, the code.live JSON Formatter and API Mock Data Generator can help you validate inputs quickly.
Key takeaways
- Apply one concrete change from this post before collecting more reading.
- Prefer browser-side tools when the work involves secrets, tokens, or PII.
- Document the why next to the how so the next reviewer inherits context.
FAQ
- Who is this guide on ai-agents for?
- Working developers who need a practical take on how to stop an ai agent from going rogue — not a marketing overview. Skim the sections, apply one tip, then come back when you hit an edge case.
- Do I need an account to use the related tools?
- No. code.live tools run in your browser with no signup. Nothing you paste is uploaded to a server for the client-side utilities linked from this post.
- How often is this article updated?
- This post was published November 21, 2026. Fundamentals stay stable; check linked tool pages and official docs when version-specific behavior matters.