SafeTool: Sandboxed Execution for LLM Agent Tool Calls

SafeTool: Sandboxed Execution for LLM Agent Tool Calls

Overview

SafeTool is the execution-safety layer inside QUACK. It governs how agent-initiated tool calls are validated, isolated, and — where necessary — reversed. The goal is a simple invariant: no agent action reaches an external system without a checked permission, a bounded blast radius, and a recovery path for anything irreversible.

Threat Model

Agent tool execution differs from traditional software because the caller (an LLM) is non-deterministic and can be influenced by untrusted input (prompt injection through retrieved documents, tool outputs, or user content). SafeTool assumes the planner may attempt any tool with any arguments, and treats every call as untrusted until validated.

Classification of Tools

Every registered tool carries a safety classification:

• Read-only — no external state change (search, retrieve, compute). Lowest gate.
• Reversible write — state change with a defined inverse (create draft, add tag). Requires permission check and an undo record.
• Irreversible — no inverse (send email, execute payment, delete record). Requires explicit approval policy and pre-execution confirmation.

Static Analysis

Before execution, SafeTool statically inspects the proposed call:

1. Argument schema validation against the tool contract.
2. Policy evaluation against the current runtime context (role, scope, environment).
3. Taint analysis — arguments derived from untrusted content are flagged and, for irreversible tools, routed to human approval.

Runtime Monitoring

During execution, tools run inside a constrained sandbox with:

• Network egress allowlists per tool.
• Resource ceilings (time, memory, call count).
• Structured audit records capturing inputs, outputs, and the agent's stated reasoning.

Rollback & Compensation

For reversible writes, SafeTool records a compensating action at execution time. If a downstream step fails or a human rejects a plan mid-flight, the runtime replays compensations in reverse order. Irreversible operations never execute without a satisfied approval policy, so they are excluded from automatic rollback by design.

Human Approval Integration

Approval gates are first-class. A pending irreversible call suspends the agent, surfaces the exact operation and its arguments to a human, and resumes only on explicit approval — with the decision written to the audit log.

Evaluation

Across an internal red-team suite of injection scenarios, SafeTool blocked 100% of attempts to trigger irreversible tools from untrusted content without approval, while adding a median of 6ms of overhead to read-only calls.

Future Work

Extending taint tracking across multi-agent handoffs, and formalizing approval-policy composition so organizations can layer team, project, and regulatory rules without conflict.