AI Agent Tool-Use System

1. Introduction

An AI agent tool-use system lets a model-backed controller take actions through tools. The tools might search documents, send emails, update tickets, run code, call APIs, inspect files, or schedule tasks. The important word is not "autonomous." The important word is "bounded."

The naive mental model is:

give the model tools and let it solve the task

That is unsafe. Tools can read private data, write production state, spend money, send messages, delete records, or make confusing partial changes. A production agent system needs a tool registry, permission model, state machine, approval flow, idempotency strategy, and trace log. Without those boundaries, the agent loop becomes an unpredictable integration surface.

2. Product Requirements

Functional Requirements

• The system can expose tools with names, schemas, descriptions, and permissions.
• The agent can decide when to call allowed tools.
• Tool results return observations that influence the next step.
• Risky actions can require human approval.
• The system can store task state and trace history.
• The agent can stop with success, failure, clarification, or escalation.
• Operators can audit what tools were called and why.

Non-Functional Requirements

• Tool calls should be permission-checked before execution.
• Repeated attempts should not duplicate unsafe side effects.
• The agent should have step, time, and cost limits.
• Tool failures should be handled explicitly.
• Sensitive tool results should not leak into unrelated contexts.
• The system should be observable enough to debug bad outcomes.
• Evaluation should inspect tool traces, not only final answers.

3. Core Engineering Challenges

Tool use turns language-model behavior into system behavior. That raises the bar. A bad answer is a quality problem. A bad tool call can be an operational incident.

4. High-Level Architecture

flowchart LR
  User[User] --> AgentAPI[Agent task API]
  AgentAPI --> Controller[Agent controller]
  Controller --> StateStore[(Task state store)]
  Controller --> Model[LLM planner]
  Model --> ToolBroker[Tool broker]
  ToolBroker --> Registry[Tool registry]
  ToolBroker --> Policy[Permission and approval policy]
  Policy --> Tools[External tools and APIs]
  Tools --> Observation[Tool observation]
  Observation --> Controller
  Controller --> TraceStore[(Trace store)]
  Policy --> Human[Human approval]

The controller owns the loop. The model proposes a tool call or final answer. The broker validates the tool call against schema and policy. The tool executes. The observation returns. The controller decides whether to continue, ask for approval, ask the user, or stop.

5. Core Components

Agent task API: creates tasks, captures user intent, and sets permissions, budget, and risk level.

Agent controller: manages steps, state, observations, stop conditions, and retries. It should not let the model run the loop without external control.

Tool registry: lists available tools, schemas, descriptions, side-effect levels, scopes, timeout rules, and approval requirements.

Tool broker: validates arguments, applies policy, executes tools, and normalizes results.

Permission policy: decides what the agent can read or write for this user, tenant, task, and environment.

Human approval flow: pauses the loop before risky actions and records the user's decision.

State store: preserves task memory, intermediate outputs, tool results, and status.

Trace store: records steps for debugging, audit, and evaluation.

6. Data Modeling

Agent systems need to model tools and tasks explicitly.

agent_task(id, user_id, goal, status, max_steps, created_at)
agent_step(id, task_id, step_number, model_input_ref, model_output_ref)
tool_definition(id, name, schema, side_effect_level, scopes)
tool_call(id, task_id, step_id, tool_id, arguments_hash, status)
tool_observation(id, tool_call_id, result_ref, error_type)
approval_request(id, task_id, tool_call_id, status, decided_by)

Side-effect level is important:

read-only
safe-write
external-write
destructive

The exact labels can differ, but the idea should exist. A calendar lookup and a production database delete should not have the same policy.

7. Request Lifecycle

1. The user gives the agent a goal.
2. The task API records constraints: tools, budget, approval mode, and allowed data.
3. The controller builds the first model context.
4. The model chooses to answer, ask a question, or call a tool.
5. The tool broker validates tool name and arguments.
6. Policy checks scopes, side effects, and approval requirements.
7. The tool executes or waits for human approval.
8. The observation returns to the controller.
9. The loop continues until success, failure, clarification, limit, or stop condition.
10. The trace is stored for audit and evaluation.

The system should treat "I need approval" as a normal state, not an exception.

8. Scaling Problems

Agent systems scale poorly when every task becomes an open-ended loop. Each step may call a model, read data, invoke APIs, and store traces. A single user request can become dozens of downstream operations.

Common scaling problems include:

• runaway loops consuming model and tool budget
• slow tools blocking task queues
• duplicate side effects during retries
• inconsistent tool schemas across integrations
• trace storage growing quickly
• approval requests becoming noisy
• long-running tasks losing context
• high variance in latency and cost

The system needs limits: max steps, max cost, max wall-clock time, max retries, and tool-specific rate limits. Background tasks may need queues separate from interactive assistant turns.

9. Distributed Systems Concepts

State machines: agent tasks move through states such as running, waiting for approval, failed, cancelled, and completed.

Idempotency: a retried tool call should not duplicate an external write.

Sagas: multi-step workflows may need compensating actions when later steps fail.

Timeouts: every tool call needs a timeout and failure classification.

Backpressure: task queues should throttle when tools or model serving are overloaded.

Audit logs: tool-based systems need durable traces for accountability.

10. Reliability & Failure Handling

Failures need clear categories.

A model failure means the model produced invalid tool arguments or weak reasoning. A tool failure means the external system timed out, rejected the request, or returned an error. A policy failure means the agent tried something outside its authority. A progress failure means the loop is repeating without useful new information.

Good systems handle each differently. Invalid arguments can be repaired once. A forbidden action should stop or ask for approval. A repeated timeout should degrade or escalate. A loop that reaches max steps should summarize what was attempted.

Useful metrics include tool-call success rate, approval rate, denied-action rate, retries by tool, average steps per task, loop-stop reason, cost per successful task, and unsafe-action eval failures.

11. Real-World Company Approaches

Companies building agent platforms often expose tools through schemas, separate read actions from write actions, require approval for important side effects, and keep detailed traces. Enterprise products usually add tenant permissions, audit logs, and integration-specific scopes.

The common production pattern is not "let the model do anything." It is "let the model choose among permitted actions while the system enforces boundaries."

12. Tradeoffs & Alternatives

The safest design usually starts narrow and earns more autonomy through evaluation and user trust.

13. Evolution Path

1. Start with read-only retrieval tools.
2. Add structured function calling for safe internal actions.
3. Add approval-gated external writes.
4. Add task state, trace logs, and cancellation.
5. Add idempotency keys for write tools.
6. Add multi-step workflows with max-step limits.
7. Add per-tool evals and red-team cases.
8. Add policy-aware routing for different risk levels.

The system evolves from assistant to operator only when boundaries mature.

14. Key Engineering Lessons

• Tool use turns AI output into real system action.
• The controller, not the model, should own loop limits and state.
• Permissions and approvals are core architecture, not UX polish.
• Idempotency matters because retries can duplicate side effects.
• Agent evals must inspect decisions, arguments, observations, and stop behavior.

15. Related Topics

• What Is An AI Agent?
• Tool Use And Function Calling
• Agent Loops
• Model Context Protocol
• Agentic RAG
• LLM Evaluation