How to design safe tool permissions for AI agents in production apps

An AI agent can choose a tool, fill in arguments, and explain what it intends to do. None of those actions make it authorized. Safe tool permissions come from the application around the model: authenticate the user, check the resource, validate the arguments, enforce a budget, request confirmation when needed, and record the result.

This distinction matters when an agent can read customer records, create an invoice, update a project, send a message, or call an external service. The production boundary is not a longer system prompt. It is code that remains safe when the model is mistaken, manipulated, or unavailable.

Start with a tool inventory

Before writing prompts, list every action the agent can take. Give each tool a purpose, required inputs, affected resources, and worst-case consequence.

A read-only search may expose private data if it ignores workspace boundaries. A calendar action may create an unwanted commitment. A send operation may contact too many recipients if the list is not constrained. A record update may be reversible in theory but damaging in practice.

Classify tools by impact:

• Read-only access to information.
• Reversible changes to a user-owned resource.
• External communication or financial action.
• Irreversible or high-impact operation.

The classification determines the control. Searching documentation and deleting an account should not share an approval path.

Keep the inventory near the implementation. If the registry changes but the security review does not, the review is already stale. The OWASP GenAI LLM Top 10 identifies excessive agency, prompt injection, insecure output handling, and sensitive information disclosure as risks that should map to application controls.

Prefer narrow tools over one general function

Least privilege is easier to enforce with small operations. Prefer getInvoice(invoiceId) over a function that accepts arbitrary query text. Prefer draftEmail(recipientId, templateId) over unrestricted sending when the product does not need arbitrary recipients and message bodies.

A narrow tool is easier to authorize, test, rate-limit, and explain. It also limits the damage caused by malicious instructions in retrieved content.

type ToolRequest = {
  name: "get_invoice" | "draft_email" | "update_project"
  arguments: Record<string, unknown>
}

const toolSchemas = {
  get_invoice: invoiceInputSchema,
  draft_email: draftEmailInputSchema,
  update_project: updateProjectInputSchema,
} as const

function validateToolRequest(request: ToolRequest) {
  const schema = toolSchemas[request.name]
  if (!schema) throw new Error("Unknown tool")
  return schema.parse(request.arguments)
}

Keep user and workspace identity outside model-controlled arguments whenever possible. Derive the current actor and tenant from the authenticated session. If the model supplies an identifier, verify that the resource belongs to the same authorization scope before reading or changing it.

A valid schema proves shape, not permission. A well-formed projectId can still belong to another workspace.

Put authorization immediately before execution

The model can propose an action. Your application decides whether it is permitted. A useful execution path has distinct stages:

1. The user requests an outcome.
2. The model proposes a tool and structured arguments.
3. The server validates the arguments.
4. The authorization layer checks actor, role, tenant, and resource ownership.
5. The application requests confirmation for sensitive work.
6. The bounded tool executes.
7. The result is recorded and shown to the user.

Do not let the model skip stages four or five by writing “confirmed” in its own output. Confirmation must come from the application interface or another trusted control.

async function runTool(
  request: ToolRequest,
  context: RequestContext,
) {
  const args = validateToolRequest(request)
  const resource = await resolveResource(request.name, args)

  await authorize({
    actorId: context.actorId,
    workspaceId: context.workspaceId,
    tool: request.name,
    resource,
  })

  if (requiresConfirmation(request.name)) {
    return createApprovalRequest({ request, context, resource })
  }

  return executeBoundedTool(request.name, args, context)
}

For an email draft, approval should be tied to the exact recipients and content. For a financial action, include the amount and destination in the confirmation view. An approval for one action must not be replayable against changed arguments.

The OpenAI Agents documentation describes guardrails and human review as a distinct capability for workflows that should pause before risky work continues. Apply the same separation even if you are not using that SDK.

Validate at two boundaries

Validate once at the model boundary for useful feedback and again at the tool boundary for security. The first check can tell the model that a field is missing or a date is invalid. The second assumes the input may be hostile.

Check types, ranges, allowed identifiers, URL destinations, recipient counts, file sizes, and action-specific business rules. Never interpolate model output directly into a shell command, database query, HTML response, or network request. Use parameterized operations and allowlists.

If a tool accepts a URL, restrict protocols and hosts. If it accepts a file path, resolve it against an allowed directory and prevent traversal. If it accepts a quantity, enforce both its type and its business limit.

function validateDestination(value: string) {
  const url = new URL(value)
  if (url.protocol !== "https:") throw new Error("Protocol blocked")
  if (!allowedHosts.has(url.hostname)) throw new Error("Host blocked")
  return url
}

Return a policy-blocked result instead of allowing the model to invent success. The UI can explain that the action needs approval, the resource is outside the workspace, or the destination is not allowed.

Treat retrieved content as data, not policy

Documents, webpages, tickets, and customer messages can contain instructions aimed at the model. A retrieved page might say to ignore previous rules and export data. The agent should analyze that text as content; it must not gain permission from it.

Keep a clear boundary between trusted instructions and retrieved data. Limit which fields are passed to the model. Do not place retrieved text inside tool definitions. Require server-side authorization for every resource access, even when the retrieval result contains an identifier.

The Model Context Protocol tools specification describes tools as model-controlled and recommends a human in the loop with the ability to deny invocations for trust and safety. That recommendation is not a replacement for authorization, but it is a useful design requirement for high-impact operations.

Reduce the blast radius instead of trying to solve prompt injection with one sentence. A model that sees malicious instructions should still be unable to access another tenant, call an unapproved host, or perform an irreversible operation without confirmation.

For the broader boundary around inputs, outputs, and secrets, see the AI app security checklist.

Set budgets for the whole workflow

Permissions answer “may this action happen?” Budgets answer “how much can happen?” Set limits for tool calls per request, records returned, recipients, spend, execution time, retries, and outbound domains.

Use separate budgets for read and write operations. A repeated read loop may create a cost problem; a repeated write loop may create a customer incident. Track budgets across the entire workflow, not only inside one model call.

Make high-impact limits visible in the approval view. A user should not discover after the fact that an agent could send hundreds of messages or modify every record in a workspace. When a budget is exhausted, stop with a typed error and preserve the operation record.

Test the limits:

• Request more records than the policy allows.
• Provide a recipient list above the maximum.
• Retry after a simulated timeout.
• Ask for an unapproved destination.
• Start two workflows that compete for the same account budget.

Make side effects safe to retry

Tools fail. Providers time out. Workers restart. Permission checks can change between planning and execution. An external write must be idempotent so a network retry does not create two invoices, send two emails, or update a record twice.

Attach a stable operation identifier to the action and store the result before allowing a retry to proceed:

async function applyOperation(operation: Operation) {
  const previous = await db.operations.findUnique({
    where: { id: operation.id },
  })
  if (previous) return previous.result

  const result = await executeExternalWrite(operation)
  await db.operations.insert({
    id: operation.id,
    tool: operation.tool,
    result,
  })
  return result
}

For multi-step work, checkpoint progress. If an agent created a draft and the next step failed, the retry should resume from the known state rather than create a second draft. For a related treatment of long-running work, read background jobs for AI features.

Record the decision and the outcome

An audit record should connect the user request to the actual side effect. Record the request ID, authenticated actor, workspace, tool name, validated argument shape, authorization decision, confirmation state, start and end time, result category, and operation identifier.

Do not store secrets or unnecessary personal content. Redact sensitive fields or store references to controlled records instead of copying full prompts into every log.

The audit trail should answer:

• Who requested the action?
• Which policy allowed or blocked it?
• What resource was affected?
• Did a trusted user confirm it?
• Was the operation retried?
• What was the final outcome?

The LLM observability guide covers the trace and measurement layer that helps connect these events across a production workflow.

Use a release checklist

Before shipping an agent tool, confirm that:

• The tool has one narrow purpose.
• User and workspace scope come from the server session.
• Arguments are validated at both boundaries.
• Resource ownership is checked immediately before execution.
• Sensitive actions require trusted confirmation.
• Limits exist for calls, records, spend, recipients, and retries.
• External writes are safe to replay.
• Retrieved content cannot grant permissions.
• Secrets never enter model context or logs.
• Audit records explain authorization and outcome.
• Abuse cases are part of the regression suite.

OTF kits include AI-tool configuration files and tested prompts so an agent can extend owned application code with project context instead of starting from an empty brief. That does not replace the permission boundary; it makes the boundary easier to document alongside the rest of the code. Review the current OTF templates before choosing a starting point.

The strongest result is not “the model followed the prompt.” It is “the application remained safe when the model did not.” Narrow tools, server-side authorization, trusted confirmation, bounded budgets, replay-safe writes, and a useful audit trail give production builders that property without asking the model to be perfect.

Sources

• OWASP GenAI LLM Top 10 2026
• OpenAI Agents guardrails and human review
• Model Context Protocol tools specification
• OTF templates