Test malformed AI tool calls before they break a production agent

A malformed tool call can fail after the model request itself returns successfully. If your test suite only covers network errors and valid tool arguments, it can miss the failure that leaves a user waiting, executes an unsafe action, or makes the next turn unusable.

A useful recovery test controls the bad response, runs it through your real client and application code, then checks what your app did. CopilotKit's aimock model-misbehavior faults can supply repeatable adverse outputs for that test. They do not implement your retry or repair path; that remains application behavior you must assert.

Test the failure your app actually ownsCopy Anchor Link

An agent tool call crosses several boundaries. The model response arrives, the client decodes it, your application parses arguments, validates them, authorizes the requested action, and decides whether a tool can run. Then your conversation code stores the result and prepares the next turn.

A successful HTTP status proves only that a response arrived. It does not prove that its arguments are valid JSON, match your schema, name an available tool, or leave the conversation reusable. A request timeout exercises a different branch. A successful-fixture test misses the error path altogether.

Aimock gives your test control over model outputs that represent faults such as invalid tool-argument JSON. Your real application still processes that response. This makes the test useful for questions such as:

• Did argument parsing reject the broken value?
• Was the tool prevented from running with invalid input?
• Did your retry limit prevent a loop?
• Could the agent continue after the failure?
• Did the user interface leave a spinner or an empty reply behind?

Conceptual illustration only; no CopilotKit/aimock interface or OTF integration is shown or included.

Configure one malformed responseCopy Anchor Link

The CopilotKit example starts an LLMock server with model misbehavior enabled and a fault for the weather tool. With times: 1, the first matching request receives truncated JSON arguments; the fixture's original arguments are available on the next matching request. Your client points at the mock server's OpenAI Chat-compatible /v1 endpoint.

import { LLMock } from "@copilotkit/aimock";

const mock = new LLMock({
  port: 0,
  enableMisbehavior: true,
  misbehavior: {
    seed: 42,
    faults: [{
      fault: "tool-args-invalid-json",
      style: "truncated",
      tool: "weather",
      providers: ["openai-chat"],
      rate: 1,
      times: 1,
    }],
  },
});

mock.onMessage("weather", {
  toolCalls: [{
    name: "weather",
    arguments: '{"city":"Paris"}',
  }],
});

Start the mock during test setup, route the real client to its URL, and send the same X-Test-Id on the initial request and retry. Stop the server in cleanup. Keep test data synthetic: the request journal records prepared tool names, argument strings, emitted IDs, and why candidate faults were skipped.

Assert each side effectCopy Anchor Link

For an app that retries invalid arguments once, assert three outcomes together. First, the malformed call does not execute the tool. Second, the retry uses the original valid fixture arguments. Third, the tool runs exactly once with the expected parsed input, such as { city: "Paris" }.

Those checks guard separate bugs. If you assert only that the second response was successful, the tool may already have run once with unsafe data. If you assert only that the tool did not run on the first response, the retry may repeat forever. If you check only a final answer, the agent might have produced it without invoking the intended tool.

Also verify how your conversation history behaves. If your implementation persists raw assistant tool-call arguments, make sure the rejected value does not poison the next request. Your test should exercise the application's real history handling: repair, remove, or restart the affected turn according to your design. Aimock can provide the malformed response, but it does not decide how your agent repairs its state.

The distinction matters when a provider rejects later requests. A reported failure pattern is that malformed arguments remain in history and subsequent requests fail. Aimock can trigger the initial bad arguments so your own code can be tested; this feature does not automatically simulate a provider's later HTTP 400 or clear the conversation for you.

Keep retries bounded and visibleCopy Anchor Link

A recovery test should define its stop condition before it defines its retry. Decide which errors are retryable, how many attempts are allowed, and what the user sees when recovery fails. For example, a single retry may be reasonable for a transient malformed argument; an unknown tool or a denied permission may require a clear error instead of another model call.

Use assertions on application effects, not only mock responses. Confirm invalid arguments never reach the side-effecting function. Confirm a retry count stays within its budget. Confirm the final UI exits its pending state and the conversation can accept another user turn. A local mock can make those branches repeatable; it cannot prove that every provider emits the same failure in production.

Select a fault that fits the boundaryCopy Anchor Link

Aimock's misbehavior set includes invalid JSON, schema violations, unknown tool names, duplicate tool-call IDs, length stops during tool calls, empty responses, refusals, content filtering, and reasoning-only output. Select one fault per test so a failure points to a clear boundary. The fault engine applies at most one fault to each response.

Configuration can be set at server, test-ID, fixture, or request-header scope. The highest-priority source wins as a whole. Rates control how often a candidate fires; limits control the number of successful firings. A numeric seed can make rate decisions repeatable for the same test ID, configuration source, fault entry, and evaluation order. Concurrent requests can change that order, so a seed does not promise deterministic scheduling for parallel tests.

Check the support matrix before relying on a fault. Aimock documents formats across OpenAI, Anthropic, Bedrock, Gemini, Cohere, and Ollama APIs; Azure OpenAI and OpenRouter use the OpenAI Chat format. Availability depends on the fault, API format, and streaming mode. Unsupported fixture or request-header configurations fail explicitly, while broader server and runtime configurations can skip an unsupported format and record why.

The request journal is useful evidence that a fault was prepared and shows where the configuration came from. It is not proof that your client received or handled the response correctly. Keep client-side assertions in place.

Add the test to your agent workflowCopy Anchor Link

Start with a tool whose accidental execution would matter. Choose one failure mode, write the expected safe behavior, and make the test exercise your real client and recovery code. Then add adjacent cases for history corruption, unknown tools, schema failures, or empty responses where they match your product's risk.

For an existing app team, this is a focused test-harness addition rather than a reason to replace the app architecture. The SaaS Dashboard starter is an optional editable UI and API starting point; it does not include CopilotKit, aimock, or this recovery logic. The kit is listed at $99 on the pricing page. If you already have a TypeScript or full-stack app, add the mock in your existing test setup and preserve the production permission checks.

If your agent's tools can change customer records, issue refunds, or send messages, keep those operations behind the same authorization and confirmation rules in tests as in production. A malformed response should never become a shortcut around those controls. See how to design safe tool permissions for AI agents for the separate authorization question.

A controlled failure is valuable only when the test proves both that the failure occurred and that your application recovered safely. Use the journal to confirm the mock prepared the intended fault, then use your app's own assertions to confirm what it received, what it executed, and whether the next turn still works.

Conceptual illustration only; no CopilotKit/aimock interface or OTF integration is shown or included.

SourcesCopy Anchor Link

• CopilotKit: HTTP 200. Broken tool call. Now you can test it with aimock. (12 October 2026)