# Supabase Evals: Benchmarking AI Agents on Real-World Tasks

> Supabase releases an open source benchmark to evaluate AI agents on real Supabase tasks like schema building and debugging.
> By Dave · 2026-08-01
> Source: https://otf-kit.dev/blog/supabase-evals-ai-benchmark

## Why this benchmark matters more than it looks

For the last eighteen months, "AI shipped a feature" has meant the LLM emitted code that compiled and looked plausible. It did not mean the schema actually ran. It did not mean the Edge Function returned what it claimed. It did not mean the RLS policy only let through the rows the spec required. Most agent evals were vibes — a human reading diffs and going "yeah, that looks right."

Supabase's [open-source Supabase Evals framework](https://www.marktechpost.com/2026/08/01/supabase-releases-evals-an-open-source-benchmark-that-scores-claude-code-codex-and-opencode-on-real-supabase-tasks/) drops that. The benchmark boots an actual hosted-like stack and a local CLI project inside containers, points Claude Code, Codex, or OpenCode at real Supabase tasks — build a schema, debug a failed Edge Function, fix a broken RLS policy — and grades the result against the running system, not the diff. Apache-2.0, runs locally, and it already powers the public leaderboard at supabase.com/evals plus an internal regression suite that's monitored daily.

That last paragraph is the one that matters. A scoring mechanism that *executes the agent's code against a real Supabase stack* and gates on it is a different category of confidence than a benchmark that grades text.



![text-based scoring vs execution-based scoring](https://cdn.otf-kit.dev/blog/supabase-evals-ai-benchmark/inline-1.png)



## The harness: three dimensions, two suites

The framework defines every scenario along three axes, and they're worth memorising because they show what "coverage" actually means here:

- **Products**: database, auth, storage, edge-functions, realtime, cron, queues, vectors, data-api
- **Topics**: RLS, security, migrations, SQL, SDK, observability, self-hosting, tests, declarative-schema
- **Stages**: build, deploy, investigate, resolve

Scenarios were picked to touch each (product, topic, stage) combo at least once — the smallest set that covers the surface — and they were grounded in real support tickets, bug reports, and GitHub issues. That design choice is doing real work: the eval isn't synthetic, it's the bugs the team has actually had to fix.

They split the scenarios into two suites with very different jobs:

| Suite | Purpose | Published | Refresh |
|-------|---------|-----------|---------|
| Benchmark | Breadth across the dimension matrix | Yes | Stable |
| Regression | Known failure modes | Internal | Daily |

The regression suite is what powers daily internal monitoring and SDK release gating. The benchmark suite is what the public leaderboard shows.

## What the scoring actually does

Every scenario is a directory with three things: a task file plus frontmatter, the scorer, and optional setup / seed starting states. When you ship a workspace, the framework boots a Docker sandbox with the real CLI installed and exposes a Management API-compatible surface backed by Postgres. The agent calls the actual MCP server and CLI — no mocks, no stubs, no "we simulated the response."

Scoring is two passes:

```ts
// scoring.ts — illustrative shape, see harness for the exact API
export const grade = async (task, agentOutput) => {
  // Pass 1: deterministic. Did the SQL parse? Does RLS deny the rows
  // that should be denied? Did the Edge Function return the schema it
  // claimed to return?
  const deterministic = await runChecks(task, agentOutput)

  // Pass 2: LLM-as-a-judge on the intent match.
  const judge = await llmJudge(task, agentOutput, trace)

  return combine(deterministic, judge)
}
```

Agents get one retry before grading — a useful concession, because real engineering involves iteration and penalising a single bad first try would reward agents that wrap themselves in two-line "looks done" outputs.

## What the leaderboard already shows

The early findings are encouraging in a way that's worth stating plainly. Agents pass most scenarios with no skill loaded. In the Build stage, Opus 5 and Kimi K3 both scored 100% unaided. That's a real signal that the model layer has caught up to routine Supabase work without bespoke scaffolding.

The work that still needs skills is the gap: Investigate and Resolve — the stages that require reading logs, following a chain of side effects, or understanding why a deploy broke. The benchmark publishes which categories each agent wins and which it doesn't, which is the part that matters if you're picking an agent for a regulated backend where a wrong investigation is the bug.

## How to run it today

The framework is deployable now, under Apache-2.0, locally. Three things the release confirms you need:

**Prerequisites:**
- Docker daemon running
- Provider API keys for whichever agents you want to grade (Anthropic for Claude Code, OpenAI for Codex, the OpenCode provider for OpenCode)
- Ports 54321–54329 free

Each eval directory is a self-contained scenario: the task plus its frontmatter, the scorer, and optional setup / seed files for the starting state. Shipping a workspace, or declaring one, boots a Docker sandbox with the real CLI installed.

A typical run:

```bash
# 1. Boot the local Supabase-like stack in containers
docker compose up -d

# 2. Run a scenario against an agent.
# (Exact CLI flags depend on the harness version; check the repo's --help.)
docker compose exec harness run \
  --suite benchmark \
  --scenario rls/broken-policy \
  --agent claude-code
```

The runtime exposes a Management API-compatible surface, so the agent calls the real MCP server and the real CLI throughout. The agent's final state is graded by the directory's scorer.

Wire the result into CI as a release gate:

```yaml
# Pattern: same shape as a migration-test or type-check gate
- name: Run Supabase Evals regression suite
  run: harness run --suite regression --agents claude-code

- name: Block release if pass rate drops
  run: |
    RATE=$(jq -r '.aggregate.pass_rate' runs/latest/report.json)
    test "$(echo "$RATE > 0.95" | bc -l)" -eq 1
```

That last block is the part that matters: the eval suite becomes a release gate the same way a migration test does. If Claude Code's score on database-migration scenarios drops 10 points after a docs edit, your edit regressed agent behaviour — and the bench caught it before the SDK shipped.

## What this enables in practice

Four patterns become cheap once the bench exists:

1. **Regression-testing docs and skill edits.** When you change a Supabase skill file or a docs page, the regression suite tells you whether your edit *broke* an agent's ability to solve a task it previously solved. That's the loop you've been missing.
2. **Gating SDK releases.** Before shipping a Supabase SDK that changes how agents call it, run the benchmark. If Codex's score on Edge Functions drops 15 points, your release regressed agent behaviour.
3. **Comparing harnesses head-to-head.** Same scenarios, two harnesses, same judge. The diff in `report.json` tells you which harness is buying you which capability.
4. **Regulated backends.** Fintech and healthcare can't ship a wrong RLS policy — it's a security incident. A bench that scores against a real Postgres with real RLS evaluation is the difference between "the agent said it did the right thing" and "the policy actually denies what it should."

## The layer underneath the layer

Supabase Evals measures whether an agent shipped correct backend work. That's half the problem.

The other half is the UI layer. When an agent generates a UI from a description, you don't want the web version to use one component, iOS to use a different one, and Android to ship a third — and you don't want the agent to have to learn three frameworks to keep them aligned. The OTF Kit gives the agent a single component contract: the same `<Card>`, `<Button>`, `<Sheet>` ships on web, iOS, and Android from one API. The agent writes against the contract once; the contract compiles to three targets. The durable part is the contract — the model swaps out underneath you and the UI stays coherent.

Use Supabase Evals to grade the backend work the agent does. Use the OTF Kit to keep the UI work portable across platforms. Two different layers, both of which have to hold for the ship to count.

## What it doesn't do yet

Three honest limitations from the current release:

- **Local-stack only.** You boot the bench on your laptop; it doesn't run as a managed service. That's by design — the scenarios need a real CLI and a real MCP server — but it means you can't bolt it onto a SaaS dashboard with one click.
- **API keys and ports.** Every agent run needs the provider's API key, and the local stack needs ports 54321–54329 free. CI runners need to be configured for both; there's no managed secrets plumbing yet.
- **The regression suite is curated.** The benchmark suite covers the dimension matrix; the regression suite is the team's known failure modes and refreshes daily. If your team's failure modes aren't yet represented, you'll need to add scenarios yourself — which is exactly the point of Apache-2.0, but it's still work.

The roadmap signal in the project is toward more scenarios and richer judge prompts. Both are the right next moves.

## The takeaway

Supabase Evals is the first benchmark in this space that scores *executing code* against a real Supabase stack, not *generated text* against a vibe check. That's a real advance, and the Apache-2.0 + local-runnable shape means you can wire it into your release pipeline this week. The fact that Opus 5 and Kimi K3 both hit 100% on unaided Build tasks is the headline — agents have caught up to routine backend work without bespoke scaffolding — and the fact that Investigate and Resolve still need skills is where the next round of work lives.

The benchmark is the part that measures the agent. The component contract is the part that holds the UI together when the model swaps out underneath you. Use both.