Supabase evals: benchmarking AI agents on real-world tasks

Why this benchmark matters more than it looks

For the last eighteen months, "AI shipped a feature" has meant the LLM emitted code that compiled and looked plausible. It did not mean the schema actually ran. It did not mean the Edge Function returned what it claimed. It did not mean the RLS policy only let through the rows the spec required. Most agent evals were vibes — a human reading diffs and going "yeah, that looks right."

Supabase's open-source Evals framework drops that. The benchmark boots an actual hosted-like stack and a local CLI project inside containers, points an agent such as Claude Code, Codex, or OpenCode at real Supabase tasks — build a schema, debug a failed Edge Function, fix a broken RLS policy — and grades the result against the running system, not the diff. Apache-2.0, runs locally, and it already powers the public leaderboard plus an internal regression suite.

That last paragraph is the one that matters. A scoring mechanism that executes the agent's code against a real Supabase stack and gates on it is a different category of confidence than a benchmark that grades text.

The harness: three dimensions, two suites

The framework defines every scenario along three axes, and they're worth memorising because they show what "coverage" actually means here:

• Products: database, auth, storage, edge-functions, realtime, cron, queues, vectors, data-api
• Topics: RLS, security, migrations, SQL, SDK, observability, self-hosting, tests, declarative-schema
• Stages: build, deploy, investigate, resolve

Scenarios were picked to touch each (product, topic, stage) combo at least once — the smallest set that covers the surface — and they were grounded in real support tickets, bug reports, and GitHub issues. That design choice is doing real work: the eval isn't synthetic, it's the bugs the team has actually had to fix.

They split the scenarios into two suites with very different jobs:

The regression suite is what powers internal monitoring and SDK release gating. The benchmark suite is what the public leaderboard shows.

What the scoring actually does

Every scenario is a directory with three things: a task file plus frontmatter, the scorer, and optional setup / seed starting states. When you ship a workspace, the framework boots a Docker sandbox with the real CLI installed and exposes a Management API-compatible surface backed by Postgres. The agent calls the actual MCP server and CLI — no mocks, no stubs, no "we simulated the response." The repo documents the eval directory layout, experiment runner, and the platform-lite Management API-compatible surface (supabase/evals on GitHub).

Scoring is two passes:

// scoring.ts — illustrative shape, see harness for the exact API
export const grade = async (task, agentOutput) => {
  // Pass 1: deterministic. Did the SQL parse? Does RLS deny the rows
  // that should be denied? Did the Edge Function return the schema it
  // claimed to return?
  const deterministic = await runChecks(task, agentOutput)

  // Pass 2: LLM-as-a-judge on the intent match.
  const judge = await llmJudge(task, agentOutput, trace)

  return combine(deterministic, judge)
}

Agents get one retry before grading — a useful concession, because real engineering involves iteration and penalising a single bad first try would reward agents that wrap themselves in two-line "looks done" outputs. Row Level Security sits underneath several of these tasks; if RLS-as-Postgres-primitive is new to you, start with the official RLS guide and the Supabase RLS production checklist.

What the leaderboard already shows

The early signal is encouraging in a way that's worth stating plainly: agents pass most Build-stage scenarios with no skill loaded. That is a real indication the model layer has caught up to routine Supabase work without bespoke scaffolding.

The work that still needs skills is the gap: Investigate and Resolve — the stages that require reading logs, following a chain of side effects, or understanding why a deploy broke. The benchmark publishes which categories each agent wins and which it doesn't, which is the part that matters if you're picking an agent for a regulated backend where a wrong investigation is the bug.

Leaderboard numbers move fast as models and scenarios update, so treat any specific score you see quoted as a snapshot: check the live leaderboard for current standings rather than trusting a number from a post — including this one.

How to run it today

The framework is deployable now, under Apache-2.0, locally. Three things you need:

Prerequisites:

• Docker daemon running
• Provider API keys for whichever agents you want to grade
• Ports 54321–54329 free

Each eval directory is a self-contained scenario: the task plus its frontmatter, the scorer, and optional setup / seed files for the starting state. Shipping a workspace, or declaring one, boots a Docker sandbox with the real CLI installed.

A typical run:

# 1. Boot the local Supabase-like stack in containers
docker compose up -d

# 2. Run a scenario against an agent.
# (Exact CLI flags depend on the harness version; check the repo's --help.)
docker compose exec harness run \
  --suite benchmark \
  --scenario rls/broken-policy \
  --agent claude-code

The runtime exposes a Management API-compatible surface, so the agent calls the real MCP server and the real CLI throughout. The agent's final state is graded by the directory's scorer.

Wire the result into CI as a release gate:

# Pattern: same shape as a migration-test or type-check gate
- name: Run Evals regression suite
  run: harness run --suite regression --agents claude-code

- name: Block release if pass rate drops
  run: |
    RATE=$(jq -r '.aggregate.pass_rate' runs/latest/report.json)
    test "$(echo "$RATE > 0.95" | bc -l)" -eq 1

That last block is the part that matters: the eval suite becomes a release gate the same way a migration test does. If an agent's score on database-migration scenarios drops after a docs edit, your edit regressed agent behaviour — and the bench caught it before the SDK shipped. For the production side of that loop, see AI production background jobs.

What this enables in practice

Four patterns become cheap once the bench exists:

1. Regression-testing docs and skill edits. When you change a skill file or a docs page, the regression suite tells you whether your edit broke an agent's ability to solve a task it previously solved. That's the loop you've been missing.
2. Gating SDK releases. Before shipping an SDK that changes how agents call it, run the benchmark. If scores on Edge Functions drop, your release regressed agent behaviour.
3. Comparing harnesses head-to-head. Same scenarios, two harnesses, same judge. The diff in report.json tells you which harness is buying you which capability.
4. Regulated backends. Fintech and healthcare can't ship a wrong RLS policy — it's a security incident. A bench that scores against a real Postgres with real RLS evaluation is the difference between "the agent said it did the right thing" and "the policy actually denies what it should."

The layer underneath the layer

Supabase Evals measures whether an agent shipped correct backend work. That's half the problem.

The other half is the UI layer. When an agent generates a UI from a description, you don't want the web version to use one component, iOS to use a different one, and Android to ship a third — and you don't want the agent to have to learn three frameworks to keep them aligned. The OTF Kit gives the agent a single component contract: the same <Card>, <Button>, <Sheet> ships on web, iOS, and Android from one API. The agent writes against the contract once; the contract compiles to three targets. The durable part is the contract — the model swaps out underneath you and the UI stays coherent. How agents read a repo is covered in agent readable repository structure.

Use Supabase Evals to grade the backend work the agent does. Use the OTF Kit to keep the UI work portable across platforms. Two different layers, both of which have to hold for the ship to count.

What it doesn't do yet

Three honest limitations of the current release:

• Local-stack only. You boot the bench on your laptop; it doesn't run as a managed service. That's by design — the scenarios need a real CLI and a real MCP server — but it means you can't bolt it onto a SaaS dashboard with one click.
• API keys and ports. Every agent run needs the provider's API key, and the local stack needs its port range free. CI runners need to be configured for both; there's no managed secrets plumbing yet.
• The regression suite is curated. The benchmark suite covers the dimension matrix; the regression suite is the team's known failure modes. If your team's failure modes aren't yet represented, you'll need to add scenarios yourself — which is exactly the point of Apache-2.0, but it's still work.

The roadmap signal in the project is toward more scenarios and richer judge prompts. Both are the right next moves.

The takeaway

Supabase Evals is the first benchmark in this space that scores executing code against a real Supabase stack, not generated text against a vibe check. That's a real advance, and the Apache-2.0 plus local-runnable shape means you can wire it into your release pipeline this week. Agents have caught up to routine backend work without bespoke scaffolding — and Investigate and Resolve is where the next round of work lives.

The benchmark is the part that measures the agent. The component contract is the part that holds the UI together when the model swaps out underneath you. Start from a kit that ships that contract: otf-kit.dev/templates.

Sources

• supabase/evals on GitHub — eval directory layout, experiment runner, platform-lite Management API-compatible surface.
• Supabase Evals leaderboard — build/deploy/investigate/resolve stages and scoring; check live for current standings.
• Supabase RLS guide — RLS-as-Postgres-primitive background behind the fix-a-broken-RLS-policy tasks.