Harbor on Vercel Sandbox: parallel Terminal-Bench without your laptop

Harbor is the open-source eval harness behind Terminal-Bench. As of 17 September 2026, harbor run --env vercel places each trial in its own Firecracker microVM on Vercel Sandbox. That is the production change: you can parallelize agent benchmarks beyond your laptop without collapsing network policy and secrets into the same guest as the agent under test.

If you already measure coding agents with a fixed bench, this update changes where the trials run — not whether you still need a written acceptance gate. Pair it with a durable evaluation loop, not a one-off screenshot of pass rates.

What shipped

The Vercel changelog is explicit:

1. Harbor 0.22.0+ can target Vercel Sandbox with --env vercel.
2. Each trial gets an isolated Firecracker microVM, so concurrency is not limited to one workstation.
3. Network policy is enforced at the sandbox firewall, outside the VM.
4. Optional credential injection attaches secrets to matching outbound requests at that firewall, so tokens do not need to live inside the trial guest.
5. AI Gateway lets one AI_GATEWAY_API_KEY reach many models; swapping models is the same command with a different --model.

The registry Harbor sits in front of is not Terminal-Bench alone — the same post names SWE-bench, tau3-bench, and OSWorld among other Harbor-backed suites. Treat that as a catalog of evals you can point at Sandbox, not as a claim that every suite is required for your product.

Why production builders should care

Laptop-bound Harbor runs break in predictable ways once an agent team grows:

• One machine cannot host eight honest concurrent Firecracker guests without thrash.
• Shared developer laptops mix personal tokens, VPN routes, and half-finished sandboxes.
• Model swaps become tribal knowledge ("I ran Claude last night on my Mac") instead of a reproducible command.

Sandbox-backed Harbor answers those failure modes with isolation and concurrency you can schedule. It does not replace product ownership. You still need a repo that defines what "good" means for your app — prompts, fixtures, and ship criteria that survive the next model marketing cycle.

Related context already on this site: size agent workspaces when Sandbox storage is large, own the Queue and Sandbox hosting seam for OpenAI Agents, and the durable LLM evaluation loop.

Concrete gate you can run this week

Install Harbor with the Vercel extra, then run a small Terminal-Bench slice before you trust a model swap in production agents.

uv tool install 'harbor[vercel]'
export VERCEL_TOKEN="<your-token>"
export AI_GATEWAY_API_KEY="<your-key>"

harbor run -d terminal-bench/terminal-bench-2-1 \
  --agent fx \
  --model vercel_ai_gateway/anthropic/claude-fable-5 \
  --env vercel \
  --n-concurrent 8

Swap --model to another Gateway-routed id (the changelog example uses vercel_ai_gateway/openai/gpt-5.6-luna) and keep every other flag fixed. That is the whole point of the release: same harness, different model, same sandbox isolation.

Pass criteria before you change production routing:

1. Harbor version is 0.22.0 or later (required for --env vercel).
2. Trials complete in Sandbox with concurrent workers greater than one.
3. Secrets used for model calls are injected at the firewall path described in the changelog — not pasted into the guest image.
4. You record model id, Harbor version, dataset id, concurrency, and date next to the score.

Fail criteria that should block a model swap:

• Scores exist only as a Slack screenshot with no command line.
• Trials ran on a developer laptop with personal API keys in the environment.
• Dataset or agent flags changed between the baseline and the candidate model.

How to keep Harbor from becoming another news-reaction post

Benchmark posts die when they only announce a vendor flag. Keep this change useful by wiring it to decisions you already make:

1. Freeze the product question first

Write one sentence: "We will switch the default coding agent model when Harbor Terminal-Bench score X is at least Y on dataset Z with agent A." If you cannot fill those letters, Sandbox concurrency will not help — you still lack an acceptance bar. The evaluation loop guide is the place to design that bar; Harbor is the runner.

2. Separate harness secrets from product secrets

Harbor needs a Vercel token and, when you use Gateway, an AI Gateway key. Product apps need their own auth, billing, and data keys. Do not unify those into one .env that agents copy into every trial. The changelog's firewall injection model is the pattern: credentials meet outbound requests at the boundary.

3. Keep owned product code outside the eval guest

Sandbox is for untrusted or agent-generated code under test. Your shipping product still lives in the repo buyers clone — kits with auth, data, payments, and agent docs on disk. Browse that catalog at otf-kit.dev/templates. Eval isolation is not a substitute for an owned monorepo.

4. Log failures the same way you log passes

A red trial is only useful if you can open the guest logs, reproduce with --n-concurrent 1, and attach the Harbor version. Treat flaky concurrent runs as infrastructure bugs, not model intelligence.

Architecture sketch (what stays outside the VM)

Think in three layers:

If a teammate proposes baking provider keys into the trial image "for speed," reject it. The release exists so you do not have to.

Decision table for this week

Standup checklist: upgrade Harbor to 0.22.0+, store VERCEL_TOKEN and AI_GATEWAY_API_KEY in the CI secret store, run one Terminal-Bench slice at --n-concurrent 8, log scores next to model ids, only then change production agent routing.

Harbor on Vercel Sandbox does not invent judgment for you. It gives each trial a microVM, pushes network policy and credential injection to the firewall, and lets AI Gateway swap models with one flag. Use that to make model changes boring — and keep the product spine in a repo you own.

Sources

• Run Terminal-Bench and other Harbor evals on Vercel Sandbox — 17 Sep 2026 changelog (Harbor 0.22.0, --env vercel, firewall injection, AI Gateway examples)
• Vercel Sandbox docs — isolation model and intended uses for agent-generated code
• OTF templates — owned full-stack kits and landing templates buyers keep