# Harbor on Vercel Sandbox: parallel Terminal-Bench without your laptop

> Harbor 0.22 runs each Terminal-Bench trial in a Firecracker microVM on Vercel Sandbox — concurrency, firewall secrets, and AI Gateway model swaps.
> By Dave · 2026-09-18
> Source: https://otf-kit.dev/blog/harbor-evals-vercel-sandbox

Harbor is the open-source eval harness behind Terminal-Bench. As of [17 September 2026](https://vercel.com/changelog/run-terminal-bench-and-other-harbor-evals-on-vercel-sandbox), `harbor run --env vercel` places each trial in its own Firecracker microVM on [Vercel Sandbox](https://vercel.com/docs/vercel-sandbox). That is the production change: you can parallelize agent benchmarks beyond your laptop without collapsing network policy and secrets into the same guest as the agent under test.

If you already measure coding agents with a fixed bench, this update changes where the trials run — not whether you still need a written acceptance gate. Pair it with a durable evaluation loop, not a one-off screenshot of pass rates.

## What shipped

The Vercel changelog is explicit:

1. **Harbor 0.22.0+** can target Vercel Sandbox with `--env vercel`.
2. **Each trial gets an isolated Firecracker microVM**, so concurrency is not limited to one workstation.
3. **Network policy is enforced at the sandbox firewall**, outside the VM.
4. **Optional credential injection** attaches secrets to matching outbound requests at that firewall, so tokens do not need to live inside the trial guest.
5. **AI Gateway** lets one `AI_GATEWAY_API_KEY` reach many models; swapping models is the same command with a different `--model`.

The registry Harbor sits in front of is not Terminal-Bench alone — the same post names SWE-bench, tau3-bench, and OSWorld among other Harbor-backed suites. Treat that as a catalog of evals you can point at Sandbox, not as a claim that every suite is required for your product.

![Official Vercel changelog: Run Terminal-Bench and other Harbor evals on Vercel Sandbox, captured 2026-09-18](https://cdn.otf-kit.dev/blog/harbor-evals-vercel-sandbox/competitor-harbor-vercel-20260918.png)

## Why production builders should care

Laptop-bound Harbor runs break in predictable ways once an agent team grows:

- One machine cannot host eight honest concurrent Firecracker guests without thrash.
- Shared developer laptops mix personal tokens, VPN routes, and half-finished sandboxes.
- Model swaps become tribal knowledge ("I ran Claude last night on my Mac") instead of a reproducible command.

Sandbox-backed Harbor answers those failure modes with isolation and concurrency you can schedule. It does not replace product ownership. You still need a repo that defines what "good" means for your app — prompts, fixtures, and ship criteria that survive the next model marketing cycle.

Related context already on this site: [size agent workspaces when Sandbox storage is large](/blog/vercel-sandbox-64gb-storage), [own the Queue and Sandbox hosting seam for OpenAI Agents](/blog/vercel-openai-agents-sandbox), and the durable [LLM evaluation loop](/blog/llm-evaluation-loop).

## Concrete gate you can run this week

Install Harbor with the Vercel extra, then run a small Terminal-Bench slice before you trust a model swap in production agents.

```bash
uv tool install 'harbor[vercel]'
export VERCEL_TOKEN="<your-token>"
export AI_GATEWAY_API_KEY="<your-key>"

harbor run -d terminal-bench/terminal-bench-2-1 \
  --agent fx \
  --model vercel_ai_gateway/anthropic/claude-fable-5 \
  --env vercel \
  --n-concurrent 8
```

Swap `--model` to another Gateway-routed id (the changelog example uses `vercel_ai_gateway/openai/gpt-5.6-luna`) and keep every other flag fixed. That is the whole point of the release: same harness, different model, same sandbox isolation.

Pass criteria before you change production routing:

1. Harbor version is **0.22.0 or later** (required for `--env vercel`).
2. Trials complete in Sandbox with concurrent workers greater than one.
3. Secrets used for model calls are injected at the firewall path described in the changelog — not pasted into the guest image.
4. You record model id, Harbor version, dataset id, concurrency, and date next to the score.

Fail criteria that should block a model swap:

- Scores exist only as a Slack screenshot with no command line.
- Trials ran on a developer laptop with personal API keys in the environment.
- Dataset or agent flags changed between the baseline and the candidate model.

![Dex and Luna run a gate: single local trial versus parallel Sandbox microVMs](https://cdn.otf-kit.dev/blog/harbor-evals-vercel-sandbox/inbody-01-gate-20260918b.png)

## How to keep Harbor from becoming another news-reaction post

Benchmark posts die when they only announce a vendor flag. Keep this change useful by wiring it to decisions you already make:

### 1. Freeze the product question first

Write one sentence: "We will switch the default coding agent model when Harbor Terminal-Bench score X is at least Y on dataset Z with agent A." If you cannot fill those letters, Sandbox concurrency will not help — you still lack an acceptance bar. The [evaluation loop guide](/blog/llm-evaluation-loop) is the place to design that bar; Harbor is the runner.

### 2. Separate harness secrets from product secrets

Harbor needs a Vercel token and, when you use Gateway, an AI Gateway key. Product apps need their own auth, billing, and data keys. Do not unify those into one `.env` that agents copy into every trial. The changelog's firewall injection model is the pattern: credentials meet outbound requests at the boundary.

### 3. Keep owned product code outside the eval guest

Sandbox is for untrusted or agent-generated code under test. Your shipping product still lives in the repo buyers clone — kits with auth, data, payments, and agent docs on disk. Browse that catalog at [otf-kit.dev/templates](https://otf-kit.dev/templates). Eval isolation is not a substitute for an owned monorepo.

### 4. Log failures the same way you log passes

A red trial is only useful if you can open the guest logs, reproduce with `--n-concurrent 1`, and attach the Harbor version. Treat flaky concurrent runs as infrastructure bugs, not model intelligence.

## Architecture sketch (what stays outside the VM)

Think in three layers:

| Layer | Owns | Harbor + Sandbox default |
| --- | --- | --- |
| Eval definition | Dataset id, agent adapter, pass/fail rubric | Harbor registry + your flags |
| Isolation | Filesystem, network, concurrency | Firecracker microVM per trial on Sandbox |
| Credentials | Model and platform tokens | Firewall injection + AI Gateway |

If a teammate proposes baking provider keys into the trial image "for speed," reject it. The release exists so you do not have to.

![Byte and Dex verify Harbor CLI, firewall credential injection, and AI Gateway model swaps](https://cdn.otf-kit.dev/blog/harbor-evals-vercel-sandbox/inbody-02-sandbox-20260918c.png)

## Decision table for this week

| Signal | Stay on local Harbor | Move trials to `--env vercel` |
| --- | --- | --- |
| Concurrent trials needed | 1–2 on a quiet machine | 4+ without thrash |
| Secret handling | Personal keys on laptop OK for a spike | Need firewall injection for shared runs |
| Model comparison | One model, one evening | Many models via AI Gateway |
| Audience for scores | Yourself | Team / release notes |

Standup checklist: upgrade Harbor to 0.22.0+, store `VERCEL_TOKEN` and `AI_GATEWAY_API_KEY` in the CI secret store, run one Terminal-Bench slice at `--n-concurrent 8`, log scores next to model ids, only then change production agent routing.

Harbor on Vercel Sandbox does not invent judgment for you. It gives each trial a microVM, pushes network policy and credential injection to the firewall, and lets AI Gateway swap models with one flag. Use that to make model changes boring — and keep the product spine in a repo you own.

## Sources

- [Run Terminal-Bench and other Harbor evals on Vercel Sandbox](https://vercel.com/changelog/run-terminal-bench-and-other-harbor-evals-on-vercel-sandbox) — 17 Sep 2026 changelog (Harbor 0.22.0, `--env vercel`, firewall injection, AI Gateway examples)
- [Vercel Sandbox docs](https://vercel.com/docs/vercel-sandbox) — isolation model and intended uses for agent-generated code
- [OTF templates](https://otf-kit.dev/templates) — owned full-stack kits and landing templates buyers keep