Methodology

The matrix

A run covers every combination of cloud × model × app × trial in benchmark.yaml, called a cell. Each cloud gets its own reusable-workflow call (cloud.yml). Each cell gets its own fresh GitHub-hosted runner and its own Claude Code session, so nothing learned or installed in one cell carries over to another. Cells of the same cloud run one at a time by default (max_parallel_per_cloud). That keeps timings free of interference from build queues, and avoids clashes over account-wide limits such as DigitalOcean's single container registry or Render's single free Postgres.

A cell, step by step

Every cell builds its app from scratch, and the build prompt names the target cloud, so the agent picks an architecture that suits the platform: serverless functions on one cloud, a container with a volume on another. That is deliberate: the work and tokens a build takes depend on the architecture a cloud calls for, so build metrics are part of the comparison. Each cell's report and the run summary show the stack that was built (frameworks, lines of code, dependencies and deploy configuration). Because builds also vary from run to run, use trials above 1 to tell platform effects from noise.

The prompts are identical for every cloud except the cloud's name and URL, its credential variable names, and the cell's resource prefix. Agents get no provider-specific hints, skills or MCP servers. Finding out how to deploy to each cloud is part of what is measured.

Claude Code runs headless (claude -p --output-format stream-json) with bypassPermissions, a pinned version, and the model and effort from benchmark.yaml. A model marked hipaa: true uses an API key from an organization with the HIPAA configuration, which rejects Claude Code's advisor tool and pre-release API features, so those are turned off for it; reports note it next to the model. Each phase has a wall-clock limit (build_timeout_minutes, deploy_timeout_minutes) and a spend cap (max_budget_usd_per_phase, which Claude Code applies per invocation, so a resumed deploy gets its own budget). The deploy prompt requires every resource name to start with the cell's prefix (for example adb-k3x9f-crud). Agents may set up whatever the cloud lets them set up themselves, saving any credentials they create in the workspace's .env, and must list anything only a person can do instead of working around it with another provider (see Human steps). Resources are found by the prefix, or because the agent declared them in deployment.json. Declared resources that existed before the cell are priced but never deleted.

When a run is cancelled or a step times out, the harness stops Claude Code and everything it started before the step ends, so teardown never races a live agent. A cancelled run is not committed.

Completion success

A cell is a success when its deployment answers on a public HTTPS URL and passes every required check for its app. It is partial when it is reachable but at least one required check fails, and failed when there is no reachable deployment. The failure stage is recorded: build, deploy, or human when the agent couldn't finish without something only a person can do. Optional checks count toward the cell's score, which is checks passed over checks run. Checks skipped because no browser was available don't count either way.

Every app shares three required checks: the app answers with 2xx within ready_timeout_minutes (which absorbs cold starts and rollouts still in progress), the URL uses HTTPS with a valid certificate, and it doesn't redirect to another host such as a login wall. Status codes are checked strictly where the spec names one, with close equivalents accepted (for example 200 or 201 on create, 400 or 422 on validation errors).

The verifiers are tested against reference implementations of all five apps, and against broken variants to prove each check catches what it should (tests/).

Screenshots

While the deployment is live, the verifier also takes screenshots in its browser so people can see what each agent built. They don't affect any score.

Each is a 1280×900 JPEG, shown in the cell's report and, for the first trial, in the run summary's gallery. Pixels can't be masked like text, so the verifier saves each page's text next to its screenshot, and finalize deletes any screenshot whose page showed a secret or that has no text.

Human steps

The benchmark is about what it takes for an agent to deploy, so the work a person has to do is measured rather than done quietly in advance. Agents start each cell with only what no agent can get for itself, the cloud's prerequisites, and nothing else is set up for them: no Marketplace integrations, and no Git hosting unless the optional GitHub scratch account is provided.

Each cell's human steps are collected from evidence by bench/human.py:

• Prerequisites, from benchmark.yaml.
• Help the deployment relies on: repositories or images in the GitHub scratch account, a Marketplace store on Vercel (a person accepted its terms in a browser), or paid resources on a cloud whose paid plans need a payment method (when_paid).
• Blockers: what the agent listed under human_steps in deployment.json, and provider errors in its tool output that only a person can resolve, such as Vercel's integration_terms_acceptance_required, Render's "You must enter payment information", or Cloudish's "API key is out of credits". Blockers count only when the cell didn't succeed: an agent that deployed a working app evidently managed without them. A cell that failed or was partial and has blockers gets the failure stage human.

The number of distinct steps is the objective measure. Each step also has an estimated time from the human_steps catalog in benchmark.yaml, for someone who already knows what to do (signing up 3 min, payment method 2, API token 2, accepting integration terms 3, Git hosting 10, adding credits 3, anything else 5). Replace the estimates with measured times as you get them. Per cloud, the summary counts each distinct step once across apps, because a person does it once per account.

Security and compliance review

After verification, while the deployment is still live, a separate Claude Code session reviews how securely the app was built and deployed, whether its hosting fits how the app will be used, and what is running. The reviewer is the same for every cell (reviewer in benchmark.yaml: Opus 5.5 at high effort), whatever model built the app, and it gets the model key only, never cloud credentials. It works from:

• a copy of the code and configuration the agent produced, with secrets masked;
• condensed transcripts of the build and deploy sessions: the agent's messages, every command and tool call, and abbreviated results. A masked value in a command shows that a secret appeared there;
• deployment.json and the resources the benchmark found;
• how the app will be used (use in benchmark.yaml: commercial for the landing page, the team todo list and the sales dashboard, personal for the notes API and the club chat) and what the benchmark could read about the cloud account, such as Vercel's team plan;
• the live URL, which it may probe like a careful outside tester, sending at most 100 requests and no brute-forcing, fuzzing or load.

It scores each criterion of security_rubric from 0 to 10, with evidence for every finding. The harness computes the 0–100 score from the criterion weights, so the arithmetic isn't left to the model:

The weights favor how the app was deployed over the app's own code. The code would have the same weaknesses on any cloud, so it says little about the cloud; it is still reviewed, but counts for 5.

Cells without a reachable deployment aren't reviewed. The reviewer's cost is reported separately and isn't added to the agent cost. Each cell's report lists the criterion scores, the rationale and the findings, and the summaries average the score per cloud. Each finding names its source, the party that would have to fix it: the platform (the cloud's own behavior or defaults), the deployment (how the agent built and deployed) or the app (the code the agent wrote).

Metrics

Hosting cost

Clouds bill in different ways: some charge a fixed price per instance, some per second while a container runs, and some per request with free allowances. A single "monthly cost" therefore only compares fairly at a stated level of traffic. Each deployment is priced for a month at two levels, defined in hosting_usage in benchmark.yaml:

Each cloud adapter in bench/clouds/ lists the resources a cell created, gives each its list price at the size it was provisioned, from the provider's live price API where there is one and otherwise from the dated tables in bench/clouds/pricing/, and then prices the whole deployment at each level:

• Cloudish bills containers per second while they run and scales idle ones to zero after 15 minutes, so a container costs its size's hourly rate for the hours it is awake: 10 at light use, 730 when busy. Cloudish lists no bandwidth or request charges.
• DigitalOcean App Platform containers always run, so they cost their fixed price at both levels. Each container size includes outbound transfer (50 GiB a month for the $5 size), pooled across the team's apps, and beyond it transfer costs $0.02 per GiB. Static-only apps get 1 GiB.
• Render paid instances cost their plan price at both levels. Free web services cost nothing while the workspace's 750 free instance hours last, and spin down after 15 idle minutes. Outbound transfer beyond the workspace plan's allowance costs $0.15 per GB. The API doesn't report the workspace plan, so the benchmark assumes Hobby, which includes 5 GB. Without a payment method on file, Render suspends the services instead of billing the extra transfer.
• Vercel Hobby is free within fixed monthly limits (1M CDN requests, 100 GB of transfer, 1M function invocations, 4 hours of active CPU, 360 GB-hours of memory). It pauses projects that exceed them, can't buy more, and is for personal, non-commercial use only. So on a Hobby team, a business's app (use: commercial in benchmark.yaml), or traffic beyond the limits, is priced on Pro: the $20 monthly platform fee, which includes $20 of usage credit, plus usage beyond the credit at Pro's on-demand rates. On a Pro team, only usage beyond the credit counts, since the fee is paid once per team.
• Databases, volumes, registries and other fixed-price resources cost their list price at both levels.
• Free tiers cost $0 even when they sleep when idle or expire (Render free Postgres lasts 30 days). The security and compliance review judges whether relying on them suits the app.
• Other account-level fees (Render workspace plans, Vercel seats beyond the first) aren't attributed to an app.

The two levels bracket what a small app costs: light use shows what scale-to-zero and free plans save when nobody visits, and busy shows what an app costs once it never idles. Each cell report explains how each figure was reached. Resources whose price can't be determined are listed as unpriced, and totals show +. Each resource's list price, always on at its provisioned size, is also recorded (hosting_monthly_usd).

Result files

Everything under results/ is plain JSON and markdown, so other tools, such as a results website, can read it directly. Paths inside the JSON files are relative to the folder given in the table.

Every file has a schema number, raised whenever fields are added, renamed or removed. Files from earlier runs keep the schema they were written with, so readers should treat fields added in later versions as optional:

Fairness and limitations

• Model knowledge differs by provider. Long-established CLIs are well covered by training data. Newer platforms such as Cloudish have to be discovered from their docs during the run. This is part of what the benchmark measures, and it is visible in each cell's web lookups.
• The security and compliance score is a model's judgment. A fixed rubric, required evidence and the same reviewer for every cell keep it comparable, but reviews vary from run to run like the deployments do. The reviewer also can't inspect cloud settings it has no credentials for, and it judges providers' terms from what it knows of them.
• Human time is estimated. The steps are collected from evidence, but their minutes come from a catalog; first-time setup usually takes longer than the estimates. A person's own model API key is not counted, since every cloud needs it equally.
• Runs are not deterministic. Use trials above 1 to report success rates rather than single outcomes.
• Persistence is checked over a minute. The checks don't restart the app, so they can't prove data survives a redeploy.
• Leak detection is heuristic. Resources created during a cell without the prefix are reported as possible leaks, but they are not deleted automatically.
• Credential scoping prevents accidents, not attacks. Each step gets only the secrets it needs, and the agent's environment excludes the runner's file commands (GITHUB_ENV, GITHUB_PATH, ...) and harness variables. But the agent runs as the runner's user with passwordless sudo, so an agent determined to misbehave, for example after a prompt injection in a page it fetched, could tamper with later steps. Use dedicated cloud accounts with spending limits.
• Secrets are masked before anything leaves the runner. That covers the provided values, passwords in URLs, and well-known token formats. Transcripts and generated source are uploaded as workflow artifacts for 30 days, and only metrics and summaries are committed.

This page is docs/methodology.md in the repository.