Each metric over runs, one line per cloud. Hover over a point for its value, and select a cloud in a legend to hide or show it.
Every run that included the realtime chat, averaged over trials.
Success rate
Share of apps whose deployment passed every required check. Higher is better.
Checks passed
Share of the verifier's checks that passed, averaged over apps. Higher is better.
Human steps
Things a person had to do, or would have to do, for the agent to deploy. Lower is better.
Human time
Estimated minutes those human steps take. Lower is better.
Security & compliance
Score of the independent security and compliance review (0-100). Higher is better.
Build time
Wall-clock time of the agent's build phase, averaged over apps. Lower is better.
Deploy time
Wall-clock time of the agent's deploy phase, averaged over apps. Lower is better.
Tokens
Tokens the agent used, including cache reads and writes; totals across apps. Lower is better.
Agent cost
Model API cost of the build and deploy sessions; totals across apps. Lower is better.
Hosting, light use
Estimated monthly hosting cost of the deployed resources: barely used: a few visits a week, about 2,000 requests a month. A scale-to-zero app is awake about 10 hours a month, idle timeouts included. Lower is better.
Hosting, busy
Estimated monthly hosting cost of the deployed resources: a public app that crawlers, uptime monitors and visitors hit around the clock: one request a second on average, so it never goes idle. Lower is better.