Vercel · Opus 5.5 · medium effort · HIPAA organization · trial 1

Real-Time Collaborative App on Vercel

Success

Checks passed13/13
Human steps25 min of a person's time
Security & compliance84/100
Build5m 02s
Deploy9m 27s
Tokens2.57M
Agent cost$1.58
Hosting / month$0.00 light use$20.00 busy

Deployed to https://adb-raatw-realtime-web.vercel.app. Every deployment is deleted after it's verified and reviewed.

Screenshots

The room the verifier's clients chatted in, with its history
The room the verifier's clients chatted in, with its history · https://adb-raatw-realtime-web.vercel.app/?room=bench-2ac8ef33

Technologies

Runtime
Node.js
Frameworks
Next.jsReact
Libraries
Upstash Redis SDKnode-redisVercel Functions helpers
Dev tools
TypeScript
Packaging
vercel.json
Hosting
Vercel project (Next.js)Vercel Functions

Named from the app's dependencies, Dockerfile and config files, and the cloud resources it created.

Checks

Black-box checks against the live URL. A cell succeeds when every required check passes. The app's spec

ResultCheckDetails
passed GET / answers 2xx within 1 min ready
passed The app is served over HTTPS with a valid certificate https
passed The app is public: no redirect to another host such as a login page public_access
passed GET / serves the chat UI as HTML ui
passed deployment.json declares a supported transport (websocket or sse) transport optional
passed Clients connect over sse connect
passed A message reaches another client in the same room within 5 s delivery
passed The sender receives its own message too echo optional
passed Messages never reach clients in other rooms room_isolation
passed Five users posting at the same time all see every message within 10 s fanout
passed GET /api/rooms/<room>/messages returns recent messages history
passed Messages over 1000 characters are rejected validation optional
passed The chat UI loads in a browser without errors browser_ui optional

Human steps

Security & compliance review

An independent Claude Code session reviewed the source, the agent's sessions and the live deployment against a fixed rubric (claude-opus-5-5, high effort).

The Vercel Hobby deployment of this personal club chat app is clean. No secrets are involved or leaked, HTTPS is enforced with HSTS, no source or config files are exposed, and input validation and output escaping are sound. The main weakness is that the public POST and long-lived SSE endpoints have no rate limiting or connection caps. Minor issues: deployment protection is turned off for previews too, the CLI and dependency ranges are unpinned, and the app sets no security headers.

CriterionWeightScoreWhy
Secrets handling 25 9/10 The app needs no secrets and none are in the source, the build context or the deployed files. The agent confirmed via the deployment file listing that .env.local wasn't uploaded. Optional Redis credentials would be read from platform env vars. One minor issue: the Vercel token was passed as a --token command-line argument (expanded from an env var).
Deployment provenance 20 8/10 Production was built by Vercel from the agent's local source tree, with package-lock.json uploaded, so dependency versions are locked. But package.json uses caret ranges and @latest installs, the Vercel CLI was run as unpinned `npx -y vercel@latest`, and there is no owner-controlled remote git repo tied to the deployment.
Terms compliance 15 9/10 This is a personal (hobby club) app on Vercel Hobby, which Vercel's terms allow for personal, non-commercial use. As far as I know, Hobby doesn't expire or sleep functions. The message store is Runtime Cache, which is ephemeral, so history can be evicted. That is a durability limit, not a terms violation.
Least privilege 15 8/10 There is no public data store, and the Runtime Cache can only be reached from the project's functions. The agent turned deployment protection off completely (ssoProtection=null) rather than using Standard protection, which keeps the production alias public but leaves preview and per-deployment URLs protected.
Transport security 10 8/10 HTTP redirects to HTTPS with a 308, the certificate is valid and the platform adds HSTS with preload. The app sets no CSP, X-Content-Type-Options or frame protection, though it could have done so in next.config headers.
Attack surface 10 9/10 Requests for .env, .git, package.json, source files and .vercel all return 404, and errors return generic JSON with no stack traces. Responses still send the `x-powered-by: Next.js` header.
Application security 5 6/10 Input validation is solid: invalid JSON, null bodies, non-string, empty and over-1000-character text, and room names over 64 characters all get 400. React escapes user content, so there is no XSS. But the public POST and SSE endpoints have no rate limiting or connection caps, and each SSE connection holds a function for up to 270s while polling the cache every 500ms, so anyone can exhaust the Hobby quota or flood rooms.

Findings

Cloud resources and cost

TrafficPer monthHow it was priced
Light use
Barely used: a few visits a week, about 2,000 requests a month. A scale-to-zero app is awake about 10 hours a month, idle timeouts included.
$0.00within Hobby's included usage
Busy
A public app that crawlers, uptime monitors and visitors hit around the clock: one request a second on average, so it never goes idle.
$20.00exceeds Hobby's limits (CDN requests, function invocations, active CPU), which pauses the project, so priced on Pro: $20 platform fee including $20 of usage credit, $5.75 of usage

Resources the agent created

KindNamePlanList price / monthPricing
projectadb-raatw-realtime-web$0.00no fixed fee; usage within the plan's included allowance

The agent estimated $0.00 a month.

Code

Code
540 lines in 15 files (TypeScript 431, JSON 94, JavaScript 92, Markdown 17, CSS 17)
Dependencies
9 npm, 0 Python
Deploy config
vercel.json

The agent's sessions

PhaseStatusTimeTurnsTokensCostShell commands
Buildcompleted5m 02s15582.9K$0.5513
Deploycompleted9m 27s291.99M$1.0327

The agent's deploy summary

The app is live at https://adb-raatw-realtime-web.vercel.app and every requirement works through that URL. Message history is held in a cache that can lose old messages, because the durable Redis store needs a person to accept terms first (see below).

What's deployed

  • Project: adb-raatw-realtime-web, Next.js on the Hobby plan (free), running in Vercel's iad1 region. I turned off Vercel Authentication, which had been blocking public access to the .vercel.app URL.
  • Real-time: Server-Sent Events, because Vercel functions can't hold WebSocket connections. deployment.json has "realtime": {"transport": "sse"}, and there is no /ws endpoint.
  • Storage: Vercel Runtime Cache, which is free on Hobby and shared by all instances of the app's functions. It has no atomic operations, so each instance writes only its own message log, and a per-room list of writers is updated with read-and-retry so simultaneous posts aren't lost.

Tested through the public URL

  • Test suite: all 4 tests pass, covering the page, message format, 50-message history in order, the 400 cases, and room isolation. 20 simultaneous posts reached 2 listeners in under 2s.
  • Stress test: run 3 times, 45 posts from parallel senders. All 4 listeners got every message each time, at most about 1.1s after sending, and history had all 45.
  • Browser test: in headless Chromium, a second user saw messages live, a user in another room didn't, and a new joiner saw history.

What didn't work

  • Durable Redis: setting up Upstash needs its marketplace terms accepted in a browser by a person. This is listed under human_steps as non-blocking.
  • Data loss risk: until that's done, history isn't durable. Cached messages can be evicted at any time and expire after 7 days. On Hobby the cache is also shared with the team's other projects.
  • One flaky run: the first full test run after this change had one history test fail. I didn't capture which check failed, and the cause is unconfirmed; a race during cold start is the likely one. Every run since has passed.

To switch to Redis, a person accepts the terms at https://vercel.com/ai-deploy-bench/~/integrations/accept-terms/upstash?source=cli. Then run vercel integration add upstash/upstash-kv --name adb-raatw-realtime-redis and vercel deploy --prod. The app picks up the Redis settings automatically.

Changes in this step

  • All room endpoints now live in one route file.
  • The browser subscribes to the stream before loading history, so no message falls in the gap.
  • Message IDs are now strings.
  • vercel.json sets the framework to Next.js.

Estimated cost is $0/month. I didn't create any credentials, so .env wasn't needed. No servers are left running, and the work is committed locally.

27 shell commands during deploy
  1. 5s cd /home/runner/work/_temp/adb-raatw-realtime; npx -y vercel@latest --version 2>&1 | tail -1; curl -s -H "Authorization: Bearer $VERCEL_TOKEN" "https://api.vercel.com/v1/storage/stores?teamId=$VERCEL_
  2. 30s cd /home/runner/work/_temp/adb-raatw-realtime; export VC="npx -y vercel@latest --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID"; $VC project add adb-raatw-realtime-web 2>&1 | tail -2; $VC link --yes --p
  3. 36s cd /home/runner/work/_temp/adb-raatw-realtime; VC="npx -y vercel@latest --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID"; $VC integration --help 2>&1 | head -40
  4. 39s cd /home/runner/work/_temp/adb-raatw-realtime; VC="npx -y vercel@latest --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID"; $VC integration add --help 2>&1 | sed -n 1,60p
  5. 42s cd /home/runner/work/_temp/adb-raatw-realtime; VC="npx -y vercel@latest --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID"; $VC integration add upstash/upstash-kv --name adb-raatw-realtime-redis --non-int
  6. 1m 11s cd /home/runner/work/_temp/adb-raatw-realtime/app/api/rooms/\[room\] && mkdir -p '[action]' && python3 - <<'EOF'
  7. 1m 15s cd /home/runner/work/_temp/adb-raatw-realtime && f='app/api/rooms/[room]/[action]/route.ts' && cat -s "$f" > /tmp/r && mv /tmp/r "$f" && npm run build 2>&1 | grep -E 'ƒ|rror' ; (npx next start -p 3123
  8. 1m 27s cd /home/runner/work/_temp/adb-raatw-realtime; VC="npx -y vercel@latest --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID"; $VC deploy --prod --yes 2>&1 | grep -vi plugin | tail -8
  9. 1m 54s cd /home/runner/work/_temp/adb-raatw-realtime; echo '{ "framework": "nextjs" }' > vercel.json; VC="npx -y vercel@latest --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID"; $VC deploy --prod --yes 2>&1 | g
  10. 2m 26s cd /home/runner/work/_temp/adb-raatw-realtime; curl -s -H "Authorization: Bearer $VERCEL_TOKEN" "https://api.vercel.com/v9/projects/adb-raatw-realtime-web?teamId=$VERCEL_TEAM_ID" | python3 -c "import
  11. 2m 32s cd /home/runner/work/_temp/adb-raatw-realtime; curl -s -XPATCH -H "Authorization: Bearer $VERCEL_TOKEN" -H 'Content-Type: application/json' "https://api.vercel.com/v9/projects/adb-raatw-realtime-web?t
  12. 4m 35s cat /tmp/claude-1001/-home-runner-work--temp-adb-raatw-realtime/880f54ab-6b61-41de-8e0c-ca9f803bf7c3/tasks/b2ynld44g.output
  13. 4m 49s cd /home/runner/work/_temp/adb-raatw-realtime; pkill -f "node --test" ; curl -s -H "Authorization: Bearer $VERCEL_TOKEN" "https://api.vercel.com/v9/projects/adb-raatw-realtime-web?teamId=$VERCEL_TEAM_
  14. 4m 53s curl -s -H "Authorization: Bearer $VERCEL_TOKEN" "https://api.vercel.com/v9/projects/adb-raatw-realtime-web?teamId=$VERCEL_TEAM_ID" | python3 -c "import json,sys;d=json.load(sys.stdin);print(d.get('re
  15. 5m 41s cd /home/runner/work/_temp/adb-raatw-realtime; npm i @vercel/functions 2>&1 | tail -1; grep -n "getCache\|interface\|export" node_modules/@vercel/functions/cache/*.d.ts 2>/dev/null | head -30 || ls no
  16. 5m 42s cd /home/runner/work/_temp/adb-raatw-realtime; sed -n 1,80p 'app/api/rooms/[room]/[action]/route.ts' | sed -n 45,110p
  17. 5m 46s cd /home/runner/work/_temp/adb-raatw-realtime; cat node_modules/@vercel/functions/cache/types.d.ts | sed -n 1,60p
  18. 6m 08s cd /home/runner/work/_temp/adb-raatw-realtime; python3 - <<'EOF'
  19. 6m 18s cd /home/runner/work/_temp/adb-raatw-realtime; (VERCEL=1 npx next start -p 3126 > /tmp/s3.log 2>&1 &); (npx next start -p 3127 > /tmp/s4.log 2>&1 &); sleep 4; for p in 3126 3127; do BASE_URL=http://lo
  20. 6m 32s cd /home/runner/work/_temp/adb-raatw-realtime; python3 - <<'EOF'
  21. 7m 03s cd /home/runner/work/_temp/adb-raatw-realtime; U=https://adb-raatw-realtime-web.vercel.app; BASE_URL=$U timeout 100 node --test --test-name-pattern="history" tests/api.test.mjs 2>&1 | grep -A12 Assert
  22. 7m 16s cd /home/runner/work/_temp/adb-raatw-realtime; U=https://adb-raatw-realtime-web.vercel.app; for i in 1 2; do BASE_URL=$U timeout 100 npm test 2>&1 | grep -E '^(✔|✖)|actual|expected|got|^\s+[+-] ' | he
  23. 7m 42s mkdir -p /tmp/stress && cat > /tmp/stress/s.mjs <<'EOF'
  24. 7m 57s cd /home/runner/work/_temp/adb-raatw-realtime; for i in 1 2; do node /tmp/stress/s.mjs https://adb-raatw-realtime-web.vercel.app; done; BASE_URL=https://adb-raatw-realtime-web.vercel.app timeout 100 n
  25. 8m 31s mkdir -p /tmp/pw && cd /tmp/pw && npm init -y >/dev/null && npm i playwright@latest >/dev/null 2>&1 && npx playwright install --with-deps chromium >/dev/null 2>&1; cat > ui.mjs <<'EOF'
  26. 9m 00s cd /home/runner/work/_temp/adb-raatw-realtime; D=$(curl -s -H "Authorization: Bearer $VERCEL_TOKEN" "https://api.vercel.com/v6/deployments?teamId=$VERCEL_TEAM_ID&projectId=adb-raatw-realtime-web&limit
  27. 9m 16s cd /home/runner/work/_temp/adb-raatw-realtime; python3 - <<'EOF'

Raw data