The Phantom Sandbox: 30 Commits to Fix a Two-Click Problem

If you read my previous article about the CRLF bug, you know my deployment pipeline has had a rough time. This one took longer to fix, and the solution was embarrassingly simple.

Background

My portfolio runs on Fresh 2 and deploys to Deno Deploy via GitLab CI. The original pipeline was straightforward: build locally, deploy with the built-in deno deploy subcommand:

deploy:
  script:
    - deno install
    - deno task build
    - deno deploy --app="$DENO_DEPLOY_PROJECT" --prod --token="$DENO_DEPLOY_TOKEN"

Over the weeks, the config grew to handle a series of problems:

1. The 100MB artifact blowout: Deno Deploy's cloud builder re-downloaded node_modules on the server, exceeding the 50MB upload limit. I fixed it by adding rm -rf node_modules before deploy.
2. The CRLF invisible character: Windows line endings in .denoignore broke ignore rules on Linux builders. I fixed it with a .gitattributes file that enforces eol=lf.

Each fix was reasonable and each one worked, but I was stacking changes on top of each other without understanding the root issue.

The New Error

On July 22, my pipeline threw something entirely new:

error: Top-level await promise never resolved
    await deployCommand.command("sandbox", sandboxCommand).reset().noExit()
    ^
    at <anonymous> (https://jsr.io/@deno/deploy/0.0.9904/main.ts:40:5)

I looked at the stack trace. The error wasn't in my code. It was inside @deno/deploy@0.0.9904, a JSR package that the deno deploy CLI delegates to internally. A sandboxCommand was starting up and its promise was hanging forever, never resolving, never rejecting.

The Debugging Frenzy

What followed was a 30+ commit marathon over two days and an embarrassing amount of CI minutes.

I started with version pinning. I pinned Deno to version 2.0.2, thinking it was a regression. I added --non-interactive to prevent the CLI from waiting for browser auth. I added main.ts as an explicit entrypoint argument. None of it worked.

Then I tried bypassing the Deno Deploy cloud builder entirely. I built locally in CI and attempted to upload pre-built assets. This required sed to remove _fresh/ from .gitignore so the build output would upload, rewriting main.ts to export { default } from "./_fresh/server.js" so Deno Deploy wouldn't need the Fresh runtime, and a bypass.ts script that rewrote deno.json to replace the build task with echo 'Build successfully bypassed on server'. The pipeline grew to 8+ steps of shell gymnastics.

I then dove into config experimentation. I added a deploy block to deno.json with org, project, and entrypoint. Deno Deploy's parser crashed on it. I removed it. I toggled nodeModulesDir between "auto" and "manual". I stripped allowScripts. I tried adding --entrypoint, DENO_NO_UPDATE_CHECK=1, and deno clean to bust cached packages.

Finally, I stripped the pipeline back to the bare minimum: just deno deploy with no build step, no sed, no bypass. It still failed with the same sandbox error.

At this point my .gitlab-ci.yml had been through so many iterations that git log --oneline -- .gitlab-ci.yml returned over 25 deploy-related commits in a single week.

The Actual Problem

After stepping away and thinking about what had actually changed, I realized the problem wasn't in my code, my config, or the CLI. It was the app itself.

I'm on Deno Deploy's free tier. At first I suspected the platform was throttling me. Maybe I'd hit some kind of build limit that comes with the free plan. The vague internal error responses and the sandbox hanging certainly felt like a rate limit or quota issue.

The clue was simpler than I expected. My app on console.deno.com had gone through an enormous number of builds over a short period. Between the debugging marathon, CI retriggers, and various config experiments, the app had accumulated a long history of deployments. I never figured out which one it was, but the app's internal state on the platform had become unrecoverable.

The @deno/deploy@0.0.9904 sandbox wasn't buggy. My best guess is that it was choking on state the platform could no longer process cleanly.

The Fix

I deleted my old app from console.deno.com and created a new one.

The next pipeline run deployed successfully with the simplest possible config:

deploy:
  stage: deploy
  script:
    - deno deploy --app="$DENO_DEPLOY_PROJECT" --prod --token="$DENO_DEPLOY_TOKEN"
  only:
    - main

No sed. No bypass.ts. No cache clearing. No entrypoint flags. No nodeModulesDir stripping. No build step in CI at all. Deno Deploy's cloud builder handles everything natively.

The Pipeline at Its Worst

At its worst, my deploy pipeline looked like this:

deploy:
  script:
    - deno --version
    - deno clean
    - deno install --reload
    - deno task build
    - rm -rf node_modules
    - sed -i 's|_fresh/||' .gitignore
    - sed -i '/"nodeModulesDir"/d' deno.json
    - echo 'export { default } from "./_fresh/server.js";' > main.ts
    - deno deploy --help || true
    - DENO_NO_UPDATE_CHECK=1 deno deploy --app="..." --entrypoint=main.ts

Ten steps. Three sed mutations. A debug --help dump. An environment variable to suppress update checks. All of this to work around a problem that had nothing to do with my code or configuration.

Conclusion

I spent two days adding layers of fixes. The actual solution took two clicks: delete, re-create.

When an opaque internal error persists across every config change you try, the problem might not be in your code at all. It might be corrupted state on the platform side. That's worth checking before writing a tenth workaround.