tinybeeai 0.0.0-stage → 0.16.1

package/docs/DEPLOY.md

@@ -0,0 +1,577 @@
+# Deploy localdost
+
+localdost is one Node process and one SQLite file. A host needs three things: a folder that survives restarts for `LOCALDOST_DATA_DIR`, a `LOCALDOST_TOKEN` for the API, and an HTTPS URL for phone push and webhooks. Pick the host by where your tests must run. Set up [backups](#back-up-and-restore) before the first real use, and read [Upgrade](#upgrade) before you move to a new tag.
+
+| Host | Good for | Test runs |
+|---|---|---|
+| [Railway](#railway) | Always on, no machine of yours | Yes, the default image has browsers, see [Railway limits](#railway-limits) |
+| [Docker Compose](#docker-compose) | A spare server or VM you own | Yes, with the `runner` image target |
+| [Your machine plus cloudflared](#your-machine-plus-cloudflared) | Fastest start, tests run in your checkout | Yes |
+
+## Settings every host needs
+
+| Variable | Value |
+|---|---|
+| `LOCALDOST_TOKEN` | Required on any host other than your own laptop. Without it the server listens on 127.0.0.1 only, and with `HOST=0.0.0.0`, as the Docker image sets, it refuses to start. Generate it with `openssl rand -hex 24` |
+| `LOCALDOST_DATA_DIR` | A persistent folder. `/data` in the Docker image |
+| `LOCALDOST_PUBLIC_URL` | The HTTPS origin, for links in PR comments and for push |
+| `LOCALDOST_POLL` | `1` polls GitHub and Linear, so no webhook setup is needed |
+| `GITHUB_WEBHOOK_SECRET`, `JIRA_WEBHOOK_SECRET`, `LINEAR_WEBHOOK_SECRET` | One per webhook source you use. A source whose secret is unset gets a 503 and queues nothing |
+| `LOCALDOST_LLM`, `ANTHROPIC_API_KEY` | `anthropic` and an API key on a server. A Claude CLI login lives on one machine, so `claude-cli` fits only your own machine |
+| `GITHUB_TOKEN` | From `gh auth token`, or a fine-grained PAT. For a GitHub App, set `GITHUB_APP_ID` and `GITHUB_PRIVATE_KEY` instead, see [GITHUB.md](GITHUB.md) |
+| `LINEAR_API_KEY`, or `JIRA_BASE_URL` with `JIRA_EMAIL` and `JIRA_API_TOKEN` | The tracker the dot watches, see [TRACKERS.md](TRACKERS.md). Jira has no poller, so it needs a public URL for its webhook |
+
+`LOCALDOST_TOKEN` and `LOCALDOST_DATA_DIR` must come from the environment. Once the server runs, you can connect the model, GitHub, the tracker, Slack, voice, and the public URL from the Setup page in the app instead of the variables above. Saved values live encrypted in `LOCALDOST_DATA_DIR`, so that folder must survive restarts. Every setting is listed in [SETUP.md](SETUP.md). Send a secret to a host from stdin or from a command. Never type it as a literal on the command line, and never paste it into a chat. Each recipe below shows how.
+
+## Railway
+
+The service, its volume, and its non-secret variables come from `.railway/railway.ts` (Railway Infrastructure as Code). Run these commands in the localdost checkout. They need Railway CLI 5.42.1 or newer. `railway login` opens a browser, so a person must do that step.
+
+```bash
+cd ~/localdost
+railway login
+# The TinyBee staging lives in the existing project `localdost`; a fresh install makes its own.
+railway link   # or: railway init --name localdost
+
+# Preview, then create the service, the 5 GB volume at /data, and the non-secret variables
+railway config plan
+railway config apply --yes
+
+# Secrets: from stdin, so the value never shows in the shell history or the chat
+openssl rand -hex 24 | railway variable set LOCALDOST_TOKEN --stdin --skip-deploys --service localdost
+gh auth token | railway variable set GITHUB_TOKEN --stdin --skip-deploys --service localdost
+printenv ANTHROPIC_API_KEY | railway variable set ANTHROPIC_API_KEY --stdin --skip-deploys --service localdost
+printenv LINEAR_API_KEY | railway variable set LINEAR_API_KEY --stdin --skip-deploys --service localdost
+
+railway up --detach --service localdost
+railway domain --port 4319 --service localdost
+```
+
+`railway domain` prints the URL, for example `https://localdost-production.up.railway.app`. Add it to the `env` block of `.railway/railway.ts` as `LOCALDOST_PUBLIC_URL`, run `railway config apply --yes` again (variables set only with the CLI and not declared in the file are removed by apply), then create the dot on the server. `railway run` passes the Railway variables, `LOCALDOST_TOKEN` included, to the local command and prints none of them:
+
+```bash
+railway run --service localdost node bin/localdost.js init --yes --dir ~/code/shop-web --server https://localdost-production.up.railway.app
+railway run --service localdost node bin/localdost.js doctor --dir ~/code/shop-web
+```
+
+To open the app on your phone you need the token value once. `railway variable list --kv --service localdost` prints it in your own terminal. Do not paste it into a chat.
+
+Set only the secrets you use. For Jira instead of Linear, add `JIRA_BASE_URL`, `JIRA_EMAIL`, and `JIRA_API_TOKEN` to the `env` block of the file with `preserve()` and set them with `railway variable set` the same way.
+
+### .railway/railway.ts
+
+The file declares two services from the same image: `localdost` (the server and the PWA) and `hive` (the TinyBee control plane, docs/tinybee). `LOCALDOST_ROLE=hive` in the `hive` service makes `scripts/docker-entrypoint.sh` start `node dist/hive/index.js` instead of the server; nothing else differs between the two containers.
+
+The file declares the project `localdost` with one service `localdost`, one replica, the start command `node dist/index.js`, a health check on `/` with a 60 second timeout, a volume `localdost-data` mounted at `/data`, and the deploy guards described under [Deploy guards](#deploy-guards). It sets `HOST`, `PORT`, `LOCALDOST_DATA_DIR`, `LOCALDOST_POLL`, `LOCALDOST_LLM`, and `RAILWAY_RUN_UID`. The four secrets are declared with `preserve()`, which keeps the value you set with `railway variable set`, so the file holds no secret. It sets no `source`: `railway up` uploads the folder, or you link a GitHub repo to the service.
+
+#### Deploy guards
+
+The `deploy` block and the service settings carry these guards:
+
+| Setting | Value | What it does |
+|---|---|---|
+| `requiredMountPath` | `/data` | A deploy fails when the volume is not mounted at `/data`, so the database and artifacts never land on the container disk and vanish on the next deploy |
+| `healthcheck`, `healthcheckTimeout` | `/`, 60 seconds | A new deploy takes over only after `/` answers within 60 seconds |
+| `restartPolicyType`, `restartPolicyMaxRetries` | `ON_FAILURE`, 5 | A crashed server restarts up to 5 times, then stays stopped so the failure is visible |
+| `drainingSeconds` | 10 | The old instance has 10 seconds after SIGTERM to finish its SQLite writes |
+| `replicas` | 1 | SQLite allows one writer |
+
+Keep them when you edit the file. They replace the same settings that the deprecated `railway.json` held.
+
+#### Non-root user
+
+The `runner` image runs as the user `localdost` (uid 10001), which owns `/app` and `/data`. The Playwright browsers in `/ms-playwright` are readable by every user. Docker and Docker Compose start the container as `localdost`.
+
+Railway mounts volumes as root, and `RAILWAY_RUN_UID=0` makes it start the container as root so `/data` is reachable. The entrypoint (`scripts/docker-entrypoint.sh`) then gives `/data` to `localdost` and switches to that user with `setpriv` before the server starts. The server, the git checkouts, and the test runs all execute as `localdost`, never as root. In a shell on the service, `ps -o user= -C node` prints `localdost`.
+
+The file has no build target. Railway builds the root `Dockerfile` and takes its last stage, which is `runner` (the Playwright image with browsers). Keep `runner` last in the `Dockerfile`; a test checks it. Edit `.railway/railway.ts`, run `railway config plan`, then `railway config apply`. The `railway` package in `devDependencies` only typechecks the file (`npm run typecheck`). The production image does not install it.
+
+Railway deprecated Config as Code (`railway.json`) and stops reading it on 2026-12-01. New services ignore it, which is why this repo no longer has one.
+
+### Railway limits
+
+- Test runs work. The default image is the `runner` target with Chromium, Firefox, and WebKit. The first run of each cloned repo installs its dependencies (`npm ci`, falling back to `npm install`), later runs with the same lockfile restore them from the cache in `/data`. The image has the browsers for Playwright 1.63; a repo that pins another Playwright version needs `npx playwright install chromium` in its `runner.setup`. Set `runner.setup` on the dot to replace that install, see [SANDBOX.md](SANDBOX.md#setup-steps). There is no Docker daemon, so the `docker` runner mode is not available.
+- Chromium needs memory. Give the service at least 2 GB of RAM, and size the volume (5 GB by default) for the database, artifacts, and the dependency cache. A volume can grow but not shrink.
+- SQLite needs exactly one replica. Keep `replicas` at 1.
+- With a volume, each redeploy stops the old instance before the new one starts. Expect a few seconds of downtime per deploy.
+
+## Docker Compose
+
+`compose.yml` starts the app on port 4319 with a named volume `localdost-data` for `/data`. Keep the token in `.env` next to `compose.yml` (gitignored), so you can read it again later:
+
+```bash
+cd ~/localdost
+printf 'LOCALDOST_TOKEN=%s\n' "$(openssl rand -hex 24)" >> .env
+docker compose up -d --build
+docker compose logs -f app
+```
+
+Put the other settings from the table above in the same `.env`. Docker Compose reads it, and `.dockerignore` keeps it out of the image.
+
+Test runs need the `runner` target, which has the Playwright browsers. Add `compose.override.yml`:
+
+```yaml
+services:
+  app:
+    build:
+      target: runner
+    image: localdost:runner
+```
+
+Create the dot from your machine with the token in your shell:
+
+```bash
+export LOCALDOST_TOKEN="$(grep '^LOCALDOST_TOKEN=' ~/localdost/.env | cut -d= -f2)"
+localdost init --yes --server http://your-server:4319
+```
+
+Back up the volume as described in [Backups on Docker Compose](#backups-on-docker-compose). A `tar` of the volume while the app runs can miss writes that sit in the `-wal` file.
+
+For HTTPS, the `tunnel` profile in `compose.yml` runs cloudflared next to the app. Create a tunnel in the Cloudflare dashboard that points to `http://app:4319`, put `CLOUDFLARE_TUNNEL_TOKEN` and `LOCALDOST_PUBLIC_URL` in `.env`, then run `docker compose --profile tunnel up -d`.
+
+## Your machine plus cloudflared
+
+The server runs from the repo the dot watches, so the dot runs tests in your checkout with your browsers.
+
+```bash
+cd ~/code/shop-web
+localdost init
+localdost doctor
+localdost start
+```
+
+Before you open the server to the internet, give it a token. On macOS, keep the token in the Keychain and reference it from `.env.localdost`:
+
+```bash
+security add-generic-password -a localdost -s LOCALDOST_TOKEN -w "$(openssl rand -hex 24)"
+```
+
+```bash
+# .env.localdost
+LOCALDOST_TOKEN=${cmd:security find-generic-password -a localdost -s LOCALDOST_TOKEN -w}
+```
+
+On Linux, `pass` works the same way: `LOCALDOST_TOKEN=${cmd:pass show localdost/token}`. A `${cmd:...}` reference runs only a read-only command (`gh auth token`, `op read`, `security find-generic-password`, or `pass show`), and never through a shell.
+
+A quick tunnel needs no Cloudflare account. Its URL changes on every start. With `cloudflared` installed, the Public URL row in Setup starts one with **Quick tunnel**. To start it yourself:
+
+```bash
+cloudflared tunnel --url http://localhost:4319
+```
+
+For a URL that stays the same, use a named tunnel on a domain in your Cloudflare account:
+
+```bash
+cloudflared tunnel login
+cloudflared tunnel create localdost
+cloudflared tunnel route dns localdost qa.example.com
+cloudflared tunnel run --url http://localhost:4319 localdost
+```
+
+Then enter `https://qa.example.com` in Setup with **My own address** on the Public URL row, or set `LOCALDOST_PUBLIC_URL=https://qa.example.com` in `.env.localdost` and restart `localdost start`. The server stays on 127.0.0.1. cloudflared reaches it there, and only the tunnel is public.
+
+## Check a deploy
+
+`localdost doctor --server <url>` asks the server for `/api/setup` and lists the dots on it. It exits 1 when the server, the model, GitHub, the tracker, or the dot is not ready, and prints a fix command for each line. Two other checks answer one question each: `localdost doctor --restore-check <folder>` says whether a backup can be restored, and `localdost doctor --idle` says whether a run or job is active. Both are in the sections below.
+
+## Back up and restore
+
+localdost keeps its state in one folder, `LOCALDOST_DATA_DIR` (`/data` in the Docker image). Back it up every day and before each upgrade. Check each backup with `localdost doctor --restore-check`. Try one restore before you need it. The host recipes at the end of this section cover [Docker Compose](#backups-on-docker-compose), [Node with a start script](#backups-on-node-with-a-start-script), [Railway](#backups-on-railway), and [Kubernetes](#backups-on-kubernetes).
+
+### What to copy
+
+| In the data folder | What it holds | Copy it? |
+|---|---|---|
+| `localdost.db` | The database: dots, runs, drafts, approvals, paired devices, and the settings saved in Setup. It runs in WAL mode, so its newest writes sit in `localdost.db-wal` and `localdost.db-shm` beside it | Always, with the SQLite backup command below. A file copy can miss those writes |
+| `secret.key` | The key that encrypts the tokens saved in Setup. It also signs the hidden marker in every comment the dot posts | Always, from the same moment as the database |
+| `vapid.json` | The push keys of this server | Always. Without it, each phone turns notifications on again |
+| `artifacts/` | Screenshots, traces, and reports of past runs | When old run pages must keep their evidence. It can be large |
+| `eval-candidates/` | Review cases saved for the eval set | When you use them |
+| `mirrors/`, `checkouts/`, `worktrees/`, `runner-cache/`, `repo-cache/`, `graph-cache/` | Git mirrors, checkouts, and caches. localdost rebuilds them | No |
+
+Two things live outside the data folder. When `LOCALDOST_DB` is set, the database is the file it names, so back up that file. The settings you give the server as variables sit in the host's own store, outside the data folder: `LOCALDOST_TOKEN`, `GITHUB_TOKEN`, `ANTHROPIC_API_KEY`, the webhook secrets, and the secret variable of each MCP connection (a connection stores only its name). That store is the `.env` next to `compose.yml`, `.env.localdost`, the Railway variables, or a Kubernetes Secret. Keep those values in your secret manager.
+
+### A backup without secret.key
+
+Setup encrypts every token it saves, and `secret.key` holds the key. A backup without that file restores the database, but every saved token reads as not set. A dot that relies on saved tokens cannot reach GitHub, the tracker, or the model until you enter them again in Setup.
+
+The same key signs the hidden marker in every comment and issue the dot posts. A restore without it makes a new key, so the markers in the dot's earlier comments no longer verify, and the dot reads its own earlier comments as written by a person, see [TRACKERS.md](TRACKERS.md). This holds when every token is a variable on the host and Setup saved none. `localdost doctor --restore-check` fails for such a backup.
+
+When you set `LOCALDOST_SECRET_KEY` on the host, the server makes no `secret.key` file. Keep that value in your secret manager with the other variables. Export the same value in the shell that runs `--restore-check`, and the check uses it in place of the file. Set it before you save the first token, because a value saved under one key cannot be read with another.
+
+A backup that holds both the database and `secret.key` lets anyone who opens it read every saved token. Keep backups where only the admins can read them.
+
+### Take a copy while the server runs
+
+SQLite copies a live database in one consistent step, and the server keeps running. Node 24 has this built in. The images have no `sqlite3` command, so the recipes use Node:
+
+The recipes below set `db` to the database file the server uses: the file `LOCALDOST_DB` names when it is set, else `localdost.db` in `LOCALDOST_DATA_DIR`. Use the same two variables as the server, or the backup copies another file.
+
+```bash
+db="${LOCALDOST_DB:-$LOCALDOST_DATA_DIR/localdost.db}"
+SNAP='const s=require("node:sqlite");const d=new s.DatabaseSync(process.argv[1],{readOnly:true});s.backup(d,process.argv[2]).then(()=>d.close())'
+node -e "$SNAP" "$db" ~/localdost-backups/now/localdost.db
+```
+
+Where the `sqlite3` command exists, `sqlite3 "$db" ".backup '$HOME/localdost-backups/now/localdost.db'"` does the same. If you can stop the server for a minute, copying the whole folder while it is stopped also works.
+
+Do not copy `localdost.db` with `cp`, `tar`, or a volume copy while the server runs. The file alone lacks the writes that sit in `localdost.db-wal`. `--restore-check` reports a copy that shows no tables at all, and a copy that has a `-wal` file beside it. It cannot report a copy that only lacks the newest writes. That copy can pass every check and still miss the last hours of work, so take every copy with the backup command above.
+
+### How often
+
+Back up every day and before every upgrade. A restore loses everything after its backup: runs, drafts, approvals, and decisions. Keep the last 7 daily copies, and keep the copy taken before the last upgrade. Run the backup from cron, launchd, or whatever scheduler your host has.
+
+### Check a backup
+
+Run this on the backup folder after each backup:
+
+```bash
+localdost doctor --restore-check ~/localdost-backups/2026-10-14-0300
+```
+
+```text
+ok    Backup folder  /home/me/localdost-backups/2026-10-14-0300
+ok    Database       opens read-only in a temp copy and passes its self-check
+ok    Schema         matches localdost 0.10.0 (34 tables)
+ok    Saved tokens   secret.key decrypts all 4 saved values
+ok    Push keys      vapid.json found
+
+This backup can be restored.
+```
+
+The check copies `localdost.db` (and its `-wal` and `-shm` files, when the backup has them) to a temp folder and opens the copy read-only, so the backup is never changed. It prints no token, key, or saved value. The exit code is 0 when no line says `FAIL`, and 1 otherwise. A `warn` line does not change the exit code.
+
+| Line | What it tests |
+|---|---|
+| Database | The file opens and passes SQLite's quick self-check. A warning follows when the backup holds a `-wal` file, because then the database was copied as a plain file. It cannot see that a plain copy lacks the newest writes |
+| Schema | Every table and column is one this version creates, the tables that the start migrates and the ones their modules create on first use (report pages, saved Setup values, MCP connections, poll cursors). An older backup passes, and the line says what the next start adds. A backup from a newer version fails on the first table or column this version does not know: restore it with that tag or a newer one |
+| Saved tokens | `secret.key` from the backup decrypts the saved values. It fails when the key is missing, also when the backup holds no saved tokens, because the same key signs the marker in every comment the dot posts. It fails when the key decrypts none of the values, and when it decrypts only some of them, since the rest would be lost. It warns when the key is there and the backup holds no saved tokens, because there is nothing to test it against. When the backup has no `secret.key`, the check uses `LOCALDOST_SECRET_KEY` from your shell |
+| Push keys | `vapid.json` is in the backup. A warning says each phone must turn notifications on again |
+
+Run it from a checkout of the version you plan to restore into. `localdost` is the command that `npm link` makes, see the README. Without it, use `node bin/localdost.js`. The Docker image has no `bin` folder, so inside a container use `node /app/dist/cli/index.js doctor --restore-check <folder>`. The check needs room in the temp folder for one copy of `localdost.db`.
+
+### Restore
+
+1. Run `localdost doctor --restore-check` on the backup folder.
+2. Stop the server. No process may have the database open.
+3. Copy the current database file, its `-wal` and `-shm` files, `secret.key`, and `vapid.json` somewhere safe. The database file is the one the server uses: `LOCALDOST_DB` when set, else `localdost.db` in the data folder. Copy the ones that exist. You may need them if the backup is wrong.
+4. Delete the `-wal` and `-shm` files that sit beside the database. An old `-wal` file beside a restored database can corrupt it.
+5. Put the database at the path the server uses, and `secret.key` and `vapid.json` from the backup in the data folder, the ones the backup holds. Add `artifacts/` if you backed it up. A host that sets `LOCALDOST_SECRET_KEY` has no `secret.key`, and a host that sets the `VAPID_*` variables has no `vapid.json`. Skip a file that is not in the backup.
+6. Give the files to the user that runs the server, and keep `secret.key` at mode 600.
+7. Start the server. Read the version at `/api/health`, then open Setup. A saved token that shows as not set means the key is wrong.
+
+The recipes below do steps 3 to 6 in that order. Each one saves the current files first, copies only the files the backup holds, and removes no file until the safety copy is made. Set `out` in each one to the backup folder that passed the check.
+
+The restored database knows nothing after the backup. The dot can review a PR again when that PR changes, and comments it already posted stay on GitHub. A backup older than an hour counts as a gap, and polling catches up as described in [Poll mode](SETUP.md#poll-mode). GitHub does not resend webhook events that a stopped server missed, unless you redeliver them in the webhook settings.
+
+### Backups on Docker Compose
+
+The data folder is the volume `localdost_localdost-data`, mounted at `/data` in the `app` service. The `db` line asks the container for the database path the server uses, which is `/data/localdost.db` unless the service sets `LOCALDOST_DB`. Copy that database, `secret.key`, and `vapid.json`. A backup without `secret.key` cannot decrypt the saved tokens. Run this once a day from the folder that holds `compose.yml`:
+
+```bash
+out=~/localdost-backups/$(date +%F-%H%M) && mkdir -p "$out" && chmod 700 "$out"
+SNAP='const s=require("node:sqlite");const d=new s.DatabaseSync(process.argv[1],{readOnly:true});s.backup(d,process.argv[2]).then(()=>d.close())'
+db=$(docker compose exec -T app sh -c 'echo "${LOCALDOST_DB:-$LOCALDOST_DATA_DIR/localdost.db}"')
+docker compose exec -T app node -e "$SNAP" "$db" /data/backup.tmp.db
+docker compose cp app:/data/backup.tmp.db "$out/localdost.db"
+docker compose exec -T app rm /data/backup.tmp.db
+docker compose cp app:/data/secret.key "$out/secret.key"
+docker compose cp app:/data/vapid.json "$out/vapid.json"
+docker compose cp app:/data/artifacts "$out/artifacts"   # optional, and can be large
+node bin/localdost.js doctor --restore-check "$out"
+```
+
+To restore, stop the app, put the files in the volume with a throwaway container, and start the app. Read the database path from the running app first, as in the backup. It must lie in the volume. The container first copies the current files to `$before` on the host. It stops before it removes anything when that copy fails. Then it copies each file the backup holds and gives it the owner of `/data` itself: user `node` in the `app` image and user `localdost` in the `runner` image. A file that the backup lacks stays as it is in the volume.
+
+```bash
+out=~/localdost-backups/2026-10-14-0300   # the backup folder that passed the check
+before=~/localdost-backups/before-restore-$(date +%F-%H%M) && mkdir -p "$before" && chmod 700 "$before"
+db=$(docker compose exec -T app sh -c 'echo "${LOCALDOST_DB:-$LOCALDOST_DATA_DIR/localdost.db}"')
+docker compose stop app
+docker run --rm -e DB="$db" -v localdost_localdost-data:/data -v "$out":/backup:ro -v "$before":/before alpine sh -ec '
+  for s in "" -wal -shm; do
+    if [ -f "$DB$s" ]; then cp -p "$DB$s" "/before/localdost.db$s"; fi
+  done
+  for f in secret.key vapid.json; do
+    if [ -f /data/$f ]; then cp -p /data/$f /before/; fi
+  done
+  rm -f "$DB" "$DB-wal" "$DB-shm"
+  if [ -f /backup/localdost.db ]; then
+    cp /backup/localdost.db "$DB"
+    chown "$(stat -c %u:%g /data)" "$DB"
+  fi
+  for f in secret.key vapid.json; do
+    if [ -f /backup/$f ]; then
+      cp /backup/$f /data/$f
+      chown "$(stat -c %u:%g /data)" /data/$f
+    fi
+  done
+  if [ -f /data/secret.key ]; then chmod 600 /data/secret.key; fi'
+docker compose start app
+```
+
+### Backups on Node with a start script
+
+This host runs `node dist/index.js`, or `localdost start`, from a checkout under a script of your own that sets the environment. For example `start.sh`. The repo does not ship that script. Set `LOCALDOST_DATA_DIR` in it, and `LOCALDOST_DB` when it sets that too, and export the same values in the shell that runs the backup. Copy the database, `secret.key`, and `vapid.json`. A backup without `secret.key` cannot decrypt the saved tokens. Put these lines in a script and run it daily:
+
+```bash
+out=~/localdost-backups/$(date +%F-%H%M) && mkdir -p "$out" && chmod 700 "$out"
+SNAP='const s=require("node:sqlite");const d=new s.DatabaseSync(process.argv[1],{readOnly:true});s.backup(d,process.argv[2]).then(()=>d.close())'
+db="${LOCALDOST_DB:-$LOCALDOST_DATA_DIR/localdost.db}"
+node -e "$SNAP" "$db" "$out/localdost.db"
+cp "$LOCALDOST_DATA_DIR/secret.key" "$LOCALDOST_DATA_DIR/vapid.json" "$out/"
+cd ~/localdost && node bin/localdost.js doctor --restore-check "$out"
+```
+
+To restore, stop the server the way your start script does, then:
+
+```bash
+out=~/localdost-backups/2026-10-14-0300   # the backup folder that passed the check
+before=~/localdost-backups/before-restore-$(date +%F-%H%M) && mkdir -p "$before" && chmod 700 "$before"
+db="${LOCALDOST_DB:-$LOCALDOST_DATA_DIR/localdost.db}"
+for s in "" -wal -shm; do
+  if [ -f "$db$s" ]; then cp -p "$db$s" "$before/localdost.db$s"; fi
+done
+for f in secret.key vapid.json; do
+  if [ -f "$LOCALDOST_DATA_DIR/$f" ]; then cp -p "$LOCALDOST_DATA_DIR/$f" "$before/"; fi
+done
+rm -f "$db-wal" "$db-shm"
+if [ -f "$out/localdost.db" ]; then cp "$out/localdost.db" "$db"; fi
+for f in secret.key vapid.json; do
+  if [ -f "$out/$f" ]; then cp "$out/$f" "$LOCALDOST_DATA_DIR/"; fi
+done
+if [ -f "$LOCALDOST_DATA_DIR/secret.key" ]; then chmod 600 "$LOCALDOST_DATA_DIR/secret.key"; fi
+```
+
+Run the commands as the user that owns the data folder, so the copied files keep that owner. The current files are in `$before`. Then start the server with `start.sh`.
+
+### Backups on Railway
+
+The data folder is the volume `localdost-data`, mounted at `/data`. The database is `/data/localdost.db` unless you set `LOCALDOST_DB` on the service, and then it is the file that variable names, which must lie in the volume. Copy the database, `secret.key`, and `vapid.json`. A backup without `secret.key` cannot decrypt the saved tokens. The other secrets are Railway variables: keep their values in your secret manager. `railway variable list --kv --service localdost` prints them in your own terminal, and you never paste the output into a chat.
+
+The snapshot runs inside the service, and `railway volume files` brings the files to your machine. Open a shell on the running service, paste the next three lines, then type `exit`. The `db` line takes `LOCALDOST_DB` from the service. Run `echo "$db"` to read it. When you set `LOCALDOST_DB` and it prints the default, the shell did not carry the variable, so set `db` to that path by hand:
+
+```bash
+railway ssh --service localdost
+db="${LOCALDOST_DB:-/data/localdost.db}"
+SNAP='const s=require("node:sqlite");const d=new s.DatabaseSync(process.argv[1],{readOnly:true});s.backup(d,process.argv[2]).then(()=>d.close())'
+node -e "$SNAP" "$db" /data/backup.tmp.db
+exit
+```
+
+```bash
+out=~/localdost-backups/$(date +%F-%H%M) && mkdir -p "$out" && chmod 700 "$out"
+railway volume files --volume localdost-data download /backup.tmp.db "$out/localdost.db"
+railway volume files --volume localdost-data download /secret.key "$out/secret.key"
+railway volume files --volume localdost-data download /vapid.json "$out/vapid.json"
+railway ssh --service localdost -- rm /data/backup.tmp.db
+node bin/localdost.js doctor --restore-check "$out"
+```
+
+To restore, remove the running deployment and download the current files to `$before`. `db` is the path the server uses, as in the backup, and `dbv` is the same path inside the volume:
+
+```bash
+out=~/localdost-backups/2026-10-14-0300   # the backup folder that passed the check
+before=~/localdost-backups/before-restore-$(date +%F-%H%M) && mkdir -p "$before" && chmod 700 "$before"
+db=/data/localdost.db   # or the file LOCALDOST_DB names, when you set that on the service. It lies inside /data
+dbv=${db#/data}
+railway down --service localdost --yes
+for s in "" -wal -shm; do
+  railway volume files --volume localdost-data download "$dbv$s" "$before/localdost.db$s"
+done
+for f in secret.key vapid.json; do
+  railway volume files --volume localdost-data download "/$f" "$before/$f"
+done
+ls -l "$before"
+```
+
+A download line prints an error when its file is not on the volume. That is normal for the `-wal` and `-shm` files of the database after a clean stop, and for `secret.key` and `vapid.json` when the host sets them as variables. Go on only when `$before` holds `localdost.db`. Then replace the files on the volume and deploy again from the tag you run:
+
+```bash
+railway volume files --volume localdost-data delete "$dbv-wal" --yes
+railway volume files --volume localdost-data delete "$dbv-shm" --yes
+if [ -f "$out/localdost.db" ]; then railway volume files --volume localdost-data upload "$out/localdost.db" "$dbv" --overwrite; fi
+for f in secret.key vapid.json; do
+  if [ -f "$out/$f" ]; then railway volume files --volume localdost-data upload "$out/$f" "/$f" --overwrite; fi
+done
+railway up --detach --service localdost
+```
+
+The entrypoint gives `/data` to `localdost` only when the folder itself belongs to someone else. If the server then fails with `attempt to write a readonly database` or `unable to open database file`, the uploaded files belong to another user. Run `railway ssh --service localdost -- chown -R localdost:localdost /data`, then `railway restart --service localdost --yes`. A delete line prints an error when its `-wal` or `-shm` file is already gone. Ignore that error.
+
+### Backups on Kubernetes
+
+This repo ships no manifests. The steps assume one Deployment `localdost` with a single replica and the strategy `Recreate`, a PersistentVolumeClaim `localdost-data` mounted at `/data`, the label `app=localdost`, a container named `localdost`, and the namespace `localdost`. SQLite allows one writer, so the Deployment must never run two pods on the volume. The database is `/data/localdost.db` unless the Deployment sets `LOCALDOST_DB`, and then it is the file that variable names, which must lie on the claim. Copy the database, `secret.key`, and `vapid.json`. A backup without `secret.key` cannot decrypt the saved tokens. Keep the other secrets in the Kubernetes Secret that the Deployment reads, and keep a copy of that Secret in your secret manager.
+
+```bash
+out=~/localdost-backups/$(date +%F-%H%M) && mkdir -p "$out" && chmod 700 "$out"
+pod=$(kubectl -n localdost get pod -l app=localdost -o jsonpath='{.items[0].metadata.name}')
+SNAP='const s=require("node:sqlite");const d=new s.DatabaseSync(process.argv[1],{readOnly:true});s.backup(d,process.argv[2]).then(()=>d.close())'
+db=$(kubectl -n localdost exec "$pod" -- sh -c 'echo "${LOCALDOST_DB:-$LOCALDOST_DATA_DIR/localdost.db}"')
+kubectl -n localdost exec "$pod" -- node -e "$SNAP" "$db" /data/backup.tmp.db
+kubectl -n localdost cp "$pod":/data/backup.tmp.db "$out/localdost.db"
+kubectl -n localdost exec "$pod" -- rm /data/backup.tmp.db
+kubectl -n localdost cp "$pod":/data/secret.key "$out/secret.key"
+kubectl -n localdost cp "$pod":/data/vapid.json "$out/vapid.json"
+node bin/localdost.js doctor --restore-check "$out"
+```
+
+A volume snapshot of the claim makes a good second copy. It does not replace this one, because it carries no check of the schema or the key.
+
+To restore, scale the Deployment to zero and start a helper pod that mounts the claim. Use the image of the Deployment, so the pod has `tar`. Use the same `runAsUser` and `fsGroup`, so the files get the right owner:
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: localdost-restore
+  namespace: localdost
+spec:
+  restartPolicy: Never
+  securityContext:
+    runAsUser: 10001
+    fsGroup: 10001
+  containers:
+    - name: shell
+      image: <the image of the Deployment>
+      command: ['sleep', '3600']
+      volumeMounts:
+        - name: data
+          mountPath: /data
+  volumes:
+    - name: data
+      persistentVolumeClaim:
+        claimName: localdost-data
+```
+
+Read the database path from the running pod, as in the backup. Then scale down, start the pod, and copy the current files to `$before` on your machine:
+
+```bash
+out=~/localdost-backups/2026-10-14-0300   # the backup folder that passed the check
+before=~/localdost-backups/before-restore-$(date +%F-%H%M) && mkdir -p "$before" && chmod 700 "$before"
+pod=$(kubectl -n localdost get pod -l app=localdost -o jsonpath='{.items[0].metadata.name}')
+db=$(kubectl -n localdost exec "$pod" -- sh -c 'echo "${LOCALDOST_DB:-$LOCALDOST_DATA_DIR/localdost.db}"')
+kubectl -n localdost scale deploy/localdost --replicas=0
+kubectl -n localdost apply -f restore-pod.yaml
+kubectl -n localdost wait --for=condition=Ready pod/localdost-restore
+for s in "" -wal -shm; do
+  if kubectl -n localdost exec localdost-restore -- test -f "$db$s"; then kubectl -n localdost cp "localdost-restore:$db$s" "$before/localdost.db$s"; fi
+done
+for f in secret.key vapid.json; do
+  if kubectl -n localdost exec localdost-restore -- test -f "/data/$f"; then kubectl -n localdost cp "localdost-restore:/data/$f" "$before/$f"; fi
+done
+ls -l "$before"
+```
+
+The listing shows what the pod held. Go on only when `$before` holds `localdost.db`. Then replace the files and scale up. The copy lines run only for a file the backup holds:
+
+```bash
+kubectl -n localdost exec localdost-restore -- rm -f "$db" "$db-wal" "$db-shm"
+if [ -f "$out/localdost.db" ]; then kubectl -n localdost cp "$out/localdost.db" "localdost-restore:$db"; fi
+for f in secret.key vapid.json; do
+  if [ -f "$out/$f" ]; then kubectl -n localdost cp "$out/$f" "localdost-restore:/data/$f"; fi
+done
+kubectl -n localdost exec localdost-restore -- sh -c 'if [ -f /data/secret.key ]; then chmod 600 /data/secret.key; fi'
+kubectl -n localdost delete pod localdost-restore
+kubectl -n localdost scale deploy/localdost --replicas=1
+```
+
+## Upgrade
+
+An upgrade is a new tag, a rebuild, and a restart. The steps are the same on every host, and only the commands differ. A restart interrupts every running run, so wait until nothing runs.
+
+1. Pick a tag. Run `git fetch --tags` in the localdost checkout, then `git tag --list 'v*' --sort=-v:refname | head -5`. The version you run now is `version` in the answer of `GET /api/health`.
+2. Read `CHANGELOG.md` in the repo for every version from yours up to the new tag. A minor version (0.x.0) adds features or breaks something. A patch version (0.x.y) only fixes bugs. Read Changed and Security first.
+3. Wait until no run or job is active, see below.
+4. Back up the data folder and check the backup, see [Back up and restore](#back-up-and-restore). This backup is what you restore on a rollback.
+5. Switch to the tag, build, and restart, with the commands for your host below.
+6. Check the result, see below.
+
+### Wait until idle
+
+`localdost doctor --idle` exits 0 when no run and no job is active, 1 when one is, and 2 when it cannot read the database. It reads the database the server uses read-only, and it prints one line each for runs and jobs. It finds that file the way the server does: `LOCALDOST_DB` when set, else `localdost.db` in `LOCALDOST_DATA_DIR`, each from your shell first and then from `.env.localdost` in `--dir` (default: the current folder). A `${cmd:...}` reference in that file is never run, so export the value in your shell instead. Run it as the user that runs the server. A run that is going, a job that is going, and a job that is ready to start all count. A job scheduled for later (a debounce wait, a retry back-off) keeps its row and starts after the restart, so it does not count.
+
+While the disk is low or a write failed, localdost holds back new test runs, and the jobs that wait for them count as waiting to start until space returns. Free disk space and the count drops. A restart does not interrupt a job that only waits, so you can also restart anyway.
+
+| Host | Command |
+|---|---|
+| Docker Compose | `docker compose exec -T app node dist/cli/index.js doctor --idle` |
+| Node with a start script | `localdost doctor --idle`, with `LOCALDOST_DATA_DIR` and `LOCALDOST_DB` exported as in your start script, or run from the folder of `.env.localdost` |
+| Railway | `railway ssh --service localdost -- env LOCALDOST_DATA_DIR=/data node /app/dist/cli/index.js doctor --idle` |
+| Kubernetes | `kubectl -n localdost exec deploy/localdost -- env LOCALDOST_DATA_DIR=/data node /app/dist/cli/index.js doctor --idle` |
+
+```text
+ok    Database  /data/localdost.db
+FAIL  Runs      1 running
+                fix: wait and run this again. A restart now interrupts a running run, and its run page offers Retry
+ok    Jobs      none running or waiting to start
+
+Not idle. See the lines above, wait, then run this again.
+```
+
+On a Node host you can wait in a loop:
+
+```bash
+until localdost doctor --idle; [ $? -ne 1 ]; do sleep 30; done
+```
+
+The loop prints each answer. It waits while the exit code is 1, and it stops on 0 (idle) and on 2 (it cannot tell, so read the line it printed). A wrong `LOCALDOST_DATA_DIR` or a wrong user gives 2. The answer is true at one moment, and a webhook can start a job a second later. If a restart cuts a run off, it stays in the feed as Interrupted with a Retry button, and a queued job waits for the new process. If the server is stopped and the check still finds a running run, a crash left it behind, and the next start marks it interrupted.
+
+### Switch to the new tag
+
+```bash
+cd ~/localdost
+git fetch --tags && git checkout v0.11.0   # the tag you picked
+```
+
+Then, for your host:
+
+```bash
+# Docker Compose: rebuilds the image and recreates the container
+docker compose up -d --build
+
+# Node with a start script: stop the server first, so the build does not replace files that a running server reads
+npm ci && npm run build
+bash ./start.sh
+
+# Railway: uploads this folder and deploys it
+railway up --detach --service localdost
+
+# Kubernetes: build and push the image of the tag, then set it. The Recreate strategy stops the old pod first
+docker build --target runner -t <registry>/localdost:v0.11.0 . && docker push <registry>/localdost:v0.11.0
+kubectl -n localdost set image deploy/localdost localdost=<registry>/localdost:v0.11.0
+kubectl -n localdost rollout status deploy/localdost
+```
+
+### What runs at start
+
+Every start applies these steps in this order. Each one is safe to run again.
+
+| Step | What it does |
+|---|---|
+| Schema update | Creates missing tables, adds missing columns, and creates missing indexes, in one transaction. It never drops a table or a column. On a full disk it rolls back whole, and the server starts on the old tables |
+| Repo sets | Moves a dot with one repo to the repo-set form and adds its repos to the workspace |
+| Stage settings | Fills the stage settings a dot lacks, and sets the one-time offer of new stages. It never changes a choice you made |
+| Test runs | Gives results from before test runs existed their automated test runs |
+| Default engine | Picks the workspace test engine when none is chosen |
+| Sweeps | Removes PR checkouts, test worktrees, and mirror metadata that a crash left behind |
+| Interrupted work | Marks every run and job that the old process left running as interrupted, just before the queue opens. A run gets a Retry button |
+
+The CHANGELOG names any release that adds a step to this list.
+
+### Check the upgrade
+
+- `GET /api/health` shows the new `version` and `commit`: `curl -s -H "authorization: Bearer $LOCALDOST_TOKEN" https://localdost.example.com/api/health`.
+- The log has `boot ready: queue open`, and `recovered N interrupted job(s)` when the restart cut runs off.
+- The Setup page shows every service you use as ready, and no saved token asks to be entered again.
+
+A restart that takes under an hour needs nothing more: polling continues from its stored cursor. After a longer gap, polling takes the open PRs and the issues that changed during it, at most 5 per dot, and each such run starts with a "Caught up after a gap" step, see [Poll mode](SETUP.md#poll-mode). GitHub does not resend webhook events that a stopped server missed, unless you redeliver them in the webhook settings.
+
+### Roll back
+
+1. Stop the server.
+2. Check out the previous tag and build it again, or set the previous image.
+3. Restore the backup taken before the upgrade, see [Restore](#restore).
+4. Start the server and read `version` at `/api/health`.
+
+Restoring the database is the safe rollback. An upgrade only adds tables and columns, so an older version can often open a newer database. Do not rely on it, because the start steps above also fill in dot settings. A rollback loses what happened after the backup, and the paragraph under [Restore](#restore) tells what that means.