forgecad 0.9.14 → 0.9.15

package/dist/docs-raw/runbook.md

@@ -1,611 +0,0 @@
-# ForgeCAD Hosted Service — Runbook
-
-Operational reference for running, testing, and troubleshooting the forgecad.io hosted service.
-
-For the full documentation index see [INDEX.md](./INDEX.md).
-For environment variables, Docker setup, and deployment details see [deployment.md](deployment.md).
-For dashboards, logs, metrics, and observability access see [platform/observability.md](platform/observability.md).
-For releasing new versions see [RELEASING.md](./RELEASING.md).
-For platform feature docs (auth, projects, sharing, admin, email) see [platform/](./platform/).
-
----
-
-## Quick Reference
-
-| What | Command |
-|------|---------|
-| Start local DB | `npm run dev:db` |
-| Stop local DB | `npm run dev:db:stop` |
-| Start API server | `npm run dev:server` |
-| Start frontend (with API proxy) | `npm run dev:studio` |
-| Run API tests | `bash scripts/test-shares-api.sh` |
-| Run unit tests | `npm run test:unit` |
-| Run full test suite | `npm test` |
-| Health check | `curl http://localhost:5174/api/health` |
-| SSH to production | `ssh hetzner` |
-| Open observability UI | Connect to Tailscale, then open `http://100.118.68.93:3000` |
-| Check observability stack | `bash scripts/prod/observability-status.sh` |
-| Roll back bad web deploy | `npm run prod:rollback -- <VERSION>` |
-| Verify npm tarball in Docker | `npm run release:docker:all` |
-
----
-
-## NPM Release Docker Verification
-
-Use this runbook before publishing a CLI release, after unpublishing/rolling back a bad npm version, or whenever a bug only appears in a clean global install.
-
-The check builds the exact npm tarball, recreates a clean `node:22-slim` container, copies the tarball and a model into it, installs the tarball globally, and runs `forgecad run` inside the container. Stop immediately if any step fails.
-
-### One-command check
-
-```bash
-npm run release:docker:all
-```
-
-Default behavior:
-
-| Setting | Default |
-|---------|---------|
-| Container | `forgecad-release-check` |
-| Docker image | `node:22-slim` |
-| Browser URL | `http://localhost:4173` |
-| Studio port mapping | host `4173` -> container `5173` |
-| Tarball output | `dist-bundles/npm-release-check/forgecad-<version>.tgz` |
-| Smoke model | `examples/api/verification-demo.forge.js` |
-| Run timeout | `20s` |
-
-For a known repro model, override the model path:
-
-```bash
-FORGECAD_RELEASE_MODEL=/Users/kostard/Downloads/plant-pot-holder-marble.forge.js \
-  npm run release:docker:all
-```
-
-To reuse a familiar container name:
-
-```bash
-FORGECAD_RELEASE_CONTAINER=forgecad-debug \
-FORGECAD_RELEASE_MODEL=/Users/kostard/Downloads/plant-pot-holder-marble.forge.js \
-  npm run release:docker:all
-```
-
-### Step-by-step check
-
-Run the lane manually when you want to inspect a specific stage:
-
-```bash
-npm run release:pack
-npm run release:docker:create
-npm run release:docker:copy
-npm run release:docker:install
-npm run release:docker:run
-```
-
-What each step does:
-
-| Step | Purpose |
-|------|---------|
-| `release:pack` | Runs `npm pack` and writes the tarball under `dist-bundles/npm-release-check/`. |
-| `release:docker:create` | Removes any existing release-check container and starts a fresh `node:22-slim` container. |
-| `release:docker:copy` | Copies the tarball to `/tmp/forgecad-release.tgz` and the model to `/tmp/<model-name>` in the container. |
-| `release:docker:install` | Runs `npm install -g /tmp/forgecad-release.tgz` inside the container and prints `forgecad --version`. |
-| `release:docker:run` | Runs `forgecad run /tmp/<model-name>` with update checks disabled and a timeout. |
-| `release:docker:studio` | Starts `forgecad studio /tmp --host 0.0.0.0` in the container. Open `http://localhost:4173` on the host. |
-
-Expected success signal:
-
-```text
-forgecad --version
-# <new version>
-
-✓ Objects: ...
-✓ Verifications: ... pass, 0 fail
-✓ Time: ...
-```
-
-### Manual checks inside the container
-
-After `release:docker:install`, open a shell for interactive checks:
-
-```bash
-docker exec -it forgecad-release-check bash
-forgecad --version
-forgecad login
-forgecad run /tmp/verification-demo.forge.js
-```
-
-If you used a custom container name, replace `forgecad-release-check`.
-
-To open Studio in your host browser without changing the internal ForgeCAD port:
-
-```bash
-npm run release:docker:studio
-```
-
-Then open:
-
-```text
-http://localhost:4173
-```
-
-The helper publishes host port `4173` to the container's normal Studio port `5173`, so `forgecad studio` still runs on its default internal port.
-
-### Cleanup
-
-Remove the release-check container when done:
-
-```bash
-npm run release:docker:clean
-```
-
-The tarball output directory is under ignored `dist-bundles/`, but it can still consume local disk. Remove it when space is tight:
-
-```bash
-rm -rf dist-bundles/npm-release-check
-```
-
-### Useful overrides
-
-| Variable | Use |
-|----------|-----|
-| `FORGECAD_RELEASE_CONTAINER` | Container name to create/use. |
-| `FORGECAD_RELEASE_IMAGE` | Docker image, for example `node:20-slim` or `node:22-slim`. |
-| `FORGECAD_RELEASE_HOST_PORT` | Host browser port, default `4173`. |
-| `FORGECAD_RELEASE_CONTAINER_PORT` | Container Studio port, default `5173`. |
-| `FORGECAD_RELEASE_STUDIO_PROJECT` | Project path inside the container for `release:docker:studio`, default `/tmp`. |
-| `FORGECAD_RELEASE_PACK_DIR` | Local tarball output directory. |
-| `FORGECAD_RELEASE_TARBALL` | Existing tarball to copy instead of the default output path. |
-| `FORGECAD_RELEASE_MODEL` | Model path to copy and run. |
-| `FORGECAD_RELEASE_TIMEOUT_SECONDS` | Timeout for `forgecad run` inside the container. |
-
----
-
-## Local Development Stack
-
-### Minimal (editor only, no auth)
-
-```bash
-npm run dev
-```
-
-No database, no login. For geometry, sketches, viewer work.
-
-### Full stack (auth, projects, sharing)
-
-Three terminals:
-
-```bash
-# 1. Database
-npm run dev:db
-
-# 2. API server (port 5174)
-npm run dev:server
-
-# 3. Frontend (port 5173, proxies /api → 5174)
-npm run dev:studio
-```
-
-Open http://localhost:5173. Dev credentials are hardcoded in `dev:server` — see [deployment.md](deployment.md#local-development).
-
-### Verify the stack is healthy
-
-```bash
-# API health
-curl -s http://localhost:5174/api/health | python3 -m json.tool
-
-# Database
-docker exec forgecad-postgresql-1 psql -U forgecad -d forgecad -c "SELECT count(*) FROM users;"
-
-# Migration state
-docker exec forgecad-postgresql-1 psql -U forgecad -d forgecad -c "SELECT * FROM _migrations ORDER BY id;"
-```
-
----
-
-## Testing
-
-### API tests (end-to-end, from terminal)
-
-These scripts exercise the real server — no mocks, real HTTP, real database.
-
-| Script | What it tests | Prerequisites |
-|--------|--------------|---------------|
-| `scripts/test-shares-api.sh` | Publish, fetch, update, unpublish, auth enforcement | `dev:db` + `dev:server` |
-
-Run:
-
-```bash
-bash scripts/test-shares-api.sh
-```
-
-The script registers a fresh test user, runs through the full lifecycle, and cleans up. Each run uses a unique email so it's safe to re-run.
-
-### Adding new API test scripts
-
-Follow the pattern in `test-shares-api.sh`:
-
-1. Name: `scripts/test-<feature>-api.sh`
-2. Check server health first (`/api/health`)
-3. Register a unique test user per run
-4. Use `acurl()` helper for authenticated requests
-5. Assert HTTP status codes and response fields
-6. Clean up (delete created resources)
-7. Print colored pass/fail per step
-
-### Unit tests
-
-```bash
-npm run test:unit           # vitest
-npm test                    # smoke invariant suite
-npm run test:smoke          # same as npm test
-npm run test:full           # broader regression sweep
-npm run check:suite         # repo invariant suite
-npm run check:suite -- --profile smoke
-npm run test:examples       # maintained fast example smoke lane
-npm run test:examples:full  # full example catalog sweep
-```
-
-### Invariant checks (CLI)
-
-```bash
-node dist-cli/forgecad.js check examples
-node dist-cli/forgecad.js check examples --profile smoke
-node dist-cli/forgecad.js check examples --profile full
-```
-
-Use the smoke profiles for everyday merge checks; use the full profiles when you want the broader, currently noisier regression sweep.
-
----
-
-## Database
-
-### Migrations
-
-Migrations run automatically on server startup (`server/db/migrate.ts`). The system:
-
-1. Maintains a `_migrations` table tracking applied migrations by sequential ID
-2. Runs unapplied migrations in order on boot
-3. Each migration is idempotent SQL (`CREATE TABLE IF NOT EXISTS`, `ALTER ... ADD COLUMN IF NOT EXISTS`)
-
-**To add a new migration:** append to the `MIGRATIONS` array in `server/db/migrate.ts`.
-
-### Inspect the database locally
-
-```bash
-docker exec -it forgecad-postgresql-1 psql -U forgecad -d forgecad
-```
-
-Common queries:
-
-```sql
--- List tables
-\dt
-
--- Users
-SELECT id, email, name, role, created_at FROM users;
-
--- Projects
-SELECT id, slug, name, visibility, owner_id FROM projects;
-
--- Shared/published models
-SELECT share_id, filename, title, owner_id, updated_at FROM shared_files;
-
--- Audit log (recent)
-SELECT action, details, created_at FROM audit_log ORDER BY created_at DESC LIMIT 20;
-
--- Migration state
-SELECT * FROM _migrations ORDER BY id;
-```
-
-### Backup production database
-
-Stream a compressed dump directly to your Mac:
-
-```bash
-ssh hetzner "docker exec \$(docker ps --filter name=postgres -q) pg_dump -Fc -U JR3CsoTkOQypFSdD forgecad" > ~/forgecad-backup-<label>.dump
-```
-
-Use a descriptive label (e.g., `pre-fileids-20260409`). Always take a backup before deploying migrations that modify existing data.
-
-### Restore production database
-
-```bash
-cat ~/forgecad-backup-<label>.dump | ssh hetzner "docker exec -i \$(docker ps --filter name=postgres -q) pg_restore -U JR3CsoTkOQypFSdD -d forgecad --clean"
-```
-
-This drops and recreates all tables from the dump. The app container will need a restart afterward to reconnect.
-
-### Reset local database
-
-```bash
-npm run dev:db:stop
-docker volume rm forgecad_postgresql-dev-data   # or the worktree-specific volume name
-npm run dev:db
-# Server will re-run all migrations on next start
-```
-
----
-
-## Production (Hetzner / Kamal)
-
-### Access
-
-```bash
-ssh hetzner    # see ~/.ssh/config for host alias
-```
-
-### Observability-first incident workflow
-
-Use the observability stack as the primary path for production incidents. Fall back to raw `docker logs` only if the stack itself is degraded.
-
-1. Make sure your laptop is connected to Tailscale.
-
-2. Check external symptoms first in Uptime Kuma:
-
-- `http://100.118.68.93:3001`
-- Confirm whether `https://forgecad.io/` and `https://forgecad.io/api/health` are down, slow, or flapping.
-
-3. Open Grafana for quick triage:
-
-- `http://100.118.68.93:3000`
-- Use the `ForgeCAD Host Overview` dashboard first to check host pressure, container pressure, and whether observability targets are up.
-- Then switch to `ForgeCAD Web Requests` if the problem looks like throughput collapse, rising 4xx/5xxs, or slow request handling in the web app.
-- Then switch to `ForgeCAD DB Queries` if the issue looks like latency, auth/session churn, admin slowness, or anything else that may be query-bound.
-
-4. Use Grafana Explore for logs and metrics:
-
-- Select `Loki` for logs.
-- Select `Prometheus` for metrics.
-
-Useful Loki queries:
-
-```text
-{container=~"forgecad-web-.*|forgecad-backend-web-.*"}
-{container=~"forgecad-web-.*|forgecad-backend-web-.*"} |= "error"
-{container=~"forgecad-web-.*|forgecad-beta-web-.*"} |= "DB slow query"
-{container=~"forgecad-web-.*|forgecad-beta-web-.*"} | json | event="db.query"
-{container=~"forgecad-beta-.*"}
-{container="kamal-proxy"}
-```
-
-### Roll back a bad prod web deploy
-
-If the latest web deploy is healthy enough for `/api/health` but breaking real
-user flows, roll back to the previous healthy web version immediately:
-
-```bash
-ssh hetzner 'docker ps -a --format "{{.Names}}\t{{.Status}}" | grep "^forgecad-web-" | sed -E "s/^forgecad-web-([a-f0-9]+)(_replaced_[^[:space:]]+)?\t/\1\t/"'
-npm run prod:rollback -- <VERSION>
-```
-
-Example:
-
-```bash
-npm run prod:rollback -- db9c49b058aaa379d9d622456cb249f9b0ddc335
-```
-
-After rollback, verify a real user-facing path, not just `/api/health`. For
-auth incidents, a dummy login should return `401 Invalid credentials`, not
-`500`.
-
-Useful Prometheus queries:
-
-```text
-up
-node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
-100 * (1 - node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{mountpoint="/",fstype!~"tmpfs|overlay"})
-sum by (container) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))
-sum(rate(forge_http_request_total[$__rate_interval]))
-sum(rate(forge_http_request_total{surface="api",status_class="5xx"}[$__rate_interval]))
-histogram_quantile(0.95, sum by (le) (rate(forge_http_request_duration_seconds_bucket[$__rate_interval])))
-topk(10, sum by (forge_env, method, route) (rate(forge_http_request_total{surface="api"}[$__rate_interval])))
-```
-
-5. Check the app health endpoint after any intervention:
-
-```bash
-curl -s https://forgecad.io/api/health | python3 -m json.tool
-```
-
-6. Use raw Docker inspection only as fallback or to verify what Grafana already suggested:
-
-- `docker logs --tail 200 <forgecad-container>`
-- `docker logs --tail 200 <forgecad-backend-container>`
-- `docker ps --format "{{.Names}}\t{{.Status}}"`
-- `df -h / && free -h`
-
-Grafana admin credentials live on Hetzner in `/home/kostard/forgecad-observability/.env`.
-
-### Find containers
-
-```bash
-docker ps --format "{{.Names}}" | grep forge
-```
-
-### Logs
-
-```bash
-# App logs (recent)
-docker logs --tail 200 <forgecad-container>
-
-# Follow live
-docker logs -f <forgecad-container>
-
-# Filter for errors
-docker logs <forgecad-container> 2>&1 | grep -i error | tail -20
-```
-
-If the observability stack is healthy, prefer Grafana Explore over direct `docker logs` because it gives you cross-container history and labels in one place.
-
-### Database access in production
-
-```bash
-# Get credentials from the app container's environment
-docker exec <forgecad-container> env | grep POSTGRES
-
-# Connect
-docker exec -it <postgres-container> psql -U <USER> -d forgecad
-```
-
-### Health check
-
-```bash
-curl -s https://forgecad.io/api/health | python3 -m json.tool
-```
-
-### Verify a deployment landed
-
-GitHub Actions deploys production on push to `mainline` by running Kamal against Hetzner. This usually takes 2–5 minutes, depending on image build and pull time.
-
-To confirm:
-
-```bash
-# 1. Check container was recently recreated (look at the "Up X minutes" status and CreatedAt)
-ssh hetzner 'docker ps --format "{{.Names}}\t{{.CreatedAt}}\t{{.Status}}" | grep -E "^forgecad-" | grep -v postgres'
-
-# 2. Check for healthy status
-ssh hetzner 'docker ps --format "{{.Names}}\t{{.Status}}" | grep -E "^forgecad-" | grep -v postgres | grep healthy'
-
-# 3. Check startup logs for errors
-ssh hetzner 'docker logs --tail 30 $(docker ps --format "{{.Names}}" | grep -E "^forgecad-" | grep -v postgres) 2>&1'
-
-# 4. Hit the health endpoint
-curl -s https://forgecad.io/api/health | python3 -m json.tool
-
-# 5. Verify specific code landed (check compiled output, not source)
-ssh hetzner 'docker exec $(docker ps --format "{{.Names}}" | grep -E "^forgecad-" | grep -v postgres) grep -c "YOUR_KEYWORD" /app/dist-server/routes/files.js'
-```
-
-Container names change on each deploy — always discover dynamically with `docker ps | grep forgecad`. The app dir is `/app/` with compiled output in `dist/`, `dist-server/`, `dist-cli/`.
-
-### Verify share links work after deploy
-
-Share links are a critical user-facing feature. After any deploy that touches shares, files, or database migrations:
-
-```bash
-# Hit the share API directly — should return JSON with shareId, code, author
-curl -s https://forgecad.io/api/shares/gUGt8bdvKg | python3 -c "
-import sys, json
-d = json.load(sys.stdin)
-if 'shareId' in d and 'code' in d:
-    print(f'OK: share {d[\"shareId\"]} resolves ({len(d[\"code\"])} bytes)')
-else:
-    print(f'BROKEN: {d}')
-    sys.exit(1)
-"
-```
-
-If it prints `BROKEN`, restore the database backup and roll back the deploy.
-
-### Restart
-
-Kamal normally handles deploys and restarts on push to `mainline`. For a manual restart:
-
-```bash
-docker restart $(docker ps --format "{{.Names}}" | grep -E "^forgecad-" | grep -v postgres)
-```
-
-### System resources
-
-```bash
-df -h       # disk
-free -h     # memory
-docker stats --no-stream   # per-container CPU/memory
-```
-
----
-
-## API Endpoints Reference
-
-### Public (no auth)
-
-| Method | Route | Purpose |
-|--------|-------|---------|
-| GET | `/api/health` | Server health + uptime |
-| GET | `/api/shares/:shareId` | Fetch a published model |
-
-### Auth
-
-| Method | Route | Purpose |
-|--------|-------|---------|
-| POST | `/api/auth/register` | Create account |
-| POST | `/api/auth/login` | Login (returns cookies) |
-| POST | `/api/auth/refresh` | Refresh access token |
-| POST | `/api/auth/logout` | Clear session |
-| GET | `/api/auth/session` | Current user info |
-| GET | `/api/auth/providers` | OAuth provider URLs |
-| POST | `/api/auth/callback/:provider` | OAuth callback |
-| POST | `/api/auth/forgot-password` | Request password reset |
-| POST | `/api/auth/reset-password` | Apply password reset |
-| GET | `/api/auth/verify-email` | Confirm email |
-| POST | `/api/auth/resend-verification` | Resend verification email |
-
-### Projects (all require auth)
-
-| Method | Route | Role | Purpose |
-|--------|-------|------|---------|
-| GET | `/api/projects` | — | List user's projects |
-| POST | `/api/projects` | — | Create project |
-| GET | `/api/projects/:id` | viewer | Project details |
-| PATCH | `/api/projects/:id` | owner | Update project |
-| DELETE | `/api/projects/:id` | owner | Delete project |
-| GET | `/api/projects/:id/members` | viewer | List members |
-| POST | `/api/projects/:id/members` | owner | Add member |
-| PATCH | `/api/projects/:id/members/:userId` | owner | Update member role |
-| DELETE | `/api/projects/:id/members/:userId` | owner | Remove member |
-
-### Files (all require auth + project role)
-
-| Method | Route | Role | Purpose |
-|--------|-------|------|---------|
-| GET | `/api/projects/:id/watch` | viewer | SSE file watcher |
-| GET | `/api/projects/:id/file-ids` | viewer | File ID map (path → fileId) |
-| POST | `/api/projects/:id/save` | editor | Save file (returns fileId) |
-| POST | `/api/projects/:id/delete` | editor | Delete file |
-| POST | `/api/projects/:id/move` | editor | Move/rename file (preserves share links) |
-| POST | `/api/projects/:id/mkdir` | editor | Create directory |
-| GET | `/api/projects/:id/read-binary` | viewer | Download mesh file |
-
-### Shares (published models)
-
-| Method | Route | Auth | Purpose |
-|--------|-------|------|---------|
-| GET | `/api/shares/:shareId` | none | Fetch published model (public) |
-| GET | `/api/shares` | required | List user's published models |
-| POST | `/api/shares` | required | Publish or update a model |
-| DELETE | `/api/shares/:shareId` | required (owner) | Unpublish |
-
----
-
-## Troubleshooting
-
-### Server won't start
-
-| Symptom | Cause | Fix |
-|---------|-------|-----|
-| `pino-pretty` transport error | Missing dev dependency in worktree | Use `NODE_ENV=production` or install pino-pretty |
-| Port 5174 already in use | Another server instance running | `lsof -i :5174` then kill the PID |
-| `FORGE_DB_URL` missing | Env var not set | Use `npm run dev:server` which sets it automatically |
-| Connection refused on :5432 | Postgres not running | `npm run dev:db` |
-
-### Auth issues
-
-| Symptom | Cause | Fix |
-|---------|-------|-----|
-| 500 on login/register | Missing `FORGE_JWT_SECRET` | Set it (any string for dev, `openssl rand -base64 32` for prod) |
-| 401 on authenticated endpoints | Expired access token (15 min) | Client should auto-refresh; check refresh token cookie |
-| CORS errors in browser | `FORGE_ALLOWED_ORIGINS` mismatch | Must match the frontend's origin exactly |
-
-### Email issues
-
-| Symptom | Cause | Fix |
-|---------|-------|-----|
-| No verification/reset email received; logs show `[email] To: user@...` | `RESEND_API_KEY` is empty — falling back to console log | Set `RESEND_API_KEY` in `.kamal/secrets` or production secrets, then redeploy. See [platform/email.md](platform/email.md) |
-
-### Database issues
-
-| Symptom | Cause | Fix |
-|---------|-------|-----|
-| Migration fails | Schema conflict from manual changes | Reset local DB (see above) |
-| `relation does not exist` | Migrations didn't run | Check `_migrations` table; restart server |
-| Quota errors on file save | User's `storage_used_bytes` exceeded 50MB | Check with `SELECT storage_used_bytes FROM users WHERE ...` |