@ngocsangairvds/vsaf 3.2.16 → 4.0.0

package/tools/vds-scripts/scheduler_orchestrator/DEPLOYMENT-RUNBOOK.md

@@ -1,655 +0,0 @@
-# VDS Scheduler — Production Deployment Runbook
-
-**Spec**: `.gpt-5.4/specs/vds-scheduler/` v1.0.0+ (Phase 0 v2.0.19 closure record at `CLOSURE-v2.0.18.md`; predecessor `ecosystem-scheduling/` consolidated 2026-05-01)
-**Date**: 2026-04-26 · revised v2.0.9 with corrected role hierarchy after live audit found `vds_sched` role suggestion was incompatible with the migration's `vds_scheduler_app` design + missing `GRANT USAGE ON SCHEMA sched` · revised v2.0.18 added §Auth Posture (TSK-N.63) · revised v2.0.19 restructured §Auth Posture into 5-operator subsections (phase219 alignment)
-**Scope**: First-time production cutover for the 4-phase ecosystem-scheduling program (merged to `vds-scripts` master at `eb365d57` on 2026-04-24)
-
----
-
-## Preface — What This Runbook Covers
-
-This runbook is the ops companion to `RETIREMENT-EXECUTED.md` and `CLOSURE-PHASE-4.md §What's NOT done`. It sequences the production actions that are **outside code scope**: applying the Alembic migrations on the production Postgres, starting the `scheduler_orchestrator` service, flipping the kill switch, retiring the Celery Beat container, and verifying the cutover.
-
-**Who runs this**: production ops engineer with:
-- SSH/access to the production host running `vds-scripts`
-- Write access to the production Postgres instance that will host the `sched.*` schema
-- Write access to `/etc/.vds/.env` (or equivalent production secret store) for the flag flip
-- Docker/compose access on the production host to stop the Celery Beat container
-
-**Local validation**: every step below has been exercised locally on `vds-ai-memory-postgres-1` against master tip `eb365d57` (plus one paramstyle bugfix in migration `d4e5f6a7b8c9` landed on 2026-04-25). See `§Local Exercise Log` at the bottom.
-
----
-
-## Preflight Checklist (must all be ✅ before cutover)
-
-- [ ] Production Postgres instance reachable with a DBA-owned role that has `CREATEDB` + `CREATE SCHEMA` privileges
-- [ ] A dedicated application DB name selected for the scheduler (recommended: `vds_scheduler`)
-- [ ] A dedicated application role for the scheduler's runtime (recommended: `vds_sched`) — separate from DBA role per ADR-19 (append-only audit via role separation)
-- [ ] An on-call window booked (recommend 2 hours; median time on local was ~20 minutes)
-- [ ] The deploy-time `DBOS_APP_VERSION` will be set to the git SHA of `eb365d57` or the next production tag (whichever matches the deployed image)
-- [ ] `docker compose --profile dev,mem0 ps` on the production host shows `beat` still running (confirms we're cutting over from the old state)
-- [ ] Tag has been cut and image built (`vds-scheduler-orchestrator:$TAG`) if using containerised service; otherwise the uvicorn process recipe in Stage 3 is the reference
-
----
-
-## Stage 0 — First-time setup: `~/.vds/.env` contract [v2.0.17 N.52 + N.54]
-
-The scheduler reads ALL non-context-dependent config from `~/.vds/.env` (the operator's user-scoped secret store). Two derived requirements:
-
-1. **`docker/.env` symlink** — Docker Compose's `${VAR}` YAML substitution reads from compose's own env (the invoking shell or compose's `.env`), NOT from the per-service `env_file:` block. To make `~/.vds/.env` feed BOTH compose-time substitution AND in-container env_file, link compose's local env to the user's:
-   ```bash
-   ln -sf ~/.vds/.env /opt/vds/vds-scripts/docker/.env
-   # or (idempotent, repo-wide):
-   make -C /opt/vds/vds-scripts env-link
-   ```
-   The symlink is gitignored (developer/operator-machine artifact).
-
-2. **Three DSN entries; one source of truth** — Post v2.0.17 N.52 consolidation, `~/.vds/.env` declares three Postgres-DSN env vars that all currently point at the SAME database (`vds_platform` / `sched` schema):
-   - `DATABASE_URL` — canonical runtime DSN (winning AliasChoices entry).
-   - `MIGRATION_DATABASE_URL` — superuser DDL DSN consumed by entrypoint's alembic step.
-   - `VDS_SCHEDULER_DATABASE_URL` — host-mode/dev fallback (Pydantic AliasChoices fallback path; preserved for backwards compatibility with v2.0.16 host CLI scripts).
-
-   All three use `localhost:5432` (resolves correctly from host-side processes via the published Postgres port). Inside the container, compose's `environment:` block overrides only the host portion to Docker-DNS `postgres:5432`; everything else flows through `env_file:` unchanged.
-
-   **Operator rule (v2.0.17 N.54)**: Do NOT remove `VDS_SCHEDULER_DATABASE_URL` from `~/.vds/.env` even though it is functionally redundant with `DATABASE_URL`. The Pydantic AliasChoices fallback path in `scheduler_orchestrator/src/vds_scheduler_orchestrator/config.py` references it, and removing the env var would surface as `vds_scheduler_database_url=None` for any host-mode dev path that doesn't also export `DATABASE_URL`. Keep it; rely on N.43's AliasChoices precedence ordering (`DATABASE_URL` wins).
-
----
-
-## Stage 1 — Create Dedicated Scheduler Database + Runtime Login Role
-
-The role design (corrected v2.0.9 after live audit): migration `c3d4e5f6a7b8` automatically creates a NOLOGIN privilege-holder role `vds_scheduler_app` (the role ADR-19 grants INSERT-only on `audit_log` to). What you create here is the **runtime LOGIN role** that the scheduler process actually authenticates as — it must be a MEMBER of `vds_scheduler_app` so it inherits the audit-log INSERT privilege via role inheritance.
-
-```bash
-# On a DBA-credentialed shell against the production Postgres
-psql -U <dba_role> -d postgres <<'SQL'
-CREATE DATABASE vds_scheduler;
-
--- Runtime LOGIN role (the scheduler process connects as this role)
-CREATE ROLE vds_sched_runtime LOGIN PASSWORD '<generated-16-byte-hex>' INHERIT;
-GRANT CONNECT ON DATABASE vds_scheduler TO vds_sched_runtime;
-
--- NOTE: vds_scheduler_app (NOLOGIN, INSERT-only on audit_log) is created
--- automatically by migration c3d4e5f6a7b8. We just need to make our runtime
--- role a member of it so the audit_log INSERT privilege is inherited.
--- Run AFTER Stage 2 migrations land:
---   GRANT vds_scheduler_app TO vds_sched_runtime;
-SQL
-```
-
-**Why this role hierarchy** (ADR-19): `sched.audit_log` is append-only by structural enforcement. Migration `c3d4e5f6a7b8` creates the privilege-holder role `vds_scheduler_app` with `GRANT INSERT, REVOKE UPDATE+DELETE` on `audit_log` (and GRANT USAGE on schema sched, added v2.0.9 — the missing USAGE was a real bug found during live audit). The runtime LOGIN role inherits these privileges through `GRANT vds_scheduler_app TO vds_sched_runtime`. An attacker who compromises the runtime role's password still cannot UPDATE or DELETE audit-log rows because the REVOKE wins at object level over any subsequent permissive GRANT.
-
-**Rollback**: `DROP DATABASE vds_scheduler; DROP ROLE vds_sched_runtime;` — zero-risk if executed before Stage 2. (Do NOT drop `vds_scheduler_app` — it's created by the migration's downgrade and may be shared by re-applied environments per the migration's design comment.)
-
----
-
-## Stage 2 — Run Alembic Migrations
-
-```bash
-# On the production host, with the repo cloned at the tagged commit
-cd /opt/vds/vds-scripts   # or wherever the repo is checked out
-
-export VDS_SCHEDULER_DATABASE_URL="postgresql://<dba_role>:<dba_password>@<pg_host>:5432/vds_scheduler"
-# NOTE: use DBA role for the migration itself (needs CREATE SCHEMA); runtime role uses vds_sched
-
-uv run --project scheduler_orchestrator \
-  alembic -c scheduler_orchestrator/alembic.ini upgrade head
-```
-
-**Expected output** (7 migrations, bottom-up):
-```
-Running upgrade  -> 0001, Create sched schema
-Running upgrade 0001 -> 04f7169497f6, create_domain_tables
-Running upgrade 04f7169497f6 -> a1b2c3d4e5f6, add_overlap_policy_to_schedules
-Running upgrade a1b2c3d4e5f6 -> b2c3d4e5f6a7, add_dispatcher_state_machine
-Running upgrade b2c3d4e5f6a7 -> c3d4e5f6a7b8, add_audit_log
-Running upgrade c3d4e5f6a7b8 -> d4e5f6a7b8c9, celery_retirement
-Running upgrade d4e5f6a7b8c9 -> e5f6a7b8c9d0, add_udf_enqueue
-```
-
-**Migration `d4e5f6a7b8c9` is the Celery retirement record** — it inserts one row into `sched.audit_log` with `trace_id='tsk-12.7-celery-retirement-d4e5f6a7b8c9'` and strips `migration_note` context keys from 6 DBOS schedule rows. Verify:
-
-```sql
-SELECT actor_capability, operation, object_type, object_id
-  FROM sched.audit_log
- WHERE trace_id = 'tsk-12.7-celery-retirement-d4e5f6a7b8c9';
--- Expect exactly 1 row:
--- alembic:d4e5f6a7b8c9 | celery_beat.retired | celery_beat_schedule | memory_orchestrator
-```
-
-**§grants** — most grants are baked into migration `c3d4e5f6a7b8` (corrected v2.0.9). The migration handles `vds_scheduler_app` role creation, schema USAGE, table-level INSERT grant, and audit-log REVOKE. You only need to (a) add the runtime role to the privilege-holder group and (b) grant operational-table DML + DBOS-schema access to the runtime role.
-
-```sql
--- Inherit audit-log INSERT-only privilege (post Stage 2 migrations)
-GRANT vds_scheduler_app TO vds_sched_runtime;
-
--- Runtime role: full DML on operational tables (these are NOT covered by the migration)
-GRANT USAGE ON SCHEMA sched TO vds_sched_runtime;
-GRANT SELECT, INSERT, UPDATE, DELETE ON
-  sched.schedules, sched.subscriptions, sched.workflow_registry,
-  sched.events_outbox, sched.dispatch_ack_log,
-  sched.chain_definitions, sched.chain_fire_log
-TO vds_sched_runtime;
-
--- DBOS system database — scheduler process needs CREATE on dbos schema for
--- its 19 system migrations on first boot
-GRANT USAGE, CREATE ON SCHEMA dbos TO vds_sched_runtime;
-GRANT ALL ON ALL TABLES IN SCHEMA dbos TO vds_sched_runtime;
-GRANT ALL ON ALL SEQUENCES IN SCHEMA dbos TO vds_sched_runtime;
-
--- DO NOT manually re-grant on sched.audit_log — that path is owned by the
--- migration. Re-granting here would not break anything but is redundant and
--- confuses ownership. The runtime role gets INSERT-only on audit_log by
--- being a member of vds_scheduler_app via the GRANT above.
-```
-
-**Verification** (run after grants):
-```sql
-SET ROLE vds_scheduler_app;
-INSERT INTO sched.audit_log (actor_capability, operation, object_type, object_id, after_state)
-VALUES ('test:cutover', 'verify.append_only', 'capability', 'preflight', '{"test":true}'::jsonb);
--- Should succeed.
-
-UPDATE sched.audit_log SET operation='hacked' WHERE 1=0;
--- Should raise: ERROR: permission denied for table audit_log (SQLSTATE 42501)
-
-DELETE FROM sched.audit_log WHERE 1=0;
--- Should raise: ERROR: permission denied for table audit_log (SQLSTATE 42501)
-
-RESET ROLE;
-```
-This is the canonical TSK-N.7 AC2/AC3/AC4 acceptance check; the integration suite at `tests/integration/test_audit_log_grants.py` automates it (now also runs in CI per chaos-smoke.yml v2.0.9).
-
-**Rollback**: `uv run --project scheduler_orchestrator alembic -c scheduler_orchestrator/alembic.ini downgrade base` — tears down all 7 migrations. Low-risk because no production traffic has flowed yet. If you've already flipped the flag in Stage 4, stop the scheduler service first.
-
----
-
-## Stage 3 — Start the Scheduler Service
-
-The scheduler is a FastAPI app (`vds_scheduler_orchestrator.app:app`) run under uvicorn. It does NOT auto-start and has NO compose service — per the Program Governance Artifact Map it is a module library with deploy-time supervision.
-
-### Option A — systemd-managed uvicorn (recommended for single-host deploys)
-
-```ini
-# /etc/systemd/system/vds-scheduler.service
-[Unit]
-Description=VDS Scheduler Orchestrator (DBOS Transact)
-After=network.target postgresql.service
-
-[Service]
-Type=simple
-User=vds
-Group=vds
-WorkingDirectory=/opt/vds/vds-scripts
-EnvironmentFile=/etc/.vds/.env
-Environment=VDS_SCHEDULER_ENABLED=true
-Environment=DBOS_APP_VERSION=eb365d57   # update per release tag
-ExecStart=/opt/vds/vds-scripts/.venv/bin/uvicorn \
-    vds_scheduler_orchestrator.app:app \
-    --host 127.0.0.1 --port 8005 --log-level info
-Restart=on-failure
-RestartSec=5
-# Hard cap on memory + workflow processes; tune per Phase-2 capacity plan
-MemoryMax=2G
-
-[Install]
-WantedBy=multi-user.target
-```
-
-```bash
-sudo systemctl daemon-reload
-sudo systemctl enable --now vds-scheduler
-sudo systemctl status vds-scheduler
-```
-
-### Option B — compose service (recommended for k8s/containerised deploys)
-
-See `scheduler_orchestrator/docs/autoscaling-k8s-hpa.yaml` for the HPA pattern wired to queue-depth metrics (Phase 4 autoscaler). Containerised service accepts the same env vars as Option A.
-
-### Verify startup
-
-```bash
-curl -sf http://127.0.0.1:8005/health
-# Expect: {"status":"ok","scheduler_enabled":true,"app_version":"eb365d57"}
-```
-
-```bash
-# Tail the systemd journal for DBOS launch banner
-journalctl -u vds-scheduler -n 80 --no-pager | grep -E "DBOS launched|Listening to.*queues"
-# Expect 2 lines:
-#   DBOS launched!
-#   Listening to 4 queues:  ops-default / agent-default / agent-partitioned / chain-fanout
-```
-
-**Rollback**: `sudo systemctl stop vds-scheduler && sudo systemctl disable vds-scheduler`. The service leaves no persistent state behind — durable workflow state in `dbos.*` remains but is inert until the service is restarted.
-
----
-
-## Stage 4 — Flip `VDS_SCHEDULER_ENABLED=true` Where It Counts
-
-The flag is read at process-start time. Two patterns are safe:
-
-**Pattern 4a — inline env (recommended)**: set `VDS_SCHEDULER_ENABLED=true` in the systemd unit file (as shown in Stage 3) or in the container's `env:` block. Leave `/etc/.vds/.env` at `VDS_SCHEDULER_ENABLED=false` so ambient `vds-cli` calls from other users on the same host don't accidentally initialise DBOS.
-
-**Pattern 4b — file-level flip**: flip `/etc/.vds/.env` to `VDS_SCHEDULER_ENABLED=true`. Do this ONLY if the scheduler is the only process on the host that imports `scheduler_orchestrator` — otherwise any future `vds-cli` invocation will try to init DBOS on a stale DSN.
-
-The vds-scripts CLAUDE.md guardrail: *"Set `VDS_SCHEDULER_ENABLED=true` only when the scheduler service is intentionally started."* Pattern 4a honours this guardrail even on shared hosts.
-
-### Step 4.x — Clean up host-mode uvicorn before/after container deploy [v2.0.17 N.45]
-
-When an operator runs the scheduler locally for dev (uvicorn-on-host, Mode B) and then deploys the container (Mode A), both bind port 8005 (uvicorn IPv4 + container IPv6/0.0.0.0). `localhost` resolution can flip-flop between them, leading to confusing dual-state where some CLI calls hit the container and others hit the stale uvicorn.
-
-Cleanup recipe before / after a Mode A deploy:
-```bash
-# Find PIDs listening on 8005 (excluding the container's bridged proxy)
-PIDS=$(lsof -nP -iTCP:8005 -sTCP:LISTEN -t)
-CONTAINER_PID=$(docker inspect vds-scheduler-orchestrator --format '{{.State.Pid}}' 2>/dev/null || true)
-for p in $PIDS; do
-  if [ "$p" != "$CONTAINER_PID" ] && [ "$(ps -p $p -o comm= 2>/dev/null)" = "python3.1" ]; then
-    echo "Killing host-mode uvicorn PID $p"
-    kill -9 "$p"
-  fi
-done
-```
-
-Run this AFTER `docker compose up -d scheduler-orchestrator` to ensure only the container owns 8005.
-
----
-
-## Stage 5 — Stop and Remove Celery Beat Container
-
-Per `RETIREMENT-EXECUTED.md`, the `beat:` service was removed from `memory_orchestrator/compose.yaml` on 2026-04-24 but any running container persists until manually stopped.
-
-```bash
-docker compose --profile dev,mem0 stop beat  2>/dev/null || true
-docker rm -f vds-ai-memory-beat-1            2>/dev/null || true
-
-# Verify absence
-docker ps --format '{{.Names}}' | grep -i beat && echo "STILL RUNNING" || echo "beat container absent ✅"
-```
-
-**Rollback**: if you need Celery Beat back before the scheduler-orchestrator is ticking its 6 migrated workflows, `git revert eb365d57` restores the `beat:` compose service and the 6 `beat_schedule` entries. Then `docker compose --profile dev,mem0 up -d beat`.
-
----
-
-## Stage 6 — Post-Cutover Verification (48h soak gate per TSK-8.4)
-
-### 6.1 — DBOS queue health
-
-```bash
-# Queue depth must not grow unbounded
-psql -U vds_sched -d vds_scheduler -tAc "
-  SELECT queue_name, count(*) AS enqueued
-  FROM dbos.workflow_status
-  WHERE status='ENQUEUED'
-  GROUP BY 1;
-"
-```
-
-### 6.2 — Periodic workflows firing
-
-The 6 formerly-Celery-Beat DBOS aliases must be producing `PENDING → SUCCESS` transitions:
-
-```bash
-# Last 10 minutes of completed workflows
-psql -U vds_sched -d vds_scheduler -c "
-  SELECT name, count(*) AS ok
-  FROM dbos.workflow_status
-  WHERE status = 'SUCCESS'
-    AND updated_at > NOW() - INTERVAL '10 minutes'
-  GROUP BY 1
-  ORDER BY 1;
-"
-```
-
-Expected aliases (at their respective cadences):
-| Alias | Cadence |
-|---|---|
-| `memory.evolution.dispatch` | `*/1 * * * *` |
-| `memory.evolution.sweep` | `0 * * * *` |
-| `memory.evolution.replay-dlq` | `*/15 * * * *` |
-| `mem0.resolve-pending-ids` | `*/5 * * * *` |
-| `mem0.repush-stalled` | `0 * * * *` |
-| `mem0.repush-local-only` | `0 */6 * * *` |
-| `evolution.auto-promote` | `*/1 * * * *` |
-
-### 6.3 — Prometheus metrics
-
-```bash
-curl -sf http://127.0.0.1:8005/metrics | grep -E 'vds_scheduler_(tick|dispatch|rate_limit)_total' | head -20
-```
-
-Grafana dashboards (Phase 2 deliverable): `scheduler_orchestrator/dashboards/*.json` — import into Grafana and watch the 7 tick metrics for 48h.
-
-### 6.3a — HTTP API surface probe (v2.0.13 TSK-N.21)
-
-The v2.0.13 cycle added a read-only `/api/v1/*` surface. Verify every route responds 200 on a fresh cutover — a 404 indicates the api/v1 router wasn't wired or the deployment is pre-v2.0.13:
-
-```bash
-for r in /api/v1/health/full /api/v1/schedules /api/v1/chains /api/v1/workflows /api/v1/workflow-registry /api/v1/events; do
-  printf "%-36s " "$r"
-  curl -s -o /dev/null -w "%{http_code}\n" "http://127.0.0.1:8005$r"
-done
-# Expect: all 200
-```
-
-Detailed health aggregate (`/api/v1/health/full`) reports per-layer status + live row counts:
-
-```bash
-curl -s http://127.0.0.1:8005/api/v1/health/full | python3 -m json.tool
-# Expect: status=ok; dbos.status=ok; db.status=ok with counts for
-# sched.{schedules, chain_definitions, workflow_registry, events_outbox}
-# + dbos.{workflow_schedules, workflow_status.ENQUEUED}
-```
-
-Alias resolution end-to-end probe (exercises the v2.0.13 TSK-N.18 registry path):
-
-```bash
-curl -s http://127.0.0.1:8005/api/v1/workflow-registry/scheduler.autoscale
-# Expect: {"alias": "scheduler.autoscale", "workflow_fqn": "vds_scheduler_orchestrator.workflows.autoscale.autoscale_check_workflow"}
-```
-
-Write endpoints (POST/PATCH/DELETE schedules; POST events) are deferred to v2.0.14 per `tasks.md` §TSK-N.24..N.27 — operators doing create/pause/resume/delete in this cycle must still use `vds-cli scheduler schedules ...` directly against the service (which invokes DBOS in-process and therefore requires DBOS to be launched in the CLI environment).
-
-### 6.4 — Gate for retirement migration commit (d4e5f6a7b8c9)
-
-The row inserted by the migration is the durable audit record. Verify once post-cutover:
-
-```sql
-SELECT occurred_at, operation, after_state
-FROM sched.audit_log
-WHERE trace_id = 'tsk-12.7-celery-retirement-d4e5f6a7b8c9';
-```
-
-If the row is missing, re-run `alembic upgrade head` — the migration is idempotent with respect to the trace_id (downgrade deletes by trace_id, upgrade re-inserts).
-
----
-
-## Appendix A — Rollback Matrix
-
-| Stage | Reached? | Rollback command(s) |
-|---|---|---|
-| 1 only | DB created | `DROP DATABASE vds_scheduler; DROP ROLE vds_sched;` |
-| 2 complete | Migrations applied | `alembic downgrade base` (no traffic) OR leave schema in place (harmless) |
-| 3 complete | Service up | `systemctl stop vds-scheduler` |
-| 4 complete | Flag flipped | unset flag, restart service if applicable |
-| 5 complete | Beat stopped | `git revert eb365d57 && docker compose up -d beat` |
-| 6 failing | Post-cutover regression | Full rollback in reverse order; page the on-call engineer (see `docs/runbooks/rollback-deploy.md`) |
-
----
-
-## Appendix B — Local Exercise Log
-
-All 6 stages exercised locally on 2026-04-25 against `vds-ai-memory-postgres-1` with the merged `eb365d57` code:
-
-- Stage 1 ✅ — `vds_scheduler` DB created
-- Stage 2 ✅ — 7 migrations applied cleanly AFTER one paramstyle bugfix in `20260424_d4e5f6a7b8c9_celery_retirement.py` (`:after_state::jsonb` → `CAST(:after_state AS jsonb)`). **Production note**: pick up the fix commit or the migration will fail on first run.
-- Stage 3 ✅ — scheduler started on port 8005; `/health` returned `{"status":"ok","scheduler_enabled":true}`; DBOS launched 19 system migrations + 4 queues
-- Stage 4 ✅ — inline-env pattern (`.env` stayed false, service got `VDS_SCHEDULER_ENABLED=true` via environment)
-- Stage 5 ✅ — `vds-ai-memory-beat-1` orphan stopped + removed on 2026-04-24
-- Stage 6 ✅ — end-to-end smoke (`POST /test/enqueue` → `SUCCESS` in 2s), 4 queues verified, workflow_registry has 5 aliases populated
-
-Local deviations from production expectations:
-- `audit.*` workflow bootstraps emit "module not found" warnings because `vds_audit_orchestrator` isn't in the local workspace sync set; production deploys that pip-install both packages will have them register cleanly
-- `evolution.auto-promote` same — requires `vds_evolution` package in the environment
-- Only `scheduler.autoscale` schedule auto-fires locally (its workflow is inside `scheduler_orchestrator` itself)
-
-## Appendix C — On-Call References
-
-- **Chaos runbook**: `scheduler_orchestrator/docs/chaos-runbook.md`
-- **DR postgres restore**: `scheduler_orchestrator/docs/runbooks/dr-postgres-restore.md`
-- **Dispatcher stall**: `scheduler_orchestrator/docs/runbooks/incident-dispatcher-stalled.md`
-- **Postgres unavailable**: `scheduler_orchestrator/docs/runbooks/incident-postgres-unavailable.md`
-- **Workflow backlog**: `scheduler_orchestrator/docs/runbooks/incident-workflow-backlog.md`
-- **Secret leak in event payload**: `scheduler_orchestrator/docs/runbooks/incident-secret-leaked-in-event.md`
-- **Rollback deploy**: `scheduler_orchestrator/docs/runbooks/rollback-deploy.md`
-- **Postmortem template**: `scheduler_orchestrator/docs/runbooks/postmortem-template.md`
-- **Error budget policy**: `scheduler_orchestrator/docs/error-budget-policy.md`
-- **Quarterly game-day cadence**: `scheduler_orchestrator/docs/quarterly-game-day.md`
-- **PITR drill**: `scheduler_orchestrator/scripts/dr/pitr-drill.sh`
-
----
-
-## §Auth Posture (TSK-N.63)
-
-> **Related runbook**: `docs/runbooks/orphan-reconciliation-alert.md` documents triage for the `sched_orphan_reconciled_total` Prometheus alert (v2.0.18 TSK-N.65).
-
-The scheduler HTTP API (`/api/v1/*`) supports opt-in API-key authentication. By
-default auth is **disabled** for backward compatibility with all existing clients.
-
-| Env Variable | Type | Default | Description |
-|---|---|---|---|
-| `VDS_SCHEDULER_AUTH_REQUIRED` | `bool` | `false` | Enable API-key validation on `/api/v1/*` routes |
-| `VDS_SCHEDULER_API_KEY` | string (secret) | _none_ | Primary API key; send via `X-VDS-Scheduler-Key` header |
-| `VDS_SCHEDULER_API_KEY_OLD` | string (secret) | _none_ | Previous key kept valid during dual-key rotation window |
-| `VDS_SCHEDULER_AUTH_HEADER_NAME` | string | `X-VDS-Scheduler-Key` | HTTP header name carrying the API key |
-
-> API keys are stored as `SecretStr` by pydantic-settings. Set them in
-> `~/.vds/.env` or inject via environment. **Never hardcode in source or compose.**
-
-### Default Posture (`auth_required=False`)
-
-When `VDS_SCHEDULER_AUTH_REQUIRED` is unset or `false` (the default), the
-`require_api_key` FastAPI dependency is a no-op — one attribute read, zero
-request overhead. All `/api/v1/*` routes are reachable without any
-`X-VDS-Scheduler-Key` header. This preserves backward compatibility for CLI
-orchestrators and internal services that do not supply API keys.
-
-`/health` and `/metrics` are always unauthenticated regardless of this setting;
-they are registered at application level, outside the `/api/v1` prefix.
-
-Reference: `security/auth.py:require_api_key` (early return when
-`cfg.vds_scheduler_auth_required` is `False`).
-
-### Required Posture (`auth_required=True`)
-
-Set `VDS_SCHEDULER_AUTH_REQUIRED=true` in `~/.vds/.env` to enforce API-key
-validation on every `/api/v1/*` route. Every inbound request **must** carry the
-`X-VDS-Scheduler-Key` header with a matching key value. A missing or invalid key
-produces `401 Unauthorized` with `WWW-Authenticate: Bearer`.
-
-Key comparison uses `hmac.compare_digest()` for constant-time validation,
-preventing timing side-channel key enumeration. Both the primary key
-(`VDS_SCHEDULER_API_KEY`) and the rotation-window key (`VDS_SCHEDULER_API_KEY_OLD`)
-are checked; if either matches, the request proceeds.
-
-Reference: `security/auth.py:require_api_key` (raises `HTTPException(401)` when
-the header is absent or matches neither configured key).
-
-### Dual-Key Rotation Procedure
-
-Zero-downtime key rotation via a two-key overlap window:
-
-1. **Stage old key**: copy the current value of `VDS_SCHEDULER_API_KEY` into
-   `VDS_SCHEDULER_API_KEY_OLD`. Redeploy. Both keys are now valid — existing
-   clients continue uninterrupted.
-2. **Mint new key**: generate a new high-entropy secret and set it as
-   `VDS_SCHEDULER_API_KEY`. Redeploy. Both primary and old key are accepted via
-   the dual-check in `require_api_key`.
-3. **Roll clients**: update every client that calls the scheduler API to present
-   the new key. Confirm each client receives `200` via
-   `vds-cli scheduler health` from its host.
-4. **Retire old key**: remove `VDS_SCHEDULER_API_KEY_OLD` from `~/.vds/.env` and
-   redeploy. Only the new primary key is now valid; the old key is invalidated.
-
-### Brute-Force Protection
-
-`AuthBruteForceMiddleware` tracks per-IP `401` failures in a sliding 60-second
-window. After **10 consecutive failures** from the same source IP, all further
-requests from that IP receive `HTTP 429 Too Many Requests` with
-`Retry-After: 60` until the window resets.
-
-The failure tracker is **in-process** (single-replica scope). For multi-replica
-deployments, shared state (Redis or equivalent) is required — deferred and
-tracked for v2.1.0+.
-
-Reference: `security/middleware.py:AuthBruteForceMiddleware`
-(`RATE_LIMIT_ATTEMPTS=10`, `RATE_LIMIT_WINDOW_SEC=60`).
-
-### Pre-Deploy Verification
-
-Before enabling `VDS_SCHEDULER_AUTH_REQUIRED=true`, confirm the scheduler API
-returns the expected status codes:
-
-```bash
-curl -s -o /dev/null -w "%{http_code}\n" \
-  -H "X-VDS-Scheduler-Key: $VDS_SCHEDULER_API_KEY" \
-  http://scheduler:8001/api/v1/schedules
-# Expect: 200. Without header: 401 (if VDS_SCHEDULER_AUTH_REQUIRED=true) or 200 (if False).
-```
-
-Run the probe twice — once with the header (expect `200`) and once without
-(expect `401` after the flag is set) — before promoting the configuration to
-production.
-
----
-
-## §v2.0.9 Deep-Audit Cross-Reference (2026-04-25)
-
-This runbook was authored at spec_version 2.0.8. The v2.0.9 deep-audit pass added two production-blocker fixes that **MUST** be present in the deployed code before Stage 4 (flag flip / scheduler enablement):
-
-1. **`vds_scheduler_orchestrator/security/audit_log.py:120`** — paramstyle fix (`CAST(:before_state AS jsonb), CAST(:after_state AS jsonb)`). Without this, every runtime audit-log write silently fails (per the function's "never raise" contract) — the production audit trail would be empty without anyone noticing.
-2. **Migration `c3d4e5f6a7b8`** — `GRANT USAGE ON SCHEMA sched TO vds_scheduler_app` (line 112). Without this, the `vds_scheduler_app` role gets `permission denied for schema sched` on every audit-INSERT despite holding table-level INSERT.
-
-**Pre-deploy verification command** (add to Stage 1 — Database Preparation):
-
-```bash
-# Confirm the v2.0.9 fixes are present in the deployed code before applying migrations
-grep -q "CAST(:before_state AS jsonb)" scheduler_orchestrator/src/vds_scheduler_orchestrator/security/audit_log.py \
-  && grep -q "GRANT USAGE ON SCHEMA sched TO vds_scheduler_app" scheduler_orchestrator/alembic/versions/20260424_c3d4e5f6a7b8_add_audit_log.py \
-  && echo "v2.0.9 fixes verified" || echo "MISSING v2.0.9 fixes — DO NOT DEPLOY"
-```
-
-If the verification fails, the deployment will silently fail at the audit-log layer in production. Abort and re-pull the master tip.
-
----
-
-## Common Deploy Gotchas (P6 WP-6.5)
-
-These are real deploy-time bugs discovered during P5 live validation that static analysis didn't catch. Each includes the exact error message so operators can grep for them.
-
-### Gotcha 1: Makefile target loadout incompleteness
-
-**Symptom**: `make up-services` or `make up-scheduler` fails with:
-```
-service "memory-worker" depends on undefined service "memory-migrate": invalid compose project
-service "scheduler-orchestrator" depends on undefined service "memory-api": invalid compose project
-```
-
-**Root Cause**: WP-5.3 moved `memory-migrate` from `docker-compose.services.yml` → `docker-compose.cli.yml`. The `make up-services` and `make up-scheduler` targets load Compose files via a Makefile variable that wasn't updated. As a result:
-- `make up-memory` loads all 3 compose files (infra + services + cli) and works.
-- `make up-services` loads only infra + services (missing cli.yml), so `memory-migrate` is undefined.
-- `make up-scheduler` loads infra + services only, causing scheduler's `depends_on: memory-api` to fail (memory-api is in the memory profile, not scheduler profile).
-
-**Fix** (commit `e8de66d2`):
-- The Makefile's `ALL` macro now expands to all 3 compose files (infra, services, cli).
-- `make up-services` uses `$(ALL)` instead of a two-file subset.
-- `make up-scheduler` activates `--profile scheduler --profile memory` (both profiles needed because scheduler's `depends_on: memory-api` crosses profile boundaries).
-
-**Prevention**: P6 WP-6.2 lands `scripts/tests/test_makefile_loadouts.py` — a Python test that runs `docker compose config` against every `make up-*` / `make run-*` target's loadout. It catches future regressions at commit time.
-
----
-
-### Gotcha 2: Dockerfile workspace-dep wiring lag
-
-**Symptom**: Service Docker build fails with:
-```
-× No solution found when resolving dependencies: Because vds-platform-core was not found in the package registry...
-```
-
-Or at runtime:
-```
-ModuleNotFoundError: No module named 'vds_platform_core'
-```
-
-**Root Cause**: WP-5.2 added `vds-platform-core` as a workspace dependency to `scheduler_orchestrator/pyproject.toml` and `progress_report_orchestrator/pyproject.toml`. However, the Dockerfiles for both services use:
-```dockerfile
-RUN uv pip install --no-sources <packages>
-```
-
-The `--no-sources` flag **disables** the `[tool.uv.sources]` workspace resolution. So when the Dockerfile runs `uv pip install`, it can't find `vds-platform-core` and fails. Even if you manually copy the workspace package into `/app/`, the `uv pip install` command doesn't know to look there — it's not in the argument list.
-
-**Fix** (commit `921c95a9`):
-- Both Dockerfiles (scheduler + progress) gain a `COPY platform_core /app/platform_core` step.
-- The `uv pip install --no-sources` command line adds `/app/platform_core` to its path:
-  ```dockerfile
-  RUN uv pip install --no-sources /app/scheduler_orchestrator /app/platform_core
-  ```
-- Now `uv` can resolve the workspace dep from disk.
-
-**Prevention**: P6 WP-6.1 lands `scripts/tests/test_dockerfile_correctness.py` — a static-analysis test that parses each service's `pyproject.toml`, extracts workspace deps (looking for `path = "..."` entries), and asserts that every one has a matching Dockerfile `COPY` statement and appears in the `uv pip install` command line. Catches future sync-lag regressions.
-
----
-
-### Gotcha 3: Cross-profile depends_on (pre-existing phase217)
-
-**Symptom**: `make up-scheduler` fails with:
-```
-service "scheduler-orchestrator" depends on undefined service "memory-api": invalid compose project
-```
-
-**Root Cause**: phase217 (commits `e7960ef9`, `8b9b6fb2`) added a runtime HTTP dependency from scheduler to memory-api and wired it in Compose:
-```yaml
-scheduler-orchestrator:
-  depends_on:
-    memory-api:  # But memory-api lives in profile: memory
-      condition: service_started
-```
-
-When you run `docker compose --profile scheduler up`, Compose loads only the `scheduler` profile. The `memory-api` service (which lives in `profile: memory`) is never loaded, so Compose sees the `depends_on: memory-api` reference as undefined.
-
-**Fix** (commit `e8de66d2`):
-- `make up-scheduler` activates **both** `--profile scheduler` **and** `--profile memory`.
-- This loads both profiles and their services, so the `depends_on` reference resolves.
-
-**Architectural Note**: This `depends_on` is a **hard startup dependency** (services cannot start before memory-api is healthy). The long-term fix (out of P6 scope) is to upgrade to Compose 2.x soft-deps:
-```yaml
-depends_on:
-  memory-api:
-    condition: service_started
-    required: false  # Compose 2.x soft-dep
-```
-
-For now, activating both profiles is the production-safe workaround.
-
----
-
-## Running Scheduler Downgrade Tests (P6 WP-6.3)
-
-The 9 testcontainers regression tests in `scheduler_orchestrator/tests/test_alembic_downgrade.py` (added by P5 WP-5.1 to verify the surgical-no-op `0001.downgrade()` pattern per ADR-INF-11 §Schema-Drop-Safe Downgrade Pattern) are **deselected by default** because they (a) require a running Docker daemon for testcontainers, and (b) take ~8 seconds per run.
-
-### Explicit Invocation
-
-```bash
-# From vds-scripts repo root:
-uv run --project scheduler_orchestrator pytest \
-  scheduler_orchestrator/tests/test_alembic_downgrade.py \
-  -m "docker and alembic" -v --timeout=120
-```
-
-Expected: **9 passed** in ~8 seconds against a fresh testcontainers Postgres 16-alpine container. Tests cover the full `upgrade head → downgrade base → upgrade head` cycle.
-
-### Smoke Deploy Integration
-
-`scripts/smoke-test-deploy.sh` step 8 runs these tests automatically post-`make verify`. To skip them in environments without Docker:
-
-```bash
-SKIP_DOWNGRADE_TESTS=1 bash scripts/smoke-test-deploy.sh --target live
-```
-
-The skip is recorded as `exit_code=0 / SKIPPED` in the JSON evidence artifact at `docs/p4-closure-evidence/smoke-<timestamp>-<sha>.json`.
-
-### What These Tests Catch
-
-If a future Alembic migration regresses to `DROP SCHEMA sched CASCADE` in `0001.downgrade()` (the C1 pattern WP-5.1 fixed), this test suite will fail at `test_03_downgrade_base_succeeds` — the post-`downgrade()` alembic-cleanup hook would raise `relation "sched.alembic_version" does not exist`. **Direct regression coverage for the WP-5.1 CRITICAL fix.**
-
-Cross-reference: `.gpt-5.4/specs/ecosystem-infrastructure-evolution/design.md` ADR-INF-11 §Schema-Drop-Safe Downgrade Pattern (the architectural decision these tests enforce).
-
----
-
-## Sign-off
-
-| Role | Name | Date | Status |
-|---|---|---|---|
-| Production ops engineer | _TBD_ | _YYYY-MM-DD_ | _pending_ |
-| Platform team lead | _TBD_ | _YYYY-MM-DD_ | _pending_ |
-| Security reviewer (ADR-19 audit grants) | _TBD_ | _YYYY-MM-DD_ | _pending_ |