@ngocsangairvds/vsaf 3.2.15 → 4.0.0

package/tools/vds-scripts/scheduler_orchestrator/docs/conductor-decision.md

@@ -1,232 +0,0 @@
-# DBOS Conductor Adoption Decision (TSK-11.10 / ADR-13)
-
-> **[HISTORICAL — POST-RETIREMENT 2026-04-24]**
-> Decision document preserved as audit trail. The line in §Background referencing
-> "Phase 3 (Celery Beat migration of 6 periodic tasks)" reflects the pre-retirement
-> two-scheduler design. ADR-14 retirement (executed 2026-04-24, commit `a0ba4179`)
-> migrated those 6 periodic tasks to DBOS schedules in `scheduler_orchestrator`;
-> see `WHO-project/vds-scripts/scheduler_orchestrator/CLOSURE-v2.0.18.md`.
-> The Conductor adoption decision itself is unaffected by retirement and remains
-> at the v2.0.6 outcome documented below.
-
-**Date**: 2026-04-24  
-**Author**: ops-dev  
-**Spec ref**: `vds-scheduler/design.md §ADR-13`  
-**Decision gate**: Phase 2
-
----
-
-## Background
-
-ADR-10 deferred DBOS Conductor to Phase 2. ADR-13 mandates a data-driven
-decision at Phase 2 planning: measure actual Phase-1 workflow volume for 30 days
-and apply the threshold rule:
-
-> IF monthly workflow count > 10 000 OR incident frequency ≥ 2/month  
-> THEN adopt paid Conductor license  
-> ELSE build a minimal custom dashboard against `dbos.*` tables
-
----
-
-## Measurement Methodology
-
-The following queries are run against the scheduler's Postgres database
-(the same DB referenced by `VDS_SCHEDULER_DATABASE_URL`).  
-They require SELECT access to the `dbos` schema.
-
-### 1. 30-day workflow volume
-
-```sql
--- Total workflows started in the last 30 days
-SELECT
-    count(*)                                             AS total_workflows,
-    count(*) FILTER (WHERE status = 'SUCCESS')           AS succeeded,
-    count(*) FILTER (WHERE status = 'ERROR')             AS errored,
-    count(*) FILTER (WHERE status = 'CANCELLED')         AS cancelled,
-    count(*) FILTER (WHERE status
-        IN ('MAX_RECOVERY_ATTEMPTS_EXCEEDED'))            AS exhausted,
-    count(*) FILTER (WHERE status IN ('PENDING','ENQUEUED','DELAYED'))
-                                                         AS in_flight
-FROM dbos.workflow_status
-WHERE created_at >= now() - interval '30 days';
-```
-
-### 2. Monthly workflow count trend (by week)
-
-```sql
-SELECT
-    date_trunc('week', created_at)  AS week_start,
-    count(*)                        AS workflows
-FROM dbos.workflow_status
-WHERE created_at >= now() - interval '30 days'
-GROUP BY 1
-ORDER BY 1;
-```
-
-### 3. Incident frequency proxy (ERROR + exhausted per week)
-
-```sql
-SELECT
-    date_trunc('week', created_at)                              AS week_start,
-    count(*) FILTER (WHERE status = 'ERROR'
-        OR status = 'MAX_RECOVERY_ATTEMPTS_EXCEEDED')           AS incident_proxy
-FROM dbos.workflow_status
-WHERE created_at >= now() - interval '30 days'
-GROUP BY 1
-ORDER BY 1;
-```
-
-### 4. Queue breakdown
-
-```sql
-SELECT
-    queue_name,
-    count(*)                                            AS total,
-    count(*) FILTER (WHERE status = 'SUCCESS')          AS succeeded,
-    round(
-        100.0 * count(*) FILTER (WHERE status = 'ERROR')
-              / nullif(count(*), 0), 2
-    )                                                   AS error_pct
-FROM dbos.workflow_status
-WHERE created_at >= now() - interval '30 days'
-GROUP BY queue_name
-ORDER BY total DESC;
-```
-
----
-
-## Phase 1 Measurement Results
-
-> **Context**: Phase 1 ran from approximately 2026-03-20 → 2026-04-24.
-> The scheduler was gated by `VDS_SCHEDULER_ENABLED=false` in production;
-> workflows were executed during:
-> - Phase 1 smoke/acceptance tests (TSK-1.4)
-> - Weekly `audit_drift_scan` cron (TSK-7.1 — weekly cadence, 1 workflow/week)
-> - Chain integration tests and load-test runs (WP-8)
-
-| Metric | Observed value |
-|--------|---------------|
-| Total workflows (30d) | **~150** (smoke + drift scans + load tests) |
-| Monthly extrapolation | **~150/month** at current cadence |
-| ERROR workflows | **< 5** (load-test failure injections) |
-| Incident proxy (ERROR/week) | **< 1/month** |
-| Peak queue depth (WP-8) | **~40 concurrent** (`agent-default`, load test) |
-| Production steady-state volume | **4–8 workflows/week** (drift scan + ops triggers) |
-
-> **Note**: Full production deployment is NOT yet live — `VDS_SCHEDULER_ENABLED`
-> remains `false` in `~/.vds/.env`. These figures are test-environment observations
-> from worktree `vds-scripts-ecosystem-scheduling-phase2`. Volume projections for
-> Phase 3 (Celery Beat migration of 6 periodic tasks) are estimated below.
-
-### Phase 3 volume projection
-
-When Phase 3 migrates the 6 `memory_evolution` tasks (TSK-12.x):
-
-| Task | Cadence | Monthly workflows |
-|------|---------|-------------------|
-| memory-evolution-dispatch | 60 s | ~43 200 |
-| memory-evolution-sweep | 1 h | ~720 |
-| memory-evolution-replay-dlq | 15 min | ~2 880 |
-| mem0-resolve-pending-ids | 5 min | ~8 640 |
-| mem0-repush-stalled | 1 h | ~720 |
-| mem0-repush-local-only | 6 h | ~120 |
-| **Total (Phase 3 steady state)** | | **~56 280 / month** |
-
----
-
-## Decision
-
-### ADR-13 gate applied
-
-| Criterion | Threshold | Phase 1 actual | Phase 3 projected |
-|-----------|-----------|----------------|-------------------|
-| Monthly workflow count | > 10 000 | ~150 ✓ below | **~56 000 ✗ above** |
-| Incident frequency | ≥ 2/month | < 1 ✓ below | unknown |
-
-**Phase 1 alone does NOT meet the paid-Conductor threshold.**  
-**Phase 3 WILL exceed the 10 K threshold** when Celery Beat tasks migrate.
-
-### Chosen path: **Build minimal custom dashboard now; defer Conductor to Phase 3 gate**
-
-**Rationale**:
-
-1. **Phase 1 volume is well below threshold** (150 vs. 10 000). Paying for
-   Conductor now would be premature per ADR-13's data-driven rule.
-
-2. **Phase 3 migration changes the picture entirely.** The 6 Celery Beat tasks
-   running at 1-minute and 5-minute cadences will push monthly volume to ~56K —
-   far above the 10K gate. However, Phase 3 work begins after Phase 2 gates, so
-   the Conductor evaluation is more appropriately done at the Phase 3 gate.
-
-3. **Custom dashboard covers current needs**. The existing `vds-cli workflows
-   list/show`, Prometheus metrics, and the new WebSocket events tail (TSK-11.9)
-   give operators sufficient visibility for Phase 1 + Phase 2.
-
-4. **Conductor integration is trivial when needed**. Per ADR-10: set
-   `DBOS_CONDUCTOR_KEY` env var and re-launch DBOS. No code changes required.
-   This keeps the Phase 3 upgrade path open at near-zero migration cost.
-
----
-
-## Minimal Custom Dashboard Build Plan
-
-### Deliverables (Phase 2)
-
-| Item | Status | Notes |
-|------|--------|-------|
-| `scheduler_orchestrator/dashboards/` module | Skeleton exists | Grafana JSON dashboards |
-| `GET /api/v1/workflows` + `GET /api/v1/workflows/{id}` | CLI stubs (TSK-6.2) | Full backend in WP-3 |
-| `GET /api/v1/events` + `WS /api/v1/events/stream` | REST stub + WS live (TSK-11.9) | WebSocket landed this phase |
-| Grafana `vds-scheduler` dashboard (FR-7.2/7.3) | TSK-6.3 metrics landed | Prometheus-backed |
-| `vds-scheduler workflows list --format table` | Live (TSK-6.2) | |
-
-### Dashboard panels (minimal viable set)
-
-| Panel | Data source | Query |
-|-------|-------------|-------|
-| Workflow throughput / min | Prometheus | `rate(vds_scheduler_workflow_duration_seconds_count[5m])` |
-| Error rate % | Prometheus | workflow_failures_total / runs_total |
-| Queue depth | Prometheus | `vds_scheduler_queue_depth{queue_name=...}` |
-| Active workflows | `dbos.workflow_status` | `WHERE status IN ('PENDING','ENQUEUED')` |
-| Recent incidents | `dbos.workflow_status` | `WHERE status IN ('ERROR','MAX_RECOVERY_ATTEMPTS_EXCEEDED') AND created_at > now()-1d` |
-| Events outbox lag | Prometheus | `vds_scheduler_event_bus_lag_seconds` |
-
-### Phase 3 gate revisit
-
-At Phase 3 planning (Celery Beat migration kickoff), re-run the ADR-13
-measurement queries. If projected monthly volume > 10K is confirmed in
-production, initiate Conductor free-trial → evaluate paid license.
-
----
-
-## Conductor quick-adopt runbook (if Phase 3 gate triggers)
-
-```bash
-# 1. Obtain Conductor API key from https://dbos.dev/dbos-conductor
-export DBOS_CONDUCTOR_KEY="<key>"
-echo "DBOS_CONDUCTOR_KEY=<key>" >> ~/.vds/.env
-
-# 2. Restart scheduler service — DBOS reads the key at launch
-docker compose restart scheduler
-
-# 3. Verify connection in logs
-docker logs vds-scheduler | grep -i conductor
-# Expected: "Connected to DBOS Conductor at conductor.dbos.dev"
-
-# 4. Visit Conductor dashboard at https://dbos.dev/conductor
-#    Select app "vds-scheduler-orchestrator"
-```
-
-No code changes are required (ADR-10 consequence). The `DBOS_CONDUCTOR_KEY` env
-var is already plumbed through `SchedulerConfig.dbos_conductor_key` in
-`config.py`.
-
----
-
-## References
-
-- ADR-10: `design.md §ADR-10 DBOS Conductor deferred to Phase 2`
-- ADR-13: `design.md §ADR-13 DBOS Conductor decision deferred to Phase 2 gate`
-- FR-7.2/7.3: Prometheus metrics + Grafana dashboard
-- TSK-11.9: WebSocket events tail (landed this phase)
-- DBOS Conductor docs: <https://docs.dbos.dev/production/conductor>

package/tools/vds-scripts/scheduler_orchestrator/docs/cross-orchestrator-rollout-checklist.md

@@ -1,223 +0,0 @@
-# Cross-Orchestrator Scheduler Rollout Checklist
-
-> TSK-14.7 — Phase 3 deliverable. Practical operator checklist for rolling out
-> the VDS ecosystem scheduler across all orchestrators that consume scheduled
-> workflows or publish/subscribe to the event bus.
-
-## Pre-Rollout Gates
-
-Before enabling `VDS_SCHEDULER_ENABLED=true` in any environment, verify:
-
-- [ ] **DB schema**: Run `vds-cli scheduler db migrate` — all Alembic migrations applied.
-  - Key tables: `sched.schedules`, `sched.workflow_registry`, `sched.events_outbox`,
-    `sched.audit_log`, `dbos.workflow_status`.
-  - Verify: `psql $VDS_SCHEDULER_DATABASE_URL -c "\dt sched.*"` shows all tables.
-- [ ] **Queue validation**: `vds-cli scheduler queues list` returns all four queues:
-  `ops-default`, `agent-default`, `agent-partitioned`, `chain-fanout`.
-- [ ] **DBOS connectivity**: `VDS_SCHEDULER_ENABLED=true` + `/health` returns
-  `{"status": "ok", "scheduler_enabled": true}`.
-- [ ] **Capability token**: `VDS_SCHEDULER_CAPABILITY` set in `~/.vds/.env`.
-  Required for schedule create/pause/delete (capability model AC-3).
-- [ ] **LangGraph checkpoint tables**: `sched.lg_checkpoints` visible (created by
-  `AsyncPostgresSaver.setup()` in lifespan). Audit workflows fail without this.
-- [ ] **Kill-switch default**: Confirm `VDS_SCHEDULER_ENABLED=false` in prod env
-  until all gates are green.
-
----
-
-## Per-Orchestrator Rollout Steps
-
-### audit_orchestrator (TSK-7.1 + TSK-14.4)
-
-Registers three scheduled audit workflows at startup.
-
-**Verify bootstrap completed:**
-
-```bash
-vds-cli scheduler workflows registry list | grep audit
-# Expect: audit.drift.weekly, audit.security.daily, audit.deep.monthly
-```
-
-**Verify schedules exist:**
-
-```bash
-vds-cli scheduler schedules list | grep audit
-# Expect: audit-drift-weekly (paused=false), audit-security-daily, audit-deep-monthly
-```
-
-**Cron expressions (all UTC):**
-
-| Schedule | Alias | Cron | Fires at |
-|---|---|---|---|
-| `audit-drift-weekly` | `audit.drift.weekly` | `0 2 * * MON` | Mon 02:00 |
-| `audit-security-daily` | `audit.security.daily` | `0 3 * * *` | Daily 03:00 |
-| `audit-deep-monthly` | `audit.deep.monthly` | `0 4 1 * *` | 1st of month 04:00 |
-
-**Manual trigger test (use mock LLM context):**
-
-```bash
-# Weekly drift — quick smoke test
-vds-cli scheduler schedule trigger-now audit-drift-weekly
-
-# Security daily — verify security profile reaches agent
-vds-cli scheduler schedule trigger-now audit-security-daily
-
-# Deep monthly — verify all collectors are loaded
-vds-cli scheduler schedule trigger-now audit-deep-monthly
-```
-
-**Verify workflow completed:**
-
-```bash
-# After trigger-now, poll status (replace <id> with returned workflow_id)
-vds-cli scheduler workflows status <id>
-# Expect: status=SUCCESS (or PENDING while running)
-```
-
-**Rollback:**
-
-```bash
-vds-cli scheduler schedule pause audit-security-daily
-vds-cli scheduler schedule pause audit-deep-monthly
-# (weekly drift was already live from Phase 1 — do not pause unless rolling back)
-```
-
----
-
-### progress_report_orchestrator (TSK-14.5)
-
-Thin event-bus subscriber — no DBOS workflow at Phase 1. Activation requires the
-event bus LISTEN loop (TSK-N.6, Phase 2).
-
-**Verify import is clean:**
-
-```bash
-cd progress_report_orchestrator
-python -c "from progress_report_orchestrator.scheduler_subscriber import on_audit_run_completed; print('OK')"
-```
-
-**Verify manual CLI:**
-
-```bash
-# Check command is visible
-vds-cli progress report --help
-
-# Dry-run manual report (requires a valid PSK)
-vds-cli progress report generate --project-storage-key <PSK> --output-dir /tmp/reports
-```
-
-**Phase 2 activation (when TSK-N.6 merges):**
-
-1. Wire `on_audit_run_completed` into the LISTEN loop in
-   `scheduler_orchestrator/events/bus.py`.
-2. Confirm the handler fires when `audit.run.completed` events are published.
-3. Un-comment the `q.enqueue(generate_report_workflow, ...)` block in
-   `scheduler_subscriber.py` and register the workflow alias.
-
-**Current status:** Handler is registered as a decorator stub. Analytics logging
-is live once the LISTEN loop is activated.
-
----
-
-### multi_agent_orchestrator (TSK-14.6)
-
-Registers `agent-pool-rebalance` (every 15 minutes) via DBOS workflow.
-
-**Verify bootstrap completed:**
-
-```bash
-vds-cli scheduler workflows registry list | grep agent
-# Expect: agent.pool.rebalance
-```
-
-**Verify schedule exists:**
-
-```bash
-vds-cli scheduler schedules list | grep agent-pool
-# Expect: agent-pool-rebalance (paused=false, cron=*/15 * * * *)
-```
-
-**Manual trigger test:**
-
-```bash
-vds-cli scheduler schedule trigger-now agent-pool-rebalance
-```
-
-**Verify rebalancing output in logs:**
-
-```
-rebalance: analytics  total_agents=N  load_pct=X.X  stale_cleaned=0  recommendations=[]
-```
-
-**Dependency check — Redis required:**
-
-```bash
-# Confirm Redis is reachable before enabling
-redis-cli -u $REDIS_URL ping   # → PONG
-```
-
-**Rollback:**
-
-```bash
-vds-cli scheduler schedule pause agent-pool-rebalance
-```
-
----
-
-## Post-Rollout Validation
-
-After all three orchestrators are live:
-
-- [ ] **Registry completeness**: `vds-cli scheduler workflows registry list` shows at least 5 aliases:
-  `audit.drift.weekly`, `audit.security.daily`, `audit.deep.monthly`,
-  `agent.pool.rebalance`, plus any chain/evolution aliases from WP-12/13.
-- [ ] **No PENDING timeouts**: `vds-cli scheduler workflows list --status PENDING` is empty or
-  clears within `concurrency × average_workflow_duration`.
-- [ ] **Audit log clean**: `vds-cli scheduler audit-log list --limit 50` shows no `denied` entries.
-- [ ] **Event outbox drains**: `vds-cli scheduler events outbox list --status pending` empties
-  within the outbox retention window (default 24h).
-- [ ] **Prometheus metrics**: `/metrics` on the scheduler endpoint shows:
-  - `vds_scheduler_schedule_fired_total` incrementing.
-  - `vds_scheduler_workflow_status_total{status="SUCCESS"}` growing.
-  - `vds_scheduler_workflow_status_total{status="ERROR"}` ≤ 5%.
-- [ ] **SLO gate**: After 24h, error budget consumption ≤ 0.5% (per `error-budget-policy.md`).
-
----
-
-## Known Gotchas
-
-| Symptom | Likely cause | Fix |
-|---|---|---|
-| `AliasNotFoundError` on trigger-now | Bootstrap didn't run (scheduler was disabled when the orchestrator started) | Enable scheduler, restart, let lifespan run bootstrap_* |
-| `UnknownQueueError` on schedule create | Queue name typo or `build_queues()` not called before `DBOS.launch()` | Check `queues.py` KNOWN_QUEUE_NAMES; verify startup order |
-| Cron schedule not firing | Schedule paused OR DBOS worker loop not running | `vds-cli scheduler schedules get <name>` → check `paused`; verify process is running |
-| `DuplicateScheduleError` on first run | Schedule row exists from a prior run but alias wasn't registered | Run `bootstrap_*()` idempotently — it silences this error |
-| Progress report `pipeline.report.run` ImportError | `pipeline/report.py` doesn't expose a `run()` coroutine | Wire `scheduler_subscriber._run_report_pipeline` to actual pipeline entry point |
-| Agent pool rebalance fails with Redis timeout | Redis unavailable | Check `redis-cli ping`; `multi_agent_orchestrator` config `REDIS_URL` |
-| `VDS_SCHEDULER_CAPABILITY` missing | Capability checks reject all mutations | Set in `~/.vds/.env`; use `vds-cli env status` to verify |
-
----
-
-## Rollout Order Recommendation
-
-```
-1. Apply DB migrations   (idempotent — always safe to run)
-2. Enable scheduler      (VDS_SCHEDULER_ENABLED=true)
-3. Deploy audit_orchestrator  → verify 3 audit aliases + schedules
-4. Deploy multi_agent_orchestrator → verify agent-pool-rebalance
-5. Deploy progress_report_orchestrator → verify CLI + stub handler import
-6. Monitor 24h          → check error budget, outbox drain, PENDING queue
-7. Activate Phase 2 bus LISTEN loop (TSK-N.6) → progress_report analytics go live
-```
-
----
-
-## References
-
-- Spec: `.gpt-5.4/specs/vds-scheduler/`
-- Runbooks: `scheduler_orchestrator/docs/runbooks/`
-- Workflow registry API: `scheduler_orchestrator/src/vds_scheduler_orchestrator/workflows/registry.py`
-- Schedules CRUD API: `scheduler_orchestrator/src/vds_scheduler_orchestrator/scheduler/schedules.py`
-- Queues: `scheduler_orchestrator/src/vds_scheduler_orchestrator/workflows/queues.py`
-- WAVE-2-GUARDRAILS: `scheduler_orchestrator/WAVE-2-GUARDRAILS.md`
-- Skill: `WHO-project/vds-skills/skills/scheduler-skill/SKILL.md`

package/tools/vds-scripts/scheduler_orchestrator/docs/dr-drill-report.md

@@ -1,173 +0,0 @@
-# DR Drill Report — PITR Template (TSK-17.4)
-
-**Runbook ref**: `docs/runbooks/dr-postgres-restore.md` (DR-001)  
-**Drill script**: `scripts/dr/pitr-drill.sh`  
-**Spec ref**: `vds-scheduler/design.md §Disaster Recovery / WP-17 TSK-17.4`
-
-> **OPS INSTRUCTIONS**: Fill in every `<fill in>` field when running the drill.
-> Commit this file to the worktree after each drill for audit trail.
-> The drill script emits a machine-readable `evidence.txt` — copy values here.
-
----
-
-## Drill Metadata
-
-| Field | Value |
-|-------|-------|
-| Drill date | `<fill in: YYYY-MM-DD>` |
-| Drill lead | `<fill in: name/alias>` |
-| Environment | `<fill in: local staging / staging-k8s / etc.>` |
-| Drill mode | `<fill in: dry / simulate / live>` |
-| Drill script version | `<fill in: git log --oneline -1 scripts/dr/pitr-drill.sh>` |
-| Postgres version | `<fill in: psql --version>` |
-| DBOS version | `<fill in: pip show dbos \| grep Version>` |
-| Backup store | `<fill in: s3://... or local path>` |
-| WAL archive path | `<fill in: s3://... or local path>` |
-| Recovery target time | `<fill in: YYYY-MM-DDTHH:MM:SSZ UTC>` |
-
----
-
-## Pre-Drill Snapshot
-
-| Table | Row count (pre-drill) |
-|-------|----------------------|
-| `sched.schedules` | `<fill in>` |
-| `sched.events_outbox` | `<fill in>` |
-| `sched.subscriptions` | `<fill in>` |
-| `sched.workflow_registry` | `<fill in>` |
-| `sched.dispatch_ack_log` | `<fill in>` |
-| `sched.chain_fire_log` | `<fill in>` |
-| `dbos.workflow_status` (total) | `<fill in>` |
-
-Anchor row seeded: `<fill in: pitr-drill-anchor-<unix-ts>>`  
-Snapshot timestamp (UTC): `<fill in>`
-
----
-
-## Restore Timeline
-
-| Phase | Wall-clock time (UTC) | Elapsed (s) |
-|-------|----------------------|-------------|
-| Drill start | `<fill in>` | 0 |
-| Kill-switch applied | `<fill in>` | `<fill in>` |
-| Base backup restore complete | `<fill in>` | `<fill in>` |
-| WAL replay started | `<fill in>` | `<fill in>` |
-| WAL replay complete (promoted) | `<fill in>` | `<fill in>` |
-| Alembic upgrade head | `<fill in>` | `<fill in>` |
-| Scheduler reconnected + healthy | `<fill in>` | `<fill in>` |
-| **Total RTO** | — | **`<fill in>` s** |
-
-RTO SLO: **3 600 s (60 min)** per design.md.  
-RTO status: `<fill in: WITHIN SLO / EXCEEDED SLO>`
-
----
-
-## Verification Checks
-
-| Check | Expected | Observed | Status |
-|-------|----------|----------|--------|
-| Anchor row present in restored DB | count = 1 | `<fill in>` | `<fill in: PASS/FAIL>` |
-| `sched.schedules` count matches snapshot | = pre-drill count | `<fill in>` | `<fill in>` |
-| `sched.events_outbox` count within tolerance | ≤ pre-drill (some may have been dispatched) | `<fill in>` | `<fill in>` |
-| `sched.subscriptions` count | = pre-drill count | `<fill in>` | `<fill in>` |
-| `dbos.workflow_status` — no orphaned PENDING | 0 rows stuck PENDING > step-timeout | `<fill in>` | `<fill in>` |
-| Alembic at head | head | `<fill in>` | `<fill in>` |
-| Scheduler `/health` response | `{"status":"ok","scheduler_enabled":true}` | `<fill in>` | `<fill in>` |
-| Smoke test enqueue → SUCCESS within 5 s | SUCCESS | `<fill in>` | `<fill in>` |
-
-### Smoke test command used
-
-```bash
-# Run after scheduler restart against the restored DB
-curl -sf -X POST "${CHAOS_API_URL}/test/enqueue" | python3 -m json.tool
-# Then poll:
-# curl -sf "${CHAOS_API_URL}/api/v1/workflows/<id>" | python3 -m json.tool
-```
-
----
-
-## Gaps and Findings
-
-| ID | Gap / Finding | Severity | Recommendation | Owner |
-|----|--------------|----------|----------------|-------|
-| `<fill in: G-1>` | `<fill in>` | `<Low/Medium/High>` | `<fill in>` | `<fill in>` |
-
-*(Add rows as needed. "None observed" is a valid finding.)*
-
----
-
-## WAL Archive Health
-
-| Metric | Observed | Notes |
-|--------|----------|-------|
-| WAL segments available | `<fill in>` | `aws s3 ls <wal-path> \| wc -l` |
-| Oldest WAL segment date | `<fill in>` | Should be ≥ 90 days (matching outbox retention) |
-| Base backup date | `<fill in>` | |
-| PITR window (oldest WAL → now) | `<fill in>` days | Target: ≥ 7 days |
-
----
-
-## Overall Drill Result
-
-| Field | Value |
-|-------|-------|
-| **Overall result** | `<fill in: PASS / FAIL / PARTIAL>` |
-| **RTO achieved** | `<fill in>` s |
-| **Data loss window** | `<fill in: 0 / N seconds (records committed after target time)>` |
-| **Gaps found** | `<fill in: count>` |
-| **Next drill scheduled** | `<fill in: first Monday of next quarter>` |
-
----
-
-## Drill Sign-Off
-
-| Role | Name | Date |
-|------|------|------|
-| Drill lead | `<fill in>` | `<fill in>` |
-| DR runbook reviewer | `<fill in>` | `<fill in>` |
-| Infra sign-off | `<fill in>` | `<fill in>` |
-
----
-
-## Appendix: Quick Commands Reference
-
-```bash
-# Kill-switch
-export VDS_SCHEDULER_ENABLED=false
-docker compose restart scheduler
-
-# Record table counts (pre/post)
-psql "$VDS_SCHEDULER_DATABASE_URL" -c "
-SELECT 'schedules' AS t, count(*) FROM sched.schedules
-UNION ALL SELECT 'events_outbox', count(*) FROM sched.events_outbox
-UNION ALL SELECT 'subscriptions', count(*) FROM sched.subscriptions
-UNION ALL SELECT 'workflow_registry', count(*) FROM sched.workflow_registry
-UNION ALL SELECT 'dispatch_ack_log', count(*) FROM sched.dispatch_ack_log
-UNION ALL SELECT 'chain_fire_log', count(*) FROM sched.chain_fire_log
-ORDER BY t;"
-
-# Check DBOS orphaned workflows
-psql "$VDS_SCHEDULER_DATABASE_URL" -c "
-SELECT count(*) AS stuck_pending
-FROM dbos.workflow_status
-WHERE status = 'PENDING'
-  AND updated_at < now() - interval '10 minutes';"
-
-# Alembic version check
-cd scheduler_orchestrator && uv run alembic current
-
-# Re-enable after restore
-export VDS_SCHEDULER_ENABLED=true
-docker compose restart scheduler
-curl -sf http://localhost:8005/health | python3 -m json.tool
-```
-
----
-
-## History
-
-| Drill date | Mode | RTO | Result | Lead |
-|------------|------|-----|--------|------|
-| *(first drill — fill in after running)* | — | — | — | — |
-
-*(Append a row after each quarterly drill. See `docs/quarterly-game-day.md` for cadence.)*