@ngocsangairvds/vsaf 3.2.16 → 4.0.0

package/tools/vds-scripts/scheduler_orchestrator/docs/evolution-daemon-retirement.md

@@ -1,214 +0,0 @@
-# Evolution Daemon Retirement Procedure
-
-> **[HISTORICAL — RETIREMENT EXECUTED 2026-04-24]**
-> ADR-14 staged 14-day soak overridden by user directive ("perform complete migration, do not keep any legacy, compatibility").
-> All 6 OS-level daemon files have been removed via `git rm` on 2026-04-24.
-> The `vds-cli evolution auto-promote` polling CLI has been replaced with a deprecation notice.
-> `AutoPromoteDaemon.run_once()` is preserved (called by `scheduler_orchestrator` DBOS step).
-> This document is preserved as decision history only.
-
-**TSK-13.2** — Procedure for retiring the OS-level `vds-evolution-auto` daemon
-after a 14-day parallel-run soak period confirms the DBOS schedule is stable.
-
----
-
-## Background
-
-Prior to Phase 3, the Evolution auto-promote loop ran as an OS-supervised
-daemon with four deployment-mode files:
-
-| Deployment mode | File |
-|---|---|
-| systemd (Linux service) | `vds-evolution-auto.service` + `vds-evolution-auto.timer` |
-| crontab (POSIX fallback) | `crontab-evolution-auto.txt` |
-| launchd (macOS) | `com.vds-ai.vds-evolution-auto.plist` |
-| Shared run script | `scripts/evolution-auto-run.sh` |
-| Supervisor installer | `scripts/install-evolution-auto.sh` |
-
-TSK-13.1 replaces all four with a single DBOS cron schedule
-(`evolution.auto-promote`, `*/1 * * * *`) running under the ecosystem scheduler.
-
-See [`evolution-retirement-files.md`](./evolution-retirement-files.md) for the
-exact paths and delete commands.
-
----
-
-## Retirement Phases
-
-### Phase 0 — Verify DBOS schedule is active
-
-Before starting the soak period, confirm the new schedule is registered and
-ticking correctly:
-
-```bash
-# Check schedule is registered
-./scripts/worktree_uv.sh run --project vds_cli vds-cli scheduler schedules list \
-  | grep evolution-auto-promote
-
-# Confirm last run (should update every ~60 seconds)
-./scripts/worktree_uv.sh run --project vds_cli vds-cli scheduler schedules show \
-  evolution-auto-promote
-
-# Watch Prometheus: evolution.auto_promote ticks should increment
-curl -s http://localhost:9090/metrics | grep evolution_auto_promote
-```
-
-Expected: schedule shows `status=active`, `cron=*/1 * * * *`,
-`last_run_at` updated within the last 90 seconds.
-
----
-
-### Phase 1 — 14-day parallel run (soak period)
-
-Run both the DBOS schedule **and** the OS daemon simultaneously.
-The daemon continues to operate; the DBOS schedule ticks in parallel.
-
-**Goal**: confirm that DBOS ticks correctly detect and promote pending
-verdicts, that `run_once()` remains idempotent under dual execution, and
-that no verdict is silently double-promoted.
-
-**Success criteria** (all must hold for 14 consecutive days):
-
-| Criterion | Pass condition |
-|---|---|
-| DBOS ticks fire every minute | `last_run_at` never stale > 90 s |
-| No double-promotion | Each verdict has exactly one `*-promoted.json` companion |
-| No stale PENDING workflows | `vds-cli scheduler workflows list --status PENDING \| grep evolution` returns empty |
-| Error rate < 1% | DBOS step retries < 1 per day on average |
-| OS daemon still promotes | Daemon log shows no errors throughout soak |
-
-**Monitoring commands (run daily during soak)**:
-
-```bash
-# Count stale DBOS workflows (should be 0)
-./scripts/worktree_uv.sh run --project vds_cli vds-cli scheduler workflows list \
-  --status PENDING | grep evolution | wc -l
-
-# Count DBOS step retry events (should be low)
-./scripts/worktree_uv.sh run --project vds_cli vds-cli scheduler workflows list \
-  --status ERROR | grep evolution
-
-# OS daemon heartbeat (systemd)
-systemctl status vds-evolution-auto.timer --no-pager
-
-# Verify no double-promotions (all verdicts have exactly one promoted file)
-find ~/.vds/evolution-log -name 'verdict-*.json' | while read vf; do
-  promoted="${vf%.json}-promoted.json"
-  if [ ! -f "$promoted" ]; then
-    echo "UNPROMOTED: $vf"
-  fi
-done
-```
-
----
-
-### Phase 2 — Daemon shutdown
-
-After 14 clean soak days, stop and disable the OS-level daemon.
-
-**systemd (Linux)**:
-
-```bash
-# Disable and stop timer + service
-sudo systemctl disable --now vds-evolution-auto.timer
-sudo systemctl disable --now vds-evolution-auto.service
-
-# Verify no more daemon ticks
-journalctl -u vds-evolution-auto.service -n 5 --no-pager
-```
-
-**crontab (POSIX fallback)**:
-
-```bash
-# Remove evolution crontab line
-crontab -l | grep -v 'evolution-auto' | crontab -
-```
-
-**launchd (macOS)**:
-
-```bash
-# Unload the plist
-launchctl unload ~/Library/LaunchAgents/com.vds-ai.vds-evolution-auto.plist
-```
-
-After stopping the daemon, run one more manual promotion pass to ensure no
-verdict slipped between the final daemon tick and DBOS taking over exclusively:
-
-```bash
-./scripts/worktree_uv.sh run --project vds_cli vds-cli evolution auto-promote --dry-run
-```
-
-Confirm output shows 0 pending verdicts, then proceed.
-
----
-
-### Phase 3 — File deletion
-
-Delete all OS-level daemon files from the repository.
-See [`evolution-retirement-files.md`](./evolution-retirement-files.md) for the
-explicit file list and per-file delete commands.
-
-Commit the deletions with a `chore(evolution)` commit message referencing TSK-13.3.
-
----
-
-### Phase 4 — Post-retirement verification
-
-Confirm the ecosystem runs cleanly on DBOS alone:
-
-```bash
-# Scheduler health
-./scripts/worktree_uv.sh run --project vds_cli vds-cli scheduler schedules show \
-  evolution-auto-promote
-
-# Force-trigger one tick and verify it completes
-./scripts/worktree_uv.sh run --project vds_cli vds-cli scheduler schedules trigger-now \
-  evolution-auto-promote
-
-# Poll for completion (workflow should reach SUCCESS within 60 s)
-./scripts/worktree_uv.sh run --project vds_cli vds-cli scheduler workflows list \
-  --status SUCCESS | grep evolution | tail -3
-```
-
----
-
-## Rollback Procedure
-
-If the DBOS schedule fails during the soak period, roll back by ensuring the
-OS daemon is still running (Phase 1 keeps it active), then pause the schedule:
-
-```bash
-./scripts/worktree_uv.sh run --project vds_cli vds-cli scheduler schedules pause \
-  evolution-auto-promote
-```
-
-Investigate the DBOS error logs, fix the underlying issue, then resume:
-
-```bash
-./scripts/worktree_uv.sh run --project vds_cli vds-cli scheduler schedules resume \
-  evolution-auto-promote
-```
-
-Do **not** delete the DBOS schedule during soak — pause/resume preserves
-the existing alias registration and avoids re-bootstrapping.
-
----
-
-## Completion Checklist
-
-- [ ] Phase 0: DBOS schedule verified active before soak start
-- [ ] Phase 1: 14-day soak completed with no double-promotions, no stale PENDING
-- [ ] Phase 2: OS daemon stopped and disabled on all hosts
-- [ ] Phase 3: Daemon files deleted from repository (TSK-13.3 commit merged)
-- [ ] Phase 4: Post-retirement trigger test passes
-- [ ] TSK-13.3 merged: `evolution-retirement-files.md` commit is on master
-- [ ] ECOSYSTEM-CHANGELOG.md updated with retirement date
-
----
-
-## References
-
-- TSK-13.1: DBOS workflow implementation (`workflows/evolution/auto_promote.py`)
-- TSK-13.3: Retirement file manifest (`docs/evolution-retirement-files.md`)
-- ADR-15: Kill-switch rule (`VDS_SCHEDULER_ENABLED=false`)
-- `AGENTS.md §Scheduler Architecture` — Two-scheduler operator note

package/tools/vds-scripts/scheduler_orchestrator/docs/evolution-retirement-files.md

@@ -1,171 +0,0 @@
-# Evolution Daemon Retirement — File Manifest
-
-> **[HISTORICAL — RETIREMENT EXECUTED 2026-04-24]**
-> All 6 files listed below have been removed via `git rm` on 2026-04-24.
-> The staged 14-day soak was overridden by user directive ("perform complete migration, do not keep any legacy, compatibility").
-> This manifest is preserved as decision history only.
-
-**TSK-13.3** — Explicit list of OS-level daemon files to delete from the
-`WHO-project/vds-scripts` repository after the 14-day soak period defined in
-[`evolution-daemon-retirement.md`](./evolution-daemon-retirement.md) (TSK-13.2).
-
-> **Do not delete these files before completing Phase 1 and Phase 2 of the
-> retirement procedure.** The OS daemon must be stopped and disabled on all
-> hosts before the repository files are removed.
-
----
-
-## File List
-
-All paths are relative to the repository root (`WHO-project/vds-scripts/`).
-
-### 1. `vds-evolution-auto.service`
-
-**Type**: systemd unit file (Linux service)
-**Purpose**: Defines the long-running `vds-evolution-auto.service` systemd unit
-that runs `scripts/evolution-auto-run.sh` as a persistent service.
-**Consumers**: `vds-evolution-auto.timer` (WantedBy), systemctl
-
-```bash
-git rm vds-evolution-auto.service
-```
-
----
-
-### 2. `vds-evolution-auto.timer`
-
-**Type**: systemd timer unit file (Linux)
-**Purpose**: Schedules the service to trigger every 60 seconds via
-`OnActiveSec=60` / `OnUnitActiveSec=60`. The timer is the cron-equivalent
-trigger for the service unit above.
-**Consumers**: systemd (`systemctl enable vds-evolution-auto.timer`)
-
-```bash
-git rm vds-evolution-auto.timer
-```
-
----
-
-### 3. `crontab-evolution-auto.txt`
-
-**Type**: crontab fragment (POSIX fallback)
-**Purpose**: `*/1 * * * *` crontab entry for environments where systemd is not
-available. Installed via `scripts/install-evolution-auto.sh --mode cron`.
-**Consumers**: `install-evolution-auto.sh`
-
-```bash
-git rm crontab-evolution-auto.txt
-```
-
----
-
-### 4. `com.vds-ai.vds-evolution-auto.plist`
-
-**Type**: launchd property list (macOS)
-**Purpose**: Defines the `com.vds-ai.vds-evolution-auto` Launch Agent that
-runs `scripts/evolution-auto-run.sh` on macOS with a 60-second `StartInterval`.
-**Consumers**: `launchctl load ~/Library/LaunchAgents/com.vds-ai.vds-evolution-auto.plist`
-
-```bash
-git rm com.vds-ai.vds-evolution-auto.plist
-```
-
----
-
-### 5. `scripts/evolution-auto-run.sh`
-
-**Type**: Shell script (shared run entrypoint)
-**Purpose**: Wrapper invoked by systemd, crontab, and launchd alike.
-Activates the vds-scripts virtualenv and calls
-`vds-cli evolution auto-promote --poll-interval 60`.
-**Consumers**: `vds-evolution-auto.service`, `crontab-evolution-auto.txt`,
-`com.vds-ai.vds-evolution-auto.plist`
-
-```bash
-git rm scripts/evolution-auto-run.sh
-```
-
----
-
-### 6. `scripts/install-evolution-auto.sh`
-
-**Type**: Supervisor installer script
-**Purpose**: One-shot installer that copies daemon files to the correct OS
-locations (`/etc/systemd/system/`, `~/Library/LaunchAgents/`, or crontab).
-Offers `--mode systemd|cron|launchd` and `--uninstall` flags.
-**Consumers**: Operator runbook; called once per host during initial setup.
-
-```bash
-git rm scripts/install-evolution-auto.sh
-```
-
----
-
-## Batch Delete Command
-
-After stopping all OS daemons (Phase 2 of the retirement procedure), run the
-following from the repository root to stage all six deletions in one step:
-
-```bash
-git rm \
-  vds-evolution-auto.service \
-  vds-evolution-auto.timer \
-  crontab-evolution-auto.txt \
-  com.vds-ai.vds-evolution-auto.plist \
-  scripts/evolution-auto-run.sh \
-  scripts/install-evolution-auto.sh
-```
-
-Then commit:
-
-```bash
-git commit -m "chore(evolution): TSK-13.3 retire OS-level daemon files post-soak
-
-Remove the six OS-supervisor files superseded by the DBOS
-evolution.auto-promote schedule (TSK-13.1). The 14-day parallel-run
-soak completed successfully per the procedure in
-docs/evolution-daemon-retirement.md (TSK-13.2).
-
-Files removed:
-  - vds-evolution-auto.service
-  - vds-evolution-auto.timer
-  - crontab-evolution-auto.txt
-  - com.vds-ai.vds-evolution-auto.plist
-  - scripts/evolution-auto-run.sh
-  - scripts/install-evolution-auto.sh"
-```
-
----
-
-## Verification After Deletion
-
-Confirm no other file in the repository references these paths:
-
-```bash
-# Should return no output
-rg --type-not markdown \
-  -e 'vds-evolution-auto\.(service|timer|plist)' \
-  -e 'crontab-evolution-auto' \
-  -e 'evolution-auto-run\.sh' \
-  -e 'install-evolution-auto\.sh' \
-  .
-```
-
-Run the CI test suite to confirm nothing breaks:
-
-```bash
-./scripts/worktree_uv.sh run --project scheduler_orchestrator \
-  pytest scheduler_orchestrator/tests/unit/test_evolution_auto_promote.py -v
-```
-
-All 15 tests should pass.
-
----
-
-## References
-
-- TSK-13.1: DBOS workflow (`scheduler_orchestrator/src/vds_scheduler_orchestrator/workflows/evolution/auto_promote.py`)
-- TSK-13.2: Retirement procedure (`docs/evolution-daemon-retirement.md`)
-- `crontab-evolution-auto.txt`: confirmed present at repo root (verified 2026-04-24)
-- `vds-evolution-auto.service` / `.timer`: confirmed present at repo root (verified 2026-04-24)
-- `com.vds-ai.vds-evolution-auto.plist`: confirmed present at repo root (verified 2026-04-24)

package/tools/vds-scripts/scheduler_orchestrator/docs/game-day-phase2.md

@@ -1,243 +0,0 @@
-# Phase 2 Chaos Game-Day Debrief (TSK-11.8)
-
-**Date**: 2026-04-24  
-**Duration**: 09:00–15:30 ICTST (6h 30m)  
-**Lead**: ops-dev  
-**Participants**: ops-dev, (chaos-dev — chaos script author, on-call for questions)  
-**Environment**: local staging (worktree `vds-scripts-ecosystem-scheduling-phase2`,
-  `VDS_SCHEDULER_ENABLED=true`, Postgres at `postgresql://vds_sched:***@localhost:5432/vds_sched_staging`)  
-**Scenarios executed**: 3 of 3 available (C-1, C-2, C-3)  
-**Spec ref**: `design.md §Chaos & Resilience Testing Plan FR-15.2`
-
----
-
-## Executive Summary
-
-| Scenario | Script | Runbook used | Outcome | TTR |
-|----------|--------|-------------|---------|-----|
-| C-1: API kill during schedule creation | `c1-api-kill.sh` | OPS-001 (rollback-deploy) | **PASS** | 4 min 12 s |
-| C-2: Executor kill mid-workflow | `c2-executor-kill.sh` | INC-003 (workflow-backlog) | **PASS** | 6 min 47 s |
-| C-3: Dispatcher kill mid-replay (100 events) | `c3-dispatcher-kill.sh` | INC-002 (dispatcher-stalled) | **PASS** | 3 min 55 s |
-
-All three scenarios passed. Two runbook gaps were identified and are tracked in
-§ Runbook Gap Findings. No data loss occurred in any scenario.
-
----
-
-## Scenario C-1 — API Kill During Schedule Creation
-
-**Script**: `scripts/chaos/c1-api-kill.sh`  
-**Runbook**: `docs/runbooks/rollback-deploy.md` (OPS-001)  
-**Chaos hypothesis**: Committing a schedule row then killing the API before the
-HTTP response reaches the client should leave the schedule in a consistent,
-persisted state. A client retry must surface `DuplicateScheduleError` — not a
-second row.
-
-### Injection
-
-```text
-09:03:00  Seeded committed row directly via psql (simulating committed-but-not-responded)
-09:03:05  Issued SIGKILL to vds-scheduler API process (PID 48712)
-09:03:05  API dead — health endpoint returning ECONNREFUSED
-09:03:10  Observation window: outbox tail via WebSocket showed no lost events
-09:05:12  Restarted API: fastapi dev scheduler_orchestrator/src/...
-09:05:20  API /health returned {"status":"ok","scheduler_enabled":true}
-09:05:30  Queried vds-cli scheduler schedules list → chaos-c1-<ts> present ✓
-09:06:05  Retry of original schedule create → DuplicateScheduleError (409) ✓
-09:07:12  Cleanup: deleted chaos schedule
-```
-
-### Observations
-
-- **Recovery**: API restart (7 s) + schedule visibility (10 s after restart). Total TTR = 4m 12s
-  (including deliberate 2 min observation window per game-day protocol).
-- **DBOS checkpoint**: DBOS.launch() re-registered all PENDING workflows on restart; no orphaned
-  workflows observed.
-- **Monitoring**: Grafana `vds_scheduler_scheduled_runs_total` flat during outage (correct — no
-  new triggers while API was down). Prometheus did not alert (outage < 1 min — below alert
-  threshold). **Gap found**: see §G-1.
-- **Log quality**: `Failed to insert: duplicate key value violates unique constraint
-  "schedules_pkey"` surfaced in structured logs; human-readable error was clear.
-- **Exit code**: `c1-api-kill.sh` → 0 (PASS).
-
----
-
-## Scenario C-2 — Executor Kill Mid-Workflow
-
-**Script**: `scripts/chaos/c2-executor-kill.sh`  
-**Runbook**: `docs/runbooks/incident-workflow-backlog.md` (INC-003)  
-**Chaos hypothesis**: Killing the executor process mid-way through a 5-step DBOS workflow must
-result in DBOS auto-resuming from the last durable checkpoint; steps 1–2 must NOT re-execute.
-Final workflow status must be SUCCESS.
-
-### Injection
-
-```text
-09:30:00  Enqueued long smoke workflow (5 DB-write steps, ~15s total)
-09:30:03  SIGKILL to executor process at step 2 (confirmed by db query: step 2 checkpoint written)
-09:30:03  API process survived (only executor thread killed via test harness isolation)
-09:30:04  Workflow status: PENDING (waiting for executor)
-09:33:10  Restarted API (full process restart to bring executor back online)
-09:33:15  DBOS.launch() output: "Resuming 1 workflow from checkpoint: step_3"
-09:35:17  Workflow status: SUCCESS ✓
-09:36:47  Audit: step 1 and 2 rows in test_step_trace NOT duplicated ✓
-```
-
-### Observations
-
-- **Recovery**: Restart (7 s) + workflow completion from step 3 (~120 s). Total TTR = 6 min 47 s.
-- **DBOS replay**: Confirmed no double-execution of steps 1–2 by querying test trace table.
-  Checkpoint granularity is per-step (correct per AC-2 acceptance criteria).
-- **Runbook INC-003**: The runbook's §Detection step (check `workflows list --status running`)
-  correctly identified the stuck PENDING workflow. §Resolution Step 2 (restart service) resolved it.
-  **Gap found**: see §G-2 — runbook lacks instruction for confirming checkpoint vs full restart.
-- **Alert gap**: No alert fired for PENDING > 2 min threshold. This is a known gap in
-  `vds_scheduler_workflow_failures_total` — only fires on ERROR/EXHAUSTED, not on
-  PENDING-stuck. Tracked as §G-2.
-- **Exit code**: `c2-executor-kill.sh` → 0 (PASS).
-
----
-
-## Scenario C-3 — Dispatcher Kill Mid-Replay
-
-**Script**: `scripts/chaos/c3-dispatcher-kill.sh`  
-**Runbook**: `docs/runbooks/incident-dispatcher-stalled.md` (INC-002)  
-**Chaos hypothesis**: Kill the dispatcher after it has acked exactly 50 of 100 seeded events.
-On restart, the dispatcher must resume from event 51 (no duplicates for 1–50; no missed events
-for 51–100). Deduplication is enforced by the two-phase claim+ack state machine in dispatcher.py.
-
-### Injection
-
-```text
-10:30:00  Seeded 100 events in sched.events_outbox (topic chaos.c3.<ts>.*)
-10:30:05  Registered test subscription (consumer_name chaos-c3-consumer-<ts>)
-10:30:10  Started dispatcher (via test harness, not full API restart)
-10:30:58  Dispatcher acked 50 events (confirmed via dispatch_ack_log query)
-10:31:00  SIGKILL to dispatcher coroutine (simulated by terminating test harness process)
-10:31:00  Grafana: outbox_backlog_seconds began climbing — 50 events still pending
-10:31:02  `vds-cli scheduler events tail` (snapshot): published_at=null for events 51–100 ✓
-10:31:30  Operator followed INC-002 §Detection: confirmed dispatcher not running
-10:32:00  Restarted API; DBOS.launch() restarted dispatcher
-10:33:05  Dispatcher replayed from event 51 (confirmed by dispatch_ack_log query)
-10:33:55  All 100 events acked; dispatch_ack_log shows no duplicate (subscription_id, event_id) pairs ✓
-10:34:05  Grafana: outbox_backlog_seconds = 0 ✓
-```
-
-### Observations
-
-- **Recovery**: Restart (7 s) + replay of 50 remaining events (~100 s). Total TTR = 3 min 55 s.
-- **Exactly-once guarantee**: No duplicate dispatch_ack_log rows for events 1–50. Exactly one
-  acked row per (subscription_id, event_id) pair — AC-3 confirmed.
-- **WebSocket live tail**: New `vds-cli events tail --follow` (TSK-11.9, landed this phase) showed
-  events 51–100 arriving in real time after restart — much better than the polling snapshot
-  which only showed lagging state. **Practical benefit validated in production-adjacent scenario.**
-- **Runbook INC-002**: All steps followed correctly. Runbook was accurate; §Detection timing
-  guidance ("lag should be < 5 s steady-state") matched observed behaviour.
-- **No alert fired** during the 60 s outage window — event_bus_lag_seconds alert threshold is
-  5 min (INC-002 §Alerting). Appropriate for this scenario. **Gap found**: see §G-3.
-- **Exit code**: `c3-dispatcher-kill.sh` → 0 (PASS).
-
----
-
-## Runbook Gap Findings
-
-### G-1 — Short API outage (< 60 s) not captured by any alert
-
-**Observed in**: Scenario C-1  
-**Gap**: The Prometheus availability alert fires only when `vds_scheduler_scheduled_runs_total`
-has no increments for > 5 min. A 7-second API restart (as in a rolling deploy) produces zero
-alert noise — which is the desired behaviour. However, the health-check probe scrape interval
-(30 s) means a 90 s total outage could go un-alerted for up to 30 s.  
-**Recommendation**: Add `probe_success{job="vds-scheduler-health"}` probe (Blackbox Exporter or
-equivalent) with 15 s scrape interval and `for: 45s` alert duration. Update INC-001
-(incident-postgres-unavailable) §Alerting section with the probe metric name.  
-**Severity**: Low (only cosmetic — short restarts during deploys should not page).  
-**Owner**: ops-dev (update INC-001 runbook + Grafana alert in Phase 3).
-
-### G-2 — INC-003 lacks guidance for "PENDING-stuck" vs "slow but progressing" distinction
-
-**Observed in**: Scenario C-2  
-**Gap**: `incident-workflow-backlog.md` §Resolution Step 2 says "restart service if workflows
-are stuck." It does not describe how to confirm a workflow is truly stuck (checkpoint not
-advancing) vs legitimately slow (long agent step). During C-2, the workflow appeared PENDING
-for 2 min, which could look identical to a legitimately running 2-min workflow.  
-**Recommendation**: Add a §Triage sub-step to INC-003: "Query `dbos.workflow_status` for
-`updated_at` — if `updated_at` has not moved for > step-timeout, the executor is stuck."
-Also recommend adding a Prometheus metric for `max(now() - updated_at) WHERE status='PENDING'`
-to give an alertable signal.  
-**Severity**: Medium — operators may unnecessarily restart a workflow that is legitimately running.  
-**Owner**: runbook-writer (update INC-003); chaos-dev (add updated_at assertion to c2-executor-kill.sh).
-
-### G-3 — No alert for dispatcher outage < 5 min
-
-**Observed in**: Scenario C-3  
-**Gap**: The 60 s dispatcher outage in C-3 produced no alert (threshold is 5 min). This is
-intentional (transient blips should not page). However, the current threshold of 5 min means
-a sustained dispatcher stall for 4 min 59 s produces no alert.  
-**Recommendation**: Add a separate, faster-firing SLO alert:
-`vds_scheduler_outbox_backlog_seconds > 120 for 2m` (P3 severity, Slack-only, no PagerDuty).
-This would have fired ~90 s into C-3, giving early warning without paging.  
-**Severity**: Low — the 5 min alert is still the canonical P2 boundary.  
-**Owner**: ops-dev (Grafana alert addition in Phase 3).
-
----
-
-## Game-Day Summary Table
-
-| # | Scenario | Script exit | Runbook followed | TTR | Gaps found |
-|---|----------|-------------|-----------------|-----|-----------|
-| 1 | C-1: API kill (schedule creation) | 0 (PASS) | OPS-001 §Phase 1 kill-switch + restart | 4m 12s | G-1 |
-| 2 | C-2: Executor kill mid-workflow | 0 (PASS) | INC-003 §Restart + verify resume | 6m 47s | G-2 |
-| 3 | C-3: Dispatcher kill mid-replay | 0 (PASS) | INC-002 §Restart + verify backlog drain | 3m 55s | G-3 |
-
-**Overall result**: PASS — all three scenarios recovered within SLO (RTO 15 min per design.md).  
-**Data loss**: None observed in any scenario.  
-**Exactly-once guarantee (AC-3)**: Confirmed — no duplicate dispatch_ack_log rows.  
-**DBOS replay (AC-2)**: Confirmed — steps not double-executed after executor kill.
-
----
-
-## Follow-Up Action Items
-
-| ID | Action | Owner | Phase |
-|----|--------|-------|-------|
-| FU-1 | Update INC-001: add Blackbox probe metric to §Alerting | ops-dev | Phase 3 |
-| FU-2 | Update INC-003: add PENDING-stuck triage step with `dbos.workflow_status.updated_at` | runbook-writer | Phase 3 |
-| FU-3 | Add `outbox_backlog_seconds > 120 for 2m` Slack-only alert to Grafana | ops-dev | Phase 3 |
-| FU-4 | Update c2-executor-kill.sh: add `updated_at` assertion to verify stuck vs slow | chaos-dev | Phase 3 |
-| FU-5 | Quarterly game-day cadence (TSK-N.11): schedule first repeat for Phase 3 cutover | ops-dev | Phase 3 kick-off |
-
----
-
-## Appendix: Environment Setup
-
-```bash
-# Staging environment used during game-day
-export VDS_SCHEDULER_ENABLED=true
-export VDS_SCHEDULER_DATABASE_URL="postgresql://vds_sched:***@localhost:5432/vds_sched_staging"
-export CHAOS_API_URL="http://localhost:8005"
-
-# Start staging API
-cd scheduler_orchestrator
-uv run fastapi dev src/vds_scheduler_orchestrator/app.py --port 8000
-
-# Apply migrations
-uv run alembic upgrade head
-
-# Run chaos scenarios
-./scripts/chaos/c1-api-kill.sh
-./scripts/chaos/c2-executor-kill.sh
-./scripts/chaos/c3-dispatcher-kill.sh
-```
-
-Postgres version: 17.2. Python: 3.13.3. DBOS: 2.19.x. asyncpg: 0.31.x.
-
----
-
-## Sign-Off
-
-| Role | Name | Date |
-|------|------|------|
-| Game-day lead | ops-dev | 2026-04-24 |
-| Chaos script review | chaos-dev | 2026-04-24 (async) |
-| Runbook review | runbook-writer | 2026-04-24 (async) |