@ngocsangairvds/vsaf 3.2.16 → 4.0.1

package/tools/vds-scripts/scheduler_orchestrator/design-notes/dedupe-design.md

@@ -1,324 +0,0 @@
-# Dedupe & Idempotency Design Review
-## TSK-N.6 — Pre-implementation gate for TSK-2.1, TSK-3.2, TSK-5.1
-
-**Author**: schedules-dev (Wave 2)  
-**Date**: 2026-04-24  
-**Status**: RESOLVED — all 5 items answered; no FR-12 wording changes required  
-**Blocks**: TSK-2.1 (schedule CRUD), TSK-3.2 (dispatcher), TSK-5.1 (chains)
-
----
-
-## Item 1 — Overlap Policy: Active-state Detection SQL
-
-### Semantics
-
-Three policies govern what happens when a new tick fires while a prior run for the
-same schedule is still active:
-
-| Policy | Meaning |
-|--------|---------|
-| `skip` | Do nothing if an active run exists; increment counter |
-| `queue` | Still enqueue the new run (queue concurrency limits rate) |
-| `cancel_previous` | Cancel the active run, then enqueue the new one |
-
-"Active" means the prior run could still execute steps — any state before terminal.
-
-### Canonical SQL (DBOS-native statuses — NOT VDS canonical names)
-
-Per design.md §DBOS Status Mapping and the WAVE-2-GUARDRAILS.md, the overlap
-detection query MUST use DBOS native status strings. VDS's operator-friendly names
-(`running`, `enqueued`, etc.) are a presentation mapping layer only; they MUST NOT
-appear in SQL against `dbos.workflow_status`.
-
-```sql
--- Active = may still execute. Uses DBOS native statuses per spec v2.0.4.
-SELECT ws.workflow_id, ws.status
-  FROM dbos.workflow_status ws
-  JOIN sched.schedules s
-    ON ws.queue_name = s.queue_name          -- broad scope
- WHERE s.schedule_name = $1
-   AND ws.status IN ('ENQUEUED', 'DELAYED', 'PENDING')
- ORDER BY ws.created_at DESC
- LIMIT 1;
-```
-
-**Implementation note**: Because `sched.workflow_runs` is a view (read-only JOIN of
-`dbos.workflow_status` with schedule metadata), the overlap check queries DBOS's own
-status table directly. The view exposes this for operators via `vds-cli workflows
-list`; the overlap policy logic reads from `dbos.workflow_status` directly at tick
-time to avoid view overhead on the hot path.
-
-**DBOS status semantics reminder** (from TSK-N.5 Check 8):
-- `PENDING` = actively executing (a worker has claimed this workflow). Counter-
-  intuitive but documented. Maps to VDS `running`.
-- `ENQUEUED` = queued, awaiting worker claim. Maps to VDS `enqueued`/`dispatched`.
-- `DELAYED` = scheduled with a future start time (one-shot / run-at). Maps to VDS
-  `dispatched`.
-
-### `cancel_previous` implementation
-
-1. Fetch active workflow_id via the SQL above.
-2. Call `DBOS.cancel_workflow(workflow_id)` — cooperative cancel; executing step
-   finishes; subsequent steps are skipped.
-3. Enqueue new run via `queue.enqueue(workflow_fn, context)`.
-
-IMPORTANT: `DBOS.cancel_workflow` does NOT guarantee synchronous termination. The
-overlap check for the NEXT tick must still query `dbos.workflow_status` — a `PENDING`
-row remains until the currently-executing step's function returns and DBOS marks it
-`CANCELLED`. The new run can coexist briefly with a `CANCELLING` run (there is no
-explicit `CANCELLING` status in DBOS; it goes directly to `CANCELLED` once the step
-acknowledges the signal).
-
----
-
-## Item 2 — Dispatch Ack Atomicity: Option B Selected
-
-### Decision: Option B (two-phase state machine)
-
-**TSK-N.5 Check 9 finding** (recorded in Wave 1): The Python DBOS SDK's
-`queue.enqueue()` does NOT support participating in an externally-managed
-SQLAlchemy/asyncpg transaction. The SDK manages its own Postgres connection
-internally. Therefore:
-
-- **Option A** (single-transaction INSERT + SDK enqueue) is NOT feasible in Phase 1.
-- **Option C** (`dbos.enqueue_workflow` PL/pgSQL UDF, Phase 2 optimization) is the
-  right long-term path per ADR-11, but deferred.
-- **Option B** is the Phase 1 choice.
-
-### Option B: Two-phase state machine on `sched.dispatch_ack_log`
-
-```
-State transitions for a single (subscription_id, event_id) pair:
-  [absent] → claimed → enqueued → acked
-                  ↘ enqueue_failed (retried by janitor)
-```
-
-**Step-by-step protocol** (per dispatcher on each inbound event):
-
-```python
-# Phase 1: claim (in a short Postgres txn)
-with session.begin():
-    inserted = session.execute(
-        INSERT INTO sched.dispatch_ack_log
-          (subscription_id, event_id, status, claimed_at)
-        VALUES ($sub_id, $event_id, 'claimed', now())
-        ON CONFLICT (subscription_id, event_id) DO NOTHING
-        RETURNING subscription_id
-    ).fetchone()
-if inserted is None:
-    return  # already claimed/acked by another dispatcher replica
-
-# Phase 2: DBOS enqueue (outside txn — SDK manages own connection)
-try:
-    handle = queue.enqueue(handler_fn, event)
-    workflow_id = handle.workflow_id
-except DBOSQueueDeduplicatedError:
-    # A prior idempotency_key collision — handler already enqueued.
-    # Look up prior workflow_id via dispatch_ack_log (GUARDRAIL 3: don't
-    # access e.workflow_id on the exception — it may not be set).
-    prior = session.execute(
-        SELECT workflow_id FROM sched.dispatch_ack_log
-         WHERE subscription_id = $sub_id AND event_id = $event_id
-           AND status = 'enqueued'
-    ).fetchone()
-    workflow_id = prior.workflow_id if prior else None
-
-# Phase 3: update row to enqueued/acked
-with session.begin():
-    session.execute(
-        UPDATE sched.dispatch_ack_log
-           SET status = 'acked', workflow_id = $workflow_id, acked_at = now()
-         WHERE subscription_id = $sub_id AND event_id = $event_id
-    )
-```
-
-**Janitor recovery**: A background task (fires every 60s) retries any row with
-`status = 'claimed'` and `claimed_at < now() - interval '30 seconds'` — these are
-crash survivors from between Phase 1 and Phase 2.
-
-**Why this is safe**: the `ON CONFLICT DO NOTHING` claim in Phase 1 is the
-authoritative dedup gate. Even if two dispatcher replicas race, only one wins the
-INSERT. The other sees zero rows returned and exits.
-
-**Option C note**: Once Phase 2 adopts the `dbos.enqueue_workflow` PL/pgSQL UDF
-(ADR-11), the hot path can replace Option B with a single atomic SQL call that
-both INSERTs the ack row and enqueues the DBOS workflow — zero separate phases.
-The dispatch_ack_log schema is compatible with both approaches.
-
----
-
-## Item 3 — Chain Fire Log Race Condition
-
-### Mechanism
-
-The `sched.chain_fire_log` table uses a structural dedup via its PK constraint:
-
-```sql
-INSERT INTO sched.chain_fire_log (chain_name, source_key, target_workflow_id, fired_at)
-VALUES ($chain_name, $source_key, $target_workflow_id, now())
-ON CONFLICT (chain_name, source_key) DO NOTHING
-RETURNING fired_at;
-```
-
-**Behavior**:
-- If `(chain_name, source_key)` is already in the table → zero rows returned.
-- Dispatcher reads the result: empty = already fired = skip enqueue.
-- Non-empty = first fire = proceed with `queue.enqueue(target_fn, input)`.
-
-**Race confirmation**: PostgreSQL's `ON CONFLICT ... DO NOTHING RETURNING` evaluates
-atomically within the statement. Two concurrent INSERT attempts for the same PK will
-have one succeed (returns a row) and one silently no-op (returns zero rows). This is
-confirmed behavior per PostgreSQL docs §INSERT ON CONFLICT. There is no window where
-both return a row for the same PK.
-
-**Single-transaction requirement**: the `INSERT ... RETURNING` and the
-`queue.enqueue()` call happen in sequence (Option B protocol above). The INSERT is
-in a short transaction; the enqueue is outside. If the process crashes between the
-INSERT commit and the enqueue call, the janitor loop retries (it detects
-`chain_fire_log` rows with no corresponding `dispatch_ack_log` acked entry). Chain
-fires are therefore at-least-once idempotent at the SQL level.
-
----
-
-## Item 4 — Idempotency Key / Deduplication ID Pass-Through
-
-### API surface
-
-The public `trigger_now()` and dispatcher `enqueue()` functions accept an optional
-`idempotency_key: str` parameter. How this maps to DBOS depends on the call site:
-
-#### 4a. Workflow enqueue path (schedule ticks, trigger_now, chain fires)
-
-```python
-from dbos import Queue, SetEnqueueOptions
-
-queue.enqueue(
-    workflow_fn,
-    *args,
-    options=SetEnqueueOptions(deduplication_id=idempotency_key),
-)
-# ↑ May raise dbos.error.DBOSQueueDeduplicatedError
-```
-
-When `DBOSQueueDeduplicatedError` is raised:
-- Do NOT access `e.workflow_id` or any attribute that assumes the error carries the
-  existing run's ID (WAVE-2-GUARDRAILS.md GUARDRAIL 3).
-- Instead, look up the existing workflow_id from our own `sched.dispatch_ack_log`
-  using `(subscription_id, event_id)` or `(schedule_name, idempotency_key)` as the
-  lookup key, depending on call site.
-- Return the existing handle as the idempotent response to the caller.
-
-#### 4b. Message send path (DBOS.send — NOT used by scheduler primitives)
-
-```python
-DBOS.send(destination_workflow_id, message, idempotency_key=key)
-```
-
-This is semantically different from workflow enqueue dedup:
-- `idempotency_key` here prevents the same message from being delivered twice to a
-  workflow's `DBOS.recv()` call.
-- `DBOSQueueDeduplicatedError` is NOT raised; instead DBOS silently drops the
-  duplicate message.
-- The scheduler primitives (schedules, event dispatcher, chains) do NOT use
-  `DBOS.send`; this path is only relevant to callers that implement custom message
-  passing between workflows.
-
-#### 4c. Which public APIs accept `idempotency_key`
-
-| API | Parameter accepted | Maps to DBOS primitive |
-|-----|--------------------|------------------------|
-| `schedules.trigger_now(name, idempotency_key=...)` | yes | `SetEnqueueOptions(deduplication_id=...)` on `queue.enqueue()` |
-| `dispatcher._enqueue_handler(sub, event, idempotency_key=...)` | yes | `SetEnqueueOptions(deduplication_id=...)` |
-| `chains._fire(chain, source_key, idempotency_key=...)` | yes | `SetEnqueueOptions(deduplication_id=...)` |
-| `bus.publish(topic, payload, idempotency_key=...)` | no — outbox row is deduplicated by `event_id` PK | N/A |
-
-#### 4d. Caller guidance when dedup error is raised
-
-```python
-try:
-    handle = queue.enqueue(fn, *args, options=SetEnqueueOptions(deduplication_id=key))
-except dboserror.DBOSQueueDeduplicatedError:
-    # Key is still active (ENQUEUED or PENDING in DBOS).
-    # Our own log table is the lookup source per GUARDRAIL 3.
-    existing_row = session.execute(
-        SELECT workflow_id FROM sched.dispatch_ack_log
-         WHERE idempotency_key = $key
-         ORDER BY claimed_at DESC LIMIT 1
-    ).fetchone()
-    workflow_id = existing_row.workflow_id if existing_row else None
-    return ExistingWorkflowResponse(workflow_id=workflow_id)
-```
-
----
-
-## Item 5 — Retention Safety: No Orphaned In-Flight Workflow References
-
-### Concern
-
-The `sched.dispatch_ack_log` and `sched.chain_fire_log` retention job (TSK-3.5)
-must NOT delete rows whose corresponding DBOS workflow is still in an active state.
-Deleting such rows would break the dedup guarantee — a retry arriving after the row
-is purged would re-enqueue an already-running workflow.
-
-### Required retention query
-
-```sql
--- Safe purge: delete ack log rows older than 30 days ONLY if the
--- corresponding DBOS workflow has reached a terminal state.
-DELETE FROM sched.dispatch_ack_log dal
- WHERE dal.acked_at < now() - ($1 * interval '1 day')
-   AND NOT EXISTS (
-       SELECT 1 FROM dbos.workflow_status ws
-        WHERE ws.workflow_id = dal.workflow_id
-          AND ws.status IN ('ENQUEUED', 'DELAYED', 'PENDING')
-   );
-```
-
-The same JOIN pattern applies to `sched.chain_fire_log`:
-
-```sql
-DELETE FROM sched.chain_fire_log cfl
- WHERE cfl.fired_at < now() - ($1 * interval '1 day')
-   AND NOT EXISTS (
-       SELECT 1 FROM dbos.workflow_status ws
-        WHERE ws.workflow_id = cfl.target_workflow_id
-          AND ws.status IN ('ENQUEUED', 'DELAYED', 'PENDING')
-   );
-```
-
-### Why this is safe
-
-- `dbos.workflow_status` is DBOS's own table, managed by DBOS.launch(). It is
-  authoritative for workflow lifecycle state.
-- The `IN ('ENQUEUED', 'DELAYED', 'PENDING')` predicate matches exactly the "active"
-  states used for overlap detection (Item 1 above) — consistent vocabulary.
-- A workflow that has reached `SUCCESS`, `ERROR`, `CANCELLED`, or
-  `MAX_RECOVERY_ATTEMPTS_EXCEEDED` will not restart without an explicit operator
-  `replay` command. Purging its ack row after the terminal state is safe.
-- DBOS's own retention for `dbos.workflow_status` rows (default 30 days per the
-  DBOS config) runs independently. If a DBOS row is purged before our ack row hits
-  retention, the `NOT EXISTS` subquery returns false (no matching row = no active
-  workflow), so the ack row WILL be deleted — correct behavior, since there's no
-  active workflow to protect.
-
-### Dedup-release timing note (GUARDRAIL 2)
-
-TSK-N.5 Check 10 confirmed that DBOS's dedup release timing on terminal state is
-inferred (not explicitly documented). However, our own `sched.dispatch_ack_log` is
-independent of DBOS's native dedup. The ack row is our authoritative dedup gate;
-DBOS's `deduplication_id` is a secondary defense. Even if DBOS's dedup ID is held
-past terminal state (permanent hold), our ack row drives the idempotency guarantee.
-
----
-
-## Resolution Summary
-
-| Item | Conclusion | Impact |
-|------|------------|--------|
-| 1 — Overlap SQL | Use `('ENQUEUED', 'DELAYED', 'PENDING')` in all active-state queries | TSK-2.1, TSK-2.2 |
-| 2 — Dispatch ack | Option B (two-phase state machine); Option C deferred to Phase 2 | TSK-3.2 |
-| 3 — Chain fire race | `ON CONFLICT DO NOTHING RETURNING` is atomically safe | TSK-5.1 |
-| 4 — Idempotency key | Three-way split: enqueue→`deduplication_id`, send→`idempotency_key`, outbox→PK | TSK-2.1, TSK-3.2 |
-| 5 — Retention safety | Purge only terminal-state rows via JOIN with `dbos.workflow_status` | TSK-3.5 |
-
-No FR-12 wording changes required. All ambiguities resolved before code is written.

package/tools/vds-scripts/scheduler_orchestrator/docs/Q3-2026-GAME-DAY-PREFLIGHT.md

@@ -1,217 +0,0 @@
-# Q3 2026 Chaos Game-Day — Pre-Flight Checklist
-
-**Game-day date**: Monday **2026-07-06** · 09:00–13:00 local
-**Type**: First post-Phase-3-cutover drill (first repeat after Celery→DBOS migration)
-**Scope**: C-4 + C-5 + C-6 mandatory; C-1 optional regression check if time permits
-**Owner**: _TBD — assign during 2-week lead-in_
-
----
-
-## Why This Game-Day Is Special
-
-Per `quarterly-game-day.md` §Scheduled Game-Days and Phase 2 debrief FU-5, the **Q3 2026 drill is the first full drill after the Celery→DBOS migration lands in production**. The executor topology has changed (Celery Beat container retired 2026-04-24; DBOS scheduled workflows now own the 6 memory-evolution cadences). Chaos scenarios executed pre-migration (Q2 2026, `game-day-phase2.md`) are not a substitute — we need fresh evidence that the new topology holds.
-
-The phase-2 drill validated the scheduler behind the scheduler kill-switch. The Q3 drill validates the scheduler **as the sole periodic executor**.
-
----
-
-## Two-Weeks-Before Checklist (due **2026-06-22**)
-
-- [ ] Calendar invite sent to: `#vds-scheduler` channel + platform-team-leads DL + infra-ops DL
-- [ ] Game-day lead assigned and confirmed (see `quarterly-game-day.md §Roles`)
-- [ ] Chaos script author on-call confirmed (async ok)
-- [ ] Runbook writer confirmed (active participant, not async)
-- [ ] Observer confirmed (silent — records timeline)
-- [ ] Infra/ops-dev confirmed (executes Patroni failover + network partition commands)
-- [ ] Staging environment reservation placed (no other maintenance windows that day)
-- [ ] Post-cutover health verified: all 6 migrated DBOS workflows are producing `SUCCESS` transitions on the staging host (query in §Post-Cutover Health Gate below)
-
----
-
-## One-Week-Before Checklist (due **2026-06-29**)
-
-### Scripts & runbooks still run
-
-Exercise each of C-4, C-5, C-6 against staging and verify `exit 0`:
-
-```bash
-cd scheduler_orchestrator
-bash scripts/chaos/c4-postgres-failover.sh
-# Expect: exit 0 within 15 min; Patroni failover completes; scheduler reconnects
-# If exit non-zero or TTR > 15 min: file gap BEFORE the game-day so the team arrives
-# with a shared understanding of the known issue
-
-bash scripts/chaos/c5-network-partition.sh
-# Expect: exit 0; scheduler enters degraded mode then recovers cleanly
-
-bash scripts/chaos/c6-outbox-storm.sh
-# Expect: exit 0; backlog clears within 2× dispatcher poll; shed ≤ 25%
-```
-
-**Host dependencies — verify present before dry-run** (found during 2026-04-25 dev-box pre-flight; C-6 calls `psql` 8 times for queue-depth and Postgres-connection probes):
-
-```bash
-which psql       || echo "INSTALL: brew install postgresql@16 (mac) or apt-get install postgresql-client (linux)"
-which python3    || echo "INSTALL: required for capture_pg_metrics.py + chaos script Python helpers"
-which docker     || echo "INSTALL: required for testcontainers-driven smoke tests + C-4 Patroni manipulation"
-which iptables   || echo "[C-5 only] required for network-partition simulation; on macOS use pfctl"
-# Patroni / etcd CLI tools (C-4 only): patronictl, etcdctl  — verify on the staging host
-```
-
-On the dev box used to validate v2.0.9, **C-6 could not be exercised because `psql` was not installed on the macOS host** — the chaos script exits with `[FAIL] psql not found` before any Postgres work. This is expected and not a regression: all chaos scripts assume the on-call host has libpq tools installed. **Production/staging hosts must satisfy these dependencies before the 2026-07-06 drill** — verify during this 1-week-before window, not the morning of.
-
-**Dry-run fail handling**: any dry-run failure is an immediate blocker — halt and fix before game-day. Running chaos against a broken scheduler produces no useful information.
-
-### Runbook cross-refs resolve
-
-The three scenarios map to runbooks per the chaos-runbook. Verify each runbook is reachable AND its cited SQL/commands still match live schema:
-
-```bash
-for rb in \
-  scheduler_orchestrator/docs/runbooks/dr-postgres-restore.md \
-  scheduler_orchestrator/docs/runbooks/incident-postgres-unavailable.md \
-  scheduler_orchestrator/docs/runbooks/incident-workflow-backlog.md \
-  scheduler_orchestrator/docs/runbooks/rollback-deploy.md; do
-    test -f "$rb" && echo "✅ $rb" || echo "❌ MISSING: $rb"
-done
-```
-
-Spot-check the top SQL snippet in each runbook against live staging schema (columns may have shifted between phases).
-
-### Dashboards still scrape
-
-```bash
-# Import scheduler_orchestrator/dashboards/*.json into staging Grafana
-ls scheduler_orchestrator/dashboards/
-# Verify Prometheus scrape on staging is hitting :8005/metrics and returning
-# vds_scheduler_tick_total, vds_scheduler_dispatch_total, vds_scheduler_rate_limit_total
-curl -sf http://staging-scheduler:8005/metrics | grep -c "^vds_scheduler_"
-# Expect: ≥ 15 metric series
-```
-
-### DBOS conductor (optional)
-
-If DBOS Conductor was enabled per ADR-10/13, confirm connection + dashboard access in staging so the game-day lead can watch workflow state transitions in real time. Not required — local `dbos.workflow_status` queries work too.
-
----
-
-## Post-Cutover Health Gate (run 1 week before, then again day-of)
-
-This gate validates that the post-retirement topology is stable enough to chaos-test. Run on staging:
-
-```bash
-# 1. All 7 migrated schedules ticking
-psql -U vds_sched -d vds_sched_staging -c "
-  SELECT
-    schedule_name,
-    extract(epoch FROM NOW() - last_run_at)::int AS secs_since_last_run
-  FROM sched.schedules
-  WHERE schedule_name IN (
-    'memory.evolution.dispatch','memory.evolution.sweep','memory.evolution.replay-dlq',
-    'mem0.resolve-pending-ids','mem0.repush-stalled','mem0.repush-local-only',
-    'evolution.auto-promote'
-  )
-  ORDER BY 1;
-"
-# Expect: each row's secs_since_last_run is well within its cadence
-# (dispatch/sweep/auto-promote: ≤ 120s;  replay-dlq: ≤ 1800s;  etc.)
-
-# 2. DBOS system healthy
-psql -U vds_sched -d vds_sched_staging -c "
-  SELECT status, count(*)
-  FROM dbos.workflow_status
-  WHERE updated_at > NOW() - INTERVAL '1 hour'
-  GROUP BY 1
-  ORDER BY 1;
-"
-# Expect: overwhelmingly SUCCESS; no MAX_RECOVERY_ATTEMPTS_EXCEEDED;
-# a handful of PENDING is ok (active workflows);
-# any ERROR rows > 0 — investigate before proceeding
-
-# 3. Outbox not backlogging
-psql -U vds_sched -d vds_sched_staging -c "
-  SELECT count(*) AS unpublished
-  FROM sched.events_outbox
-  WHERE published_at IS NULL;
-"
-# Expect: ≤ 100 rows at any instant; persistently growing = dispatcher gap
-```
-
-If any health gate fails: POSTPONE the game-day. Running chaos against a sick scheduler creates false negatives on recovery behavior.
-
----
-
-## Day-Of Checklist (2026-07-06)
-
-### 08:30 — Pre-start
-
-- [ ] All participants joined the call
-- [ ] Staging environment per `quarterly-game-day.md §Environment Setup` — green `/health`
-- [ ] Post-cutover health gate re-run (steps 1-3 above) — green
-- [ ] Timer and notes doc open (shared gdoc or markdown buffer)
-
-### 09:00–13:00 — Execute per agenda template
-
-Follow `quarterly-game-day.md §Agenda Template` — 30–35 min per scenario with 10 min buffer. Record:
-
-- Script start time (wall-clock)
-- Exit code
-- TTR (time-to-recovery to SLO-compliant state)
-- Runbook step-by-step adherence
-- Any deviation reasons
-- Residual artifacts check (`chaos-c*-*` schedules in `sched.schedules`, dangling test processes)
-
-### Acceptance (from `quarterly-game-day.md §Acceptance Criteria Per Scenario`)
-
-- **C-4**: Patroni failover completes; new primary accepts writes; scheduler reconnects without manual intervention; data loss = 0 rows
-- **C-5**: Scheduler enters degraded mode during partition; recovers fully when partition heals; no data loss
-- **C-6**: Outbox backlog clears within 2× dispatcher poll interval; shed ≤ 25%; exit 0
-- **All**: TTR ≤ 15 min
-
-### 13:00 — Debrief (commit within 48h)
-
-Create `scheduler_orchestrator/docs/game-day-q3-2026.md` using the debrief structure template in `quarterly-game-day.md §Debrief Structure`. Append a row to `§History`.
-
----
-
-## Post-Game-Day Follow-Up
-
-- [ ] Any FAIL re-run within 2 weeks (by **2026-07-20**)
-- [ ] Any gaps filed as GitHub issues with owner + target phase
-- [ ] Runbook fixes merged before Q4 game-day (**2026-10-05**)
-- [ ] Q4 scenarios confirmed in the Q3 debrief doc
-
----
-
-## Rollback / Abort
-
-If during execution any scenario triggers an **unplanned production-level incident** (e.g., chaos leaks past staging boundary, or post-recovery state does not match expectations within 2× TTR target):
-
-1. STOP the current scenario — `Ctrl-C` the chaos script; run its `_cleanup_fn` if one exists
-2. Follow `docs/runbooks/rollback-deploy.md` to restore pre-game-day state
-3. Page the on-call engineer
-4. File an incident report; game-day counts as PARTIAL or FAIL
-5. Do NOT run remaining scenarios — stable environment first
-
----
-
-## References
-
-- `quarterly-game-day.md` — cadence, roles, agenda, debrief template (this doc's parent)
-- `chaos-runbook.md` — per-scenario hypothesis + observe-expected tables
-- `game-day-phase2.md` — Q2 2026 debrief (reference for what a completed game-day looks like)
-- `DEPLOYMENT-RUNBOOK.md` — production cutover steps (how we got to the post-Celery topology we're testing)
-- `error-budget-policy.md` — burn-rate thresholds used in the §Post-Cutover Health Gate
-- `RETIREMENT-EXECUTED.md` — the 2026-04-24 cutover that makes this drill meaningful
-
----
-
-## Sign-off
-
-| Checkpoint | Date due | Responsible | Status |
-|---|---|---|---|
-| Two-weeks-before checklist complete | 2026-06-22 | game-day lead | ⬜ |
-| One-week-before checklist complete | 2026-06-29 | game-day lead + ops | ⬜ |
-| Post-cutover health gate — green | 2026-06-29 + 2026-07-06 | ops | ⬜ |
-| Day-of checklist complete | 2026-07-06 | all participants | ⬜ |
-| Debrief doc committed | 2026-07-08 (48h after) | game-day lead | ⬜ |