npm - @ngocsangairvds/vsaf - Versions diffs - 3.2.15 → 4.0.0 - Mend

@ngocsangairvds/vsaf 3.2.15 → 4.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (2337) hide show

package/tools/vds-scripts/scheduler_orchestrator/docs/quarterly-game-day.md DELETED Viewed

@@ -1,206 +0,0 @@
-# Quarterly Chaos Game-Day — Cadence and Template (TSK-N.11)
-**Spec ref**: `vds-scheduler/design.md §Chaos & Resilience Testing Plan FR-15.2`
-**Phase 2 debrief**: `docs/game-day-phase2.md`
-**Chaos runbook**: `docs/chaos-runbook.md`
-**Chaos scenarios**: `scripts/chaos/c1-api-kill.sh` … `c6-outbox-storm.sh`
----
-## Cadence
-| Parameter | Value |
-|-----------|-------|
-| **Frequency** | Once per quarter |
-| **Day** | First Monday of the first month of each quarter |
-| **Quarters** | Q1: January · Q2: April · Q3: July · Q4: October |
-| **Duration** | Half-day (09:00–13:00 local) — full-day if new scenarios are being validated |
-| **Notification lead time** | 2 weeks before drill date (calendar invite + Slack `#vds-scheduler` channel) |
-### Scheduled Game-Days
-| Quarter | Date | Status |
-|---------|------|--------|
-| Q2 2026 | 2026-04-06 (Monday) | ✅ Phase 2 — completed (`docs/game-day-phase2.md`) |
-| Q3 2026 | 2026-07-06 (Monday) | ⏳ Scheduled — first repeat post-Phase 3 cutover |
-| Q4 2026 | 2026-10-05 (Monday) | Planned |
-| Q1 2027 | 2027-01-04 (Monday) | Planned |
-> **Phase 3 cutover note** (from game-day-phase2.md FU-5): The Q3 2026 game-day
-> is the first post-Celery-migration drill. All 6 chaos scenarios should be executed
-> (not just 3) because the Celery-to-DBOS migration changes the executor topology.
----
-## Scope: Scenarios Per Game-Day
-Each game-day executes **at least 3 scenarios** selected from C-1 through C-6.
-The selection rotates to ensure all scenarios are exercised at least once per year.
-### Scenario Catalogue
-| ID | Script | What it tests | Default quarter |
-|----|--------|--------------|-----------------|
-| C-1 | `c1-api-kill.sh` | API crash after committed INSERT; duplicate guard | Q2 (April) |
-| C-2 | `c2-executor-kill.sh` | DBOS executor kill mid-workflow; checkpoint replay | Q2 (April) |
-| C-3 | `c3-dispatcher-kill.sh` | Dispatcher kill mid-replay; exactly-once dedup | Q2 (April) |
-| C-4 | `c4-postgres-failover.sh` | Postgres primary failover (DR-A path) | Q3 (July) |
-| C-5 | `c5-network-partition.sh` | Network partition between API and Postgres | Q3 (July) |
-| C-6 | `c6-outbox-storm.sh` | Outbox event storm; dispatcher shed and recovery | Q4 (October) |
-### Rotation Schedule
-| Quarter | Mandatory scenarios | Optional (if time permits) |
-|---------|--------------------|-----------------------------|
-| Q2 (April) | C-1, C-2, C-3 | C-4 (first-Monday timing allows full-day) |
-| Q3 (July) | C-4, C-5, C-6 | C-1 (regression check after Phase 3 cutover) |
-| Q4 (October) | C-3, C-5, C-6 | C-2 |
-| Q1 (January) | C-1, C-2, C-4 | C-3, C-5 |
-> Game-day lead may deviate from the rotation if a specific scenario was
-> implicated in a recent incident — that scenario takes priority.
----
-## Roles
-| Role | Responsibility |
-|------|---------------|
-| **Game-day lead** | Owns agenda, declares pass/fail, files post-game debrief |
-| **Chaos script author** | On-call async; answers questions about script internals |
-| **Runbook writer** | Verifies runbook steps are followed; records gaps |
-| **Infra / ops-dev** | Executes infrastructure commands (Patroni failover, network partition) |
-| **Observer** | Silent participant — records timeline and command output |
----
-## Agenda Template
-```
-09:00  Kick-off — confirm environment, roles, and scenario list for today
-09:15  Scenario 1 — inject + observe + runbook execution + TTR measurement
-09:45  Debrief Scenario 1 — gaps, findings, immediate remediation notes
-10:00  [Break]
-10:15  Scenario 2 — inject + observe + runbook execution
-10:50  Debrief Scenario 2
-11:05  Scenario 3 — inject + observe + runbook execution
-11:45  Debrief Scenario 3
-12:00  Joint debrief — aggregate findings, assign owners, draft follow-up tasks
-12:30  Sign-off — lead declares overall result; commit debrief doc to repo
-13:00  End
-```
-*Adjust slots: 30–35 min per scenario (inject 5 min + recover to SLO + 10 min buffer)*
-*Full-day variant: add Scenario 4 at 13:30 with 45 min slot.*
----
-## Environment Setup
-```bash
-# Staging environment (VDS_SCHEDULER_ENABLED=true required)
-export VDS_SCHEDULER_ENABLED=true
-export VDS_SCHEDULER_DATABASE_URL="postgresql://vds_sched:***@localhost:5432/vds_sched_staging"
-export CHAOS_API_URL="http://localhost:8005"
-# Start staging API
-cd scheduler_orchestrator
-uv run fastapi dev src/vds_scheduler_orchestrator/app.py --port 8000
-# Apply migrations (confirm at head)
-uv run alembic current
-uv run alembic upgrade head
-# Verify healthy before first scenario
-curl -sf "$CHAOS_API_URL/health" | python3 -m json.tool
-# Expected: {"status":"ok","scheduler_enabled":true}
-```
----
-## Scenario Execution Pattern
-For each scenario:
-1. **State the hypothesis** (from design.md §Chaos & Resilience Testing Plan).
-2. **Run the script** — record exact start time.
-3. **Follow the runbook** — ops leads execution; observer records commands and timestamps.
-4. **Measure TTR** — time from injection to system back to SLO-compliant state.
-5. **Exit code check** — script exit 0 = PASS, exit 1 = FAIL.
-6. **Cleanup** — confirm no leftover chaos artifacts (`chaos-c*-*` schedules, dangling processes).
-7. **Fill in the execution table** (see §Debrief Structure below).
----
-## Acceptance Criteria Per Scenario
-| Scenario | Pass condition |
-|----------|---------------|
-| C-1 | Schedule persists after API crash; retry returns DuplicateScheduleError; exit 0 |
-| C-2 | Workflow resumes from correct DBOS checkpoint; no double-execution of completed steps; final status SUCCESS; exit 0 |
-| C-3 | Dispatcher replays from first unacked event (no missed, no duplicate); `dispatch_ack_log` has exactly one row per (subscription_id, event_id); exit 0 |
-| C-4 | Patroni failover completes; new primary accepts writes; scheduler reconnects without manual intervention; data loss = 0 rows |
-| C-5 | Scheduler enters degraded mode during partition; recovers fully when partition heals; no data loss |
-| C-6 | Outbox backlog clears within 2× dispatcher poll interval after storm ends; shed 429 rate ≤ 25%; exit 0 |
-| **All** | TTR ≤ 900 s (15 min) per design.md RTO target |
----
-## Debrief Structure
-### Execution Table (fill in per game-day)
-| # | Scenario | Script exit | Runbook followed | TTR | Gaps found |
-|---|----------|-------------|-----------------|-----|-----------|
-| 1 | `<scenario>` | `<0=PASS/1=FAIL>` | `<runbook ID + section>` | `<Xm Ys>` | `<gap IDs or "none">` |
-| 2 | | | | | |
-| 3 | | | | | |
-### Gap Findings Table (fill in per gap)
-| ID | Scenario | Gap description | Severity | Recommendation | Owner | Target phase |
-|----|----------|----------------|----------|----------------|-------|-------------|
-| G-1 | | | Low/Medium/High | | | |
-### Overall Result
-| Field | Value |
-|-------|-------|
-| **Overall result** | `<PASS / FAIL / PARTIAL>` |
-| **Data loss in any scenario** | `<None / N rows>` |
-| **Exactly-once guarantee (C-3)** | `<Confirmed / Violation detected>` |
-| **DBOS replay (C-2)** | `<Confirmed / Double-execution detected>` |
-| **Worst TTR** | `<Xm Ys for scenario C-N>` |
-| **All TTRs within 15 min SLO** | `<Yes / No>` |
-| **Gaps found** | `<count>` |
----
-## Follow-Up Process
-1. Game-day lead creates GitHub issues for each gap (or adds to the next phase's task list).
-2. Runbook gaps are fixed before the next game-day (owners assigned in the gap table above).
-3. If any scenario FAILED, re-run that single scenario within 2 weeks of the game-day.
-4. Debrief doc is committed to the repo within 48 hours of the game-day.
-5. Next game-day date is confirmed in the debrief doc before sign-off.
----
-## Sign-Off
-| Role | Name | Date |
-|------|------|------|
-| Game-day lead | `<fill in>` | `<fill in>` |
-| Chaos script review | `<fill in>` | `<fill in (async ok)>` |
-| Runbook review | `<fill in>` | `<fill in (async ok)>` |
----
-## History
-| Quarter | Date | Scenarios | Overall | Debrief doc |
-|---------|------|-----------|---------|-------------|
-| Q2 2026 | 2026-04-24 | C-1, C-2, C-3 | ✅ PASS | `docs/game-day-phase2.md` |
-| Q3 2026 | 2026-07-06 | C-4, C-5, C-6 | *(pending)* | *(to be created)* |
-*(Append a row after each game-day.)*

package/tools/vds-scripts/scheduler_orchestrator/docs/runbooks/dr-postgres-restore.md DELETED Viewed

@@ -1,268 +0,0 @@
-# DR: Postgres Restore (Including Point-in-Time Recovery)
-**Runbook ID:** DR-001
-**Severity:** P0 — Full platform data loss or Postgres corruption.
-**RTO target:** 60 minutes (per design.md C-4 target; full PITR may take longer — see per-phase estimates).
-**Last updated:** 2026-04-24
----
-## Scope
-This runbook covers two scenarios:
-| Scenario | Description |
-|---|---|
-| **DR-A: Primary failure with intact replica** | Postgres primary is dead; streaming replica is healthy. Failover in < 60 s via Patroni. |
-| **DR-B: Full data loss / corruption** | Both primary and replica are gone, or data is corrupted. Restore from base backup + WAL replay (PITR). |
-The VDS scheduler stores all durable state in Postgres:
-- `sched.*` tables (7 tables: schedules, events_outbox, subscriptions, chain_definitions, workflow_registry, dispatch_ack_log, chain_fire_log)
-- `dbos.*` system tables (workflow_status, workflow_inputs, workflow_queue, etc.)
-- LangGraph checkpoint tables (created by `AsyncPostgresSaver.setup()` at startup)
----
-## Pre-Recovery Checklist
-- [ ] Kill-switch the scheduler immediately: `VDS_SCHEDULER_ENABLED=false` + `docker compose restart scheduler`
-- [ ] Alert the VDS infra team and security lead (potential data loss event).
-- [ ] Identify the target recovery timestamp (the last known-good point before corruption/loss).
-- [ ] Locate the most recent base backup and WAL archive in the backup store.
-- [ ] Confirm the backup store is accessible from the restore target host.
-- [ ] Record current `DBOS_APP_VERSION` from `/health` endpoint (needed to re-confirm version after restore).
----
-## Scenario DR-A: Primary Failure with Intact Replica
-### Step 1: Verify replica is healthy and up-to-date
-```bash
-# On the replica host
-pg_controldata $PGDATA | grep -E "state|checkpoint"
-# Expected: "Database cluster state: in archive recovery" or "standby mode"
-# Check replication lag (run on the replica)
-psql -c "SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;"
-# Acceptable: < 60 s
-```
-### Step 2: Trigger Patroni failover (or manual promotion)
-```bash
-# Patroni-managed cluster (preferred)
-patronictl -c /etc/patroni/config.yml failover <cluster_name> --master <old_primary> --candidate <replica_name> --force
-# Confirm the new primary
-patronictl -c /etc/patroni/config.yml topology
-```
-If Patroni is not available, promote the replica manually:
-```bash
-# On the replica host
-touch $PGDATA/promote_trigger  # or use pg_ctl
-# pg_ctl promote -D $PGDATA
-```
-### Step 3: Update VDS_SCHEDULER_DATABASE_URL to point at the new primary
-```bash
-# In ~/.vds/.env
-VDS_SCHEDULER_DATABASE_URL="postgresql://vds_scheduler:<password>@<new_primary_host>:5432/vds_scheduler"
-# Re-enable and restart
-VDS_SCHEDULER_ENABLED=true
-docker compose restart scheduler
-# Confirm connectivity
-vds-cli scheduler schedules list --format table
-curl -sf http://localhost:8005/health | python3 -m json.tool
-```
-### Step 4: Verify no data loss
-```bash
-# Compare schedule count against last known good value
-psql "$VDS_SCHEDULER_DATABASE_URL" -c "SELECT count(*) FROM sched.schedules;"
-psql "$VDS_SCHEDULER_DATABASE_URL" -c "SELECT count(*) FROM sched.events_outbox WHERE published_at IS NULL;"
-```
----
-## Scenario DR-B: Full Restore from Base Backup + PITR
-### Step 1: Stop all services that write to Postgres
-```bash
-# Kill-switch the scheduler
-VDS_SCHEDULER_ENABLED=false
-docker compose stop scheduler
-# Stop all other VDS services writing to the database
-docker compose stop  # or stop individual services
-```
-### Step 2: Provision a fresh Postgres instance
-```bash
-# On the restore host, initialize a new Postgres cluster
-initdb -D /var/lib/postgresql/data
-# Confirm Postgres version matches the backup's version
-psql --version
-# Must match the pg_dump / WAL archive version
-```
-### Step 3: Restore the base backup
-```bash
-# Example using pg_basebackup backup stored in the VDS backup store
-# (replace with your actual backup storage path)
-aws s3 sync s3://vds-pg-backups/<cluster_name>/base/<latest_backup>/ /var/lib/postgresql/data/
-# or: rsync from NFS mount
-rsync -az <backup_host>:/backups/pg/base/<latest_backup>/ /var/lib/postgresql/data/
-# Set ownership
-chown -R postgres:postgres /var/lib/postgresql/data
-chmod 700 /var/lib/postgresql/data
-```
-### Step 4: Configure Point-in-Time Recovery (PITR)
-Create `$PGDATA/recovery.conf` (Postgres < 12) or `postgresql.conf` + `recovery.signal` (Postgres ≥ 12):
-**Postgres ≥ 12 (recommended):**
-```bash
-# Create recovery signal file
-touch /var/lib/postgresql/data/recovery.signal
-# Add to postgresql.conf:
-cat >> /var/lib/postgresql/data/postgresql.conf << 'EOF'
-restore_command = 'aws s3 cp s3://vds-pg-backups/<cluster_name>/wal/%f %p'
-# Or: restore_command = 'cp /mnt/wal_archive/%f %p'
-# --- Target recovery time (use the last known-good timestamp) ---
-recovery_target_time = '2026-04-24 14:30:00 UTC'  # REPLACE with actual target
-recovery_target_action = 'promote'
-# Optional: stop at a specific transaction ID instead of time
-# recovery_target_xid = '<xid>'
-EOF
-```
-**Postgres < 12:**
-```bash
-cat > /var/lib/postgresql/data/recovery.conf << 'EOF'
-restore_command = 'cp /mnt/wal_archive/%f %p'
-recovery_target_time = '2026-04-24 14:30:00 UTC'  # REPLACE
-recovery_target_action = 'promote'
-EOF
-```
-### Step 5: Start Postgres and monitor WAL replay
-```bash
-pg_ctl start -D /var/lib/postgresql/data -l /var/log/postgresql/recovery.log
-# Tail the log to watch PITR progress
-tail -f /var/log/postgresql/recovery.log
-# Expected lines:
-#   LOG:  starting point-in-time recovery to ...
-#   LOG:  restored log file "0000000100000000000000XY" from archive
-#   LOG:  recovery stopping before commit of transaction ...
-#   LOG:  pausing at the end of recovery
-#   LOG:  database system is ready to accept read only connections
-```
-### Step 6: Verify PITR stopped at the correct point
-```bash
-psql -h localhost -U postgres -c "SELECT now(), pg_last_xact_replay_timestamp();"
-# pg_last_xact_replay_timestamp should match your recovery_target_time (± a few seconds)
-```
-### Step 7: Promote the restored instance to read-write
-```bash
-# If recovery_target_action = 'promote' was set, Postgres promotes automatically.
-# If not:
-pg_ctl promote -D /var/lib/postgresql/data
-```
-### Step 8: Verify sched.* and dbos.* tables are intact
-```sql
--- Check sched schema
-SELECT table_name, pg_size_pretty(pg_total_relation_size(quote_ident(table_name)::regclass)) AS size
-  FROM information_schema.tables
- WHERE table_schema = 'sched'
- ORDER BY table_name;
--- Expected: 7 tables (schedules, events_outbox, subscriptions, chain_definitions,
---           workflow_registry, dispatch_ack_log, chain_fire_log)
--- Check row counts for sanity
-SELECT 'schedules' AS t, count(*) FROM sched.schedules
-UNION ALL
-SELECT 'events_outbox', count(*) FROM sched.events_outbox
-UNION ALL
-SELECT 'subscriptions', count(*) FROM sched.subscriptions
-UNION ALL
-SELECT 'workflow_registry', count(*) FROM sched.workflow_registry;
-```
-### Step 9: Run Alembic to confirm schema is up to date
-```bash
-cd scheduler_orchestrator
-uv run alembic current
-# Expected: head (or the latest revision hash)
-# If behind head, run migrations:
-uv run alembic upgrade head
-```
-### Step 10: Reconnect the scheduler and verify
-```bash
-VDS_SCHEDULER_DATABASE_URL="postgresql://vds_scheduler:<password>@<restore_host>:5432/vds_scheduler"
-VDS_SCHEDULER_ENABLED=true
-docker compose restart scheduler
-# Health check
-curl -sf http://localhost:8005/health | python3 -m json.tool
-# Schedules present
-vds-cli scheduler schedules list --format table
-# Smoke test
-curl -sf -X POST http://localhost:8005/test/enqueue | python3 -m json.tool
-# Poll status/<id> within 5 s → Expected: SUCCESS
-```
----
-## WAL Archive Retention and Management
-The `VDS_SCHEDULER_OUTBOX_RETENTION_DAYS` (default 90) controls only event row purging, not WAL archive retention. WAL archive management is an infrastructure-level concern:
-```bash
-# Example: list WAL archive segments
-aws s3 ls s3://vds-pg-backups/<cluster_name>/wal/ | tail -20
-# Retention policy: keep WAL archives for at least max(90 days, recovery window SLA)
-# Never delete WAL archives that post-date your oldest base backup
-```
-> **Security note:** If a security incident required a credential purge (see `incident-secret-leaked-in-event.md`), WAL archives contain the credential in any transaction committed during the exposure window. Coordinate a WAL purge with the infra team after PITR is no longer needed for that window.
----
-## Post-DR
-- Confirm the Alembic revision at `head`.
-- Run the game-day verification checklist (all 6 chaos scenarios from design.md §Chaos & Resilience Testing Plan) to confirm the restored system behaves correctly.
-- Measure and record actual RTO; compare to the 60 s SLO for DR-A and the documented estimate for DR-B.
-- Update `VDS_SCHEDULER_DATABASE_URL` in all relevant config stores (Docker secrets, Consul, Kubernetes secrets) to the new host.
-- Set up streaming replication from the restored primary to a new replica before declaring DR complete.
-- File a postmortem (use `postmortem-template.md`) documenting the data loss window, root cause, and recovery timeline.

package/tools/vds-scripts/scheduler_orchestrator/docs/runbooks/evolution-auto-promote-runbook.md DELETED Viewed

@@ -1,194 +0,0 @@
-# Runbook: evolution.auto-promote Schedule Operations
-**TSK-13.4** — Operator runbook for day-two operations on the
-`evolution.auto-promote` DBOS cron schedule introduced in TSK-13.1.
-Audience: on-call engineers, release operators.
----
-## Quick Reference
-| Item | Value |
-|---|---|
-| Schedule name | `evolution-auto-promote` |
-| Alias | `evolution.auto-promote` |
-| Cron | `*/1 * * * *` (every minute) |
-| Queue | `ops-default` |
-| Overlap policy | `skip` |
-| Workflow FQN | `vds_scheduler_orchestrator.workflows.evolution.auto_promote.evolution_auto_promote_workflow` |
-| Owner orchestrator | `evolution_orchestrator` |
-| Kill-switch | `VDS_SCHEDULER_ENABLED=false` |
----
-## Common Operations
-All commands use the worktree-safe wrapper. From the repo root:
-```bash
-WUSH="./scripts/worktree_uv.sh run --project vds_cli vds-cli"
-```
-### Check schedule status
-```bash
-$WUSH scheduler schedules show evolution-auto-promote
-```
-Expected output includes `status: active`, `cron: */1 * * * *`,
-`last_run_at` within the last 90 seconds.
-### List recent workflow runs
-```bash
-$WUSH scheduler workflows list | grep evolution
-```
-### View a specific workflow run
-```bash
-$WUSH scheduler workflows show <workflow_id>
-```
-### Trigger an immediate (out-of-cycle) tick
-```bash
-$WUSH scheduler schedules trigger-now evolution-auto-promote
-```
-This enqueues one tick on `ops-default` immediately.
-Check completion via `scheduler workflows list --status SUCCESS`.
-### Pause the schedule
-Use when investigating a recurring failure or during planned maintenance:
-```bash
-$WUSH scheduler schedules pause evolution-auto-promote
-```
-The OS daemon (if still running during the soak period) continues to
-operate unaffected. After DBOS is paused, no new ticks fire.
-### Resume the schedule
-```bash
-$WUSH scheduler schedules resume evolution-auto-promote
-```
-### Force-promote a specific verdict (bypass scheduler)
-```bash
-$WUSH evolution promote --verdict /path/to/verdict-<id>.json
-```
-Use when a verdict is stuck and the next scheduled tick is more than 60
-seconds away.
----
-## Failure Scenarios
-### Scenario 1: `last_run_at` is stale (> 5 minutes)
-**Symptoms**: Schedule shows `last_run_at` > 5 minutes ago;
-Prometheus counter `evolution_auto_promote_ticks_total` not incrementing.
-**Investigation**:
-```bash
-# Check for ERROR-status workflows
-$WUSH scheduler workflows list --status ERROR | grep evolution
-# Check DBOS system logs
-docker logs vds-scheduler-orchestrator --tail 50 2>&1 | grep evolution
-# Check if VDS_SCHEDULER_ENABLED is still true
-$WUSH scheduler schedules list | grep evolution
-```
-**Resolution**:
-1. If vds_evolution is not importable in the scheduler's venv, install it:
-   `uv pip install vds-evolution` inside the scheduler container.
-2. If the DBOS worker is dead, restart the scheduler container.
-3. If the schedule is paused accidentally, run `schedule resume` (above).
----
-### Scenario 2: Verdict stuck as unpromoted despite DBOS ticks succeeding
-**Symptoms**: DBOS shows `promoted=0` on every tick; a verdict file exists
-without a companion `*-promoted.json` file.
-**Investigation**:
-```bash
-# Check for unpromoted verdicts
-find ~/.vds/evolution-log -name 'verdict-*.json' | while read vf; do
-  promoted="${vf%.json}-promoted.json"
-  [ ! -f "$promoted" ] && echo "UNPROMOTED: $vf"
-done
-# Run promote in dry-run mode
-$WUSH evolution auto-promote --dry-run --poll-interval 1
-```
-**Resolution**:
-1. Run `vds-cli evolution promote --verdict <path>` to manually promote.
-2. Check if the verdict file is malformed JSON (`cat <verdict-path> | python3 -m json.tool`).
-3. If `vds-evolution promote` fails, check DBOS step error logs for
-   `subprocess.CalledProcessError` — the `vds-evolution` CLI may be absent.
----
-### Scenario 3: Double-promotion during parallel-run soak
-**Symptoms**: A verdict has both been promoted by the OS daemon AND the DBOS
-schedule; the `*-promoted.json` companion file exists but promote was called twice.
-**Impact**: Generally benign — `promote` is idempotent (re-running it on an
-already-promoted verdict is a no-op or a harmless re-apply).
-**Resolution**:
-1. Confirm idempotency by re-running `vds-cli evolution promote --verdict <path>`
-   on the affected file and checking the output.
-2. If the double-promotion caused side effects, consult the Evolution loop
-   operator guide in `vds_evolution/AGENTS.md`.
----
-### Scenario 4: DBOS step max retries exceeded
-**Symptoms**: Workflow shows `status=MAX_RECOVERY_ATTEMPTS_EXCEEDED`.
-**Resolution**:
-1. Inspect the error: `$WUSH scheduler workflows show <workflow_id>`.
-2. Fix the underlying issue (importable `vds_evolution`, reachable verdict root).
-3. Manually trigger a fresh tick: `$WUSH scheduler schedules trigger-now evolution-auto-promote`.
-4. DBOS does not automatically restart `MAX_RECOVERY_ATTEMPTS_EXCEEDED` workflows —
-   only new ticks (triggered by cron or `trigger-now`) create new workflow instances.
----
-## Metrics
-The following Prometheus metrics are exported by the scheduler orchestrator:
-| Metric | Description |
-|---|---|
-| `dbos_workflow_executions_total{name="evolution_auto_promote_workflow"}` | Total ticks fired |
-| `dbos_workflow_duration_seconds{name="evolution_auto_promote_workflow"}` | Per-tick execution time |
-| `dbos_step_executions_total{name="_run_auto_promote_once"}` | Step-level invocations (incl. retries) |
-Access via `GET http://localhost:<METRICS_PORT>/metrics` on the scheduler container.
----
-## References
-- TSK-13.1: Workflow source (`workflows/evolution/auto_promote.py`)
-- TSK-13.2: Retirement procedure (`docs/evolution-daemon-retirement.md`)
-- TSK-13.3: File manifest (`docs/evolution-retirement-files.md`)
-- `AGENTS.md §Scheduler Architecture` — Two-scheduler operator note
-- `docs/chaos-runbook.md` — General chaos engineering procedures