@ngocsangairvds/vsaf 3.2.15 → 4.0.0

package/tools/vds-scripts/scheduler_orchestrator/docs/runbooks/incident-dispatcher-stalled.md

@@ -1,188 +0,0 @@
-# Incident: Dispatcher Stalled
-
-**Runbook ID:** INC-002  
-**Severity:** P2 — Event bus stops delivering; published events accumulate in `sched.events_outbox` without being dispatched to subscribers.  
-**Last updated:** 2026-04-24  
-
----
-
-## Symptoms
-
-- Grafana: `vds_scheduler_event_bus_lag_seconds{topic="*"}` grows continuously (should be < 5 s steady-state).
-- Grafana: `vds_scheduler_outbox_backlog_seconds{subscription_id="*"}` rising — oldest unpublished event getting older by the minute.
-- `vds-cli scheduler events tail` shows events with `published_at = null` that are many minutes old.
-- Downstream workflows that depend on chain triggers (`sched.chain_definitions`) are not firing; chain_fires_total is flat.
-- Application logs: `dispatcher: LISTEN sched_events active` is absent (meaning the LISTEN connection is not established), or repeated `asyncpg exceptions` during the LISTEN loop.
-- `sched.dispatch_ack_log` has rows with `status = 'claimed'` and `claimed_at` > 5 minutes ago (crash survivors the janitor has not yet cleaned).
-
----
-
-## Quick Diagnosis (≤ 5 min)
-
-**Step 1 — Tail the outbox for stuck events:**
-```bash
-vds-cli scheduler events tail --since 30m --limit 100 --format table
-# Look for rows where "Published At" column shows "pending" (published_at IS NULL)
-# If > 0 such rows older than 5 minutes → dispatcher is stalled
-```
-
-**Step 2 — Count stuck claimed rows in dispatch_ack_log:**
-```sql
-SELECT count(*), min(claimed_at) AS oldest_stuck
-  FROM sched.dispatch_ack_log
- WHERE status = 'claimed'
-   AND claimed_at < now() - interval '30 seconds';
--- > 0 rows = janitor hasn't cleaned up or dispatcher crashed between claim and enqueue
-```
-
-**Step 3 — Check if the LISTEN loop is alive (look for the startup log line):**
-```bash
-docker compose logs --tail=200 scheduler | grep -E "dispatcher|LISTEN|sched_events"
-# Healthy:    dispatcher: LISTEN sched_events active
-# Stalled:    No such line, or repeated connection errors
-```
-
-**Step 4 — Check for pg_notify channel activity from Postgres:**
-```sql
--- Run on the Postgres primary; shows recent notify activity
-SELECT pid, query_start, query
-  FROM pg_stat_activity
- WHERE query ILIKE '%sched_events%' OR query ILIKE '%LISTEN%'
- ORDER BY query_start DESC
- LIMIT 10;
-```
-> **Interpretation:** A healthy dispatcher has exactly one row here with `LISTEN sched_events`. Zero rows confirm the LISTEN connection is gone.
-
-**Step 5 — Check for stuck subscription paused state:**
-```sql
-SELECT subscription_id, consumer_name, topic_pattern, paused
-  FROM sched.subscriptions
- WHERE paused = true;
--- Paused subscriptions will not receive dispatches; confirm intentional vs accidental
-```
-
----
-
-## Mitigation (≤ 30 min)
-
-### M-1: Restart the scheduler service (fastest recovery)
-
-The dispatcher's `_listen_loop` runs as a background asyncio task inside the scheduler process. A clean restart re-establishes the LISTEN connection and automatically replays any outbox events missed since `last_acked_event_id` — the replay is at-least-once, deduplicated by `ON CONFLICT DO NOTHING` in `sched.dispatch_ack_log`.
-
-```bash
-docker compose restart scheduler
-# or: systemctl restart vds-scheduler
-
-# Confirm LISTEN loop is re-established:
-docker compose logs --tail=50 scheduler | grep "dispatcher: LISTEN"
-# Expected:  dispatcher: LISTEN sched_events active
-```
-> **Rollback M-1:** If the restart makes things worse (service fails to start), revert to `VDS_SCHEDULER_ENABLED=false` (see `incident-postgres-unavailable.md` M-1).
-
-### M-2: Manually trigger janitor cleanup if many stuck 'claimed' rows remain
-
-The janitor runs every 60 s automatically, but if the process is still alive and the janitor is also stalled, reset stuck rows directly so the next replay pass can re-dispatch:
-
-```sql
--- Reset stuck 'claimed' rows > 5 minutes old
--- (safe: dispatcher will re-claim and re-dispatch them on next notify or replay)
-DELETE FROM sched.dispatch_ack_log
- WHERE status = 'claimed'
-   AND claimed_at < now() - interval '5 minutes';
-```
-> **Rollback M-2:** Rows deleted; the dispatcher re-inserts them on next dispatch attempt. This is idempotent.
-
-### M-3: If a specific subscription is wedged, pause and resume it
-
-This forces a checkpoint reset, causing the dispatcher to replay from the last successfully acked event:
-
-```sql
--- Pause the subscription (stops it receiving new dispatches temporarily)
-UPDATE sched.subscriptions
-   SET paused = true
- WHERE consumer_name = '<wedged_consumer_name>';
-
--- Restart dispatcher (M-1 above)
-
--- Resume the subscription after restart confirms LISTEN is active
-UPDATE sched.subscriptions
-   SET paused = false
- WHERE consumer_name = '<wedged_consumer_name>';
-```
-> **Rollback M-3:** Set `paused = false` on the subscription row.
-
-### M-4: Verify dispatch resumes after restart
-
-```bash
-# Watch events tail — published_at should start filling in within 30 s
-vds-cli scheduler events tail --since 10m --limit 20 --format table
-
-# Check event_bus_lag_seconds is falling on Grafana
-# Alert threshold: > 30 s sustained is P1; < 5 s is healthy
-```
-
----
-
-## Alerting
-
-> **G-3 gap fix (game-day-phase2.md §G-3, 2026-04-24):** During game-day C-3,
-> a 60-second dispatcher outage produced no alert because the existing threshold
-> is 5 minutes. This means a sustained stall for up to 4 min 59 s goes un-alerted.
-> The fix adds a **second, faster-firing SLO alert** at a lower threshold.
-
-### Alert tiers for dispatcher health
-
-| Alert | Metric | Threshold | Duration | Severity | Destination |
-|---|---|---|---|---|---|
-| `VdsDispatcherEarlyWarning` (new — G-3) | `vds_scheduler_outbox_backlog_seconds` | > 120 s | `for: 2m` | P3 | Slack only |
-| `VdsDispatcherStalled` (existing P2) | `vds_scheduler_event_bus_lag_seconds` | > 30 s | `for: 5m` | P2 | PagerDuty + Slack |
-
-Add the following Grafana alert rule alongside the existing P2 rule:
-
-```yaml
-# Grafana alert rule — fast early-warning (P3, Slack-only, no PagerDuty)
-- alert: VdsDispatcherEarlyWarning
-  expr: vds_scheduler_outbox_backlog_seconds > 120
-  for: 2m
-  labels:
-    severity: P3
-    team: vds-platform
-    notify: slack-only
-  annotations:
-    summary: "Dispatcher backlog > 120 s for 2+ minutes — early warning"
-    description: |
-      outbox_backlog_seconds for subscription {{ $labels.subscription_id }}
-      has exceeded 120 s for more than 2 minutes. The dispatcher may be
-      stalling. Use the Quick Diagnosis steps to confirm.
-      The canonical P2 alert fires at 5 min; this fires at ~2 min for
-      early visibility without waking on-call.
-```
-
-**Rationale:** The 5-minute P2 threshold is intentional (transient blips should
-not page). This P3 early-warning fires at 2 minutes, giving operators visibility
-~90 seconds earlier than the P2 alert without introducing alert fatigue.
-
----
-
-## Permanent Fix
-
-1. **Dispatcher health probe alert:** Configure a Grafana alert on `vds_scheduler_event_bus_lag_seconds` > 30 s for 2 consecutive minutes → page on-call.
-2. **outbox_backlog_seconds alert (two-tier):** See §Alerting above — P3 early-warning at 120 s / 2 min (Slack-only) + existing P2 at 30 s / 5 min (PagerDuty).
-3. **Dispatcher restart policy:** Ensure the process manager (systemd `Restart=always`, Docker `restart: unless-stopped`) restarts the scheduler within 10 s of a crash.
-4. **Asyncpg connection keep-alive:** Confirm the LISTEN connection is configured with TCP keep-alive to detect silent network partitions. Add `server_settings={"keepalives_idle": "30"}` to the asyncpg DSN if needed.
-5. **Janitor interval tuning:** Default janitor fires every 60 s. For higher-volume deployments, lower `janitor_interval_s` in `Dispatcher.__init__` to 15 s to reduce stuck-row latency.
-
----
-
-## Post-Incident
-
-- Record which subscription(s) were affected, the lag duration, and root cause (crash vs. network vs. asyncpg bug).
-- If the janitor was slow to clean up, verify the partial index `ix_dispatch_ack_log_claimed_janitor` exists and is being used (run `EXPLAIN` on the janitor query).
-- Add the specific asyncpg exception or error variant to the Symptoms section.
-- If the outage lasted > SLO error-budget window, open a postmortem (use `postmortem-template.md`).
-- Consider whether downstream workflows that failed to fire need manual trigger-now:
-  ```bash
-  vds-cli scheduler schedules trigger-now <schedule_name>
-  # or replay a specific workflow that got stuck:
-  vds-cli scheduler workflows replay <workflow_id>
-  ```

package/tools/vds-scripts/scheduler_orchestrator/docs/runbooks/incident-postgres-unavailable.md

@@ -1,206 +0,0 @@
-# Incident: Postgres Unavailable
-
-**Runbook ID:** INC-001  
-**Severity:** P1 — Entire scheduler is inoperational while Postgres is unreachable.  
-**Last updated:** 2026-04-24  
-
----
-
-## Alerting
-
-> **G-1 gap fix (game-day-phase2.md §G-1, 2026-04-24):** A 7-second API restart
-> produces zero alert noise from `vds_scheduler_scheduled_runs_total` (by design —
-> the counter only fires on a > 5 min gap). A 90-second outage could go un-alerted
-> for up to 30 s due to the 30-second scrape interval. The fix below adds a
-> fast-firing HTTP probe alert.
-
-### Recommended probe alert
-
-Configure a **Blackbox Exporter** (or Grafana synthetic monitoring) probe on
-the scheduler's `/health` endpoint with a **15-second scrape interval**:
-
-```yaml
-# Prometheus scrape config (blackbox_exporter job)
-- job_name: vds-scheduler-health
-  metrics_path: /probe
-  params:
-    module: [http_2xx]
-  static_configs:
-    - targets:
-        - http://vds-scheduler:8000/health
-  relabel_configs:
-    - source_labels: [__address__]
-      target_label: __param_target
-    - source_labels: [__param_target]
-      target_label: instance
-    - target_label: __address__
-      replacement: blackbox-exporter:9115
-  scrape_interval: 15s
-```
-
-Grafana alert rule:
-
-```yaml
-# Alert: scheduler health probe failed for 45+ seconds
-- alert: VdsSchedulerHealthProbeFailed
-  expr: probe_success{job="vds-scheduler-health"} == 0
-  for: 45s
-  labels:
-    severity: P3
-    team: vds-platform
-  annotations:
-    summary: "VDS Scheduler health probe failed for > 45 s"
-    description: |
-      The /health endpoint at {{ $labels.instance }} is not returning HTTP 2xx.
-      Outages < 45 s (e.g., rolling deploys) will NOT page.
-      Investigate using the Quick Diagnosis steps below.
-```
-
-> **Severity**: P3 / Slack-only for < 2 min outages. Escalate to P1 if Postgres
-> is confirmed down (see M-3). Short rolling-deploy restarts (< 45 s) produce no
-> alert — this is intentional.
-
----
-
-## Symptoms
-
-- `vds-cli scheduler schedules list` exits with `OperationalError: could not connect to server` or `Cannot connect to scheduler service`.
-- Application logs contain `asyncpg.exceptions.TooManyConnectionsError` or `connection refused` on the `VDS_SCHEDULER_DATABASE_URL` DSN.
-- `/health` endpoint returns HTTP 503 or the process never finishes starting (if `VDS_SCHEDULER_ENABLED=true`).
-- Grafana: `vds_scheduler_scheduled_runs_total` counter stops incrementing; `vds_scheduler_workflow_failures_total{failure_stage="enqueue"}` spikes.
-- Downstream orchestrators depending on scheduled triggers are silent (no new workflow runs dispatched).
-- DBOS logs: `DBOSUninitializedError` or `RuntimeError: VDS_SCHEDULER_DATABASE_URL must be set` on startup.
-
----
-
-## Quick Diagnosis (≤ 5 min)
-
-**Step 1 — Verify the scheduler process health:**
-```bash
-# From the scheduler host or k8s pod
-curl -sf http://localhost:8005/health | python3 -m json.tool
-# Expected when healthy:  {"status": "ok", "scheduler_enabled": true, "app_version": "..."}
-# Unhealthy: connection refused, 503, or {"scheduler_enabled": false}
-```
-
-**Step 2 — Test Postgres reachability directly:**
-```bash
-psql "$VDS_SCHEDULER_DATABASE_URL" -c "SELECT 1 AS ping;"
-# Expected: ping = 1
-# If psql is unavailable, use pg_isready:
-pg_isready -d "$VDS_SCHEDULER_DATABASE_URL"
-# Expected:  accepting connections
-```
-
-**Step 3 — Check Postgres connection counts (run from a working psql session):**
-```sql
-SELECT count(*), state
-  FROM pg_stat_activity
- WHERE datname = current_database()
- GROUP BY state
- ORDER BY count DESC;
-```
-> **Interpretation:** If `active` + `idle` approaches `max_connections`, the scheduler's connection pool is exhausted rather than the server being down.
-
-**Step 4 — Check scheduler process logs for root cause:**
-```bash
-# Docker-compose deployment
-docker compose logs --tail=100 scheduler
-
-# Systemd
-journalctl -u vds-scheduler --since "10 minutes ago"
-```
-Look for: `TooManyConnectionsError`, `connection refused`, `authentication failed`, `SSL SYSCALL error`.
-
-**Step 5 — Inspect in-flight workflow state:**
-```bash
-# List workflows stuck in non-terminal states (requires a working Postgres connection via a different host)
-psql "$VDS_SCHEDULER_DATABASE_URL" -c \
-  "SELECT workflow_id, status, created_at FROM dbos.workflow_status
-   WHERE status NOT IN ('SUCCESS','ERROR','CANCELLED','MAX_RECOVERY_ATTEMPTS_EXCEEDED')
-   ORDER BY created_at DESC LIMIT 20;"
-```
-
----
-
-## Mitigation (≤ 30 min)
-
-### M-1: Kill-switch the scheduler immediately to stop log spam and reconnect storms
-
-```bash
-# Set in ~/.vds/.env or export directly in the running environment
-export VDS_SCHEDULER_ENABLED=false
-
-# Restart the service in stub mode (health endpoint stays up, DBOS not launched)
-docker compose restart scheduler
-# or: systemctl restart vds-scheduler
-```
-> **Rollback M-1:** Re-enable with `VDS_SCHEDULER_ENABLED=true` and restart once Postgres is restored.
-
-### M-2: If connection limit exhausted — recycle idle connections
-
-```sql
--- Run from a DBA session on the Postgres primary
-SELECT pg_terminate_backend(pid)
-  FROM pg_stat_activity
- WHERE datname = current_database()
-   AND state = 'idle'
-   AND pid <> pg_backend_pid()
-   AND application_name LIKE '%scheduler%';
-```
-> **Rollback M-2:** No rollback needed; `pg_terminate_backend` is non-destructive for idle connections.
-
-### M-3: If Postgres primary is down — wait for automatic replica promotion
-
-The VDS Postgres cluster uses streaming replication. Primary failover is automated (RTO ≤ 60 s per design.md Scenario C-4). Confirm the new primary's DSN:
-
-```bash
-# Ask VDS infra team or check the PgBouncer/patroni status page
-patronictl -c /etc/patroni/config.yml topology
-
-# Once new primary is confirmed, update the env var:
-export VDS_SCHEDULER_DATABASE_URL="postgresql://vds_scheduler:<pass>@<new-primary-host>/vds_scheduler"
-
-# Restart scheduler with new DSN
-docker compose restart scheduler
-```
-> **Rollback M-3:** Revert `VDS_SCHEDULER_DATABASE_URL` to the original primary if failover is reversed.
-
-### M-4: Re-enable and verify recovery
-
-```bash
-export VDS_SCHEDULER_ENABLED=true
-docker compose restart scheduler
-
-# Confirm health
-curl -sf http://localhost:8005/health | python3 -m json.tool
-# Expected: {"status": "ok", "scheduler_enabled": true}
-
-# Verify schedules are visible again
-vds-cli scheduler schedules list --format table
-```
-
-DBOS auto-resumes any PENDING workflows from their last committed checkpoint — no manual replay is needed.
-
----
-
-## Permanent Fix
-
-1. **Connection pool tuning:** Set `VDS_SCHEDULER_OPS_CONCURRENCY` to ≤ 20 % of `max_connections`. For a shared 100-connection Postgres, cap ops-default concurrency at 20.
-2. **PgBouncer:** Route `VDS_SCHEDULER_DATABASE_URL` through PgBouncer in transaction-pool mode to multiplex application connections.
-3. **Postgres HA:** Confirm Patroni or equivalent automatic failover is configured and tested (see `dr-postgres-restore.md` for PITR).
-4. **Circuit breaker alert:** Configure a Grafana alert on `vds_scheduler_workflow_failures_total{failure_stage="enqueue"}` rate > 5/min to page on-call before the outage becomes invisible.
-5. **VDS_SCHEDULER_ENABLED as a Consul/etcd flag:** Enabling kill-switch without a process restart requires dynamic config — plan this for Phase 2.
-
----
-
-## Post-Incident
-
-- Add the specific Postgres error variant encountered to the Symptoms section.
-- If automatic replica promotion took > 60 s, file a ticket with the VDS infra team to investigate Patroni configuration.
-- Update `VDS_SCHEDULER_DATABASE_URL` documentation if the DSN format or hostname changed.
-- Measure and record actual RTO; if > 60 s, update the SLO breach threshold in `docs/error-budget-policy.md`.
-- Check whether any schedules missed ticks while Postgres was down. If `automatic_backfill=true` is set on affected schedules, missed runs will self-heal on resume. For `automatic_backfill=false` schedules, manually trigger if needed:
-  ```bash
-  vds-cli scheduler schedules trigger-now <schedule_name>
-  ```

package/tools/vds-scripts/scheduler_orchestrator/docs/runbooks/incident-secret-leaked-in-event.md

@@ -1,170 +0,0 @@
-# Incident: Secret / Credential Found in Event Payload
-
-**Runbook ID:** INC-004  
-**Severity:** P1 — Potential credential exposure. Treat as a security incident from the moment of detection. Engage the VDS security lead immediately.  
-**Last updated:** 2026-04-24  
-
----
-
-## Symptoms
-
-- Application logs contain: `SecretBlockedError: Event payload contains <pattern_name> — publish blocked. sha256_of_violating_span=<hex>`  
-  This is the **blocked** case: the scanner caught it before insert. The payload never reached `sched.events_outbox`. No credential was persisted.
-- Alternatively (worst case): a credential appears in `sched.events_outbox.payload` or in DBOS step arguments stored in `dbos.workflow_inputs`. This can happen if the scanner was not active at the time of publish (pre-scanner era, scanner bypass, or misconfiguration).
-- Grafana or log-based alert fires on the pattern `SecretBlockedError` in the scheduler logs.
-- A credential is discovered via a SIEM rule scanning Postgres JSONB columns.
-
-> **Pattern classes the scanner covers** (from `security/secret_scanner.py`):
-> - `aws_key` — `AKIA[0-9A-Z]{16}`
-> - `anthropic_key` — `sk-ant-...`
-> - `openai_key` — `sk-...`
-> - `postgres_dsn` — `postgresql[+asyncpg]://user:pass@host/...`
-> - `bearer_token` — `Bearer <token>`
-
----
-
-## Quick Diagnosis (≤ 5 min)
-
-**Step 1 — Confirm whether the credential was blocked (never persisted) or slipped through:**
-```bash
-# Search scheduler logs for SecretBlockedError (blocked case — good news)
-docker compose logs scheduler | grep -i "SecretBlockedError"
-# Example line: SecretBlockedError: Event payload contains openai_key — publish blocked. sha256=...
-
-# Search for the sha256 digest in logs to correlate to a trace_id
-docker compose logs scheduler | grep "<sha256_of_violating_span>"
-```
-
-**Step 2 — If you suspect the credential slipped into the outbox (scan was bypassed), query the outbox:**
-```sql
--- WARNING: this query materialises payload text; run in a restricted psql session
--- with no screen-sharing and redirect output to a file.
--- Do NOT paste results into Slack or a ticket.
-\o /tmp/secret_scan_$(date +%s).txt
-SELECT event_id,
-       topic,
-       committed_at,
-       -- Show first 200 chars of payload serialised to text for manual scan:
-       left(payload::text, 200) AS payload_prefix
-  FROM sched.events_outbox
- WHERE payload::text ~* '(AKIA[0-9A-Z]{16}|sk-ant-|sk-[A-Za-z0-9]{32,}|postgresql\+?[a-z]*://[^@]+@|Bearer\s+[A-Za-z0-9_\-.]{8,})'
- ORDER BY committed_at DESC
- LIMIT 50;
-\o
-```
-
-**Step 3 — Check DBOS workflow inputs for credential propagation:**
-```sql
--- DBOS stores workflow inputs in dbos.workflow_inputs (system table)
--- This is a wider blast-radius check
-SELECT workflow_id, created_at, left(input::text, 300) AS input_prefix
-  FROM dbos.workflow_inputs
- WHERE input::text ~* '(AKIA[0-9A-Z]{16}|sk-ant-|sk-[A-Za-z0-9]{32,})'
- ORDER BY created_at DESC
- LIMIT 20;
-```
-
-**Step 4 — Identify the publishing caller:**
-```bash
-# The trace_id in the event row links back to the originating request
-psql "$VDS_SCHEDULER_DATABASE_URL" -c \
-  "SELECT event_id, topic, trace_id, committed_at FROM sched.events_outbox
-   WHERE event_id = '<suspected_event_id>';"
-
-# Use the trace_id to find the originating service in your tracing system (Jaeger, Tempo)
-```
-
----
-
-## Mitigation (≤ 30 min)
-
-### CRITICAL — Perform these steps in order. Do not skip ahead.
-
-### M-1: Rotate the exposed credential immediately
-
-> This is the **highest priority** step and must happen in parallel with investigation.
-
-| Credential type | Rotation action |
-|---|---|
-| `aws_key` | AWS IAM console → Deactivate key → Create new key → Update `~/.vds/.env` |
-| `anthropic_key` | Anthropic console → Delete key → Issue new key → Update `~/.vds/.env` |
-| `openai_key` | OpenAI dashboard → Revoke key → Create new key → Update `~/.vds/.env` |
-| `postgres_dsn` | `ALTER ROLE vds_scheduler PASSWORD '<new_password>';` → Update `VDS_SCHEDULER_DATABASE_URL` |
-| `bearer_token` | Revoke in issuing service → Re-issue → Update consumer secrets |
-
-### M-2: Purge the credential-containing event(s) from the outbox
-
-> Only execute if Step 2 of Quick Diagnosis confirmed the credential reached the outbox.
-
-```sql
--- Begin a transaction so you can review before committing
-BEGIN;
-
--- Delete the specific event rows containing the credential
-DELETE FROM sched.events_outbox
- WHERE event_id IN ('<event_id_1>', '<event_id_2>');
-
--- Also remove any dispatch_ack_log entries that reference those events
-DELETE FROM sched.dispatch_ack_log
- WHERE event_id IN ('<event_id_1>', '<event_id_2>');
-
--- Confirm the delete is correct, then commit
-SELECT count(*) FROM sched.events_outbox WHERE event_id IN ('<event_id_1>', '<event_id_2>');
--- Expected: 0
-
-COMMIT;
--- If in doubt: ROLLBACK;
-```
-> **Rollback M-2:** Events cannot be un-deleted. Take a full snapshot of the affected rows before deleting (pg_dump specific event_ids to a file).
-
-### M-3: Purge DBOS workflow inputs if credential appeared there
-
-```sql
--- DBOS system table — confirm with DBOS team before mutating in production
--- Only applicable if Step 3 of Quick Diagnosis returned results
--- Phase 1 mitigation: set the input column to a redacted placeholder
-UPDATE dbos.workflow_inputs
-   SET input = '"<REDACTED_SECURITY_INCIDENT>"'
- WHERE workflow_id IN ('<wf_id_1>', '<wf_id_2>');
-```
-
-### M-4: Identify and fix the publishing caller
-
-Trace back to the code that called `bus.publish()` with the offending payload:
-
-```python
-# Caller pattern to search for:
-# from vds_scheduler_orchestrator.events import bus
-# bus.publish(topic, payload_containing_credential)
-```
-- If this was triggered by user input, add input sanitisation before the event payload is constructed.
-- If it was a code bug (e.g., accidentally serialising a config object), fix the serialisation path.
-- Add a test: publish an event containing a mock credential → assert `SecretBlockedError` is raised.
-
-### M-5: Restart the scheduler if any config change affected `VDS_SCHEDULER_DATABASE_URL`
-
-```bash
-docker compose restart scheduler
-curl -sf http://localhost:8005/health | python3 -m json.tool
-```
-
----
-
-## Permanent Fix
-
-1. **Pre-publish hook test:** Add a unit test asserting `scan_payload()` raises `SecretBlockedError` for each of the 5 pattern classes.
-2. **Outbox column encryption:** For Phase 2+, evaluate column-level encryption on `sched.events_outbox.payload` using pgcrypto, so even if a secret slips the scanner it is not readable in plaintext from a Postgres shell.
-3. **SIEM rule:** Add a log-based alert on `SecretBlockedError` in the scheduler log stream → P1 page with 0-minute SLA.
-4. **Credential audit:** After rotation, scan all `~/.vds/.env`, Docker secrets, and CI/CD secret stores for the old credential and confirm it is gone everywhere.
-5. **Scanner coverage review:** Quarterly review of `_PATTERNS` in `secret_scanner.py` to add new credential formats (GitHub tokens, Slack tokens, etc.).
-
----
-
-## Post-Incident
-
-- File an internal security incident ticket within 1 hour of detection.
-- Determine blast radius: was the credential stored in the outbox only, or also in DBOS workflow inputs, Postgres WAL archives, or application logs?
-- If WAL archives contain the credential (they will if it was persisted even briefly), coordinate a WAL purge window with the VDS infra team (see `dr-postgres-restore.md` for WAL management).
-- Complete a postmortem (use `postmortem-template.md`) within 5 business days.
-- If the credential was an API key used by an external provider, verify with that provider's audit logs that no unauthorised calls were made.
-- Update the `sha256_of_violating_span` list in the incident ticket (never the plaintext credential).