@ngocsangairvds/vsaf 3.2.16 → 4.0.0

package/tools/vds-scripts/scheduler_orchestrator/docs/runbooks/rollback-deploy.md

@@ -1,209 +0,0 @@
-# Rollback: Scheduler Deployment
-
-**Runbook ID:** OPS-001  
-**Last updated:** 2026-04-24  
-
----
-
-## Overview
-
-The VDS Ecosystem Scheduler uses a defence-in-depth rollback model:
-
-- **Phase 1:** Kill-switch via `VDS_SCHEDULER_ENABLED=false` — zero data loss, zero Celery Beat impact.
-- **Phase 2:** Revert observability/feature additions; core platform is unaffected.
-- **Phase 3:** Reverse Celery Beat migration tasks one-by-one.
-- **Phase 4:** Disable autoscaling or trigger-enqueue flags per schedule.
-
-DBOS's blue-green versioning (`DBOS_APP_VERSION` / `dbos_app_version`) ensures in-flight workflows complete before new workers pick up tasks, preventing split-brain during rollback.
-
----
-
-## Pre-Rollback Checklist
-
-Before rolling back any deployment, complete these steps:
-
-- [ ] Identify the target git tag / image tag you are rolling back to.
-- [ ] Confirm `DBOS_APP_VERSION` of the current (broken) deploy: `curl -sf http://localhost:8005/health | python3 -m json.tool`.
-- [ ] Snapshot in-flight workflow count: `vds-cli scheduler workflows list --status running | wc -l`.
-- [ ] Notify on-call and team lead via the incident channel.
-- [ ] Open this runbook's corresponding postmortem ticket.
-
----
-
-## Phase 1 Rollback — Kill-Switch (Fastest; Zero Data Loss)
-
-**When to use:** Any Phase 1 scheduler bug, database connection issue, DBOS startup failure, or unexpected behaviour.
-
-### Step 1: Disable the scheduler via kill-switch
-
-```bash
-# Set in ~/.vds/.env
-VDS_SCHEDULER_ENABLED=false
-
-# Restart the service — it starts in stub mode (health endpoint alive, DBOS not launched)
-docker compose restart scheduler
-
-# Confirm stub mode
-curl -sf http://localhost:8005/health | python3 -m json.tool
-# Expected: {"status": "ok", "scheduler_enabled": false, ...}
-```
-
-> **Effect:** All cron triggers stop immediately. In-flight workflows that were already PENDING complete normally — DBOS resumes them when the scheduler next starts in enabled mode. `sched.schedules` data is preserved; no rows are deleted.
-> **Celery Beat:** Completely unaffected. The 6 `memory_evolution` periodic tasks continue running normally.
-
-### Step 2: Roll back the code / container image
-
-```bash
-# Docker-compose: pin to the previous image tag
-docker compose pull scheduler:<previous_tag>
-docker compose up -d scheduler
-
-# Confirm image version
-docker inspect scheduler | grep -i image
-```
-
-### Step 3: Re-enable the scheduler on the rolled-back image
-
-```bash
-VDS_SCHEDULER_ENABLED=true
-docker compose restart scheduler
-
-# Health check
-curl -sf http://localhost:8005/health | python3 -m json.tool
-# Expected: {"status": "ok", "scheduler_enabled": true, "app_version": "<previous_tag>"}
-
-# Verify schedules are still present
-vds-cli scheduler schedules list --format table
-```
-
-### Step 4: Run smoke test
-
-```bash
-# Trigger the smoke workflow and confirm it reaches SUCCESS within 10 s
-curl -sf -X POST http://localhost:8005/test/enqueue | python3 -m json.tool
-# Note the workflow_id
-
-curl -sf http://localhost:8005/test/status/<workflow_id> | python3 -m json.tool
-# Expected: {"status": "SUCCESS", ...}
-```
-
----
-
-## Phase 2 Rollback — Revert Observability / Feature Changes
-
-**When to use:** Phase 2 added a feature (e.g., Grafana dashboard, websocket events, Conductor integration) that is broken, but the core Phase 1 scheduler is healthy.
-
-### Step 1: Confirm Phase 1 core is healthy
-
-```bash
-curl -sf http://localhost:8005/health | python3 -m json.tool
-vds-cli scheduler schedules list --format table
-```
-
-### Step 2: Revert the specific Phase 2 change
-
-The Phase 2 changes are additive and flag-gated where possible:
-
-```bash
-# Disable Conductor (if adopted and problematic)
-unset DBOS_CONDUCTOR_KEY
-docker compose restart scheduler
-
-# Disable MCP surface (if enabled and broken)
-VDS_SCHEDULER_MCP_ENABLED=false
-docker compose restart scheduler
-```
-
-If the change was a code deployment, roll back the image (see Phase 1 Step 2) but keep `VDS_SCHEDULER_ENABLED=true`.
-
----
-
-## Phase 3 Rollback — Reverse Celery Beat Migration
-
-**When to use:** A task that was migrated from Celery Beat to `scheduler_orchestrator` is broken in the scheduler. Restore Celery Beat for that specific task.
-
-### Step 1: Re-enable the Celery Beat task
-
-```python
-# In memory_orchestrator/celery_app.py (or equivalent):
-# Un-comment the beat_schedule entry for the migrated task.
-# The task's scheduler_orchestrator schedule should remain paused (not deleted)
-# during the 30-day parallel-run observation window.
-```
-
-```bash
-# Restart the Celery Beat container
-docker compose restart celery-beat
-docker compose logs --tail=20 celery-beat | grep "<task_name>"
-```
-
-### Step 2: Pause (do not delete) the corresponding scheduler_orchestrator schedule
-
-```bash
-vds-cli scheduler schedules pause <schedule_name>
-# Do NOT delete — the schedule row serves as the audit record of the migration attempt
-```
-
-### Step 3: Monitor parallel execution
-
-Observe both the Celery Beat task and the paused scheduler schedule for 7 days before deciding whether to re-attempt the migration or abort permanently.
-
----
-
-## Phase 4 Rollback — Disable Autoscaling or UDF-Trigger Enqueue
-
-**When to use:** Phase 4 autoscaling or Postgres-UDF-trigger enqueue is causing unexpected behaviour.
-
-### Disable autoscaling
-
-```bash
-# Remove or set to false the autoscaling flag (Phase 4 env var TBD at Phase 4 planning)
-VDS_SCHEDULER_AUTOSCALING_ENABLED=false
-docker compose restart scheduler
-```
-
-### Disable UDF-trigger enqueue for a specific schedule
-
-```sql
--- Opt out of UDF enqueue for the affected schedule (Phase 4 flag column)
-UPDATE sched.schedules
-   SET use_udf_enqueue = false
- WHERE schedule_name = '<schedule_name>';
--- No restart required; takes effect on next tick
-```
-
----
-
-## Rollback Verification
-
-After any rollback, confirm the following within 10 minutes:
-
-```bash
-# 1. Health endpoint
-curl -sf http://localhost:8005/health | python3 -m json.tool
-
-# 2. Schedules accessible
-vds-cli scheduler schedules list --format table
-
-# 3. No ERROR workflows accumulating
-vds-cli scheduler workflows list --status failed --format table
-
-# 4. Metrics endpoint live
-curl -sf http://localhost:8005/metrics | grep vds_scheduler_scheduled_runs_total
-
-# 5. No dispatcher stall (event bus lag)
-curl -sf http://localhost:8005/metrics | grep vds_scheduler_event_bus_lag_seconds
-# Expected: value < 5.0 for all topics
-```
-
----
-
-## Post-Rollback
-
-- Update the DBOS_APP_VERSION in the environment to match the rolled-back image tag.
-- File a postmortem ticket (use `postmortem-template.md`) if the deployment caused a user-visible incident or an SLO breach.
-- Determine root cause before re-deploying the rolled-back change.
-- If schedules missed ticks during rollback, check `automatic_backfill` column and trigger manual runs where required:
-  ```bash
-  vds-cli scheduler schedules trigger-now <schedule_name>
-  ```

package/tools/vds-scripts/scheduler_orchestrator/docs/scaling-decision-phase4.md

@@ -1,176 +0,0 @@
-# Scaling Decision — Phase 4 (WP-17 TSK-17.3)
-
-**Date**: 2026-04-24  
-**Author**: sharding-dr-dev  
-**Spec ref**: `vds-scheduler/design.md §Capacity & Scaling Plan / WP-17`  
-**Decision gate**: Phase 4  
-**Status**: ✅ DECISION — Sharding deferred (DEFAULT path)
-
----
-
-## Background
-
-WP-17 (Phase 4) requires a data-driven sharding decision:
-
-> IF measured peak throughput approaches the single-Postgres ceiling  
-> OR monthly workflow volume exceeds 1 000 000  
-> THEN proceed to sharding implementation (WP-17 ALTERNATE path)  
-> ELSE document the deferral rationale and set a re-evaluation trigger (DEFAULT path)
-
-ADR-14 (`design.md §ADR-14`) defers Postgres sharding to Phase 4 on the condition that
-Phase 1–3 volume data confirm headroom remains. This document records that assessment.
-
----
-
-## Volume Data
-
-### Phase 1 Actual Volume (from `conductor-decision.md`)
-
-| Metric | Observed |
-|--------|----------|
-| Production steady-state | **4–8 workflows / week** (drift scan + ops triggers) |
-| Total workflows in 30-day test window | ~150 (smoke + load tests) |
-| Peak concurrent queue depth (load test) | ~40 (agent-default queue) |
-| ERROR / incident frequency | < 1 / month |
-
-> **Context**: `VDS_SCHEDULER_ENABLED=false` in production during Phase 1.
-> Figures are from controlled test-environment runs, not production traffic.
-
-### Phase 3 Projection Post-Celery Migration (from `conductor-decision.md`)
-
-| Task | Cadence | Monthly workflows |
-|------|---------|-------------------|
-| memory-evolution-dispatch | 60 s | ~43 200 |
-| memory-evolution-sweep | 1 h | ~720 |
-| memory-evolution-replay-dlq | 15 min | ~2 880 |
-| mem0-resolve-pending-ids | 5 min | ~8 640 |
-| mem0-repush-stalled | 1 h | ~720 |
-| mem0-repush-local-only | 6 h | ~120 |
-| **Total Phase 3 steady state** | | **~56 280 / month** |
-
-Expressed as a rate:
-- **Daily**: ~1 876 workflows / day
-- **Peak hourly** (assuming 60 s dispatch task drives 43 200/month): ~1 440 / hour ≈ **~0.4 / second**
-
----
-
-## Postgres Ceiling
-
-DBOS documentation and design.md cite:
-
-> **1 000 – 10 000 workflows / second** per single Postgres instance (depending on hardware tier).
-
-Hardware reference (design.md §Capacity):
-- **4 vCPU / 16 GB** Postgres instance
-- **3 executor replicas** (4 vCPU / 8 GB each)
-
-Phase 2 performance baseline (`docs/performance-baseline.md`) demonstrated:
-- p95 workflow start ≤ 2 000 ms at 5 rps sustained
-- p95 workflow start ≤ 5 000 ms at 50 rps burst (Phase 2 ceiling)
-
----
-
-## Headroom Analysis
-
-| Dimension | Phase 3 Projection | DBOS Ceiling | Headroom |
-|-----------|-------------------|--------------|----------|
-| Peak throughput | ~0.4 wf/s | 1 000 wf/s | **~2 500×** |
-| Monthly volume | ~56 000 | — | — |
-| vs. Phase 1 actual (4–8/week) | 28–56× higher | — | — |
-
-Even at the conservative DBOS floor of 1 000 wf/s:
-- Phase 3 peak rate (0.4 wf/s) is **~2 500×** below the single-instance ceiling.
-- A 10× growth over Phase 3 projection still leaves **250×** headroom.
-
----
-
-## Re-measure Procedure (TSK-17.1)
-
-`scripts/benchmarks/phase4-sharding-remeasure.sh` re-runs Phase 2's Locust harness
-at 10× scale (50 rps sustained, 500 rps burst) and validates NFR gates.
-
-**Status of live re-measure**: ⏳ **PENDING — ops task post-staging-deployment.**
-
-The scheduler is not yet deployed to a production-scale Postgres instance
-(`VDS_SCHEDULER_ENABLED=false` in `~/.vds/.env`). Once Phase 3 Celery migration is
-live on staging, run:
-
-```bash
-export VDS_SCHEDULER_ENABLED=true
-export VDS_SCHEDULER_DATABASE_URL="postgresql://vds_scheduler:<pass>@<staging-host>/vds_scheduler"
-export BASE_URL="http://<staging-api>:8000"
-
-bash scheduler_orchestrator/scripts/benchmarks/phase4-sharding-remeasure.sh
-```
-
-Fill in the table below with measured values once the run completes.
-
-### Re-measure Results (fill in post-staging-run)
-
-| Scenario | Target | p95 Measured | p99 Measured | rps Achieved | Status |
-|----------|--------|-------------|-------------|-------------|--------|
-| S2-P4 (50 rps, 30 min) | p95 ≤ 2 000 ms | TBD | TBD | TBD | — |
-| S3-P4 (500 rps, 10 min) | p95 ≤ 5 000 ms | TBD | TBD | TBD | — |
-| S4-P4 (2 000 rps outbox, 5 min) | p95 ≤ 2 000 ms | TBD | TBD | TBD | — |
-
----
-
-## Decision
-
-### Criterion Applied
-
-| Criterion | Threshold | Current | Decision |
-|-----------|-----------|---------|----------|
-| Monthly volume | ≥ 1 000 000 workflows/month | ~56 000 (Phase 3 projection) | ✅ Below — 18× headroom |
-| Peak throughput vs. ceiling | > 10% of DBOS 1K wf/s floor | 0.4 wf/s (~0.04%) | ✅ Far below |
-| Live remeasure NFR breach | S2-P4 p95 > 2 000 ms | TBD (pending ops run) | ⏳ Pending |
-
-### Chosen Path: **DEFAULT — Sharding DEFERRED**
-
-**Rationale**:
-
-1. **Phase 3 projection is 56K workflows/month** — equivalent to ~0.4 wf/s peak.
-   The DBOS + Postgres ceiling is 1 000–10 000 wf/s. Headroom exceeds 2 000×.
-   Sharding at this scale would add operational complexity with zero throughput benefit.
-
-2. **Phase 2 NFR gates cleared at 50 rps burst** (`docs/performance-baseline.md`).
-   Phase 4's 10× remeasure target of 500 rps is the sharding-trigger threshold;
-   until the ops run confirms an NFR breach there, no action is warranted.
-
-3. **DBOS workflow partitioning path exists** (`docs/sharding-implementation-plan.md`).
-   Separate-Postgres-instance partitioning can be introduced in a future phase with
-   minimal code change (one queue-per-shard mapping layer). The future path is documented
-   and cost-of-delay is low.
-
-4. **Volume re-evaluation trigger is clear** — see §Re-evaluation Trigger below.
-
-**Code impact**: None. No sharding code is introduced in Phase 4. See
-`docs/sharding-implementation-plan.md` for the future implementation skeleton.
-
----
-
-## Re-evaluation Trigger
-
-Re-open the sharding decision if **any** of the following are observed in production:
-
-| Signal | Threshold | Action |
-|--------|-----------|--------|
-| Monthly workflow volume | Exceeds **1 000 000 / month** (~20× Phase 3 projection) | Begin sharding design sprint |
-| Sustained Postgres CPU | > 70% at peak for 7+ consecutive days | Evaluate vertical scale first; then sharding |
-| p95 workflow start latency | > 2 000 ms at steady-state (< 50 rps) | Investigate indexes / pool tuning before sharding |
-| Live Phase 4 remeasure | Any S2-P4/S3-P4 NFR breach | Escalate to team-lead; re-open WP-17 ALTERNATE path |
-| DBOS Conductor adoption | Phase 3 gate triggers Conductor license | Reassess if Conductor adds multi-cluster routing |
-
-**Next scheduled review**: Phase 5 planning gate, or immediately upon any trigger above.
-
----
-
-## References
-
-- `conductor-decision.md` — Phase 1 volume observations and Phase 3 projections
-- `docs/performance-baseline.md` — Phase 2 NFR measurements (S1–S5)
-- `docs/sharding-implementation-plan.md` — ALTERNATE path skeleton (DBOS partitioning)
-- `scripts/benchmarks/phase4-sharding-remeasure.sh` — 10× Locust re-measure script
-- `design.md §ADR-14` — Sharding deferred to Phase 4 gate
-- `design.md §Capacity & Scaling Plan` — Hardware reference and scale-up triggers
-- DBOS throughput reference: <https://docs.dbos.dev/production/performance>

package/tools/vds-scripts/scheduler_orchestrator/docs/secret-rotation-drill.md

@@ -1,202 +0,0 @@
-# Secret Rotation Drill
-
-**Owner**: security-dev | **TSK-N.8** | **ADR-19**  
-**Last run**: 2026-04-24 (dry-run, testcontainers simulation)
-
----
-
-## Overview
-
-This document describes the procedure for rotating three secret classes used by the
-VDS Ecosystem Scheduler without dropping any in-flight workflows or causing scheduler
-downtime:
-
-1. **Postgres DSN** (`VDS_SCHEDULER_DATABASE_URL`)
-2. **LLM API keys** (OpenAI / Anthropic keys consumed by agent workflows)
-3. **Capability tokens** (`VDS_SCHEDULER_CAPABILITY` — orchestrator identity label)
-
-All rotation steps follow the **zero-downtime, connection-pool-aware** pattern: the
-old connection pool is drained gracefully, the env var is updated, and the new pool
-opens to the new target. At no point are running DBOS workflows interrupted because
-DBOS persists workflow state in Postgres — a brief connection gap leaves workflow
-progress intact.
-
----
-
-## Prerequisites
-
-- Access to `~/.vds/.env` (secrets file; never committed to git).
-- Shell with `vds-cli` on PATH (or use `./scripts/worktree_uv.sh run --project vds_cli vds-cli`).
-- Docker Compose stack running (`COMPOSE_PROFILES=dev` or production equivalent).
-- Operator Postgres superuser credentials for the `CREATE ROLE` / `REVOKE` commands.
-
----
-
-## Rotation 1 — Postgres DSN
-
-### When to rotate
-- Quarterly password rotation policy.
-- After a potential credential exposure (see `incident-secret-leaked-in-event.md`).
-- Before promoting a new Postgres replica to primary.
-
-### Procedure
-
-```bash
-# Step 1: Drain in-flight DBOS workflows (graceful shutdown)
-#   The scheduler's /health endpoint reports active workflow count.
-#   Wait until active_workflows = 0 before proceeding, or set a quiescence timeout.
-vds-cli scheduler health --json | jq '.active_workflows'
-
-# Step 2: Generate / receive the new Postgres password from your secrets manager.
-NEW_PG_PASSWORD="$(op item get vds-scheduler-db --field password)"  # 1Password example
-
-# Step 3: Update the Postgres password (run as superuser on the DB host)
-psql "$OLD_DSN" -c "ALTER ROLE vds_scheduler_app PASSWORD '$NEW_PG_PASSWORD';"
-
-# Step 4: Update the env file (never commit this file)
-sed -i '' "s|VDS_SCHEDULER_DATABASE_URL=.*|VDS_SCHEDULER_DATABASE_URL=postgresql://vds_scheduler_app:${NEW_PG_PASSWORD}@db-host:5432/vds_scheduler|" ~/.vds/.env
-
-# Step 5: Restart the scheduler service (hot-reload is not supported for DSN changes)
-#   Docker Compose:
-docker compose -p vds-scheduler --profile dev restart scheduler
-
-#   Or if running via systemd:
-# systemctl restart vds-scheduler
-
-# Step 6: Verify health
-vds-cli scheduler health --json | jq '.status'
-# Expected: "healthy" or "degraded" (check degraded reason before proceeding)
-
-# Step 7: Smoke-test a schedule mutation
-vds-cli scheduler schedules list
-```
-
-### Zero-downtime guarantee
-DBOS workflows store their continuation state in `dbos.workflow_status` (Postgres).
-During the connection gap (Steps 4→5), any in-flight workflow retries until the new
-pool is available. The retry window is configurable via `DBOS_WORKFLOW_TIMEOUT`. For
-production, set this to at least twice the service restart time (typically < 30s).
-
-### Rollback
-```bash
-# Restore old DSN in ~/.vds/.env and restart
-sed -i '' "s|VDS_SCHEDULER_DATABASE_URL=.*|VDS_SCHEDULER_DATABASE_URL=$OLD_DSN|" ~/.vds/.env
-docker compose -p vds-scheduler --profile dev restart scheduler
-```
-
----
-
-## Rotation 2 — LLM API Keys
-
-LLM keys (OpenAI / Anthropic) are consumed by agent-step workflows inside DBOS.
-They are read from the environment at process startup and cached in memory.
-
-### When to rotate
-- After key exposure event.
-- On a 90-day rotation schedule.
-
-### Procedure
-
-```bash
-# Step 1: Provision new key from provider dashboard / secrets manager.
-NEW_ANTHROPIC_KEY="sk-ant-..."    # Anthropic Console
-NEW_OPENAI_KEY="sk-..."           # OpenAI Dashboard
-
-# Step 2: Update ~/.vds/.env
-sed -i '' "s|ANTHROPIC_AUTH_TOKEN=.*|ANTHROPIC_AUTH_TOKEN=$NEW_ANTHROPIC_KEY|" ~/.vds/.env
-sed -i '' "s|OPENAI_API_KEY=.*|OPENAI_API_KEY=$NEW_OPENAI_KEY|" ~/.vds/.env
-
-# Step 3: Rolling restart (one replica at a time for HA deployments)
-docker compose -p vds-scheduler --profile dev up -d --no-deps --force-recreate scheduler
-
-# Step 4: Verify agent workflows succeed
-vds-cli scheduler schedules trigger-now audit-drift-daily --dry-run
-```
-
-### Hot rotation (advanced)
-The agent_step workflow (workflows/agent_step.py) accepts an optional `llm_api_key`
-in its context_json. This allows key rotation without process restart by updating the
-schedule's `context_json`:
-
-```bash
-vds-cli scheduler schedules update audit-drift-daily \
-  --context-json '{"llm_api_key": "sk-ant-...<new_key>"}'
-```
-
-**Security note**: context_json is stored in Postgres. For highly sensitive keys,
-prefer process-restart rotation over context_json embedding.
-
----
-
-## Rotation 3 — Capability Tokens
-
-`VDS_SCHEDULER_CAPABILITY` identifies the orchestrator process (e.g. `audit_orchestrator`,
-`memory_orchestrator`). This is an identity label, not an authentication secret. It is
-read at process start from the environment and is not time-limited.
-
-### When to rotate
-- When an orchestrator process is decommissioned and its schedules transferred.
-- When splitting a combined orchestrator into two separate capability identities.
-
-### Procedure
-
-```bash
-# Step 1: Transfer ownership of existing schedules to the new capability.
-#   (platform capability required)
-vds-cli scheduler schedules list --owner old-orchestrator-name | \
-  jq -r '.[].schedule_name' | \
-  xargs -I{} vds-cli scheduler schedules update {} --owner new-orchestrator-name
-
-# Step 2: Update capability label in env file.
-sed -i '' "s|VDS_SCHEDULER_CAPABILITY=.*|VDS_SCHEDULER_CAPABILITY=new-orchestrator-name|" ~/.vds/.env
-
-# Step 3: Restart the scheduler process for the new capability to take effect.
-docker compose -p vds-scheduler --profile dev restart scheduler
-
-# Step 4: Verify the new capability can mutate its schedules.
-vds-cli scheduler schedules pause <any-transferred-schedule>
-vds-cli scheduler schedules resume <any-transferred-schedule>
-```
-
-### Zero-downtime note
-Capability changes do NOT require workflow quiescence. In-flight workflows inherit
-their enqueue-time context and are unaffected by runtime capability changes.
-
----
-
-## Audit Log
-
-All secret rotation events are logged to `sched.audit_log` via the `platform` capability:
-
-```sql
-SELECT audit_id, actor_capability, operation, object_id, occurred_at
-FROM sched.audit_log
-WHERE operation IN ('schedule.update', 'schedule.create', 'schedule.delete')
-  AND occurred_at > now() - interval '1 hour'
-ORDER BY occurred_at DESC;
-```
-
----
-
-## Dry-Run Test Results
-
-Integration test: `tests/integration/test_secret_rotation_drill.py`
-
-| Step | Result |
-|------|--------|
-| Seed 2 schedules and 1 event | PASS |
-| Simulate DSN "rotation" (reset session factory) | PASS |
-| Data still accessible post-rotation | PASS — 0 rows lost |
-| Config reload picks up new CAPABILITY | PASS |
-| Old sessions drain without workflow drops | PASS |
-
-Last verified: 2026-04-24 via testcontainers Postgres 16.
-
----
-
-## References
-
-- `incident-secret-leaked-in-event.md` — response procedure for key leaks in event payloads
-- `security/secret_scanner.py` — payload scan rejecting DSN / API key patterns before outbox insert
-- `security/audit_log.py` — append-only audit log emission
-- ADR-19 — append-only audit trail with Postgres role enforcement