@ngocsangairvds/vsaf 3.2.16 → 4.0.1

package/tools/vds-scripts/scheduler_orchestrator/docs/sharding-decision.md

@@ -1,111 +0,0 @@
-# Sharding Decision — VDS Ecosystem Scheduler (Phase 4 / TSK-17.1..17.3)
-
-**Status:** **DEFERRED** — single-Postgres deployment; sharding gate documented for re-evaluation at scale threshold.
-**Date:** 2026-04-25
-**Spec ref:** `.gpt-5.4/specs/vds-scheduler/tasks.md` line 883 (TSK-17.1..17.3); `design.md` §Capacity & Scaling Plan (lines 1416–1446); ADR-INF-2 (shared-Postgres canonical deployment).
-**Companion doc:** `scaling-decision-phase4.md` (this file refines its strategy + rejection sections).
-
----
-
-## Decision Summary
-
-The scheduler runs against a single `vds_platform` Postgres instance with multi-schema isolation (`sched`, `dbos`, `audit_state`, `memory`, `progress`, `public`) shared with three sibling orchestrators (audit, memory, progress). **Sharding is DEFERRED** because (a) measured and projected workflow volume is multiple orders of magnitude below the DBOS+Postgres single-instance ceiling, (b) DBOS workflow throughput at current volume shows zero visible lock contention or queue backlog in the Phase-2 baseline runs, and (c) sharding would introduce N×operational surface (migrations, backups, dashboards, event-bus subscription fan-out) for no observed throughput benefit.
-
-The shared-Postgres + multi-schema layout introduced in v2.0.17 (ADR-INF-2 / ADR-23) actually *strengthens* the deferral case: cross-orchestrator state is co-located, operationally visible, and queryable as single-DB joins instead of cross-DB ETL. Splitting would reverse a deliberate consolidation that just landed.
-
-This decision is reversible. The triggers in §Sharding Triggers below are measurable and tied to existing SLO dashboard panels and Postgres telemetry; when any trigger fires, the implementation skeleton in `sharding-implementation-plan.md` becomes the next step.
-
----
-
-## Current State (Data-Backed)
-
-All numbers below cite the source they are drawn from. Where exact production data is unavailable (the scheduler runs with `VDS_SCHEDULER_ENABLED=false` in production by default), the source is a controlled smoke or Phase-2 load-test run.
-
-| Dimension | Observed / Projected | Source |
-|---|---|---|
-| Steady-state workflow firings | ~4–8 / week production; ~30+ / smoke-matrix run | `scaling-decision-phase4.md` §Volume Data; v2.0.17 closure smoke matrix (zero permission errors) |
-| Phase-3 projected monthly volume (post-Celery retirement) | ~56 280 workflows/month (~1 876/day, ~0.4 wf/s peak) | `scaling-decision-phase4.md` §Phase 3 Projection |
-| `dbos.workflow_status` row count | Estimated ~10³–10⁴ rows in staging; production schema empty until `VDS_SCHEDULER_ENABLED=true` | inferred from smoke-matrix run cadence |
-| Peak ENQUEUED queue depth | ~40 rows during Phase-2 load test | `scaling-decision-phase4.md` §Phase 1 Actual Volume |
-| Active schedules registered | ~7 (6 ex-Celery memory-evolution + `evolution.auto-promote` + autoscaler tick) | `vds-scripts/CLAUDE.md` §Scheduler Architecture |
-| Workflow-start p95 latency | ≤ 2 000 ms at 5 rps; ≤ 5 000 ms at 50 rps burst | `performance-baseline.md` (Phase-2 NFR gates) |
-| Lock-contention on `sched.*` / `dbos.*` | None observed in v2.0.17 smoke matrix (30+ firings, zero errors) | v2.0.17 closure note |
-| Postgres connection pool saturation | Not observed; pool sized for 4 orchestrators on shared `vds_platform` | ADR-INF-2 / ADR-23 |
-
-**Headroom vs DBOS-cited ceiling (1 000–10 000 wf/s per Postgres instance):** roughly **2 500× at the conservative floor**, even before any vertical scaling.
-
----
-
-## Sharding Triggers (When to Revisit)
-
-This decision MUST be re-opened when **any** of the following are observed sustained for the indicated window. All are measurable from the existing SLO dashboard (`scheduler_orchestrator/dashboards/slo-dashboard.json`) or standard Postgres telemetry.
-
-| Trigger | Threshold | Window | Dashboard Panel / Source |
-|---|---|---|---|
-| Monthly workflow volume | ≥ 1 000 000 / month (~20× Phase-3 projection) | rolling 30 d | `dbos.workflow_status` row growth |
-| `dbos.workflow_status` table size | ≥ 50 M rows OR > 100 GB on disk | snapshot | `pg_stat_user_tables` |
-| Workflow-start p99 latency | > 500 ms sustained at < 50 rps | 7 consecutive days | SLO dashboard "Workflow Start Latency p95/p99" |
-| Postgres CPU at peak | > 70% sustained | 7 consecutive days | host / `pg_stat_database` |
-| Postgres IOPS saturated | host IOPS quota hit | any | host metrics |
-| INSERT lock contention on `dbos.workflow_status` / `sched.*` | > 5% of transaction time blocked | 24 h | `pg_stat_activity` `wait_event_type = 'Lock'` |
-| Connection pool saturation | pgBouncer / DBOS pool > 90% utilization | 7 d sustained | pool exporter |
-| Phase-4 10× re-measure NFR breach | any S2-P4/S3-P4 row in `scaling-decision-phase4.md` §Re-measure Results fails | one run | `scripts/benchmarks/phase4-sharding-remeasure.sh` |
-
-A single trigger fires the design sprint described in `sharding-implementation-plan.md`; it does not auto-execute sharding.
-
----
-
-## Sharding Strategy Options (If/When Triggered)
-
-Three options, ranked by operational simplicity. Selection happens at trigger-fire time based on which dimension is saturating.
-
-### (a) Schema-level partitioning — `sched.workflow_status` partitioned by `created_at` month
-- **Pros:** Lowest operational lift; Postgres-native declarative partitioning; backups and migrations stay single-instance; no event-bus topology change; old partitions become drop-in archival units.
-- **Cons:** Does not relieve INSERT-hotspot or connection-pool saturation (all writes still hit one instance); only buys runway against table-size and read-scan growth.
-- **Best for triggers:** `workflow_status ≥ 50 M rows` or read-side scan latency.
-
-### (b) Database-level shard split — multiple `vds_platform_N` databases
-- **Pros:** Linear horizontal capacity for both writes and connections; clean blast-radius separation per shard.
-- **Cons:** N× migrations, N× backups, N× dashboards; cross-shard joins for cross-orchestrator state (reverses the v2.0.17 / ADR-INF-2 consolidation); event-bus subscribers fan out O(N) per topic, with cross-shard correlation queries becoming O(N²) in the worst case; routing layer (shard-key → DSN) becomes a new failure surface.
-- **Best for triggers:** Sustained Postgres CPU / IOPS / connection-pool saturation that vertical scale cannot absorb.
-
-### (c) DBOS multi-tenant split — separate DBOS instances per orchestrator
-- **Pros:** Aligns shard boundary to ownership boundary (audit, memory, progress, scheduler each get their own DBOS+Postgres); maps to DBOS's documented multi-instance partitioning path; minimal code change (queue-per-shard mapping layer; see `sharding-implementation-plan.md`).
-- **Cons:** Loses the shared-Postgres single-source-of-truth that ADR-INF-2 explicitly designed for; cross-orchestrator workflows (chains, event-bus) now traverse instance boundaries; no help if the scheduler itself is the hot orchestrator.
-- **Best for triggers:** A non-scheduler orchestrator (e.g., memory) generating the volume pressure on the shared instance.
-
----
-
-## Rejection Criteria for Premature Sharding
-
-We **reject** sharding now because every justification fails:
-
-1. **No observed bottleneck.** Smoke matrix (v2.0.17) and Phase-2 load test (5–50 rps) show zero lock contention, zero error rate, p95 within NFR. Sharding solves a problem that does not exist in the measurements.
-2. **Operational complexity multiplier.** Each shard adds a migration target, a backup target, a monitoring target, and a runbook variant. With 4 orchestrators and N shards the operator surface is O(N×4); the current operator surface is O(1).
-3. **Event-bus fan-out cost.** FR-13 (backpressure) and the outbox/dispatcher contract assume a single publish substrate. Cross-shard event-bus subscriptions become O(N²) in the worst case (N publishers × N subscribers).
-4. **Reverses v2.0.17 / ADR-INF-2 consolidation.** The shared `vds_platform` layout was just adopted to *eliminate* cross-DB ETL. Sharding now would reintroduce the cost ADR-INF-2 paid down.
-5. **DBOS partitioning is a future drop-in.** The path exists (`sharding-implementation-plan.md`) and is documented; cost-of-delay is low because the trigger-thresholds give us advance warning.
-
----
-
-## Re-evaluation Cadence
-
-- **Routine review:** Phase 5 planning gate. Pull latest `dbos.workflow_status` row count, monthly workflow volume, p95/p99 workflow-start latency, and Postgres CPU peak — log them in this file's "Current State" table.
-- **Trigger-driven review:** Immediate, on any §Sharding Triggers row firing. Open a sharding design sprint per `sharding-implementation-plan.md`.
-- **Auto-prompt review:** When `dbos.workflow_status` reaches 10 M rows OR when SLO dashboard shows workflow-start p99 > 100 ms for 7 consecutive days — these are early-warning thresholds that precede the hard triggers above.
-
-Each review either re-affirms DEFERRED (with a refreshed Current-State snapshot) or escalates to ALTERNATE-path implementation.
-
----
-
-## References
-
-- **ADR-INF-2 — Shared `vds_platform` Postgres** — `design.md` §v2.0.17 / ADR-23 (lines 1716–1788); rationale for single-DB multi-schema deployment.
-- **FR-13 — Backpressure & Flow Control** — `requirements.md` lines 776–782; outbox rate limit, backlog alerting, dispatcher batch mode, opt-in shed policy. Sharding interacts with FR-13.4's per-topic shed policy.
-- **Capacity & Scaling Plan** — `design.md` lines 1416–1446; baseline scenarios, scale-up triggers, Phase-4 sharding gate.
-- **Companion: `scaling-decision-phase4.md`** — Volume data, Phase-3 projections, NFR re-measure procedure, criterion table.
-- **Implementation skeleton: `sharding-implementation-plan.md`** — DBOS partitioning ALTERNATE path.
-- **SLO dashboard:** `scheduler_orchestrator/dashboards/slo-dashboard.json` — availability, burn rate, tick jitter, workflow-start latency, event-dispatch latency, outbox lag panels referenced as trigger sources.
-- **Performance baseline:** `scheduler_orchestrator/docs/performance-baseline.md` — Phase-2 NFR measurements (S1–S5).
-- **Re-measure script:** `scripts/benchmarks/phase4-sharding-remeasure.sh` — Phase-4 10× Locust harness.
-- **DBOS production performance reference:** https://docs.dbos.dev/production/performance — cited 1K–10K wf/s ceiling.

package/tools/vds-scripts/scheduler_orchestrator/docs/sharding-implementation-plan.md

@@ -1,189 +0,0 @@
-# Sharding Implementation Plan — Future Reference (TSK-17.2)
-
-**Status**: ⏭ SKIPPED — per TSK-17.3 scaling decision, sharding is NOT needed for Phase 4 scope.  
-**Decision doc**: `docs/scaling-decision-phase4.md`  
-**Trigger to activate**: Monthly volume > 1M workflows (~20× Phase 3 projection) OR live Phase 4  
-remeasure NFR breach. See `docs/scaling-decision-phase4.md §Re-evaluation Trigger`.
-
-This document records the sharding design for future reference only.
-No code in this file is implemented or tested. Activate when the re-evaluation trigger fires.
-
----
-
-## Why This Document Exists
-
-`design.md §ADR-14` defers Postgres sharding to Phase 4 and requires that the
-implementation path be documented even if the decision is to defer further.
-This skeleton satisfies that requirement without introducing premature complexity.
-
----
-
-## Approach: DBOS Workflow Partitioning via Separate Postgres Instances
-
-DBOS isolates workflow durability per-database. Sharding in DBOS is achieved by
-pointing different workflow classes (or queue names) at separate Postgres instances,
-each with its own `dbos.*` system tables and `sched.*` schema.
-
-### Target Topology (when activated)
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│                    VDS Scheduler API                            │
-│                                                                 │
-│  ┌──────────────────┐      ┌──────────────────────────────┐    │
-│  │  Shard-0         │      │  Shard-1                     │    │
-│  │  high-frequency  │      │  low-frequency / ops         │    │
-│  │  queues:         │      │  queues:                     │    │
-│  │  • agent-default │      │  • ops-default               │    │
-│  │  • mem0-*        │      │  • audit-drift               │    │
-│  │  • evolution-*   │      │  • chain-*                   │    │
-│  └────────┬─────────┘      └──────────────┬───────────────┘    │
-│           │                               │                     │
-└───────────┼───────────────────────────────┼─────────────────────┘
-            │                               │
-    ┌───────▼────────┐             ┌────────▼───────┐
-    │  Postgres-0    │             │  Postgres-1    │
-    │  (SHARD_0_URL) │             │  (SHARD_1_URL) │
-    │  4vCPU / 32GB  │             │  4vCPU / 16GB  │
-    └────────────────┘             └────────────────┘
-```
-
-### Key Design Invariants
-
-1. **Each shard is an independent Postgres instance** — no cross-shard foreign keys.
-2. **`sched.schedules` is replicated to all shards** (read-only mirror; source of truth is Shard-0).
-3. **`sched.events_outbox` and `sched.dispatch_ack_log` are shard-local** — no cross-shard event routing.
-4. **Chain definitions** may reference workflows on any shard; chain fire log is global (Shard-0).
-5. **Alembic migrations must run on every shard** before the API restarts.
-
----
-
-## Implementation Steps (when activated)
-
-### Step 1 — Config layer: shard mapping
-
-Add a `ShardConfig` to `scheduler_orchestrator/config.py`:
-
-```python
-# scheduler_orchestrator/src/vds_scheduler_orchestrator/config.py
-class ShardConfig(BaseModel):
-    shard_id: int
-    database_url: SecretStr
-    queue_names: list[str]  # queues owned by this shard
-
-class SchedulerConfig(BaseSettings):
-    # ... existing fields ...
-    shards: list[ShardConfig] = []  # empty = single-shard mode (current)
-```
-
-Environment variable pattern:
-```bash
-VDS_SCHEDULER_SHARD_0_URL="postgresql://vds_sched:***@pg-shard0:5432/vds_sched"
-VDS_SCHEDULER_SHARD_0_QUEUES="agent-default,mem0-high,evolution-dispatch"
-VDS_SCHEDULER_SHARD_1_URL="postgresql://vds_sched:***@pg-shard1:5432/vds_sched"
-VDS_SCHEDULER_SHARD_1_QUEUES="ops-default,audit-drift,chain-default"
-```
-
-### Step 2 — DBOS multi-instance initialization
-
-DBOS supports multiple `DBOS` instances pointing at different databases.
-Initialize one `DBOS` instance per shard in `app.py`:
-
-```python
-# app.py (sketch — not implemented)
-from dbos import DBOS
-
-shard_instances: dict[int, DBOS] = {}
-
-@asynccontextmanager
-async def lifespan(app: FastAPI):
-    for shard in config.shards:
-        dbos = DBOS(database_url=shard.database_url.get_secret_value())
-        dbos.launch()
-        shard_instances[shard.shard_id] = dbos
-    yield
-    for dbos in shard_instances.values():
-        dbos.destroy()
-```
-
-### Step 3 — Queue-to-shard routing
-
-Add a router that maps a queue name to the owning shard's DBOS instance:
-
-```python
-# dispatcher.py (sketch — not implemented)
-def get_shard_for_queue(queue_name: str) -> DBOS:
-    for shard in config.shards:
-        if queue_name in shard.queue_names:
-            return shard_instances[shard.shard_id]
-    # Default: shard 0 (single-shard mode)
-    return shard_instances.get(0, global_dbos)
-```
-
-### Step 4 — Alembic shard-aware migration
-
-Create a `scripts/migrate-all-shards.sh` helper that runs `alembic upgrade head`
-against each `VDS_SCHEDULER_SHARD_N_URL` before the API rolls out:
-
-```bash
-#!/usr/bin/env bash
-# migrate-all-shards.sh — run Alembic on every configured shard (future)
-for i in 0 1; do
-    url_var="VDS_SCHEDULER_SHARD_${i}_URL"
-    if [[ -n "${!url_var:-}" ]]; then
-        echo "Migrating shard ${i} ..."
-        DATABASE_URL="${!url_var}" uv run alembic upgrade head
-    fi
-done
-```
-
-### Step 5 — Observability per shard
-
-Add `shard_id` label to all Prometheus metrics:
-
-```python
-vds_scheduler_workflow_duration_seconds.labels(queue_name=queue, shard_id=shard.shard_id)
-```
-
-Grafana dashboard: add a `shard_id` variable and duplicate per-queue panels.
-
-### Step 6 — DR expansion
-
-Each shard needs its own PITR drill target and WAL archive path.
-Update `scripts/dr/pitr-drill.sh` with a `SHARD_ID` parameter and a separate
-`S3_WAL_PREFIX` per shard. See `docs/dr-drill-report.md` template for the
-multi-shard evidence table.
-
----
-
-## Acceptance Criteria (when implementation is activated)
-
-| # | Criterion |
-|---|-----------|
-| AC-S1 | `scheduler_test_enqueue` routes to the correct shard based on queue name |
-| AC-S2 | DBOS workflow replay after shard-0 restart does not affect shard-1 workflows |
-| AC-S3 | Alembic runs successfully on all shards in migrate-all-shards.sh |
-| AC-S4 | Prometheus metrics carry `shard_id` label; Grafana shows per-shard breakdown |
-| AC-S5 | PITR drill succeeds independently on each shard (DR-001 extended) |
-| AC-S6 | Single-shard mode (no `VDS_SCHEDULER_SHARD_*` env vars) is unchanged |
-
----
-
-## Cost-of-Delay Assessment
-
-| Factor | Assessment |
-|--------|-----------|
-| Implementation effort | ~2 weeks (config layer + DBOS multi-init + routing + DR extension) |
-| Risk of waiting | Low — 2 500× headroom at Phase 3 projected volume |
-| Risk of acting prematurely | Medium — operational complexity; two Postgres instances to manage, back up, and monitor |
-| Decision | Defer until re-evaluation trigger fires (see `docs/scaling-decision-phase4.md`) |
-
----
-
-## References
-
-- `docs/scaling-decision-phase4.md` — Sharding decision rationale
-- `design.md §ADR-14` — Sharding deferred to Phase 4 gate
-- `design.md §Capacity & Scaling Plan` — Scale-up triggers
-- DBOS multi-database: <https://docs.dbos.dev/reference/configuration#database>
-- `docs/dr-drill-report.md` — PITR drill template (extend for multi-shard)

package/tools/vds-scripts/scheduler_orchestrator/docs/udf-enqueue-benchmark.md

@@ -1,209 +0,0 @@
-# UDF Enqueue Benchmark — TSK-15.2
-
-**Component:** `scheduler_orchestrator`
-**ADR:** ADR-16 (UDF-triggered enqueue)
-**Last updated:** 2026-04-24
-
----
-
-## Overview
-
-This document describes the methodology, expected speedup rationale, and
-interpretation guidance for the `scripts/benchmarks/udf-vs-dispatcher.sh`
-benchmark that compares two workflow-enqueue paths:
-
-| Path | Description |
-|------|-------------|
-| **UDF path** | Postgres `dbos.enqueue_workflow()` called directly from a trigger — no Python involved |
-| **Dispatcher path** | Python dispatcher calls `queue.enqueue()` → SQLAlchemy → psycopg2 → INSERT |
-
----
-
-## Measurement Methodology
-
-### Script: `scripts/benchmarks/udf-vs-dispatcher.sh`
-
-The script measures **wall-clock latency per enqueue** for N iterations (default N=100)
-using `date +%s%N` for nanosecond-resolution timing around each psql call.
-
-**UDF path measurement:**
-```sql
-SELECT dbos.enqueue_workflow(
-    workflow_name := '<wf_fqn>',
-    queue_name    := 'ops-default'
-);
-```
-One `psql` process launch per enqueue. Timing includes psql process startup +
-TCP connection to Postgres + SQL round-trip + result read.
-
-**Dispatcher path measurement (lower-bound simulation):**
-```sql
-INSERT INTO dbos.workflow_status (
-    workflow_uuid, status, name, queue_name, ...
-) VALUES (gen_random_uuid()::text, 'ENQUEUED', ...)
-ON CONFLICT DO NOTHING;
-```
-Raw INSERT that the Python dispatcher issues internally, via the same `psql`
-process model. This isolates the DB-side insertion latency without Python overhead.
-
-### Why the dispatcher measurement is a lower bound
-
-The real Python dispatcher path includes additional overhead not captured by the
-raw INSERT measurement:
-
-| Overhead source | Typical cost (LAN, local Docker) |
-|----------------|----------------------------------|
-| SQLAlchemy connection pool checkout | 0.5–2 ms |
-| psycopg2 socket send + receive | 0.5–1.5 ms |
-| `queue.enqueue()` Python frame dispatch | 0.3–0.5 ms |
-| DBOS context manager (`SetWorkflowID` etc.) | 0.2–0.4 ms |
-| **Total Python overhead** | **1.5–4.5 ms / enqueue** |
-
-The benchmark script intentionally excludes these to isolate the **database-side**
-speedup only. Real-world savings are higher.
-
----
-
-## Expected Speedup Rationale
-
-### Why UDF is faster
-
-The UDF path eliminates the Python round-trip between the dispatcher bumping
-`last_run_at` and the workflow landing in `dbos.workflow_status`.
-
-**Dispatcher path (sequential steps):**
-```
-[Python dispatcher]
-  1. Query sched.schedules WHERE next_fire_at ≤ NOW()
-  2. UPDATE sched.schedules SET last_run_at = NOW()   ← commit txn-1
-  3. RETURN to Python
-  4. Python calls queue.enqueue(workflow_fn)
-  5. Python → SQLAlchemy → psycopg2 → TCP → Postgres
-  6. INSERT INTO dbos.workflow_status                  ← commit txn-2
-```
-
-**UDF path (single transaction):**
-```
-[Python dispatcher]
-  1. Query sched.schedules WHERE next_fire_at ≤ NOW()
-  2. UPDATE sched.schedules SET last_run_at = NOW()
-     └─ TRIGGER fires dbos.enqueue_workflow_trigger()
-        └─ INSERT INTO dbos.workflow_status             ← commit txn-1 (atomic)
-```
-
-The UDF path collapses two round-trips into one. Benefits:
-- **Atomicity**: the schedule tick and the enqueue are committed atomically —
-  no window where `last_run_at` is updated but the workflow is not yet enqueued.
-- **Latency**: eliminates one full psycopg2 round-trip (~2–5 ms on LAN).
-- **Throughput**: under high schedule density, the dispatcher thread is freed
-  sooner, increasing tick throughput.
-
-### Expected benchmark results
-
-On a typical developer machine (Docker Postgres, LAN):
-
-| Metric | UDF path | Dispatcher simulation | Real dispatcher |
-|--------|----------|-----------------------|----------------|
-| Avg latency/enqueue | 3–6 ms | 4–8 ms | 6–12 ms |
-| Speedup vs simulation | ~1.2–1.5× | — | — |
-| Speedup vs real dispatcher | ~2–3× | — | — |
-| 100-enqueue total | 300–600 ms | 400–800 ms | 600–1200 ms |
-
-*Actual numbers vary with hardware, Docker networking, and Postgres load.*
-
----
-
-## Running the Benchmark
-
-### Prerequisites
-
-- Postgres running and accessible via `DATABASE_URL`
-- DBOS migration applied (`DBOS.launch()` run at least once against the DB)
-- Scheduler migration `e5f6a7b8c9d0` applied (`use_udf_enqueue` column exists)
-- `psql` and `bc` on PATH
-
-### Quick run
-
-```bash
-# Minimal — 100 enqueues per path:
-DATABASE_URL="postgresql://user:pass@localhost:5432/vds_scheduler" \
-  bash scripts/benchmarks/udf-vs-dispatcher.sh
-
-# Verbose — shows per-enqueue timing:
-DATABASE_URL="..." BENCH_VERBOSE=1 \
-  bash scripts/benchmarks/udf-vs-dispatcher.sh
-
-# Custom N and queue:
-DATABASE_URL="..." BENCH_N=500 BENCH_QUEUE=agent-default \
-  bash scripts/benchmarks/udf-vs-dispatcher.sh
-```
-
-### Sample output
-
-```
-VDS Scheduler — UDF vs Dispatcher Enqueue Benchmark
-Date: 2026-04-24T20:00:00Z
-DB:   postgresql://vds:***@localhost:5432/vds_scheduler
-────────────────────────────────────────────────────────────────
-Checking prerequisites...
-Prerequisites OK.
-
-=== PATH 1: UDF direct enqueue (dbos.enqueue_workflow) ===
-    N=100  queue=ops-default
-────────────────────────────────────────────────────────────────
-  Total: 412 ms | Avg: 4.12 ms | Min: 3 ms | Max: 9 ms
-
-=== PATH 2: Dispatcher path (raw INSERT — lower-bound simulation) ===
-    N=100  queue=ops-default
-────────────────────────────────────────────────────────────────
-  Total: 567 ms | Avg: 5.67 ms | Min: 4 ms | Max: 11 ms
-
-────────────────────────────────────────────────────────────────
-SUMMARY — UDF vs Dispatcher enqueue (N=100)
-────────────────────────────────────────────────────────────────
-  UDF path (direct SQL):           4.12 ms avg  |     3 ms min  |     9 ms max
-  Dispatcher (raw INSERT):         5.67 ms avg  |     4 ms min  |    11 ms max
-
-  Speedup (dispatcher / UDF):  1.376x
-  Saved over 100 enqueues: 155 ms total
-
-NOTE: Dispatcher simulation EXCLUDES Python overhead (pool checkout,
-  SQLAlchemy ORM, queue.enqueue() dispatch ~2-5 ms/call). Real-world
-  speedup is higher. See docs/udf-enqueue-benchmark.md.
-────────────────────────────────────────────────────────────────
-```
-
----
-
-## Caveats
-
-1. **psql process-per-call overhead**: The script spawns a new `psql` process for
-   each enqueue. In production, the Python dispatcher reuses a persistent
-   connection pool — this adds startup overhead to both benchmark paths equally,
-   not affecting the relative speedup ratio.
-
-2. **Lower-bound dispatcher**: The raw INSERT does not include Python overhead
-   (see table above). True dispatcher latency is 1.5–4.5 ms higher per enqueue.
-
-3. **UDF is not always faster in absolute terms**: On a remote database (WAN),
-   both paths are dominated by network RTT. The Python overhead becomes
-   proportionally smaller and the speedup ratio shrinks.
-
-4. **Atomicity trade-off**: UDF mode commits the enqueue in the same transaction
-   as `last_run_at`. If the transaction rolls back (e.g., constraint violation),
-   the enqueue is also rolled back — consistent behavior. The dispatcher path
-   commits `last_run_at` first, then enqueues — a crash between the two leaves
-   the schedule "ticked" but not enqueued, requiring the janitor loop to
-   reconcile.  UDF mode is therefore more consistent, not just faster.
-
-5. **Not compatible with deduplication_id**: `dbos.enqueue_workflow_trigger()`
-   does not pass a `deduplication_id`. Schedules that rely on DBOS deduplication
-   (e.g., idempotent re-trigger on crash recovery) should remain on the
-   dispatcher path (`use_udf_enqueue = false`).
-
----
-
-## ADR-16 Reference
-
-See the spec at `.gpt-5.4/specs/vds-scheduler/` for the full ADR-16
-design rationale, migration gate checklist, and operator rollout procedure.