@mndrk/agx 2.0.25 → 2.0.32

package/cloud-runtime/standalone/docs/migration-spec.md

@@ -1,345 +0,0 @@
-# AGX Cloud — SQLite Migration Specification
-
-## §0.1 SQLite Runtime & Environment Contract
-
-### 0.1.1 Minimum SQLite Version
-
-| Requirement | Min Version | Reason |
-|---|---|---|
-| UPSERT (`ON CONFLICT … DO UPDATE`) | 3.24.0 | Idempotent writes |
-| `RETURNING` clause | 3.35.0 | Avoid extra SELECT after INSERT |
-| JSON operators (`->`, `->>`) | 3.38.0 | Ergonomic JSON queries |
-
-**Effective minimum: 3.35.0**
-
-The project uses `better-sqlite3@^12.6.2` which bundles **SQLite 3.51.2** — well above the floor. The bundled SQLite is the *only* supported runtime; system-installed SQLite is never used at query time.
-
-### 0.1.2 Required Compile-Time Extensions
-
-All of these are enabled in the `better-sqlite3` bundled build (verified in `deps/defines.gypi`):
-
-| Extension | Compile Flag | Used For |
-|---|---|---|
-| JSON1 | `SQLITE_ENABLE_JSON1` | JSONB columns, `json_extract()`, `->` / `->>` |
-| FTS5 | `SQLITE_ENABLE_FTS5` | Full-text search (future: task search) |
-| GEOPOLY | `SQLITE_ENABLE_GEOPOLY` | Reserved (not currently required) |
-| Math functions | `SQLITE_ENABLE_MATH_FUNCTIONS` | Reserved |
-
-### 0.1.3 Required PRAGMAs
-
-Every database connection **MUST** execute these PRAGMAs before any application SQL:
-
-```sql
-PRAGMA journal_mode = WAL;          -- write-ahead logging for concurrent reads
-PRAGMA synchronous = NORMAL;        -- safe with WAL; full durability on checkpoint
-PRAGMA foreign_keys = ON;           -- enforce FK constraints at runtime
-PRAGMA busy_timeout = 5000;         -- wait up to 5 s for writer lock
-PRAGMA cache_size = -16000;         -- 16 MB page cache (negative = KiB)
-```
-
-**Rationale for each value:**
-
-| PRAGMA | Value | Why |
-|---|---|---|
-| `journal_mode` | `WAL` | Enables concurrent readers + single writer. Required for dashboard read paths while worker writes. |
-| `synchronous` | `NORMAL` | With WAL, `NORMAL` guarantees durability on checkpoint. `FULL` adds ~2× write latency for marginal benefit on local SSD. |
-| `foreign_keys` | `ON` | SQLite defaults to OFF. Without this, FK constraints are silently ignored. |
-| `busy_timeout` | `5000` | Prevents immediate `SQLITE_BUSY` errors under concurrent access. 5 s is generous for local disk. |
-| `cache_size` | `-16000` | 16 MB. Matches the `better-sqlite3` compiled default (`SQLITE_DEFAULT_CACHE_SIZE=-16000`). |
-
-### 0.1.4 Startup Validation
-
-On application boot, the database initialization function **MUST** verify:
-
-1. **SQLite version** — `SELECT sqlite_version()` ≥ `3.35.0`. Fail fast with a clear error if not met.
-2. **JSON1** — `SELECT json('{}')` succeeds.
-3. **FTS5** — `CREATE VIRTUAL TABLE IF NOT EXISTS _fts5_check USING fts5(x)` succeeds, then `DROP TABLE _fts5_check`.
-4. **WAL mode active** — `PRAGMA journal_mode` returns `'wal'`.
-5. **Foreign keys active** — `PRAGMA foreign_keys` returns `1`.
-
-If any check fails, the process **MUST** exit with a non-zero code and a human-readable diagnostic.
-
-### 0.1.5 Version Pinning
-
-| Environment | How SQLite Version is Controlled |
-|---|---|
-| **Dev** | `better-sqlite3@^12.6.2` in `package.json` (bundles SQLite 3.51.2) |
-| **CI** | Same `package.json` + `npm ci` — deterministic via `package-lock.json` |
-| **Prod** | Same `package.json` + `npm ci` — identical binary |
-
-The `better-sqlite3` package compiles SQLite from source at install time, so the version is locked by the npm package version, **not** by the host OS.
-
-**Pin rule:** `better-sqlite3` patch version **MUST** be locked in `package-lock.json`. Major/minor bumps require explicit review of the bundled SQLite version.
-
-### 0.1.6 Filesystem Requirements
-
-| Requirement | Rationale |
-|---|---|
-| **Local block device (SSD)** | WAL mode requires shared-memory (`-shm` file) backed by `mmap()`. Network filesystems (NFS, SMB/CIFS, EFS, GlusterFS) do not guarantee POSIX `fcntl()` locking semantics and **will corrupt the database**. |
-| **No network filesystems** | NFS/SMB/EFS are **banned** for the primary DB file, WAL file, and SHM file. |
-| **Validated `fsync`** | The filesystem must honor `fsync()` / `fdatasync()`. Disable write-back caching on the block device or use a battery-backed controller. |
-| **Same directory** | The `-wal` and `-shm` files **MUST** reside in the same directory as the main `.db` file (SQLite default). Do not symlink them to different mounts. |
-
-### 0.1.7 WAL-Safe Backup & Restore
-
-#### Online Backup Flow
-
-Use the SQLite Online Backup API (exposed via `better-sqlite3`'s `db.backup(destination)` method):
-
-```typescript
-await sourceDb.backup(`${backupDir}/agx-queue-${timestamp}.db`);
-```
-
-This produces a consistent snapshot without stopping writes. The backup destination **MUST** also be on local disk.
-
-#### Checkpoint Policy
-
-| Event | Action |
-|---|---|
-| Graceful shutdown | `PRAGMA wal_checkpoint(TRUNCATE)` — merges WAL into main DB and truncates WAL file to zero bytes. |
-| Periodic (every 5 min while idle) | `PRAGMA wal_checkpoint(PASSIVE)` — non-blocking, best-effort. |
-| WAL size > 50 MB | `PRAGMA wal_checkpoint(RESTART)` — blocks new writers briefly, resets WAL. |
-
-#### Restore Procedure
-
-1. Stop all processes accessing the database.
-2. Delete or move aside the existing `.db`, `-wal`, and `-shm` files.
-3. Copy the backup `.db` file into place.
-4. Run `PRAGMA integrity_check` on the restored file — abort if result is not `ok`.
-5. Start the application — it will create fresh `-wal` and `-shm` files.
-6. Verify with startup checks (§0.1.4).
-
-**Restore-time objective (RTO):** < 60 seconds for databases up to 1 GB (local SSD copy speed).
-
-### 0.1.8 Current Codebase Compliance
-
-Files that open SQLite connections today:
-
-| File | WAL | FK | busy_timeout | Status |
-|---|---|---|---|---|
-| `lib/queue/sqlite-adapter.ts` | Yes | — | — | **Non-compliant** (missing FK, busy_timeout) |
-| `lib/attachment-store.ts` | Yes | — | — | **Non-compliant** |
-| `lib/participants-store.ts` | Yes | — | — | **Non-compliant** |
-| `lib/history-store.ts` | Yes | — | — | **Non-compliant** |
-| `scripts/bench-queue.ts` | Yes | — | — | Script only |
-
-**Action required:** Centralize PRAGMA application into a single `openDatabase()` helper and migrate all call sites.
-
-## §0.2 Type-Semantics Contract
-
-SQLite uses dynamic typing with type affinities. All PG-to-SQLite type mappings must follow these conventions to ensure consistent behavior across the application.
-
-### 0.2.1 Mapping Table
-
-| Type (PG) | SQLite Affinity | Encoding / Convention | Category |
-|-----------|----------------|----------------------|----------|
-| `uuid` | `TEXT` | Lowercase hex with hyphens (`xxxxxxxx-xxxx-…`) | direct |
-| `timestamptz` | `TEXT` | ISO 8601 with `Z` suffix, always UTC | needs convention |
-| `timestamp` | `TEXT` | ISO 8601 without offset (local/unzoned) | needs convention |
-| `date` | `TEXT` | `YYYY-MM-DD` | needs convention |
-| `time` | `TEXT` | `HH:MM:SS` | needs convention |
-| `interval` | `TEXT` | ISO 8601 duration (`P1DT2H3M`) or app-level integer seconds | needs convention |
-| `boolean` | `INTEGER` | `0` / `1` only | needs convention |
-| `numeric(p,s)` | `REAL` | IEEE 754 double; accept precision loss (see §0.2.4) | lossy |
-| `integer` | `INTEGER` | Same; includes `pid`, `exit_code`, `priority`, `version`, `position`, token counts | direct |
-| `text` | `TEXT` | Same | direct |
-| `jsonb` | `TEXT` | `JSON.stringify` on write, `JSON.parse` on read. Use `json_valid()` CHECK where feasible | needs convention |
-| `uuid[]` | `TEXT` | JSON array of UUID strings: `["uuid1","uuid2"]` | needs convention |
-| `text[]` | `TEXT` | JSON array of strings: `["a","b"]` | needs convention |
-| `inet` | `TEXT` | Store as string, no validation at DB level | lossy |
-| `enum` | `TEXT` | `CHECK (col IN ('val1','val2',…))` | needs convention |
-| `serial` / `bigserial` | `INTEGER` | `INTEGER PRIMARY KEY` | needs convention |
-
-### 0.2.2 Timestamps & Timezones
-
-PG `timestamptz` stores instants in UTC internally and renders in the session timezone. SQLite has no date type — all temporal values are stored as `TEXT`.
-
-#### Storage Format Table
-
-| PG Type | SQLite Type | Format | Example |
-|---------|-------------|--------|---------|
-| `TIMESTAMPTZ` | `TEXT` | ISO 8601 with `Z` suffix (always UTC) | `2026-02-21T14:30:00.000Z` |
-| `TIMESTAMP` | `TEXT` | ISO 8601 without offset | `2026-02-21T14:30:00.000` |
-| `DATE` | `TEXT` | `YYYY-MM-DD` | `2026-02-21` |
-| `TIME` | `TEXT` | `HH:MM:SS` | `14:30:00` |
-| `INTERVAL` | `TEXT` | ISO 8601 duration or integer seconds | `P1DT2H30M` or `95400` |
-
-#### UTC Normalization Rule
-
-**All `TIMESTAMPTZ` values MUST be normalized to UTC before storage.** The application layer is responsible for this — SQLite has no timezone awareness.
-
-- **On write**: Call `new Date(val).toISOString()` which always produces a UTC `Z`-suffixed string.
-- **On read**: Parse with `new Date(text_value)` which correctly interprets `Z`-suffixed strings as UTC.
-- **Never store**: local-time strings, bare offsets like `+03:00`, or Unix epoch integers.
-- **Precision**: Milliseconds (3 fractional digits). PG supports microseconds; we truncate. Acceptable since no column requires sub-ms precision.
-
-#### Example: Correct INSERT/SELECT Encoding
-
-```sql
--- INSERT: application code prepares the value
--- JS: const now = new Date().toISOString();  // "2026-02-21T14:30:00.123Z"
-INSERT INTO agx_tasks (slug, created_at, updated_at)
-VALUES ('my-task', '2026-02-21T14:30:00.123Z', '2026-02-21T14:30:00.123Z');
-
--- SELECT: values come back as TEXT, parse in app layer
--- JS: const createdAt = new Date(row.created_at);
-SELECT slug, created_at, updated_at FROM agx_tasks WHERE slug = 'my-task';
-
--- Filtering by date range works with string comparison
-SELECT * FROM agx_tasks
-WHERE created_at >= '2026-02-01T00:00:00.000Z'
-  AND created_at <  '2026-03-01T00:00:00.000Z';
-```
-
-#### String Ordering Guarantee
-
-ISO 8601 UTC strings are lexicographically sortable in chronological order, so `ORDER BY created_at ASC` works correctly in SQLite without date functions.
-
-#### SQLite Date/Time Function Compatibility
-
-SQLite built-in date/time functions (`datetime()`, `strftime()`, `date()`, `time()`, `julianday()`) understand ISO 8601 input and can be used in queries when needed:
-
-```sql
--- Extract date part
-SELECT strftime('%Y-%m-%d', created_at) AS day FROM agx_tasks;
-
--- Compute age in seconds
-SELECT (julianday('now') - julianday(created_at)) * 86400 AS age_seconds
-FROM agx_tasks;
-
--- Filter: tasks created in the last 24 hours
-SELECT * FROM agx_tasks
-WHERE created_at > strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day');
-```
-
-**Prefer application-level date math** over SQLite date functions for consistency, but these functions are available for ad-hoc queries and migration scripts.
-
-### 0.2.3 Booleans
-
-PG `boolean` is a true type with `TRUE`/`FALSE` literals. SQLite has no boolean type.
-
-- **Storage**: `INTEGER` with values `0` (false) and `1` (true) only.
-- **CHECK constraints**: Add `CHECK (col IN (0, 1))` on boolean columns (`is_default`, `is_public`, `swarm`, `had_graph_before`).
-- **Writes**: Application code must coerce JS booleans → `value ? 1 : 0` before binding.
-- **Reads**: Coerce back → `Boolean(row.col)` or `row.col === 1`. Note: `Boolean(0)` returns `false` in JS, so `Boolean(row.col)` works correctly.
-- **Never store**: string `"true"`/`"false"`, or `NULL` for false. Columns with `DEFAULT false` become `DEFAULT 0`.
-
-### 0.2.4 Numeric Precision
-
-PG `numeric(p,s)` is exact arbitrary precision. SQLite `REAL` is IEEE 754 64-bit float.
-
-- **Affected column**: `task_costs.estimated_cost` — stores per-request LLM cost estimates.
-- **Precision loss**: Float64 has ~15 significant digits, so values up to 999999.999999 are representable without meaningful loss. Acceptable for cost tracking (sub-cent precision preserved).
-- **Alternative considered**: Store as `TEXT` for exact decimal fidelity. Rejected because cost aggregation (`SUM()`) requires numeric type, and the precision loss is negligible for this use case.
-- **Application enforcement**: Use `parseFloat()` on read. For display, always `toFixed(6)` to match the original PG scale.
-
-### 0.2.5 NULL Sorting Behavior
-
-- **PG default**: NULLs sort **last** in ASC, **first** in DESC (`NULLS LAST` for ASC is the default).
-- **SQLite default**: NULLs sort **first** in ASC, **last** in DESC (smallest value).
-- **Impact**: Queries using `ORDER BY` on nullable columns (`claimed_by`, `project_id`, `error`, `exit_code`, `workflow_id`, etc.) will return NULLs in a different position.
-- **Mitigation — preferred**: Use explicit `NULLS FIRST` / `NULLS LAST` (supported in SQLite ≥ 3.30.0, bundled version is 3.51.2):
-  ```sql
-  SELECT * FROM agx_tasks ORDER BY project_id ASC NULLS LAST;
-  ```
-- **Mitigation — legacy fallback** (SQLite < 3.30):
-  ```sql
-  SELECT * FROM agx_tasks
-  ORDER BY CASE WHEN project_id IS NULL THEN 1 ELSE 0 END, project_id ASC;
-  ```
-- **Recommendation**: Audit `ORDER BY` clauses on nullable columns during Phase 2 (Query Compatibility Matrix) and add explicit NULL ordering where user-visible sort order could differ.
-
-### 0.2.6 Application Layer Conventions
-
-All type coercion between PG semantics and SQLite storage **MUST** be handled in a single shared module — the **type-coercion layer** — rather than scattered across individual query sites.
-
-#### Coercion Layer Responsibility
-
-The coercion layer lives in `lib/db/type-coerce.ts` and is invoked by the `DbAdapter` on every write (serialize) and read (deserialize). No query-building code outside this module should perform type conversions.
-
-| Responsibility | Where |
-|---|---|
-| Serialize JS values → SQLite binding values | `serialize(tableName, columnName, value)` |
-| Deserialize SQLite result rows → JS types | `deserialize(tableName, columnName, value)` |
-| Column type registry (which columns need coercion) | Static map derived from schema metadata |
-
-The adapter calls `serialize` before every `INSERT`/`UPDATE` binding and `deserialize` on every result row. This ensures a single code path regardless of whether the caller is a REST handler, a queue worker, or a migration script.
-
-#### Round-Trip Invariants
-
-Every type mapping in §0.2.1 **MUST** satisfy this invariant:
-
-> For any valid PG value `v` of type `T`, writing `v` through the serialize path and reading it back through the deserialize path produces a value equal to `v` (within documented precision bounds).
-
-Concrete invariants per type family:
-
-| Type Family | Serialize | Deserialize | Round-Trip Equality |
-|---|---|---|---|
-| **Temporal** (`timestamptz`) | `new Date(v).toISOString()` → `TEXT` | `new Date(text)` → `Date` | `original.getTime() === roundTripped.getTime()` (ms precision) |
-| **Temporal** (`timestamp`, `date`, `time`) | String format as §0.2.2 → `TEXT` | Parse back to string | Exact string equality |
-| **Temporal** (`interval`) | ISO 8601 duration string → `TEXT` | Parse back to string | Exact string equality |
-| **Boolean** | `v ? 1 : 0` → `INTEGER` | `v === 1` → `boolean` | `original === roundTripped` |
-| **Numeric** (`numeric(p,s)`) | `Number(v)` → `REAL` | `parseFloat(text)` → `number` | `Math.abs(original - roundTripped) < 1e-9` |
-| **UUID** | Lowercase string → `TEXT` | String → `string` | Exact string equality |
-| **JSON** (`jsonb`) | `JSON.stringify(v)` → `TEXT` | `JSON.parse(text)` → `object` | `deepEqual(original, roundTripped)` |
-| **Arrays** (`uuid[]`, `text[]`) | `JSON.stringify(v)` → `TEXT` | `JSON.parse(text)` → `array` | `deepEqual(original, roundTripped)` |
-| **Enum** | String pass-through → `TEXT` | String → `string` | Exact string equality |
-| **Text / Integer** | Pass-through | Pass-through | Exact equality |
-
-#### Testing Strategy
-
-Type coercion correctness **MUST** be verified with round-trip tests — one test per type family listed above.
-
-**Test structure** (property-based where applicable):
-
-```typescript
-// __tests__/db/type-coerce.roundtrip.test.ts
-
-describe('type coercion round-trip', () => {
-  // One test per type family
-  test.each([
-    ['timestamptz', new Date('2026-02-21T14:30:00.123Z')],
-    ['timestamp',   '2026-02-21T14:30:00.000'],
-    ['date',        '2026-02-21'],
-    ['time',        '14:30:00'],
-    ['interval',    'P1DT2H30M'],
-    ['boolean',     true],
-    ['boolean',     false],
-    ['numeric',     123.456789],
-    ['uuid',        'a1b2c3d4-e5f6-7890-abcd-ef1234567890'],
-    ['jsonb',       { key: 'value', nested: [1, 2, 3] }],
-    ['uuid[]',      ['uuid-1', 'uuid-2']],
-    ['text[]',      ['hello', 'world']],
-    ['enum',        'pending'],
-    ['text',        'plain string'],
-    ['integer',     42],
-  ])('%s round-trips correctly: %p', (typeName, original) => {
-    const serialized = serialize('test_table', typeName, original);
-    const deserialized = deserialize('test_table', typeName, serialized);
-    expectRoundTripEqual(typeName, original, deserialized);
-  });
-});
-```
-
-**Property-based tests** (recommended for temporal and numeric families):
-
-- **Temporal**: Generate random `Date` objects within a wide range (1970–2100). Assert `getTime()` equality after round-trip.
-- **Numeric**: Generate random floats with up to 6 decimal places. Assert the round-trip value is within `1e-9` of the original.
-- **Boolean**: Exhaustive — only two values, both must round-trip.
-- **JSON**: Generate random nested objects (depth ≤ 3). Assert `deepEqual` after round-trip.
-
-**Integration tests** (in the parity test harness, Phase 4):
-
-- For each type family, INSERT a row into the real SQLite database via the adapter, SELECT it back, and verify the round-trip invariant holds end-to-end (not just at the coercion module level).
-
-#### Edge Cases to Document and Test
-
-| Edge Case | Expected Behavior |
-|---|---|
-| `NULL` values for any type | Serialize → `null`, deserialize → `null`. No coercion applied. |
-| Empty string vs `NULL` | Preserved as-is. `''` does not become `NULL`. |
-| Empty JSON array `[]` | Stored as `'[]'`, not `NULL`. |
-| `timestamptz` at epoch (`1970-01-01T00:00:00.000Z`) | Must round-trip correctly, not become `NULL` or `0`. |
-| Boolean `false` (falsy in JS) | Stored as `0`, not `NULL`. `deserialize` must return `false`, not `undefined`. |
-| Numeric `0` and `0.000000` | Both stored as `REAL 0.0`, read back as `0`. |
-| Unicode in text/JSON | Preserved through `JSON.stringify`/`JSON.parse`. No encoding loss. |