@mndrk/agx 2.0.26 → 2.0.33

package/cloud-runtime/standalone/docs/pg-sqlite-compatibility-matrix.md

@@ -1,554 +0,0 @@
-# PostgreSQL → SQLite Compatibility Matrix
-
-Every Postgres-specific SQL pattern in agx-cloud, with its SQLite rewrite.
-
----
-
-## 1. ILIKE → LIKE COLLATE NOCASE
-
-**Locations:** `lib/db-adapter.ts:233`, `src/graph/migration-job.ts:161`, `mcp/src/tools/learnings.ts:95`
-
-### Before (PG)
-```sql
-SELECT * FROM tasks WHERE title ILIKE '%deploy%';
-```
-
-### After (SQLite)
-```sql
-SELECT * FROM tasks WHERE title LIKE '%deploy%' COLLATE NOCASE;
-```
-
-**Notes:** SQLite's `LIKE` is case-insensitive for ASCII by default when no collation is set, but explicit `COLLATE NOCASE` guarantees it. For programmatic queries in `db-adapter.ts`, replace the `ILIKE` operator branch with `LIKE ... COLLATE NOCASE`.
-
----
-
-## 2. @> JSON Containment → json_each()
-
-**Locations:** `lib/db-adapter.ts:275`
-
-### Before (PG)
-```sql
--- Check if tags array contains 'urgent'
-SELECT * FROM tasks WHERE metadata @> '["urgent"]'::jsonb;
-
--- Check if labels column contains specific values
-SELECT * FROM tasks WHERE labels @> ARRAY['bug','critical'];
-```
-
-### After (SQLite)
-```sql
--- Using json_each to check containment
-SELECT DISTINCT t.* FROM tasks t, json_each(t.metadata) je
-WHERE je.value = 'urgent';
-
--- Multiple containment checks
-SELECT t.* FROM tasks t
-WHERE (SELECT COUNT(*) FROM json_each(t.labels) je
-       WHERE je.value IN ('bug','critical')) = 2;
-```
-
-**Notes:** For the `contains` operator in `db-adapter.ts`, emit a subquery using `json_each()` instead of `@>`. If checking a single value, a simpler form works:
-```sql
-SELECT * FROM tasks WHERE EXISTS (
-  SELECT 1 FROM json_each(tasks.labels) WHERE value = 'urgent'
-);
-```
-
----
-
-## 3. RETURNING → Separate SELECT
-
-**Locations:** `lib/db-adapter.ts:398`, `mcp/src/db.ts:159`, `src/graph/migration-job.ts:361`, `src/graph/store.ts:251,436,520`
-
-### Before (PG)
-```sql
--- INSERT RETURNING
-INSERT INTO agx.execution_graphs (id, task_id, nodes, edges)
-VALUES ($1, $2, $3::jsonb, $4::jsonb)
-RETURNING *;
-
--- UPDATE RETURNING
-UPDATE agx.execution_graphs
-SET updated_at = NOW()
-WHERE id = $1
-RETURNING graph_version, updated_at;
-
--- DELETE RETURNING
-DELETE FROM agx.execution_graphs
-WHERE task_id = $1
-RETURNING id;
-```
-
-### After (SQLite)
-```sql
--- INSERT + SELECT (use last_insert_rowid() or known id)
-INSERT INTO execution_graphs (id, task_id, nodes, edges)
-VALUES (?, ?, json(?), json(?));
-SELECT * FROM execution_graphs WHERE id = ?;
-
--- UPDATE + SELECT
-UPDATE execution_graphs
-SET updated_at = datetime('now')
-WHERE id = ?;
-SELECT graph_version, updated_at FROM execution_graphs WHERE id = ?;
-
--- DELETE: capture ids first, then delete
-SELECT id FROM execution_graphs WHERE task_id = ?;
-DELETE FROM execution_graphs WHERE task_id = ?;
-```
-
-**Notes:** Wrap INSERT+SELECT or DELETE(select)+DELETE in a transaction to ensure atomicity. SQLite supports `RETURNING` since version 3.35.0 (2021-03-12) — if targeting modern SQLite, `RETURNING` works as-is. Check your runtime version.
-
----
-
-## 4. Array Types → JSON Arrays
-
-**Locations:** `001_agx_board_schema.sql:123,261`, `src/graph/store.ts:377`, `src/graph/migration-job.ts:137,149,258,312`
-
-### Before (PG)
-```sql
--- Schema
-outputs text[],
-depends_on uuid[] DEFAULT '{}'::uuid[],
-
--- Query with ANY()
-SELECT * FROM graph_nodes
-WHERE graph_id = $1 AND node_id = ANY($2::text[]);
-
--- Query with array parameter
-SELECT * FROM tasks WHERE id = ANY($1::uuid[]);
-```
-
-### After (SQLite)
-```sql
--- Schema: store as JSON text
-outputs TEXT DEFAULT '[]',  -- JSON array
-depends_on TEXT DEFAULT '[]',  -- JSON array of uuids
-
--- Query: replace ANY() with json_each subquery
-SELECT * FROM graph_nodes
-WHERE graph_id = ? AND node_id IN (SELECT value FROM json_each(?));
-
--- Same pattern for uuid arrays
-SELECT * FROM tasks WHERE id IN (SELECT value FROM json_each(?));
-```
-
-**Notes:** Pass JSON-encoded arrays (`'["a","b","c"]'`) as parameters. The `json_each()` table-valued function unpacks them for `IN` clauses.
-
----
-
-## 5. increment_version Stored Procedure → Application-Level CAS
-
-**Locations:** `001_agx_board_schema.sql:43-50,551-558` (triggers), `store.ts` (relies on auto-incremented versions)
-
-### Before (PG)
-```sql
--- Trigger function
-CREATE OR REPLACE FUNCTION agx.increment_task_version()
-RETURNS trigger LANGUAGE plpgsql AS $$
-BEGIN
-  NEW.version = OLD.version + 1;
-  RETURN NEW;
-END;
-$$;
-
-CREATE TRIGGER trg_increment_task_version
-BEFORE UPDATE ON agx.tasks
-FOR EACH ROW EXECUTE FUNCTION agx.increment_task_version();
-
--- Usage: just UPDATE, version increments automatically
-UPDATE agx.tasks SET title = 'new title' WHERE id = $1;
-```
-
-### After (SQLite)
-```sql
--- Option A: SQLite trigger (closest equivalent)
-CREATE TRIGGER trg_increment_task_version
-BEFORE UPDATE ON tasks
-FOR EACH ROW
-BEGIN
-  UPDATE tasks SET version = OLD.version + 1 WHERE id = NEW.id;
-END;
-
--- Option B: Application-level compare-and-swap (recommended)
--- Read current version, then update with version check:
-UPDATE tasks
-SET title = ?, version = version + 1
-WHERE id = ? AND version = ?;
--- If rows_affected = 0, conflict detected → retry or error
-```
-
-**Notes:** Option B (CAS) is preferred because it provides explicit optimistic concurrency control. The caller reads the current version, attempts the update with `AND version = ?`, and retries on conflict. This avoids hidden trigger behavior and makes concurrency explicit.
-
-The same pattern applies to `increment_graph_version`:
-```sql
-UPDATE execution_graphs
-SET graph_version = graph_version + 1, nodes = json(?)
-WHERE id = ? AND graph_version = ?;
-```
-
----
-
-## 6. update_updated_at Trigger → SQLite Trigger or Application Code
-
-**Locations:** `001_agx_board_schema.sql:52-59`
-
-### Before (PG)
-```sql
-CREATE OR REPLACE FUNCTION agx.update_updated_at()
-RETURNS trigger LANGUAGE plpgsql AS $$
-BEGIN
-  NEW.updated_at = now();
-  RETURN NEW;
-END;
-$$;
-
-CREATE TRIGGER trg_user_settings_updated_at
-BEFORE UPDATE ON agx.user_settings
-FOR EACH ROW EXECUTE FUNCTION agx.update_updated_at();
-```
-
-### After (SQLite)
-```sql
-CREATE TRIGGER trg_user_settings_updated_at
-AFTER UPDATE ON user_settings
-FOR EACH ROW
-BEGIN
-  UPDATE user_settings SET updated_at = datetime('now')
-  WHERE id = NEW.id;
-END;
-```
-
-**Notes:** SQLite triggers use `BEGIN...END` blocks instead of PL/pgSQL functions. Alternatively, set `updated_at` in application code on every UPDATE.
-
----
-
-## 7. ON CONFLICT (Upsert) Differences
-
-**Locations:** `lib/db-adapter.ts:376-395`, `mcp/src/db.ts:563`, `src/graph/store.ts:247`, `src/graph/migration-job.ts:263,389`
-
-### Before (PG)
-```sql
-INSERT INTO agx.rate_limits (user_id, endpoint, window_start, request_count)
-VALUES ($1, $2, $3, 1)
-ON CONFLICT (user_id, endpoint, window_start)
-DO UPDATE SET request_count = rate_limits.request_count + 1
-RETURNING request_count;
-```
-
-### After (SQLite)
-```sql
-INSERT INTO rate_limits (user_id, endpoint, window_start, request_count)
-VALUES (?, ?, ?, 1)
-ON CONFLICT (user_id, endpoint, window_start)
-DO UPDATE SET request_count = rate_limits.request_count + 1;
--- Then SELECT if needed:
-SELECT request_count FROM rate_limits
-WHERE user_id = ? AND endpoint = ? AND window_start = ?;
-```
-
-**Notes:** SQLite supports `ON CONFLICT ... DO UPDATE SET` and `DO NOTHING` with the same syntax as PG. The key differences:
-- Replace `EXCLUDED` with `excluded` (SQLite uses lowercase, but is case-insensitive so both work)
-- Remove `RETURNING` (see section 3)
-- Use `?` placeholders instead of `$1`, `$2`
-
----
-
-## 8. gen_random_uuid() → Application-Generated UUIDs
-
-**Locations:** `001_agx_board_schema.sql` (19 table definitions)
-
-### Before (PG)
-```sql
-id uuid DEFAULT gen_random_uuid() NOT NULL PRIMARY KEY,
-```
-
-### After (SQLite)
-```sql
--- Option A: Use SQLite's randomblob to generate UUID
-id TEXT NOT NULL PRIMARY KEY DEFAULT (
-  lower(hex(randomblob(4))) || '-' ||
-  lower(hex(randomblob(2))) || '-4' ||
-  substr(lower(hex(randomblob(2))),2) || '-' ||
-  substr('89ab', abs(random()) % 4 + 1, 1) ||
-  substr(lower(hex(randomblob(2))),2) || '-' ||
-  lower(hex(randomblob(6)))
-),
-
--- Option B (recommended): Generate UUID in application code
-id TEXT NOT NULL PRIMARY KEY,
--- Application generates: crypto.randomUUID() before INSERT
-```
-
-**Notes:** Option B is simpler and more portable. Generate UUIDs in TypeScript with `crypto.randomUUID()` and pass them as parameters.
-
----
-
-## 9. NOW() / CURRENT_TIMESTAMP → datetime('now')
-
-**Locations:** Throughout schema and queries
-
-### Before (PG)
-```sql
-created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
-updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW() NOT NULL,
-
--- In queries
-UPDATE agx.execution_graphs SET updated_at = NOW() WHERE id = $1;
-```
-
-### After (SQLite)
-```sql
-created_at TEXT DEFAULT (datetime('now')),
-updated_at TEXT DEFAULT (datetime('now')) NOT NULL,
-
--- In queries
-UPDATE execution_graphs SET updated_at = datetime('now') WHERE id = ?;
-```
-
-**Notes:** SQLite stores timestamps as TEXT in ISO-8601 format. Use `datetime('now')` for UTC. For comparison operations, ISO-8601 strings sort correctly as text.
-
----
-
-## 10. Type Casts (::type) → Removed or json()
-
-**Locations:** `src/graph/store.ts`, `src/graph/migration-job.ts`
-
-### Before (PG)
-```sql
-$5::jsonb, $6::jsonb, $7::timestamptz
-$2::text[]
-$1::uuid[]
-```
-
-### After (SQLite)
-```sql
-json(?), json(?), ?
--- Array casts: pass JSON string, use json_each() for queries
--- UUID/text casts: just use ? (SQLite is dynamically typed)
-```
-
-**Notes:** SQLite is dynamically typed. Remove all `::type` casts. For JSONB columns, wrap with `json()` to validate/normalize JSON. For arrays, pass JSON strings.
-
----
-
-## 11. INTERVAL / date_trunc / extract → SQLite Date Functions
-
-**Locations:** `001_agx_board_schema.sql:31` (check_rate_limit function)
-
-### Before (PG)
-```sql
-v_window_start := date_trunc('minute', now()) -
-  ((extract(minute from now())::int % (p_window_seconds / 60)) || ' minutes')::interval;
-```
-
-### After (SQLite)
-```sql
--- Compute window start in application code (recommended)
--- Or use strftime:
-SELECT strftime('%Y-%m-%d %H:%M:00',
-  datetime('now', '-' || (strftime('%M','now') % (? / 60)) || ' minutes')
-) AS window_start;
-```
-
-**Notes:** The entire `check_rate_limit` function should be moved to application code since SQLite lacks PL/pgSQL. Compute the window start in TypeScript and pass it as a parameter.
-
----
-
-## 12. json_populate_record() → Application-Level Deserialization
-
-**Locations:** `src/graph/migration-job.ts:351`
-
-### Before (PG)
-```sql
-SELECT * FROM json_populate_record(NULL::agx.tasks, $2::json);
-```
-
-### After (SQLite)
-```sql
--- No equivalent. Deserialize in application code:
--- const task = JSON.parse(jsonString);
--- Then construct INSERT from the parsed object.
-```
-
----
-
-## 13. Schema Namespace (agx.) → Removed
-
-**Locations:** All schema and query files
-
-### Before (PG)
-```sql
-CREATE SCHEMA IF NOT EXISTS agx;
-SELECT * FROM agx.tasks WHERE id = $1;
-```
-
-### After (SQLite)
-```sql
--- SQLite has no schemas. Remove the agx. prefix.
-SELECT * FROM tasks WHERE id = ?;
-```
-
----
-
-## 14. Parameterized Query Placeholders
-
-**Locations:** All query files
-
-### Before (PG)
-```sql
-SELECT * FROM tasks WHERE id = $1 AND project_id = $2;
-```
-
-### After (SQLite)
-```sql
-SELECT * FROM tasks WHERE id = ? AND project_id = ?;
-```
-
-**Notes:** SQLite uses positional `?` or `?NNN` (e.g., `?1`, `?2`). Most SQLite drivers for Node.js accept `?` placeholders.
-
----
-
-## 15. Idempotency, Crash Recovery & Transaction Semantics
-
-### 15.1 Enqueue Idempotency
-
-Duplicate task submissions are rejected via a unique constraint on a deduplication key (e.g., `project_id + slug` or an explicit `dedup_key` column).
-
-**PG:**
-```sql
-INSERT INTO agx.task_queue (task_id, action, dedup_key)
-VALUES ($1, $2, $3)
-ON CONFLICT (dedup_key) DO NOTHING
-RETURNING id;
--- If RETURNING yields no rows → duplicate was silently ignored
-```
-
-**SQLite:**
-```sql
-INSERT OR IGNORE INTO task_queue (task_id, action, dedup_key)
-VALUES (?, ?, ?);
--- Check changes() to detect duplicate:
--- If changes() = 0, the row already existed (duplicate enqueue)
-SELECT id FROM task_queue WHERE dedup_key = ?;
-```
-
-**Semantics:** Both engines treat duplicate enqueue as a no-op. The caller checks whether a row was actually inserted (PG: empty RETURNING; SQLite: `changes() = 0`) and returns the existing task ID.
-
-### 15.2 Retry Idempotency with CAS (Compare-and-Swap)
-
-Every state transition uses optimistic concurrency via the `version` column:
-
-```sql
--- Claim a queued task for execution (identical SQL on both engines)
-UPDATE tasks
-SET stage = 'PROGRESS', version = version + 1, updated_at = datetime('now')
-WHERE id = ? AND stage = 'QUEUED' AND version = ?;
--- rows_affected = 1 → successfully claimed
--- rows_affected = 0 → another worker claimed it or version changed → skip
-```
-
-**Duplicate retry scenario:** Worker A crashes after claiming task T (version 5→6). Worker B tries to claim the same task. Because T is now `stage = 'PROGRESS'` and `version = 6`, Worker B's `WHERE stage = 'QUEUED' AND version = 5` matches zero rows. The retry is safely rejected.
-
-### 15.3 Crash Recovery Scenarios
-
-| Scenario | State After Crash | Recovery Action |
-|---|---|---|
-| Worker crashes after claiming task | `stage='PROGRESS'`, stale `updated_at` | Reaper reclaims (see below) |
-| Worker crashes mid-INSERT (before COMMIT) | Row never persisted | No cleanup needed — transaction rolled back |
-| Worker crashes after completing work but before marking done | `stage='PROGRESS'`, work artifacts exist | Reaper reclaims; worker re-executes (must be idempotent) |
-| DB crashes mid-WAL write | Partial WAL frames | SQLite WAL recovery replays committed frames on next open |
-
-**Reaper query** (run periodically by a supervisor):
-```sql
--- Reclaim tasks stuck in PROGRESS with no heartbeat for 5 minutes
-UPDATE tasks
-SET stage = 'QUEUED', version = version + 1, updated_at = datetime('now')
-WHERE stage = 'PROGRESS'
-  AND updated_at < datetime('now', '-5 minutes');
-```
-
-The CAS version bump ensures that if the original worker is actually still alive (network partition, not a real crash), its next UPDATE will fail the version check and it will back off.
-
-### 15.4 SQLite WAL Mode Configuration
-
-SQLite **must** run in WAL (Write-Ahead Logging) mode for concurrent read/write access:
-
-```sql
-PRAGMA journal_mode = WAL;        -- Enable WAL (persistent, set once)
-PRAGMA busy_timeout = 5000;       -- Wait up to 5s for locks instead of failing immediately
-PRAGMA synchronous = NORMAL;      -- Safe with WAL; FULL is unnecessary overhead
-PRAGMA foreign_keys = ON;         -- Not on by default in SQLite
-```
-
-**Why WAL matters:**
-- **Without WAL** (default rollback journal): Readers block writers and writers block readers. A single slow read can stall all enqueue/dequeue operations.
-- **With WAL:** Readers never block writers. Multiple readers can proceed concurrently with a single writer. This is essential for the task queue pattern where a reaper, workers, and the API all access the DB.
-
-**Limitation:** SQLite allows only **one writer at a time**, even in WAL mode. Concurrent write transactions are serialized (one proceeds, others get `SQLITE_BUSY` and retry per `busy_timeout`). This is fine for single-node agx but prevents multi-process write scaling.
-
-### 15.5 Transaction Isolation: PG MVCC vs SQLite Serialized
-
-| Property | PostgreSQL | SQLite (WAL mode) |
-|---|---|---|
-| Default isolation | READ COMMITTED | SERIALIZED (effectively) |
-| Concurrent readers | Yes (MVCC snapshots) | Yes (WAL snapshots) |
-| Concurrent writers | Yes (row-level locks) | **No** — single writer at a time |
-| Dirty reads | No | No |
-| Phantom reads | Possible at READ COMMITTED | Not possible (serialized) |
-| Deadlocks | Possible (detected + aborted) | Not possible (single writer) |
-| Retry on conflict | Rare (row locks wait) | Common (`SQLITE_BUSY`) |
-
-**Implications for agx:**
-- **PG:** Multiple workers can claim different tasks concurrently via row-level locks. Two concurrent `UPDATE ... WHERE version = ?` on different rows both succeed.
-- **SQLite:** Two concurrent writes serialize. The second writer waits (up to `busy_timeout`), then proceeds. For the CAS pattern, this is actually **safer** — no possibility of lost updates or deadlocks.
-- **Application code must handle `SQLITE_BUSY`:** Wrap write operations in a retry loop (the `busy_timeout` PRAGMA handles most cases, but application-level retry with backoff is a safety net).
-
-### 15.6 Idempotency Guarantees Summary
-
-| Operation | Idempotency Mechanism | PG | SQLite |
-|---|---|---|---|
-| Enqueue task | `ON CONFLICT (dedup_key) DO NOTHING` | `RETURNING` empty = dup | `changes() = 0` = dup |
-| Claim task | CAS: `WHERE version = ? AND stage = 'QUEUED'` | Same | Same |
-| Complete task | CAS: `WHERE version = ? AND stage = 'PROGRESS'` | Same | Same |
-| Retry after crash | Reaper bumps version, re-queues | Same | Same |
-| Version increment | PG trigger / SQLite CAS in UPDATE | Trigger-based | Application CAS |
-
----
-
-## 16. Error Code Mapping: PostgreSQL ↔ SQLite
-
-| Error Condition | PostgreSQL | SQLite | Detection in Node.js |
-|---|---|---|---|
-| Unique violation | `23505` (`unique_violation`) | `SQLITE_CONSTRAINT_UNIQUE` (2067) | PG: `err.code === '23505'`; SQLite: `err.code === 'SQLITE_CONSTRAINT_UNIQUE'` |
-| Foreign key violation | `23503` (`foreign_key_violation`) | `SQLITE_CONSTRAINT_FOREIGNKEY` (787) | PG: `err.code === '23503'`; SQLite: `err.code === 'SQLITE_CONSTRAINT_FOREIGNKEY'` |
-| Not null violation | `23502` (`not_null_violation`) | `SQLITE_CONSTRAINT_NOTNULL` (1299) | PG: `err.code === '23502'`; SQLite: `err.code === 'SQLITE_CONSTRAINT_NOTNULL'` |
-| Check constraint | `23514` (`check_violation`) | `SQLITE_CONSTRAINT_CHECK` (275) | PG: `err.code === '23514'`; SQLite: `err.code === 'SQLITE_CONSTRAINT_CHECK'` |
-| Database locked / busy | `55P03` (`lock_not_available`) | `SQLITE_BUSY` (5) | PG: `err.code === '55P03'`; SQLite: `err.code === 'SQLITE_BUSY'` |
-| Serialization failure | `40001` (`serialization_failure`) | N/A (single writer) | PG: `err.code === '40001'` → retry transaction |
-| Deadlock | `40P01` (`deadlock_detected`) | N/A (single writer) | PG: `err.code === '40P01'` → retry transaction |
-
-**Adapter pattern:** The DB adapter should normalize error codes into a common enum:
-```typescript
-enum DbError {
-  UNIQUE_VIOLATION = 'UNIQUE_VIOLATION',
-  FK_VIOLATION = 'FK_VIOLATION',
-  NOT_NULL_VIOLATION = 'NOT_NULL_VIOLATION',
-  BUSY = 'BUSY',
-}
-
-function classifyError(err: any, dialect: 'pg' | 'sqlite'): DbError | null {
-  if (dialect === 'pg') {
-    if (err.code === '23505') return DbError.UNIQUE_VIOLATION;
-    if (err.code === '23503') return DbError.FK_VIOLATION;
-    if (err.code === '23502') return DbError.NOT_NULL_VIOLATION;
-    if (err.code === '55P03' || err.code === '40001') return DbError.BUSY;
-  } else {
-    if (err.code === 'SQLITE_CONSTRAINT_UNIQUE') return DbError.UNIQUE_VIOLATION;
-    if (err.code === 'SQLITE_CONSTRAINT_FOREIGNKEY') return DbError.FK_VIOLATION;
-    if (err.code === 'SQLITE_CONSTRAINT_NOTNULL') return DbError.NOT_NULL_VIOLATION;
-    if (err.code === 'SQLITE_BUSY') return DbError.BUSY;
-  }
-  return null;
-}
-```