@mndrk/agx 2.0.38 → 2.0.39

package/cloud-runtime/standalone/docs/pg-sqlite-compatibility-matrix.md

@@ -0,0 +1,554 @@
+# PostgreSQL → SQLite Compatibility Matrix
+
+Every Postgres-specific SQL pattern in agx-cloud, with its SQLite rewrite.
+
+---
+
+## 1. ILIKE → LIKE COLLATE NOCASE
+
+**Locations:** `lib/db-adapter.ts:233`, `src/graph/migration-job.ts:161`, `mcp/src/tools/learnings.ts:95`
+
+### Before (PG)
+```sql
+SELECT * FROM tasks WHERE title ILIKE '%deploy%';
+```
+
+### After (SQLite)
+```sql
+SELECT * FROM tasks WHERE title LIKE '%deploy%' COLLATE NOCASE;
+```
+
+**Notes:** SQLite's `LIKE` is case-insensitive for ASCII by default when no collation is set, but explicit `COLLATE NOCASE` guarantees it. For programmatic queries in `db-adapter.ts`, replace the `ILIKE` operator branch with `LIKE ... COLLATE NOCASE`.
+
+---
+
+## 2. @> JSON Containment → json_each()
+
+**Locations:** `lib/db-adapter.ts:275`
+
+### Before (PG)
+```sql
+-- Check if tags array contains 'urgent'
+SELECT * FROM tasks WHERE metadata @> '["urgent"]'::jsonb;
+
+-- Check if labels column contains specific values
+SELECT * FROM tasks WHERE labels @> ARRAY['bug','critical'];
+```
+
+### After (SQLite)
+```sql
+-- Using json_each to check containment
+SELECT DISTINCT t.* FROM tasks t, json_each(t.metadata) je
+WHERE je.value = 'urgent';
+
+-- Multiple containment checks
+SELECT t.* FROM tasks t
+WHERE (SELECT COUNT(*) FROM json_each(t.labels) je
+       WHERE je.value IN ('bug','critical')) = 2;
+```
+
+**Notes:** For the `contains` operator in `db-adapter.ts`, emit a subquery using `json_each()` instead of `@>`. If checking a single value, a simpler form works:
+```sql
+SELECT * FROM tasks WHERE EXISTS (
+  SELECT 1 FROM json_each(tasks.labels) WHERE value = 'urgent'
+);
+```
+
+---
+
+## 3. RETURNING → Separate SELECT
+
+**Locations:** `lib/db-adapter.ts:398`, `mcp/src/db.ts:159`, `src/graph/migration-job.ts:361`, `src/graph/store.ts:251,436,520`
+
+### Before (PG)
+```sql
+-- INSERT RETURNING
+INSERT INTO agx.execution_graphs (id, task_id, nodes, edges)
+VALUES ($1, $2, $3::jsonb, $4::jsonb)
+RETURNING *;
+
+-- UPDATE RETURNING
+UPDATE agx.execution_graphs
+SET updated_at = NOW()
+WHERE id = $1
+RETURNING graph_version, updated_at;
+
+-- DELETE RETURNING
+DELETE FROM agx.execution_graphs
+WHERE task_id = $1
+RETURNING id;
+```
+
+### After (SQLite)
+```sql
+-- INSERT + SELECT (use last_insert_rowid() or known id)
+INSERT INTO execution_graphs (id, task_id, nodes, edges)
+VALUES (?, ?, json(?), json(?));
+SELECT * FROM execution_graphs WHERE id = ?;
+
+-- UPDATE + SELECT
+UPDATE execution_graphs
+SET updated_at = datetime('now')
+WHERE id = ?;
+SELECT graph_version, updated_at FROM execution_graphs WHERE id = ?;
+
+-- DELETE: capture ids first, then delete
+SELECT id FROM execution_graphs WHERE task_id = ?;
+DELETE FROM execution_graphs WHERE task_id = ?;
+```
+
+**Notes:** Wrap INSERT+SELECT or DELETE(select)+DELETE in a transaction to ensure atomicity. SQLite supports `RETURNING` since version 3.35.0 (2021-03-12) — if targeting modern SQLite, `RETURNING` works as-is. Check your runtime version.
+
+---
+
+## 4. Array Types → JSON Arrays
+
+**Locations:** `001_agx_board_schema.sql:123,261`, `src/graph/store.ts:377`, `src/graph/migration-job.ts:137,149,258,312`
+
+### Before (PG)
+```sql
+-- Schema
+outputs text[],
+depends_on uuid[] DEFAULT '{}'::uuid[],
+
+-- Query with ANY()
+SELECT * FROM graph_nodes
+WHERE graph_id = $1 AND node_id = ANY($2::text[]);
+
+-- Query with array parameter
+SELECT * FROM tasks WHERE id = ANY($1::uuid[]);
+```
+
+### After (SQLite)
+```sql
+-- Schema: store as JSON text
+outputs TEXT DEFAULT '[]',  -- JSON array
+depends_on TEXT DEFAULT '[]',  -- JSON array of uuids
+
+-- Query: replace ANY() with json_each subquery
+SELECT * FROM graph_nodes
+WHERE graph_id = ? AND node_id IN (SELECT value FROM json_each(?));
+
+-- Same pattern for uuid arrays
+SELECT * FROM tasks WHERE id IN (SELECT value FROM json_each(?));
+```
+
+**Notes:** Pass JSON-encoded arrays (`'["a","b","c"]'`) as parameters. The `json_each()` table-valued function unpacks them for `IN` clauses.
+
+---
+
+## 5. increment_version Stored Procedure → Application-Level CAS
+
+**Locations:** `001_agx_board_schema.sql:43-50,551-558` (triggers), `store.ts` (relies on auto-incremented versions)
+
+### Before (PG)
+```sql
+-- Trigger function
+CREATE OR REPLACE FUNCTION agx.increment_task_version()
+RETURNS trigger LANGUAGE plpgsql AS $$
+BEGIN
+  NEW.version = OLD.version + 1;
+  RETURN NEW;
+END;
+$$;
+
+CREATE TRIGGER trg_increment_task_version
+BEFORE UPDATE ON agx.tasks
+FOR EACH ROW EXECUTE FUNCTION agx.increment_task_version();
+
+-- Usage: just UPDATE, version increments automatically
+UPDATE agx.tasks SET title = 'new title' WHERE id = $1;
+```
+
+### After (SQLite)
+```sql
+-- Option A: SQLite trigger (closest equivalent)
+CREATE TRIGGER trg_increment_task_version
+BEFORE UPDATE ON tasks
+FOR EACH ROW
+BEGIN
+  UPDATE tasks SET version = OLD.version + 1 WHERE id = NEW.id;
+END;
+
+-- Option B: Application-level compare-and-swap (recommended)
+-- Read current version, then update with version check:
+UPDATE tasks
+SET title = ?, version = version + 1
+WHERE id = ? AND version = ?;
+-- If rows_affected = 0, conflict detected → retry or error
+```
+
+**Notes:** Option B (CAS) is preferred because it provides explicit optimistic concurrency control. The caller reads the current version, attempts the update with `AND version = ?`, and retries on conflict. This avoids hidden trigger behavior and makes concurrency explicit.
+
+The same pattern applies to `increment_graph_version`:
+```sql
+UPDATE execution_graphs
+SET graph_version = graph_version + 1, nodes = json(?)
+WHERE id = ? AND graph_version = ?;
+```
+
+---
+
+## 6. update_updated_at Trigger → SQLite Trigger or Application Code
+
+**Locations:** `001_agx_board_schema.sql:52-59`
+
+### Before (PG)
+```sql
+CREATE OR REPLACE FUNCTION agx.update_updated_at()
+RETURNS trigger LANGUAGE plpgsql AS $$
+BEGIN
+  NEW.updated_at = now();
+  RETURN NEW;
+END;
+$$;
+
+CREATE TRIGGER trg_user_settings_updated_at
+BEFORE UPDATE ON agx.user_settings
+FOR EACH ROW EXECUTE FUNCTION agx.update_updated_at();
+```
+
+### After (SQLite)
+```sql
+CREATE TRIGGER trg_user_settings_updated_at
+AFTER UPDATE ON user_settings
+FOR EACH ROW
+BEGIN
+  UPDATE user_settings SET updated_at = datetime('now')
+  WHERE id = NEW.id;
+END;
+```
+
+**Notes:** SQLite triggers use `BEGIN...END` blocks instead of PL/pgSQL functions. Alternatively, set `updated_at` in application code on every UPDATE.
+
+---
+
+## 7. ON CONFLICT (Upsert) Differences
+
+**Locations:** `lib/db-adapter.ts:376-395`, `mcp/src/db.ts:563`, `src/graph/store.ts:247`, `src/graph/migration-job.ts:263,389`
+
+### Before (PG)
+```sql
+INSERT INTO agx.rate_limits (user_id, endpoint, window_start, request_count)
+VALUES ($1, $2, $3, 1)
+ON CONFLICT (user_id, endpoint, window_start)
+DO UPDATE SET request_count = rate_limits.request_count + 1
+RETURNING request_count;
+```
+
+### After (SQLite)
+```sql
+INSERT INTO rate_limits (user_id, endpoint, window_start, request_count)
+VALUES (?, ?, ?, 1)
+ON CONFLICT (user_id, endpoint, window_start)
+DO UPDATE SET request_count = rate_limits.request_count + 1;
+-- Then SELECT if needed:
+SELECT request_count FROM rate_limits
+WHERE user_id = ? AND endpoint = ? AND window_start = ?;
+```
+
+**Notes:** SQLite supports `ON CONFLICT ... DO UPDATE SET` and `DO NOTHING` with the same syntax as PG. The key differences:
+- Replace `EXCLUDED` with `excluded` (SQLite uses lowercase, but is case-insensitive so both work)
+- Remove `RETURNING` (see section 3)
+- Use `?` placeholders instead of `$1`, `$2`
+
+---
+
+## 8. gen_random_uuid() → Application-Generated UUIDs
+
+**Locations:** `001_agx_board_schema.sql` (19 table definitions)
+
+### Before (PG)
+```sql
+id uuid DEFAULT gen_random_uuid() NOT NULL PRIMARY KEY,
+```
+
+### After (SQLite)
+```sql
+-- Option A: Use SQLite's randomblob to generate UUID
+id TEXT NOT NULL PRIMARY KEY DEFAULT (
+  lower(hex(randomblob(4))) || '-' ||
+  lower(hex(randomblob(2))) || '-4' ||
+  substr(lower(hex(randomblob(2))),2) || '-' ||
+  substr('89ab', abs(random()) % 4 + 1, 1) ||
+  substr(lower(hex(randomblob(2))),2) || '-' ||
+  lower(hex(randomblob(6)))
+),
+
+-- Option B (recommended): Generate UUID in application code
+id TEXT NOT NULL PRIMARY KEY,
+-- Application generates: crypto.randomUUID() before INSERT
+```
+
+**Notes:** Option B is simpler and more portable. Generate UUIDs in TypeScript with `crypto.randomUUID()` and pass them as parameters.
+
+---
+
+## 9. NOW() / CURRENT_TIMESTAMP → datetime('now')
+
+**Locations:** Throughout schema and queries
+
+### Before (PG)
+```sql
+created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
+updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW() NOT NULL,
+
+-- In queries
+UPDATE agx.execution_graphs SET updated_at = NOW() WHERE id = $1;
+```
+
+### After (SQLite)
+```sql
+created_at TEXT DEFAULT (datetime('now')),
+updated_at TEXT DEFAULT (datetime('now')) NOT NULL,
+
+-- In queries
+UPDATE execution_graphs SET updated_at = datetime('now') WHERE id = ?;
+```
+
+**Notes:** SQLite stores timestamps as TEXT in ISO-8601 format. Use `datetime('now')` for UTC. For comparison operations, ISO-8601 strings sort correctly as text.
+
+---
+
+## 10. Type Casts (::type) → Removed or json()
+
+**Locations:** `src/graph/store.ts`, `src/graph/migration-job.ts`
+
+### Before (PG)
+```sql
+$5::jsonb, $6::jsonb, $7::timestamptz
+$2::text[]
+$1::uuid[]
+```
+
+### After (SQLite)
+```sql
+json(?), json(?), ?
+-- Array casts: pass JSON string, use json_each() for queries
+-- UUID/text casts: just use ? (SQLite is dynamically typed)
+```
+
+**Notes:** SQLite is dynamically typed. Remove all `::type` casts. For JSONB columns, wrap with `json()` to validate/normalize JSON. For arrays, pass JSON strings.
+
+---
+
+## 11. INTERVAL / date_trunc / extract → SQLite Date Functions
+
+**Locations:** `001_agx_board_schema.sql:31` (check_rate_limit function)
+
+### Before (PG)
+```sql
+v_window_start := date_trunc('minute', now()) -
+  ((extract(minute from now())::int % (p_window_seconds / 60)) || ' minutes')::interval;
+```
+
+### After (SQLite)
+```sql
+-- Compute window start in application code (recommended)
+-- Or use strftime:
+SELECT strftime('%Y-%m-%d %H:%M:00',
+  datetime('now', '-' || (strftime('%M','now') % (? / 60)) || ' minutes')
+) AS window_start;
+```
+
+**Notes:** The entire `check_rate_limit` function should be moved to application code since SQLite lacks PL/pgSQL. Compute the window start in TypeScript and pass it as a parameter.
+
+---
+
+## 12. json_populate_record() → Application-Level Deserialization
+
+**Locations:** `src/graph/migration-job.ts:351`
+
+### Before (PG)
+```sql
+SELECT * FROM json_populate_record(NULL::agx.tasks, $2::json);
+```
+
+### After (SQLite)
+```sql
+-- No equivalent. Deserialize in application code:
+-- const task = JSON.parse(jsonString);
+-- Then construct INSERT from the parsed object.
+```
+
+---
+
+## 13. Schema Namespace (agx.) → Removed
+
+**Locations:** All schema and query files
+
+### Before (PG)
+```sql
+CREATE SCHEMA IF NOT EXISTS agx;
+SELECT * FROM agx.tasks WHERE id = $1;
+```
+
+### After (SQLite)
+```sql
+-- SQLite has no schemas. Remove the agx. prefix.
+SELECT * FROM tasks WHERE id = ?;
+```
+
+---
+
+## 14. Parameterized Query Placeholders
+
+**Locations:** All query files
+
+### Before (PG)
+```sql
+SELECT * FROM tasks WHERE id = $1 AND project_id = $2;
+```
+
+### After (SQLite)
+```sql
+SELECT * FROM tasks WHERE id = ? AND project_id = ?;
+```
+
+**Notes:** SQLite uses positional `?` or `?NNN` (e.g., `?1`, `?2`). Most SQLite drivers for Node.js accept `?` placeholders.
+
+---
+
+## 15. Idempotency, Crash Recovery & Transaction Semantics
+
+### 15.1 Enqueue Idempotency
+
+Duplicate task submissions are rejected via a unique constraint on a deduplication key (e.g., `project_id + slug` or an explicit `dedup_key` column).
+
+**PG:**
+```sql
+INSERT INTO agx.task_queue (task_id, action, dedup_key)
+VALUES ($1, $2, $3)
+ON CONFLICT (dedup_key) DO NOTHING
+RETURNING id;
+-- If RETURNING yields no rows → duplicate was silently ignored
+```
+
+**SQLite:**
+```sql
+INSERT OR IGNORE INTO task_queue (task_id, action, dedup_key)
+VALUES (?, ?, ?);
+-- Check changes() to detect duplicate:
+-- If changes() = 0, the row already existed (duplicate enqueue)
+SELECT id FROM task_queue WHERE dedup_key = ?;
+```
+
+**Semantics:** Both engines treat duplicate enqueue as a no-op. The caller checks whether a row was actually inserted (PG: empty RETURNING; SQLite: `changes() = 0`) and returns the existing task ID.
+
+### 15.2 Retry Idempotency with CAS (Compare-and-Swap)
+
+Every state transition uses optimistic concurrency via the `version` column:
+
+```sql
+-- Claim a queued task for execution (identical SQL on both engines)
+UPDATE tasks
+SET stage = 'PROGRESS', version = version + 1, updated_at = datetime('now')
+WHERE id = ? AND stage = 'QUEUED' AND version = ?;
+-- rows_affected = 1 → successfully claimed
+-- rows_affected = 0 → another worker claimed it or version changed → skip
+```
+
+**Duplicate retry scenario:** Worker A crashes after claiming task T (version 5→6). Worker B tries to claim the same task. Because T is now `stage = 'PROGRESS'` and `version = 6`, Worker B's `WHERE stage = 'QUEUED' AND version = 5` matches zero rows. The retry is safely rejected.
+
+### 15.3 Crash Recovery Scenarios
+
+| Scenario | State After Crash | Recovery Action |
+|---|---|---|
+| Worker crashes after claiming task | `stage='PROGRESS'`, stale `updated_at` | Reaper reclaims (see below) |
+| Worker crashes mid-INSERT (before COMMIT) | Row never persisted | No cleanup needed — transaction rolled back |
+| Worker crashes after completing work but before marking done | `stage='PROGRESS'`, work artifacts exist | Reaper reclaims; worker re-executes (must be idempotent) |
+| DB crashes mid-WAL write | Partial WAL frames | SQLite WAL recovery replays committed frames on next open |
+
+**Reaper query** (run periodically by a supervisor):
+```sql
+-- Reclaim tasks stuck in PROGRESS with no heartbeat for 5 minutes
+UPDATE tasks
+SET stage = 'QUEUED', version = version + 1, updated_at = datetime('now')
+WHERE stage = 'PROGRESS'
+  AND updated_at < datetime('now', '-5 minutes');
+```
+
+The CAS version bump ensures that if the original worker is actually still alive (network partition, not a real crash), its next UPDATE will fail the version check and it will back off.
+
+### 15.4 SQLite WAL Mode Configuration
+
+SQLite **must** run in WAL (Write-Ahead Logging) mode for concurrent read/write access:
+
+```sql
+PRAGMA journal_mode = WAL;        -- Enable WAL (persistent, set once)
+PRAGMA busy_timeout = 5000;       -- Wait up to 5s for locks instead of failing immediately
+PRAGMA synchronous = NORMAL;      -- Safe with WAL; FULL is unnecessary overhead
+PRAGMA foreign_keys = ON;         -- Not on by default in SQLite
+```
+
+**Why WAL matters:**
+- **Without WAL** (default rollback journal): Readers block writers and writers block readers. A single slow read can stall all enqueue/dequeue operations.
+- **With WAL:** Readers never block writers. Multiple readers can proceed concurrently with a single writer. This is essential for the task queue pattern where a reaper, workers, and the API all access the DB.
+
+**Limitation:** SQLite allows only **one writer at a time**, even in WAL mode. Concurrent write transactions are serialized (one proceeds, others get `SQLITE_BUSY` and retry per `busy_timeout`). This is fine for single-node agx but prevents multi-process write scaling.
+
+### 15.5 Transaction Isolation: PG MVCC vs SQLite Serialized
+
+| Property | PostgreSQL | SQLite (WAL mode) |
+|---|---|---|
+| Default isolation | READ COMMITTED | SERIALIZED (effectively) |
+| Concurrent readers | Yes (MVCC snapshots) | Yes (WAL snapshots) |
+| Concurrent writers | Yes (row-level locks) | **No** — single writer at a time |
+| Dirty reads | No | No |
+| Phantom reads | Possible at READ COMMITTED | Not possible (serialized) |
+| Deadlocks | Possible (detected + aborted) | Not possible (single writer) |
+| Retry on conflict | Rare (row locks wait) | Common (`SQLITE_BUSY`) |
+
+**Implications for agx:**
+- **PG:** Multiple workers can claim different tasks concurrently via row-level locks. Two concurrent `UPDATE ... WHERE version = ?` on different rows both succeed.
+- **SQLite:** Two concurrent writes serialize. The second writer waits (up to `busy_timeout`), then proceeds. For the CAS pattern, this is actually **safer** — no possibility of lost updates or deadlocks.
+- **Application code must handle `SQLITE_BUSY`:** Wrap write operations in a retry loop (the `busy_timeout` PRAGMA handles most cases, but application-level retry with backoff is a safety net).
+
+### 15.6 Idempotency Guarantees Summary
+
+| Operation | Idempotency Mechanism | PG | SQLite |
+|---|---|---|---|
+| Enqueue task | `ON CONFLICT (dedup_key) DO NOTHING` | `RETURNING` empty = dup | `changes() = 0` = dup |
+| Claim task | CAS: `WHERE version = ? AND stage = 'QUEUED'` | Same | Same |
+| Complete task | CAS: `WHERE version = ? AND stage = 'PROGRESS'` | Same | Same |
+| Retry after crash | Reaper bumps version, re-queues | Same | Same |
+| Version increment | PG trigger / SQLite CAS in UPDATE | Trigger-based | Application CAS |
+
+---
+
+## 16. Error Code Mapping: PostgreSQL ↔ SQLite
+
+| Error Condition | PostgreSQL | SQLite | Detection in Node.js |
+|---|---|---|---|
+| Unique violation | `23505` (`unique_violation`) | `SQLITE_CONSTRAINT_UNIQUE` (2067) | PG: `err.code === '23505'`; SQLite: `err.code === 'SQLITE_CONSTRAINT_UNIQUE'` |
+| Foreign key violation | `23503` (`foreign_key_violation`) | `SQLITE_CONSTRAINT_FOREIGNKEY` (787) | PG: `err.code === '23503'`; SQLite: `err.code === 'SQLITE_CONSTRAINT_FOREIGNKEY'` |
+| Not null violation | `23502` (`not_null_violation`) | `SQLITE_CONSTRAINT_NOTNULL` (1299) | PG: `err.code === '23502'`; SQLite: `err.code === 'SQLITE_CONSTRAINT_NOTNULL'` |
+| Check constraint | `23514` (`check_violation`) | `SQLITE_CONSTRAINT_CHECK` (275) | PG: `err.code === '23514'`; SQLite: `err.code === 'SQLITE_CONSTRAINT_CHECK'` |
+| Database locked / busy | `55P03` (`lock_not_available`) | `SQLITE_BUSY` (5) | PG: `err.code === '55P03'`; SQLite: `err.code === 'SQLITE_BUSY'` |
+| Serialization failure | `40001` (`serialization_failure`) | N/A (single writer) | PG: `err.code === '40001'` → retry transaction |
+| Deadlock | `40P01` (`deadlock_detected`) | N/A (single writer) | PG: `err.code === '40P01'` → retry transaction |
+
+**Adapter pattern:** The DB adapter should normalize error codes into a common enum:
+```typescript
+enum DbError {
+  UNIQUE_VIOLATION = 'UNIQUE_VIOLATION',
+  FK_VIOLATION = 'FK_VIOLATION',
+  NOT_NULL_VIOLATION = 'NOT_NULL_VIOLATION',
+  BUSY = 'BUSY',
+}
+
+function classifyError(err: any, dialect: 'pg' | 'sqlite'): DbError | null {
+  if (dialect === 'pg') {
+    if (err.code === '23505') return DbError.UNIQUE_VIOLATION;
+    if (err.code === '23503') return DbError.FK_VIOLATION;
+    if (err.code === '23502') return DbError.NOT_NULL_VIOLATION;
+    if (err.code === '55P03' || err.code === '40001') return DbError.BUSY;
+  } else {
+    if (err.code === 'SQLITE_CONSTRAINT_UNIQUE') return DbError.UNIQUE_VIOLATION;
+    if (err.code === 'SQLITE_CONSTRAINT_FOREIGNKEY') return DbError.FK_VIOLATION;
+    if (err.code === 'SQLITE_CONSTRAINT_NOTNULL') return DbError.NOT_NULL_VIOLATION;
+    if (err.code === 'SQLITE_BUSY') return DbError.BUSY;
+  }
+  return null;
+}
+```