codeforge-dev 1.4.0

package/.devcontainer/plugins/devs-marketplace/plugins/codedirective-skills/skills/sqlite/references/python-patterns.md

@@ -0,0 +1,354 @@
+# Python SQLite Patterns -- Deep Dive
+
+## 1. Connection Management for Multi-Threaded Apps
+
+SQLite connections are not thread-safe by default. For multi-threaded applications, create one connection per thread or use a connection pool:
+
+```python
+import sqlite3
+import threading
+from contextlib import contextmanager
+
+class ConnectionPool:
+    def __init__(self, db_path: str, max_connections: int = 5):
+        self.db_path = db_path
+        self.semaphore = threading.Semaphore(max_connections)
+        self.local = threading.local()
+
+    @contextmanager
+    def get_connection(self):
+        self.semaphore.acquire()
+        try:
+            if not hasattr(self.local, "conn"):
+                self.local.conn = self._create_connection()
+            yield self.local.conn
+        finally:
+            self.semaphore.release()
+
+    def _create_connection(self) -> sqlite3.Connection:
+        conn = sqlite3.connect(self.db_path)
+        conn.row_factory = sqlite3.Row
+        conn.execute("PRAGMA journal_mode=WAL")
+        conn.execute("PRAGMA foreign_keys=ON")
+        conn.execute("PRAGMA busy_timeout=5000")
+        return conn
+```
+
+Thread-local storage ensures each thread reuses its own connection. The semaphore limits total concurrent connections, preventing resource exhaustion.
+
+For `check_same_thread=False` (sharing a connection across threads), external locking is required:
+
+```python
+conn = sqlite3.connect("app.db", check_same_thread=False)
+lock = threading.Lock()
+
+def execute_safely(sql, params=()):
+    with lock:
+        return conn.execute(sql, params).fetchall()
+```
+
+This pattern is simpler but serializes all database access. Prefer per-thread connections for concurrent workloads.
+
+---
+
+## 2. Custom Aggregates and Collations
+
+### Custom Aggregate Functions
+
+Register Python functions as SQL aggregate functions:
+
+```python
+class MedianAggregate:
+    def __init__(self):
+        self.values = []
+
+    def step(self, value):
+        if value is not None:
+            self.values.append(value)
+
+    def finalize(self):
+        if not self.values:
+            return None
+        self.values.sort()
+        n = len(self.values)
+        mid = n // 2
+        if n % 2 == 0:
+            return (self.values[mid - 1] + self.values[mid]) / 2
+        return self.values[mid]
+
+conn.create_aggregate("median", 1, MedianAggregate)
+# SELECT median(price) FROM products;
+```
+
+### Custom Collations
+
+Define custom sorting logic for text comparisons:
+
+```python
+import unicodedata
+
+def collate_normalized(a: str, b: str) -> int:
+    na = unicodedata.normalize("NFKD", a).casefold()
+    nb = unicodedata.normalize("NFKD", b).casefold()
+    return (na > nb) - (na < nb)
+
+conn.create_collation("NORMALIZED", collate_normalized)
+# SELECT * FROM users ORDER BY name COLLATE NORMALIZED;
+```
+
+### Scalar Functions
+
+```python
+import json
+
+def json_array_length(value):
+    if value is None:
+        return 0
+    return len(json.loads(value))
+
+conn.create_function("json_arr_len", 1, json_array_length)
+# SELECT json_arr_len(tags) FROM posts;
+```
+
+---
+
+## 3. Backup API
+
+Copy a live database without locking writes:
+
+```python
+import sqlite3
+
+def backup_database(source_path: str, backup_path: str):
+    source = sqlite3.connect(source_path)
+    dest = sqlite3.connect(backup_path)
+    with dest:
+        source.backup(dest, pages=100, progress=backup_progress)
+    dest.close()
+    source.close()
+
+def backup_progress(status, remaining, total):
+    print(f"Backup: {total - remaining}/{total} pages copied")
+```
+
+The `pages` parameter controls how many pages are copied per step. Between steps, other connections can write. This makes `backup()` suitable for hot backups of production databases.
+
+### In-Memory to Disk (and Vice Versa)
+
+```python
+# Load disk database into memory for fast operations
+disk_db = sqlite3.connect("app.db")
+mem_db = sqlite3.connect(":memory:")
+disk_db.backup(mem_db)
+
+# Save in-memory database to disk
+mem_db.backup(disk_db)
+```
+
+---
+
+## 4. Datetime Adapters
+
+SQLite has no native datetime type. Store timestamps as ISO 8601 text and register adapters for automatic conversion:
+
+```python
+import sqlite3
+from datetime import datetime, timezone
+
+def adapt_datetime(dt: datetime) -> str:
+    return dt.isoformat()
+
+def convert_datetime(value: bytes) -> datetime:
+    return datetime.fromisoformat(value.decode())
+
+sqlite3.register_adapter(datetime, adapt_datetime)
+sqlite3.register_converter("TIMESTAMP", convert_datetime)
+
+conn = sqlite3.connect("app.db", detect_types=sqlite3.PARSE_DECLTYPES)
+```
+
+With `PARSE_DECLTYPES`, a column declared as `TIMESTAMP` automatically uses the registered converter. This keeps datetime handling transparent to application code.
+
+---
+
+## 5. Blob I/O
+
+For large binary data, use incremental blob I/O instead of loading the entire value into memory:
+
+```python
+import sqlite3
+
+# Write a blob
+with open("image.png", "rb") as f:
+    data = f.read()
+    conn.execute("INSERT INTO files (name, data) VALUES (?, ?)", ("image.png", data))
+
+# Read a blob incrementally
+row = conn.execute("SELECT rowid, length(data) FROM files WHERE name = ?",
+                   ("image.png",)).fetchone()
+rowid, size = row
+
+blob = conn.blobopen("main", "files", "data", rowid, readonly=True)
+chunk_size = 65536
+with open("output.png", "wb") as f:
+    while True:
+        chunk = blob.read(chunk_size)
+        if not chunk:
+            break
+        f.write(chunk)
+blob.close()
+```
+
+Incremental blob I/O avoids loading multi-megabyte files into Python memory. The `blobopen()` method returns a file-like object that reads directly from the database page cache.
+
+---
+
+## 6. aiosqlite + FastAPI Integration
+
+### Dependency Pattern
+
+```python
+import aiosqlite
+from contextlib import asynccontextmanager
+from fastapi import FastAPI, Depends, Request
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    app.state.db = await aiosqlite.connect("app.db")
+    app.state.db.row_factory = aiosqlite.Row
+    await app.state.db.execute("PRAGMA journal_mode=WAL")
+    await app.state.db.execute("PRAGMA foreign_keys=ON")
+    yield
+    await app.state.db.close()
+
+app = FastAPI(lifespan=lifespan)
+
+async def get_db(request: Request) -> aiosqlite.Connection:
+    return request.app.state.db
+
+@app.get("/users")
+async def list_users(db: aiosqlite.Connection = Depends(get_db)):
+    async with db.execute("SELECT * FROM users") as cursor:
+        rows = await cursor.fetchall()
+        return [dict(row) for row in rows]
+```
+
+### Transaction Pattern
+
+```python
+async def create_user_with_profile(db: aiosqlite.Connection, user_data: dict):
+    try:
+        await db.execute("BEGIN")
+        cursor = await db.execute(
+            "INSERT INTO users (email, name) VALUES (?, ?) RETURNING id",
+            (user_data["email"], user_data["name"]),
+        )
+        row = await cursor.fetchone()
+        user_id = row[0]
+        await db.execute(
+            "INSERT INTO profiles (user_id, bio) VALUES (?, ?)",
+            (user_id, user_data.get("bio", "")),
+        )
+        await db.commit()
+        return user_id
+    except Exception:
+        await db.rollback()
+        raise
+```
+
+---
+
+## 7. Migration Patterns
+
+### Simple Version Tracking
+
+```python
+import sqlite3
+
+MIGRATIONS = [
+    # Version 1
+    """
+    CREATE TABLE users (
+        id INTEGER PRIMARY KEY,
+        email TEXT NOT NULL UNIQUE,
+        created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
+    ) STRICT;
+    """,
+    # Version 2
+    """
+    ALTER TABLE users ADD COLUMN display_name TEXT;
+    CREATE INDEX idx_users_email ON users(email);
+    """,
+    # Version 3
+    """
+    CREATE TABLE posts (
+        id INTEGER PRIMARY KEY,
+        user_id INTEGER NOT NULL REFERENCES users(id),
+        title TEXT NOT NULL,
+        body TEXT NOT NULL
+    ) STRICT;
+    """,
+]
+
+def migrate(conn: sqlite3.Connection):
+    conn.execute("CREATE TABLE IF NOT EXISTS schema_version (version INTEGER)")
+    row = conn.execute("SELECT MAX(version) FROM schema_version").fetchone()
+    current = row[0] if row[0] is not None else 0
+
+    for i, sql in enumerate(MIGRATIONS[current:], start=current + 1):
+        conn.executescript(sql)
+        conn.execute("INSERT INTO schema_version (version) VALUES (?)", (i,))
+        conn.commit()
+```
+
+This pattern works for small-to-medium applications.
+
+### Reversible Migrations
+
+Structure migrations as `(up_sql, down_sql)` tuples to support rollback during development:
+
+```python
+MIGRATIONS = [
+    # Version 1
+    (
+        """
+        CREATE TABLE users (
+            id INTEGER PRIMARY KEY,
+            email TEXT NOT NULL UNIQUE,
+            created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
+        ) STRICT;
+        """,
+        """
+        DROP TABLE users;
+        """,
+    ),
+    # Version 2
+    (
+        """
+        ALTER TABLE users ADD COLUMN display_name TEXT;
+        CREATE INDEX idx_users_email ON users(email);
+        """,
+        """
+        DROP INDEX idx_users_email;
+        ALTER TABLE users DROP COLUMN display_name;
+        """,
+    ),
+]
+
+def rollback(conn: sqlite3.Connection, target_version: int):
+    row = conn.execute("SELECT MAX(version) FROM schema_version").fetchone()
+    current = row[0] if row[0] is not None else 0
+
+    if target_version >= current:
+        return
+
+    for i in range(current, target_version, -1):
+        _up, down_sql = MIGRATIONS[i - 1]
+        conn.executescript(down_sql)
+        conn.execute("DELETE FROM schema_version WHERE version = ?", (i,))
+        conn.commit()
+```
+
+**Safety constraints:** `DROP COLUMN` is only available in SQLite 3.35.0+. Data-destructive operations (dropping columns, changing types, deleting rows) are not safely reversible — the down migration can undo the schema change but not restore lost data. Prefer forward-fix migrations in production; use rollback as a development convenience for iterating on schema changes.
+
+For larger projects, use a dedicated migration tool (Alembic, yoyo-migrations) that provides migration file management and dependency tracking.

package/.devcontainer/plugins/devs-marketplace/plugins/codedirective-skills/skills/sqlite/references/schema-and-pragmas.md

@@ -0,0 +1,326 @@
+# Schema and PRAGMAs -- Deep Dive
+
+## 1. Recommended PRAGMAs
+
+Apply these at connection startup for production workloads. Each PRAGMA has a specific purpose and tradeoff:
+
+| PRAGMA | Value | Rationale |
+|--------|-------|-----------|
+| `journal_mode` | `WAL` | Concurrent reads during writes, better performance for mixed workloads |
+| `foreign_keys` | `ON` | Enforce referential integrity (off by default for backward compatibility) |
+| `busy_timeout` | `5000` | Wait 5 seconds for locks instead of failing immediately with SQLITE_BUSY |
+| `synchronous` | `NORMAL` | Safe with WAL -- fsync on checkpoint only. `FULL` fsyncs every commit |
+| `cache_size` | `-64000` | 64MB page cache in memory. Negative value = kilobytes. Default is ~2MB |
+| `temp_store` | `MEMORY` | Temp tables and indexes in RAM instead of disk |
+| `mmap_size` | `268435456` | Memory-map up to 256MB of the database file for faster reads |
+| `page_size` | `4096` | Match filesystem block size (set before creating the database) |
+
+### Persistence Rules
+
+| PRAGMA | Persists in DB file? | Set per connection? |
+|--------|---------------------|-------------------|
+| `journal_mode` | Yes (WAL persists) | Set once, all connections inherit |
+| `foreign_keys` | No | Must set every connection |
+| `busy_timeout` | No | Must set every connection |
+| `synchronous` | No | Must set every connection |
+| `cache_size` | No | Must set every connection |
+| `page_size` | Yes | Set before first table creation |
+
+### Connection Initialization Template
+
+```sql
+-- Apply in this order at connection startup
+PRAGMA journal_mode = WAL;
+PRAGMA foreign_keys = ON;
+PRAGMA busy_timeout = 5000;
+PRAGMA synchronous = NORMAL;
+PRAGMA cache_size = -64000;
+PRAGMA temp_store = MEMORY;
+PRAGMA mmap_size = 268435456;
+```
+
+---
+
+## 2. WAL Internals
+
+### How WAL Works
+
+In WAL mode, writes append to a separate WAL file (`database.db-wal`) instead of modifying the main database file. Readers see a consistent snapshot of the database as of the start of their transaction, even while a writer is appending new data.
+
+The WAL file grows until a **checkpoint** transfers committed pages back to the main database file. Checkpointing is automatic by default (after 1000 pages, ~4MB with 4KB page size).
+
+### WAL File Lifecycle
+
+```
+database.db       -- main database file (read by readers via mmap)
+database.db-wal   -- write-ahead log (appended by writers)
+database.db-shm   -- shared memory for WAL index (mmap'd by all connections)
+```
+
+The `-shm` file is a shared-memory index that allows readers to find pages in the WAL file efficiently. Both `-wal` and `-shm` files are automatically created and managed.
+
+### Checkpoint Modes
+
+```sql
+-- Passive: checkpoint pages not currently locked by readers
+PRAGMA wal_checkpoint(PASSIVE);
+
+-- Full: wait for readers to finish, then checkpoint all pages
+PRAGMA wal_checkpoint(FULL);
+
+-- Truncate: like FULL, then truncate WAL file to zero bytes
+PRAGMA wal_checkpoint(TRUNCATE);
+```
+
+Use `PASSIVE` for regular maintenance (non-blocking). Use `TRUNCATE` before backups or when disk space is a concern.
+
+### WAL Limitations
+
+- Only one writer at a time (multiple concurrent readers are fine).
+- WAL does not work on network filesystems (NFS, SMB) -- the shared memory file requires POSIX locking.
+- The WAL file can grow large under sustained write pressure without checkpointing.
+
+---
+
+## 3. FTS5 Tokenizers
+
+### Built-In Tokenizers
+
+| Tokenizer | Behavior | Best For |
+|-----------|----------|----------|
+| `unicode61` | Unicode-aware, folds diacritics, lowercases | General text in any language |
+| `porter unicode61` | Unicode61 + Porter stemming | English-language search |
+| `ascii` | ASCII-only tokenization | ASCII-only data, slightly faster |
+| `trigram` | 3-character substrings | Substring matching, autocomplete |
+
+### Configuring Tokenizers
+
+```sql
+-- Porter stemming with Unicode folding (default for English)
+CREATE VIRTUAL TABLE docs_fts USING fts5(
+    title, body,
+    tokenize='porter unicode61'
+);
+
+-- Trigram for substring search
+CREATE VIRTUAL TABLE names_fts USING fts5(
+    name,
+    tokenize='trigram'
+);
+
+-- Case-sensitive trigram
+CREATE VIRTUAL TABLE code_fts USING fts5(
+    source,
+    tokenize='trigram case_sensitive 1'
+);
+```
+
+### Prefix Indexes
+
+Enable prefix search (autocomplete) by configuring `prefix`:
+
+```sql
+CREATE VIRTUAL TABLE search_fts USING fts5(
+    title, body,
+    tokenize='porter unicode61',
+    prefix='2,3'     -- index 2- and 3-character prefixes
+);
+
+-- Query with prefix
+SELECT * FROM search_fts WHERE search_fts MATCH 'sqli*';
+```
+
+Prefix indexes increase FTS index size but make prefix queries instant instead of scanning all tokens.
+
+---
+
+## 4. External Content Tables
+
+An external-content FTS table stores no content of its own -- it reads from a regular table. This avoids data duplication:
+
+```sql
+CREATE TABLE posts (
+    id INTEGER PRIMARY KEY,
+    title TEXT NOT NULL,
+    body TEXT NOT NULL,
+    created_at TEXT NOT NULL
+);
+
+CREATE VIRTUAL TABLE posts_fts USING fts5(
+    title, body,
+    content='posts',
+    content_rowid='id'
+);
+```
+
+### Keeping External Content in Sync
+
+External-content tables do not update automatically. Use triggers to maintain synchronization:
+
+```sql
+-- After INSERT: add to FTS
+CREATE TRIGGER posts_fts_insert AFTER INSERT ON posts BEGIN
+    INSERT INTO posts_fts(rowid, title, body)
+    VALUES (new.id, new.title, new.body);
+END;
+
+-- After DELETE: remove from FTS
+CREATE TRIGGER posts_fts_delete AFTER DELETE ON posts BEGIN
+    INSERT INTO posts_fts(posts_fts, rowid, title, body)
+    VALUES ('delete', old.id, old.title, old.body);
+END;
+
+-- After UPDATE: remove old, add new
+CREATE TRIGGER posts_fts_update AFTER UPDATE ON posts BEGIN
+    INSERT INTO posts_fts(posts_fts, rowid, title, body)
+    VALUES ('delete', old.id, old.title, old.body);
+    INSERT INTO posts_fts(rowid, title, body)
+    VALUES (new.id, new.title, new.body);
+END;
+```
+
+The `INSERT INTO fts_table(fts_table, ...)` syntax with the table name as the first value is FTS5's delete command. This is not a regular insert -- it instructs FTS5 to remove the specified row from the index.
+
+### Rebuilding the FTS Index
+
+If the FTS index becomes out of sync with the content table, rebuild it:
+
+```sql
+INSERT INTO posts_fts(posts_fts) VALUES ('rebuild');
+```
+
+This scans the entire content table and rebuilds the FTS index from scratch.
+
+---
+
+## 5. FTS5 Query Syntax
+
+### Boolean Operators
+
+```sql
+-- AND (implicit or explicit)
+SELECT * FROM posts_fts WHERE posts_fts MATCH 'sqlite database';
+SELECT * FROM posts_fts WHERE posts_fts MATCH 'sqlite AND database';
+
+-- OR
+SELECT * FROM posts_fts WHERE posts_fts MATCH 'sqlite OR postgres';
+
+-- NOT
+SELECT * FROM posts_fts WHERE posts_fts MATCH 'sqlite NOT tutorial';
+
+-- Phrase search
+SELECT * FROM posts_fts WHERE posts_fts MATCH '"full text search"';
+
+-- Column filter
+SELECT * FROM posts_fts WHERE posts_fts MATCH 'title:sqlite';
+
+-- NEAR (within N tokens)
+SELECT * FROM posts_fts WHERE posts_fts MATCH 'NEAR(sqlite database, 5)';
+```
+
+### Ranking with bm25()
+
+```sql
+SELECT *, bm25(posts_fts) AS relevance
+FROM posts_fts
+WHERE posts_fts MATCH 'search query'
+ORDER BY relevance;
+```
+
+`bm25()` returns negative values where more negative means more relevant. Pass column weights to prioritize title matches over body matches:
+
+```sql
+-- Weight title 10x more than body
+SELECT *, bm25(posts_fts, 10.0, 1.0) AS relevance
+FROM posts_fts
+WHERE posts_fts MATCH 'search query'
+ORDER BY relevance;
+```
+
+### Highlight and Snippet
+
+```sql
+SELECT
+    highlight(posts_fts, 0, '<mark>', '</mark>') AS title,
+    snippet(posts_fts, 1, '<mark>', '</mark>', '...', 64) AS body_preview
+FROM posts_fts
+WHERE posts_fts MATCH 'sqlite';
+```
+
+`highlight()` wraps all matching terms in the specified markers. `snippet()` extracts a relevant fragment with context, truncated to the specified token count.
+
+---
+
+## 6. Partial and Expression Indexes
+
+### Partial Indexes
+
+Reduce index size by indexing only rows that meet a condition:
+
+```sql
+-- Index only active products
+CREATE INDEX idx_products_active ON products(name, price)
+    WHERE status = 'active';
+
+-- Index only future events
+CREATE INDEX idx_events_upcoming ON events(start_date)
+    WHERE start_date > date('now');
+```
+
+The query planner uses a partial index only when the query's `WHERE` clause implies the index condition. Add the same condition to queries:
+
+```sql
+-- Uses idx_products_active
+SELECT * FROM products WHERE status = 'active' AND name LIKE 'A%';
+
+-- Does NOT use idx_products_active (missing status condition)
+SELECT * FROM products WHERE name LIKE 'A%';
+```
+
+### Expression Indexes
+
+```sql
+-- Index on lowercase email for case-insensitive lookup
+CREATE INDEX idx_users_email_lower ON users(lower(email));
+
+-- Index on JSON field
+CREATE INDEX idx_users_theme ON users(json_extract(metadata, '$.theme'));
+
+-- Index on date extracted from timestamp
+CREATE INDEX idx_orders_date ON orders(date(created_at));
+```
+
+---
+
+## 7. ANALYZE
+
+`ANALYZE` collects statistics about table and index contents, stored in the `sqlite_stat1` table. The query planner uses these statistics to choose optimal execution plans:
+
+```sql
+-- Analyze all tables and indexes
+ANALYZE;
+
+-- Analyze a specific table
+ANALYZE users;
+
+-- Analyze a specific index
+ANALYZE idx_users_email;
+```
+
+### When to Run ANALYZE
+
+- After bulk data loads that significantly change data distribution.
+- After creating indexes on populated tables.
+- Periodically in long-running applications as data distribution shifts.
+
+Without `ANALYZE`, the query planner assumes uniform distribution and may choose suboptimal plans for skewed data.
+
+### Inspecting Statistics
+
+```sql
+SELECT * FROM sqlite_stat1;
+-- tbl: table name, idx: index name, stat: "nrow n1 n2 ..."
+-- nrow = rows in table, n1 = avg rows per first index column value
+```
+
+Low selectivity values (high n1 relative to nrow) indicate the index is not selective enough for certain queries. Consider a composite index or a different query strategy.