dirsql 0.3.65 → 0.3.67

package/docs/guide/async.md

@@ -1,268 +0,0 @@
----
-canonical: https://thekevinscott.github.io/dirsql/guide/async
----
-
-# Async API
-
-> Online: <https://thekevinscott.github.io/dirsql/guide/async>
-
-`DirSQL` is async by default in Python. The initial directory scan runs in a background thread so it does not block the event loop.
-
-## Basic usage
-
-::: code-group
-
-```python [Python]
-import asyncio
-import json
-from dirsql import DirSQL, Table
-
-async def main():
-    db = DirSQL(
-        "./my-project",
-        tables=[
-            Table(
-                ddl="CREATE TABLE items (name TEXT, value INTEGER)",
-                glob="data/*.json",
-                extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
-            ),
-        ],
-    )
-    await db.ready()
-
-    # Query (runs in a thread, does not block the event loop)
-    results = await db.query("SELECT * FROM items WHERE value > 10")
-    print(results)
-
-asyncio.run(main())
-```
-
-```rust [Rust]
-use dirsql::{DirSQL, Table, Value};
-use std::collections::HashMap;
-
-// See `row_from_json` in getting-started.md for a reusable helper that
-// turns a JSON object into a dirsql row (dirsql::Value is not Deserialize,
-// so a row can't be produced by serde_json::from_str directly).
-fn row_from_json(raw: &str) -> HashMap<String, Value> {
-    let v: serde_json::Value = serde_json::from_str(raw).unwrap();
-    let serde_json::Value::Object(obj) = v else { return HashMap::new() };
-    obj.into_iter()
-        .map(|(k, val)| {
-            let v = match val {
-                serde_json::Value::String(s) => Value::Text(s),
-                serde_json::Value::Number(n) => n
-                    .as_i64()
-                    .map(Value::Integer)
-                    .unwrap_or_else(|| Value::Real(n.as_f64().unwrap_or(0.0))),
-                serde_json::Value::Bool(b) => Value::Integer(b as i64),
-                serde_json::Value::Null => Value::Null,
-                other => Value::Text(other.to_string()),
-            };
-            (k, v)
-        })
-        .collect()
-}
-
-#[tokio::main]
-async fn main() -> Result<(), Box<dyn std::error::Error>> {
-    let db = DirSQL::new(
-        "./my-project",
-        vec![
-            Table::new(
-                "CREATE TABLE items (name TEXT, value INTEGER)",
-                "data/*.json",
-                |path| vec![row_from_json(&std::fs::read_to_string(path).unwrap())],
-            ),
-        ],
-    )?;
-
-    let results = db.query("SELECT * FROM items WHERE value > 10")?;
-    println!("{:?}", results);
-    Ok(())
-}
-```
-
-```typescript [TypeScript]
-import { readFileSync } from 'node:fs';
-import { DirSQL, Table } from 'dirsql';
-
-const db = new DirSQL({
-  root: './my-project',
-  tables: [
-    new Table({
-      ddl: 'CREATE TABLE items (name TEXT, value INTEGER)',
-      glob: 'data/*.json',
-      extract: (path) => [JSON.parse(readFileSync(path, 'utf8'))],
-    }),
-  ],
-});
-
-// Query
-const results = await db.query('SELECT * FROM items WHERE value > 10');
-console.log(results);
-```
-
-:::
-
-## Constructor
-
-```python
-DirSQL(root=None, *, tables=None, ignore=None, config=None, persist=False, persist_path=None)
-```
-
-The constructor immediately starts scanning in a background thread via `asyncio.ensure_future`. The constructor itself returns immediately without blocking.
-
-## `await db.ready()`
-
-Waits until the initial directory scan is complete. If the scan raised an exception (invalid DDL, unreadable files, etc.), `ready()` re-raises it.
-
-`ready()` can be called multiple times safely. After the first completion, subsequent calls return immediately.
-
-```python
-db = DirSQL("./data", tables=[...])
-
-# Do other setup work here while the scan runs in the background
-setup_logging()
-connect_websocket()
-
-# Now wait for the scan to finish before querying
-await db.ready()
-```
-
-## `await db.query(sql)`
-
-Runs a SQL query in a background thread. Returns a list of dicts keyed by column name.
-
-```python
-results = await db.query("SELECT COUNT(*) as n FROM items")
-```
-
-## `async for event in db.watch()`
-
-Returns an async iterable of `RowEvent` objects. The watcher is started automatically on the first iteration.
-
-::: code-group
-
-```python [Python]
-async for event in db.watch():
-    if event.action == "insert":
-        print(f"New row in {event.table}: {event.row}")
-    elif event.action == "update":
-        print(f"Updated row in {event.table}: {event.row}")
-    elif event.action == "delete":
-        print(f"Deleted row from {event.table}: {event.row}")
-    elif event.action == "error":
-        print(f"Error: {event.error}")
-```
-
-```rust [Rust]
-// `RowEvent` is an enum; match on the variant to destructure its fields.
-// `StreamExt` (for `.next()`) comes from the `futures` crate, which is only a
-// dirsql dependency under its `cli` feature -- add it to your own project:
-//
-//     cargo add futures
-use dirsql::RowEvent;
-use futures::StreamExt;
-
-let mut stream = db.watch()?; // watch() returns Result<WatchStream>
-while let Some(event) = stream.next().await {
-    match event {
-        RowEvent::Insert { table, row, file_path } => {
-            println!("New row in {table} ({file_path}): {row:?}")
-        }
-        RowEvent::Update { table, new_row, file_path, .. } => {
-            println!("Updated row in {table} ({file_path}): {new_row:?}")
-        }
-        RowEvent::Delete { table, row, file_path } => {
-            println!("Deleted row from {table} ({file_path}): {row:?}")
-        }
-        RowEvent::Error { file_path, error, .. } => {
-            eprintln!("Error on {file_path:?}: {error}")
-        }
-    }
-}
-```
-
-```typescript [TypeScript]
-for await (const event of db.watch()) {
-  switch (event.action) {
-    case 'insert':
-      console.log(`New row in ${event.table}:`, event.row);
-      break;
-    case 'update':
-      console.log(`Updated row in ${event.table}:`, event.row);
-      break;
-    case 'delete':
-      console.log(`Deleted row from ${event.table}:`, event.row);
-      break;
-    case 'error':
-      console.error(`Error: ${event.error}`);
-      break;
-  }
-}
-```
-
-:::
-
-The async iterator polls for filesystem events with a 200ms timeout internally. It yields events as they arrive and never terminates on its own -- use `break` or cancellation to stop.
-
-## Combining with other async code
-
-The async API works alongside other concurrent code without blocking:
-
-::: code-group
-
-```python [Python]
-async def watch_and_serve(db):
-    async for event in db.watch():
-        await notify_clients(event)
-
-async def main():
-    db = DirSQL("./data", tables=[...])
-    await asyncio.gather(
-        watch_and_serve(db),
-        run_web_server(),
-    )
-```
-
-```rust [Rust]
-// `.next()` needs `StreamExt` from the `futures` crate (`cargo add futures`).
-use futures::StreamExt;
-
-async fn watch_and_serve(db: &DirSQL) -> Result<(), Box<dyn std::error::Error>> {
-    let mut stream = db.watch()?; // watch() returns Result<WatchStream>
-    while let Some(event) = stream.next().await {
-        notify_clients(&event).await;
-    }
-    Ok(())
-}
-
-#[tokio::main]
-async fn main() -> Result<(), Box<dyn std::error::Error>> {
-    let db = DirSQL::new("./data", vec![/* tables */])?;
-
-    tokio::join!(
-        watch_and_serve(&db),
-        run_web_server(),
-    );
-    Ok(())
-}
-```
-
-```typescript [TypeScript]
-async function watchAndServe(db: DirSQL) {
-  for await (const event of db.watch()) {
-    await notifyClients(event);
-  }
-}
-
-const db = new DirSQL({ root: './data', tables: [/* tables */] });
-
-await Promise.all([
-  watchAndServe(db),
-  runWebServer(),
-]);
-```
-
-:::

package/docs/guide/crdt.md

@@ -1,161 +0,0 @@
----
-canonical: https://thekevinscott.github.io/dirsql/guide/crdt
----
-
-# Collaboration with CRDTs
-
-> Online: <https://thekevinscott.github.io/dirsql/guide/crdt>
-
-`dirsql` treats the filesystem as the source of truth. That works well when a single process (or a single human) is writing, but breaks down for multi-writer collaboration: two peers editing the same file concurrently produce a merge conflict, not a merged result.
-
-[Conflict-free Replicated Data Types](https://crdt.tech/) (CRDTs) solve that merge problem at the data-structure level, not the filesystem level. Two replicas that apply the same set of edits -- in any order, with any network partitions in between -- converge on the same final state, without a central arbiter.
-
-This guide is **opinionated**. It picks one library, explains the integration pattern with `dirsql`, and names the alternatives so you can steer if your situation is different.
-
-## Recommendation: Automerge
-
-Use [Automerge](https://automerge.org/) (specifically the 2.x series with [automerge-repo](https://automerge.org/automerge-repo/) for sync).
-
-Why Automerge over the alternatives:
-
-- **JSON-shaped document model.** Automerge docs look like nested maps, lists, and text. That matches `dirsql`'s one-object-per-file workflow -- each Automerge document is the thing your `extract` function projects into rows.
-- **Cross-language SDKs that mirror `dirsql`'s parity story.** First-class Rust ([`automerge`](https://crates.io/crates/automerge)), TypeScript ([`@automerge/automerge`](https://www.npmjs.com/package/@automerge/automerge)), and Python ([`automerge`](https://pypi.org/project/automerge/)) implementations exist today, all driven by the same Rust core. If you already have all three `dirsql` SDKs in play, Automerge won't force a language monoculture on you.
-- **Filesystem-friendly sync primitives.** `automerge-repo` ships a [`NodeFSStorageAdapter`](https://automerge.org/docs/repositories/storage/) that shards document history into regular files under a directory. That directory is exactly the kind of tree `dirsql` is designed to watch.
-- **Binary format with a deterministic JSON view.** You never hand-edit a CRDT file, but every replica projects the same canonical JSON from the binary state. That canonical JSON is what you feed to `dirsql`'s extract function, so two peers that have synced will produce identical rows.
-
-## The integration shape
-
-There are two files per logical "document":
-
-```
-workspace/
-  posts/
-    hello/
-      doc.automerge   <-- binary CRDT state (the source of truth for writers)
-      view.json       <-- materialized JSON snapshot (written on each merge)
-```
-
-- Writers (editors, sync peers, etc.) mutate `doc.automerge` through Automerge APIs.
-- After every mutation, the writer serializes the current document to `view.json`. This is the file `dirsql` indexes.
-- `dirsql` watches `view.json`, not `doc.automerge`. The CRDT file is an implementation detail of how the JSON got there.
-
-This keeps `dirsql`'s extract function oblivious to CRDT semantics: it reads a plain JSON file, exactly as it would without Automerge.
-
-::: tip Why not `extract` directly from `.automerge`?
-You could -- the Rust and Python Automerge SDKs let you load a binary doc and walk its fields. But it couples your table schema to the CRDT library version, makes `extract` non-pure (it allocates CRDT state on every file change), and buys nothing: the writer is the only place that can produce a valid Automerge blob, so it might as well produce the JSON view at the same time.
-:::
-
-### Example: posts as Automerge documents
-
-::: code-group
-
-```python [Python]
-from dirsql import DirSQL, Table
-import json
-
-db = DirSQL(
-    "./workspace",
-    tables=[
-        Table(
-            ddl="CREATE TABLE posts (id TEXT, title TEXT, body TEXT, updated INTEGER)",
-            # Match the JSON view, not the raw CRDT binary.
-            glob="posts/*/view.json",
-            extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
-        ),
-    ],
-)
-```
-
-```rust [Rust]
-use dirsql::{DirSQL, Table};
-// See `row_from_json` in getting-started.md.
-
-let db = DirSQL::new(
-    "./workspace",
-    vec![
-        Table::new(
-            "CREATE TABLE posts (id TEXT, title TEXT, body TEXT, updated INTEGER)",
-            "posts/*/view.json",
-            |path| vec![row_from_json(&std::fs::read_to_string(path).unwrap())],
-        ),
-    ],
-)?;
-```
-
-```typescript [TypeScript]
-import { readFileSync } from 'node:fs';
-import { DirSQL, type TableDef } from 'dirsql';
-
-const tables: TableDef[] = [
-  {
-    ddl: 'CREATE TABLE posts (id TEXT, title TEXT, body TEXT, updated INTEGER)',
-    glob: 'posts/*/view.json',
-    extract: (path) => [JSON.parse(readFileSync(path, 'utf8'))],
-  },
-];
-
-const db = new DirSQL({ root: './workspace', tables });
-```
-
-:::
-
-The Automerge writer (sketch, TypeScript):
-
-```typescript
-import * as Automerge from '@automerge/automerge';
-import { writeFileSync, readFileSync } from 'node:fs';
-
-// Load (or create) the CRDT doc.
-const bytes = readFileSync('workspace/posts/hello/doc.automerge');
-let doc = Automerge.load<Post>(bytes);
-
-// Apply an edit.
-doc = Automerge.change(doc, (d) => {
-  d.title = 'Hello, world';
-  d.updated = Date.now();
-});
-
-// Persist both the CRDT state and the materialized view.
-writeFileSync('workspace/posts/hello/doc.automerge', Automerge.save(doc));
-writeFileSync('workspace/posts/hello/view.json', JSON.stringify(doc));
-```
-
-`dirsql`'s watcher picks up the change to `view.json`, re-runs `extract`, and emits an `update` row event. Queries over `posts` reflect the merged state without `dirsql` knowing Automerge exists.
-
-## Multi-writer, in practice
-
-1. Each peer runs an `automerge-repo` instance with the filesystem storage adapter pointed at its local `workspace/`.
-2. Peers sync via any transport `automerge-repo` supports ([WebSocket](https://automerge.org/docs/repositories/networking/), [BroadcastChannel](https://automerge.org/docs/repositories/networking/), or a custom adapter).
-3. On every sync, the repo applies incoming ops to the local CRDT, writes the updated `doc.automerge`, and the writer code re-serializes `view.json`.
-4. Every peer's `dirsql` sees the same eventual `view.json` and produces the same rows.
-
-The key invariant: **`view.json` is a deterministic projection of `doc.automerge`**. Two peers that have converged on the CRDT state must produce byte-identical (or at least semantically-identical) JSON views. Otherwise you get spurious `update` events that flap with sync order. Use `JSON.stringify` with sorted keys (or `json.dumps(..., sort_keys=True)` in Python) to guarantee this.
-
-## Tradeoffs vs plain files
-
-When **plain files** are the right answer:
-
-- Single writer. A solo user editing `posts/*.json` will never hit a merge conflict. Adding a CRDT is overhead.
-- Human-readable history matters. `git diff` on a JSON file tells a story; `git diff` on a CRDT binary does not.
-- Schema churn is frequent. Renaming a field in plain JSON is a `sed`; in a CRDT it's a migration.
-
-When **CRDTs** earn their complexity:
-
-- Multi-writer without a central server (local-first, peer-to-peer).
-- Offline edits that need to merge on reconnect.
-- Fine-grained collaborative editing (cursor-level merging of a shared text field).
-
-Hybrid is common: keep configuration and reference data as plain files, and use CRDTs only for the documents that genuinely have multiple writers.
-
-## Alternatives we considered
-
-- [**Yjs**](https://docs.yjs.dev/) -- the dominant JS CRDT, excellent for rich-text collaboration (it backs many of the production collab editors you've used). Skipped as the primary recommendation because its Rust port ([`yrs`](https://crates.io/crates/yrs)) and Python bindings lag the JS implementation. If your workload is browser-first and text-heavy, prefer Yjs.
-- [**Loro**](https://loro.dev/) -- Rust-first CRDT with a clean API and good cross-language story. Worth watching; we'd consider it once its Python bindings are GA. Try it if you're Rust-centric and don't need Automerge's ecosystem.
-- **Operational Transform / hand-rolled merge logic** -- don't. OT is correct but hard to implement right, and you lose the offline-peer story that CRDTs give you for free.
-- **Git as the merge engine** -- tempting because `dirsql` already lives on the filesystem, but three-way merges of structured JSON produce garbage conflict markers that no extract function can parse. Use a CRDT.
-
-## See also
-
-- [Ink & Switch's local-first essay](https://www.inkandswitch.com/local-first/) -- the design space CRDTs sit in.
-- [Automerge documentation](https://automerge.org/docs/) -- API reference and sync-adapter guides.
-- [`crdt.tech`](https://crdt.tech/) -- library survey across languages.

package/docs/guide/persistence.md

@@ -1,177 +0,0 @@
-# Persistence
-
-By default `dirsql` keeps its SQLite database in memory and rebuilds it from scratch every time the process starts. For large directories this can take seconds to minutes -- nearly all of which is spent re-parsing files that haven't changed since the previous run.
-
-Persistence stores the SQLite database on disk so that subsequent startups only re-parse the files that have actually changed.
-
-::: tip Same answers, faster startup
-The rows returned by `query()` after a persistent startup are equivalent to those produced by a from-scratch rebuild. Persistence is a startup-time optimization, not a correctness compromise. The reconcile algorithm is the same one `git status` uses to decide which files have changed since the last index write.
-:::
-
-## Quick start
-
-::: code-group
-
-```toml [.dirsql.toml]
-[dirsql]
-persist = true
-```
-
-```python [Python]
-from dirsql import DirSQL
-
-db = DirSQL("./my-project", tables=[...], persist=True)
-await db.ready()
-```
-
-```rust [Rust]
-use dirsql::DirSQL;
-
-let db = DirSQL::builder()
-    .root("./my-project")
-    .tables(vec![/* ... */])
-    .persist(true)
-    .build()?;
-```
-
-```typescript [TypeScript]
-import { DirSQL } from "dirsql";
-
-const db = new DirSQL({ root: "./my-project", tables: [/* ... */], persist: true });
-await db.ready;
-```
-
-:::
-
-That's it. The first run writes the database to `./my-project/.dirsql/cache.db`. Every subsequent startup uses the cache.
-
-## Configuration
-
-| Option | Type | Default | Meaning |
-|---|---|---|---|
-| `persist` | boolean | `false` | Enable persistent on-disk storage. |
-| `persist_path` (Python, Rust) / `persistPath` (TypeScript) | string | `<root>/.dirsql/cache.db` | Override the database file path. Ignored when `persist` is `false`. |
-
-The default location keeps the cache alongside the data it indexes, which means it follows the project around (clone, copy, move) without extra setup. Override `persist_path` if you want the cache somewhere else -- a CI cache directory, a tmpfs mount, an XDG cache dir, etc.
-
-::: code-group
-
-```toml [.dirsql.toml]
-[dirsql]
-persist = true
-persist_path = "/var/cache/dirsql/myproject.db"
-```
-
-```python [Python]
-db = DirSQL(
-    "./my-project",
-    tables=[...],
-    persist=True,
-    persist_path="/var/cache/dirsql/myproject.db",
-)
-```
-
-```rust [Rust]
-let db = DirSQL::builder()
-    .root("./my-project")
-    .tables(vec![/* ... */])
-    .persist(true)
-    .persist_path("/var/cache/dirsql/myproject.db")
-    .build()?;
-```
-
-```typescript [TypeScript]
-const db = new DirSQL({
-  root: "./my-project",
-  tables: [/* ... */],
-  persist: true,
-  persistPath: "/var/cache/dirsql/myproject.db",
-});
-```
-
-:::
-
-## The `.dirsql/` directory
-
-`dirsql` reserves the top-level `.dirsql/` directory inside every scanned root. It is **unconditionally excluded from the directory walk**, whether persistence is enabled or not. This means:
-
-- The default cache path `<root>/.dirsql/cache.db` cannot accidentally be ingested as a data file.
-- You can place additional `dirsql`-related files in `.dirsql/` (e.g. a project-local config snapshot) without them being parsed.
-- You should not put your own data files in `.dirsql/` -- they will be silently ignored.
-
-If you persist into `.dirsql/`, add it to your `.gitignore`:
-
-```
-.dirsql/
-```
-
-The cache file should never be committed -- it is reproducible from the source tree and frequently large.
-
-## How the startup reconcile works
-
-When a persistent cache exists, `dirsql` does not blindly trust it. On startup it:
-
-1. **Checks compatibility metadata.** If the cached `dirsql` version, schema version, glob configuration, parser versions, or canonical root path differs from the current build, the cache is wiped and rebuilt from scratch.
-2. **Walks the tree and stats every matching file.** This is metadata-only -- no file contents are read.
-3. **For each file, compares the live `(size, mtime, ctime, inode, dev)` tuple against the cached row:**
-   - **Trust the cache** when every field matches *and* the file's mtime is older than the cache's snapshot time (outside the racy window).
-   - **Hash-confirm** when the tuple matches but the file's mtime falls inside the racy window. `dirsql` reads and hashes the file; if the hash matches the cached hash, the cache is trusted.
-   - **Re-parse** when any field of the tuple differs.
-4. **Deletes** rows for files that were in the cache but are no longer on disk.
-5. **Inserts** rows for files that are on disk but were not in the cache.
-
-This is the same algorithm `git status` uses to decide which files have changed since the last index write. The "racy window" handling is what closes the gap when a file is modified within the same filesystem-timestamp resolution as the cache write.
-
-## When `dirsql` does a full rebuild
-
-Any of the following will cause the cache to be discarded and rebuilt from scratch on the next startup:
-
-- The `dirsql` library was upgraded between runs.
-- The glob configuration changed (a new table, a removed table, a modified glob, a changed `ignore` list).
-- A built-in parser version changed (this generally only happens on `dirsql` upgrades).
-- The cache was written for a different root directory than the one currently configured.
-- The internal schema of the cache changed (i.e. you upgraded `dirsql` across a schema version bump).
-
-Full rebuilds take exactly as long as a non-persistent startup -- there is no penalty for them, only a missed optimization.
-
-## Limitations
-
-### Network filesystems
-
-NFS, SMB/CIFS, and similar network filesystems cache file attributes on the client and can return stale `stat` results. Persistent mode is **not supported** on network filesystems and may produce stale rows. Use in-memory mode (the default) if your `root` lives on a network mount.
-
-### The mtime-preservation edge case
-
-Racy-stat detection misses changes only when **all** of the following are true:
-
-- A file's contents are modified.
-- The file's size after modification is identical to its size before.
-- The file's `mtime` is externally reset to a value older than the cache's snapshot time (e.g. via `touch -r` or a backup-restore tool that preserves mtime).
-
-If you cannot tolerate this edge case, disable persistence (`persist = false`). This is the same trade-off `git` makes with `core.trustctime` / `core.checkStat`.
-
-### Single writer
-
-Only one `dirsql` process should write to a given cache file at a time. Multiple read-only processes can query the same file safely once the writer finishes the initial reconcile. Coordinated multi-writer access is not supported in v0.3.0.
-
-## Inspecting the cache
-
-The persistent database is a normal SQLite file. You can open it with any SQLite client:
-
-```bash
-sqlite3 .dirsql/cache.db
-```
-
-```sql
-.tables
--- comments  documents  metrics  _dirsql_files  _dirsql_meta
-
-SELECT * FROM _dirsql_meta;
--- schema_version    | 1
--- dirsql_version    | 0.3.0
--- glob_config_hash  | <hex>
--- parser_versions   | {"json":"1","jsonl":"1","csv":"1",...}
--- root_canonical    | /home/alice/my-project
-```
-
-The `_dirsql_files` and `_dirsql_meta` tables are managed by `dirsql`. Do not modify them by hand -- on the next startup, `dirsql` will detect the inconsistency and rebuild from scratch.