dirsql 0.2.4 → 0.2.8

package/docs/api/index.md

@@ -0,0 +1,232 @@
+---
+canonical: https://thekevinscott.github.io/dirsql/api/
+---
+
+# API Reference
+
+> Online: <https://thekevinscott.github.io/dirsql/api/>
+
+## DirSQL
+
+### Import
+
+::: code-group
+
+```python [Python]
+from dirsql import DirSQL
+```
+
+```rust [Rust]
+use dirsql::DirSQL;
+```
+
+```typescript [TypeScript]
+import { DirSQL } from 'dirsql';
+```
+
+:::
+
+### Constructor
+
+::: code-group
+
+```python [Python]
+DirSQL(
+    root: str | None = None,
+    *,
+    tables: list[Table] | None = None,
+    ignore: list[str] | None = None,
+    config: str | None = None,
+)
+```
+
+```rust [Rust]
+DirSQL::builder()
+    .root(root)                 // optional
+    .tables(tables)             // optional; append with .table(t)
+    .ignore(patterns)           // optional
+    .config(config_toml_path)   // optional
+    .build()                    // -> Result<DirSQL>
+```
+
+```typescript [TypeScript]
+new DirSQL(configPath: string)
+// or
+new DirSQL({
+    root?: string,
+    tables?: TableDef[],
+    ignore?: string[],
+    config?: string,
+})
+```
+
+:::
+
+Creates an in-memory SQLite index over the given directory. At least one of `root` or `config` must be supplied.
+
+When both `root` and `config` are supplied -- or when `config` declares `[dirsql].root` -- the explicit `root` wins and a warning is emitted on stderr. A `[dirsql].root` declared in the config file is resolved relative to the config file's parent directory.
+
+In Python, the constructor starts scanning in a background thread and returns immediately. Call `await db.ready()` before querying. In Rust, `.build()` scans synchronously; use `.build_async()` (via `AsyncDirSQL`) for the tokio-driven equivalent. In TypeScript, scanning starts immediately and `db.ready` resolves when the scan finishes.
+
+**Parameters:**
+
+- `root` -- Path to the directory to index. Optional if `config` is supplied.
+- `tables` -- List of `Table` definitions. Each defines a SQLite table, a glob pattern, and an extract function.
+- `ignore` -- Optional list of glob patterns. Files matching any ignore pattern are skipped regardless of table globs.
+- `config` -- Optional path to a `.dirsql.toml` config file. Its `[[table]]` entries are appended to any programmatic `tables`; its `[dirsql].ignore` patterns are appended to any explicit `ignore`; its optional `[dirsql].root` supplies the root directory when `root` is not passed explicitly.
+
+### Methods
+
+#### `ready`
+
+::: code-group
+
+```python [Python]
+await db.ready() -> None
+```
+
+```rust [Rust]
+db.ready().await -> Result<()>
+```
+
+```typescript [TypeScript]
+await db.ready  // awaitable property
+```
+
+:::
+
+Wait for the initial scan to complete. Re-raises any exception from the scan. Safe to call multiple times.
+
+#### `query`
+
+::: code-group
+
+```python [Python]
+await db.query(sql: str) -> list[dict]
+```
+
+```rust [Rust]
+db.query(sql: &str) -> Result<Vec<HashMap<String, Value>>>
+```
+
+```typescript [TypeScript]
+await db.query(sql: string): Promise<Record<string, unknown>[]>
+```
+
+:::
+
+Execute a SQL query against the in-memory database. Returns results keyed by column name. Internal tracking columns (`_dirsql_file_path`, `_dirsql_row_index`) are excluded from results.
+
+#### `watch`
+
+::: code-group
+
+```python [Python]
+async for event in db.watch():  # AsyncIterator[RowEvent]
+    ...
+```
+
+```rust [Rust]
+let mut stream = db.watch();  // impl Stream<Item = RowEvent>
+while let Some(event) = stream.next().await { ... }
+```
+
+```typescript [TypeScript]
+for await (const event of db.watch()) {  // AsyncIterable<RowEvent>
+    ...
+}
+```
+
+:::
+
+Returns an async iterable of `RowEvent` objects. The file watcher starts automatically on first iteration. The iterator never terminates on its own.
+
+---
+
+## Table
+
+### Import
+
+::: code-group
+
+```python [Python]
+from dirsql import Table
+```
+
+```rust [Rust]
+use dirsql::Table;
+```
+
+```typescript [TypeScript]
+import { Table } from 'dirsql';
+```
+
+:::
+
+### Constructor
+
+::: code-group
+
+```python [Python]
+Table(*, ddl: str, glob: str, extract: Callable[[str, str], list[dict]])
+```
+
+```rust [Rust]
+Table::new(ddl: &str, glob: &str, extract: fn(&str, &str) -> Vec<Value>)
+```
+
+```typescript [TypeScript]
+new Table({ ddl: string, glob: string, extract: (path: string, content: string) => Record<string, unknown>[] })
+```
+
+:::
+
+Defines a mapping from files to SQLite table rows.
+
+**Parameters:**
+
+- `ddl` -- A `CREATE TABLE` statement. The table name is parsed from this DDL.
+- `glob` -- A glob pattern matched against file paths relative to the root directory.
+- `extract` -- A callable `(path, content) -> list[dict]`. Receives the relative file path and file content as strings. Returns a list of dicts/maps mapping column names to values. Return an empty list to skip a file.
+
+**Attributes:**
+
+- `ddl` -- The DDL string (read-only).
+- `glob` -- The glob pattern (read-only).
+
+---
+
+## RowEvent
+
+### Import
+
+::: code-group
+
+```python [Python]
+from dirsql import RowEvent
+```
+
+```rust [Rust]
+use dirsql::RowEvent;
+```
+
+```typescript [TypeScript]
+import { RowEvent } from 'dirsql';
+```
+
+:::
+
+Emitted by the watch stream. Represents a change to a row in the database caused by a filesystem event.
+
+**Attributes:**
+
+| Attribute | Python | Rust | TypeScript |
+|-----------|--------|------|------------|
+| Table name | `table: str` | `table: String` | `table: string` |
+| Action | `action: str` | `action: Action` | `action: string` |
+| Current/new row | `row: dict \| None` | `row: Option<HashMap>` | `row?: Record` |
+| Previous row | `old_row: dict \| None` | `old_row: Option<HashMap>` | `oldRow?: Record` |
+| Error message | `error: str \| None` | `error: Option<String>` | `error?: string` |
+| File path | `file_path: str \| None` | `file_path: Option<String>` | `filePath?: string` |
+
+Action values: `"insert"`, `"update"`, `"delete"`, `"error"`.

package/docs/getting-started.md

@@ -0,0 +1,186 @@
+---
+canonical: https://thekevinscott.github.io/dirsql/getting-started
+---
+
+# Getting Started
+
+> Online: <https://thekevinscott.github.io/dirsql/getting-started>
+
+## Installation
+
+::: code-group
+
+```bash [Python]
+pip install dirsql
+```
+
+```bash [Rust]
+cargo add dirsql
+```
+
+```bash [TypeScript]
+pnpm add dirsql
+```
+
+```bash [CLI]
+# Pick whichever install path you already have handy
+npx dirsql --version
+uvx dirsql --version
+cargo install dirsql --features cli
+```
+
+:::
+
+See the [CLI guide](./guide/cli.md) for details on the command-line interface, and the [Rust library README](https://github.com/thekevinscott/dirsql/tree/main/packages/rust) for the library-vs-CLI feature split.
+
+## Quick start
+
+Suppose you have a directory of JSON files representing blog posts:
+
+```
+my-blog/
+  posts/
+    hello.json      # {"title": "Hello World", "author": "alice"}
+    second.json     # {"title": "Second Post", "author": "bob"}
+  authors/
+    alice.json      # {"id": "alice", "name": "Alice"}
+    bob.json        # {"id": "bob", "name": "Bob"}
+```
+
+Index and query them with `dirsql`:
+
+::: code-group
+
+```python [Python]
+import asyncio
+import json
+from dirsql import DirSQL, Table
+
+async def main():
+    db = DirSQL(
+        "./my-blog",
+        tables=[
+            Table(
+                ddl="CREATE TABLE posts (title TEXT, author TEXT)",
+                glob="posts/*.json",
+                extract=lambda path, content: [json.loads(content)],
+            ),
+            Table(
+                ddl="CREATE TABLE authors (id TEXT, name TEXT)",
+                glob="authors/*.json",
+                extract=lambda path, content: [json.loads(content)],
+            ),
+        ],
+    )
+    await db.ready()
+
+    # Query all posts
+    posts = await db.query("SELECT * FROM posts")
+    # [{"title": "Hello World", "author": "alice"}, {"title": "Second Post", "author": "bob"}]
+
+    # Join across tables
+    results = await db.query("""
+        SELECT posts.title, authors.name
+        FROM posts
+        JOIN authors ON posts.author = authors.id
+    """)
+    # [{"title": "Hello World", "name": "Alice"}, {"title": "Second Post", "name": "Bob"}]
+
+asyncio.run(main())
+```
+
+```rust [Rust]
+use dirsql::{DirSQL, Table, Value};
+use std::collections::HashMap;
+
+// Convert a JSON object string into a dirsql row.
+fn row_from_json(raw: &str) -> HashMap<String, Value> {
+    let v: serde_json::Value = serde_json::from_str(raw).unwrap();
+    let serde_json::Value::Object(obj) = v else { return HashMap::new() };
+    obj.into_iter()
+        .map(|(k, val)| {
+            let v = match val {
+                serde_json::Value::String(s) => Value::Text(s),
+                serde_json::Value::Number(n) => n
+                    .as_i64()
+                    .map(Value::Integer)
+                    .unwrap_or_else(|| Value::Real(n.as_f64().unwrap_or(0.0))),
+                serde_json::Value::Bool(b) => Value::Integer(b as i64),
+                serde_json::Value::Null => Value::Null,
+                other => Value::Text(other.to_string()),
+            };
+            (k, v)
+        })
+        .collect()
+}
+
+let db = DirSQL::new(
+    "./my-blog",
+    vec![
+        Table::new(
+            "CREATE TABLE posts (title TEXT, author TEXT)",
+            "posts/*.json",
+            |_path, content| vec![row_from_json(content)],
+        ),
+        Table::new(
+            "CREATE TABLE authors (id TEXT, name TEXT)",
+            "authors/*.json",
+            |_path, content| vec![row_from_json(content)],
+        ),
+    ],
+)?;
+
+let posts = db.query("SELECT * FROM posts")?;
+
+let results = db.query(
+    "SELECT posts.title, authors.name \
+     FROM posts JOIN authors ON posts.author = authors.id"
+)?;
+```
+
+```typescript [TypeScript]
+import { DirSQL, type TableDef } from 'dirsql';
+
+const tables: TableDef[] = [
+  {
+    ddl: 'CREATE TABLE posts (title TEXT, author TEXT)',
+    glob: 'posts/*.json',
+    extract: (_path, content) => [JSON.parse(content)],
+  },
+  {
+    ddl: 'CREATE TABLE authors (id TEXT, name TEXT)',
+    glob: 'authors/*.json',
+    extract: (_path, content) => [JSON.parse(content)],
+  },
+];
+
+const db = new DirSQL({ root: './my-blog', tables });
+
+const posts = await db.query('SELECT * FROM posts');
+
+const results = await db.query(`
+  SELECT posts.title, authors.name
+  FROM posts JOIN authors ON posts.author = authors.id
+`);
+```
+
+:::
+
+## What happens at startup
+
+1. `dirsql` walks the directory tree
+2. Files matching each table's glob pattern are read
+3. The `extract` function converts file content into rows
+4. Rows are inserted into an in-memory SQLite database
+5. SQL queries run against that database
+
+The filesystem is always the source of truth. The database is rebuilt from files at startup.
+
+## Next steps
+
+- [Defining Tables](./guide/tables.md) -- DDL, globs, and extract functions in detail
+- [Querying](./guide/querying.md) -- SQL queries and return format
+- [File Watching](./guide/watching.md) -- real-time change events
+- [Async API](./guide/async.md) -- async ready(), query(), and watch()
+- [Command-Line Interface](./guide/cli.md) -- `dirsql` runs an HTTP server (`POST /query`, `GET /events` SSE)
+- [Collaboration with CRDTs](./guide/crdt.md) -- multi-writer document merging alongside `dirsql`

package/docs/guide/async.md

@@ -0,0 +1,224 @@
+---
+canonical: https://thekevinscott.github.io/dirsql/guide/async
+---
+
+# Async API
+
+> Online: <https://thekevinscott.github.io/dirsql/guide/async>
+
+`DirSQL` is async by default in Python. The initial directory scan runs in a background thread so it does not block the event loop.
+
+## Basic usage
+
+::: code-group
+
+```python [Python]
+import asyncio
+import json
+from dirsql import DirSQL, Table
+
+async def main():
+    db = DirSQL(
+        "./my-project",
+        tables=[
+            Table(
+                ddl="CREATE TABLE items (name TEXT, value INTEGER)",
+                glob="data/*.json",
+                extract=lambda path, content: [json.loads(content)],
+            ),
+        ],
+    )
+
+    # Query (runs in a thread, does not block the event loop)
+    results = await db.query("SELECT * FROM items WHERE value > 10")
+    print(results)
+
+asyncio.run(main())
+```
+
+```rust [Rust]
+use dirsql::{DirSQL, Table};
+
+#[tokio::main]
+async fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let db = DirSQL::new(
+        "./my-project",
+        vec![
+            Table::new(
+                "CREATE TABLE items (name TEXT, value INTEGER)",
+                "data/*.json",
+                |_path, content| vec![serde_json::from_str(content).unwrap()],
+            ),
+        ],
+    )?;
+
+    let results = db.query("SELECT * FROM items WHERE value > 10")?;
+    println!("{:?}", results);
+    Ok(())
+}
+```
+
+```typescript [TypeScript]
+import { DirSQL, Table } from 'dirsql';
+
+const db = new DirSQL({
+  root: './my-project',
+  tables: [
+    new Table({
+      ddl: 'CREATE TABLE items (name TEXT, value INTEGER)',
+      glob: 'data/*.json',
+      extract: (_path, content) => [JSON.parse(content)],
+    }),
+  ],
+});
+
+// Query
+const results = await db.query('SELECT * FROM items WHERE value > 10');
+console.log(results);
+```
+
+:::
+
+## Constructor
+
+```python
+DirSQL(root=None, *, tables=None, ignore=None, config=None)
+```
+
+The constructor immediately starts scanning in a background thread via `asyncio.ensure_future`. The constructor itself returns immediately without blocking.
+
+## `await db.ready()`
+
+Waits until the initial directory scan is complete. If the scan raised an exception (invalid DDL, unreadable files, etc.), `ready()` re-raises it.
+
+`ready()` can be called multiple times safely. After the first completion, subsequent calls return immediately.
+
+```python
+db = DirSQL("./data", tables=[...])
+
+# Do other setup work here while the scan runs in the background
+setup_logging()
+connect_websocket()
+
+# Now wait for the scan to finish before querying
+await db.ready()
+```
+
+## `await db.query(sql)`
+
+Runs a SQL query in a background thread. Returns a list of dicts keyed by column name.
+
+```python
+results = await db.query("SELECT COUNT(*) as n FROM items")
+```
+
+## `async for event in db.watch()`
+
+Returns an async iterable of `RowEvent` objects. The watcher is started automatically on the first iteration.
+
+::: code-group
+
+```python [Python]
+async for event in db.watch():
+    if event.action == "insert":
+        print(f"New row in {event.table}: {event.row}")
+    elif event.action == "update":
+        print(f"Updated row in {event.table}: {event.row}")
+    elif event.action == "delete":
+        print(f"Deleted row from {event.table}: {event.row}")
+    elif event.action == "error":
+        print(f"Error: {event.error}")
+```
+
+```rust [Rust]
+use futures::StreamExt;
+
+let mut stream = db.watch();
+while let Some(event) = stream.next().await {
+    match event.action {
+        Action::Insert => println!("New row in {}: {:?}", event.table, event.row),
+        Action::Update => println!("Updated row in {}: {:?}", event.table, event.row),
+        Action::Delete => println!("Deleted row from {}: {:?}", event.table, event.row),
+        Action::Error => eprintln!("Error: {:?}", event.error),
+    }
+}
+```
+
+```typescript [TypeScript]
+for await (const event of db.watch()) {
+  switch (event.action) {
+    case 'insert':
+      console.log(`New row in ${event.table}:`, event.row);
+      break;
+    case 'update':
+      console.log(`Updated row in ${event.table}:`, event.row);
+      break;
+    case 'delete':
+      console.log(`Deleted row from ${event.table}:`, event.row);
+      break;
+    case 'error':
+      console.error(`Error: ${event.error}`);
+      break;
+  }
+}
+```
+
+:::
+
+The async iterator polls for filesystem events with a 200ms timeout internally. It yields events as they arrive and never terminates on its own -- use `break` or cancellation to stop.
+
+## Combining with other async code
+
+The async API works alongside other concurrent code without blocking:
+
+::: code-group
+
+```python [Python]
+async def watch_and_serve(db):
+    async for event in db.watch():
+        await notify_clients(event)
+
+async def main():
+    db = DirSQL("./data", tables=[...])
+    await asyncio.gather(
+        watch_and_serve(db),
+        run_web_server(),
+    )
+```
+
+```rust [Rust]
+async fn watch_and_serve(db: &DirSQL) {
+    let mut stream = db.watch();
+    while let Some(event) = stream.next().await {
+        notify_clients(&event).await;
+    }
+}
+
+#[tokio::main]
+async fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let db = DirSQL::new("./data", vec![...])?;
+
+    tokio::join!(
+        watch_and_serve(&db),
+        run_web_server(),
+    );
+    Ok(())
+}
+```
+
+```typescript [TypeScript]
+async function watchAndServe(db: DirSQL) {
+  for await (const event of db.watch()) {
+    await notifyClients(event);
+  }
+}
+
+const db = new DirSQL({ root: './data', tables: [/* tables */] });
+
+await Promise.all([
+  watchAndServe(db),
+  runWebServer(),
+]);
+```
+
+:::