dirsql 0.3.69 → 0.3.71

package/README.md

@@ -89,7 +89,7 @@ Each event has `.action` (`'insert'` | `'update'` | `'delete'` | `'error'`), `.t
 
 ## CLI
 
-`npx dirsql` runs an HTTP server exposing the SDK over HTTP: `POST /query` for SQL and `GET /events` for a Server-Sent Events change stream. Requires **Node >= 20.11**. See the [CLI guide](https://thekevinscott.github.io/dirsql/cli/).
+`npx dirsql` runs an HTTP server exposing the SDK over HTTP: `POST /query` for SQL and `GET /events` for a Server-Sent Events change stream. Requires **Node >= 20.11**. See the [CLI reference](https://thekevinscott.github.io/dirsql/reference/cli).
 
 ## License
 

package/docs/explanation.md

@@ -0,0 +1,5 @@
+---
+title: How dirsql thinks
+---
+
+<!--@include: ../ARCHITECTURE.md-->

package/docs/howto/columns-from-paths.md

@@ -0,0 +1,63 @@
+# Derive columns from file paths
+
+Directory layouts often encode real data — an author, a year, a thread ID —
+as path segments. A `{name}` capture in a table's glob turns such a segment
+into a queryable column, no extraction code required.
+
+## 1. Name the segment in the glob
+
+Suppose photos are filed by year and month:
+
+```
+photos/2024/05/beach.jpg
+photos/2024/11/hike.jpg
+photos/2025/01/snow.jpg
+```
+
+Capture both directory levels in `.dirsql.toml`:
+
+```toml
+[[table]]
+ddl  = "CREATE TABLE photos (year TEXT, month TEXT, _basename TEXT)"
+glob = "photos/{year}/{month}/*.jpg"
+```
+
+A capture only populates a column when the DDL declares one with the same
+name — here `year` and `month`. The capture rules (valid names, matching
+within one path segment) are in
+[glob captures](../reference/columns.md#glob-captures).
+
+## 2. Query the captured columns
+
+Start the server (`npx dirsql` / `uvx dirsql`) and query:
+
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT year, month, _basename FROM photos ORDER BY year, month"}'
+```
+
+```json
+[{"_basename":"beach.jpg","month":"05","year":"2024"},{"_basename":"hike.jpg","month":"11","year":"2024"},{"_basename":"snow.jpg","month":"01","year":"2025"}]
+```
+
+Captures are real SQL columns, so aggregation works:
+
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT year, COUNT(*) AS photos FROM photos GROUP BY year"}'
+```
+
+```json
+[{"photos":2,"year":"2024"},{"photos":1,"year":"2025"}]
+```
+
+## Going further
+
+- Captures combine freely with [virtual columns](../reference/columns.md#virtual-columns)
+  (`_basename` above) — both are filesystem facts merged onto every row.
+- The [tutorial](../getting-started.md) walks the same idea with an
+  `{author}` capture, starting from zero.
+- When the value you need lives inside the file rather than in its path,
+  see [Extract rows from file contents](./extract-from-contents.md).

package/docs/howto/define-tables.md

@@ -0,0 +1,70 @@
+# Define tables for your files
+
+Map a glob of files to a named SQL table so you query exactly the files you
+care about, with exactly the columns you care about — instead of the
+catch-all `files` table that [zero-config mode](../reference/cli.md#zero-config-mode)
+serves.
+
+## 1. Create a config next to your files
+
+Suppose your blog posts live under `posts/`, one markdown file each. In the
+directory you want to index, create a `.dirsql.toml` with one
+[`[[table]]`](../reference/config.md#table) entry:
+
+```toml
+[[table]]
+ddl  = "CREATE TABLE posts (_path TEXT, _size INTEGER, _mtime INTEGER)"
+glob = "posts/**/*.md"
+```
+
+- `glob` selects the files: every `.md` under `posts/`, at any depth,
+  relative to the directory containing the config.
+- `ddl` is a plain SQLite `CREATE TABLE` naming the columns you want. Here
+  all three are [virtual columns](../reference/columns.md#virtual-columns) —
+  filesystem facts `dirsql` computes for every file. Facts are opt-in by
+  DDL: only the ones you declare become columns.
+
+## 2. Start the server and query
+
+::: code-group
+
+```bash [npm]
+npx dirsql
+```
+
+```bash [PyPI]
+uvx dirsql
+```
+
+:::
+
+Each matched file is one row:
+
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT _path, _size FROM posts ORDER BY _path"}'
+```
+
+```json
+[{"_path":"posts/2024/hello.md","_size":21},{"_path":"posts/2025/again.md","_size":55}]
+```
+
+Files that don't match the glob (a `README.txt` next to `posts/`, say) are
+simply not in the table. Once a config file exists, it fully replaces the
+zero-config default — only the tables you define are served.
+
+## Multiple tables
+
+Add one `[[table]]` entry per table. When a file matches several globs, the
+first matching table wins — see [`[[table]]`](../reference/config.md#table)
+for that and the remaining keys (`strict`, `on-file`).
+
+## Going further
+
+- Your directory layout encodes data (authors, dates, IDs)? Capture path
+  segments as columns — [Derive columns from file paths](./columns-from-paths.md).
+- Need columns from *inside* the files? A plain table never reads file
+  contents — [Extract rows from file contents](./extract-from-contents.md).
+- Why one row per file, rebuilt from disk? See
+  [how `dirsql` thinks](../explanation.md).

package/docs/howto/embed.md

@@ -0,0 +1,196 @@
+# Embed `dirsql` in your application
+
+Run the same engine in-process instead of as a server: construct a `DirSQL`
+over a directory, hand it your tables, and query it like any library — no
+HTTP, no separate process. The SDKs for Python, TypeScript, and Rust are
+thin bindings over one shared core, so behavior matches the CLI exactly.
+
+## 1. Install the SDK
+
+::: code-group
+
+```bash [Python]
+uv add dirsql
+```
+
+```bash [TypeScript]
+pnpm add dirsql
+```
+
+```bash [Rust]
+cargo add dirsql
+```
+
+:::
+
+## 2. Index a directory and query it
+
+Suppose comments live as JSON files filed by thread:
+
+```
+comments/t1/c1.json    # {"body": "first!", "author": "alice"}
+comments/t1/c2.json    # {"body": "nice post", "author": "bob"}
+comments/t2/c1.json    # {"body": "following up", "author": "alice"}
+```
+
+Unlike a config-file table, a programmatic table takes an `extract`
+callback — your code reads each matched file and returns its rows, with
+[glob captures and virtual columns](../reference/columns.md) merged on
+automatically (here, `{thread}` from the path):
+
+::: code-group
+
+```python [Python]
+import asyncio
+import json
+
+from dirsql import DirSQL, Table
+
+
+def extract(path: str) -> list[dict]:
+    with open(path, encoding="utf-8") as f:
+        return [json.load(f)]
+
+
+async def main() -> None:
+    db = DirSQL(
+        "./comments-root",
+        tables=[
+            Table(
+                ddl="CREATE TABLE comments (thread TEXT, author TEXT, body TEXT)",
+                glob="comments/{thread}/*.json",
+                extract=extract,
+            )
+        ],
+    )
+    rows = await db.query("SELECT thread, author, body FROM comments ORDER BY thread, author")
+    for row in rows:
+        print(f"[{row['thread']}] {row['author']}: {row['body']}")
+
+
+asyncio.run(main())
+```
+
+```typescript [TypeScript]
+import { readFileSync } from "node:fs";
+import { DirSQL } from "dirsql";
+
+const db = new DirSQL({
+  root: "./comments-root",
+  tables: [
+    {
+      ddl: "CREATE TABLE comments (thread TEXT, author TEXT, body TEXT)",
+      glob: "comments/{thread}/*.json",
+      extract: (path) => [JSON.parse(readFileSync(path, "utf8"))],
+    },
+  ],
+});
+
+const rows = await db.query(
+  "SELECT thread, author, body FROM comments ORDER BY thread, author",
+);
+for (const row of rows) {
+  console.log(`[${row.thread}] ${row.author}: ${row.body}`);
+}
+```
+
+```rust [Rust]
+use std::collections::HashMap;
+
+use dirsql::{DirSQL, Table, Value};
+
+fn extract(path: &str) -> Vec<HashMap<String, Value>> {
+    let raw = std::fs::read_to_string(path).unwrap_or_default();
+    let Ok(serde_json::Value::Object(obj)) = serde_json::from_str(&raw) else {
+        return vec![];
+    };
+    let mut row = HashMap::new();
+    for (key, value) in obj {
+        if let serde_json::Value::String(s) = value {
+            row.insert(key, Value::Text(s));
+        }
+    }
+    vec![row]
+}
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let db = DirSQL::builder()
+        .root("./comments-root")
+        .table(Table::new(
+            "CREATE TABLE comments (thread TEXT, author TEXT, body TEXT)",
+            "comments/{thread}/*.json",
+            extract,
+        ))
+        .build()?;
+
+    let rows = db.query("SELECT thread, author, body FROM comments ORDER BY thread, author")?;
+    for row in rows {
+        if let (Value::Text(thread), Value::Text(author), Value::Text(body)) =
+            (&row["thread"], &row["author"], &row["body"])
+        {
+            println!("[{thread}] {author}: {body}");
+        }
+    }
+    Ok(())
+}
+```
+
+:::
+
+Rows come back keyed by column name (a `dict` / object /
+`HashMap<String, Value>` per row); all three programs print:
+
+```
+[t1] alice: first!
+[t1] bob: nice post
+[t2] alice: following up
+```
+
+A few things happened implicitly. In Python and TypeScript the constructor
+returned immediately and the scan ran in the background — `query` awaited
+readiness for you. In Rust, `.build()` scanned synchronously (there is a
+[`build_async()`](../reference/sdk.md#asyncdirsql-rust) for the
+non-blocking equivalent). Queries are read-only in every SDK. The complete
+contract — constructor parameters, `ready`, value type mappings, error
+shapes — is the [SDK reference](../reference/sdk.md).
+
+## 3. React to changes in-process
+
+`watch()` yields the same row events the server streams over
+[`/events`](./react-to-changes.md), as your language's native async
+iteration:
+
+::: code-group
+
+```python [Python]
+async for event in db.watch():
+    print(event.action, event.table, event.row)
+```
+
+```typescript [TypeScript]
+for await (const event of db.watch()) {
+  console.log(event.action, event.table, event.row);
+}
+```
+
+```rust [Rust]
+use futures::StreamExt;
+
+let mut stream = db.watch()?;
+while let Some(event) = stream.next().await {
+    println!("{event:?}");
+}
+```
+
+:::
+
+Event fields and actions are under
+[`RowEvent`](../reference/sdk.md#rowevent).
+
+## Reusing your CLI setup
+
+Everything the config file expresses is available programmatically —
+`ignore` patterns, `persist`, extensions — and the constructor's `config`
+parameter loads a `.dirsql.toml` directly, so an application can share the
+exact table definitions the CLI serves
+([constructor reference](../reference/sdk.md#constructor)).

package/docs/howto/extract-from-contents.md

@@ -0,0 +1,90 @@
+# Extract rows from file contents
+
+Paths and stat metadata only get you so far — when the columns you want live
+*inside* the files (JSON fields, frontmatter, log lines), add an
+[`on-file`](../reference/config.md#table) command: it runs once per matched
+file and its stdout becomes the file's rows.
+
+## 1. Point a command at the files
+
+Suppose each book is a JSON file:
+
+```json
+{"title": "Middlemarch", "author": "George Eliot", "year": 1871}
+```
+
+Any program that reads a file and prints a **JSON array of row objects** on
+stdout works. With [`jq`](https://jqlang.org/):
+
+```toml
+[[table]]
+ddl     = "CREATE TABLE books (title TEXT, author TEXT, year INTEGER, _path TEXT)"
+glob    = "books/*.json"
+on-file = "jq -c '[{title, author, year}]' {path}"
+```
+
+`{path}` is the matched file, relative to the index root — one of the
+placeholders defined by the
+[command hook contract](../reference/hooks.md#on-file), which also covers
+the argv splitting, working directory, stdout protocol, and timeout shared
+by every hook.
+
+## 2. Query the extracted columns
+
+Start the server (`npx dirsql` / `uvx dirsql`) and query:
+
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT title, author, year, _path FROM books ORDER BY year"}'
+```
+
+```json
+[{"_path":"books/bleak-house.json","author":"Charles Dickens","title":"Bleak House","year":1852},{"_path":"books/middlemarch.json","author":"George Eliot","title":"Middlemarch","year":1871}]
+```
+
+Filesystem facts are still merged onto every row — `_path` above comes from
+`dirsql`, not from `jq`. When the command emits a key that collides with a
+fact, the command wins
+([precedence](../reference/columns.md#precedence)).
+
+## Multiple rows per file
+
+Each object in the printed array is one row. To turn a JSONL file into one
+row per line, slurp it:
+
+```toml
+[[table]]
+ddl     = "CREATE TABLE events (event TEXT, user TEXT, _path TEXT)"
+glob    = "logs/*.jsonl"
+on-file = "jq -c -s '.' {path}"
+```
+
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT event, user FROM events"}'
+```
+
+```json
+[{"event":"login","user":"alice"},{"event":"logout","user":"alice"},{"event":"login","user":"bob"}]
+```
+
+## When a file fails
+
+A file whose command errors (or prints something that isn't a JSON array of
+objects) contributes no rows: `dirsql` warns on stderr and the scan
+continues — one bad file never takes down the index. Details in
+[failure semantics](../reference/hooks.md#failure-semantics); the JSON to
+SQLite value mapping is under
+[`on-file` row mapping](../reference/config.md#on-file-row-mapping).
+
+## Going further
+
+- The command re-runs on every startup and on every change to a matched
+  file. If it is expensive, [keep the index across restarts](./persist.md).
+- The flagship use of `on-file` — computing embeddings — is
+  [Search documents by meaning](./search-by-meaning.md).
+- Embedding `dirsql` in a program instead? The SDK's `extract` callback
+  fills the same role in-process — see
+  [Embed `dirsql` in your application](./embed.md).

package/docs/howto/load-extension.md

@@ -0,0 +1,91 @@
+# Load a SQLite extension
+
+SQLite extensions add functions to your query surface — vector distances,
+crypto digests, statistics. `dirsql` loads them at startup from a
+[`[[dirsql.extension]]`](../reference/config.md#dirsql-extension) entry,
+named either by file path or by installed package.
+
+## Option A: a literal file path
+
+Works everywhere — the `uvx`/`npx` launchers, the standalone
+`cargo install` binary, and every SDK. Point `path` at the extension's
+shared library (here [`sqlite-vec`](https://github.com/asg017/sqlite-vec)'s
+`vec0.so`, copied into the project):
+
+```toml
+[[dirsql.extension]]
+path       = "./ext/vec0.so"
+entrypoint = "sqlite3_vec_init"
+```
+
+Relative paths resolve against the config file's directory. `entrypoint`
+overrides the init symbol when it doesn't match the filename-derived
+default — `sqlite-vec` is exactly such a case
+([reference](../reference/config.md#dirsql-extension)).
+
+Start the server (`npx dirsql` / `uvx dirsql`) and the extension's
+functions are callable:
+
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT vec_version() AS vec_version"}'
+```
+
+```json
+[{"vec_version":"v0.1.9"}]
+```
+
+## Option B: a package name
+
+When the extension ships as a pip or npm package, name the package and let
+the launcher find the loadable inside it — no path to copy or keep in sync.
+How a bare name resolves (file-first probing, one-loadable rule) is defined
+under [`[[dirsql.extension]]`](../reference/config.md#dirsql-extension);
+what to actually write differs per runtime:
+
+**Python (`uvx dirsql`, Python SDK).** Use the *importable module name* —
+for the `sqlite-vec` pip package that is `sqlite_vec` (underscore), which
+is looked up via `importlib`:
+
+```toml
+[[dirsql.extension]]
+path       = "sqlite_vec"
+entrypoint = "sqlite3_vec_init"
+```
+
+```bash
+uvx --with sqlite-vec dirsql
+```
+
+**Node (`npx dirsql`, TypeScript SDK).** Use the *npm package name* — but
+note the named package must itself contain the loadable file. `sqlite-vec`
+on npm is a meta-package whose loadable lives in a platform-specific
+sub-package, so name that one (and install it) instead:
+
+```toml
+[[dirsql.extension]]
+path       = "sqlite-vec-linux-x64"   # or -darwin-arm64, etc.
+entrypoint = "sqlite3_vec_init"
+```
+
+If a platform-specific config is awkward, fall back to a literal path
+(Option A).
+
+**Rust (standalone binary, Rust SDK).** File paths only — there is no
+interpreter to resolve package names with
+([reference](../reference/config.md#dirsql-extension)).
+
+## Notes
+
+- Extensions add **functions**. An extension-backed virtual table cannot be
+  declared as a `[[table]]` — `dirsql` tables are per-file row tables
+  ([reference](../reference/config.md#dirsql-extension)).
+- Loading happens before any table DDL runs, and the SQL
+  `load_extension()` function is never exposed to queries
+  ([reference](../reference/config.md#dirsql-extension)).
+- Embedding `dirsql` in a program? The SDK constructor takes the same
+  specs via its `extensions` parameter
+  ([SDK reference](../reference/sdk.md#constructor)).
+- The payoff use case — `sqlite-vec` powering semantic search — is
+  [Search documents by meaning](./search-by-meaning.md).

package/docs/howto/persist.md

@@ -0,0 +1,52 @@
+# Keep the index across restarts
+
+By default the database is ephemeral: rebuilt from your files on every
+startup and discarded on exit. [`persist`](../reference/config.md#dirsql-keys) keeps the
+SQLite index on disk instead, so a restart only re-parses files that
+actually changed — the difference between seconds and milliseconds on large
+trees, and between re-running and skipping expensive
+[`on-file`](./extract-from-contents.md) commands.
+
+## 1. Turn it on
+
+```toml
+[dirsql]
+persist = true
+```
+
+That's the whole change. On the next run the cache is written to
+`.dirsql/cache.db` under the root; runs after that start from it. To put
+the cache elsewhere (a CI cache dir, a tmpfs), set
+[`persist_path`](../reference/config.md#dirsql-keys).
+
+## 2. Keep the cache out of git
+
+The cache is derived data — reproducible from the tree and frequently
+large. Add it to `.gitignore`:
+
+```
+.dirsql/
+```
+
+The top-level `.dirsql/` directory is reserved for `dirsql`'s metadata and
+is never scanned as data, so the cache can't index itself
+([config reference](../reference/config.md#dirsql-keys)).
+
+## What survives, what rebuilds
+
+On startup `dirsql` validates the cache rather than trusting it blindly:
+files whose stat metadata is unchanged keep their rows without being
+re-read; changed, added, and deleted files are reconciled. When the cache
+can't be trusted at all — the table/ignore configuration changed, or the
+`dirsql` version did — it is discarded and rebuilt from scratch
+automatically. You never need to delete it by hand; a full rebuild costs
+exactly what a non-persistent startup does.
+
+Persistence is a startup-time optimization, not a change in meaning: the
+database remains a derived view of your files, and queries return the same
+rows either way ([how `dirsql` thinks](../explanation.md)).
+
+## Embedding `dirsql`?
+
+The SDK constructors expose the same switch as `persist` / `persistPath`
+parameters — see the [SDK reference](../reference/sdk.md#constructor).

package/docs/howto/react-to-changes.md

@@ -0,0 +1,67 @@
+# React to file changes
+
+`dirsql` watches the directory it indexes, so your tables update as files
+change — and [`GET /events`](../reference/http-api.md#get-events) pushes
+every row-level change to you as it happens. No polling, no diffing on your
+side.
+
+## 1. Open the stream
+
+With the server running (`npx dirsql` / `uvx dirsql`), subscribe from
+another terminal:
+
+```bash
+curl -N http://localhost:7117/events
+```
+
+The server immediately confirms the subscription is attached:
+
+```
+event: ready
+data: {}
+```
+
+## 2. Change a file, receive an event
+
+Create a file under the indexed directory:
+
+```bash
+echo 'second' > inbox/two.txt
+```
+
+The stream delivers the resulting row change:
+
+```
+event: row
+data: {"action":"insert","file_path":"inbox/two.txt","old_row":null,"row":{"_basename":"two.txt","_ctime":1783170226,"_dir":"inbox","_ext":"txt","_mtime":1783170226,"_path":"inbox/two.txt","_size":7},"table":"files"}
+```
+
+Edits arrive as `update` events carrying both the old and new row;
+deletions as `delete`. The full payload schema — including `error` events —
+is in [event payloads](../reference/http-api.md#event-payloads).
+
+## Consuming it
+
+Anything that speaks
+[Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events)
+works — `EventSource` in a browser, an SSE client library, or plain
+`curl -N` in a shell pipeline. Two semantics worth designing around
+([stream semantics](../reference/http-api.md#stream-semantics)):
+
+- **Errors don't end the stream.** A malformed file produces an `error`
+  event and the stream continues — handle the event, don't reconnect.
+- **Slow consumers skip.** A subscriber that falls behind misses the
+  overflowed events rather than stalling the server. If you must not miss
+  anything, re-query after catching up.
+
+## Notes
+
+- Events reflect *row* changes, not raw filesystem events: a file edit
+  that leaves a table's rows identical emits nothing, and one file change
+  can emit several events. The diffing model is part of
+  [how `dirsql` thinks](../explanation.md).
+- Ignored files ([Skip files you don't want indexed](./skip-files.md))
+  never generate events.
+- Embedding `dirsql` in a program? The SDK's `watch()` yields the same
+  events in-process — see the [SDK reference](../reference/sdk.md#watch)
+  and [Embed `dirsql` in your application](./embed.md).