dirsql 0.3.70 → 0.3.71

package/docs/howto/search-by-meaning.md

@@ -0,0 +1,148 @@
+# Search documents by meaning
+
+Ask a question in plain language and get the closest documents back — even
+when they share no keywords with it. Three pieces you already have compose
+into semantic search: a SQLite vector extension for the distance math, an
+[`on-file`](../reference/hooks.md#on-file) command to embed each file at
+index time, and a [`pre-query`](../reference/hooks.md#pre-query) command to
+embed each question at query time.
+
+## How the pieces fit
+
+1. **[`[[dirsql.extension]]`](../reference/config.md#dirsql-extension)**
+   loads [`sqlite-vec`](https://github.com/asg017/sqlite-vec), making
+   `vec_distance_cosine()` callable in queries.
+2. **`on-file`** runs an embedding command once per matched file; the
+   vector lands in an `embedding` column next to the text.
+3. **`pre-query`** receives each raw `POST /query` body, embeds the
+   question, and prints the nearest-neighbor SQL to run.
+
+`dirsql` never sees a model — both hooks are commands you own, so any
+embedding model works. This guide uses
+[`model2vec`](https://github.com/MinishLab/model2vec), a small, fast,
+CPU-only embedding library (its `potion-base-8M` model downloads ~30 MB
+from Hugging Face on first use).
+
+## 1. The embedding scripts
+
+Suppose short notes live in `notes/*.md`:
+
+```
+notes/pasta.md      # boiling spaghetti, olive oil, garlic
+notes/branches.md   # git feature branches and pull requests
+notes/tomatoes.md   # planting tomato seedlings after the last frost
+```
+
+Next to them, `embed.py` turns one file into one row carrying its text and
+its embedding (a JSON array, stored as TEXT — `sqlite-vec` accepts JSON
+vectors directly):
+
+```python
+"""Embed one file's text; print a dirsql row array on stdout."""
+import json
+import sys
+
+from model2vec import StaticModel
+
+path = sys.argv[1]
+text = open(path, encoding="utf-8").read()
+model = StaticModel.from_pretrained("minishlab/potion-base-8M")
+vector = model.encode([text])[0]
+print(json.dumps([{"text": text, "embedding": json.dumps([round(float(x), 6) for x in vector])}]))
+```
+
+And `search.py` turns a `{"q": "..."}` request body into SQL, embedding the
+question with the same model:
+
+```python
+"""Turn a {"q": "..."} request body into a nearest-neighbor SQL query."""
+import json
+import sys
+
+from model2vec import StaticModel
+
+body = json.loads(sys.argv[1])
+model = StaticModel.from_pretrained("minishlab/potion-base-8M")
+vector = model.encode([body["q"]])[0]
+needle = json.dumps([round(float(x), 6) for x in vector])
+print(
+    "SELECT _path, ROUND(vec_distance_cosine(embedding, '%s'), 3) AS distance "
+    "FROM notes ORDER BY distance LIMIT 3" % needle
+)
+```
+
+::: warning The hook owns SQL safety
+Whatever SQL `pre-query` prints is executed as-is. Here the interpolated
+value is a numeric vector the script itself produced; never splice raw
+request text into SQL. See [`pre-query`](../reference/hooks.md#pre-query).
+:::
+
+## 2. Wire them up in `.dirsql.toml`
+
+```toml
+[dirsql]
+pre-query = "uv run --with model2vec python search.py {args}"
+hook-timeout = 300   # headroom for the first-run model download
+
+[[dirsql.extension]]
+path       = "sqlite_vec"          # Python module name; see note below
+entrypoint = "sqlite3_vec_init"
+
+[[table]]
+ddl     = "CREATE TABLE notes (_path TEXT, text TEXT, embedding TEXT)"
+glob    = "notes/*.md"
+on-file = "uv run --with model2vec python embed.py {path}"
+```
+
+The extension is named by package: the Python launcher resolves the
+installed `sqlite_vec` module to its bundled loadable. Naming rules per
+runtime — and the literal-path alternative that works everywhere — are in
+[Load a SQLite extension](./load-extension.md).
+
+## 3. Start the server and ask questions
+
+Launch with `sqlite-vec` available to the launcher's environment:
+
+```bash
+uvx --with sqlite-vec dirsql
+```
+
+The initial scan runs `embed.py` once per note. Then ask:
+
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"q": "how do I cook pasta?"}'
+```
+
+```json
+[{"_path":"notes/pasta.md","distance":0.315},{"_path":"notes/tomatoes.md","distance":0.881},{"_path":"notes/branches.md","distance":0.92}]
+```
+
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"q": "reviewing code on github"}'
+```
+
+```json
+[{"_path":"notes/branches.md","distance":0.51},{"_path":"notes/pasta.md","distance":1.033},{"_path":"notes/tomatoes.md","distance":1.074}]
+```
+
+Neither question shares a keyword with its top note — "cook" appears
+nowhere in `pasta.md`, "github" nowhere in `branches.md`. The distance
+ranking is doing the work.
+
+Because `pre-query` is set, the request body is *not* the usual
+`{"sql": …}` — the raw body goes to your script, which decides what SQL
+runs ([hook interactions](../reference/http-api.md#hook-interactions)).
+
+## Recomputing vs. caching
+
+Embeddings are recomputed on every startup, because the database is a
+derived, ephemeral view of the files
+([how `dirsql` thinks](../explanation.md)). Once the model is cached this
+is fast for small trees; for large ones, enable persistence so unchanged
+files keep their stored embeddings across restarts —
+[Keep the index across restarts](./persist.md). Editing a note re-embeds
+just that file, automatically.

package/docs/howto/skip-files.md

@@ -0,0 +1,60 @@
+# Skip files you don't want indexed
+
+Drafts, build output, and editor droppings don't belong in your tables.
+[`ignore`](../reference/config.md#dirsql-keys) globs exclude files entirely —
+from the initial scan and from watch events alike.
+
+## 1. Add `ignore` patterns
+
+Suppose finished notes live in `notes/`, but drafts and scratch files hide
+among them:
+
+```
+notes/final.md
+notes/drafts/wip.md
+notes/scratch.md.tmp
+```
+
+Exclude the noise in `.dirsql.toml`:
+
+```toml
+[dirsql]
+ignore = ["notes/drafts/**", "**/*.tmp"]
+
+[[table]]
+ddl  = "CREATE TABLE notes (_path TEXT)"
+glob = "notes/**/*"
+```
+
+Patterns match against root-relative paths, the same way table globs do. An
+ignored file never reaches any table — even one whose glob would match it.
+
+## 2. Confirm what made it in
+
+Start the server (`npx dirsql` / `uvx dirsql`) and check:
+
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT _path FROM notes ORDER BY _path"}'
+```
+
+```json
+[{"_path":"notes/final.md"}]
+```
+
+## Notes
+
+- `ignore` lives in a config file, so it needs one:
+  [zero-config mode](../reference/cli.md#zero-config-mode) indexes
+  everything with no ignores.
+- The top-level `.dirsql/` directory is always excluded, ignore list or
+  not — it is reserved for `dirsql`'s own metadata
+  ([config reference](../reference/config.md#dirsql-keys)).
+- Narrow table globs are the other half of the story: a file matching no
+  table's glob contributes no rows either. Use `ignore` for things that
+  should never be looked at; use precise globs to shape what each table
+  sees ([Define tables for your files](./define-tables.md)).
+- Embedding `dirsql` instead? The SDK constructor takes the same patterns
+  via its `ignore` parameter
+  ([SDK reference](../reference/sdk.md#constructor)).

package/docs/reference/cli.md

@@ -0,0 +1,132 @@
+# CLI
+
+The `dirsql` binary has two modes:
+
+| Invocation | Behavior |
+|---|---|
+| `dirsql` (no subcommand) | Start a long-lived HTTP server exposing a SQL view of a directory. See [HTTP API](./http-api.md). |
+| `dirsql init` | Generate a starter `.dirsql.toml` by running `claude` over a directory. |
+
+## Installation
+
+::: code-group
+
+```bash [npm]
+npx dirsql
+```
+
+```bash [PyPI]
+uvx dirsql
+```
+
+```bash [Cargo]
+# The `cli` feature is opt-in; this installs the binary only.
+cargo install dirsql --features cli
+dirsql
+```
+
+:::
+
+The npm launcher requires **Node ≥ 20.11**.
+
+## Server mode
+
+```bash
+dirsql
+# Running at localhost:7117
+```
+
+On startup the server prints `Running at <host>:<port>` to stdout. It runs
+until it receives `SIGINT` (Ctrl-C) or `SIGTERM`, then drains in-flight
+requests, closes open `/events` streams, and exits.
+
+### Flags
+
+| Flag | Default | Description |
+|---|---|---|
+| `--config <path>` | `./.dirsql.toml` | Path to the [config file](./config.md). The index is rooted at the directory containing this file (unless the config sets `[dirsql].root`). When the file does not exist, the server runs in [zero-config mode](#zero-config-mode). |
+| `--host <addr>` | `localhost` | Bind address. |
+| `--port <n>` | `7117` | TCP port to bind. |
+| `--extension <path>` | none | Load a SQLite extension by literal path, overriding the config's `[[dirsql.extension]]` entries. Repeatable. Format: `<path>` or `<path>::<entrypoint>`. Internal plumbing for the pip/npm launchers, which resolve package-name extensions and pass the resolved paths here — not intended for direct use. When any `--extension` is present, the config file's own extension entries are not loaded. |
+| `--version` | | Print the version and exit. |
+| `--help` | | Print usage and exit. |
+
+### Defaults
+
+- Per-query timeout: **30 seconds**. A query exceeding it returns
+  `408 Request Timeout`.
+- Command hooks (`on-file`, `pre-query`, `post-query`) default to a
+  **30-second** timeout each, overridable with the config key
+  [`[dirsql].hook-timeout`](./config.md#dirsql-keys).
+
+### Zero-config mode
+
+When the config file named by `--config` does not exist, the server indexes
+the directory that would have contained it (the current directory for the
+default `./.dirsql.toml`) with a single table named `files`:
+
+- Glob: `**/*` — every file under the root, at any depth, no ignores.
+- One row per file, with all seven
+  [virtual columns](./columns.md): `_path`, `_basename`, `_dir`, `_ext`,
+  `_size`, `_mtime`, `_ctime`.
+
+```bash
+curl -s localhost:7117/query -H 'content-type: application/json' \
+  -d '{"sql":"SELECT _basename, _size FROM files ORDER BY _size DESC LIMIT 5"}'
+```
+
+A config file, when present, fully overrules this default. A *missing*
+config is not an error; only a config that exists but fails to load is (see
+below).
+
+### Degraded mode
+
+When the config file exists but cannot be resolved or loaded (unreadable
+path, invalid TOML, schema errors), the server still starts and binds, but
+every request to `/query` and `/events` returns `503 Service Unavailable`
+with a JSON body describing the failure:
+
+```json
+{"error": "failed to load config: ..."}
+```
+
+### Exit codes
+
+| Code | Meaning |
+|---|---|
+| `0` | Clean shutdown after `SIGINT` / `SIGTERM`. |
+| `1` | Failed to bind `host:port`, or an error during shutdown. |
+
+## `dirsql init`
+
+Generates a `.dirsql.toml` by running the `claude` CLI over the target
+directory. The generated config contains only filesystem-fact tables
+(`[[table]]` entries whose columns come from [glob captures and virtual
+columns](./columns.md)) — never content-derived columns.
+
+```bash
+dirsql init
+```
+
+### Flags
+
+| Flag | Default | Description |
+|---|---|---|
+| `--root <path>` | current directory | Directory to scan. |
+| `--output <path>` | `<root>/.dirsql.toml` | Where to write the generated config. |
+| `--force` | off | Overwrite the output file if it already exists. |
+
+### Requirements and failure modes
+
+`init` requires `claude` on `PATH`, signed in; there is no separate API key.
+All failures exit `1` with a message on stderr:
+
+| Condition | Behavior |
+|---|---|
+| Output file exists and `--force` not passed | Fails before invoking `claude` (no LLM call is made). |
+| `claude` not found on `PATH` | Fails with a pointer to the Claude Code install docs. |
+| `claude` exits non-zero | Fails with `claude`'s stderr; no partial config is written. |
+| `claude` produces non-UTF-8 output | Fails; nothing is written. |
+
+On success, `claude`'s stdout is written verbatim to the output path and
+`init` exits `0`.

package/docs/reference/columns.md

@@ -0,0 +1,68 @@
+# Virtual columns and glob captures
+
+Every table — config-defined or programmatic — gets filesystem facts merged
+onto its rows automatically: seven reserved **virtual columns** derived from
+the file's path and stat metadata, plus one column per **`{name}` capture**
+in the table's glob.
+
+Facts are **opt-in by DDL**: only facts whose name appears as a column in
+the table's `CREATE TABLE` are populated; the rest are silently dropped.
+Declaring them requires nothing else.
+
+## Virtual columns
+
+| Column | Type | Value |
+|---|---|---|
+| `_path` | TEXT | The file's path relative to the scan root (e.g. `posts/hello.md`). |
+| `_basename` | TEXT | The filename, including extension (`hello.md`). |
+| `_dir` | TEXT | The parent directory relative to the root (`posts`); the empty string for files directly under the root. |
+| `_ext` | TEXT | The file extension without the leading dot (`md`). Original case is preserved — `Photo.JPG` yields `JPG`; use `LOWER(_ext)` for case-insensitive matching. `NULL` when the file has no extension. |
+| `_size` | INTEGER | File size in bytes. |
+| `_mtime` | INTEGER | Last-modified time, Unix seconds. |
+| `_ctime` | INTEGER | Creation (birth) time, Unix seconds. `NULL` when the platform or filesystem cannot supply it. |
+
+A fact that cannot be computed (an unreadable file's `_size`/`_mtime`/
+`_ctime`, a missing extension's `_ext`) is absent from the row: `NULL` in
+the default relaxed mode, a missing-column error for a
+[`strict`](./config.md#table) table that declares it.
+
+```sql
+SELECT _basename, _size
+FROM posts
+WHERE _mtime > strftime('%s', '2024-01-01')
+ORDER BY _mtime DESC;
+```
+
+## Glob captures
+
+A `{name}` segment in a table's glob captures part of each matched path as
+a TEXT column named `name`:
+
+```toml
+[[table]]
+ddl  = "CREATE TABLE comments (thread_id TEXT, _basename TEXT, _mtime INTEGER)"
+glob = "_comments/{thread_id}/*.jsonl"
+```
+
+A file at `_comments/abc123/2024-05-05.jsonl` produces a row with
+`thread_id = "abc123"`.
+
+- A capture name must be a valid identifier: a letter or underscore
+  followed by letters, digits, or underscores (`[a-zA-Z_][a-zA-Z0-9_]*`).
+- A capture matches **within a single path segment** — one or more
+  characters, never a `/`. For matching purposes, `{name}` behaves like
+  `*`.
+- A glob may contain multiple captures (`{year}/{month}/*.jpg`).
+- Like virtual columns, a capture populates a column only when the DDL
+  declares a column of the same name.
+
+## Precedence
+
+Values produced by a table's own row source — an `on-file` command's JSON
+output or an SDK `extract` callback's return value — **win** over
+auto-injected facts of the same name. An extract that explicitly emits
+`_path` is honored.
+
+Injection order per row: virtual columns first, then glob captures, then
+the row source's own values, each layer overwriting the previous, all
+filtered to the DDL's declared columns.

package/docs/reference/config.md

@@ -0,0 +1,166 @@
+# Configuration file (`.dirsql.toml`)
+
+`.dirsql.toml` is a TOML file with one optional `[dirsql]` section, zero or
+more `[[dirsql.extension]]` entries, and zero or more `[[table]]` entries.
+An empty file is valid. A missing `[dirsql]` section behaves as an
+all-defaults one. Unknown keys are ignored.
+
+The [CLI](./cli.md) loads `./.dirsql.toml` by default (`--config <path>`
+overrides). The [SDKs](./sdk.md) load a config via the `config` constructor
+parameter.
+
+**Path resolution.** Relative paths in the config (`root`, `persist_path`,
+`[[dirsql.extension]]` `path`) resolve against the config file's parent
+directory. The scan root defaults to that same directory when `root` is not
+set.
+
+## `[dirsql]` keys
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `root` | string | config file's parent directory | Directory to index. Relative values resolve against the config file's parent. An explicit `root` passed to an SDK constructor overrides this (a warning is emitted on stderr). |
+| `ignore` | array of strings | `[]` | Glob patterns matched against root-relative paths. Matched files are skipped entirely — excluded from the initial scan and from watch events. |
+| `persist` | boolean | `false` | Keep the SQLite index on disk between runs. When `false`, the index is ephemeral: rebuilt from your files on every startup and discarded on exit. |
+| `persist_path` | string | `<root>/.dirsql/cache.db` | Location of the on-disk cache. Relative values resolve against the config file's parent. Ignored unless `persist = true`. |
+| `pre-query` | string | none | Server-wide command hook: the raw `POST /query` request body is passed to this command as `{args}`, and the plain-text SQL it prints is executed instead of parsing the body as `{"sql": …}`. CLI server only; the SDKs ignore it. Must be non-empty. See [Command hooks](./hooks.md#pre-query). |
+| `post-query` | string | none | Server-wide command hook: each successful `POST /query` result set is handed to this command (as a JSON array on stdin, and as `{args}` up to 96 KiB), and the JSON body it prints is returned instead of the bare row array. CLI server only; the SDKs ignore it. Must be non-empty. See [Command hooks](./hooks.md#post-query). |
+| `hook-timeout` | integer (seconds) | `30` | One global per-run timeout for every command hook — `on-file`, `pre-query`, and `post-query` alike. Positive whole seconds; zero and negative values are a config error. See [Command hooks](./hooks.md#timeout). |
+
+The top-level `.dirsql/` directory under the root is always excluded from
+scanning, whether or not it appears in `ignore` — it is reserved for
+`dirsql`'s own metadata (the persist cache lives there by default). Only the
+top-level `.dirsql/` is reserved; a nested `sub/.dirsql/` is an ordinary
+directory.
+
+```toml
+[dirsql]
+root = "../data"
+ignore = ["node_modules/**", ".git/**"]
+persist = true
+persist_path = ".dirsql/cache.db"   # the default; shown for illustration
+hook-timeout = 300
+```
+
+## `[[dirsql.extension]]`
+
+Each entry declares a SQLite extension to load at startup. Extensions are
+loaded onto the connection before any `CREATE TABLE` runs; loading is
+enabled only for the duration of each load and disabled again afterwards, so
+the SQL `load_extension()` function is never exposed to queries.
+
+| Key | Required | Description |
+|---|---|---|
+| `path` | yes (non-empty) | The extension's shared library. Either a **file path** (`.so` / `.dylib` / `.dll`; relative paths resolve against the config file's parent directory) or a bare **package name** (no path separator, no loadable-file suffix). The package name is runtime-specific — see [package-name resolution](#package-name-resolution) below. |
+| `entrypoint` | no | Init-symbol override. When omitted, SQLite derives the entry point from the filename (`sqlite3_<filename>_init`); set this when that default does not match (e.g. `sqlite-vec`'s entry point is `sqlite3_vec_init`). |
+
+```toml
+[[dirsql.extension]]
+path       = "./ext/vec0.dylib"
+entrypoint = "sqlite3_vec_init"
+```
+
+### Package-name resolution
+
+A `path` naming a package is resolved from the
+installed package in the runtime environment: the Python launcher and SDK
+use `importlib`, the Node launcher and SDK use `require.resolve` against
+`node_modules`. Resolution is file-first (a same-named local file wins) and
+errors when the package contains zero or multiple loadables for the current
+platform. The **standalone Rust binary and the Rust SDK are
+file-path-only** — they have no interpreter to resolve package names with.
+
+The name must be what the runtime's resolver knows, which is not always the
+name you installed:
+
+- **Python** resolves the **importable module name** — underscores, not the
+  pip distribution name. `pip install sqlite-vec` is loaded as
+  `path = "sqlite_vec"`; `path = "sqlite-vec"` does not resolve.
+- **Node** resolves the package whose install actually **contains the
+  loadable**. Meta-packages that split binaries per platform resolve via the
+  platform package — e.g. `npm install sqlite-vec` is loaded as
+  `path = "sqlite-vec-linux-x64"` (matching your platform), not
+  `path = "sqlite-vec"`, whose meta-package ships no loadable.
+
+Extensions add **functions** callable in queries and in a table's DDL. An
+extension-backed **virtual table** cannot be declared as a `[[table]]` —
+`dirsql` tables are per-file row tables, so a `CREATE VIRTUAL TABLE` DDL is
+rejected; call the extension's functions in queries instead.
+
+## `[[table]]`
+
+Each entry maps a glob pattern to a SQL table. Every matched file produces
+rows whose columns come from filesystem facts — [glob captures and virtual
+columns](./columns.md) — plus, when `on-file` is set, the output of a
+per-file command.
+
+| Key | Required | Description |
+|---|---|---|
+| `ddl` | yes | A SQLite `CREATE TABLE` statement. The table name is parsed from it. Only columns declared here are populated; auto-injected facts not in the DDL are dropped. |
+| `glob` | yes | Glob pattern matched against root-relative paths. May contain `{name}` [capture segments](./columns.md#glob-captures). First matching table wins when a file matches several globs. |
+| `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied (by the command/extract output, a glob capture, or a virtual column). When `false`, extra keys are dropped and missing columns become `NULL`. |
+| `on-file` | no | A command run once per matched file; its stdout (a JSON array of row objects) becomes the file's rows. Must be non-empty. See [Command hooks](./hooks.md#on-file). |
+
+Without `on-file`, a table produces exactly one row per matched file, built
+entirely from filesystem facts. Content interpretation (frontmatter, JSON
+fields, CSV parsing) is out of scope for plain config tables — use
+`on-file`, or a programmatic [SDK table](./sdk.md#table) with an `extract`
+callback.
+
+```toml
+[[table]]
+ddl  = "CREATE TABLE comments (thread_id TEXT, _basename TEXT, _mtime INTEGER)"
+glob = "_comments/{thread_id}/*.jsonl"
+
+[[table]]
+ddl     = "CREATE TABLE papers (paper_id TEXT, title TEXT)"
+glob    = "**/meta.json"
+on-file = "uv run python extract_papers.py {path}"
+strict  = true
+```
+
+### `on-file` row mapping
+
+The command prints a JSON array of objects; each object becomes one row.
+JSON values map to SQLite as: `null` → `NULL`; `true`/`false` → `1`/`0`; an
+integral number → `INTEGER`, any other number → `REAL`; a string → `TEXT`; a
+nested array or object → its JSON text as `TEXT`.
+
+Filesystem facts are still merged onto every `on-file` row; a column emitted
+by the command wins over a same-named fact. Output that is not a JSON array
+of objects is a per-file failure: the file is skipped with a stderr warning
+and the scan continues (see [failure semantics](./hooks.md#failure-semantics)).
+
+## Parse errors
+
+Loading fails (the CLI enters [degraded mode](./cli.md#degraded-mode); the
+SDKs raise/reject) when:
+
+- The TOML is malformed.
+- A `[[table]]` entry omits `ddl` or `glob`.
+- A `[[dirsql.extension]]` entry omits `path`, or `path` is empty.
+- `on-file`, `pre-query`, or `post-query` is present but empty/whitespace.
+- `hook-timeout` is zero or negative.
+
+## Full example
+
+```toml
+[dirsql]
+ignore = ["node_modules/**", ".git/**", "dist/**"]
+persist = true
+pre-query = "uv run python to_sql.py {args}"
+post-query = "jq -c '{results: .}'"
+hook-timeout = 120
+
+[[dirsql.extension]]
+path       = "sqlite_vec"            # Python module name; on Node use the
+                                     # platform package, e.g. sqlite-vec-linux-x64
+entrypoint = "sqlite3_vec_init"
+
+[[table]]
+ddl  = "CREATE TABLE comments (thread_id TEXT, _basename TEXT, _mtime INTEGER)"
+glob = "_comments/{thread_id}/*.jsonl"
+
+[[table]]
+ddl  = "CREATE TABLE documents (_path TEXT, _basename TEXT, _size INTEGER)"
+glob = "**/index.md"
+```