dirsql 0.4.63__tar.gz → 0.4.66__tar.gz

{dirsql-0.4.63 → dirsql-0.4.66}/Cargo.lock

@@ -552,7 +552,7 @@ checksum = "6184e33543162437515c2e2b48714794e37845ec9851711914eec9d308f6ebe8"
 
 [[package]]
 name = "dirsql"
-version = "0.4.63"
+version = "0.4.66"
 dependencies = [
  "assert_cmd",
  "axum",
@@ -600,7 +600,7 @@ dependencies = [
 
 [[package]]
 name = "dirsql-py-ext"
-version = "0.4.63"
+version = "0.4.66"
 dependencies = [
  "dirsql",
  "pyo3",

{dirsql-0.4.63 → dirsql-0.4.66}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dirsql
-Version: 0.4.63
+Version: 0.4.66
 Requires-Dist: bin-shim>=0.1
 Summary: Ephemeral SQL index over a local directory
 Keywords: sql,filesystem,directory,sqlite,index
@@ -13,7 +13,7 @@ Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
 
 Ephemeral SQL index over a local directory. `dirsql` watches a filesystem, ingests structured files into an in-memory SQLite database, and exposes a SQL query interface -- the filesystem is always the source of truth.
 
-[Documentation](https://thekevinscott.github.io/dirsql/?lang=python)
+[Documentation](https://dirsql.dev/?lang=python)
 
 Also available as [`dirsql` on crates.io](https://crates.io/crates/dirsql) and [`dirsql` on npm](https://www.npmjs.com/package/dirsql).
 
@@ -135,7 +135,7 @@ Each event has `.action` (`"insert"`, `"update"`, `"delete"`, or `"error"`), `.t
 
 ## CLI
 
-`pip install dirsql` also installs a `dirsql` console script. `dirsql "<sql>"` (or `uvx dirsql "<sql>"`) runs one query and prints the rows as JSON — the default. `dirsql server` starts an HTTP server exposing the SDK over HTTP: `POST /query` for SQL and `GET /events` for a Server-Sent Events change stream. See the [CLI reference](https://thekevinscott.github.io/dirsql/reference/cli).
+`pip install dirsql` also installs a `dirsql` console script. `dirsql "<sql>"` (or `uvx dirsql "<sql>"`) runs one query and prints the rows as JSON — the default. `dirsql server` starts an HTTP server exposing the SDK over HTTP: `POST /query` for SQL and `GET /events` for a Server-Sent Events change stream. See the [CLI reference](https://dirsql.dev/reference/cli).
 
 ## License
 

{dirsql-0.4.63/packages/python → dirsql-0.4.66}/README.md

@@ -2,7 +2,7 @@
 
 Ephemeral SQL index over a local directory. `dirsql` watches a filesystem, ingests structured files into an in-memory SQLite database, and exposes a SQL query interface -- the filesystem is always the source of truth.
 
-[Documentation](https://thekevinscott.github.io/dirsql/?lang=python)
+[Documentation](https://dirsql.dev/?lang=python)
 
 Also available as [`dirsql` on crates.io](https://crates.io/crates/dirsql) and [`dirsql` on npm](https://www.npmjs.com/package/dirsql).
 
@@ -124,7 +124,7 @@ Each event has `.action` (`"insert"`, `"update"`, `"delete"`, or `"error"`), `.t
 
 ## CLI
 
-`pip install dirsql` also installs a `dirsql` console script. `dirsql "<sql>"` (or `uvx dirsql "<sql>"`) runs one query and prints the rows as JSON — the default. `dirsql server` starts an HTTP server exposing the SDK over HTTP: `POST /query` for SQL and `GET /events` for a Server-Sent Events change stream. See the [CLI reference](https://thekevinscott.github.io/dirsql/reference/cli).
+`pip install dirsql` also installs a `dirsql` console script. `dirsql "<sql>"` (or `uvx dirsql "<sql>"`) runs one query and prints the rows as JSON — the default. `dirsql server` starts an HTTP server exposing the SDK over HTTP: `POST /query` for SQL and `GET /events` for a Server-Sent Events change stream. See the [CLI reference](https://dirsql.dev/reference/cli).
 
 ## License
 

{dirsql-0.4.63 → dirsql-0.4.66}/docs/AGENTS.md

@@ -57,7 +57,7 @@ migrated -- a page survives only if a slot wants its content):
   `dirsql` in an application.
 - **Reference** (`reference/`) -- CLI flags and defaults; the complete
   `.dirsql.toml` schema; the command hook contract (placeholders, stdout
-  protocol, exit codes, timeouts); the columns model (named tables get their
+  protocol, failure semantics); the columns model (named tables get their
   hook's output; path-tables get stat columns); the
   HTTP API; the SDK page (`reference/sdk.md`, one page with
   Python/TypeScript/Rust code-groups -- the sole SDK home).

{dirsql-0.4.63 → dirsql-0.4.66}/docs/getting-started.md

@@ -68,11 +68,11 @@ this directory anyway — from inside `my-notes`, run one command:
 ::: code-group
 
 ```bash [npm]
-npx dirsql "SELECT COUNT(*) AS files FROM './'"
+npx dirsql "SELECT COUNT(*) AS files FROM './**'"
 ```
 
 ```bash [PyPI]
-uvx dirsql "SELECT COUNT(*) AS files FROM './'"
+uvx dirsql "SELECT COUNT(*) AS files FROM './**'"
 ```
 
 :::
@@ -91,10 +91,11 @@ files
 Three files, three rows. That one command scanned the directory, handed
 SQLite one row per file, ran your SQL, and printed the answer.
 
-There is no named table here — you never declared one. `'./'` is a
+There is no named table here — you never declared one. `'./**'` is a
 [path-table](./reference/path-tables.md): a quoted path written where a table
-name goes. `'./'` means everything under the directory you ran the command
-in. The path *is* the query.
+name goes. `'./**'` means every file under the directory you ran the command
+in, at any depth; `'./'` alone would list only the files directly inside it,
+like `ls`. The path *is* the query.
 
 ## 3. Select some columns
 
@@ -105,11 +106,11 @@ pretty-print. Ask for two columns instead of a count:
 ::: code-group
 
 ```bash [npm]
-npx dirsql query "SELECT path, size FROM './' ORDER BY path" | jq
+npx dirsql query "SELECT path, size FROM './**' ORDER BY path" | jq
 ```
 
 ```bash [PyPI]
-uvx dirsql query "SELECT path, size FROM './' ORDER BY path" | jq
+uvx dirsql query "SELECT path, size FROM './**' ORDER BY path" | jq
 ```
 
 :::

{dirsql-0.4.63/packages/python → dirsql-0.4.66}/docs/howto/define-tables.md

@@ -116,8 +116,8 @@ templates, with the trigger mistakes that fail silently, are in
 
 ## Going further
 
-- The parser mechanics — placeholders, stdout protocol, per-file failure
-  isolation — are the [`on-file` hook contract](../reference/hooks.md#on-file);
+- The parser mechanics — placeholders, stdout protocol, failure semantics —
+  are the [`on-file` hook contract](../reference/hooks.md#on-file);
   [Extract rows from file contents](./extract-from-contents.md) is the fuller
   recipe.
 - Your directory layout encodes data (authors, dates, IDs)? Split the path in

{dirsql-0.4.63/packages/rust → dirsql-0.4.66}/docs/howto/extract-from-contents.md

@@ -2,8 +2,9 @@
 
 Paths and stat metadata only get you so far — when the columns you want live
 *inside* the files (JSON fields, frontmatter, log lines), add an
-[`on-file`](../reference/config.md#table) command: it runs once per matched
-file and its stdout becomes the file's rows.
+[`on-file`](../reference/config.md#table) command: it runs once for the
+table, with every matched file as an argument, and the JSON array it prints
+becomes the table's rows.
 
 ## 1. Point a command at the files
 
@@ -13,8 +14,8 @@ Suppose each book is a JSON file:
 {"title": "Middlemarch", "author": "George Eliot", "year": 1871}
 ```
 
-Any program that reads a file and prints a **JSON array of row objects** on
-stdout works. With [`jq`](https://jqlang.org/):
+Any program that reads the files named by its arguments and prints a **JSON
+array of row objects** on stdout works. With [`jq`](https://jqlang.org/):
 
 ```toml
 [[table]]
@@ -27,8 +28,8 @@ on-file = "jq -c -n '[inputs | {title, author, year}]'"
 The command runs once for the table, with every matched file's absolute path
 appended as an argument, and prints one array for all of them — the
 [command hook contract](../reference/hooks.md#on-file), which also covers
-the argv splitting, working directory, stdout protocol, and timeout shared
-by every hook.
+the argv splitting, working directory, and stdout protocol shared by every
+hook.
 
 ## 2. Query the extracted columns
 
@@ -68,13 +69,21 @@ dirsql query "SELECT event, user FROM events" -c ./.dirsql.toml
 [{"event":"login","user":"alice"},{"event":"logout","user":"alice"},{"event":"login","user":"bob"}]
 ```
 
-## When a file fails
+## When the command fails
 
-A file whose command errors (or prints something that isn't a JSON array of
-objects) contributes no rows: `dirsql` warns on stderr and the scan
-continues — one bad file never takes down the index. Details in
-[failure semantics](../reference/hooks.md#failure-semantics); the JSON to
-SQLite value mapping is under
+The command runs once for the whole table, so its failure is the table's: a
+non-zero exit, no output, or output that is not a JSON array of objects fails
+the build, and `dirsql query` exits `1` with the command's stderr tail on
+stderr and nothing on stdout.
+
+```
+dirsql query: failed to load config: table `books`: on-file command failed: command `jq -c -n '[inputs | {title, author, year}]'` failed (exit 5): jq: error (at /home/me/library/books/broken.json:1): Cannot index string with string "title"
+```
+
+A file the command cannot parse is the command's to handle — skip it, or
+emit a row with `NULL`s — because dirsql has no per-file view of the rows.
+Details in [failure semantics](../reference/hooks.md#failure-semantics); the
+JSON to SQLite value mapping is under
 [`on-file` row mapping](../reference/config.md#on-file-row-mapping).
 
 ## Going further
@@ -82,8 +91,9 @@ SQLite value mapping is under
 - Just want to read a field or two out of some JSON files? SQLite's JSON
   operators work directly on a path-table's `content` column, with no config
   and no command — [Query JSON file contents](./query-json.md).
-- The command re-runs on every startup and on every change to a matched
-  file. If it is expensive, [keep the index across restarts](./persist.md).
+- The command re-runs on every startup and, over all of the table's files, on
+  every change under the table's glob. If it is expensive,
+  [keep the index across restarts](./persist.md).
 - Embedding `dirsql` in a program instead? The SDK's `on_file` callback
   fills the same role in-process — see
   [Embed `dirsql` in your application](./embed.md).

{dirsql-0.4.63/packages/rust → dirsql-0.4.66}/docs/howto/parse-files-into-columns.md

@@ -66,12 +66,17 @@ dirsql query "SELECT title, author FROM './posts/*.md' ORDER BY title" \
 Now the parser's output *is* the table. `--on-file` runs the command once,
 with every matched file's absolute path appended as an argument, under the
 shared [`on-file` hook contract](../reference/hooks.md#on-file) (argv
-splitting, timeout, and failure semantics all come from there). The stat
-columns are no longer reachable — a parser that wants the path emits it, since
-it already has the path. See
+splitting and failure semantics come from there). It runs in the directory
+you ran `dirsql` from, which is why `python3 extract.py` finds the script. The
+stat columns are no longer reachable — a parser that wants the path emits it,
+since it already has the path. See
 [Parsing rows with `--on-file`](../reference/path-tables.md#parsing-rows-with-on-file)
 for the full behavior.
 
+One run, one process: on a thousand files the parser starts once, not a
+thousand times, so a Python or Node parser costs its startup once per query.
+That is the shape to write — loop over `sys.argv[1:]`, collect, print once.
+
 `--on-file` applies to every path-table in the query and may be given at most
 once. It is a `query`-only flag: there is no config file involved yet, so it is
 the fastest way to see whether a parser produces the rows you expect.
@@ -114,12 +119,74 @@ gives every path-table one parser. In both spellings the table's columns are
 exactly what the parser emits: `dirsql` merges no filesystem facts back on. A
 row that needs the file's `path` emits it (the parser has the path).
 
+## The two shapes
+
+Every parser is one of two shapes. The array length is the row count either
+way; what differs is how many files feed it.
+
+**One file, many rows.** A lab notebook is a single Markdown file of dated
+`## YYYY-MM-DD` sections, and the question is about the sections. Name the one
+file as the path-table; the parser receives one path and prints one row per
+section. `notebook.py`:
+
+```python
+import json, re, sys
+
+rows = []
+for path in sys.argv[1:]:
+    text = open(path, encoding="utf-8").read()
+    for date, body in re.findall(r"^## (\d{4}-\d\d-\d\d)\n(.*?)(?=^## |\Z)", text, re.S | re.M):
+        rows.append({"date": date, "words": len(body.split())})
+print(json.dumps(rows))
+```
+
+```bash
+dirsql query "SELECT date, words FROM './notebook.md' ORDER BY date DESC LIMIT 2" \
+  --on-file 'python3 notebook.py'
+```
+
+```json
+[{"date":"2026-09-23","words":10},{"date":"2026-09-22","words":3}]
+```
+
+**Many files, many rows.** A tree of JSONL session logs under
+`projects/<name>/`, and the question is how many messages each project has per
+role. The parser receives every log's path and prints one row per message,
+carrying the project name it derived from the path, because dirsql will not
+attach it. `messages.py`:
+
+```python
+import json, os, sys
+
+rows = []
+for path in sys.argv[1:]:
+    project = os.path.relpath(path, "projects").split(os.sep)[0]
+    for line in open(path, encoding="utf-8"):
+        entry = json.loads(line)
+        if entry["type"] in ("user", "assistant"):
+            rows.append({"project": project, "role": entry["type"], "text": entry["message"]["content"]})
+print(json.dumps(rows))
+```
+
+```bash
+dirsql query "SELECT project, role, count(*) AS n FROM './projects/**/*.jsonl' GROUP BY project, role ORDER BY project, role" \
+  --on-file 'python3 messages.py'
+```
+
+```json
+[{"project":"alpha","role":"assistant","n":2},{"project":"alpha","role":"user","n":1},{"project":"beta","role":"user","n":2}]
+```
+
+The same two scripts work unchanged as a `[[table]]`'s `on-file` key. The
+loop is the same in both shapes, so a parser written for one file already
+handles a glob.
+
 ## Going further
 
-- The full stat-vs-parsed behavior, failure isolation, and skip rules:
+- The full stat-vs-parsed behavior, failure semantics, and skip rules:
   [Parsing rows with `--on-file`](../reference/path-tables.md#parsing-rows-with-on-file).
 - Starting from a declared table instead of a path-table?
   [Extract rows from file contents](./extract-from-contents.md) covers the
   config-first path.
-- One row per record *within* a file (JSONL, multiple frontmatter blocks): the
-  parser prints one object per row — the array length is the row count.
+- A very large table may reach the parser in several invocations; see
+  [Argument-list limits](../reference/hooks.md#argument-list-limits).

{dirsql-0.4.63/packages/rust → dirsql-0.4.66}/docs/howto/persist.md

@@ -4,8 +4,8 @@ By default the database is ephemeral: rebuilt from your files on every
 startup and discarded on exit. The [`--persist [PATH]`](../reference/cli.md#dirsql-server)
 flag keeps the SQLite index on disk instead, so a restart only re-parses
 files that actually changed — the difference between seconds and
-milliseconds on large trees, and between re-running and skipping expensive
-[`on-file`](./extract-from-contents.md) commands.
+milliseconds on large trees, and between re-running and skipping an expensive
+declared-table [`on-file`](./extract-from-contents.md) command.
 
 Whether and where to cache is a machine-local operational choice — it
 belongs to the command you run, not to the shared `.dirsql.toml`. That is
@@ -58,15 +58,11 @@ exactly what a non-persistent startup does.
 A run that changes nothing writes nothing: the cache file is read and left
 exactly as it was.
 
-This covers both kinds of table. Declared tables keep their indexed rows, and
-a [path-table](../reference/path-tables.md) parsed with `--on-file` keeps each
-file's parser output — so the second run over an unchanged tree spawns no
-parser process at all. A path-table's cached rows are keyed by the tree, the
-glob, the parser command and the `dirsql` version together, so changing any of
-them parses afresh rather than serving rows the current command never
-produced. A *stat* path-table caches nothing, because there is nothing to
-save: its columns are the metadata the scan already collected, and `content`
-is read live by design.
+This covers declared tables, which keep their indexed rows. A
+[path-table](../reference/path-tables.md) is not cached: a *stat* path-table
+has nothing to save, since its columns are the metadata the scan already
+collected and `content` is read live by design, and a path-table parsed with
+`--on-file` runs its parser on every start.
 
 Persistence is a startup-time optimization, not a change in meaning: the
 database remains a derived view of your files, and queries return the same

{dirsql-0.4.63/packages/rust → dirsql-0.4.66}/docs/howto/query-without-config.md

@@ -67,5 +67,5 @@ why indexes belong to only one of them, are in
 - Paths outside the root resolve too — `'/var/log/*.log'`, `'../notes'`,
   `'~/notes/*.md'` — reporting absolute paths
   ([path-tables reference](../reference/path-tables.md#paths-outside-the-index-root)).
-- `node_modules/` and `.git/` are skipped by default so a bare `'./'` does not
-  drown in machinery ([skip rules](../reference/path-tables.md#skip-rules)).
+- `node_modules/` and `.git/` are skipped by default so a bare `'./**'` does
+  not drown in machinery ([skip rules](../reference/path-tables.md#skip-rules)).

{dirsql-0.4.63/packages/python → dirsql-0.4.66}/docs/howto/search-by-meaning.md

@@ -120,6 +120,25 @@ ORDER BY vec_distance_cosine(emb, embed('local private models'))
 LIMIT 10
 ```
 
+A field that needs real parsing — frontmatter, a log line — comes out with a
+[parser attached](../reference/path-tables.md#parsing-rows-with-on-file) by
+`--on-file`; the plugin's `embed()` then works on the parser's columns, since a
+discovered plugin and an inline parser compose. With `title.py` printing one
+`{"path", "title"}` row per note from its frontmatter:
+
+```bash
+uvx --with dirsql-plugin-embeddings dirsql "
+  SELECT title,
+         vec_distance_cosine(embed(title), embed('how do I cook pasta?')) AS distance
+  FROM './notes/*.md'
+  ORDER BY distance
+  LIMIT 3" --on-file 'python3 title.py'
+```
+
+```json
+[{"title":"Boiling spaghetti","distance":0.491},{"title":"Planting tomatoes","distance":0.836},{"title":"Feature branches","distance":1.064}]
+```
+
 The same projection works as an `on-file` hook feeding the indexed shape:
 parse the field in the hook, store it as a column, and the trigger embeds it
 once instead of on every query.

{dirsql-0.4.63 → dirsql-0.4.66}/docs/reference/cli.md

@@ -88,7 +88,7 @@ front.
 ### Output format
 
 Rows go where they are useful: a **table** when stdout is a terminal, the
-**JSON array** when it is piped or redirected. `SELECT * FROM './'` in a
+**JSON array** when it is piped or redirected. `SELECT * FROM './**'` in a
 5000-file tree should not put a 5000-element JSON array in front of a person,
 and `dirsql "…" | jq` should not have to parse a table.
 
@@ -223,11 +223,7 @@ every statement, so it is reported once and exits `1` before the first prompt.
 | Code | Meaning |
 |---|---|
 | `0` | Clean EOF (Ctrl-D, `exit`, `quit`, or the end of a piped script) — **including when statements failed**. Matches interactive `sqlite3`; use [`dirsql query`](#dirsql-query) when a script needs a statement's exit status. |
-| `1` | The index could not be built (a bad `-c`, an unresolvable `--on-file`), or stdin could not be read. Nothing was executed. |
-
-`23` (partial scan) is not produced here: skipped files are reported before the
-first prompt, and a session's exit code describes the session rather than one
-scan.
+| `1` | The index could not be built (a bad `-c`, an unresolvable `--on-file`, a failed `on-file` command), or stdin could not be read. Nothing was executed. |
 
 ## `dirsql server`
 
@@ -283,7 +279,7 @@ curl -s localhost:7117/query -H 'content-type: application/json' \
 ```
 
 Earlier versions served an implicit table named `files` here. It is gone; a
-`SELECT ... FROM files` with no config now fails and points at `FROM './'`.
+`SELECT ... FROM files` with no config now fails and points at `FROM './**'`.
 
 Passing a config with `-c` fully overrules this default. A `-c` naming a file
 that does not exist is an error (not a fallback to the default); a config that
@@ -365,8 +361,9 @@ uses**, so behavior is identical to `POST /query` by construction:
 #### `--on-file <command>`
 
 Attach a parser to every [path-table](./path-tables.md#parsing-rows-with-on-file)
-in the query, so each matched file yields the rows the command prints (a JSON
-array of row objects) instead of the stat columns:
+in the query: the command runs once per path-table, with every matched file's
+path as a trailing argument, and the JSON array of row objects it prints
+replaces the stat columns:
 
 ```sh
 dirsql query "SELECT title, author FROM './posts/*.md'" \
@@ -375,17 +372,21 @@ dirsql query "SELECT title, author FROM './posts/*.md'" \
 
 The command follows the [`on-file` hook contract](./hooks.md#on-file) — argv
 splitting, the `{root}` placeholder, every matched path as a trailing
-argument, the failure semantics, and the timeout. The parser's output is the whole schema; the stat columns are not
-reachable on a parsed path-table. `--on-file` may be given **at most once** (a
-repeat is an error pointing at config files) and never touches config-declared
-tables. It is a `query`-only flag — server mode rejects it as an unknown
+argument, and the failure semantics — with one difference from the config
+key: there is no config file, so the command runs in the index root (the
+directory `dirsql query` was run in). The parser's output is the whole
+schema; the stat columns are not reachable on a parsed path-table.
+`--on-file` may be given **at most once** (a repeat is an error pointing at
+config files) and never touches config-declared tables. It is a `query`-only flag — server mode rejects it as an unknown
 argument. The command string is copy-paste identical to a `[[table]]`
 `on-file` key, so an inline parser graduates to a config file unchanged — see
 [Parse your files into columns](../howto/parse-files-into-columns.md).
 
 Errors print the same diagnostic the HTTP `{"error": …}` body carries —
-config failures, SQL errors, rejected reads, hook failures, timeouts — to
-stderr, with exit code `1`.
+config failures, SQL errors, rejected reads, hook failures — to stderr, with
+exit code `1`. A failed `on-file` command is a hook failure: nothing is
+printed on stdout, and the diagnostic carries the command's exit status and
+the tail of its stderr.
 
 #### `--format {auto,table,json}`
 
@@ -398,13 +399,13 @@ takes, with the same `auto` default. A one-shot query is usually piped, so
 | Code | Meaning |
 |---|---|
 | `0` | Query succeeded; rows printed on stdout. |
-| `1` | Any failure: config, SQL, rejected read, hook, or timeout. The diagnostic is on stderr. |
+| `1` | Any failure: config, SQL, rejected read, or a failed `on-file` command. The diagnostic is on stderr. |
 
 ## `dirsql init`
 
 Writes a starter `.dirsql.toml` as a scaffold to edit. It does **not**
-duplicate the zero-config floor (`SELECT * FROM './'` already lists every file
-with no config); instead it shows the **escalation**: one named `[[table]]`
+duplicate the zero-config floor (`SELECT * FROM './**'` already lists every
+file with no config); instead it shows the **escalation**: one named `[[table]]`
 with a glob, a schema, and a real `on-file` hook that pulls structured rows
 out of your files.
 
@@ -467,9 +468,9 @@ fragment is a launcher error naming the package — never a silent skip.
 ## Progress reporting
 
 Building the index over a large tree is not instant: the walk visits every
-file, then each matched file costs one `on-file` round trip plus whatever the
-table's `ddl` fires on insert. On a big corpus that is minutes. dirsql reports
-the two phases on **stderr** while they run:
+file, then each table runs its `on-file` command over all of its files, plus
+whatever the table's `ddl` fires on insert. On a big corpus that is minutes.
+dirsql reports the two phases on **stderr** while they run:
 
 ```
 dirsql: scanning 128413 files

{dirsql-0.4.63/packages/rust → dirsql-0.4.66}/docs/reference/config.md

@@ -24,7 +24,7 @@ the config file's location. See [`--config`](./cli.md#flags).
 
 | Key | Type | Default | Description |
 |---|---|---|---|
-| `ignore` | array of strings | `[]` | Glob patterns matched against root-relative paths. Matched files are skipped entirely — excluded from the initial scan and from watch events. |
+| `ignore` | array of strings | `[]` | Glob patterns matched against root-relative paths, under the [one glob rule](#glob-rule): `*` matches one level, `**` any depth. Matched files are skipped entirely — excluded from the initial scan and from watch events. |
 
 There is no timeout key. `on-file` hook runs are unbounded; to bound one, wrap
 its command in `timeout(1)` (see [Command hooks](./hooks.md#bounding-a-hook)).
@@ -43,6 +43,16 @@ directory.
 ignore = ["node_modules/**", ".git/**"]
 ```
 
+### Glob rule
+
+Every glob in dirsql — `ignore`, a `[[table]]`'s `glob`, and a
+[path-table](./path-tables.md#writing-the-path) — reads as the shell does:
+`*` and `?` match within one path segment and never cross `/`; `**` matches
+any depth. So `ignore = ["*"]` hides only the files directly inside the root,
+`ignore = ["build/*"]` hides `build/a.o` but not `build/sub/b.o`, and
+`glob = "*.json"` selects only the top-level `.json` files; write
+`**/*.json` to select them at every depth.
+
 Persistence is not a config key. Keep the SQLite index on disk between runs
 with the [`--persist [PATH]` CLI flag](./cli.md#dirsql-server) — a machine-local
 operational choice that belongs to the runner, not to shareable config.
@@ -220,7 +230,7 @@ what its required `on-file` command emits — dirsql injects nothing (see
 |---|---|---|
 | `name` | yes | The table's SQL name — the name you query it by. Declared, never derived from `ddl`: dirsql does not read the DDL text. The `ddl` must create a table by this name; if it doesn't, loading fails. |
 | `ddl` | yes | A SQL batch, run verbatim — any number of statements. It must create a table called `name`; that table holds the file rows, and only the columns it declares are kept (keys the `on-file` command emits that are not declared are dropped). The rest of the batch is yours: indexes, virtual tables, triggers. See [Batch `ddl`](#batch-ddl). |
-| `glob` | yes | Glob pattern matched against root-relative paths. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). |
+| `glob` | yes | Glob pattern matched against root-relative paths, under the [one glob rule](#glob-rule): `*` matches one level, `**` any depth. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). |
 | `on-file` | **yes** | A command run once per table, with every matched file's absolute path appended as a trailing argument; its stdout (one JSON array of row objects) is the table's rows. Must be non-empty. A `[[table]]` with no `on-file` is a load error (see [parse errors](#parse-errors)). See [Command hooks](./hooks.md#on-file). |
 | `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied by the `on-file` output. When `false`, extra keys are dropped and missing columns become `NULL`. |
 
@@ -320,9 +330,9 @@ integral number → `INTEGER`, any other number → `REAL`; a string → `TEXT`;
 nested array or object → its JSON text as `TEXT`.
 
 A row's columns are exactly the keys the command emits, narrowed to the DDL;
-dirsql merges nothing else in. Output that is not a JSON array of objects is a
-per-file failure: the file is skipped with a stderr warning and the scan
-continues (see [failure semantics](./hooks.md#failure-semantics)).
+dirsql merges nothing else in. Output that is not a JSON array of objects fails
+the table, and a failed table fails the build (see
+[failure semantics](./hooks.md#failure-semantics)).
 
 ## Composing multiple configs
 
@@ -386,7 +396,7 @@ SDKs raise/reject) when:
 - A `[[table]]` entry omits `on-file` (or it is empty/whitespace). The error
   names the offending glob and points at the fix:
 
-  > `[[table]] '**/*.md' has no on-file hook, so every row would be all-NULL. Add an `on-file` hook that emits the columns, or, for stat columns with no code, query the path directly: `FROM './'``
+  > `[[table]] '**/*.md' has no on-file hook, so every row would be all-NULL. Add an `on-file` hook that emits the columns, or, for stat columns with no code, query the path directly: `FROM './**'``
 
 - A `[[dirsql.extension]]` entry omits `path`, or `path` is empty.
 - A `[[dirsql.function]]` entry omits `name`, `command`, or `args` (or

{dirsql-0.4.63 → dirsql-0.4.66}/docs/reference/hooks.md

@@ -37,10 +37,13 @@ The available placeholders are listed under the
 
 ### Working directory and environment
 
-The command runs in the **config file's directory**, so relative paths in
-the command resolve predictably regardless of where `dirsql` was launched.
-It inherits `dirsql`'s environment, so tools like `uvx --with …` / `npx …`
-resolve their dependencies as usual.
+A command named by a `[[table]]` runs in the **config file's directory**, so
+relative paths in the command resolve predictably regardless of where
+`dirsql` was launched. A command named by the `--on-file` flag has no config
+file: it runs in the **index root**, the directory `dirsql query` was run in,
+which is also the directory a `./` path-table is relative to. In both forms
+`{root}` is the index root. The command inherits `dirsql`'s environment, so
+tools like `uvx --with …` / `npx …` resolve their dependencies as usual.
 
 ### stdout protocol
 
@@ -69,8 +72,8 @@ on-file = "timeout 30 my-extractor"
 ```
 
 When the wrapper kills an overrunning command, the run exits non-zero and
-the ordinary [failure semantics](#failure-semantics) apply — the file is
-skipped, the scan continues.
+the ordinary [failure semantics](#failure-semantics) apply — the table's
+command failed, so the build fails.
 
 ::: warning Windows
 Windows's built-in `timeout` command is a *sleep*, not a bound — it cannot
@@ -94,20 +97,39 @@ A hook run fails when the command:
 - exits zero but prints no non-empty stdout line,
 - or prints output that does not parse as a JSON array of row objects.
 
-What a failure *means*: **per-file isolation.** The file contributes no rows
-and is reported as skipped; the scan indexes every other file and commits.
-The CLI names up to ten skipped files on stderr, then `... and N more`, and
-exits `23` — distinct from `0` (clean) and `1` (the run failed), so a caller
-can tell a partial index from a complete one. A row the table rejects under
-`strict` counts as the same kind of failure.
+What a failure *means*: the command ran once for the whole table, so its
+failure is the table's, and a table that cannot be filled fails the build.
+The error names the table and carries the command's exit status and stderr
+tail; the CLI prints it and exits `1`. A row the table rejects under `strict`
+counts as the same kind of failure.
 
 ## `on-file` contract
 
-Runs once per file matched by the table's `glob`, at initial scan and on
-every watched change. The command reads the file itself and prints a JSON
-array of row objects; see [`[[table]]`](./config.md#table) for the
+Runs **once per table**, with every file matched by the table's `glob`
+appended to the command as a trailing argument, and prints **one JSON array
+of row objects** for the whole table. That is the whole contract. The command
+reads the files itself; see [`[[table]]`](./config.md#table) for the
 row-mapping rules.
 
+One process over all the files is the shape to write: a loop over the
+arguments that collects rows and prints them once at the end. Parsing is the
+command's cost, not dirsql's — dirsql spawns the command, waits, and reads
+its stdout. A table over no matched files spawns nothing.
+
+```python
+import json, sys
+
+rows = []
+for path in sys.argv[1:]:
+    rows.extend(parse(path))
+print(json.dumps(rows))
+```
+
+There is no row-to-file attribution: dirsql does not know which file a row
+came from, and a row that needs the path carries it as a column the command
+emitted. In watch mode a change to any file under the table's glob re-runs
+the command over all of the table's files and replaces the table's rows.
+
 The same command is attachable two ways, over the same contract: the
 `on-file` config key on a `[[table]]`, and the
 [`--on-file <command>`](./cli.md#on-file-command) flag on `dirsql query`,
@@ -126,5 +148,20 @@ the path or stat metadata emits it (it has the paths).
 The matched files' **absolute** paths are not placeholders: they are appended
 to the command as trailing arguments, after everything written in the
 command, so `on-file = "extract.py"` receives them as `sys.argv[1:]` — one run
-per table, self-sufficient from any working directory. A command that still
-spells `{path}` is rejected at startup.
+per table, self-sufficient from any working directory. A tool that needs a
+path anywhere other than last gets a wrapper script. A command that spells
+`{path}` is rejected at startup:
+
+```
+on-file command `python3 extract.py {path}` uses `{path}`, but an on-file command now runs once per table with every matched path appended as trailing arguments and prints one JSON array of row objects. Remove `{path}` and read the paths from the command's arguments.
+```
+
+### Argument-list limits
+
+Every platform caps the bytes one process may receive as arguments. When a
+table's paths outgrow that cap, dirsql splits them into the fewest
+consecutive runs that fit, runs the command once per run, and concatenates
+the arrays it prints, in order. The command cannot tell: each run is an
+ordinary invocation with a subset of the paths. A command must therefore not
+assume one invocation sees every path — a count, a cross-file join, or a
+dedupe over `sys.argv[1:]` is per run, not per table. Do that work in SQL.