dirsql 0.4.69 → 0.4.70

package/docs/howto/extract-from-contents.md

@@ -2,8 +2,9 @@
 
 Paths and stat metadata only get you so far — when the columns you want live
 *inside* the files (JSON fields, frontmatter, log lines), add an
-[`on-file`](../reference/config.md#table) command: it runs once per matched
-file and its stdout becomes the file's rows.
+[`on-file`](../reference/config.md#table) command: it runs once for the
+table, with every matched file as an argument, and the JSON array it prints
+becomes the table's rows.
 
 ## 1. Point a command at the files
 
@@ -13,8 +14,8 @@ Suppose each book is a JSON file:
 {"title": "Middlemarch", "author": "George Eliot", "year": 1871}
 ```
 
-Any program that reads a file and prints a **JSON array of row objects** on
-stdout works. With [`jq`](https://jqlang.org/):
+Any program that reads the files named by its arguments and prints a **JSON
+array of row objects** on stdout works. With [`jq`](https://jqlang.org/):
 
 ```toml
 [[table]]
@@ -27,8 +28,8 @@ on-file = "jq -c -n '[inputs | {title, author, year}]'"
 The command runs once for the table, with every matched file's absolute path
 appended as an argument, and prints one array for all of them — the
 [command hook contract](../reference/hooks.md#on-file), which also covers
-the argv splitting, working directory, stdout protocol, and timeout shared
-by every hook.
+the argv splitting, working directory, and stdout protocol shared by every
+hook.
 
 ## 2. Query the extracted columns
 
@@ -68,13 +69,21 @@ dirsql query "SELECT event, user FROM events" -c ./.dirsql.toml
 [{"event":"login","user":"alice"},{"event":"logout","user":"alice"},{"event":"login","user":"bob"}]
 ```
 
-## When a file fails
+## When the command fails
 
-A file whose command errors (or prints something that isn't a JSON array of
-objects) contributes no rows: `dirsql` warns on stderr and the scan
-continues — one bad file never takes down the index. Details in
-[failure semantics](../reference/hooks.md#failure-semantics); the JSON to
-SQLite value mapping is under
+The command runs once for the whole table, so its failure is the table's: a
+non-zero exit, no output, or output that is not a JSON array of objects fails
+the build, and `dirsql query` exits `1` with the command's stderr tail on
+stderr and nothing on stdout.
+
+```
+dirsql query: failed to load config: table `books`: on-file command failed: command `jq -c -n '[inputs | {title, author, year}]'` failed (exit 5): jq: error (at /home/me/library/books/broken.json:1): Cannot index string with string "title"
+```
+
+A file the command cannot parse is the command's to handle — skip it, or
+emit a row with `NULL`s — because dirsql has no per-file view of the rows.
+Details in [failure semantics](../reference/hooks.md#failure-semantics); the
+JSON to SQLite value mapping is under
 [`on-file` row mapping](../reference/config.md#on-file-row-mapping).
 
 ## Going further
@@ -82,8 +91,9 @@ SQLite value mapping is under
 - Just want to read a field or two out of some JSON files? SQLite's JSON
   operators work directly on a path-table's `content` column, with no config
   and no command — [Query JSON file contents](./query-json.md).
-- The command re-runs on every startup and on every change to a matched
-  file. If it is expensive, [keep the index across restarts](./persist.md).
+- The command re-runs on every startup and, over all of the table's files, on
+  every change under the table's glob. If it is expensive,
+  [keep the index across restarts](./persist.md).
 - Embedding `dirsql` in a program instead? The SDK's `on_file` callback
   fills the same role in-process — see
   [Embed `dirsql` in your application](./embed.md).

package/docs/howto/parse-files-into-columns.md

@@ -66,12 +66,17 @@ dirsql query "SELECT title, author FROM './posts/*.md' ORDER BY title" \
 Now the parser's output *is* the table. `--on-file` runs the command once,
 with every matched file's absolute path appended as an argument, under the
 shared [`on-file` hook contract](../reference/hooks.md#on-file) (argv
-splitting, timeout, and failure semantics all come from there). The stat
-columns are no longer reachable — a parser that wants the path emits it, since
-it already has the path. See
+splitting and failure semantics come from there). It runs in the directory
+you ran `dirsql` from, which is why `python3 extract.py` finds the script. The
+stat columns are no longer reachable — a parser that wants the path emits it,
+since it already has the path. See
 [Parsing rows with `--on-file`](../reference/path-tables.md#parsing-rows-with-on-file)
 for the full behavior.
 
+One run, one process: on a thousand files the parser starts once, not a
+thousand times, so a Python or Node parser costs its startup once per query.
+That is the shape to write — loop over `sys.argv[1:]`, collect, print once.
+
 `--on-file` applies to every path-table in the query and may be given at most
 once. It is a `query`-only flag: there is no config file involved yet, so it is
 the fastest way to see whether a parser produces the rows you expect.
@@ -114,12 +119,74 @@ gives every path-table one parser. In both spellings the table's columns are
 exactly what the parser emits: `dirsql` merges no filesystem facts back on. A
 row that needs the file's `path` emits it (the parser has the path).
 
+## The two shapes
+
+Every parser is one of two shapes. The array length is the row count either
+way; what differs is how many files feed it.
+
+**One file, many rows.** A lab notebook is a single Markdown file of dated
+`## YYYY-MM-DD` sections, and the question is about the sections. Name the one
+file as the path-table; the parser receives one path and prints one row per
+section. `notebook.py`:
+
+```python
+import json, re, sys
+
+rows = []
+for path in sys.argv[1:]:
+    text = open(path, encoding="utf-8").read()
+    for date, body in re.findall(r"^## (\d{4}-\d\d-\d\d)\n(.*?)(?=^## |\Z)", text, re.S | re.M):
+        rows.append({"date": date, "words": len(body.split())})
+print(json.dumps(rows))
+```
+
+```bash
+dirsql query "SELECT date, words FROM './notebook.md' ORDER BY date DESC LIMIT 2" \
+  --on-file 'python3 notebook.py'
+```
+
+```json
+[{"date":"2026-09-23","words":10},{"date":"2026-09-22","words":3}]
+```
+
+**Many files, many rows.** A tree of JSONL session logs under
+`projects/<name>/`, and the question is how many messages each project has per
+role. The parser receives every log's path and prints one row per message,
+carrying the project name it derived from the path, because dirsql will not
+attach it. `messages.py`:
+
+```python
+import json, os, sys
+
+rows = []
+for path in sys.argv[1:]:
+    project = os.path.relpath(path, "projects").split(os.sep)[0]
+    for line in open(path, encoding="utf-8"):
+        entry = json.loads(line)
+        if entry["type"] in ("user", "assistant"):
+            rows.append({"project": project, "role": entry["type"], "text": entry["message"]["content"]})
+print(json.dumps(rows))
+```
+
+```bash
+dirsql query "SELECT project, role, count(*) AS n FROM './projects/**/*.jsonl' GROUP BY project, role ORDER BY project, role" \
+  --on-file 'python3 messages.py'
+```
+
+```json
+[{"project":"alpha","role":"assistant","n":2},{"project":"alpha","role":"user","n":1},{"project":"beta","role":"user","n":2}]
+```
+
+The same two scripts work unchanged as a `[[table]]`'s `on-file` key. The
+loop is the same in both shapes, so a parser written for one file already
+handles a glob.
+
 ## Going further
 
-- The full stat-vs-parsed behavior, failure isolation, and skip rules:
+- The full stat-vs-parsed behavior, failure semantics, and skip rules:
   [Parsing rows with `--on-file`](../reference/path-tables.md#parsing-rows-with-on-file).
 - Starting from a declared table instead of a path-table?
   [Extract rows from file contents](./extract-from-contents.md) covers the
   config-first path.
-- One row per record *within* a file (JSONL, multiple frontmatter blocks): the
-  parser prints one object per row — the array length is the row count.
+- A very large table may reach the parser in several invocations; see
+  [Argument-list limits](../reference/hooks.md#argument-list-limits).

package/docs/howto/search-by-meaning.md

@@ -120,6 +120,25 @@ ORDER BY vec_distance_cosine(emb, embed('local private models'))
 LIMIT 10
 ```
 
+A field that needs real parsing — frontmatter, a log line — comes out with a
+[parser attached](../reference/path-tables.md#parsing-rows-with-on-file) by
+`--on-file`; the plugin's `embed()` then works on the parser's columns, since a
+discovered plugin and an inline parser compose. With `title.py` printing one
+`{"path", "title"}` row per note from its frontmatter:
+
+```bash
+uvx --with dirsql-plugin-embeddings dirsql "
+  SELECT title,
+         vec_distance_cosine(embed(title), embed('how do I cook pasta?')) AS distance
+  FROM './notes/*.md'
+  ORDER BY distance
+  LIMIT 3" --on-file 'python3 title.py'
+```
+
+```json
+[{"title":"Boiling spaghetti","distance":0.491},{"title":"Planting tomatoes","distance":0.836},{"title":"Feature branches","distance":1.064}]
+```
+
 The same projection works as an `on-file` hook feeding the indexed shape:
 parse the field in the hook, store it as a column, and the trigger embeds it
 once instead of on every query.

package/docs/reference/cli.md

@@ -361,8 +361,9 @@ uses**, so behavior is identical to `POST /query` by construction:
 #### `--on-file <command>`
 
 Attach a parser to every [path-table](./path-tables.md#parsing-rows-with-on-file)
-in the query, so each matched file yields the rows the command prints (a JSON
-array of row objects) instead of the stat columns:
+in the query: the command runs once per path-table, with every matched file's
+path as a trailing argument, and the JSON array of row objects it prints
+replaces the stat columns:
 
 ```sh
 dirsql query "SELECT title, author FROM './posts/*.md'" \
@@ -371,17 +372,21 @@ dirsql query "SELECT title, author FROM './posts/*.md'" \
 
 The command follows the [`on-file` hook contract](./hooks.md#on-file) — argv
 splitting, the `{root}` placeholder, every matched path as a trailing
-argument, the failure semantics, and the timeout. The parser's output is the whole schema; the stat columns are not
-reachable on a parsed path-table. `--on-file` may be given **at most once** (a
-repeat is an error pointing at config files) and never touches config-declared
-tables. It is a `query`-only flag — server mode rejects it as an unknown
+argument, and the failure semantics — with one difference from the config
+key: there is no config file, so the command runs in the index root (the
+directory `dirsql query` was run in). The parser's output is the whole
+schema; the stat columns are not reachable on a parsed path-table.
+`--on-file` may be given **at most once** (a repeat is an error pointing at
+config files) and never touches config-declared tables. It is a `query`-only flag — server mode rejects it as an unknown
 argument. The command string is copy-paste identical to a `[[table]]`
 `on-file` key, so an inline parser graduates to a config file unchanged — see
 [Parse your files into columns](../howto/parse-files-into-columns.md).
 
 Errors print the same diagnostic the HTTP `{"error": …}` body carries —
-config failures, SQL errors, rejected reads, hook failures, timeouts — to
-stderr, with exit code `1`.
+config failures, SQL errors, rejected reads, hook failures — to stderr, with
+exit code `1`. A failed `on-file` command is a hook failure: nothing is
+printed on stdout, and the diagnostic carries the command's exit status and
+the tail of its stderr.
 
 #### `--format {auto,table,json}`
 
@@ -394,7 +399,7 @@ takes, with the same `auto` default. A one-shot query is usually piped, so
 | Code | Meaning |
 |---|---|
 | `0` | Query succeeded; rows printed on stdout. |
-| `1` | Any failure: config, SQL, rejected read, hook, or timeout. The diagnostic is on stderr. |
+| `1` | Any failure: config, SQL, rejected read, or a failed `on-file` command. The diagnostic is on stderr. |
 
 ## `dirsql init`
 
@@ -463,9 +468,9 @@ fragment is a launcher error naming the package — never a silent skip.
 ## Progress reporting
 
 Building the index over a large tree is not instant: the walk visits every
-file, then each matched file costs one `on-file` round trip plus whatever the
-table's `ddl` fires on insert. On a big corpus that is minutes. dirsql reports
-the two phases on **stderr** while they run:
+file, then each table runs its `on-file` command over all of its files, plus
+whatever the table's `ddl` fires on insert. On a big corpus that is minutes.
+dirsql reports the two phases on **stderr** while they run:
 
 ```
 dirsql: scanning 128413 files

package/docs/reference/hooks.md

@@ -37,10 +37,13 @@ The available placeholders are listed under the
 
 ### Working directory and environment
 
-The command runs in the **config file's directory**, so relative paths in
-the command resolve predictably regardless of where `dirsql` was launched.
-It inherits `dirsql`'s environment, so tools like `uvx --with …` / `npx …`
-resolve their dependencies as usual.
+A command named by a `[[table]]` runs in the **config file's directory**, so
+relative paths in the command resolve predictably regardless of where
+`dirsql` was launched. A command named by the `--on-file` flag has no config
+file: it runs in the **index root**, the directory `dirsql query` was run in,
+which is also the directory a `./` path-table is relative to. In both forms
+`{root}` is the index root. The command inherits `dirsql`'s environment, so
+tools like `uvx --with …` / `npx …` resolve their dependencies as usual.
 
 ### stdout protocol
 
@@ -102,11 +105,31 @@ counts as the same kind of failure.
 
 ## `on-file` contract
 
-Runs once per table, over every file matched by the table's `glob`, at
-initial scan and on every watched change. The command reads the files itself
-and prints a JSON array of row objects; see [`[[table]]`](./config.md#table) for the
+Runs **once per table**, with every file matched by the table's `glob`
+appended to the command as a trailing argument, and prints **one JSON array
+of row objects** for the whole table. That is the whole contract. The command
+reads the files itself; see [`[[table]]`](./config.md#table) for the
 row-mapping rules.
 
+One process over all the files is the shape to write: a loop over the
+arguments that collects rows and prints them once at the end. Parsing is the
+command's cost, not dirsql's — dirsql spawns the command, waits, and reads
+its stdout. A table over no matched files spawns nothing.
+
+```python
+import json, sys
+
+rows = []
+for path in sys.argv[1:]:
+    rows.extend(parse(path))
+print(json.dumps(rows))
+```
+
+There is no row-to-file attribution: dirsql does not know which file a row
+came from, and a row that needs the path carries it as a column the command
+emitted. In watch mode a change to any file under the table's glob re-runs
+the command over all of the table's files and replaces the table's rows.
+
 The same command is attachable two ways, over the same contract: the
 `on-file` config key on a `[[table]]`, and the
 [`--on-file <command>`](./cli.md#on-file-command) flag on `dirsql query`,
@@ -125,5 +148,20 @@ the path or stat metadata emits it (it has the paths).
 The matched files' **absolute** paths are not placeholders: they are appended
 to the command as trailing arguments, after everything written in the
 command, so `on-file = "extract.py"` receives them as `sys.argv[1:]` — one run
-per table, self-sufficient from any working directory. A command that still
-spells `{path}` is rejected at startup.
+per table, self-sufficient from any working directory. A tool that needs a
+path anywhere other than last gets a wrapper script. A command that spells
+`{path}` is rejected at startup:
+
+```
+on-file command `python3 extract.py {path}` uses `{path}`, but an on-file command now runs once per table with every matched path appended as trailing arguments and prints one JSON array of row objects. Remove `{path}` and read the paths from the command's arguments.
+```
+
+### Argument-list limits
+
+Every platform caps the bytes one process may receive as arguments. When a
+table's paths outgrow that cap, dirsql splits them into the fewest
+consecutive runs that fit, runs the command once per run, and concatenates
+the arrays it prints, in order. The command cannot tell: each run is an
+ordinary invocation with a subset of the paths. A command must therefore not
+assume one invocation sees every path — a count, a cross-file join, or a
+dedupe over `sys.argv[1:]` is per run, not per table. Do that work in SQL.

package/docs/reference/path-tables.md

@@ -79,6 +79,25 @@ SELECT * FROM ./;
 -- hint: paths used as table names must be quoted; did you mean "./"?
 ```
 
+### Symlinks
+
+Symlinks are followed the way `bash -O globstar` follows them. A symlinked
+file is a file: `'./*'` lists it, and so does naming it. A symlinked directory
+is entered by any component of the path except `**`, which never walks into
+one. A broken link lists nothing.
+
+With `linkdir -> real`:
+
+| You write | dirsql scans |
+| --- | --- |
+| `'./*/*'` | `linkdir/r.md` and `real/r.md` |
+| `'./**'` | everything under `real/`, nothing under `linkdir/` |
+| `'./**/*.md'` | `linkdir/r.md` too: `**` may stop on `linkdir` for `*.md` to enter |
+| `'./linkdir/**'` | everything under `linkdir/` |
+
+Each link entered uses up one component of the path, so a symlink cycle is
+followed at most once per component and never hangs a scan.
+
 ### Paths outside the index root
 
 Three other prefixes resolve, with their usual shell meanings:
@@ -153,16 +172,27 @@ dirsql query "SELECT title, author FROM './posts/*.md'" \
 The command runs once, with every matched file's absolute path appended as an
 argument, and prints one JSON array of row objects, exactly like a declared
 table's [`on-file` hook](/reference/hooks) — same argv splitting, same `{root}`
-placeholder, same timeout. Its output *is* the table:
+placeholder, same [argument-list splitting](/reference/hooks#argument-list-limits)
+over a very large table. Its output *is* the table:
 
 - **The parser supplies the whole schema.** Columns are inferred from the keys
   across the emitted rows. The stat columns (`path`, `size`, …) are **not**
   reachable on a parsed path-table — a parser that wants the path emits it (it
   has the paths). The two modes stay cleanly separate.
-- **A parser failure is the table's failure.** A run that fails (spawn,
-  non-zero exit, a `timeout(1)` kill, or no output) or whose output is not a
-  JSON array of rows fails the query with an error naming the table; there is
-  no partial table.
+- **The parser runs from the index root.** There is no config file to anchor
+  it, so the command's working directory is the directory `dirsql query` was
+  run in — the same directory the `./` path is relative to — and `{root}`
+  names it. `python3 extract.py` finds `extract.py` in that directory,
+  whatever directory the path-table scans.
+- **A parser failure is the query's failure.** A run that fails (spawn,
+  non-zero exit, or no output) or whose output is not a JSON array of row
+  objects fails the query with the command's stderr tail; there is no partial
+  table, and the CLI exits `1`. A run that succeeds but prints an empty array
+  fails too, because there are no keys to infer columns from:
+
+  ```
+  dirsql_parsed: parser produced no rows for `posts/*.md`; cannot infer a schema
+  ```
 - **The skip rules still apply.** A parsed scan honors the same `node_modules`
   /`.git`/`ignore` rules a stat scan does (see below).
 - **The parser runs on every start.** A parsed path-table is not kept in the
@@ -180,12 +210,14 @@ A path-table is scanned when the statement runs, so it always reflects the
 filesystem as it is *now* — unlike declared tables, which are indexed on build
 and updated by the watcher. A file created a moment ago shows up immediately.
 
-The scan is live all the way down to `content`: a file's body is read when the
-query names the `content` column, not when the row is discovered. A file
-deleted *after* the scan finds it but *before* its `content` is read yields
-`NULL` content — the same NULL an unreadable or non-UTF-8 file gives — rather
-than failing the query. This is an accepted consequence of reading live, not a
-bug to design around.
+The scan is live all the way down to `content`: a file's body is read only
+when the query names the `content` column, not when the row is discovered. A
+query that names it has the bodies of the rows it selects read all at once,
+several files at a time, before the first row comes back. A file deleted
+*after* the scan finds it but *before* its `content` is read yields `NULL`
+content — the same NULL an unreadable or non-UTF-8 file gives — rather than
+failing the query. This is an accepted consequence of reading live, not a bug
+to design around.
 
 The table itself is per-connection: it lives in `temp`, so it cannot leak into
 `sqlite_master` or survive a restart. Under `--persist` a *parsed* table's rows

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dirsql",
-  "version": "0.4.69",
+  "version": "0.4.70",
   "description": "Ephemeral SQL index over a local directory",
   "license": "MIT",
   "repository": "https://github.com/thekevinscott/dirsql",
@@ -221,10 +221,10 @@
     ]
   },
   "optionalDependencies": {
-    "@dirsql/lib-linux-x64-gnu": "0.4.69",
-    "@dirsql/lib-linux-arm64-gnu": "0.4.69",
-    "@dirsql/lib-darwin-x64": "0.4.69",
-    "@dirsql/lib-darwin-arm64": "0.4.69",
-    "@dirsql/lib-win32-x64-msvc": "0.4.69"
+    "@dirsql/lib-linux-x64-gnu": "0.4.70",
+    "@dirsql/lib-linux-arm64-gnu": "0.4.70",
+    "@dirsql/lib-darwin-x64": "0.4.70",
+    "@dirsql/lib-darwin-arm64": "0.4.70",
+    "@dirsql/lib-win32-x64-msvc": "0.4.70"
   }
 }