dirsql 0.4.108 → 0.4.110

package/docs/getting-started.md

@@ -152,22 +152,16 @@ That parser is the point: a named table's columns are exactly what its
 `on-file` command emits. `dirsql` adds nothing on its own — so this is where
 you pull a value out of each file. Still inside `my-notes`, write a tiny
 parser that reads a note's title line (the `# Heading`) and its author (the
-folder name), and prints them as a JSON row:
+folder name), and prints them as a JSON row, one object per line:
 
 ```bash
 cat > note.sh <<'EOF'
 #!/usr/bin/env sh
-printf '['
-sep=''
 for f; do
   title=$(sed -n 's/^# //p' "$f" | head -n1)
   author=$(basename "$(dirname "$f")")
-  printf '%s{"title":%s,"author":%s}' "$sep" \
-    "$(jq -Rn --arg t "$title" '$t')" \
-    "$(jq -Rn --arg a "$author" '$a')"
-  sep=','
+  jq -cn --arg t "$title" --arg a "$author" '{title: $t, author: $a}'
 done
-printf ']'
 EOF
 ```
 
@@ -189,7 +183,7 @@ Three keys define the table:
   `notes/`, relative to the directory the config sits in.
 - `ddl` is ordinary `CREATE TABLE` SQL naming the columns you want to keep.
 - `on-file` is the command run once for the table, with every matched file's
-  path appended as an argument; the JSON array it prints is the table's rows.
+  path appended as an argument; the JSON objects it prints, one per line, are the table's rows.
   The columns are exactly what it emits — `title` from the heading, `author`
   from the folder.
 

package/docs/howto/columns-from-paths.md

@@ -17,19 +17,17 @@ photos/2025/01/snow.jpg
 
 Write a small parser, `pathcols.py`, that turns each path into a row. It
 receives the matched files' absolute paths as its arguments and prints one
-JSON array of row objects:
+JSON object per line, one row each:
 
 ```python
 #!/usr/bin/env python3
 import json, os, sys
 
-rows = []
 for path in sys.argv[1:]:
     parts = path.split(os.sep)
     # .../photos/<year>/<month>/<file>
-    rows.append({"year": parts[-3], "month": parts[-2],
-                 "basename": os.path.basename(path)})
-print(json.dumps(rows))
+    print(json.dumps({"year": parts[-3], "month": parts[-2],
+                     "basename": os.path.basename(path)}))
 ```
 
 Point a table at it in `.dirsql.toml`:

package/docs/howto/define-tables.md

@@ -11,18 +11,16 @@ Suppose your blog posts live under `posts/`, one markdown file each. A named
 table needs three keys: a `glob` that selects the files, a `ddl` that names
 the columns, and an [`on-file`](../reference/config.md#table) hook that emits
 the table's rows. Put a small parser next to the config — `extract.py`, which
-reads each post's title line and prints one JSON array of row objects:
+reads each post's title line and prints one JSON row object per line:
 
 ```python
 #!/usr/bin/env python3
 import json, os, sys
 
-rows = []
 for path in sys.argv[1:]:
     text = open(path, encoding="utf-8").read()
     title = next((l[2:].strip() for l in text.splitlines() if l.startswith("# ")), None)
-    rows.append({"title": title, "slug": os.path.basename(path)[:-3]})
-print(json.dumps(rows))
+    print(json.dumps({"title": title, "slug": os.path.basename(path)[:-3]}))
 ```
 
 Then declare the table in `.dirsql.toml`:

package/docs/howto/extract-from-contents.md

@@ -3,8 +3,8 @@
 Paths and stat metadata only get you so far — when the columns you want live
 *inside* the files (JSON fields, frontmatter, log lines), add an
 [`on-file`](../reference/config.md#table) command: it runs once for the
-table, with every matched file as an argument, and the JSON array it prints
-becomes the table's rows.
+table, with every matched file as an argument, and the JSON objects it prints,
+one per line, become the table's rows.
 
 ## 1. Point a command at the files
 
@@ -14,19 +14,19 @@ Suppose each book is a JSON file:
 {"title": "Middlemarch", "author": "George Eliot", "year": 1871}
 ```
 
-Any program that reads the files named by its arguments and prints a **JSON
-array of row objects** on stdout works. With [`jq`](https://jqlang.org/):
+Any program that reads the files named by its arguments and prints **one JSON
+object per line** on stdout works. With [`jq`](https://jqlang.org/):
 
 ```toml
 [[table]]
 name = "books"
 ddl     = "CREATE TABLE books (title TEXT, author TEXT, year INTEGER)"
 glob    = "books/*.json"
-on-file = "jq -c -n '[inputs | {title, author, year}]'"
+on-file = "jq -c '{title, author, year}'"
 ```
 
 The command runs once for the table, with every matched file's absolute path
-appended as an argument, and prints one array for all of them — the
+appended as an argument, and prints one object per line for all of them — the
 [command hook contract](../reference/hooks.md#on-file), which also covers
 the argv splitting, working directory, and stdout protocol shared by every
 hook.
@@ -50,15 +50,15 @@ The table's columns are exactly what the command emits, narrowed to the DDL —
 
 ## Multiple rows per file
 
-Each object in the printed array is one row. To turn a JSONL file into one
-row per line, slurp it:
+Each printed line is one row. A JSONL file is already one row per line, so
+project the fields you want:
 
 ```toml
 [[table]]
 name = "events"
 ddl     = "CREATE TABLE events (event TEXT, user TEXT)"
 glob    = "logs/*.jsonl"
-on-file = "jq -c -s '.'"
+on-file = "jq -c '{event, user}'"
 ```
 
 ```bash
@@ -72,12 +72,12 @@ dirsql query "SELECT event, user FROM events" -c ./.dirsql.toml
 ## When the command fails
 
 The command runs once for the whole table, so its failure is the table's: a
-non-zero exit, no output, or output that is not a JSON array of objects fails
-the build, and `dirsql query` exits `1` with the command's stderr tail on
+non-zero exit, or output that is not one JSON object per line (an array
+is rejected) fails the build, and `dirsql query` exits `1` with the command's stderr tail on
 stderr and nothing on stdout.
 
 ```
-dirsql query: failed to load config: table `books`: on-file command failed: command `jq -c -n '[inputs | {title, author, year}]'` failed (exit 5): jq: error (at /home/me/library/books/broken.json:1): Cannot index string with string "title"
+dirsql query: failed to load config: table `books`: on-file command failed: command `jq -c '{title, author, year}'` failed (exit 5): jq: error (at /home/me/library/books/broken.json:1): Cannot index string with string "title"
 ```
 
 A file the command cannot parse is the command's to handle — skip it, or

package/docs/howto/parse-files-into-columns.md

@@ -32,15 +32,14 @@ To get them, you need a parser.
 
 ## 2. Attach a parser with `--on-file`
 
-Any program that reads the files named by its arguments and prints a **JSON
-array of row objects** on stdout is a parser. Here is a small one,
+Any program that reads the files named by its arguments and prints **one JSON
+object per line** on stdout is a parser. Here is a small one,
 `extract.py`, that reads each post's frontmatter:
 
 ```python
 #!/usr/bin/env python3
 import json, re, sys
 
-rows = []
 for path in sys.argv[1:]:
     text = open(path, encoding="utf-8").read()
     m = re.match(r"^---\n(.*?)\n---", text, re.DOTALL)
@@ -48,8 +47,7 @@ for path in sys.argv[1:]:
         (k.strip(), v.strip())
         for k, _, v in (line.partition(":") for line in (m.group(1).splitlines() if m else []))
     )
-    rows.append({"title": fields.get("title"), "author": fields.get("author")})
-print(json.dumps(rows))
+    print(json.dumps({"title": fields.get("title"), "author": fields.get("author")}))
 ```
 
 Point the path-table at it with `--on-file`:
@@ -75,7 +73,7 @@ for the full behavior.
 
 One run, one process: on a thousand files the parser starts once, not a
 thousand times, so a Python or Node parser costs its startup once per query.
-That is the shape to write — loop over `sys.argv[1:]`, collect, print once.
+That is the shape to write — loop over `sys.argv[1:]` and print each row as you go.
 
 `--on-file` applies to every path-table in the query and may be given at most
 once. It is a `query`-only flag: there is no config file involved yet, so it is
@@ -121,8 +119,8 @@ row that needs the file's `path` emits it (the parser has the path).
 
 ## The two shapes
 
-Every parser is one of two shapes. The array length is the row count either
-way; what differs is how many files feed it.
+Every parser is one of two shapes. The number of lines printed is the row
+count either way; what differs is how many files feed it.
 
 **One file, many rows.** A lab notebook is a single Markdown file of dated
 `## YYYY-MM-DD` sections, and the question is about the sections. Name the one
@@ -132,12 +130,10 @@ section. `notebook.py`:
 ```python
 import json, re, sys
 
-rows = []
 for path in sys.argv[1:]:
     text = open(path, encoding="utf-8").read()
     for date, body in re.findall(r"^## (\d{4}-\d\d-\d\d)\n(.*?)(?=^## |\Z)", text, re.S | re.M):
-        rows.append({"date": date, "words": len(body.split())})
-print(json.dumps(rows))
+        print(json.dumps({"date": date, "words": len(body.split())}))
 ```
 
 ```bash
@@ -158,14 +154,12 @@ attach it. `messages.py`:
 ```python
 import json, os, sys
 
-rows = []
 for path in sys.argv[1:]:
     project = os.path.relpath(path, "projects").split(os.sep)[0]
     for line in open(path, encoding="utf-8"):
         entry = json.loads(line)
         if entry["type"] in ("user", "assistant"):
-            rows.append({"project": project, "role": entry["type"], "text": entry["message"]["content"]})
-print(json.dumps(rows))
+            print(json.dumps({"project": project, "role": entry["type"], "text": entry["message"]["content"]}))
 ```
 
 ```bash

package/docs/howto/react-to-changes.md

@@ -17,7 +17,7 @@ prints each file's basename:
 name = "files"
 ddl     = "CREATE TABLE files (basename TEXT)"
 glob    = "**/*"
-on-file = '''sh -c 'printf "["; sep=""; for p; do printf "%s{\"basename\":\"%s\"}" "$sep" "${p##*/}"; sep=","; done; printf "]"' sh'''
+on-file = '''sh -c 'for p; do printf "{\"basename\":\"%s\"}\n" "${p##*/}"; done' sh'''
 ```
 
 ## 1. Open the stream

package/docs/howto/skip-files.md

@@ -25,7 +25,7 @@ ignore = ["notes/drafts/**", "**/*.tmp"]
 name = "notes"
 ddl     = "CREATE TABLE notes (basename TEXT)"
 glob    = "notes/**/*"
-on-file = '''sh -c 'printf "["; sep=""; for p; do printf "%s{\"basename\":\"%s\"}" "$sep" "${p##*/}"; sep=","; done; printf "]"' sh'''
+on-file = '''sh -c 'for p; do printf "{\"basename\":\"%s\"}\n" "${p##*/}"; done' sh'''
 ```
 
 Patterns match against root-relative paths, the same way table globs do. An

package/docs/howto/write-a-plugin.md

@@ -121,7 +121,7 @@ embedding. dirsql injects no columns, so the script emits the path itself,
 from the `{root}` the hook passes in and the paths it appends:
 
 ```python
-"""Embed each file's text; print one dirsql row array on stdout."""
+"""Embed each file's text; print one dirsql row per line on stdout."""
 import json
 import os
 import sys
@@ -131,12 +131,9 @@ from model2vec import StaticModel
 root, paths = sys.argv[1], sys.argv[2:]
 texts = [open(path, encoding="utf-8").read() for path in paths]
 model = StaticModel.from_pretrained("minishlab/potion-base-8M")
-rows = [
-    {"path": os.path.relpath(path, root), "text": text,
-     "embedding": json.dumps([round(float(x), 6) for x in vector])}
-    for path, text, vector in zip(paths, texts, model.encode(texts))
-]
-print(json.dumps(rows))
+for path, text, vector in zip(paths, texts, model.encode(texts)):
+    print(json.dumps({"path": os.path.relpath(path, root), "text": text,
+                      "embedding": json.dumps([round(float(x), 6) for x in vector])}))
 ```
 
 The relative `embed.py` above resolves against the fragment directory, which

package/docs/reference/cli.md

@@ -383,8 +383,8 @@ uses**, so behavior is identical to `POST /query` by construction:
 
 Attach a parser to every [path-table](./path-tables.md#parsing-rows-with-on-file)
 in the query: the command runs once per path-table, with every matched file's
-path as a trailing argument, and the JSON array of row objects it prints
-replaces the stat columns:
+path as a trailing argument, and the JSON row objects it prints, one per line,
+replace the stat columns:
 
 ```sh
 dirsql query "SELECT title, author FROM './posts/*.md'" \

package/docs/reference/config.md

@@ -226,7 +226,7 @@ what its required `on-file` command emits — dirsql injects nothing (see
 | `name` | yes | The table's SQL name — the name you query it by. Declared, never derived from `ddl`: dirsql does not read the DDL text. The `ddl` must create a table by this name; if it doesn't, loading fails. |
 | `ddl` | yes | A SQL batch, run verbatim — any number of statements. It must create a table called `name`; that table holds the file rows, and only the columns it declares are kept (keys the `on-file` command emits that are not declared are dropped). The rest of the batch is yours: indexes, virtual tables, triggers. See [Batch `ddl`](#batch-ddl). |
 | `glob` | yes | Glob pattern, [anchored](#glob-anchor) at the config file's directory, under the [one glob rule](#glob-rule): `*` matches one level, `**` any depth. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` is a literal, as in bash and a path-table; the [glob page](./glob.md) has the syntax. |
-| `on-file` | **yes** | A command run once per table, with every matched file's absolute path appended as a trailing argument; its stdout (one JSON array of row objects) is the table's rows. Must be non-empty. A `[[table]]` with no `on-file` is a load error (see [parse errors](#parse-errors)). See [Command hooks](./hooks.md#on-file). |
+| `on-file` | **yes** | A command run once per table, with every matched file's absolute path appended as a trailing argument; its stdout (one JSON object per line) is the table's rows. Must be non-empty. A `[[table]]` with no `on-file` is a load error (see [parse errors](#parse-errors)). See [Command hooks](./hooks.md#on-file). |
 | `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied by the `on-file` output. When `false`, extra keys are dropped and missing columns become `NULL`. |
 
 `on-file` is required because a table's rows come from nowhere else. dirsql
@@ -339,13 +339,15 @@ Two consequences of `ddl` running **once, when the table is created**:
 
 ### `on-file` row mapping
 
-The command prints a JSON array of objects; each object becomes one row.
+The command prints one JSON object per line (NDJSON); each line becomes one
+row. Blank lines are skipped, and a command that prints nothing yields a table
+with no rows. An array on a line is an error, not a list of rows.
 JSON values map to SQLite as: `null` → `NULL`; `true`/`false` → `1`/`0`; an
 integral number → `INTEGER`, any other number → `REAL`; a string → `TEXT`; a
 nested array or object → its JSON text as `TEXT`.
 
 A row's columns are exactly the keys the command emits, narrowed to the DDL;
-dirsql merges nothing else in. Output that is not a JSON array of objects fails
+dirsql merges nothing else in. Output that is not one JSON object per line fails
 the table, and a failed table fails the build (see
 [failure semantics](./hooks.md#failure-semantics)).
 

package/docs/reference/glob.md

@@ -134,6 +134,8 @@ A test runs every row against dirsql as a path-table and, with the leading
 | `./d*/*.js` | none | gitignore |
 | `./dist/bundle.js` | `dist/bundle.js` |  |
 | `./dist/sub/o.js` | `dist/sub/o.js` |  |
+| `./dist` | `dist/bundle.js` | directory-name |
+| `./dist/` | `dist/bundle.js` | directory-name |
 | `./**/*.js` | `node_modules/pkg/index.js` |  |
 | `./node_modules/*/index.js` | `node_modules/pkg/index.js` |  |
 | `./docs/**/*.md` | `docs/api.md`, `docs/guide.md`, `docs/nested/deep.md` |  |

package/docs/reference/hooks.md

@@ -48,18 +48,18 @@ tools like `uvx --with …` / `npx …` resolve their dependencies as usual.
 
 ### stdout protocol
 
-The command's result payload is the **last non-empty line of stdout**,
-trimmed. Any log or chatter lines above it are ignored. A command that
-exits successfully but prints no non-empty line is a failure ("produced no
-output on stdout").
+The command's stdout is **NDJSON: one JSON object per line**, one row each.
+Blank lines are skipped. A command that exits successfully and prints nothing
+yields no rows. Every non-blank line must be a JSON object: a line that is
+an array, a scalar, or log chatter fails the table, and the error names the
+line number. There is no array form.
 
 stderr is never data — it is captured only to enrich error messages (the
-last 2 000 characters are attached to failures).
+last 2 000 characters are attached to failures). Send logs there.
 
-::: tip Print single-line output
-Because only the last non-empty line is the payload, multi-line output loses
-everything above the last line. `jq` users: pass `-c` so the JSON is emitted
-compactly on one line.
+::: tip Print one object per line
+`jq` users: pass `-c` so each object is emitted compactly on one line. Do not
+slurp into an array (`jq -s`, `jq -n '[inputs]'`).
 :::
 
 ### Bounding a hook
@@ -95,8 +95,8 @@ A hook run fails when the command:
 - exits non-zero (the exit code — or `signal`, if killed by one — and the
   stderr tail are reported; a `timeout(1)` wrapper killing an overrun lands
   here),
-- exits zero but prints no non-empty stdout line,
-- or prints output that does not parse as a JSON array of row objects.
+- or prints output where a non-blank line is not a JSON object (an array
+  line is rejected with a message saying to print one object per line).
 
 What a failure *means*: the command ran once for the whole table, so its
 failure is the table's, and a table that cannot be filled fails the build.
@@ -107,23 +107,22 @@ counts as the same kind of failure.
 ## `on-file` contract
 
 Runs **once per table**, with every file matched by the table's `glob`
-appended to the command as a trailing argument, and prints **one JSON array
-of row objects** for the whole table. That is the whole contract. The command
+appended to the command as a trailing argument, and prints **one JSON object
+per line** for the whole table, one row each. That is the whole contract. The command
 reads the files itself; see [`[[table]]`](./config.md#table) for the
 row-mapping rules.
 
 One process over all the files is the shape to write: a loop over the
-arguments that collects rows and prints them once at the end. Parsing is the
+arguments that prints each row as it is parsed. Parsing is the
 command's cost, not dirsql's — dirsql spawns the command, waits, and reads
 its stdout. A table over no matched files spawns nothing.
 
 ```python
 import json, sys
 
-rows = []
 for path in sys.argv[1:]:
-    rows.extend(parse(path))
-print(json.dumps(rows))
+    for row in parse(path):
+        print(json.dumps(row))
 ```
 
 There is no row-to-file attribution: dirsql does not know which file a row
@@ -154,7 +153,7 @@ path anywhere other than last gets a wrapper script. A command that spells
 `{path}` is rejected at startup:
 
 ```
-on-file command `python3 extract.py {path}` uses `{path}`, but an on-file command now runs once per table with every matched path appended as trailing arguments and prints one JSON array of row objects. Remove `{path}` and read the paths from the command's arguments.
+on-file command `python3 extract.py {path}` uses `{path}`, but an on-file command now runs once per table with every matched path appended as trailing arguments and prints one JSON object per line. Remove `{path}` and read the paths from the command's arguments.
 ```
 
 ### Argument-list limits
@@ -162,7 +161,7 @@ on-file command `python3 extract.py {path}` uses `{path}`, but an on-file comman
 Every platform caps the bytes one process may receive as arguments. When a
 table's paths outgrow that cap, dirsql splits them into the fewest
 consecutive runs that fit, runs the command once per run, and concatenates
-the arrays it prints, in order. The command cannot tell: each run is an
+the rows it prints, in order. The command cannot tell: each run is an
 ordinary invocation with a subset of the paths. A command must therefore not
 assume one invocation sees every path — a count, a cross-file join, or a
 dedupe over `sys.argv[1:]` is per run, not per table. Do that work in SQL.

package/docs/reference/path-tables.md

@@ -171,7 +171,7 @@ dirsql query "SELECT title, author FROM './posts/*.md'" \
 ```
 
 The command runs once, with every matched file's absolute path appended as an
-argument, and prints one JSON array of row objects, exactly like a declared
+argument, and prints one JSON object per line, exactly like a declared
 table's [`on-file` hook](/reference/hooks) — same argv splitting, same `{root}`
 placeholder, same [argument-list splitting](/reference/hooks#argument-list-limits)
 over a very large table. Its output *is* the table:
@@ -185,11 +185,10 @@ over a very large table. Its output *is* the table:
   run in — the same directory the `./` path is relative to — and `{root}`
   names it. `python3 extract.py` finds `extract.py` in that directory,
   whatever directory the path-table scans.
-- **A parser failure is the query's failure.** A run that fails (spawn,
-  non-zero exit, or no output) or whose output is not a JSON array of row
-  objects fails the query with the command's stderr tail; there is no partial
-  table, and the CLI exits `1`. A run that succeeds but prints an empty array
-  fails too, because there are no keys to infer columns from:
+- **A parser failure is the query's failure.** A run that fails (spawn
+  or non-zero exit) or whose output is not one JSON object per line fails the
+  query with the command's stderr tail; there is no partial table, and the CLI
+  exits `1`. A run that succeeds but prints nothing fails too, because there are no keys to infer columns from:
 
   ```
   dirsql_parsed: parser produced no rows for `posts/*.md`; cannot infer a schema

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dirsql",
-  "version": "0.4.108",
+  "version": "0.4.110",
   "description": "Ephemeral SQL index over a local directory",
   "license": "MIT",
   "repository": "https://github.com/thekevinscott/dirsql",
@@ -221,10 +221,10 @@
     ]
   },
   "optionalDependencies": {
-    "@dirsql/lib-linux-x64-gnu": "0.4.108",
-    "@dirsql/lib-linux-arm64-gnu": "0.4.108",
-    "@dirsql/lib-darwin-x64": "0.4.108",
-    "@dirsql/lib-darwin-arm64": "0.4.108",
-    "@dirsql/lib-win32-x64-msvc": "0.4.108"
+    "@dirsql/lib-linux-x64-gnu": "0.4.110",
+    "@dirsql/lib-linux-arm64-gnu": "0.4.110",
+    "@dirsql/lib-darwin-x64": "0.4.110",
+    "@dirsql/lib-darwin-arm64": "0.4.110",
+    "@dirsql/lib-win32-x64-msvc": "0.4.110"
   }
 }