dirsql 0.3.128 → 0.4.0

package/docs/getting-started.md

@@ -1,12 +1,12 @@
 # Your first dirsql database
 
 In this tutorial you will turn a directory of three tiny markdown files into
-a SQL database — with a single command, and without writing any code. You
-will:
+a SQL database. You will:
 
 1. Create the directory and files.
-2. Query them straight away with zero configuration.
-3. Declare your own table to name and reuse a shape, and query it.
+2. Query them straight away with zero configuration and no code.
+3. Declare your own named table — with a tiny parser that pulls a column out
+   of each file — and query it.
 
 It takes about five minutes.
 
@@ -140,25 +140,44 @@ A declared table fixes a shape once: you give it a name, scope it to exactly
 the files you care about, and then query it by name instead of repeating a
 path in every question. It is also the on-ramp to everything a path-table
 can't do — a named table can be kept live by the watcher, persisted across
-restarts, and given a parser that reads inside your files.
+restarts, and given a parser that reads *inside* your files.
 
-Still inside `my-notes`, create a `.dirsql.toml`:
+That parser is the point: a named table's columns are exactly what its
+`on-file` command emits. `dirsql` adds nothing on its own — so this is where
+you pull a value out of each file. Still inside `my-notes`, write a tiny
+parser that reads a note's title line (the `# Heading`) and its author (the
+folder name), and prints them as a JSON row:
+
+```bash
+cat > note.sh <<'EOF'
+#!/usr/bin/env sh
+title=$(sed -n 's/^# //p' "$1" | head -n1)
+author=$(basename "$(dirname "$1")")
+printf '[{"title":%s,"author":%s}]' \
+  "$(jq -Rn --arg t "$title" '$t')" \
+  "$(jq -Rn --arg a "$author" '$a')"
+EOF
+```
+
+Now create a `.dirsql.toml` that points a table at it:
 
 ```bash
 cat > .dirsql.toml <<'EOF'
 [[table]]
-ddl  = "CREATE TABLE notes (dir TEXT, basename TEXT, size INTEGER)"
-glob = "notes/**/*.md"
+ddl     = "CREATE TABLE notes (title TEXT, author TEXT)"
+glob    = "notes/**/*.md"
+on-file = "sh note.sh {path}"
 EOF
 ```
 
-Two keys define the table:
+Three keys define the table:
 
 - `glob` selects which files feed the table — every `.md` at any depth under
   `notes/`, relative to the directory the config sits in.
 - `ddl` is ordinary `CREATE TABLE` SQL naming the columns you want to keep.
-  Each is a [stat column](./reference/columns.md#stat-columns) `dirsql`
-  computes for every file.
+- `on-file` is the command run once per matched file; `{path}` is the file's
+  path, and its printed JSON row becomes the file's row. The columns are
+  exactly what it emits — `title` from the heading, `author` from the folder.
 
 ## 5. Query the table
 
@@ -168,11 +187,11 @@ pass it explicitly with `-c`, **after** the SQL:
 ::: code-group
 
 ```bash [npm]
-npx dirsql "SELECT dir, basename, size FROM notes ORDER BY dir, basename" -c .dirsql.toml | jq
+npx dirsql "SELECT title, author FROM notes ORDER BY author, title" -c .dirsql.toml | jq
 ```
 
 ```bash [PyPI]
-uvx dirsql "SELECT dir, basename, size FROM notes ORDER BY dir, basename" -c .dirsql.toml | jq
+uvx dirsql "SELECT title, author FROM notes ORDER BY author, title" -c .dirsql.toml | jq
 ```
 
 :::
@@ -180,35 +199,33 @@ uvx dirsql "SELECT dir, basename, size FROM notes ORDER BY dir, basename" -c .di
 ```json
 [
   {
-    "basename": "ideas.md",
-    "dir": "notes/alice",
-    "size": 52
+    "author": "alice",
+    "title": "Ideas"
   },
   {
-    "basename": "welcome.md",
-    "dir": "notes/alice",
-    "size": 66
+    "author": "alice",
+    "title": "Welcome"
   },
   {
-    "basename": "reading-list.md",
-    "dir": "notes/bob",
-    "size": 41
+    "author": "bob",
+    "title": "Reading list"
   }
 ]
 ```
 
-You queried `FROM notes` by name — no path, no glob to repeat. And because
-`dir` is a real SQL column, you can aggregate on it. Count each author's
-notes by their folder:
+You queried `FROM notes` by name — no path, no glob to repeat — and `title`
+came from *inside* each file, something a path-table can't reach. Because
+`author` is a real SQL column, you can aggregate on it. Count each author's
+notes:
 
 ::: code-group
 
 ```bash [npm]
-npx dirsql query "SELECT dir, COUNT(*) AS notes FROM notes GROUP BY dir ORDER BY dir" -c .dirsql.toml | jq
+npx dirsql query "SELECT author, COUNT(*) AS notes FROM notes GROUP BY author ORDER BY author" -c .dirsql.toml | jq
 ```
 
 ```bash [PyPI]
-uvx dirsql query "SELECT dir, COUNT(*) AS notes FROM notes GROUP BY dir ORDER BY dir" -c .dirsql.toml | jq
+uvx dirsql query "SELECT author, COUNT(*) AS notes FROM notes GROUP BY author ORDER BY author" -c .dirsql.toml | jq
 ```
 
 :::
@@ -216,11 +233,11 @@ uvx dirsql query "SELECT dir, COUNT(*) AS notes FROM notes GROUP BY dir ORDER BY
 ```json
 [
   {
-    "dir": "notes/alice",
+    "author": "alice",
     "notes": 2
   },
   {
-    "dir": "notes/bob",
+    "author": "bob",
     "notes": 1
   }
 ]
@@ -234,7 +251,7 @@ configuration, and a declared table when you want a named shape to reuse.
 - [Query files without a config](./howto/query-without-config.md) — more
   path-table questions you can ask with no setup at all.
 - [Define tables for your files](./howto/define-tables.md) — the full
-  `[[table]]` recipe: multiple tables, ignore patterns.
+  `[[table]]` recipe: multiple tables, each with its own `on-file` parser.
 - [Extract rows from file contents](./howto/extract-from-contents.md) —
   pull columns out of *inside* your files with an `on-file` parser.
 - [CLI](./reference/cli.md) — every flag, plus running `dirsql` as a

package/docs/howto/columns-from-paths.md

@@ -1,10 +1,11 @@
 # Derive columns from file paths
 
 Directory layouts often encode real data — an author, a year, a thread ID —
-as path segments. A `{name}` capture in a table's glob turns such a segment
-into a queryable column, no extraction code required.
+as path segments. An [`on-file`](../reference/config.md#table) hook receives
+each file's path and can split it into columns, so a query can group and
+filter on those segments.
 
-## 1. Name the segment in the glob
+## 1. Split the path in a hook
 
 Suppose photos are filed by year and month:
 
@@ -14,20 +15,35 @@ photos/2024/11/hike.jpg
 photos/2025/01/snow.jpg
 ```
 
-Capture both directory levels in `.dirsql.toml`:
+Write a small parser, `pathcols.py`, that turns the path into a row. It
+receives the file's absolute path as its argument and prints a JSON array of
+row objects:
+
+```python
+#!/usr/bin/env python3
+import json, os, sys
+
+parts = sys.argv[1].split(os.sep)
+# .../photos/<year>/<month>/<file>
+print(json.dumps([{"year": parts[-3], "month": parts[-2],
+                   "basename": os.path.basename(sys.argv[1])}]))
+```
+
+Point a table at it in `.dirsql.toml`:
 
 ```toml
 [[table]]
-ddl  = "CREATE TABLE photos (year TEXT, month TEXT, basename TEXT)"
-glob = "photos/{year}/{month}/*.jpg"
+ddl     = "CREATE TABLE photos (year TEXT, month TEXT, basename TEXT)"
+glob    = "photos/*/*/*.jpg"
+on-file = "python3 pathcols.py {path}"
 ```
 
-A capture only populates a column when the DDL declares one with the same
-name — here `year` and `month`. The capture rules (valid names, matching
-within one path segment) are in
-[glob captures](../reference/columns.md#glob-captures).
+The hook emits every column the table has — dirsql injects nothing. `{path}`
+is the matched file's absolute path, one of the placeholders in the
+[command hook contract](../reference/hooks.md#on-file). A column appears only
+because the DDL declares it *and* the hook emits it.
 
-## 2. Query the captured columns
+## 2. Query the derived columns
 
 Pass the config with [`-c`](../reference/cli.md#flags) (`dirsql` does not
 auto-load a `.dirsql.toml` from the current directory):
@@ -40,7 +56,7 @@ dirsql query "SELECT year, month, basename FROM photos ORDER BY year, month" -c
 [{"basename":"beach.jpg","month":"05","year":"2024"},{"basename":"hike.jpg","month":"11","year":"2024"},{"basename":"snow.jpg","month":"01","year":"2025"}]
 ```
 
-Captures are real SQL columns, so aggregation works:
+They are real SQL columns, so aggregation works:
 
 ```bash
 dirsql query "SELECT year, COUNT(*) AS photos FROM photos GROUP BY year" -c ./.dirsql.toml
@@ -52,9 +68,10 @@ dirsql query "SELECT year, COUNT(*) AS photos FROM photos GROUP BY year" -c ./.d
 
 ## Going further
 
-- Captures combine freely with [stat columns](../reference/columns.md#stat-columns)
-  (`basename` above) — both are filesystem facts merged onto every row.
-- The [tutorial](../getting-started.md) walks the same idea with an
-  `{author}` capture, starting from zero.
+- Only need the plain filesystem stat columns (`path`, `basename`, `dir`,
+  `ext`, `size`, …) with no code? Query the path directly — a
+  [path-table](../reference/path-tables.md) gives them for free, no config.
+- The [tutorial](../getting-started.md) walks the same idea, deriving an
+  author from the folder name, starting from zero.
 - When the value you need lives inside the file rather than in its path,
   see [Extract rows from file contents](./extract-from-contents.md).

package/docs/howto/define-tables.md

@@ -1,28 +1,44 @@
 # Define tables for your files
 
 Map a glob of files to a named SQL table so you query exactly the files you
-care about, with exactly the columns you care about — instead of the
-ad-hoc [path-tables](../reference/path-tables.md)
-[configless mode](../reference/cli.md#configless-mode) leaves you with.
+care about, by name — a shape you can reuse, keep live with the watcher, and
+persist across restarts, instead of repeating an ad-hoc
+[path-table](../reference/path-tables.md) path in every query.
 
-## 1. Create a config next to your files
+## 1. Create a config with a table
 
-Suppose your blog posts live under `posts/`, one markdown file each. In the
-directory you want to index, create a `.dirsql.toml` with one
-[`[[table]]`](../reference/config.md#table) entry:
+Suppose your blog posts live under `posts/`, one markdown file each. A named
+table needs three keys: a `glob` that selects the files, a `ddl` that names
+the columns, and an [`on-file`](../reference/config.md#table) hook that emits
+each file's rows. Put a small parser next to the config — `extract.py`, which
+reads a post's title line and prints a JSON array of row objects:
+
+```python
+#!/usr/bin/env python3
+import json, os, sys
+
+text = open(sys.argv[1], encoding="utf-8").read()
+title = next((l[2:].strip() for l in text.splitlines() if l.startswith("# ")), None)
+print(json.dumps([{"title": title, "slug": os.path.basename(sys.argv[1])[:-3]}]))
+```
+
+Then declare the table in `.dirsql.toml`:
 
 ```toml
 [[table]]
-ddl  = "CREATE TABLE posts (path TEXT, size INTEGER, mtime INTEGER)"
-glob = "posts/**/*.md"
+ddl     = "CREATE TABLE posts (title TEXT, slug TEXT)"
+glob    = "posts/**/*.md"
+on-file = "python3 extract.py {path}"
 ```
 
-- `glob` selects the files: every `.md` under `posts/`, at any depth,
-  relative to the directory containing the config.
-- `ddl` is a plain SQLite `CREATE TABLE` naming the columns you want. Here
-  all three are [stat columns](../reference/columns.md#stat-columns) —
-  filesystem facts `dirsql` computes for every file. Facts are opt-in by
-  DDL: only the ones you declare become columns.
+- `glob` selects the files: every `.md` under `posts/`, at any depth, relative
+  to the directory containing the config.
+- `ddl` is a plain SQLite `CREATE TABLE` naming the columns you want to keep.
+- `on-file` is **required** — it is where the table's rows come from. dirsql
+  injects nothing; the hook emits every column, reading the file (it has
+  `{path}`) and deriving whatever it needs. A `[[table]]` with no `on-file` is
+  a [config error](../reference/config.md#parse-errors). For plain stat
+  columns with no code, query the path directly with a path-table instead.
 
 ## 2. Query the table
 
@@ -31,11 +47,11 @@ auto-load a `.dirsql.toml` from the current directory. Each matched file is
 one row:
 
 ```bash
-dirsql query "SELECT path, size FROM posts ORDER BY path" -c ./.dirsql.toml
+dirsql query "SELECT title, slug FROM posts ORDER BY slug" -c ./.dirsql.toml
 ```
 
 ```json
-[{"path":"posts/2024/hello.md","size":21},{"path":"posts/2025/again.md","size":55}]
+[{"slug":"again","title":"On Recursion"},{"slug":"hello","title":"Hello World"}]
 ```
 
 Files that don't match the glob (a `README.txt` next to `posts/`, say) are
@@ -43,16 +59,18 @@ simply not in the table. Only the tables you define are served.
 
 ## Multiple tables
 
-Add one `[[table]]` entry per table. When a file matches several globs, it
-populates every matching table — each table is an independent view. See
-[`[[table]]`](../reference/config.md#table) for that and the remaining keys
-(`strict`, `on-file`).
+Add one `[[table]]` entry per table — each with its own `glob`, `ddl`, and
+`on-file`. When a file matches several globs, it populates every matching
+table — each table is an independent view. See
+[`[[table]]`](../reference/config.md#table) for the remaining key, `strict`.
 
 ## Going further
 
-- Your directory layout encodes data (authors, dates, IDs)? Capture path
-  segments as columns — [Derive columns from file paths](./columns-from-paths.md).
-- Need columns from *inside* the files? A plain table never reads file
-  contents — [Extract rows from file contents](./extract-from-contents.md).
+- The parser mechanics — placeholders, stdout protocol, per-file failure
+  isolation — are the [`on-file` hook contract](../reference/hooks.md#on-file);
+  [Extract rows from file contents](./extract-from-contents.md) is the fuller
+  recipe.
+- Your directory layout encodes data (authors, dates, IDs)? Split the path in
+  the hook — [Derive columns from file paths](./columns-from-paths.md).
 - Why one row per file, rebuilt from disk? See
   [how `dirsql` thinks](../explanation.md).

package/docs/howto/embed.md

@@ -34,22 +34,25 @@ comments/t2/c1.json    # {"body": "following up", "author": "alice"}
 ```
 
 Unlike a config-file table, a programmatic table takes an `on_file`
-callback — your code reads each matched file and returns its rows, with
-[glob captures and stat columns](../reference/columns.md) merged on
-automatically (here, `{thread}` from the path):
+callback — your code reads each matched file and returns its rows. The row's
+columns are exactly what the callback returns; `dirsql` merges nothing on top
+(see [Columns](../reference/columns.md)), so the callback derives the `thread`
+from the file's path itself:
 
 ::: code-group
 
 ```python [Python]
 import asyncio
 import json
+import os
 
 from dirsql import DirSQL, Table
 
 
 def on_file(path: str) -> list[dict]:
+    thread = os.path.basename(os.path.dirname(path))
     with open(path, encoding="utf-8") as f:
-        return [json.load(f)]
+        return [{**json.load(f), "thread": thread}]
 
 
 async def main() -> None:
@@ -58,7 +61,7 @@ async def main() -> None:
         tables=[
             Table(
                 ddl="CREATE TABLE comments (thread TEXT, author TEXT, body TEXT)",
-                glob="comments/{thread}/*.json",
+                glob="comments/*/*.json",
                 on_file=on_file,
             )
         ],
@@ -73,6 +76,7 @@ asyncio.run(main())
 
 ```typescript [TypeScript]
 import { readFileSync } from "node:fs";
+import { basename, dirname } from "node:path";
 import { DirSQL } from "dirsql";
 
 const db = new DirSQL({
@@ -80,8 +84,10 @@ const db = new DirSQL({
   tables: [
     {
       ddl: "CREATE TABLE comments (thread TEXT, author TEXT, body TEXT)",
-      glob: "comments/{thread}/*.json",
-      onFile: (path) => [JSON.parse(readFileSync(path, "utf8"))],
+      glob: "comments/*/*.json",
+      onFile: (path) => [
+        { ...JSON.parse(readFileSync(path, "utf8")), thread: basename(dirname(path)) },
+      ],
     },
   ],
 });
@@ -110,6 +116,12 @@ fn on_file(path: &str) -> Vec<HashMap<String, Value>> {
             row.insert(key, Value::Text(s));
         }
     }
+    let thread = std::path::Path::new(path)
+        .parent()
+        .and_then(|p| p.file_name())
+        .and_then(|n| n.to_str())
+        .unwrap_or_default();
+    row.insert("thread".into(), Value::Text(thread.into()));
     vec![row]
 }
 
@@ -118,7 +130,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
         .root("./comments-root")
         .table(Table::new(
             "CREATE TABLE comments (thread TEXT, author TEXT, body TEXT)",
-            "comments/{thread}/*.json",
+            "comments/*/*.json",
             on_file,
         ))
         .build()?;

package/docs/howto/extract-from-contents.md

@@ -18,7 +18,7 @@ stdout works. With [`jq`](https://jqlang.org/):
 
 ```toml
 [[table]]
-ddl     = "CREATE TABLE books (title TEXT, author TEXT, year INTEGER, path TEXT)"
+ddl     = "CREATE TABLE books (title TEXT, author TEXT, year INTEGER)"
 glob    = "books/*.json"
 on-file = "jq -c '[{title, author, year}]' {path}"
 ```
@@ -35,17 +35,16 @@ Pass the config with [`-c`](../reference/cli.md#flags) (`dirsql` does not
 auto-load a `.dirsql.toml` from the current directory):
 
 ```bash
-dirsql query "SELECT title, author, year, path FROM books ORDER BY year" -c ./.dirsql.toml
+dirsql query "SELECT title, author, year FROM books ORDER BY year" -c ./.dirsql.toml
 ```
 
 ```json
-[{"path":"books/bleak-house.json","author":"Charles Dickens","title":"Bleak House","year":1852},{"path":"books/middlemarch.json","author":"George Eliot","title":"Middlemarch","year":1871}]
+[{"author":"Charles Dickens","title":"Bleak House","year":1852},{"author":"George Eliot","title":"Middlemarch","year":1871}]
 ```
 
-Filesystem facts are still merged onto every row — `path` above comes from
-`dirsql`, not from `jq`. When the command emits a key that collides with a
-fact, the command wins
-([precedence](../reference/columns.md#precedence)).
+The table's columns are exactly what the command emits, narrowed to the DDL —
+`dirsql` adds nothing. To include the file's `path`, have the command emit it
+(it has `{path}`); dirsql will not merge it in for you.
 
 ## Multiple rows per file
 
@@ -54,7 +53,7 @@ row per line, slurp it:
 
 ```toml
 [[table]]
-ddl     = "CREATE TABLE events (event TEXT, user TEXT, path TEXT)"
+ddl     = "CREATE TABLE events (event TEXT, user TEXT)"
 glob    = "logs/*.jsonl"
 on-file = "jq -c -s '.' {path}"
 ```

package/docs/howto/parse-files-into-columns.md

@@ -106,10 +106,9 @@ Same command, same rows. The difference is what a declared table brings: it is
 indexed on build, kept fresh by the watcher, survives restarts with
 [`--persist`](./persist.md), and a config can declare
 [many tables](./define-tables.md) — each with its own `on-file` — where the flag
-gives every path-table one parser. A declared table also merges the filesystem
-facts back onto each row, so a `path TEXT` column in the DDL is populated by
-`dirsql` even though the parser did not emit it
-([precedence](../reference/columns.md#precedence)).
+gives every path-table one parser. In both spellings the table's columns are
+exactly what the parser emits: `dirsql` merges no filesystem facts back on. A
+row that needs the file's `path` emits it (the parser has `{path}`).
 
 ## Going further
 

package/docs/howto/react-to-changes.md

@@ -7,13 +7,16 @@ side.
 
 Row events are emitted for **named tables**, so this flow needs a config —
 [path-tables](../reference/path-tables.md) are scanned per query and are not
-watched. Define one next to your files:
+watched. Define one next to your files. A named table's columns are whatever
+its [`on-file`](../reference/config.md#table) hook emits; here a minimal hook
+prints each file's basename:
 
 ```toml
 # .dirsql.toml
 [[table]]
-ddl  = "CREATE TABLE files (path TEXT, basename TEXT, dir TEXT, ext TEXT, size INTEGER, mtime INTEGER, ctime INTEGER)"
-glob = "**/*"
+ddl     = "CREATE TABLE files (basename TEXT)"
+glob    = "**/*"
+on-file = '''sh -c 'printf "[{\"basename\":\"%s\"}]" "${1##*/}"' sh {path}'''
 ```
 
 ## 1. Open the stream
@@ -44,7 +47,7 @@ The stream delivers the resulting row change:
 
 ```
 event: row
-data: {"action":"insert","file_path":"inbox/two.txt","old_row":null,"row":{"basename":"two.txt","ctime":1783170226,"dir":"inbox","ext":"txt","mtime":1783170226,"path":"inbox/two.txt","size":7},"table":"files"}
+data: {"action":"insert","file_path":"inbox/two.txt","old_row":null,"row":{"basename":"two.txt"},"table":"files"}
 ```
 
 Edits arrive as `update` events carrying both the old and new row;

package/docs/howto/search-by-meaning.md

@@ -41,21 +41,25 @@ notes/tomatoes.md   # planting tomato seedlings after the last frost
 ```
 
 Next to them, `embed.py` turns one file into one row carrying its text and
-its embedding (a JSON array, stored as TEXT — `sqlite-vec` accepts JSON
-vectors directly):
+its `path`, text, and embedding (a JSON array, stored as TEXT — `sqlite-vec`
+accepts JSON vectors directly). dirsql injects no columns, so the script emits
+the path itself, deriving it from the `{path}`/`{root}` the hook passes in:
 
 ```python
 """Embed one file's text; print a dirsql row array on stdout."""
 import json
+import os
 import sys
 
 from model2vec import StaticModel
 
-path = sys.argv[1]
+path, root = sys.argv[1], sys.argv[2]
 text = open(path, encoding="utf-8").read()
 model = StaticModel.from_pretrained("minishlab/potion-base-8M")
 vector = model.encode([text])[0]
-print(json.dumps([{"text": text, "embedding": json.dumps([round(float(x), 6) for x in vector])}]))
+row = {"path": os.path.relpath(path, root), "text": text,
+       "embedding": json.dumps([round(float(x), 6) for x in vector])}
+print(json.dumps([row]))
 ```
 
 And `search.py` turns a `{"q": "..."}` request body into SQL, embedding the
@@ -98,7 +102,7 @@ entrypoint = "sqlite3_vec_init"
 [[table]]
 ddl     = "CREATE TABLE notes (path TEXT, text TEXT, embedding TEXT)"
 glob    = "notes/*.md"
-on-file = "uv run --with model2vec python embed.py {path}"
+on-file = "uv run --with model2vec python embed.py {path} {root}"
 ```
 
 The extension is named by package: the Python launcher resolves the

package/docs/howto/skip-files.md

@@ -22,8 +22,9 @@ Exclude the noise in `.dirsql.toml`:
 ignore = ["notes/drafts/**", "**/*.tmp"]
 
 [[table]]
-ddl  = "CREATE TABLE notes (path TEXT)"
-glob = "notes/**/*"
+ddl     = "CREATE TABLE notes (basename TEXT)"
+glob    = "notes/**/*"
+on-file = '''sh -c 'printf "[{\"basename\":\"%s\"}]" "${1##*/}"' sh {path}'''
 ```
 
 Patterns match against root-relative paths, the same way table globs do. An
@@ -35,11 +36,11 @@ Pass the config with [`-c`](../reference/cli.md#flags) (`dirsql` does not
 auto-load a `.dirsql.toml` from the current directory):
 
 ```bash
-dirsql query "SELECT path FROM notes ORDER BY path" -c ./.dirsql.toml
+dirsql query "SELECT basename FROM notes ORDER BY basename" -c ./.dirsql.toml
 ```
 
 ```json
-[{"path":"notes/final.md"}]
+[{"basename":"final.md"}]
 ```
 
 ## Notes

package/docs/howto/write-a-plugin.md

@@ -72,7 +72,7 @@ Both hook command styles from the [hook contract](../reference/hooks.md) work
 in a plugin fragment:
 
 - **Console scripts** — a `bin`-style entry point your package installs on
-  `PATH` (`embed-file {path}`). Recommended for published plugins: the command
+  `PATH` (`embed-file {path} {root}`). Recommended for published plugins: the command
   is bound to your package's interpreter and dependencies, and it is
   language-neutral (the fragment names a command, not a Python file).
 - **Relative scripts** — a path resolved against the fragment's own directory
@@ -116,23 +116,28 @@ entrypoint = "sqlite3_vec_init"
 [[table]]
 ddl     = "CREATE TABLE notes (path TEXT, text TEXT, embedding TEXT)"
 glob    = "notes/*.md"
-on-file = "uv run --with model2vec python embed.py {path}"
+on-file = "uv run --with model2vec python embed.py {path} {root}"
 ```
 
-`embed.py` turns one file into one row carrying its text and its embedding:
+`embed.py` turns one file into one row carrying its path, text, and embedding.
+dirsql injects no columns, so the script emits the path itself, from the
+`{path}`/`{root}` the hook passes in:
 
 ```python
 """Embed one file's text; print a dirsql row array on stdout."""
 import json
+import os
 import sys
 
 from model2vec import StaticModel
 
-path = sys.argv[1]
+path, root = sys.argv[1], sys.argv[2]
 text = open(path, encoding="utf-8").read()
 model = StaticModel.from_pretrained("minishlab/potion-base-8M")
 vector = model.encode([text])[0]
-print(json.dumps([{"text": text, "embedding": json.dumps([round(float(x), 6) for x in vector])}]))
+row = {"path": os.path.relpath(path, root), "text": text,
+       "embedding": json.dumps([round(float(x), 6) for x in vector])}
+print(json.dumps([row]))
 ```
 
 `search.py` turns a `{"q": "..."}` request body into nearest-neighbor SQL:

package/docs/reference/columns.md

@@ -1,31 +1,37 @@
-# Stat columns and glob captures
+# Columns
 
-Every table — config-defined or programmatic — gets filesystem facts merged
-onto its rows automatically: seven **stat columns** derived from the file's
-path and stat metadata, plus one column per **`{name}` capture** in the
-table's glob.
+Where a dirsql table's columns come from depends on which kind of table it is.
+The rule that governs both is the same: **dirsql never injects a column your
+table did not produce.** There is no automatic path/size/mtime merge, and glob
+`{name}` segments capture nothing.
 
-These are ordinary, physically stored `TEXT`/`INTEGER` columns, computed
-once per file at scan time and written like any other column value — not
-SQLite's `GENERATED ... VIRTUAL` columns (computed on the fly, never stored)
-and not part of a `CREATE VIRTUAL TABLE` (dirsql tables are always real
-tables). "Stat" describes where the value comes from — the file's path and
-`stat` metadata, as opposed to its content — not how it's stored.
+## Named tables: exactly the hook's output
 
-Facts are **opt-in by DDL**: only facts whose name appears as a column in
-the table's `CREATE TABLE` are populated; the rest are silently dropped.
-Declaring them requires nothing else. The names below aren't a protected or
-enforced namespace — nothing stops you from declaring a column with one of
-these names for an unrelated purpose, in which case dirsql's computed value
-lands there like any other fact (unless your own row source — an `on-file`
-command or SDK `on_file` callback — supplies its own value for that name;
-see [Precedence](#precedence)).
+A named [`[[table]]`](./config.md#table) — or an SDK
+[`Table`](./sdk.md#table) — has exactly the columns its
+[`on-file` hook](./hooks.md#on-file) emits, narrowed to the columns the DDL
+declares. dirsql adds nothing on top: no `path`, no `size`, no value derived
+from the filename. A hook that wants any of those computes them itself. The
+hook receives the file's `{path}` (an SDK `on_file` callback receives the same
+path as its argument) and may stat or read the file however it likes.
 
-## Stat columns
+The hook is **required**. A `[[table]]` with no `on-file` is a
+[config-load error](./config.md#parse-errors): with nothing supplying columns,
+every row would be all-NULL. The error points at the fix — add a hook, or, for
+plain stat columns with no code, query the path directly with a path-table.
+
+## Path-tables: stat columns
+
+A [path-table](./path-tables.md) (`FROM './'`) is the one place dirsql supplies
+columns for you. Each matched file becomes one row carrying seven **stat
+columns** — derived from the file's path and its `stat` metadata — plus a
+lazily-read hidden [`content`](./path-tables.md#columns) column.
+
+### Stat columns
 
 | Column | Type | Value |
 |---|---|---|
-| `path` | TEXT | The file's path relative to the scan root (e.g. `posts/hello.md`). |
+| `path` | TEXT | The file's path relative to the scan root (e.g. `posts/hello.md`); absolute for a `/`, `../` or `~/` path-table. |
 | `basename` | TEXT | The filename, including extension (`hello.md`). |
 | `dir` | TEXT | The parent directory relative to the root (`posts`); the empty string for files directly under the root. |
 | `ext` | TEXT | The file extension without the leading dot (`md`). Original case is preserved — `Photo.JPG` yields `JPG`; use `LOWER(ext)` for case-insensitive matching. `NULL` when the file has no extension. |
@@ -33,48 +39,32 @@ see [Precedence](#precedence)).
 | `mtime` | INTEGER | Last-modified time, Unix seconds. |
 | `ctime` | INTEGER | Creation (birth) time, Unix seconds. `NULL` when the platform or filesystem cannot supply it. |
 
-A fact that cannot be computed (an unreadable file's `size`/`mtime`/
-`ctime`, a missing extension's `ext`) is absent from the row: `NULL` in
-the default relaxed mode, a missing-column error for a
-[`strict`](./config.md#table) table that declares it.
+These are ordinary stored `TEXT`/`INTEGER` values, computed once per file at
+scan time — not SQLite `GENERATED ... VIRTUAL` columns and not part of a
+`CREATE VIRTUAL TABLE`. "Stat" describes where the value comes from — the
+file's path and `stat` metadata, as opposed to its content — not how it is
+stored.
+
+A stat value that cannot be computed (an unreadable file's `size`, a missing
+extension's `ext`) is `NULL`.
 
 ```sql
 SELECT basename, size
-FROM posts
+FROM './posts'
 WHERE mtime > strftime('%s', '2024-01-01')
 ORDER BY mtime DESC;
 ```
 
-## Glob captures
-
-A `{name}` segment in a table's glob captures part of each matched path as
-a TEXT column named `name`:
-
-```toml
-[[table]]
-ddl  = "CREATE TABLE comments (thread_id TEXT, basename TEXT, mtime INTEGER)"
-glob = "_comments/{thread_id}/*.jsonl"
-```
-
-A file at `_comments/abc123/2024-05-05.jsonl` produces a row with
-`thread_id = "abc123"`.
-
-- A capture name must be a valid identifier: a letter or underscore
-  followed by letters, digits, or underscores (`[a-zA-Z_][a-zA-Z0-9_]*`).
-- A capture matches **within a single path segment** — one or more
-  characters, never a `/`. For matching purposes, `{name}` behaves like
-  `*`.
-- A glob may contain multiple captures (`{year}/{month}/*.jpg`).
-- Like stat columns, a capture populates a column only when the DDL
-  declares a column of the same name.
-
-## Precedence
+[Attaching a parser](./path-tables.md#parsing-rows-with-on-file) to a
+path-table with `--on-file` **replaces** these stat columns with the parser's
+own output — the two modes stay cleanly separate, exactly as for a named
+table. A parser that wants the path emits it; it has `{path}`.
 
-Values produced by a table's own row source — an `on-file` command's JSON
-output or an SDK `on_file` callback's return value — **win** over
-auto-injected facts of the same name. An `on_file` callback that explicitly emits
-`path` is honored.
+## Deriving columns from the path
 
-Injection order per row: stat columns first, then glob captures, then
-the row source's own values, each layer overwriting the previous, all
-filtered to the DDL's declared columns.
+To turn path segments (an author, a year, a thread id) into columns, a hook
+splits `{path}` and emits the pieces — the same as any other column it
+produces. dirsql does not do this for you: a `{name}` segment in a glob is
+rewritten to `*` and matches a single path segment, but captures no value.
+[Derive columns from file paths](../howto/columns-from-paths.md) walks a
+worked example.

package/docs/reference/config.md

@@ -90,28 +90,29 @@ rejected; call the extension's functions in queries instead.
 
 ## `[[table]]`
 
-Each entry maps a glob pattern to a SQL table. Every matched file produces
-rows whose columns come from filesystem facts — [glob captures and virtual
-columns](./columns.md) — plus, when `on-file` is set, the output of a
-per-file command.
+Each entry maps a glob pattern to a SQL table. A table's columns are exactly
+what its required `on-file` command emits — dirsql injects nothing (see
+[Columns](./columns.md)).
 
 | Key | Required | Description |
 |---|---|---|
-| `ddl` | yes | A SQLite `CREATE TABLE` statement. The table name is parsed from it. Only columns declared here are populated; auto-injected facts not in the DDL are dropped. |
-| `glob` | yes | Glob pattern matched against root-relative paths. May contain `{name}` [capture segments](./columns.md#glob-captures). Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. |
-| `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied (by the command/on-file output, a glob capture, or a stat column). When `false`, extra keys are dropped and missing columns become `NULL`. |
-| `on-file` | no | A command run once per matched file; its stdout (a JSON array of row objects) becomes the file's rows. Must be non-empty. See [Command hooks](./hooks.md#on-file). |
-
-Without `on-file`, a table produces exactly one row per matched file, built
-entirely from filesystem facts. Content interpretation (frontmatter, JSON
-fields, CSV parsing) is out of scope for plain config tables — use
-`on-file`, or a programmatic [SDK table](./sdk.md#table) with an `on_file`
-callback.
+| `ddl` | yes | A SQLite `CREATE TABLE` statement. The table name is parsed from it. Only the columns declared here are kept; keys the `on-file` command emits that are not declared are dropped. |
+| `glob` | yes | Glob pattern matched against root-relative paths. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). |
+| `on-file` | **yes** | A command run once per matched file; its stdout (a JSON array of row objects) becomes the file's rows. Must be non-empty. A `[[table]]` with no `on-file` is a load error (see [parse errors](#parse-errors)). See [Command hooks](./hooks.md#on-file). |
+| `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied by the `on-file` output. When `false`, extra keys are dropped and missing columns become `NULL`. |
+
+`on-file` is required because a table's rows come from nowhere else. dirsql
+does not read file contents or merge filesystem facts on your behalf: the
+command reads the file (it receives `{path}`) and prints the rows, and those
+rows — filtered to the DDL — are the table. For plain stat columns with no
+command, query the path directly with a [path-table](./path-tables.md)
+instead of declaring a table.
 
 ```toml
 [[table]]
-ddl  = "CREATE TABLE comments (path TEXT, basename TEXT, mtime INTEGER)"
-glob = "_comments/*/*.jsonl"
+ddl     = "CREATE TABLE comments (path TEXT, author TEXT, body TEXT)"
+glob    = "_comments/*/*.jsonl"
+on-file = "jq -c -s '.' {path}"
 
 [[table]]
 ddl     = "CREATE TABLE papers (paper_id TEXT, title TEXT)"
@@ -127,10 +128,10 @@ JSON values map to SQLite as: `null` → `NULL`; `true`/`false` → `1`/`0`; an
 integral number → `INTEGER`, any other number → `REAL`; a string → `TEXT`; a
 nested array or object → its JSON text as `TEXT`.
 
-Filesystem facts are still merged onto every `on-file` row; a column emitted
-by the command wins over a same-named fact. Output that is not a JSON array
-of objects is a per-file failure: the file is skipped with a stderr warning
-and the scan continues (see [failure semantics](./hooks.md#failure-semantics)).
+A row's columns are exactly the keys the command emits, narrowed to the DDL;
+dirsql merges nothing else in. Output that is not a JSON array of objects is a
+per-file failure: the file is skipped with a stderr warning and the scan
+continues (see [failure semantics](./hooks.md#failure-semantics)).
 
 ## Composing multiple configs
 
@@ -174,8 +175,13 @@ SDKs raise/reject) when:
 - Any table contains an unknown key (top level, `[dirsql]`, `[[table]]`, or
   `[[dirsql.extension]]`). The error names the offending key.
 - A `[[table]]` entry omits `ddl` or `glob`.
+- A `[[table]]` entry omits `on-file` (or it is empty/whitespace). The error
+  names the offending glob and points at the fix:
+
+  > `[[table]] '**/*.md' has no on-file hook, so every row would be all-NULL. Add an `on-file` hook that emits the columns, or, for stat columns with no code, query the path directly: `FROM './'``
+
 - A `[[dirsql.extension]]` entry omits `path`, or `path` is empty.
-- `on-file`, `pre-query`, or `post-query` is present but empty/whitespace.
+- `pre-query` or `post-query` is present but empty/whitespace.
 - `hook-timeout` is zero or negative.
 
 ## Full example
@@ -193,10 +199,12 @@ path       = "sqlite_vec"            # Python module name; on Node use the
 entrypoint = "sqlite3_vec_init"
 
 [[table]]
-ddl  = "CREATE TABLE comments (path TEXT, basename TEXT, mtime INTEGER)"
-glob = "_comments/*/*.jsonl"
+ddl     = "CREATE TABLE comments (author TEXT, body TEXT)"
+glob    = "_comments/*/*.jsonl"
+on-file = "jq -c -s '.' {path}"
 
 [[table]]
-ddl  = "CREATE TABLE documents (path TEXT, basename TEXT, size INTEGER)"
-glob = "**/index.md"
+ddl     = "CREATE TABLE documents (title TEXT, summary TEXT)"
+glob    = "**/index.md"
+on-file = "uv run python extract_doc.py {path}"
 ```

package/docs/reference/hooks.md

@@ -105,10 +105,10 @@ The same command is attachable two ways, over the same contract: the
 which attaches it to every [path-table](./path-tables.md#parsing-rows-with-on-file)
 in the query. The command string is identical between the two spellings — the
 flag is the inline form, the config key the declared form (see
-[Parse your files into columns](../howto/parse-files-into-columns.md)). The one
-behavioral difference is not in this contract but in the surrounding table: a
-declared `[[table]]` merges the filesystem facts onto each parsed row, while a
-parsed path-table exposes only the parser's output.
+[Parse your files into columns](../howto/parse-files-into-columns.md)). In both
+spellings the table's columns are exactly what the command emits, narrowed to
+the DDL — `dirsql` injects no filesystem facts either way. A command that wants
+the path or stat metadata emits it (it has `{path}`).
 
 | Placeholder | Value |
 |---|---|

package/docs/reference/sdk.md

@@ -338,17 +338,17 @@ Maps files to table rows.
 
 - `ddl` — A SQLite `CREATE TABLE` statement; the table name is parsed from
   it. Table names must be unique across all tables.
-- `glob` — Glob pattern matched against root-relative paths. May contain
-  `{name}` [captures](./columns.md#glob-captures). Every table whose glob
-  matches a file receives that file's rows — a file can populate multiple
-  tables.
+- `glob` — Glob pattern matched against root-relative paths. Every table whose
+  glob matches a file receives that file's rows — a file can populate multiple
+  tables. A `{name}` segment is rewritten to `*` (it matches one path segment
+  but captures nothing).
 - `on_file` (`on_file` / `onFile`) — Callback receiving the matched file's full path (the root
   joined with the file's relative path — absolute when `root` is absolute)
-  and returning the rows that file contributes. `dirsql` never reads file
-  contents itself; a callback that needs the body reads the path. Return
-  an empty list to skip a file. [Stat columns and glob
-  captures](./columns.md) are merged onto each returned row; values the
-  callback emits win over same-named facts.
+  and returning the rows that file contributes. Required. A row's columns are
+  exactly what the callback returns, narrowed to the DDL — `dirsql` injects
+  nothing (see [Columns](./columns.md)). A callback that wants the path, stat
+  metadata, or file body computes it from the path it receives; `dirsql` never
+  reads file contents itself. Return an empty list to skip a file.
 - `strict` — Default off: extra row keys are dropped and missing declared
   columns become `NULL`. When on, any extra or missing key is an error
   (see [`strict`](./config.md#table)).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dirsql",
-  "version": "0.3.128",
+  "version": "0.4.0",
   "description": "Ephemeral SQL index over a local directory",
   "license": "MIT",
   "repository": "https://github.com/thekevinscott/dirsql",
@@ -212,15 +212,15 @@
     ]
   },
   "optionalDependencies": {
-    "@dirsql/lib-linux-x64-gnu": "0.3.128",
-    "@dirsql/lib-linux-arm64-gnu": "0.3.128",
-    "@dirsql/lib-darwin-x64": "0.3.128",
-    "@dirsql/lib-darwin-arm64": "0.3.128",
-    "@dirsql/lib-win32-x64-msvc": "0.3.128",
-    "@dirsql/cli-linux-x64-gnu": "0.3.128",
-    "@dirsql/cli-linux-arm64-gnu": "0.3.128",
-    "@dirsql/cli-darwin-x64": "0.3.128",
-    "@dirsql/cli-darwin-arm64": "0.3.128",
-    "@dirsql/cli-win32-x64-msvc": "0.3.128"
+    "@dirsql/lib-linux-x64-gnu": "0.4.0",
+    "@dirsql/lib-linux-arm64-gnu": "0.4.0",
+    "@dirsql/lib-darwin-x64": "0.4.0",
+    "@dirsql/lib-darwin-arm64": "0.4.0",
+    "@dirsql/lib-win32-x64-msvc": "0.4.0",
+    "@dirsql/cli-linux-x64-gnu": "0.4.0",
+    "@dirsql/cli-linux-arm64-gnu": "0.4.0",
+    "@dirsql/cli-darwin-x64": "0.4.0",
+    "@dirsql/cli-darwin-arm64": "0.4.0",
+    "@dirsql/cli-win32-x64-msvc": "0.4.0"
   }
 }