dirsql 0.4.25 → 0.4.27

package/docs/howto/define-tables.md

@@ -65,6 +65,50 @@ Add one `[[table]]` entry per table — each with its own `glob`, `ddl`, and
 table — each table is an independent view. See
 [`[[table]]`](../reference/config.md#table) for the remaining key, `strict`.
 
+## Add indexes and full-text search
+
+`ddl` is a SQL batch, not a single statement — SQLite runs the whole thing. So
+a table can arrive with its own index, and with an FTS5 index kept current by
+two triggers:
+
+```toml
+[[table]]
+name    = "posts"
+glob    = "posts/**/*.md"
+on-file = "python3 extract.py {path}"
+ddl     = '''
+CREATE TABLE posts (title TEXT, slug TEXT, body TEXT);
+CREATE INDEX posts_slug ON posts(slug);
+
+CREATE VIRTUAL TABLE posts_fts
+  USING fts5(body, content='posts', content_rowid='rowid');
+CREATE TRIGGER posts_ai AFTER INSERT ON posts BEGIN
+  INSERT INTO posts_fts(rowid, body) VALUES (new.rowid, new.body);
+END;
+CREATE TRIGGER posts_ad AFTER DELETE ON posts BEGIN
+  INSERT INTO posts_fts(posts_fts, rowid, body)
+    VALUES ('delete', old.rowid, old.body);
+END;
+'''
+```
+
+```bash
+dirsql query "SELECT slug FROM posts JOIN posts_fts ON posts.rowid = posts_fts.rowid
+              WHERE posts_fts MATCH 'recursion'" -c ./.dirsql.toml
+```
+
+```json
+[{"slug":"again"}]
+```
+
+dirsql writes rows with plain `INSERT` and `DELETE`, so those triggers hold
+through the initial load and every [watcher](./react-to-changes.md) event — you
+maintain nothing. `name` still has to be the row table (`posts`); the virtual
+table lives beside it under its own name. The batch runs once, when the table
+is created, and editing any part of it rebuilds a
+[persistent cache](./persist.md) from scratch. Full rules:
+[Batch `ddl`](../reference/config.md#batch-ddl).
+
 ## Going further
 
 - The parser mechanics — placeholders, stdout protocol, per-file failure

package/docs/howto/load-extension.md

@@ -76,9 +76,11 @@ interpreter to resolve package names with
 
 ## Notes
 
-- Extensions add **functions**. An extension-backed virtual table cannot be
-  declared as a `[[table]]` — `dirsql` tables are per-file row tables
-  ([reference](../reference/config.md#dirsql-extension)).
+- Extensions add **functions**, and virtual tables a table's `ddl` batch can
+  create. A `[[table]]`'s own `name` may not be a virtual table — that one is
+  the per-file row table `dirsql` inserts into — so declare the virtual table
+  alongside it in the same batch
+  ([reference](../reference/config.md#batch-ddl)).
 - Loading happens before any table DDL runs, and the SQL
   `load_extension()` function is never exposed to queries
   ([reference](../reference/config.md#dirsql-extension)).

package/docs/howto/search-by-meaning.md

@@ -95,10 +95,12 @@ LIMIT 10
 
 ::: tip Top-k is `LIMIT k`
 If you know `sqlite-vec` you may reach for its `MATCH … AND k = 10` idiom.
-That syntax belongs to `sqlite-vec`'s `vec0` virtual table, which `dirsql`
-does not use — `dirsql` tables are per-file row tables. For plain
-expressions, `sqlite-vec`'s own documented pattern is exactly what this
-guide uses: `ORDER BY vec_distance_cosine(...) LIMIT k`.
+That syntax belongs to `sqlite-vec`'s `vec0` virtual table, which the table
+above does not use: a `[[table]]`'s own `name` is always a per-file row table.
+For plain expressions, `sqlite-vec`'s own documented pattern is exactly what
+this guide uses: `ORDER BY vec_distance_cosine(...) LIMIT k`. To get the `vec0`
+idiom instead, declare the `vec0` table alongside the row table in the same
+[`ddl` batch](../reference/config.md#batch-ddl) and fill it from a trigger.
 :::
 
 ## Repeat runs are cheap

package/docs/reference/cli.md

@@ -9,9 +9,9 @@ The `dirsql` binary has these modes:
 | `dirsql query "<sql>"` | Explicit synonym for the default one-shot query. |
 | `dirsql server` | Start a long-lived HTTP server exposing a SQL view of a directory. See [HTTP API](./http-api.md). |
 | `dirsql init` | Generate a `.dirsql.toml`. |
+| `dirsql` (bare) | Open a [REPL](#the-repl) over the current directory, reading statements until EOF. |
 
-Bare `dirsql` with no SQL is a usage error pointing at `dirsql server` — it
-does **not** start the server.
+Bare `dirsql` does **not** start the server — that is `dirsql server`.
 
 ## Installation
 
@@ -47,6 +47,186 @@ dirsql "SELECT basename, size FROM './' ORDER BY size DESC LIMIT 5"
 pipeline, same flags, same output. See that section for config discovery,
 `--persist`, `--on-file`, hooks, and exit codes.
 
+## The REPL
+
+`dirsql` with no subcommand and no SQL reads statements until EOF:
+
+```bash
+dirsql
+# dirsql 0.2.7 — this directory is a database.
+#
+#   SELECT basename, size FROM './' ORDER BY size DESC LIMIT 5;
+#   SELECT path FROM './**/*.md' WHERE content LIKE '%TODO%';
+#
+# `exit`, `quit`, or Ctrl-D to leave.
+#
+# dirsql> SELECT count(*) AS files FROM './';
+# files
+# -----
+# 128
+#
+# 1 row
+# dirsql>
+```
+
+Statements go through the same pipeline as [`dirsql query`](#dirsql-query) and
+`POST /query`, so a statement typed at the prompt and one passed on the command
+line return identical rows. Every config flag the default mode takes — `-c`,
+`--persist`, `--no-ignore`, `--on-file` — applies unchanged:
+
+```bash
+dirsql -c .dirsql.toml --persist
+```
+
+The index is built **once**, before the first prompt: statements share one scan
+rather than re-walking the directory each time, and the live watcher keeps it
+fresh between them. Files the scan had to skip are named on stderr once, up
+front.
+
+### Output format
+
+Rows go where they are useful: a **table** when stdout is a terminal, the
+**JSON array** when it is piped or redirected. `SELECT * FROM './'` in a
+5000-file tree should not put a 5000-element JSON array in front of a person,
+and `dirsql "…" | jq` should not have to parse a table.
+
+`--format` overrides that, in both directions, and is valid in the REPL and in
+[`dirsql query`](#dirsql-query) alike:
+
+| Value | Renders |
+|---|---|
+| `auto` (default) | Table if stdout is a terminal, JSON otherwise. |
+| `table` | Always a table — including into a pipe or a file. |
+| `json` | Always the JSON array — including at a terminal. |
+
+```bash
+dirsql "SELECT basename, size FROM './' ORDER BY basename" --format table
+# basename  size
+# --------  ----
+# a.md      6
+# bb.md     10
+#
+# 2 rows
+```
+
+There is no `.mode`: dirsql has no dot-commands to extend (see
+[Leaving](#leaving)), and a flag serves the one-shot query too. `dirsql server`
+does not take `--format` — its transport is JSON over HTTP.
+
+**`auto` keys on stdout, not stdin.** `dirsql > rows.json` typed at a terminal
+is still headed for a file, and the file gets JSON.
+
+Table rendering is deliberately plain: aligned columns, a rule under the
+header, a row count, and `NULL` spelled out so it cannot be confused with an
+empty string. Two things happen to a value on its way into a cell, both
+because a `content` column holds a whole file: **newlines, tabs and other
+control characters are escaped** (`\n`, `\t`, `\u{…}`) so one row cannot span
+several lines, and **anything longer than 60 characters is truncated with
+`…`** so one column cannot set the width of every row. `--format json`
+returns the values unaltered.
+
+Laying the table out to the terminal's width, and paging a long result, are
+both out of scope; pipe to `less` for the latter.
+
+### Where a statement ends
+
+At its semicolon — the same rule `sqlite3` uses, and **SQLite's own tokenizer**
+decides where that semicolon is. So a statement can be laid out over as many
+lines as it needs, and a `;` inside a string literal, a comment, or a
+`BEGIN … END` body is not mistaken for the end of one:
+
+```
+dirsql> SELECT basename, size
+   ...> FROM './'
+   ...> ORDER BY size DESC
+   ...> LIMIT 5;
+```
+
+The `...>` prompt says the statement is still open. `exit`, `quit`, and a blank
+line are not SQL, so they are taken as typed rather than waiting for a
+terminator.
+
+### Editing and history
+
+The prompt is a full line editor ([reedline](https://github.com/nushell/reedline)),
+with the emacs bindings a shell prompt has:
+
+| Key | Does |
+|---|---|
+| ↑ / ↓ | Walk back and forth through history. |
+| Ctrl-R | Reverse-search history; type to narrow, Enter to accept. |
+| Ctrl-A / Ctrl-E | Jump to the start / end of the line. |
+| Alt-B / Alt-F | Move back / forward a word. |
+| Ctrl-W, Ctrl-K, Ctrl-Y | Kill the previous word, kill to end of line, yank it back. |
+| Ctrl-C | Abandon the line and return to a fresh prompt. **Does not exit.** |
+| Ctrl-D | Leave. |
+
+History is kept in one file for every directory — a query worked out in one
+project is worth recalling in the next, the same way `sqlite3` keeps a single
+`~/.sqlite_history`. It holds the last 1000 statements, at
+`$XDG_DATA_HOME/dirsql/history`, falling back to
+`~/.local/share/dirsql/history` (`%APPDATA%\dirsql\history` on Windows). If
+none of those resolve, history is kept in memory for the session only.
+
+### Terminal vs. pipe
+
+The prompt, banner, editor, and history exist only when **stdin is a terminal**.
+From a pipe or a redirect there is none of that, and the terminator rule does
+not apply either: a redirected script is not being typed, so there is no
+continuation prompt to hang it off. **One statement per line, no `;` needed:**
+
+```bash
+printf "SELECT 1 AS n\nSELECT 2 AS n\n" | dirsql
+# [{"n":1}]
+# [{"n":2}]
+
+dirsql < queries.sql > rows.jsonl
+```
+
+Blank lines do nothing in either mode.
+
+### Leaving
+
+`exit`, `quit` (either case), or Ctrl-D. There are no dot-commands: the `.`
+prefix exists in `sqlite3` to namespace meta-commands against SQL, and with no
+meta-commands there is nothing to namespace. Schema questions are ordinary SQL:
+
+```sql
+SELECT name FROM sqlite_master WHERE type = 'table';
+```
+
+### Errors
+
+A statement that fails prints its diagnostic — the same string the HTTP
+`{"error": …}` body carries — to stderr, and **the session continues**. This is
+the one behavioral difference from `dirsql query`, which exits `1` on the first
+failure:
+
+```
+dirsql> SELECT nope FROM missing;
+dirsql: SQLite error: no such table: missing
+dirsql> SELECT 1 AS n;
+n
+-
+1
+
+1 row
+```
+
+A config that cannot be loaded is different in kind: it fails identically for
+every statement, so it is reported once and exits `1` before the first prompt.
+
+### Exit codes
+
+| Code | Meaning |
+|---|---|
+| `0` | Clean EOF (Ctrl-D, `exit`, `quit`, or the end of a piped script) — **including when statements failed**. Matches interactive `sqlite3`; use [`dirsql query`](#dirsql-query) when a script needs a statement's exit status. |
+| `1` | The index could not be built (a bad `-c`, an unresolvable `--on-file`), or stdin could not be read. Nothing was executed. |
+
+`23` (partial scan) is not produced here: skipped files are reported before the
+first prompt, and a session's exit code describes the session rather than one
+scan.
+
 ## `dirsql server`
 
 ```bash
@@ -204,6 +384,12 @@ Errors print the same diagnostic the HTTP `{"error": …}` body carries —
 config failures, SQL errors, rejected reads, hook failures, timeouts — to
 stderr, with exit code `1`.
 
+#### `--format {auto,table,json}`
+
+How to render the result rows — the same flag [the REPL](#output-format)
+takes, with the same `auto` default. A one-shot query is usually piped, so
+`auto` usually means JSON; `--format table` is there for the times it is not.
+
 ### Exit codes
 
 | Code | Meaning |

package/docs/reference/config.md

@@ -87,10 +87,11 @@ name you installed:
   `path = "sqlite-vec-linux-x64"` (matching your platform), not
   `path = "sqlite-vec"`, whose meta-package ships no loadable.
 
-Extensions add **functions** callable in queries and in a table's DDL. An
-extension-backed **virtual table** cannot be declared as a `[[table]]` —
-`dirsql` tables are per-file row tables, so a `CREATE VIRTUAL TABLE` DDL is
-rejected; call the extension's functions in queries instead.
+Extensions add **functions** callable in queries and in a table's DDL, and
+**virtual tables** a table's [`ddl` batch](#batch-ddl) can create. What a
+`[[table]]`'s own `name` may not be is a virtual table: that one is the
+per-file row table `dirsql` inserts into. Create the virtual table alongside
+it, under its own name.
 
 ## `[[dirsql.function]]`
 
@@ -167,7 +168,7 @@ what its required `on-file` command emits — dirsql injects nothing (see
 | Key | Required | Description |
 |---|---|---|
 | `name` | yes | The table's SQL name — the name you query it by. Declared, never derived from `ddl`: dirsql does not read the DDL text. The `ddl` must create a table by this name; if it doesn't, loading fails. |
-| `ddl` | yes | A SQLite `CREATE TABLE` statement, run verbatim. Only the columns declared here are kept; keys the `on-file` command emits that are not declared are dropped. |
+| `ddl` | yes | A SQL batch, run verbatim — any number of statements. It must create a table called `name`; that table holds the file rows, and only the columns it declares are kept (keys the `on-file` command emits that are not declared are dropped). The rest of the batch is yours: indexes, virtual tables, triggers. See [Batch `ddl`](#batch-ddl). |
 | `glob` | yes | Glob pattern matched against root-relative paths. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). |
 | `on-file` | **yes** | A command run once per matched file; its stdout (a JSON array of row objects) becomes the file's rows. Must be non-empty. A `[[table]]` with no `on-file` is a load error (see [parse errors](#parse-errors)). See [Command hooks](./hooks.md#on-file). |
 | `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied by the `on-file` output. When `false`, extra keys are dropped and missing columns become `NULL`. |
@@ -194,6 +195,72 @@ on-file = "uv run python extract_papers.py {path}"
 strict  = true
 ```
 
+### Batch `ddl`
+
+`ddl` is handed to SQLite whole, so a table declaration is not limited to one
+statement:
+
+```toml
+[[table]]
+name    = "messages"
+glob    = "sessions/*/messages/*.json"
+on-file = "jq -c '.' {path}"
+ddl     = '''
+CREATE TABLE messages (session TEXT, idx INT, role TEXT, text TEXT);
+CREATE INDEX messages_session ON messages(session);
+
+CREATE VIRTUAL TABLE messages_fts
+  USING fts5(text, content='messages', content_rowid='rowid');
+CREATE TRIGGER messages_ai AFTER INSERT ON messages BEGIN
+  INSERT INTO messages_fts(rowid, text) VALUES (new.rowid, new.text);
+END;
+CREATE TRIGGER messages_ad AFTER DELETE ON messages BEGIN
+  INSERT INTO messages_fts(messages_fts, rowid, text)
+    VALUES ('delete', old.rowid, old.text);
+END;
+'''
+```
+
+```bash
+dirsql query "SELECT text FROM messages_fts WHERE messages_fts MATCH 'deploy'" \
+  -c ./.dirsql.toml
+```
+
+Those two triggers are all a keyword index needs. dirsql writes file rows with
+plain `INSERT` and `DELETE` — an update is a delete and an insert in one
+transaction, and there is no `UPDATE` path on user rows — so triggers you
+declare here stay current through the initial load and every
+[watcher](../howto/react-to-changes.md) event. The same shape with a `vec0`
+virtual table and an [`embed()`](#dirsql-function) call in the trigger gives
+you stored vectors.
+
+**dirsql never reads the batch.** After it runs, SQLite's own catalog
+(`pragma_table_list`) settles what it produced:
+
+- No table called `name` → a load error that lists what the batch *did*
+  create, so a typo is obvious.
+- `name` is a **virtual** table → a load error. The declared table is the one
+  dirsql inserts file rows into, so it has to be a real row table. Create the
+  virtual table alongside it, under its own name.
+- `name` is **`WITHOUT ROWID`** → a warning. Internal row bookkeeping is keyed
+  on rowid, so these will be rejected in a future release.
+
+The whole batch runs in one transaction: if any statement fails, none of them
+took effect, and the error is SQLite's own, prefixed with the config entry —
+`table 'messages': SQLite error: near "(": syntax error`. Context, never
+interpretation.
+
+Two consequences of `ddl` running **once, when the table is created**:
+
+- **Rows the batch inserts itself are not file-tracked.** No file owns them, so
+  they survive file deletions — and they vanish on any rebuild.
+- **Editing `ddl` at all rebuilds a
+  [persistent cache](../howto/persist.md).** The config hash covers the entire
+  batch, so a new index, a different FTS5
+  tokenizer or a changed embedding model id forces a full sweep and re-ingest.
+  That is the only invalidation lane: dirsql tracks no ownership of what the
+  batch made.
+
 ### `on-file` row mapping
 
 The command prints a JSON array of objects; each object becomes one row.
@@ -247,13 +314,24 @@ SDKs raise/reject) when:
 - A `[[table]]` entry omits `name`, `ddl`, or `glob` (or `name` is
   empty/whitespace).
 - A `[[table]]` entry's `ddl` runs but creates no table by its `name`. The
-  error carries the entry's name and points at the fix:
+  error carries the entry's name, lists what the batch did create, and points
+  at the fix:
 
-  > `table 'messages': its `ddl` ran but created no table called 'messages'. Set `name` to the table the `ddl` creates.`
+  > `table 'messages': its `ddl` ran but created no table called 'messages' (it created: mesages, mesages_fts). Set `name` to the table the `ddl` creates.`
 
   dirsql asks SQLite's catalog rather than interpreting the DDL, so quoted
   (`CREATE TABLE "messages"`), schema-qualified (`main.messages`) and
   `IF NOT EXISTS` forms all match a plain `name = "messages"`.
+- A `[[table]]` entry's `name` names a **virtual** table. The declared table
+  holds the file rows, so it must be a real row table:
+
+  > `table 'messages': its `ddl` created a virtual table called 'messages'. The declared table holds the file rows, so it must be a real row table; create the virtual table alongside it, under its own name.`
+
+- A `[[table]]` entry's `ddl` is rejected by SQLite. Nothing the batch did
+  takes effect, and SQLite's own message is passed through under the entry's
+  name:
+
+  > `table 'messages': SQLite error: near "(": syntax error`
 - A `[[table]]` entry omits `on-file` (or it is empty/whitespace). The error
   names the offending glob and points at the fix:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dirsql",
-  "version": "0.4.25",
+  "version": "0.4.27",
   "description": "Ephemeral SQL index over a local directory",
   "license": "MIT",
   "repository": "https://github.com/thekevinscott/dirsql",
@@ -213,10 +213,10 @@
     ]
   },
   "optionalDependencies": {
-    "@dirsql/lib-linux-x64-gnu": "0.4.25",
-    "@dirsql/lib-linux-arm64-gnu": "0.4.25",
-    "@dirsql/lib-darwin-x64": "0.4.25",
-    "@dirsql/lib-darwin-arm64": "0.4.25",
-    "@dirsql/lib-win32-x64-msvc": "0.4.25"
+    "@dirsql/lib-linux-x64-gnu": "0.4.27",
+    "@dirsql/lib-linux-arm64-gnu": "0.4.27",
+    "@dirsql/lib-darwin-x64": "0.4.27",
+    "@dirsql/lib-darwin-arm64": "0.4.27",
+    "@dirsql/lib-win32-x64-msvc": "0.4.27"
   }
 }