dirsql 0.3.113 → 0.3.114

package/docs/getting-started.md

@@ -9,6 +9,10 @@ a SQL database you can query over HTTP — without writing any code. You will:
 
 It takes about five minutes.
 
+`dirsql` only ever reads your files — it never writes, moves, or changes
+them — so it is safe to point at a real directory of your own once you are
+done here. See [Read-only by design](./explanation#read-only-by-design).
+
 **You need:** a terminal with `curl` and [`jq`](https://jqlang.org/), and
 Node ≥ 20.11 (for `npx`). Every `npx dirsql` step below also has a `uvx`
 tab that behaves identically, if you prefer Python tooling
@@ -16,33 +20,23 @@ tab that behaves identically, if you prefer Python tooling
 
 ## 1. Create three files
 
-Make a working directory with two subfolders — one per note author:
+Paste this whole block into your terminal. It makes a working directory
+with a subfolder per note author and writes three tiny markdown notes:
 
 ```bash
 mkdir -p my-notes/notes/alice my-notes/notes/bob
 cd my-notes
-```
-
-Create the three notes by pasting each block exactly as shown:
-
-```bash
 cat > notes/alice/welcome.md <<'EOF'
 # Welcome
 
 Start here. This folder is about to become a database.
 EOF
-```
-
-```bash
 cat > notes/alice/ideas.md <<'EOF'
 # Ideas
 
 - query files with SQL
 - watch for changes
 EOF
-```
-
-```bash
 cat > notes/bob/reading-list.md <<'EOF'
 # Reading list
 
@@ -50,6 +44,9 @@ cat > notes/bob/reading-list.md <<'EOF'
 EOF
 ```
 
+(Any directory of files works with `dirsql` — the rest of this tutorial
+assumes exactly these three so your output matches ours.)
+
 Check that all three files are in place:
 
 ```bash

package/docs/howto/search-by-meaning.md

@@ -7,6 +7,13 @@ into semantic search: a SQLite vector extension for the distance math, an
 index time, and a [`pre-query`](../reference/hooks.md#pre-query) command to
 embed each question at query time.
 
+::: tip Just want it working?
+[`dirsql-plugin-embeddings`](https://pypi.org/project/dirsql-plugin-embeddings/)
+packages exactly what this guide builds, ready to install:
+`uvx --with dirsql-plugin-embeddings dirsql`. Keep reading to see how it's
+built — the same three pieces, from scratch.
+:::
+
 ## How the pieces fit
 
 1. **[`[[dirsql.extension]]`](../reference/config.md#dirsql-extension)**

package/docs/howto/write-a-plugin.md

@@ -0,0 +1,239 @@
+# Write a plugin
+
+Ship a ready-made set of `dirsql` tables — an embeddings index, a
+log-line extractor, a metrics view — so a teammate gets them by installing a
+package, with zero config edits. A **plugin** is an ordinary pip package that
+carries a `dirsql.toml` config fragment and declares itself with a `dirsql`
+entry point; when it is installed in the same environment as the `uvx`/`pip`
+launcher, the launcher [discovers it and loads its fragment
+automatically](../reference/cli.md#plugins). Nothing in `dirsql` is
+plugin-aware — a plugin is just config plus a naming convention.
+
+## Package layout
+
+A plugin is a normal Python package. Two things make it a plugin:
+
+1. A **`dirsql.toml`** fragment shipped inside a top-level module.
+2. A **`[project.entry-points.dirsql]`** declaration pointing the launcher at
+   that module.
+
+```
+dirsql-embeddings/
+├── pyproject.toml
+└── src/
+    └── dirsql_embeddings/
+        ├── __init__.py
+        ├── dirsql.toml       # the config fragment
+        ├── embed.py          # on-file hook
+        └── search.py         # pre-query hook
+```
+
+The entry point maps a **source label** (the name) to the **module that
+ships `dirsql.toml`** (the value):
+
+```toml
+# pyproject.toml
+[project]
+name = "dirsql-embeddings"
+version = "0.1.0"
+dependencies = ["model2vec", "sqlite-vec"]
+
+[project.entry-points.dirsql]
+embeddings = "dirsql_embeddings"
+```
+
+The entry-point **name** (`embeddings`) is the source label the launcher uses
+to identify the plugin in diagnostics; the **value** (`dirsql_embeddings`) is
+the importable module the launcher resolves to find `dirsql.toml` beside it.
+Installing the package is all it takes — there is no enable step and no
+filename convention beyond `dirsql.toml`.
+
+## The fragment is an ordinary config
+
+A plugin's `dirsql.toml` is a plain [config file](../reference/config.md) —
+structurally **identical to a user config**. It may declare
+[`[[table]]`](../reference/config.md#table) (with
+[`on-file`](../reference/hooks.md#on-file)),
+[`[[dirsql.extension]]`](../reference/config.md#dirsql-extension),
+[`ignore`](../reference/config.md#dirsql-keys),
+[`pre-query`/`post-query`](../reference/hooks.md#pre-query), and
+[`hook-timeout`](../reference/hooks.md#timeout).
+
+There are **no plugin-specific keys and no plugin-specific restrictions**. The
+config schema is content-only: the index `root` and `--persist` are
+[runner-owned flags](../reference/config.md#dirsql-keys) (`--root`, `--persist
+[PATH]`), decided by whoever runs `dirsql`, never by a config file — so a
+plugin has nothing to say about them. Whatever you can put in your own
+`.dirsql.toml`, a plugin can put in its fragment, and vice-versa.
+
+## Hook commands
+
+Both hook command styles from the [hook contract](../reference/hooks.md) work
+in a plugin fragment:
+
+- **Console scripts** — a `bin`-style entry point your package installs on
+  `PATH` (`embed-file {path}`). Recommended for published plugins: the command
+  is bound to your package's interpreter and dependencies, and it is
+  language-neutral (the fragment names a command, not a Python file).
+- **Relative scripts** — a path resolved against the fragment's own directory
+  (`uv run python embed.py {path}`). Convenient while developing the plugin
+  in-tree.
+
+Two facts from the [execution contract](../reference/hooks.md#execution-contract)
+matter most for a published plugin:
+
+- **A hook runs in its declaring config's directory.** For a plugin that is
+  the installed fragment's directory — inside **site-packages**. That is a
+  read-only, shared location: **run from it, never write to it.** Use the
+  absolute [`{path}`](../reference/hooks.md#on-file) placeholder to read the
+  matched file, and [`{root}`](../reference/hooks.md#on-file) to reach the
+  user's project directory. Write any cache to `{root}` or a real cache dir,
+  never next to the fragment.
+- **`{path}` is absolute and `{root}` is the index root**, so a command is
+  self-sufficient from any working directory — it works whether the plugin
+  lives in the project or in site-packages.
+
+## Worked example: an embeddings plugin
+
+Here is the whole plugin — vector search over a directory of notes, buildable
+in about fifty lines. It composes the same three pieces as
+[Search documents by meaning](./search-by-meaning.md): the
+[`sqlite-vec`](https://github.com/asg017/sqlite-vec) extension for the
+distance math, an `on-file` command to embed each file, and a `pre-query`
+command to embed the question.
+
+The fragment, `src/dirsql_embeddings/dirsql.toml`:
+
+```toml
+[dirsql]
+pre-query    = "uv run --with model2vec python search.py {args}"
+hook-timeout = 300   # headroom for the first-run model download
+
+[[dirsql.extension]]
+path       = "sqlite_vec"
+entrypoint = "sqlite3_vec_init"
+
+[[table]]
+ddl     = "CREATE TABLE notes (path TEXT, text TEXT, embedding TEXT)"
+glob    = "notes/*.md"
+on-file = "uv run --with model2vec python embed.py {path}"
+```
+
+`embed.py` turns one file into one row carrying its text and its embedding:
+
+```python
+"""Embed one file's text; print a dirsql row array on stdout."""
+import json
+import sys
+
+from model2vec import StaticModel
+
+path = sys.argv[1]
+text = open(path, encoding="utf-8").read()
+model = StaticModel.from_pretrained("minishlab/potion-base-8M")
+vector = model.encode([text])[0]
+print(json.dumps([{"text": text, "embedding": json.dumps([round(float(x), 6) for x in vector])}]))
+```
+
+`search.py` turns a `{"q": "..."}` request body into nearest-neighbor SQL:
+
+```python
+"""Turn a {"q": "..."} request body into a nearest-neighbor SQL query."""
+import json
+import sys
+
+from model2vec import StaticModel
+
+body = json.loads(sys.argv[1])
+model = StaticModel.from_pretrained("minishlab/potion-base-8M")
+vector = model.encode([body["q"]])[0]
+needle = json.dumps([round(float(x), 6) for x in vector])
+print(
+    "SELECT path, ROUND(vec_distance_cosine(embedding, '%s'), 3) AS distance "
+    "FROM notes ORDER BY distance LIMIT 3" % needle
+)
+```
+
+::: warning The hook owns SQL safety
+Whatever SQL `pre-query` prints is executed as-is. Here the interpolated
+value is a numeric vector the script itself produced; never splice raw request
+text into SQL. See [`pre-query`](../reference/hooks.md#pre-query).
+:::
+
+The relative `embed.py` / `search.py` above resolve against the fragment
+directory, which is convenient during development. For a published plugin,
+promote them to console scripts (`[project.scripts]` → `embed-file`,
+`embed-search`) so the commands carry their own interpreter and dependencies
+and no longer depend on `uv run --with`.
+
+Once the package is installed alongside the launcher, its `notes` table is
+queryable with no config edits:
+
+```bash
+uvx --with dirsql-embeddings dirsql query '{"q": "how do I cook pasta?"}'
+```
+
+The launcher discovers the installed plugin, composes its fragment, and the
+`pre-query` hook turns the question into vector-distance SQL.
+
+## SDK-style convention: expose the config
+
+The launcher's auto-discovery is [launcher-only](#boundaries): an
+[SDK](../reference/sdk.md) consumer never gets a plugin's tables automatically.
+The convention that bridges this — **zero `dirsql` code, purely a plugin
+courtesy** — is to also expose the fragment's path programmatically, so an
+application can pass it to the constructor's
+[`config`](../reference/sdk.md#constructor) parameter by hand:
+
+```python
+# src/dirsql_embeddings/__init__.py
+from importlib.resources import files
+
+def config_path() -> str:
+    """Absolute path to this plugin's dirsql.toml, for SDK consumers."""
+    return str(files(__package__) / "dirsql.toml")
+```
+
+```python
+from dirsql import DirSQL
+from dirsql_embeddings import config_path
+
+db = DirSQL("./project", config=config_path())
+```
+
+This is not a `dirsql` feature — it is a naming convention a well-behaved
+plugin follows so its config is reachable both ways: auto-discovered by the
+launcher, and hand-passed to an SDK.
+
+## Boundaries
+
+Discovery is deliberately narrow. Know exactly who does what:
+
+- **The `cargo`-installed binary does no discovery.** It loads configs only
+  from [`-c/--config`](../reference/cli.md#flags). A standalone binary user
+  passes a plugin's fragment explicitly, like any other config.
+- **The `uvx`/`pip` launcher injects `-c <fragment>` per installed plugin.**
+  Each discovered plugin's fragment is appended after your own `-c` configs, so
+  your config still takes ordering precedence
+  ([composing configs](../reference/config.md#composing-multiple-configs)). When
+  you pass no `-c` of your own, the launcher also keeps the
+  [baked-in default](../reference/cli.md#default-mode) `files` table (an
+  internal `--include-default`), so plugins **add** tables rather than
+  replacing the default. Discovery is **pip/uvx only** for now — the `npx`
+  launcher does not yet discover — and is switched off per invocation with
+  [`--no-plugin` or `DIRSQL_NO_PLUGIN=1`](../reference/cli.md#plugins).
+- **The SDK never auto-discovers.** Pass a plugin's config explicitly (the
+  [convention above](#sdk-style-convention-expose-the-config)).
+- **Name collisions are a hard error.** Because fragments compose like any
+  [multiple configs](../reference/config.md#composing-multiple-configs), two
+  plugins (or a plugin and your config) defining a table of the same name — or
+  the same query hook — fail loudly, naming the conflict. It is never a silent
+  last-writer-wins.
+
+::: warning Config flags are subcommand-local
+For `dirsql query`, pass `-c` and friends **after** the subcommand
+(`dirsql query "<sql>" -c <cfg>`); before it they are a hard error. Plugin
+discovery is unaffected — the launcher injects its `-c` flags in the right
+position — but it matters when you also pass a config of your own. See
+[the CLI reference](../reference/cli.md#dirsql-query).
+:::

package/docs/index.md

@@ -22,6 +22,8 @@ But querying across many files is slow.
 
 `dirsql` bridges this gap. The filesystem remains the source of truth, but you get SQL queries and real-time change events for free. Define tables with glob patterns and on-file callbacks, and `dirsql` handles the rest.
 
+**`dirsql` never modifies your files.** It opens them for reading and nothing else — no writes, no moves, no deletes, no rewrites in place. Point it at anything and the worst it can do is read. This is permanent by design, not unimplemented; see [Read-only by design](./explanation#read-only-by-design) for its exact scope.
+
 ::: code-group
 
 ```python [Python]

package/docs/reference/cli.md

@@ -146,7 +146,9 @@ uses**, so behavior is identical to `POST /query` by construction:
   [`[dirsql].hook-timeout`](./config.md#dirsql-keys) apply identically.
 - The **30-second query timeout**, the **read-only rule**, and the
   `_dirsql_*` **internal-table denial** apply identically. A rejected read
-  is an error, not empty output.
+  is an error, not empty output. The read-only rule here governs SQL
+  statements; dirsql separately never modifies the files it indexes — see
+  [Read-only by design](../explanation#read-only-by-design).
 
 Errors print the same diagnostic the HTTP `{"error": …}` body carries —
 config failures, SQL errors, rejected reads, hook failures, timeouts — to
@@ -193,3 +195,29 @@ All failures exit `1` with a message on stderr:
 | Output path unwritable (e.g. missing parent directory) | Fails with the underlying I/O error. |
 
 On success, `init` exits `0`.
+
+## Plugins
+
+A **plugin** is an ordinary Python package that ships a `dirsql.toml` config
+fragment and declares itself via a `dirsql` entry point. When such a package is
+installed in the same environment as `dirsql` (`pip install …`, or
+`uvx --with …`), the `uvx`/`pip` launcher **discovers it automatically** and
+loads its fragment — its tables are queryable with zero config edits.
+Installed = active: there is no enable step and no naming convention. The
+fragment is composed *after* your own `-c` configs (so your config takes
+precedence in ordering), and the baked-in `files` table is preserved.
+
+Discovery is **launcher-only** — the standalone `cargo`-installed binary does no
+discovery, and the SDKs never auto-discover (pass a plugin's config explicitly
+instead). It is **pip/uvx only** for now; the `npx` launcher does not yet
+discover.
+
+Turn discovery off with either:
+
+| | Effect |
+|---|---|
+| `--no-plugin` | Skip plugin discovery for this invocation. Consumed by the launcher; never forwarded to the binary. |
+| `DIRSQL_NO_PLUGIN=1` | Same, via the environment. |
+
+A plugin that declares itself but is missing its module or its `dirsql.toml`
+fragment is a launcher error naming the package — never a silent skip.

package/docs/reference/http-api.md

@@ -5,7 +5,11 @@ exposes two endpoints: `POST /query` and `GET /events`.
 
 ## `POST /query`
 
-Run a read-only SQL query. Request body is JSON:
+Run a read-only SQL query — statements SQLite classifies as writes are
+rejected. That rule governs SQL against the index; dirsql separately never
+modifies the files it indexes, which is
+[permanent by design](../explanation#read-only-by-design). Request body is
+JSON:
 
 ```json
 {"sql": "SELECT title, author FROM posts WHERE draft = 0"}

package/docs/reference/sdk.md

@@ -216,6 +216,9 @@ Executes a SQL query and returns rows keyed by column name.
   before producing rows. Rust surfaces this as
   `DirSqlError::WriteForbidden`; Python raises a `RuntimeError` and
   TypeScript rejects with an `Error` carrying a "read-only" message.
+  This is the query-layer half of dirsql's broader guarantee that it never
+  modifies your files — see
+  [Read-only by design](../explanation#read-only-by-design).
 - Internal tracking columns (`_dirsql_file_path`, `_dirsql_row_index`) are
   excluded from `SELECT *` results; name them explicitly to see them.
 - SQLite values map back to language types:
@@ -341,7 +344,7 @@ Maps files to table rows.
   joined with the file's relative path — absolute when `root` is absolute)
   and returning the rows that file contributes. `dirsql` never reads file
   contents itself; a callback that needs the body reads the path. Return
-  an empty list to skip a file. [Virtual columns and glob
+  an empty list to skip a file. [Stat columns and glob
   captures](./columns.md) are merged onto each returned row; values the
   callback emits win over same-named facts.
 - `strict` — Default off: extra row keys are dropped and missing declared

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dirsql",
-  "version": "0.3.113",
+  "version": "0.3.114",
   "description": "Ephemeral SQL index over a local directory",
   "license": "MIT",
   "repository": "https://github.com/thekevinscott/dirsql",
@@ -212,15 +212,15 @@
     ]
   },
   "optionalDependencies": {
-    "@dirsql/lib-linux-x64-gnu": "0.3.113",
-    "@dirsql/lib-linux-arm64-gnu": "0.3.113",
-    "@dirsql/lib-darwin-x64": "0.3.113",
-    "@dirsql/lib-darwin-arm64": "0.3.113",
-    "@dirsql/lib-win32-x64-msvc": "0.3.113",
-    "@dirsql/cli-linux-x64-gnu": "0.3.113",
-    "@dirsql/cli-linux-arm64-gnu": "0.3.113",
-    "@dirsql/cli-darwin-x64": "0.3.113",
-    "@dirsql/cli-darwin-arm64": "0.3.113",
-    "@dirsql/cli-win32-x64-msvc": "0.3.113"
+    "@dirsql/lib-linux-x64-gnu": "0.3.114",
+    "@dirsql/lib-linux-arm64-gnu": "0.3.114",
+    "@dirsql/lib-darwin-x64": "0.3.114",
+    "@dirsql/lib-darwin-arm64": "0.3.114",
+    "@dirsql/lib-win32-x64-msvc": "0.3.114",
+    "@dirsql/cli-linux-x64-gnu": "0.3.114",
+    "@dirsql/cli-linux-arm64-gnu": "0.3.114",
+    "@dirsql/cli-darwin-x64": "0.3.114",
+    "@dirsql/cli-darwin-arm64": "0.3.114",
+    "@dirsql/cli-win32-x64-msvc": "0.3.114"
   }
 }