dirsql 0.4.99 → 0.4.101

package/docs/howto/query-without-config.md

@@ -67,5 +67,6 @@ why indexes belong to only one of them, are in
 - Paths outside the root resolve too — `'/var/log/*.log'`, `'../notes'`,
   `'~/notes/*.md'` — reporting absolute paths
   ([path-tables reference](../reference/path-tables.md#paths-outside-the-index-root)).
-- `node_modules/` and `.git/` are skipped by default so a bare `'./**'` does
-  not drown in machinery ([skip rules](../reference/path-tables.md#skip-rules)).
+- `.git/` is skipped by default, and a git repo's `.gitignore` hides what it
+  lists, so a bare `'./**'` does not drown in machinery
+  ([skip rules](../reference/path-tables.md#skip-rules)).

package/docs/reference/cli.md

@@ -249,7 +249,7 @@ Config flags are subcommand-local: pass them after `server`
 | `--port <n>` | `7117` | TCP port to bind. |
 | `--cors-origin <origin>` | off | Let browser pages on `<origin>` (an exact origin such as `http://localhost:3202`, or `*` for any) call `/query` and open `/events`. See [Cross-origin requests](#cross-origin-requests). |
 | `--persist [<path>]` | off | Keep the SQLite index on disk between runs so a restart only re-parses files that actually changed. Bare `--persist` caches at `<root>/.dirsql/cache.db`; `--persist <path>` caches at `<path>`. Off by default (the index is ephemeral). Also available on [`dirsql query`](#dirsql-query). See [Keep the index across restarts](../howto/persist.md). |
-| `--no-ignore` | off | Scan files a `.gitignore` would hide. [Path-tables](./path-tables.md#skip-rules) respect `.gitignore` files by default; this flag restores the full walk. The built-in skips (`node_modules`/`.git`) and configured `ignore` patterns still apply. Also available on [`dirsql query`](#dirsql-query). |
+| `--no-ignore` | off | Scan files a `.gitignore` would hide. [Path-tables](./path-tables.md#skip-rules) respect `.gitignore` files by default; this flag restores the full walk. The built-in `.git` skip and configured `ignore` patterns still apply. Also available on [`dirsql query`](#dirsql-query). |
 | `--extension <path>` | none | Load a SQLite extension by literal path, overriding the config's `[[dirsql.extension]]` entries. Repeatable. Format: `<path>` or `<path>::<entrypoint>`. Internal plumbing for the pip/npm launchers, which resolve package-name extensions and pass the resolved paths here — not intended for direct use. When any `--extension` is present, the config file's own extension entries are not loaded. |
 | `--version` | | Print the version and exit. |
 | `--help` | | Print usage and exit. |

package/docs/reference/config.md

@@ -47,13 +47,12 @@ ignore = ["node_modules/**", ".git/**"]
 
 ### Glob rule
 
-Every glob in dirsql — `ignore`, a `[[table]]`'s `glob`, and a
-[path-table](./path-tables.md#writing-the-path) — reads as the shell does:
-`*` and `?` match within one path segment and never cross `/`; `**` matches
-any depth. So `ignore = ["*"]` hides only the files directly inside the root,
-`ignore = ["build/*"]` hides `build/a.o` but not `build/sub/b.o`, and
-`glob = "*.json"` selects only the top-level `.json` files; write
-`**/*.json` to select them at every depth.
+Every glob in dirsql, whether `ignore`, a `[[table]]`'s `glob` or a
+[path-table](./path-tables.md#writing-the-path), reads as bash does with
+`globstar` on. The [glob page](./glob.md) has the rules, examples and every
+divergence. In short, `*` and `?` stay within one path segment, `**` crosses
+directories, and `glob = "*.json"` selects only top-level `.json` files; write
+`**/*.json` for every depth.
 
 Persistence is not a config key. Keep the SQLite index on disk between runs
 with the [`--persist [PATH]` CLI flag](./cli.md#dirsql-server) — a machine-local
@@ -232,7 +231,7 @@ what its required `on-file` command emits — dirsql injects nothing (see
 |---|---|---|
 | `name` | yes | The table's SQL name — the name you query it by. Declared, never derived from `ddl`: dirsql does not read the DDL text. The `ddl` must create a table by this name; if it doesn't, loading fails. |
 | `ddl` | yes | A SQL batch, run verbatim — any number of statements. It must create a table called `name`; that table holds the file rows, and only the columns it declares are kept (keys the `on-file` command emits that are not declared are dropped). The rest of the batch is yours: indexes, virtual tables, triggers. See [Batch `ddl`](#batch-ddl). |
-| `glob` | yes | Glob pattern, [anchored](#glob-anchor) at the config file's directory, under the [one glob rule](#glob-rule): `*` matches one level, `**` any depth. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). |
+| `glob` | yes | Glob pattern, [anchored](#glob-anchor) at the config file's directory, under the [one glob rule](#glob-rule): `*` matches one level, `**` any depth. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` is a literal, as in bash and a path-table; the [glob page](./glob.md) has the syntax. |
 | `on-file` | **yes** | A command run once per table, with every matched file's absolute path appended as a trailing argument; its stdout (one JSON array of row objects) is the table's rows. Must be non-empty. A `[[table]]` with no `on-file` is a load error (see [parse errors](#parse-errors)). See [Command hooks](./hooks.md#on-file). |
 | `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied by the `on-file` output. When `false`, extra keys are dropped and missing columns become `NULL`. |
 

package/docs/reference/glob.md

@@ -0,0 +1,139 @@
+# Globs
+
+A glob is the string you write where a table name goes (`FROM './docs/*.md'`)
+or in a config `[[table]]`'s `glob`. dirsql reads it the way `bash` does with
+`globstar` on and `dotglob` off, and every place it does not is listed under
+[Divergences from bash](#divergences-from-bash).
+
+## Why bash
+
+Bash with `globstar` is the most popular glob behavior. Bash, zsh, node-glob
+and minimatch (npm, ESLint, VS Code, GitHub Actions) and Python's `glob` agree
+on one core: `*`, `?` and `[...]`, recursive `**`, `{a,b}` braces, `*` hiding
+dotfiles, and `**` not entering symlinked directories. Python has no braces
+and follows symlinks; bash breaks that tie. POSIX has no `**` or braces. So
+there is one standard nobody has to learn, the same string works in a
+[path-table](./path-tables.md) and in a [config](./config.md) `glob`, and each
+divergence is a written decision.
+
+## The fixture
+
+Every example below runs against this tree. A name ending in `/` is an empty
+directory, `a -> b` is a symlink, and `.gitignore` holds `*.log`.
+
+<!-- conformance-tree -->
+```text
+top.md
+a.md
+b.md
+c.md
+1.md
+2.md
+3.md
+4.md
+x.txt
+ignored.log
+{q}.md
+[a].md
+[z.md
+my file.md
+.env
+.git/
+.config/app.toml
+.config/.secret
+.config/sub/deep.md
+docs/guide.md
+docs/api.md
+docs/.draft.md
+docs/nested/deep.md
+docs/nested/.cache/x.md
+node_modules/pkg/index.js
+real/r.md
+linkdir -> real
+link.md -> top.md
+broken -> nowhere
+```
+
+## Examples
+
+Each row lists the files the pattern returns, sorted. Every row without a
+divergence was produced by `bash -O globstar -O nullglob` with `dotglob` off,
+run from the fixture's root, keeping only files. The `Divergence` column names
+the rule from [below](#divergences-from-bash) that makes a row differ from
+bash's raw output. `$ROOT` is the fixture's directory and `$BASE` its parent.
+
+A test runs every row against dirsql as a path-table and, with the leading
+`./` dropped, as a config `glob`, so this table cannot drift from the code.
+
+<!-- conformance-table -->
+| Pattern | Rows | Divergence |
+| --- | --- | --- |
+| `./*.md` | `1.md`, `2.md`, `3.md`, `4.md`, `[a].md`, `[z.md`, `a.md`, `b.md`, `c.md`, `link.md`, `my file.md`, `top.md`, `{q}.md` |  |
+| `./**/*.md` | `1.md`, `2.md`, `3.md`, `4.md`, `[a].md`, `[z.md`, `a.md`, `b.md`, `c.md`, `docs/api.md`, `docs/guide.md`, `docs/nested/deep.md`, `link.md`, `linkdir/r.md`, `my file.md`, `real/r.md`, `top.md`, `{q}.md` |  |
+| `./**` | `1.md`, `2.md`, `3.md`, `4.md`, `[a].md`, `[z.md`, `a.md`, `b.md`, `c.md`, `docs/api.md`, `docs/guide.md`, `docs/nested/deep.md`, `link.md`, `my file.md`, `node_modules/pkg/index.js`, `real/r.md`, `top.md`, `x.txt`, `{q}.md` |  |
+| `./?.md` | `1.md`, `2.md`, `3.md`, `4.md`, `a.md`, `b.md`, `c.md` |  |
+| `./[ab].md` | `a.md`, `b.md` |  |
+| `./[!a].md` | `1.md`, `2.md`, `3.md`, `4.md`, `b.md`, `c.md` |  |
+| `./\[a\].md` | `[a].md` |  |
+| `./[z.md` | `[z.md` |  |
+| `./{a,b}.md` | `a.md`, `b.md` |  |
+| `./{q}.md` | `{q}.md` |  |
+| `./{a.md` | none |  |
+| `./{1..3}.md` | `1.md`, `2.md`, `3.md` |  |
+| `./my file.md` | `my file.md` |  |
+| `./.*` | `.env`, `.gitignore` |  |
+| `./**/.*` | `.env`, `.gitignore`, `docs/.draft.md` |  |
+| `./.env` | `.env` |  |
+| `./.config/**` | `.config/app.toml`, `.config/sub/deep.md` |  |
+| `./docs/.*` | `docs/.draft.md` |  |
+| `./link.md` | `link.md` |  |
+| `./linkdir/*` | `linkdir/r.md` |  |
+| `./*/r.md` | `linkdir/r.md`, `real/r.md` |  |
+| `./**/r.md` | `linkdir/r.md`, `real/r.md` |  |
+| `./real/**` | `real/r.md` |  |
+| `./docs` | `docs/api.md`, `docs/guide.md` | directory-name |
+| `./docs/` | `docs/api.md`, `docs/guide.md` | directory-name |
+| `./*/` | `docs/api.md`, `docs/guide.md`, `linkdir/r.md`, `real/r.md` | directory-name |
+| `./d*` | none | files-only |
+| `./*.log` | none | gitignore |
+| `./**/*.js` | `node_modules/pkg/index.js` |  |
+| `./node_modules/*/index.js` | `node_modules/pkg/index.js` |  |
+| `./docs/**/*.md` | `docs/api.md`, `docs/guide.md`, `docs/nested/deep.md` |  |
+| `./docs/*.md` | `docs/api.md`, `docs/guide.md` |  |
+| `$ROOT/docs/*.md` | `$ROOT/docs/api.md`, `$ROOT/docs/guide.md` | absolute |
+| `~/docs/*.md` | `$ROOT/docs/api.md`, `$ROOT/docs/guide.md` | absolute |
+| `../proj/docs/*.md` | `$BASE/proj/docs/api.md`, `$BASE/proj/docs/guide.md` | absolute |
+
+## Divergences from bash
+
+| Tag | Rule | Why |
+| --- | --- | --- |
+| `gitignore` | Inside a git repo, `.gitignore` files hide matching files and directories. `--no-ignore` turns it off. | A query over a repo should not drown in build output. |
+| `directory-name` | A pattern's last component that names a directory lists the files directly inside it, like `ls docs`. A trailing `/` is `*` appended. | A table is rows of files, and `ls` is the shell habit. |
+| `files-only` | A pattern that matches a directory but no file returns nothing. | Every row is a file. |
+| `absolute` | `/`, `~/` and `../` patterns report absolute `path` values, with `..` folded out textually. | The scanned root is derived from the pattern, so there is no relative base. |
+
+`.git` needs no rule of its own: the dotfile rule hides it.
+
+## Where the glob is anchored
+
+A path-table pattern is relative to the index root and starts with `./`. A config `[[table]] glob`
+is relative to the directory holding its config file, whatever directory you
+run `dirsql` from, and is written without the `./`: `./docs/*.md` in a path-table
+is `glob = "docs/*.md"` in a config. A config table whose glob matches no files prints
+`dirsql: table 'name': glob 'pattern' matched no files under <anchor>` to
+stderr; the exit code is unchanged. Path-tables never warn.
+
+## Edge cases
+
+- A `{`, `}` or `[` with no partner is a literal character.
+- A brace group with no comma and no `..` is literal (`{q}`); `{1..3}` and
+  `{a..c}` expand, as do zero-padded
+  `{01..10}` and stepped `{1..10..2}` ranges. Groups nest.
+- `\[` is a literal bracket.
+- A dot-named file or directory is listed only where a component spells the
+  dot at that depth: `./**/.*` finds `docs/.draft.md` but does not enter
+  `.config/`.
+- A symlinked file is a file. A symlinked directory is entered by any component
+  except `**`.
+- A broken link lists nothing.

package/docs/reference/path-tables.md

@@ -52,16 +52,8 @@ depth, as in the shell. A trailing `/` is `*` appended, so `'./*/'` is
 | `'./notes/today.md'` | exactly that one file — one file is one row |
 
 A path containing `*`, `?`, `[` or `{` is a glob and is used exactly as
-written: `*` matches within a single directory, `**` crosses directories.
-
-Braces expand as bash expands them. A group holding a comma is an
-alternation: `'./docs/{a,b}.md'` is `docs/a.md` and `docs/b.md`, and groups
-nest. A group holding a `..` range counts: `'./log/{1..3}.txt'`,
-`'./{a..c}.md'`, `'./{01..10}.md'` (zero-padded) and `'./{1..10..2}.md'`
-(every second value). Any other brace is literal text, so `'./br/{q}.md'`
-names the file `br/{q}.md`, and `\{` is always a literal brace. A `{`, `}`
-or `[` with no partner is literal too, in a path-table and in a config
-`[[table]] glob`.
+written. The [glob page](./glob.md) has the syntax, examples and every
+divergence from bash, including braces and unbalanced brackets.
 
 The scan starts at the last directory named outright before the first glob
 component, and descends only as deep as the pattern can match --
@@ -206,8 +198,8 @@ over a very large table. Its output *is* the table:
   ```
   dirsql_parsed: parser produced no rows for `posts/*.md`; cannot infer a schema
   ```
-- **The skip rules still apply.** A parsed scan honors the same `node_modules`
-  /`.git`/`ignore` rules a stat scan does (see below).
+- **The skip rules still apply.** A parsed scan honors the same `.git`
+  /`ignore` rules a stat scan does (see below).
 - **The parser runs on every start.** A parsed path-table is not kept in the
   [persistent cache](/howto/persist); declared tables are.
 
@@ -259,14 +251,9 @@ choice is stated as one trade in
 A path-table scan applies the same [`ignore`](/reference/config) patterns your
 declared tables use — matched against root-relative paths under the same
 [glob rule](/reference/config#glob-rule) as the path itself, `*` one level and
-`**` any depth — plus two built-in defaults so a zero-config
-`SELECT * FROM './**'` does not drown in machinery:
-
-- `**/node_modules/**`
-- `**/.git/**`
-
-Both apply at any depth, so a `node_modules` nested inside a subdirectory is
-skipped just like one at the top.
+`**` any depth — plus one built-in default, `**/.git/**`, which applies at any depth.
+`node_modules` is an ordinary directory: a `.gitignore` that lists it hides it,
+as it hides any other.
 
 ### `.gitignore`
 
@@ -274,9 +261,7 @@ Path-table scans also respect `.gitignore` files by default, inside a git
 repo, the way git, fd and ripgrep do: a `.gitignore` applies below its own
 directory, deeper files override shallower ones, `!pattern` re-includes, and
 an ignored directory is pruned rather than walked. In a typical repo this
-excludes build output, virtualenvs, and caches with zero ceremony. The
-built-in defaults above remain as a floor for directories with no
-`.gitignore` at all.
+excludes build output, virtualenvs, and caches with zero ceremony.
 
 A `.gitignore` is in force only when a directory holding `.git` encloses it.
 Outside a repo none applies, at the scan's start, above it or below it: a
@@ -288,7 +273,7 @@ applies, including those above the directory a scan starts in, so
 Pass [`--no-ignore`](./cli.md#flags) to restore the full walk — the
 determinism switch for scripted use, since results otherwise depend on
 `.gitignore` state. It disables only the `.gitignore` respect; the built-in
-defaults and configured `ignore` patterns still apply.
+default and configured `ignore` patterns still apply.
 
 ### Naming a skipped directory
 
@@ -296,23 +281,13 @@ Skip rules are judged from the directory the scan starts in, so pointing at a
 skipped directory — built-in or gitignored — still scans it:
 
 ```sql
-SELECT path FROM './**';                   -- no node_modules rows
-SELECT path FROM './node_modules/*/package.json';  -- scans it anyway
+SELECT path FROM './.git/*';               -- scans .git anyway
 SELECT path FROM './dist';                 -- scans dist/ even when gitignored
 ```
 
 The `.gitignore` files in force there, including those above it, still
 filter what lies beneath it.
 
-A `node_modules` component after a glob names it too, at any depth:
-
-```sql
-SELECT path FROM './**/node_modules/*/package.json';  -- every node_modules
-SELECT path FROM './*/node_modules/**';               -- one level down
-```
-
-Only the literal name counts; `'./**/*.js'` still skips every `node_modules`.
-
 ### Hidden files
 
 A path component starting with `.` is skipped unless the path spells it, the

package/docs/reference/sdk.md

@@ -123,7 +123,7 @@ shortcut was removed in #603 — use
   matched files are skipped entirely (scan and watch).
 - `no_ignore` / `noIgnore` — Scan files a `.gitignore` would hide.
   [Path-tables](./path-tables.md#skip-rules) respect `.gitignore` files by
-  default; the built-in `node_modules`/`.git` skips and any `ignore` patterns
+  default; the built-in `.git` skip and any `ignore` patterns
   still apply. The CLI's [`--no-ignore`](./cli.md#flags).
 - `config` — Path to a [`.dirsql.toml`](./config.md). Its `[[table]]`
   entries are appended after any programmatic `tables`; its `ignore`

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dirsql",
-  "version": "0.4.99",
+  "version": "0.4.101",
   "description": "Ephemeral SQL index over a local directory",
   "license": "MIT",
   "repository": "https://github.com/thekevinscott/dirsql",
@@ -221,10 +221,10 @@
     ]
   },
   "optionalDependencies": {
-    "@dirsql/lib-linux-x64-gnu": "0.4.99",
-    "@dirsql/lib-linux-arm64-gnu": "0.4.99",
-    "@dirsql/lib-darwin-x64": "0.4.99",
-    "@dirsql/lib-darwin-arm64": "0.4.99",
-    "@dirsql/lib-win32-x64-msvc": "0.4.99"
+    "@dirsql/lib-linux-x64-gnu": "0.4.101",
+    "@dirsql/lib-linux-arm64-gnu": "0.4.101",
+    "@dirsql/lib-darwin-x64": "0.4.101",
+    "@dirsql/lib-darwin-arm64": "0.4.101",
+    "@dirsql/lib-win32-x64-msvc": "0.4.101"
   }
 }