dirsql 0.3.117 → 0.3.119

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -88,14 +88,15 @@ open a **second terminal** for the next step.
88
88
 
89
89
  ## 3. Query your files
90
90
 
91
- You gave `dirsql` no configuration, so it serves a single default table
92
- named `files` ([default mode](./reference/cli.md#default-mode)).
93
- Ask it how many rows it has:
91
+ You gave `dirsql` no configuration, so no named tables exist. Query the
92
+ filesystem directly with a [path-table](./reference/path-tables.md) — a
93
+ quoted path where a table name goes. `'./'` means everything under the
94
+ index root. Ask it how many files there are:
94
95
 
95
96
  ```bash
96
97
  curl -s http://localhost:7117/query \
97
98
  -H 'content-type: application/json' \
98
- -d '{"sql":"SELECT COUNT(*) AS files FROM files"}'
99
+ -d '{"sql":"SELECT COUNT(*) AS files FROM \'./\'"}'
99
100
  ```
100
101
 
101
102
  ```
@@ -109,7 +110,7 @@ through `jq` to pretty-print. Now select some columns:
109
110
  ```bash
110
111
  curl -s http://localhost:7117/query \
111
112
  -H 'content-type: application/json' \
112
- -d '{"sql":"SELECT path, size FROM files ORDER BY path"}' \
113
+ -d '{"sql":"SELECT path, size FROM \'./\' ORDER BY path"}' \
113
114
  | jq
114
115
  ```
115
116
 
@@ -185,8 +186,7 @@ Running at localhost:7117
185
186
  ```
186
187
 
187
188
  This time `dirsql` loaded your `.dirsql.toml` and served the `notes` table
188
- you defined instead of the default `files` table. Query it from the second
189
- terminal:
189
+ you defined. Query it from the second terminal:
190
190
 
191
191
  ```bash
192
192
  curl -s http://localhost:7117/query \
@@ -2,8 +2,8 @@
2
2
 
3
3
  Map a glob of files to a named SQL table so you query exactly the files you
4
4
  care about, with exactly the columns you care about — instead of the
5
- catch-all `files` table that [default mode](../reference/cli.md#default-mode)
6
- serves.
5
+ ad-hoc [path-tables](../reference/path-tables.md)
6
+ [configless mode](../reference/cli.md#configless-mode) leaves you with.
7
7
 
8
8
  ## 1. Create a config next to your files
9
9
 
@@ -39,8 +39,7 @@ dirsql query "SELECT path, size FROM posts ORDER BY path" -c ./.dirsql.toml
39
39
  ```
40
40
 
41
41
  Files that don't match the glob (a `README.txt` next to `posts/`, say) are
42
- simply not in the table. Passing a config with `-c` fully replaces the
43
- default `files` table — only the tables you define are served.
42
+ simply not in the table. Only the tables you define are served.
44
43
 
45
44
  ## Multiple tables
46
45
 
@@ -5,10 +5,21 @@ change — and [`GET /events`](../reference/http-api.md#get-events) pushes
5
5
  every row-level change to you as it happens. No polling, no diffing on your
6
6
  side.
7
7
 
8
+ Row events are emitted for **named tables**, so this flow needs a config —
9
+ [path-tables](../reference/path-tables.md) are scanned per query and are not
10
+ watched. Define one next to your files:
11
+
12
+ ```toml
13
+ # .dirsql.toml
14
+ [[table]]
15
+ ddl = "CREATE TABLE files (path TEXT, basename TEXT, dir TEXT, ext TEXT, size INTEGER, mtime INTEGER, ctime INTEGER)"
16
+ glob = "**/*"
17
+ ```
18
+
8
19
  ## 1. Open the stream
9
20
 
10
- With the server running (`npx dirsql` / `uvx dirsql`), subscribe from
11
- another terminal:
21
+ With the server running (`npx dirsql -c ./.dirsql.toml` /
22
+ `uvx dirsql -c ./.dirsql.toml`), subscribe from another terminal:
12
23
 
13
24
  ```bash
14
25
  curl -N http://localhost:7117/events
@@ -45,7 +45,7 @@ dirsql query "SELECT path FROM notes ORDER BY path" -c ./.dirsql.toml
45
45
  ## Notes
46
46
 
47
47
  - `ignore` lives in a config file, so it needs one:
48
- [default mode](../reference/cli.md#default-mode) indexes
48
+ [configless mode](../reference/cli.md#configless-mode) indexes
49
49
  everything with no ignores.
50
50
  - The top-level `.dirsql/` directory is always excluded, ignore list or
51
51
  not — it is reserved for `dirsql`'s own metadata
@@ -217,9 +217,8 @@ Discovery is deliberately narrow. Know exactly who does what:
217
217
  your config still takes ordering precedence
218
218
  ([composing configs](../reference/config.md#composing-multiple-configs)). When
219
219
  you pass no `-c` of your own, the launcher also keeps the
220
- [baked-in default](../reference/cli.md#default-mode) `files` table (an
221
- internal `--include-default`), so plugins **add** tables rather than
222
- replacing the default. Discovery is **pip/uvx only** for now — the `npx`
220
+ shipped starter `files` table (an internal `--include-default`), so plugins
221
+ **add** tables rather than standing alone. Discovery is **pip/uvx only** for now — the `npx`
223
222
  launcher does not yet discover — and is switched off per invocation with
224
223
  [`--no-plugin` or `DIRSQL_NO_PLUGIN=1`](../reference/cli.md#plugins).
225
224
  - **The SDK never auto-discovers.** Pass a plugin's config explicitly (the
@@ -45,7 +45,7 @@ requests, closes open `/events` streams, and exits.
45
45
 
46
46
  | Flag | Default | Description |
47
47
  |---|---|---|
48
- | `-c, --config <path>` | baked-in default | Path to a [config file](./config.md). **Repeatable** (`-c a -c b`): the configs load and merge in argv order — see [Composing multiple configs](./config.md#composing-multiple-configs). The index is always rooted at the **invocation directory** (the current working directory), regardless of where a config lives — so `--config /elsewhere/.dirsql.toml` still indexes the directory you ran `dirsql` from. With none given, the [baked-in default](#default-mode) `files` table is served — a `./.dirsql.toml` on disk is **not** auto-loaded; pass it explicitly. A `-c` naming a file that does not exist is an [error](#degraded-mode), not a silent fallback to the default. |
48
+ | `-c, --config <path>` | none | Path to a [config file](./config.md). **Repeatable** (`-c a -c b`): the configs load and merge in argv order — see [Composing multiple configs](./config.md#composing-multiple-configs). The index is always rooted at the **invocation directory** (the current working directory), regardless of where a config lives — so `--config /elsewhere/.dirsql.toml` still indexes the directory you ran `dirsql` from. With none given, **no named tables are defined** — query the filesystem with a [path-table](./path-tables.md) (`FROM './'`). A `./.dirsql.toml` on disk is **not** auto-loaded; pass it explicitly. A `-c` naming a file that does not exist is an [error](#degraded-mode). |
49
49
  | `--host <addr>` | `localhost` | Bind address. |
50
50
  | `--port <n>` | `7117` | TCP port to bind. |
51
51
  | `--persist [<path>]` | off | Keep the SQLite index on disk between runs so a restart only re-parses files that actually changed. Bare `--persist` caches at `<root>/.dirsql/cache.db`; `--persist <path>` caches at `<path>`. Off by default (the index is ephemeral). Also available on [`dirsql query`](#dirsql-query), passed after the subcommand. See [Keep the index across restarts](../howto/persist.md). |
@@ -61,25 +61,26 @@ requests, closes open `/events` streams, and exits.
61
61
  **30-second** timeout each, overridable with the config key
62
62
  [`[dirsql].hook-timeout`](./config.md#dirsql-keys).
63
63
 
64
- ### Default mode
64
+ ### Configless mode
65
65
 
66
- With no `-c/--config`, the server serves the **baked-in default** — the
67
- shipped config compiled into the binary, indexing the invocation directory
68
- with a single table named `files`. This is a fixed default, *not* a
69
- `.dirsql.toml` read from disk: a `./.dirsql.toml` sitting in the current
70
- directory is **not** auto-loaded (pass it with `-c ./.dirsql.toml` to use
71
- it). The default `files` table:
66
+ With no `-c/--config`, the server indexes the invocation directory but
67
+ defines **no named tables**. Filesystem queries go through
68
+ [path-tables](./path-tables.md): a quoted path in place of a table name,
69
+ scanned live. A `./.dirsql.toml` sitting in the current directory is **not**
70
+ auto-loaded (pass it with `-c ./.dirsql.toml` to use it).
72
71
 
73
- - Glob: `**/*` — every file under the root, at any depth, no ignores.
74
- - One row per file, with all seven
75
- [stat columns](./columns.md): `path`, `basename`, `dir`, `ext`,
76
- `size`, `mtime`, `ctime`.
72
+ `'./'` is the whole root — every file at any depth, one row per file, with
73
+ all seven [stat columns](./columns.md): `path`, `basename`, `dir`, `ext`,
74
+ `size`, `mtime`, `ctime`.
77
75
 
78
76
  ```bash
79
77
  curl -s localhost:7117/query -H 'content-type: application/json' \
80
- -d '{"sql":"SELECT basename, size FROM files ORDER BY size DESC LIMIT 5"}'
78
+ -d '{"sql":"SELECT basename, size FROM \'./\' ORDER BY size DESC LIMIT 5"}'
81
79
  ```
82
80
 
81
+ Earlier versions served an implicit table named `files` here. It is gone; a
82
+ `SELECT ... FROM files` with no config now fails and points at `FROM './'`.
83
+
83
84
  Passing a config with `-c` fully overrules this default. A `-c` naming a file
84
85
  that does not exist is an error (not a fallback to the default); a config that
85
86
  exists but fails to load degrades the server (see below).
@@ -111,8 +112,8 @@ non-zero exit with the diagnostic on stderr.
111
112
  Run a SQL query from the shell:
112
113
 
113
114
  ```bash
114
- # No -c: the baked-in default `files` table.
115
- dirsql query "SELECT basename, size FROM files ORDER BY size DESC LIMIT 5"
115
+ # No -c: query the filesystem with a path-table.
116
+ dirsql query "SELECT basename, size FROM './' ORDER BY size DESC LIMIT 5"
116
117
  # [{"basename":"model.bin","size":104857600}, …]
117
118
 
118
119
  # A config table (`posts`) needs its config passed explicitly, AFTER the subcommand.
@@ -135,7 +136,7 @@ response body), and exits `0`.
135
136
  uses**, so behavior is identical to `POST /query` by construction:
136
137
 
137
138
  - **Config discovery** honors `--config` passed after the subcommand (with none
138
- given, the [baked-in default](#default-mode)), and `--extension` overrides,
139
+ given, [no named tables](#configless-mode)), and `--extension` overrides,
139
140
  exactly as server mode does.
140
141
  - **`--persist [<path>]`** is honored, so a repeated `dirsql query` reuses the
141
142
  on-disk cache. Because its value is optional, place a bare `--persist` after
@@ -163,8 +164,8 @@ stderr, with exit code `1`.
163
164
 
164
165
  ## `dirsql init`
165
166
 
166
- Writes a starter `.dirsql.toml` — the same table the [baked-in
167
- default](#default-mode) serves — as a scaffold to edit:
167
+ Writes a starter `.dirsql.toml` defining a catch-all `files` table, as a
168
+ scaffold to edit:
168
169
 
169
170
  ```bash
170
171
  dirsql init
@@ -205,7 +206,7 @@ installed in the same environment as `dirsql` (`pip install …`, or
205
206
  loads its fragment — its tables are queryable with zero config edits.
206
207
  Installed = active: there is no enable step and no naming convention. The
207
208
  fragment is composed *after* your own `-c` configs (so your config takes
208
- precedence in ordering), and the baked-in `files` table is preserved.
209
+ precedence in ordering), and the shipped starter `files` table is preserved.
209
210
 
210
211
  Discovery is **launcher-only** — the standalone `cargo`-installed binary does no
211
212
  discovery, and the SDKs never auto-discover (pass a plugin's config explicitly
@@ -8,7 +8,7 @@ all-defaults one. Unknown keys are a parse error at every level (top level,
8
8
  fails loudly, naming the offending key, rather than silently no-opping.
9
9
 
10
10
  The [CLI](./cli.md) loads a config only when you pass it with `-c/--config`;
11
- with none given it serves the [baked-in default](./cli.md#default-mode) (a
11
+ with none given [no named tables](./cli.md#configless-mode) are defined (a
12
12
  `./.dirsql.toml` on disk is **not** auto-loaded). The [SDKs](./sdk.md) load a
13
13
  config via the `config` constructor parameter.
14
14
 
@@ -110,8 +110,8 @@ callback.
110
110
 
111
111
  ```toml
112
112
  [[table]]
113
- ddl = "CREATE TABLE comments (thread_id TEXT, basename TEXT, mtime INTEGER)"
114
- glob = "_comments/{thread_id}/*.jsonl"
113
+ ddl = "CREATE TABLE comments (path TEXT, basename TEXT, mtime INTEGER)"
114
+ glob = "_comments/*/*.jsonl"
115
115
 
116
116
  [[table]]
117
117
  ddl = "CREATE TABLE papers (paper_id TEXT, title TEXT)"
@@ -161,8 +161,8 @@ The configs load and merge in **argv order**:
161
161
  error**, naming the table.
162
162
 
163
163
  The index [root](./cli.md#flags) is the invocation directory regardless of where
164
- any config lives. With no `-c`, the [baked-in default](./cli.md#default-mode) is
165
- served (no `./.dirsql.toml` auto-discovery); a single `-c` behaves exactly as
164
+ any config lives. With no `-c`, [no named tables](./cli.md#configless-mode) are
165
+ defined (no `./.dirsql.toml` auto-discovery); a single `-c` behaves exactly as
166
166
  before.
167
167
 
168
168
  ## Parse errors
@@ -193,8 +193,8 @@ path = "sqlite_vec" # Python module name; on Node use the
193
193
  entrypoint = "sqlite3_vec_init"
194
194
 
195
195
  [[table]]
196
- ddl = "CREATE TABLE comments (thread_id TEXT, basename TEXT, mtime INTEGER)"
197
- glob = "_comments/{thread_id}/*.jsonl"
196
+ ddl = "CREATE TABLE comments (path TEXT, basename TEXT, mtime INTEGER)"
197
+ glob = "_comments/*/*.jsonl"
198
198
 
199
199
  [[table]]
200
200
  ddl = "CREATE TABLE documents (path TEXT, basename TEXT, size INTEGER)"
@@ -32,26 +32,61 @@ Two consequences follow directly:
32
32
 
33
33
  ## Writing the path
34
34
 
35
- The path is relative to the **index root** — the directory dirsql is indexing,
36
- not your shell's working directory.
35
+ A `./` path is relative to the **index root** — the directory dirsql is
36
+ indexing, not your shell's working directory.
37
+
38
+ **Directories are recursive by default.** Naming a directory scans everything
39
+ beneath it; the non-recursive form is spelled explicitly with `*`.
37
40
 
38
41
  | You write | dirsql scans |
39
42
  | --- | --- |
40
43
  | `'./'` | every file under the index root, recursively |
44
+ | `'./docs'` | every file under `docs/`, recursively |
45
+ | `'./*'` | files directly inside the index root, and no deeper |
41
46
  | `'./docs/*.md'` | markdown files directly inside `docs/` |
42
47
  | `'./docs/**/*.md'` | markdown files at any depth under `docs/` |
48
+ | `'./notes/today.md'` | exactly that one file — one file is one row |
49
+
50
+ A path containing `*`, `?` or `[` is a glob and is used exactly as written: `*`
51
+ matches within a single directory, `**` crosses directories.
52
+
53
+ A path naming a single file yields exactly one row. dirsql never splits a file
54
+ into rows on its own — that is what a table's `on_file` hook is for.
43
55
 
44
- The `./` is required. A bare glob is rejected with a hint rather than silently
45
- accepted:
56
+ The `./` is required for index-relative paths. A bare glob is rejected with a
57
+ hint rather than silently accepted:
46
58
 
47
59
  ```
48
60
  SELECT * FROM '**/*.md';
49
61
  -- no such table: **/*.md; did you mean './**/*.md'?
50
62
  ```
51
63
 
52
- Absolute (`/var/log/*.log`), parent-relative (`../notes`) and home-relative
53
- (`~/notes`) path-tables are recognized but not yet resolved; they report that
54
- they are unsupported rather than returning wrong rows.
64
+ ### Paths outside the index root
65
+
66
+ Three other prefixes resolve, with their usual shell meanings:
67
+
68
+ | You write | dirsql scans |
69
+ | --- | --- |
70
+ | `'/var/log/*.log'` | an absolute path |
71
+ | `'../notes'` | relative to the index root's parent |
72
+ | `'~/notes/*.md'` | relative to your home directory |
73
+
74
+ `..` is folded out textually, not followed through symlinks, so the directory
75
+ scanned is a function of the string you wrote.
76
+
77
+ **These report absolute `path` values.** A `./` path-table reports paths
78
+ relative to the index root, matching every other dirsql table; a `/`, `../` or
79
+ `~/` path-table has no meaningful relative base — the root it scans is derived
80
+ from the pattern, not named by you — so it reports the full path instead. The
81
+ value you get back is one you can paste into another command:
82
+
83
+ ```sql
84
+ SELECT path FROM '/var/log/*.log';
85
+ -- /var/log/syslog
86
+ ```
87
+
88
+ On a system with no home directory, a `~/` path-table reports that it cannot
89
+ resolve rather than guessing.
55
90
 
56
91
  ## Columns
57
92
 
@@ -60,7 +95,7 @@ table:
60
95
 
61
96
  | Column | Type | Meaning |
62
97
  | --- | --- | --- |
63
- | `path` | TEXT | path relative to the index root |
98
+ | `path` | TEXT | path relative to the index root (absolute for `/`, `../`, `~/` tables) |
64
99
  | `basename` | TEXT | filename with extension |
65
100
  | `dir` | TEXT | parent directory, relative to the index root |
66
101
  | `ext` | TEXT | extension without the dot |
@@ -89,6 +124,26 @@ Path-tables are per-connection and are never written to a persistent cache, so
89
124
  they cannot leak into `sqlite_master` or survive a restart. The reserved
90
125
  top-level `.dirsql/` directory is excluded from the scan, as everywhere else.
91
126
 
127
+ ## Skip rules
128
+
129
+ A path-table scan applies the same [`ignore`](/reference/config) patterns your
130
+ declared tables use, plus two built-in defaults so a zero-config
131
+ `SELECT * FROM './'` does not drown in machinery:
132
+
133
+ - `node_modules/**`
134
+ - `.git/**`
135
+
136
+ Skip rules are judged on the part of the path *below* what you named outright,
137
+ so pointing at a skipped directory still scans it:
138
+
139
+ ```sql
140
+ SELECT path FROM './'; -- no node_modules rows
141
+ SELECT path FROM './node_modules/*/package.json'; -- scans it anyway
142
+ ```
143
+
144
+ Dotfiles are ordinary files: `'./'` and `'./*'` include them. Add an `ignore`
145
+ pattern if you would rather not see them.
146
+
92
147
  ## Joining against declared tables
93
148
 
94
149
  Path-tables are ordinary SQLite tables once resolved, so they join freely:
@@ -97,10 +97,12 @@ Creates a SQLite index over a directory. The index root is the explicit
97
97
  `root` when given, else the **process cwd** — the `config` file's location
98
98
  never sets the root.
99
99
 
100
- Constructing with **neither a `config` nor programmatic `tables`** serves the
101
- **baked-in default** `files` table (one row per file, over the root) — the
102
- same shipped default the CLI serves with no [`-c`](./cli.md#default-mode), not
103
- an empty index. There is no implicit `<root>/.dirsql.toml` discovery: to read a
100
+ Constructing with **neither a `config` nor programmatic `tables`** defines
101
+ **no named tables** — the same as the CLI with no
102
+ [`-c`](./cli.md#configless-mode). Query the filesystem with a
103
+ [path-table](./path-tables.md) (`SELECT * FROM './'`); a `SELECT ... FROM
104
+ files` in that state fails with a hint pointing at that form. There is no
105
+ implicit `<root>/.dirsql.toml` discovery: to read a
104
106
  config on disk, pass its path via `config` (Rust: `.config(path)` /
105
107
  `DirSQL::from_config_path(path)`). The root-joining `DirSQL::from_config(root)`
106
108
  shortcut was removed in #603 — use
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dirsql",
3
- "version": "0.3.117",
3
+ "version": "0.3.119",
4
4
  "description": "Ephemeral SQL index over a local directory",
5
5
  "license": "MIT",
6
6
  "repository": "https://github.com/thekevinscott/dirsql",
@@ -212,15 +212,15 @@
212
212
  ]
213
213
  },
214
214
  "optionalDependencies": {
215
- "@dirsql/lib-linux-x64-gnu": "0.3.117",
216
- "@dirsql/lib-linux-arm64-gnu": "0.3.117",
217
- "@dirsql/lib-darwin-x64": "0.3.117",
218
- "@dirsql/lib-darwin-arm64": "0.3.117",
219
- "@dirsql/lib-win32-x64-msvc": "0.3.117",
220
- "@dirsql/cli-linux-x64-gnu": "0.3.117",
221
- "@dirsql/cli-linux-arm64-gnu": "0.3.117",
222
- "@dirsql/cli-darwin-x64": "0.3.117",
223
- "@dirsql/cli-darwin-arm64": "0.3.117",
224
- "@dirsql/cli-win32-x64-msvc": "0.3.117"
215
+ "@dirsql/lib-linux-x64-gnu": "0.3.119",
216
+ "@dirsql/lib-linux-arm64-gnu": "0.3.119",
217
+ "@dirsql/lib-darwin-x64": "0.3.119",
218
+ "@dirsql/lib-darwin-arm64": "0.3.119",
219
+ "@dirsql/lib-win32-x64-msvc": "0.3.119",
220
+ "@dirsql/cli-linux-x64-gnu": "0.3.119",
221
+ "@dirsql/cli-linux-arm64-gnu": "0.3.119",
222
+ "@dirsql/cli-darwin-x64": "0.3.119",
223
+ "@dirsql/cli-darwin-arm64": "0.3.119",
224
+ "@dirsql/cli-win32-x64-msvc": "0.3.119"
225
225
  }
226
226
  }