dirsql 0.3.112 → 0.3.114
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/getting-started.md +9 -12
- package/docs/howto/columns-from-paths.md +2 -2
- package/docs/howto/define-tables.md +1 -1
- package/docs/howto/extract-from-contents.md +2 -2
- package/docs/howto/load-extension.md +1 -1
- package/docs/howto/search-by-meaning.md +9 -2
- package/docs/howto/skip-files.md +1 -1
- package/docs/howto/write-a-plugin.md +239 -0
- package/docs/index.md +2 -0
- package/docs/reference/cli.md +43 -7
- package/docs/reference/http-api.md +5 -1
- package/docs/reference/sdk.md +4 -1
- package/package.json +11 -11
package/docs/getting-started.md
CHANGED
|
@@ -9,6 +9,10 @@ a SQL database you can query over HTTP — without writing any code. You will:
|
|
|
9
9
|
|
|
10
10
|
It takes about five minutes.
|
|
11
11
|
|
|
12
|
+
`dirsql` only ever reads your files — it never writes, moves, or changes
|
|
13
|
+
them — so it is safe to point at a real directory of your own once you are
|
|
14
|
+
done here. See [Read-only by design](./explanation#read-only-by-design).
|
|
15
|
+
|
|
12
16
|
**You need:** a terminal with `curl` and [`jq`](https://jqlang.org/), and
|
|
13
17
|
Node ≥ 20.11 (for `npx`). Every `npx dirsql` step below also has a `uvx`
|
|
14
18
|
tab that behaves identically, if you prefer Python tooling
|
|
@@ -16,33 +20,23 @@ tab that behaves identically, if you prefer Python tooling
|
|
|
16
20
|
|
|
17
21
|
## 1. Create three files
|
|
18
22
|
|
|
19
|
-
|
|
23
|
+
Paste this whole block into your terminal. It makes a working directory
|
|
24
|
+
with a subfolder per note author and writes three tiny markdown notes:
|
|
20
25
|
|
|
21
26
|
```bash
|
|
22
27
|
mkdir -p my-notes/notes/alice my-notes/notes/bob
|
|
23
28
|
cd my-notes
|
|
24
|
-
```
|
|
25
|
-
|
|
26
|
-
Create the three notes by pasting each block exactly as shown:
|
|
27
|
-
|
|
28
|
-
```bash
|
|
29
29
|
cat > notes/alice/welcome.md <<'EOF'
|
|
30
30
|
# Welcome
|
|
31
31
|
|
|
32
32
|
Start here. This folder is about to become a database.
|
|
33
33
|
EOF
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
```bash
|
|
37
34
|
cat > notes/alice/ideas.md <<'EOF'
|
|
38
35
|
# Ideas
|
|
39
36
|
|
|
40
37
|
- query files with SQL
|
|
41
38
|
- watch for changes
|
|
42
39
|
EOF
|
|
43
|
-
```
|
|
44
|
-
|
|
45
|
-
```bash
|
|
46
40
|
cat > notes/bob/reading-list.md <<'EOF'
|
|
47
41
|
# Reading list
|
|
48
42
|
|
|
@@ -50,6 +44,9 @@ cat > notes/bob/reading-list.md <<'EOF'
|
|
|
50
44
|
EOF
|
|
51
45
|
```
|
|
52
46
|
|
|
47
|
+
(Any directory of files works with `dirsql` — the rest of this tutorial
|
|
48
|
+
assumes exactly these three so your output matches ours.)
|
|
49
|
+
|
|
53
50
|
Check that all three files are in place:
|
|
54
51
|
|
|
55
52
|
```bash
|
|
@@ -33,7 +33,7 @@ Pass the config with [`-c`](../reference/cli.md#flags) (`dirsql` does not
|
|
|
33
33
|
auto-load a `.dirsql.toml` from the current directory):
|
|
34
34
|
|
|
35
35
|
```bash
|
|
36
|
-
dirsql
|
|
36
|
+
dirsql query "SELECT year, month, basename FROM photos ORDER BY year, month" -c ./.dirsql.toml
|
|
37
37
|
```
|
|
38
38
|
|
|
39
39
|
```json
|
|
@@ -43,7 +43,7 @@ dirsql -c ./.dirsql.toml query "SELECT year, month, basename FROM photos ORDER B
|
|
|
43
43
|
Captures are real SQL columns, so aggregation works:
|
|
44
44
|
|
|
45
45
|
```bash
|
|
46
|
-
dirsql
|
|
46
|
+
dirsql query "SELECT year, COUNT(*) AS photos FROM photos GROUP BY year" -c ./.dirsql.toml
|
|
47
47
|
```
|
|
48
48
|
|
|
49
49
|
```json
|
|
@@ -31,7 +31,7 @@ auto-load a `.dirsql.toml` from the current directory. Each matched file is
|
|
|
31
31
|
one row:
|
|
32
32
|
|
|
33
33
|
```bash
|
|
34
|
-
dirsql
|
|
34
|
+
dirsql query "SELECT path, size FROM posts ORDER BY path" -c ./.dirsql.toml
|
|
35
35
|
```
|
|
36
36
|
|
|
37
37
|
```json
|
|
@@ -35,7 +35,7 @@ Pass the config with [`-c`](../reference/cli.md#flags) (`dirsql` does not
|
|
|
35
35
|
auto-load a `.dirsql.toml` from the current directory):
|
|
36
36
|
|
|
37
37
|
```bash
|
|
38
|
-
dirsql
|
|
38
|
+
dirsql query "SELECT title, author, year, path FROM books ORDER BY year" -c ./.dirsql.toml
|
|
39
39
|
```
|
|
40
40
|
|
|
41
41
|
```json
|
|
@@ -60,7 +60,7 @@ on-file = "jq -c -s '.' {path}"
|
|
|
60
60
|
```
|
|
61
61
|
|
|
62
62
|
```bash
|
|
63
|
-
dirsql
|
|
63
|
+
dirsql query "SELECT event, user FROM events" -c ./.dirsql.toml
|
|
64
64
|
```
|
|
65
65
|
|
|
66
66
|
```json
|
|
@@ -27,7 +27,7 @@ The extension's functions are callable (pass the config with
|
|
|
27
27
|
[`-c`](../reference/cli.md#flags) so its `[[dirsql.extension]]` entry loads):
|
|
28
28
|
|
|
29
29
|
```bash
|
|
30
|
-
dirsql
|
|
30
|
+
dirsql query "SELECT vec_version() AS vec_version" -c ./.dirsql.toml
|
|
31
31
|
```
|
|
32
32
|
|
|
33
33
|
```json
|
|
@@ -7,6 +7,13 @@ into semantic search: a SQLite vector extension for the distance math, an
|
|
|
7
7
|
index time, and a [`pre-query`](../reference/hooks.md#pre-query) command to
|
|
8
8
|
embed each question at query time.
|
|
9
9
|
|
|
10
|
+
::: tip Just want it working?
|
|
11
|
+
[`dirsql-plugin-embeddings`](https://pypi.org/project/dirsql-plugin-embeddings/)
|
|
12
|
+
packages exactly what this guide builds, ready to install:
|
|
13
|
+
`uvx --with dirsql-plugin-embeddings dirsql`. Keep reading to see how it's
|
|
14
|
+
built — the same three pieces, from scratch.
|
|
15
|
+
:::
|
|
16
|
+
|
|
10
17
|
## How the pieces fit
|
|
11
18
|
|
|
12
19
|
1. **[`[[dirsql.extension]]`](../reference/config.md#dirsql-extension)**
|
|
@@ -108,7 +115,7 @@ scan runs `embed.py` once per note, then the query argument goes straight to
|
|
|
108
115
|
`.dirsql.toml` from the current directory:
|
|
109
116
|
|
|
110
117
|
```bash
|
|
111
|
-
uvx --with sqlite-vec dirsql
|
|
118
|
+
uvx --with sqlite-vec dirsql query '{"q": "how do I cook pasta?"}' -c ./.dirsql.toml
|
|
112
119
|
```
|
|
113
120
|
|
|
114
121
|
```json
|
|
@@ -116,7 +123,7 @@ uvx --with sqlite-vec dirsql -c ./.dirsql.toml query '{"q": "how do I cook pasta
|
|
|
116
123
|
```
|
|
117
124
|
|
|
118
125
|
```bash
|
|
119
|
-
uvx --with sqlite-vec dirsql
|
|
126
|
+
uvx --with sqlite-vec dirsql query '{"q": "reviewing code on github"}' -c ./.dirsql.toml
|
|
120
127
|
```
|
|
121
128
|
|
|
122
129
|
```json
|
package/docs/howto/skip-files.md
CHANGED
|
@@ -35,7 +35,7 @@ Pass the config with [`-c`](../reference/cli.md#flags) (`dirsql` does not
|
|
|
35
35
|
auto-load a `.dirsql.toml` from the current directory):
|
|
36
36
|
|
|
37
37
|
```bash
|
|
38
|
-
dirsql
|
|
38
|
+
dirsql query "SELECT path FROM notes ORDER BY path" -c ./.dirsql.toml
|
|
39
39
|
```
|
|
40
40
|
|
|
41
41
|
```json
|
|
@@ -0,0 +1,239 @@
|
|
|
1
|
+
# Write a plugin
|
|
2
|
+
|
|
3
|
+
Ship a ready-made set of `dirsql` tables — an embeddings index, a
|
|
4
|
+
log-line extractor, a metrics view — so a teammate gets them by installing a
|
|
5
|
+
package, with zero config edits. A **plugin** is an ordinary pip package that
|
|
6
|
+
carries a `dirsql.toml` config fragment and declares itself with a `dirsql`
|
|
7
|
+
entry point; when it is installed in the same environment as the `uvx`/`pip`
|
|
8
|
+
launcher, the launcher [discovers it and loads its fragment
|
|
9
|
+
automatically](../reference/cli.md#plugins). Nothing in `dirsql` is
|
|
10
|
+
plugin-aware — a plugin is just config plus a naming convention.
|
|
11
|
+
|
|
12
|
+
## Package layout
|
|
13
|
+
|
|
14
|
+
A plugin is a normal Python package. Two things make it a plugin:
|
|
15
|
+
|
|
16
|
+
1. A **`dirsql.toml`** fragment shipped inside a top-level module.
|
|
17
|
+
2. A **`[project.entry-points.dirsql]`** declaration pointing the launcher at
|
|
18
|
+
that module.
|
|
19
|
+
|
|
20
|
+
```
|
|
21
|
+
dirsql-embeddings/
|
|
22
|
+
├── pyproject.toml
|
|
23
|
+
└── src/
|
|
24
|
+
└── dirsql_embeddings/
|
|
25
|
+
├── __init__.py
|
|
26
|
+
├── dirsql.toml # the config fragment
|
|
27
|
+
├── embed.py # on-file hook
|
|
28
|
+
└── search.py # pre-query hook
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The entry point maps a **source label** (the name) to the **module that
|
|
32
|
+
ships `dirsql.toml`** (the value):
|
|
33
|
+
|
|
34
|
+
```toml
|
|
35
|
+
# pyproject.toml
|
|
36
|
+
[project]
|
|
37
|
+
name = "dirsql-embeddings"
|
|
38
|
+
version = "0.1.0"
|
|
39
|
+
dependencies = ["model2vec", "sqlite-vec"]
|
|
40
|
+
|
|
41
|
+
[project.entry-points.dirsql]
|
|
42
|
+
embeddings = "dirsql_embeddings"
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
The entry-point **name** (`embeddings`) is the source label the launcher uses
|
|
46
|
+
to identify the plugin in diagnostics; the **value** (`dirsql_embeddings`) is
|
|
47
|
+
the importable module the launcher resolves to find `dirsql.toml` beside it.
|
|
48
|
+
Installing the package is all it takes — there is no enable step and no
|
|
49
|
+
filename convention beyond `dirsql.toml`.
|
|
50
|
+
|
|
51
|
+
## The fragment is an ordinary config
|
|
52
|
+
|
|
53
|
+
A plugin's `dirsql.toml` is a plain [config file](../reference/config.md) —
|
|
54
|
+
structurally **identical to a user config**. It may declare
|
|
55
|
+
[`[[table]]`](../reference/config.md#table) (with
|
|
56
|
+
[`on-file`](../reference/hooks.md#on-file)),
|
|
57
|
+
[`[[dirsql.extension]]`](../reference/config.md#dirsql-extension),
|
|
58
|
+
[`ignore`](../reference/config.md#dirsql-keys),
|
|
59
|
+
[`pre-query`/`post-query`](../reference/hooks.md#pre-query), and
|
|
60
|
+
[`hook-timeout`](../reference/hooks.md#timeout).
|
|
61
|
+
|
|
62
|
+
There are **no plugin-specific keys and no plugin-specific restrictions**. The
|
|
63
|
+
config schema is content-only: the index `root` and `--persist` are
|
|
64
|
+
[runner-owned flags](../reference/config.md#dirsql-keys) (`--root`, `--persist
|
|
65
|
+
[PATH]`), decided by whoever runs `dirsql`, never by a config file — so a
|
|
66
|
+
plugin has nothing to say about them. Whatever you can put in your own
|
|
67
|
+
`.dirsql.toml`, a plugin can put in its fragment, and vice-versa.
|
|
68
|
+
|
|
69
|
+
## Hook commands
|
|
70
|
+
|
|
71
|
+
Both hook command styles from the [hook contract](../reference/hooks.md) work
|
|
72
|
+
in a plugin fragment:
|
|
73
|
+
|
|
74
|
+
- **Console scripts** — a `bin`-style entry point your package installs on
|
|
75
|
+
`PATH` (`embed-file {path}`). Recommended for published plugins: the command
|
|
76
|
+
is bound to your package's interpreter and dependencies, and it is
|
|
77
|
+
language-neutral (the fragment names a command, not a Python file).
|
|
78
|
+
- **Relative scripts** — a path resolved against the fragment's own directory
|
|
79
|
+
(`uv run python embed.py {path}`). Convenient while developing the plugin
|
|
80
|
+
in-tree.
|
|
81
|
+
|
|
82
|
+
Two facts from the [execution contract](../reference/hooks.md#execution-contract)
|
|
83
|
+
matter most for a published plugin:
|
|
84
|
+
|
|
85
|
+
- **A hook runs in its declaring config's directory.** For a plugin that is
|
|
86
|
+
the installed fragment's directory — inside **site-packages**. That is a
|
|
87
|
+
read-only, shared location: **run from it, never write to it.** Use the
|
|
88
|
+
absolute [`{path}`](../reference/hooks.md#on-file) placeholder to read the
|
|
89
|
+
matched file, and [`{root}`](../reference/hooks.md#on-file) to reach the
|
|
90
|
+
user's project directory. Write any cache to `{root}` or a real cache dir,
|
|
91
|
+
never next to the fragment.
|
|
92
|
+
- **`{path}` is absolute and `{root}` is the index root**, so a command is
|
|
93
|
+
self-sufficient from any working directory — it works whether the plugin
|
|
94
|
+
lives in the project or in site-packages.
|
|
95
|
+
|
|
96
|
+
## Worked example: an embeddings plugin
|
|
97
|
+
|
|
98
|
+
Here is the whole plugin — vector search over a directory of notes, buildable
|
|
99
|
+
in about fifty lines. It composes the same three pieces as
|
|
100
|
+
[Search documents by meaning](./search-by-meaning.md): the
|
|
101
|
+
[`sqlite-vec`](https://github.com/asg017/sqlite-vec) extension for the
|
|
102
|
+
distance math, an `on-file` command to embed each file, and a `pre-query`
|
|
103
|
+
command to embed the question.
|
|
104
|
+
|
|
105
|
+
The fragment, `src/dirsql_embeddings/dirsql.toml`:
|
|
106
|
+
|
|
107
|
+
```toml
|
|
108
|
+
[dirsql]
|
|
109
|
+
pre-query = "uv run --with model2vec python search.py {args}"
|
|
110
|
+
hook-timeout = 300 # headroom for the first-run model download
|
|
111
|
+
|
|
112
|
+
[[dirsql.extension]]
|
|
113
|
+
path = "sqlite_vec"
|
|
114
|
+
entrypoint = "sqlite3_vec_init"
|
|
115
|
+
|
|
116
|
+
[[table]]
|
|
117
|
+
ddl = "CREATE TABLE notes (path TEXT, text TEXT, embedding TEXT)"
|
|
118
|
+
glob = "notes/*.md"
|
|
119
|
+
on-file = "uv run --with model2vec python embed.py {path}"
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
`embed.py` turns one file into one row carrying its text and its embedding:
|
|
123
|
+
|
|
124
|
+
```python
|
|
125
|
+
"""Embed one file's text; print a dirsql row array on stdout."""
|
|
126
|
+
import json
|
|
127
|
+
import sys
|
|
128
|
+
|
|
129
|
+
from model2vec import StaticModel
|
|
130
|
+
|
|
131
|
+
path = sys.argv[1]
|
|
132
|
+
text = open(path, encoding="utf-8").read()
|
|
133
|
+
model = StaticModel.from_pretrained("minishlab/potion-base-8M")
|
|
134
|
+
vector = model.encode([text])[0]
|
|
135
|
+
print(json.dumps([{"text": text, "embedding": json.dumps([round(float(x), 6) for x in vector])}]))
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
`search.py` turns a `{"q": "..."}` request body into nearest-neighbor SQL:
|
|
139
|
+
|
|
140
|
+
```python
|
|
141
|
+
"""Turn a {"q": "..."} request body into a nearest-neighbor SQL query."""
|
|
142
|
+
import json
|
|
143
|
+
import sys
|
|
144
|
+
|
|
145
|
+
from model2vec import StaticModel
|
|
146
|
+
|
|
147
|
+
body = json.loads(sys.argv[1])
|
|
148
|
+
model = StaticModel.from_pretrained("minishlab/potion-base-8M")
|
|
149
|
+
vector = model.encode([body["q"]])[0]
|
|
150
|
+
needle = json.dumps([round(float(x), 6) for x in vector])
|
|
151
|
+
print(
|
|
152
|
+
"SELECT path, ROUND(vec_distance_cosine(embedding, '%s'), 3) AS distance "
|
|
153
|
+
"FROM notes ORDER BY distance LIMIT 3" % needle
|
|
154
|
+
)
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
::: warning The hook owns SQL safety
|
|
158
|
+
Whatever SQL `pre-query` prints is executed as-is. Here the interpolated
|
|
159
|
+
value is a numeric vector the script itself produced; never splice raw request
|
|
160
|
+
text into SQL. See [`pre-query`](../reference/hooks.md#pre-query).
|
|
161
|
+
:::
|
|
162
|
+
|
|
163
|
+
The relative `embed.py` / `search.py` above resolve against the fragment
|
|
164
|
+
directory, which is convenient during development. For a published plugin,
|
|
165
|
+
promote them to console scripts (`[project.scripts]` → `embed-file`,
|
|
166
|
+
`embed-search`) so the commands carry their own interpreter and dependencies
|
|
167
|
+
and no longer depend on `uv run --with`.
|
|
168
|
+
|
|
169
|
+
Once the package is installed alongside the launcher, its `notes` table is
|
|
170
|
+
queryable with no config edits:
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
uvx --with dirsql-embeddings dirsql query '{"q": "how do I cook pasta?"}'
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
The launcher discovers the installed plugin, composes its fragment, and the
|
|
177
|
+
`pre-query` hook turns the question into vector-distance SQL.
|
|
178
|
+
|
|
179
|
+
## SDK-style convention: expose the config
|
|
180
|
+
|
|
181
|
+
The launcher's auto-discovery is [launcher-only](#boundaries): an
|
|
182
|
+
[SDK](../reference/sdk.md) consumer never gets a plugin's tables automatically.
|
|
183
|
+
The convention that bridges this — **zero `dirsql` code, purely a plugin
|
|
184
|
+
courtesy** — is to also expose the fragment's path programmatically, so an
|
|
185
|
+
application can pass it to the constructor's
|
|
186
|
+
[`config`](../reference/sdk.md#constructor) parameter by hand:
|
|
187
|
+
|
|
188
|
+
```python
|
|
189
|
+
# src/dirsql_embeddings/__init__.py
|
|
190
|
+
from importlib.resources import files
|
|
191
|
+
|
|
192
|
+
def config_path() -> str:
|
|
193
|
+
"""Absolute path to this plugin's dirsql.toml, for SDK consumers."""
|
|
194
|
+
return str(files(__package__) / "dirsql.toml")
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
```python
|
|
198
|
+
from dirsql import DirSQL
|
|
199
|
+
from dirsql_embeddings import config_path
|
|
200
|
+
|
|
201
|
+
db = DirSQL("./project", config=config_path())
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
This is not a `dirsql` feature — it is a naming convention a well-behaved
|
|
205
|
+
plugin follows so its config is reachable both ways: auto-discovered by the
|
|
206
|
+
launcher, and hand-passed to an SDK.
|
|
207
|
+
|
|
208
|
+
## Boundaries
|
|
209
|
+
|
|
210
|
+
Discovery is deliberately narrow. Know exactly who does what:
|
|
211
|
+
|
|
212
|
+
- **The `cargo`-installed binary does no discovery.** It loads configs only
|
|
213
|
+
from [`-c/--config`](../reference/cli.md#flags). A standalone binary user
|
|
214
|
+
passes a plugin's fragment explicitly, like any other config.
|
|
215
|
+
- **The `uvx`/`pip` launcher injects `-c <fragment>` per installed plugin.**
|
|
216
|
+
Each discovered plugin's fragment is appended after your own `-c` configs, so
|
|
217
|
+
your config still takes ordering precedence
|
|
218
|
+
([composing configs](../reference/config.md#composing-multiple-configs)). When
|
|
219
|
+
you pass no `-c` of your own, the launcher also keeps the
|
|
220
|
+
[baked-in default](../reference/cli.md#default-mode) `files` table (an
|
|
221
|
+
internal `--include-default`), so plugins **add** tables rather than
|
|
222
|
+
replacing the default. Discovery is **pip/uvx only** for now — the `npx`
|
|
223
|
+
launcher does not yet discover — and is switched off per invocation with
|
|
224
|
+
[`--no-plugin` or `DIRSQL_NO_PLUGIN=1`](../reference/cli.md#plugins).
|
|
225
|
+
- **The SDK never auto-discovers.** Pass a plugin's config explicitly (the
|
|
226
|
+
[convention above](#sdk-style-convention-expose-the-config)).
|
|
227
|
+
- **Name collisions are a hard error.** Because fragments compose like any
|
|
228
|
+
[multiple configs](../reference/config.md#composing-multiple-configs), two
|
|
229
|
+
plugins (or a plugin and your config) defining a table of the same name — or
|
|
230
|
+
the same query hook — fail loudly, naming the conflict. It is never a silent
|
|
231
|
+
last-writer-wins.
|
|
232
|
+
|
|
233
|
+
::: warning Config flags are subcommand-local
|
|
234
|
+
For `dirsql query`, pass `-c` and friends **after** the subcommand
|
|
235
|
+
(`dirsql query "<sql>" -c <cfg>`); before it they are a hard error. Plugin
|
|
236
|
+
discovery is unaffected — the launcher injects its `-c` flags in the right
|
|
237
|
+
position — but it matters when you also pass a config of your own. See
|
|
238
|
+
[the CLI reference](../reference/cli.md#dirsql-query).
|
|
239
|
+
:::
|
package/docs/index.md
CHANGED
|
@@ -22,6 +22,8 @@ But querying across many files is slow.
|
|
|
22
22
|
|
|
23
23
|
`dirsql` bridges this gap. The filesystem remains the source of truth, but you get SQL queries and real-time change events for free. Define tables with glob patterns and on-file callbacks, and `dirsql` handles the rest.
|
|
24
24
|
|
|
25
|
+
**`dirsql` never modifies your files.** It opens them for reading and nothing else — no writes, no moves, no deletes, no rewrites in place. Point it at anything and the worst it can do is read. This is permanent by design, not unimplemented; see [Read-only by design](./explanation#read-only-by-design) for its exact scope.
|
|
26
|
+
|
|
25
27
|
::: code-group
|
|
26
28
|
|
|
27
29
|
```python [Python]
|
package/docs/reference/cli.md
CHANGED
|
@@ -48,7 +48,7 @@ requests, closes open `/events` streams, and exits.
|
|
|
48
48
|
| `-c, --config <path>` | baked-in default | Path to a [config file](./config.md). **Repeatable** (`-c a -c b`): the configs load and merge in argv order — see [Composing multiple configs](./config.md#composing-multiple-configs). The index is always rooted at the **invocation directory** (the current working directory), regardless of where a config lives — so `--config /elsewhere/.dirsql.toml` still indexes the directory you ran `dirsql` from. With none given, the [baked-in default](#default-mode) `files` table is served — a `./.dirsql.toml` on disk is **not** auto-loaded; pass it explicitly. A `-c` naming a file that does not exist is an [error](#degraded-mode), not a silent fallback to the default. |
|
|
49
49
|
| `--host <addr>` | `localhost` | Bind address. |
|
|
50
50
|
| `--port <n>` | `7117` | TCP port to bind. |
|
|
51
|
-
| `--persist [<path>]` | off | Keep the SQLite index on disk between runs so a restart only re-parses files that actually changed. Bare `--persist` caches at `<root>/.dirsql/cache.db`; `--persist <path>` caches at `<path>`. Off by default (the index is ephemeral).
|
|
51
|
+
| `--persist [<path>]` | off | Keep the SQLite index on disk between runs so a restart only re-parses files that actually changed. Bare `--persist` caches at `<root>/.dirsql/cache.db`; `--persist <path>` caches at `<path>`. Off by default (the index is ephemeral). Also available on [`dirsql query`](#dirsql-query), passed after the subcommand. See [Keep the index across restarts](../howto/persist.md). |
|
|
52
52
|
| `--extension <path>` | none | Load a SQLite extension by literal path, overriding the config's `[[dirsql.extension]]` entries. Repeatable. Format: `<path>` or `<path>::<entrypoint>`. Internal plumbing for the pip/npm launchers, which resolve package-name extensions and pass the resolved paths here — not intended for direct use. When any `--extension` is present, the config file's own extension entries are not loaded. |
|
|
53
53
|
| `--version` | | Print the version and exit. |
|
|
54
54
|
| `--help` | | Print usage and exit. |
|
|
@@ -115,10 +115,18 @@ Run a SQL query from the shell:
|
|
|
115
115
|
dirsql query "SELECT basename, size FROM files ORDER BY size DESC LIMIT 5"
|
|
116
116
|
# [{"basename":"model.bin","size":104857600}, …]
|
|
117
117
|
|
|
118
|
-
# A config table (`posts`) needs its config passed explicitly.
|
|
119
|
-
dirsql
|
|
118
|
+
# A config table (`posts`) needs its config passed explicitly, AFTER the subcommand.
|
|
119
|
+
dirsql query "SELECT COUNT(*) AS n FROM posts" -c ./.dirsql.toml | jq '.[0].n'
|
|
120
120
|
```
|
|
121
121
|
|
|
122
|
+
::: warning Config flags are subcommand-local
|
|
123
|
+
Pass `-c`/`--config`, `--persist`, and `--extension` **after** `query`
|
|
124
|
+
(`dirsql query "<sql>" -c <cfg>`). A config flag placed *before* the subcommand
|
|
125
|
+
is a hard error — `error: the subcommand 'query' cannot be used with
|
|
126
|
+
'--config <CONFIG>'` — never silently dropped. (In server mode, with no
|
|
127
|
+
subcommand, the same flags are passed directly: `dirsql -c <cfg>`.)
|
|
128
|
+
:::
|
|
129
|
+
|
|
122
130
|
The subcommand builds the index, runs the SQL, prints the result rows as a
|
|
123
131
|
JSON array on stdout (byte-identical to the [`POST /query`](./http-api.md)
|
|
124
132
|
response body), and exits `0`.
|
|
@@ -126,9 +134,9 @@ response body), and exits `0`.
|
|
|
126
134
|
`dirsql query` is a thin adapter over the **same query pipeline the server
|
|
127
135
|
uses**, so behavior is identical to `POST /query` by construction:
|
|
128
136
|
|
|
129
|
-
- **Config discovery** honors `--config` (with none
|
|
130
|
-
[baked-in default](#default-mode)), and `--extension` overrides,
|
|
131
|
-
server mode does.
|
|
137
|
+
- **Config discovery** honors `--config` passed after the subcommand (with none
|
|
138
|
+
given, the [baked-in default](#default-mode)), and `--extension` overrides,
|
|
139
|
+
exactly as server mode does.
|
|
132
140
|
- **`--persist [<path>]`** is honored, so a repeated `dirsql query` reuses the
|
|
133
141
|
on-disk cache. Because its value is optional, place a bare `--persist` after
|
|
134
142
|
the SQL (`dirsql query "SELECT …" --persist`) or use the `=` form
|
|
@@ -138,7 +146,9 @@ uses**, so behavior is identical to `POST /query` by construction:
|
|
|
138
146
|
[`[dirsql].hook-timeout`](./config.md#dirsql-keys) apply identically.
|
|
139
147
|
- The **30-second query timeout**, the **read-only rule**, and the
|
|
140
148
|
`_dirsql_*` **internal-table denial** apply identically. A rejected read
|
|
141
|
-
is an error, not empty output.
|
|
149
|
+
is an error, not empty output. The read-only rule here governs SQL
|
|
150
|
+
statements; dirsql separately never modifies the files it indexes — see
|
|
151
|
+
[Read-only by design](../explanation#read-only-by-design).
|
|
142
152
|
|
|
143
153
|
Errors print the same diagnostic the HTTP `{"error": …}` body carries —
|
|
144
154
|
config failures, SQL errors, rejected reads, hook failures, timeouts — to
|
|
@@ -185,3 +195,29 @@ All failures exit `1` with a message on stderr:
|
|
|
185
195
|
| Output path unwritable (e.g. missing parent directory) | Fails with the underlying I/O error. |
|
|
186
196
|
|
|
187
197
|
On success, `init` exits `0`.
|
|
198
|
+
|
|
199
|
+
## Plugins
|
|
200
|
+
|
|
201
|
+
A **plugin** is an ordinary Python package that ships a `dirsql.toml` config
|
|
202
|
+
fragment and declares itself via a `dirsql` entry point. When such a package is
|
|
203
|
+
installed in the same environment as `dirsql` (`pip install …`, or
|
|
204
|
+
`uvx --with …`), the `uvx`/`pip` launcher **discovers it automatically** and
|
|
205
|
+
loads its fragment — its tables are queryable with zero config edits.
|
|
206
|
+
Installed = active: there is no enable step and no naming convention. The
|
|
207
|
+
fragment is composed *after* your own `-c` configs (so your config takes
|
|
208
|
+
precedence in ordering), and the baked-in `files` table is preserved.
|
|
209
|
+
|
|
210
|
+
Discovery is **launcher-only** — the standalone `cargo`-installed binary does no
|
|
211
|
+
discovery, and the SDKs never auto-discover (pass a plugin's config explicitly
|
|
212
|
+
instead). It is **pip/uvx only** for now; the `npx` launcher does not yet
|
|
213
|
+
discover.
|
|
214
|
+
|
|
215
|
+
Turn discovery off with either:
|
|
216
|
+
|
|
217
|
+
| | Effect |
|
|
218
|
+
|---|---|
|
|
219
|
+
| `--no-plugin` | Skip plugin discovery for this invocation. Consumed by the launcher; never forwarded to the binary. |
|
|
220
|
+
| `DIRSQL_NO_PLUGIN=1` | Same, via the environment. |
|
|
221
|
+
|
|
222
|
+
A plugin that declares itself but is missing its module or its `dirsql.toml`
|
|
223
|
+
fragment is a launcher error naming the package — never a silent skip.
|
|
@@ -5,7 +5,11 @@ exposes two endpoints: `POST /query` and `GET /events`.
|
|
|
5
5
|
|
|
6
6
|
## `POST /query`
|
|
7
7
|
|
|
8
|
-
Run a read-only SQL query
|
|
8
|
+
Run a read-only SQL query — statements SQLite classifies as writes are
|
|
9
|
+
rejected. That rule governs SQL against the index; dirsql separately never
|
|
10
|
+
modifies the files it indexes, which is
|
|
11
|
+
[permanent by design](../explanation#read-only-by-design). Request body is
|
|
12
|
+
JSON:
|
|
9
13
|
|
|
10
14
|
```json
|
|
11
15
|
{"sql": "SELECT title, author FROM posts WHERE draft = 0"}
|
package/docs/reference/sdk.md
CHANGED
|
@@ -216,6 +216,9 @@ Executes a SQL query and returns rows keyed by column name.
|
|
|
216
216
|
before producing rows. Rust surfaces this as
|
|
217
217
|
`DirSqlError::WriteForbidden`; Python raises a `RuntimeError` and
|
|
218
218
|
TypeScript rejects with an `Error` carrying a "read-only" message.
|
|
219
|
+
This is the query-layer half of dirsql's broader guarantee that it never
|
|
220
|
+
modifies your files — see
|
|
221
|
+
[Read-only by design](../explanation#read-only-by-design).
|
|
219
222
|
- Internal tracking columns (`_dirsql_file_path`, `_dirsql_row_index`) are
|
|
220
223
|
excluded from `SELECT *` results; name them explicitly to see them.
|
|
221
224
|
- SQLite values map back to language types:
|
|
@@ -341,7 +344,7 @@ Maps files to table rows.
|
|
|
341
344
|
joined with the file's relative path — absolute when `root` is absolute)
|
|
342
345
|
and returning the rows that file contributes. `dirsql` never reads file
|
|
343
346
|
contents itself; a callback that needs the body reads the path. Return
|
|
344
|
-
an empty list to skip a file. [
|
|
347
|
+
an empty list to skip a file. [Stat columns and glob
|
|
345
348
|
captures](./columns.md) are merged onto each returned row; values the
|
|
346
349
|
callback emits win over same-named facts.
|
|
347
350
|
- `strict` — Default off: extra row keys are dropped and missing declared
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dirsql",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.114",
|
|
4
4
|
"description": "Ephemeral SQL index over a local directory",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": "https://github.com/thekevinscott/dirsql",
|
|
@@ -212,15 +212,15 @@
|
|
|
212
212
|
]
|
|
213
213
|
},
|
|
214
214
|
"optionalDependencies": {
|
|
215
|
-
"@dirsql/lib-linux-x64-gnu": "0.3.
|
|
216
|
-
"@dirsql/lib-linux-arm64-gnu": "0.3.
|
|
217
|
-
"@dirsql/lib-darwin-x64": "0.3.
|
|
218
|
-
"@dirsql/lib-darwin-arm64": "0.3.
|
|
219
|
-
"@dirsql/lib-win32-x64-msvc": "0.3.
|
|
220
|
-
"@dirsql/cli-linux-x64-gnu": "0.3.
|
|
221
|
-
"@dirsql/cli-linux-arm64-gnu": "0.3.
|
|
222
|
-
"@dirsql/cli-darwin-x64": "0.3.
|
|
223
|
-
"@dirsql/cli-darwin-arm64": "0.3.
|
|
224
|
-
"@dirsql/cli-win32-x64-msvc": "0.3.
|
|
215
|
+
"@dirsql/lib-linux-x64-gnu": "0.3.114",
|
|
216
|
+
"@dirsql/lib-linux-arm64-gnu": "0.3.114",
|
|
217
|
+
"@dirsql/lib-darwin-x64": "0.3.114",
|
|
218
|
+
"@dirsql/lib-darwin-arm64": "0.3.114",
|
|
219
|
+
"@dirsql/lib-win32-x64-msvc": "0.3.114",
|
|
220
|
+
"@dirsql/cli-linux-x64-gnu": "0.3.114",
|
|
221
|
+
"@dirsql/cli-linux-arm64-gnu": "0.3.114",
|
|
222
|
+
"@dirsql/cli-darwin-x64": "0.3.114",
|
|
223
|
+
"@dirsql/cli-darwin-arm64": "0.3.114",
|
|
224
|
+
"@dirsql/cli-win32-x64-msvc": "0.3.114"
|
|
225
225
|
}
|
|
226
226
|
}
|