dirsql 0.3.70 → 0.3.71
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/docs/explanation.md +5 -0
- package/docs/howto/columns-from-paths.md +63 -0
- package/docs/howto/define-tables.md +70 -0
- package/docs/howto/embed.md +196 -0
- package/docs/howto/extract-from-contents.md +90 -0
- package/docs/howto/load-extension.md +91 -0
- package/docs/howto/persist.md +52 -0
- package/docs/howto/react-to-changes.md +67 -0
- package/docs/howto/search-by-meaning.md +148 -0
- package/docs/howto/skip-files.md +60 -0
- package/docs/reference/cli.md +132 -0
- package/docs/reference/columns.md +68 -0
- package/docs/reference/config.md +166 -0
- package/docs/reference/hooks.md +141 -0
- package/docs/reference/http-api.md +120 -0
- package/docs/reference/sdk.md +357 -0
- package/package.json +11 -11
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
# Command hooks
|
|
2
|
+
|
|
3
|
+
Three [config keys](./config.md) run an external command: `on-file` (per
|
|
4
|
+
`[[table]]`), and the server-wide `pre-query` and `post-query` (under
|
|
5
|
+
`[dirsql]`; CLI server only). All three share one execution contract.
|
|
6
|
+
|
|
7
|
+
## Execution contract
|
|
8
|
+
|
|
9
|
+
### argv, not a shell
|
|
10
|
+
|
|
11
|
+
The command string is split into an argv with shell-like quoting: whitespace
|
|
12
|
+
separates arguments, and single or double quotes group them (so
|
|
13
|
+
`sh -c 'grep foo {path} | sort'` keeps the quoted script as a single
|
|
14
|
+
argument). **No shell is invoked** — there is no globbing, piping, `$VAR`
|
|
15
|
+
expansion, or `&&`/`;` chaining. To get shell features, ask for a shell
|
|
16
|
+
explicitly with `sh -c '…'`.
|
|
17
|
+
|
|
18
|
+
A command that is empty, whitespace-only, or has unbalanced quotes is
|
|
19
|
+
invalid (empty/whitespace commands are already rejected at
|
|
20
|
+
[config parse time](./config.md#parse-errors)).
|
|
21
|
+
|
|
22
|
+
### Placeholders
|
|
23
|
+
|
|
24
|
+
A `{name}` in the command is substituted with its value, in every
|
|
25
|
+
occurrence, within whole argv tokens, in a single left-to-right pass:
|
|
26
|
+
|
|
27
|
+
- A substituted value is always exactly one argv element — a value
|
|
28
|
+
containing spaces, quotes, or shell metacharacters stays a single
|
|
29
|
+
argument. This makes untrusted input (file paths, request bodies)
|
|
30
|
+
injection-safe at the argv level.
|
|
31
|
+
- Substituted values are never re-scanned: a value that itself contains
|
|
32
|
+
`{…}` is inert.
|
|
33
|
+
- An unrecognized `{…}` is left literal.
|
|
34
|
+
|
|
35
|
+
Which placeholders exist depends on the hook (see
|
|
36
|
+
[per-hook contracts](#per-hook-contracts) below).
|
|
37
|
+
|
|
38
|
+
### Working directory and environment
|
|
39
|
+
|
|
40
|
+
The command runs in the **config file's directory**, so relative paths in
|
|
41
|
+
the command resolve predictably regardless of where `dirsql` was launched.
|
|
42
|
+
It inherits `dirsql`'s environment, so tools like `uvx --with …` / `npx …`
|
|
43
|
+
resolve their dependencies as usual.
|
|
44
|
+
|
|
45
|
+
### stdout protocol
|
|
46
|
+
|
|
47
|
+
The command's result payload is the **last non-empty line of stdout**,
|
|
48
|
+
trimmed. Any log or chatter lines above it are ignored. A command that
|
|
49
|
+
exits successfully but prints no non-empty line is a failure ("produced no
|
|
50
|
+
output on stdout").
|
|
51
|
+
|
|
52
|
+
stderr is never data — it is captured only to enrich error messages (the
|
|
53
|
+
last 2 000 characters are attached to failures).
|
|
54
|
+
|
|
55
|
+
::: tip Print single-line output
|
|
56
|
+
Because only the last non-empty line is the payload, multi-line output loses
|
|
57
|
+
everything above the last line. `jq` users: pass `-c` so the JSON is emitted
|
|
58
|
+
compactly on one line.
|
|
59
|
+
:::
|
|
60
|
+
|
|
61
|
+
### Timeout
|
|
62
|
+
|
|
63
|
+
Every hook run is bounded by a **30-second** default timeout. A run
|
|
64
|
+
exceeding it is killed and treated as a failure. One global config key
|
|
65
|
+
raises (or tightens) the bound for **all** hooks:
|
|
66
|
+
|
|
67
|
+
```toml
|
|
68
|
+
[dirsql]
|
|
69
|
+
hook-timeout = 300 # positive whole seconds
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Zero and negative values are a config error.
|
|
73
|
+
|
|
74
|
+
### Failure semantics
|
|
75
|
+
|
|
76
|
+
A hook run fails when the command:
|
|
77
|
+
|
|
78
|
+
- cannot be spawned (e.g. the program is not found),
|
|
79
|
+
- exits non-zero (the exit code — or `signal`, if killed by one — and the
|
|
80
|
+
stderr tail are reported),
|
|
81
|
+
- exceeds the timeout (killed; stderr tail reported),
|
|
82
|
+
- exits zero but prints no non-empty stdout line,
|
|
83
|
+
- or (per hook, below) prints output that does not parse as expected.
|
|
84
|
+
|
|
85
|
+
What a failure *means* differs per hook:
|
|
86
|
+
|
|
87
|
+
| Hook | On failure |
|
|
88
|
+
|---|---|
|
|
89
|
+
| `on-file` | **Per-file isolation.** The file contributes no rows; a one-line warning naming the file and error goes to stderr; the scan continues. One bad file never aborts the scan. |
|
|
90
|
+
| `pre-query` | The request returns `500 Internal Server Error` with the command's stderr tail in the JSON `error` body. |
|
|
91
|
+
| `post-query` | The request returns `500 Internal Server Error` with the command's stderr tail (or, for unparseable output, `post-query did not return valid JSON: <err>`) in the JSON `error` body. |
|
|
92
|
+
|
|
93
|
+
## Per-hook contracts
|
|
94
|
+
|
|
95
|
+
### `on-file`
|
|
96
|
+
|
|
97
|
+
Runs once per file matched by the table's `glob`, at initial scan and on
|
|
98
|
+
every watched change. The command reads the file itself and prints a JSON
|
|
99
|
+
array of row objects; see [`[[table]]`](./config.md#table) for the
|
|
100
|
+
row-mapping rules.
|
|
101
|
+
|
|
102
|
+
| Placeholder | Value |
|
|
103
|
+
|---|---|
|
|
104
|
+
| `{path}` | The matched file's path **relative to the index root**. Appended automatically as a final argument when the command omits it, so `extract.py` and `extract.py {path}` behave identically. |
|
|
105
|
+
| `{abspath}` | The matched file's absolute path. |
|
|
106
|
+
| `{root}` | The index root directory. |
|
|
107
|
+
|
|
108
|
+
### `pre-query`
|
|
109
|
+
|
|
110
|
+
Runs once per `POST /query` request, before the query. The raw request body
|
|
111
|
+
goes in; plain-text SQL comes out (the stdout payload line). `dirsql` runs
|
|
112
|
+
that SQL and returns rows as usual. With no `pre-query` key, the body is
|
|
113
|
+
parsed as `{"sql": …}` instead — see [HTTP API](./http-api.md#post-query).
|
|
114
|
+
|
|
115
|
+
| Placeholder | Value |
|
|
116
|
+
|---|---|
|
|
117
|
+
| `{args}` | The raw `POST /query` request body, verbatim, as one argv token. |
|
|
118
|
+
|
|
119
|
+
**The hook owns SQL safety.** The `{args}` substitution keeps the untrusted
|
|
120
|
+
body inert *as an argv token*, but whatever SQL string the hook prints is
|
|
121
|
+
executed as-is. Validate, escape, or parameterize inside the hook.
|
|
122
|
+
|
|
123
|
+
### `post-query`
|
|
124
|
+
|
|
125
|
+
Runs once per successful `POST /query`, after the query. The result rows
|
|
126
|
+
are serialized to a JSON array and delivered two ways:
|
|
127
|
+
|
|
128
|
+
- **On stdin** — always, unbounded. This is the recommended path.
|
|
129
|
+
- **As `{args}`** — only when the serialized payload is ≤ **96 KiB**. Above
|
|
130
|
+
that, `{args}` is substituted with an **empty string** and a stderr
|
|
131
|
+
warning naming the byte size directs the operator to stdin. The full
|
|
132
|
+
payload is still on stdin — this is a fallback, not truncation.
|
|
133
|
+
|
|
134
|
+
| Placeholder | Value |
|
|
135
|
+
|---|---|
|
|
136
|
+
| `{args}` | The result rows as a JSON array, as one argv token; emptied (with a stderr warning) when the payload exceeds 96 KiB. |
|
|
137
|
+
|
|
138
|
+
The stdout payload line is parsed as JSON and returned verbatim as the
|
|
139
|
+
`200 application/json` response body. A payload that is not valid JSON
|
|
140
|
+
fails the request (`500`). With no `post-query` key, the bare row array is
|
|
141
|
+
returned — see [HTTP API](./http-api.md#post-query).
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
# HTTP API
|
|
2
|
+
|
|
3
|
+
The [`dirsql` server](./cli.md#server-mode) (default `localhost:7117`)
|
|
4
|
+
exposes two endpoints: `POST /query` and `GET /events`.
|
|
5
|
+
|
|
6
|
+
## `POST /query`
|
|
7
|
+
|
|
8
|
+
Run a read-only SQL query. Request body is JSON:
|
|
9
|
+
|
|
10
|
+
```json
|
|
11
|
+
{"sql": "SELECT title, author FROM posts WHERE draft = 0"}
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
The `200` response is a JSON array of row objects keyed by column name:
|
|
15
|
+
|
|
16
|
+
```json
|
|
17
|
+
[
|
|
18
|
+
{"title": "Hello World", "author": "alice"},
|
|
19
|
+
{"title": "Second Post", "author": "bob"}
|
|
20
|
+
]
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
curl -s http://localhost:7117/query \
|
|
25
|
+
-H 'content-type: application/json' \
|
|
26
|
+
-d '{"sql":"SELECT COUNT(*) AS n FROM files"}' \
|
|
27
|
+
| jq
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
### Value serialization
|
|
31
|
+
|
|
32
|
+
| SQLite value | JSON |
|
|
33
|
+
|---|---|
|
|
34
|
+
| `NULL` | `null` |
|
|
35
|
+
| `INTEGER` | number |
|
|
36
|
+
| `REAL` | number; `NaN` / `±Infinity` become `null` |
|
|
37
|
+
| `TEXT` | string |
|
|
38
|
+
| `BLOB` | lowercase hex string (e.g. `"deadbeef"`) |
|
|
39
|
+
|
|
40
|
+
Internal tracking columns (`_dirsql_file_path`, `_dirsql_row_index`) are
|
|
41
|
+
excluded from `SELECT *` results.
|
|
42
|
+
|
|
43
|
+
### Status codes
|
|
44
|
+
|
|
45
|
+
| Status | When |
|
|
46
|
+
|---|---|
|
|
47
|
+
| `200` | Query succeeded. Body: array of row objects (or the [`post-query`](./hooks.md#post-query) hook's JSON). |
|
|
48
|
+
| `400` | Malformed JSON body, missing or empty `sql` field, or a SQL error (syntax error, unknown table, or a statement SQLite classifies as a write — queries are read-only). |
|
|
49
|
+
| `405` | `GET /query`. Plain-text body `method not allowed`. |
|
|
50
|
+
| `408` | The query exceeded the 30-second per-query timeout. |
|
|
51
|
+
| `500` | Internal server fault, or a failed [`pre-query`](./hooks.md#pre-query) / [`post-query`](./hooks.md#post-query) hook. |
|
|
52
|
+
| `503` | The server is in [degraded mode](./cli.md#degraded-mode) (the config file exists but failed to load). |
|
|
53
|
+
|
|
54
|
+
All error responses (except `405`) are `application/json`:
|
|
55
|
+
|
|
56
|
+
```json
|
|
57
|
+
{"error": "syntax error near \"SLECT\""}
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
### Hook interactions
|
|
61
|
+
|
|
62
|
+
- With [`[dirsql].pre-query`](./config.md#dirsql-keys) configured, the
|
|
63
|
+
request body is **not** parsed as `{"sql": …}`; the raw body is passed to
|
|
64
|
+
the hook, which prints the SQL to run. Hook failure → `500`.
|
|
65
|
+
- With [`[dirsql].post-query`](./config.md#dirsql-keys) configured, the
|
|
66
|
+
`200` body is whatever JSON the hook prints instead of the bare row
|
|
67
|
+
array. Hook failure or non-JSON output → `500`.
|
|
68
|
+
|
|
69
|
+
## `GET /events`
|
|
70
|
+
|
|
71
|
+
Opens a [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events)
|
|
72
|
+
stream of row-change events.
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
curl -N http://localhost:7117/events
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
On open, the server emits a single `ready` frame signalling the
|
|
79
|
+
subscription is attached. Every subsequent frame is named `row`:
|
|
80
|
+
|
|
81
|
+
```
|
|
82
|
+
event: ready
|
|
83
|
+
data: {}
|
|
84
|
+
|
|
85
|
+
event: row
|
|
86
|
+
data: {"action":"insert","table":"posts","file_path":"posts/hello.json","row":{"title":"Hello World"},"old_row":null}
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
### Event payloads
|
|
90
|
+
|
|
91
|
+
The `data:` payload is one JSON object per row-level change:
|
|
92
|
+
|
|
93
|
+
| Field | `insert` | `update` | `delete` | `error` |
|
|
94
|
+
|---|---|---|---|---|
|
|
95
|
+
| `action` | `"insert"` | `"update"` | `"delete"` | `"error"` |
|
|
96
|
+
| `table` | table name | table name | table name | table name, or `null` when the failure isn't tied to one table |
|
|
97
|
+
| `file_path` | path relative to the root | same | same | same |
|
|
98
|
+
| `row` | the new row | the new row | the deleted row | — (absent) |
|
|
99
|
+
| `old_row` | `null` | the previous row | `null` | — (absent) |
|
|
100
|
+
| `error` | — | — | — | message string |
|
|
101
|
+
|
|
102
|
+
Row values serialize as in [`POST /query`](#value-serialization).
|
|
103
|
+
|
|
104
|
+
### Stream semantics
|
|
105
|
+
|
|
106
|
+
- **Errors do not terminate the stream.** A malformed file produces an
|
|
107
|
+
`error` event; the stream continues.
|
|
108
|
+
- **Slow consumers skip, not crash.** A subscriber that lags behind the
|
|
109
|
+
event buffer silently misses the overflowed events and keeps receiving
|
|
110
|
+
new ones.
|
|
111
|
+
- The server sends periodic SSE keep-alive comments.
|
|
112
|
+
- The stream closes when the server shuts down.
|
|
113
|
+
|
|
114
|
+
### Status codes
|
|
115
|
+
|
|
116
|
+
| Status | When |
|
|
117
|
+
|---|---|
|
|
118
|
+
| `200` | Stream opened (`text/event-stream`). |
|
|
119
|
+
| `405` | `POST /events`. Plain-text body `method not allowed`. |
|
|
120
|
+
| `503` | The server is in [degraded mode](./cli.md#degraded-mode). JSON `{"error": …}` body. |
|
|
@@ -0,0 +1,357 @@
|
|
|
1
|
+
# SDK
|
|
2
|
+
|
|
3
|
+
`dirsql` ships as a library for Python (PyPI: `dirsql`), TypeScript (npm:
|
|
4
|
+
`dirsql`), and Rust (crates.io: `dirsql`). All three are thin bindings over
|
|
5
|
+
one shared Rust core, so behavior is identical; only the calling conventions
|
|
6
|
+
differ.
|
|
7
|
+
|
|
8
|
+
## Install
|
|
9
|
+
|
|
10
|
+
::: code-group
|
|
11
|
+
|
|
12
|
+
```bash [Python]
|
|
13
|
+
uv add dirsql
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
```bash [TypeScript]
|
|
17
|
+
pnpm add dirsql
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
```bash [Rust]
|
|
21
|
+
cargo add dirsql
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
:::
|
|
25
|
+
|
|
26
|
+
## Import
|
|
27
|
+
|
|
28
|
+
::: code-group
|
|
29
|
+
|
|
30
|
+
```python [Python]
|
|
31
|
+
from dirsql import DirSQL, Table, RowEvent
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
```typescript [TypeScript]
|
|
35
|
+
import { DirSQL, Table } from "dirsql";
|
|
36
|
+
import type { TableDef, RowEvent, ExtensionSpec, DirSQLOptions } from "dirsql";
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
```rust [Rust]
|
|
40
|
+
use dirsql::{DirSQL, AsyncDirSQL, Table, Row, RowEvent, Value, Extension, DirSqlError};
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
:::
|
|
44
|
+
|
|
45
|
+
## `DirSQL`
|
|
46
|
+
|
|
47
|
+
### Constructor
|
|
48
|
+
|
|
49
|
+
::: code-group
|
|
50
|
+
|
|
51
|
+
```python [Python]
|
|
52
|
+
DirSQL(
|
|
53
|
+
root: str | None = None,
|
|
54
|
+
*,
|
|
55
|
+
tables: list[Table] | None = None,
|
|
56
|
+
ignore: list[str] | None = None,
|
|
57
|
+
config: str | None = None,
|
|
58
|
+
persist: bool = False,
|
|
59
|
+
persist_path: str | None = None,
|
|
60
|
+
extensions: list[dict] | None = None, # [{ "path": str, "entrypoint"?: str }]
|
|
61
|
+
)
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
```typescript [TypeScript]
|
|
65
|
+
new DirSQL(configPath: string)
|
|
66
|
+
// or
|
|
67
|
+
new DirSQL({
|
|
68
|
+
root?: string,
|
|
69
|
+
tables?: TableDef[],
|
|
70
|
+
ignore?: string[],
|
|
71
|
+
config?: string,
|
|
72
|
+
persist?: boolean,
|
|
73
|
+
persistPath?: string,
|
|
74
|
+
extensions?: ExtensionSpec[], // [{ path: string, entrypoint?: string }]
|
|
75
|
+
})
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
```rust [Rust]
|
|
79
|
+
DirSQL::builder()
|
|
80
|
+
.root(root) // optional
|
|
81
|
+
.tables(tables) // optional; append one with .table(t)
|
|
82
|
+
.ignore(patterns) // optional
|
|
83
|
+
.config(config_toml_path) // optional
|
|
84
|
+
.persist(true) // optional; default false
|
|
85
|
+
.persist_path(path) // optional
|
|
86
|
+
.extensions(extensions) // optional; append one with .extension(e)
|
|
87
|
+
.poll_interval(duration) // optional; watch-loop cadence, default 200ms
|
|
88
|
+
.build() // -> Result<DirSQL> (synchronous scan)
|
|
89
|
+
// Shortcuts: DirSQL::new(root, tables), DirSQL::with_ignore(root, tables, ignore),
|
|
90
|
+
// DirSQL::from_config(root), DirSQL::from_config_path(path)
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
:::
|
|
94
|
+
|
|
95
|
+
Creates a SQLite index over a directory. At least one of `root` or `config`
|
|
96
|
+
must be supplied; with neither, construction fails with a "no root
|
|
97
|
+
directory" error.
|
|
98
|
+
|
|
99
|
+
**Parameters:**
|
|
100
|
+
|
|
101
|
+
- `root` — Directory to index. Optional if `config` is supplied.
|
|
102
|
+
- `tables` — Programmatic [`Table`](#table) definitions.
|
|
103
|
+
- `ignore` — Glob patterns matched against root-relative paths; matched
|
|
104
|
+
files are skipped entirely (scan and watch).
|
|
105
|
+
- `config` — Path to a [`.dirsql.toml`](./config.md). Its `[[table]]`
|
|
106
|
+
entries are appended after any programmatic `tables`; its `ignore`
|
|
107
|
+
patterns and `[[dirsql.extension]]` entries are appended likewise; its
|
|
108
|
+
`[dirsql].root` supplies the root when no explicit `root` is passed.
|
|
109
|
+
When both an explicit `root` and a config root are present, the explicit
|
|
110
|
+
value wins and a warning is emitted on stderr.
|
|
111
|
+
- `persist` — Keep the SQLite index on disk between runs (default off:
|
|
112
|
+
ephemeral, rebuilt every startup). The cache lives at
|
|
113
|
+
`<root>/.dirsql/cache.db` by default; on restart, only files whose stat
|
|
114
|
+
changed are re-parsed. `persist = true` in the config also enables it.
|
|
115
|
+
- `persist_path` / `persistPath` — Override the cache location. An
|
|
116
|
+
explicit value wins over the config's `persist_path`. Ignored when
|
|
117
|
+
persistence is off. (Config-file values resolve relative to the config's
|
|
118
|
+
parent directory; constructor values are used as given.)
|
|
119
|
+
- `extensions` — SQLite extensions to load at startup, before any table
|
|
120
|
+
DDL (enable → load → disable, so SQL `load_extension()` is never
|
|
121
|
+
exposed). Each entry pairs a shared-library `path` with an optional
|
|
122
|
+
`entrypoint` init symbol. In Python and TypeScript, `path` may be a bare
|
|
123
|
+
**package name**, resolved from the installed package (`importlib` /
|
|
124
|
+
`node_modules`) — for both the programmatic list and a `config` file's
|
|
125
|
+
`[[dirsql.extension]]` entries. The Rust SDK is file-path-only.
|
|
126
|
+
Programmatic entries load first, then the config's. A relative
|
|
127
|
+
programmatic path is used verbatim (resolved by the OS against the
|
|
128
|
+
process working directory); config-file paths resolve against the config
|
|
129
|
+
file's parent.
|
|
130
|
+
|
|
131
|
+
**Construction is asynchronous in Python and TypeScript.** The constructor
|
|
132
|
+
returns immediately and the initial scan runs in the background; `query`
|
|
133
|
+
and the other methods transparently await [readiness](#ready). In Rust,
|
|
134
|
+
`.build()` scans synchronously; use `.build_async()` → [`AsyncDirSQL`](#asyncdirsql-rust)
|
|
135
|
+
for the non-blocking equivalent.
|
|
136
|
+
|
|
137
|
+
::: details Rust builder extras
|
|
138
|
+
- `Table` appears in the builder via `.table(t)` (append) or `.tables(v)`
|
|
139
|
+
(replace).
|
|
140
|
+
- `.poll_interval(Duration)` sets the channel-based `watch()` loop's poll
|
|
141
|
+
cadence (default 200 ms): lower values give tighter event latency at
|
|
142
|
+
higher idle CPU.
|
|
143
|
+
- `.suppress_config_extensions(true)` skips loading the config file's own
|
|
144
|
+
`[[dirsql.extension]]` entries. Launcher plumbing (used with
|
|
145
|
+
pre-resolved `.extensions(...)`); not needed in application code.
|
|
146
|
+
- `Extension { path: PathBuf, entrypoint: Option<String> }` is the
|
|
147
|
+
extension spec type.
|
|
148
|
+
:::
|
|
149
|
+
|
|
150
|
+
### `ready`
|
|
151
|
+
|
|
152
|
+
::: code-group
|
|
153
|
+
|
|
154
|
+
```python [Python]
|
|
155
|
+
await db.ready() -> None
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
```typescript [TypeScript]
|
|
159
|
+
await db.ready // awaitable property
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
```rust [Rust]
|
|
163
|
+
// AsyncDirSQL only; DirSQL::build() is already complete when it returns.
|
|
164
|
+
db.ready().await -> Result<()>
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
:::
|
|
168
|
+
|
|
169
|
+
Waits for the initial scan to complete, re-raising any construction error.
|
|
170
|
+
Safe to call multiple times. In Python and TypeScript every other method
|
|
171
|
+
awaits readiness internally, so an explicit `ready` is only needed to
|
|
172
|
+
observe construction errors before issuing a query.
|
|
173
|
+
|
|
174
|
+
### `query`
|
|
175
|
+
|
|
176
|
+
::: code-group
|
|
177
|
+
|
|
178
|
+
```python [Python]
|
|
179
|
+
await db.query(sql: str) -> list[dict]
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
```typescript [TypeScript]
|
|
183
|
+
await db.query(sql: string) -> Record<string, unknown>[]
|
|
184
|
+
// Runs on the libuv threadpool; the JS event loop stays responsive.
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
```rust [Rust]
|
|
188
|
+
db.query(sql: &str) -> Result<Vec<Row>> // Row = HashMap<String, Value>
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
:::
|
|
192
|
+
|
|
193
|
+
Executes a SQL query and returns rows keyed by column name.
|
|
194
|
+
|
|
195
|
+
- **Read-only.** Each statement is classified by SQLite itself
|
|
196
|
+
(`sqlite3_stmt_readonly`); anything classified as a write — `INSERT`,
|
|
197
|
+
`UPDATE`, `DELETE`, `DROP`, `CREATE`, `ALTER`, `VACUUM`, … — is rejected
|
|
198
|
+
before producing rows. Rust surfaces this as
|
|
199
|
+
`DirSqlError::WriteForbidden`; Python raises a `RuntimeError` and
|
|
200
|
+
TypeScript rejects with an `Error` carrying a "read-only" message.
|
|
201
|
+
- Internal tracking columns (`_dirsql_file_path`, `_dirsql_row_index`) are
|
|
202
|
+
excluded from `SELECT *` results; name them explicitly to see them.
|
|
203
|
+
- SQLite values map back to language types:
|
|
204
|
+
|
|
205
|
+
| SQLite | Python | TypeScript | Rust |
|
|
206
|
+
|---|---|---|---|
|
|
207
|
+
| TEXT | `str` | `string` | `Value::Text` |
|
|
208
|
+
| INTEGER | `int` | `number` | `Value::Integer` |
|
|
209
|
+
| REAL | `float` | `number` | `Value::Real` |
|
|
210
|
+
| BLOB | `bytes` | `Buffer` | `Value::Blob` |
|
|
211
|
+
| NULL | `None` | `null` | `Value::Null` |
|
|
212
|
+
|
|
213
|
+
### `watch`
|
|
214
|
+
|
|
215
|
+
::: code-group
|
|
216
|
+
|
|
217
|
+
```python [Python]
|
|
218
|
+
async for event in db.watch(): # AsyncIterator[RowEvent]
|
|
219
|
+
...
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
```typescript [TypeScript]
|
|
223
|
+
for await (const event of db.watch()) { // AsyncIterable<RowEvent>
|
|
224
|
+
...
|
|
225
|
+
}
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
```rust [Rust]
|
|
229
|
+
use futures::StreamExt;
|
|
230
|
+
let mut stream = db.watch()?; // WatchStream: impl Stream<Item = RowEvent>
|
|
231
|
+
while let Some(event) = stream.next().await { ... }
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
:::
|
|
235
|
+
|
|
236
|
+
Returns a stream of [`RowEvent`](#rowevent)s reflecting filesystem changes
|
|
237
|
+
under the root. The watcher starts on first iteration (Python/TypeScript)
|
|
238
|
+
or at the `watch()` call (Rust). The stream never terminates on its own;
|
|
239
|
+
stop consuming it to stop.
|
|
240
|
+
|
|
241
|
+
Rust: `watch()` may be called once per instance and is mutually exclusive
|
|
242
|
+
with the polling API below — mixing them returns an error, since both drain
|
|
243
|
+
the same underlying filesystem watcher.
|
|
244
|
+
|
|
245
|
+
### Low-level watch primitives
|
|
246
|
+
|
|
247
|
+
::: code-group
|
|
248
|
+
|
|
249
|
+
```typescript [TypeScript]
|
|
250
|
+
await db.startWatcher() // idempotent; must precede pollEvents
|
|
251
|
+
await db.pollEvents(timeoutMs) -> RowEvent[]
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
```rust [Rust]
|
|
255
|
+
db.start_watching() -> Result<()> // idempotent; implied by poll_events/watch
|
|
256
|
+
db.poll_events(timeout: Duration) -> Result<Vec<RowEvent>>
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
:::
|
|
260
|
+
|
|
261
|
+
`pollEvents` / `poll_events` blocks up to the timeout for the first event,
|
|
262
|
+
then drains everything that arrived, applies it to the database, and
|
|
263
|
+
returns the batch (possibly empty). Safe to call in a loop. TypeScript runs
|
|
264
|
+
the poll on the libuv threadpool. Python does not expose public polling
|
|
265
|
+
primitives — use `watch()`.
|
|
266
|
+
|
|
267
|
+
## `AsyncDirSQL` (Rust)
|
|
268
|
+
|
|
269
|
+
Rust's non-blocking wrapper: the constructor returns immediately while the
|
|
270
|
+
scan runs on a background thread. Python and TypeScript need no equivalent —
|
|
271
|
+
their `DirSQL` is already async-by-default.
|
|
272
|
+
|
|
273
|
+
```rust
|
|
274
|
+
let db = DirSQL::builder().root("./data").tables(tables).build_async()?; // -> AsyncDirSQL
|
|
275
|
+
db.ready().await?; // required before use
|
|
276
|
+
let rows = db.query("SELECT ...").await?; // spawn_blocking under the hood
|
|
277
|
+
let stream = db.watch()?; // same WatchStream as DirSQL
|
|
278
|
+
let sync: DirSQL = db.sync()?; // unwrap the inner sync handle
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
Shortcuts mirror `DirSQL`: `AsyncDirSQL::new`, `with_ignore`,
|
|
282
|
+
`from_config`, `from_config_path`. Unlike the Python/TypeScript SDKs,
|
|
283
|
+
methods called before `ready().await` completes return a "not ready" error
|
|
284
|
+
rather than waiting — an intentional, language-idiomatic difference.
|
|
285
|
+
|
|
286
|
+
## `Table`
|
|
287
|
+
|
|
288
|
+
::: code-group
|
|
289
|
+
|
|
290
|
+
```python [Python]
|
|
291
|
+
Table(*, ddl: str, glob: str, extract: Callable[[str], list[dict]], strict: bool = False)
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
```typescript [TypeScript]
|
|
295
|
+
new Table({ ddl, glob, extract, strict? })
|
|
296
|
+
// or a plain object — TableDef and Table are interchangeable:
|
|
297
|
+
{ ddl: string, glob: string, extract: (path: string) => Record<string, unknown>[], strict?: boolean }
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
```rust [Rust]
|
|
301
|
+
Table::new(ddl, glob, extract) // extract: Fn(&str) -> Vec<Row>, infallible
|
|
302
|
+
Table::try_new(ddl, glob, extract) // extract: Fn(&str) -> Result<Vec<Row>, _>
|
|
303
|
+
Table::strict(ddl, glob, extract) // Table::new with strict = true
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
:::
|
|
307
|
+
|
|
308
|
+
Maps files to table rows.
|
|
309
|
+
|
|
310
|
+
- `ddl` — A SQLite `CREATE TABLE` statement; the table name is parsed from
|
|
311
|
+
it. Table names must be unique across all tables.
|
|
312
|
+
- `glob` — Glob pattern matched against root-relative paths. May contain
|
|
313
|
+
`{name}` [captures](./columns.md#glob-captures).
|
|
314
|
+
- `extract` — Callback receiving the matched file's full path (the root
|
|
315
|
+
joined with the file's relative path — absolute when `root` is absolute)
|
|
316
|
+
and returning the rows that file contributes. `dirsql` never reads file
|
|
317
|
+
contents itself; a callback that needs the body reads the path. Return
|
|
318
|
+
an empty list to skip a file. [Virtual columns and glob
|
|
319
|
+
captures](./columns.md) are merged onto each returned row; values the
|
|
320
|
+
callback emits win over same-named facts.
|
|
321
|
+
- `strict` — Default off: extra row keys are dropped and missing declared
|
|
322
|
+
columns become `NULL`. When on, any extra or missing key is an error
|
|
323
|
+
(see [`strict`](./config.md#table)).
|
|
324
|
+
|
|
325
|
+
Readable attributes: `ddl`, `glob` (all SDKs), plus `strict`.
|
|
326
|
+
|
|
327
|
+
## `RowEvent`
|
|
328
|
+
|
|
329
|
+
Emitted by [`watch`](#watch) and the polling primitives; one event per
|
|
330
|
+
row-level change caused by a filesystem event.
|
|
331
|
+
|
|
332
|
+
| Field | Python | Rust | TypeScript |
|
|
333
|
+
|---|---|---|---|
|
|
334
|
+
| Action | `action: str` | enum variant (`Insert` / `Update` / `Delete` / `Error`) | `action: string` |
|
|
335
|
+
| Table | `table: str \| None` | `table: String` (`Option<String>` on `Error`) | `table: string \| null` |
|
|
336
|
+
| New/current row | `row: dict \| None` | `row` / `new_row` on the variant | `row?: Record \| null` |
|
|
337
|
+
| Previous row | `old_row: dict \| None` | `old_row` on `Update` | `oldRow?: Record \| null` |
|
|
338
|
+
| Error message | `error: str \| None` | `error: String` on `Error` | `error?: string \| null` |
|
|
339
|
+
| File path | `file_path: str \| None` | `file_path` (`String`; `PathBuf` on `Error`) | `filePath?: string \| null` |
|
|
340
|
+
|
|
341
|
+
Actions: `insert` (new row; no previous), `update` (row changed in place;
|
|
342
|
+
carries both old and new), `delete` (row removed), `error` (an extraction
|
|
343
|
+
or watch failure — carries `error`, never `row`/`old_row`; `table` is
|
|
344
|
+
`None`/`null` when the failure isn't tied to one table). `file_path` is
|
|
345
|
+
relative to the root. Errors never terminate the stream.
|
|
346
|
+
|
|
347
|
+
## Value types
|
|
348
|
+
|
|
349
|
+
An `extract` callback (and the SQLite columns it feeds) accepts:
|
|
350
|
+
|
|
351
|
+
| SQLite column | Python value | TypeScript value | Rust `Value` |
|
|
352
|
+
|---|---|---|---|
|
|
353
|
+
| TEXT | `str` | `string` | `Value::Text(String)` |
|
|
354
|
+
| INTEGER | `int`, `bool` | `number`, `boolean` | `Value::Integer(i64)` |
|
|
355
|
+
| REAL | `float` | `number` | `Value::Real(f64)` |
|
|
356
|
+
| BLOB | `bytes` | `Buffer` / `Uint8Array` | `Value::Blob(Vec<u8>)` |
|
|
357
|
+
| NULL | `None` | `null` | `Value::Null` |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dirsql",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.71",
|
|
4
4
|
"description": "Ephemeral SQL index over a local directory",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": "https://github.com/thekevinscott/dirsql",
|
|
@@ -230,15 +230,15 @@
|
|
|
230
230
|
]
|
|
231
231
|
},
|
|
232
232
|
"optionalDependencies": {
|
|
233
|
-
"@dirsql/lib-linux-x64-gnu": "0.3.
|
|
234
|
-
"@dirsql/lib-linux-arm64-gnu": "0.3.
|
|
235
|
-
"@dirsql/lib-darwin-x64": "0.3.
|
|
236
|
-
"@dirsql/lib-darwin-arm64": "0.3.
|
|
237
|
-
"@dirsql/lib-win32-x64-msvc": "0.3.
|
|
238
|
-
"@dirsql/cli-linux-x64-gnu": "0.3.
|
|
239
|
-
"@dirsql/cli-linux-arm64-gnu": "0.3.
|
|
240
|
-
"@dirsql/cli-darwin-x64": "0.3.
|
|
241
|
-
"@dirsql/cli-darwin-arm64": "0.3.
|
|
242
|
-
"@dirsql/cli-win32-x64-msvc": "0.3.
|
|
233
|
+
"@dirsql/lib-linux-x64-gnu": "0.3.71",
|
|
234
|
+
"@dirsql/lib-linux-arm64-gnu": "0.3.71",
|
|
235
|
+
"@dirsql/lib-darwin-x64": "0.3.71",
|
|
236
|
+
"@dirsql/lib-darwin-arm64": "0.3.71",
|
|
237
|
+
"@dirsql/lib-win32-x64-msvc": "0.3.71",
|
|
238
|
+
"@dirsql/cli-linux-x64-gnu": "0.3.71",
|
|
239
|
+
"@dirsql/cli-linux-arm64-gnu": "0.3.71",
|
|
240
|
+
"@dirsql/cli-darwin-x64": "0.3.71",
|
|
241
|
+
"@dirsql/cli-darwin-arm64": "0.3.71",
|
|
242
|
+
"@dirsql/cli-win32-x64-msvc": "0.3.71"
|
|
243
243
|
}
|
|
244
244
|
}
|