dirsql 0.3.65 → 0.3.67

package/docs/getting-started.md

@@ -1,190 +1,257 @@
----
-canonical: https://thekevinscott.github.io/dirsql/getting-started
----
+# Your first dirsql database
 
-# Getting Started
+In this tutorial you will turn a directory of three tiny markdown files into
+a SQL database you can query over HTTP — without writing any code. You will:
 
-> Online: <https://thekevinscott.github.io/dirsql/getting-started>
+1. Create the directory and files.
+2. Start `dirsql` with zero configuration and query it with `curl`.
+3. Define your own table in a `.dirsql.toml` and query the new shape.
 
-## Installation
+It takes about five minutes.
 
-::: code-group
+**You need:** a terminal with `curl` and [`jq`](https://jqlang.org/), and
+Node ≥ 20.11 (for `npx`). Every `npx dirsql` step below also has a `uvx`
+tab that behaves identically, if you prefer Python tooling
+([`uv`](https://docs.astral.sh/uv/)).
+
+## 1. Create three files
+
+Make a working directory with two subfolders — one per note author:
+
+```bash
+mkdir -p my-notes/notes/alice my-notes/notes/bob
+cd my-notes
+```
+
+Create the three notes by pasting each block exactly as shown:
+
+```bash
+cat > notes/alice/welcome.md <<'EOF'
+# Welcome
+
+Start here. This folder is about to become a database.
+EOF
+```
+
+```bash
+cat > notes/alice/ideas.md <<'EOF'
+# Ideas
+
+- query files with SQL
+- watch for changes
+EOF
+```
+
+```bash
+cat > notes/bob/reading-list.md <<'EOF'
+# Reading list
+
+- The SQLite file format
+EOF
+```
 
-```bash [Python]
-pip install dirsql
+Check that all three files are in place:
+
+```bash
+find notes -type f | sort
 ```
 
-```bash [Rust]
-cargo add dirsql
 ```
+notes/alice/ideas.md
+notes/alice/welcome.md
+notes/bob/reading-list.md
+```
+
+## 2. Start the server
+
+From inside `my-notes`, start `dirsql`:
 
-```bash [TypeScript]
-# The npm CLI requires **Node ≥ 20.11**.
-npm add dirsql
+::: code-group
+
+```bash [npm]
+npx dirsql
 ```
 
-```bash [CLI]
-# Pick whichever install path you already have handy
-npx dirsql --version
-uvx dirsql --version
-cargo install dirsql --features cli
+```bash [PyPI]
+uvx dirsql
 ```
 
 :::
 
+The first run downloads the package (`npx` asks for confirmation — answer
+`y`; `uvx` prints download progress), then the server starts:
 
+```
+Running at localhost:7117
+```
 
-See the [CLI section](./cli/) for details on the command-line interface, and the [Rust library README](https://github.com/thekevinscott/dirsql/tree/main/packages/rust) for the library-vs-CLI feature split.
+That one command scanned the directory, built an in-memory SQLite database
+with one row per file, and started an HTTP server. Leave it running and
+open a **second terminal** for the next step.
 
-## Quick start
+## 3. Query your files
 
-Suppose you have a directory of JSON files representing blog posts:
+You gave `dirsql` no configuration, so it serves a single default table
+named `files` ([zero-config mode](./reference/cli.md#zero-config-mode)).
+Ask it how many rows it has:
 
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT COUNT(*) AS files FROM files"}'
 ```
-my-blog/
-  posts/
-    hello.json      # {"title": "Hello World", "author": "alice"}
-    second.json     # {"title": "Second Post", "author": "bob"}
-  authors/
-    alice.json      # {"id": "alice", "name": "Alice"}
-    bob.json        # {"id": "bob", "name": "Bob"}
+
+```
+[{"files":3}]
 ```
 
-Index and query them with `dirsql`:
+Three files, three rows. The response is always a JSON array of row
+objects ([HTTP API](./reference/http-api.md)) — from here on we pipe it
+through `jq` to pretty-print. Now select some columns:
 
-::: code-group
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT _path, _size FROM files ORDER BY _path"}' \
+  | jq
+```
 
-```python [Python]
-import asyncio
-import json
-from dirsql import DirSQL, Table
-
-async def main():
-    db = DirSQL(
-        "./my-blog",
-        tables=[
-            Table(
-                ddl="CREATE TABLE posts (title TEXT, author TEXT)",
-                glob="posts/*.json",
-                extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
-            ),
-            Table(
-                ddl="CREATE TABLE authors (id TEXT, name TEXT)",
-                glob="authors/*.json",
-                extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
-            ),
-        ],
-    )
-    await db.ready()
-
-    # Query all posts
-    posts = await db.query("SELECT * FROM posts")
-    # [{"title": "Hello World", "author": "alice"}, {"title": "Second Post", "author": "bob"}]
-
-    # Join across tables
-    results = await db.query("""
-        SELECT posts.title, authors.name
-        FROM posts
-        JOIN authors ON posts.author = authors.id
-    """)
-    # [{"title": "Hello World", "name": "Alice"}, {"title": "Second Post", "name": "Bob"}]
-
-asyncio.run(main())
-```
-
-```rust [Rust]
-use dirsql::{DirSQL, Table, Value};
-use std::collections::HashMap;
-
-// Convert a JSON object string into a dirsql row.
-fn row_from_json(raw: &str) -> HashMap<String, Value> {
-    let v: serde_json::Value = serde_json::from_str(raw).unwrap();
-    let serde_json::Value::Object(obj) = v else { return HashMap::new() };
-    obj.into_iter()
-        .map(|(k, val)| {
-            let v = match val {
-                serde_json::Value::String(s) => Value::Text(s),
-                serde_json::Value::Number(n) => n
-                    .as_i64()
-                    .map(Value::Integer)
-                    .unwrap_or_else(|| Value::Real(n.as_f64().unwrap_or(0.0))),
-                serde_json::Value::Bool(b) => Value::Integer(b as i64),
-                serde_json::Value::Null => Value::Null,
-                other => Value::Text(other.to_string()),
-            };
-            (k, v)
-        })
-        .collect()
-}
-
-let db = DirSQL::new(
-    "./my-blog",
-    vec![
-        Table::new(
-            "CREATE TABLE posts (title TEXT, author TEXT)",
-            "posts/*.json",
-            |path| vec![row_from_json(&std::fs::read_to_string(path).unwrap())],
-        ),
-        Table::new(
-            "CREATE TABLE authors (id TEXT, name TEXT)",
-            "authors/*.json",
-            |path| vec![row_from_json(&std::fs::read_to_string(path).unwrap())],
-        ),
-    ],
-)?;
-
-let posts = db.query("SELECT * FROM posts")?;
-
-let results = db.query(
-    "SELECT posts.title, authors.name \
-     FROM posts JOIN authors ON posts.author = authors.id"
-)?;
-```
-
-```typescript [TypeScript]
-import { readFileSync } from 'node:fs';
-import { DirSQL, type TableDef } from 'dirsql';
-
-const tables: TableDef[] = [
+```json
+[
   {
-    ddl: 'CREATE TABLE posts (title TEXT, author TEXT)',
-    glob: 'posts/*.json',
-    extract: (path) => [JSON.parse(readFileSync(path, 'utf8'))],
+    "_path": "notes/alice/ideas.md",
+    "_size": 52
   },
   {
-    ddl: 'CREATE TABLE authors (id TEXT, name TEXT)',
-    glob: 'authors/*.json',
-    extract: (path) => [JSON.parse(readFileSync(path, 'utf8'))],
+    "_path": "notes/alice/welcome.md",
+    "_size": 66
   },
-];
+  {
+    "_path": "notes/bob/reading-list.md",
+    "_size": 41
+  }
+]
+```
+
+`_path` and `_size` are two of the built-in file columns `dirsql` collects
+for every file — see [virtual columns](./reference/columns.md#virtual-columns)
+for the full list. (The `_size` values are byte counts; they match the
+output above because you pasted the files exactly.)
 
-const db = new DirSQL({ root: './my-blog', tables });
+You have a working SQL database over your files. Next, teach it the
+structure your folders already encode.
 
-const posts = await db.query('SELECT * FROM posts');
+## 4. Define a table
 
-const results = await db.query(`
-  SELECT posts.title, authors.name
-  FROM posts JOIN authors ON posts.author = authors.id
-`);
+Look at the paths again: `notes/alice/ideas.md`, `notes/bob/reading-list.md`
+— the author's name is a directory segment. A config file can capture it as
+a real column.
+
+In your second terminal, still inside `my-notes`, create a `.dirsql.toml`:
+
+```bash
+cat > .dirsql.toml <<'EOF'
+[[table]]
+ddl  = "CREATE TABLE notes (author TEXT, _basename TEXT, _size INTEGER)"
+glob = "notes/{author}/*.md"
+EOF
+```
+
+Two keys define the table:
+
+- `glob` selects which files feed the table, and `{author}` is a
+  [glob capture](./reference/columns.md#glob-captures): whatever directory
+  name matches that segment becomes the row's `author` value.
+- `ddl` is ordinary `CREATE TABLE` SQL naming the columns you want to keep.
+
+## 5. Restart and query the new shape
+
+Config is read at startup, so go back to the **first terminal**, stop the
+server with `Ctrl-C`, and start it again:
+
+::: code-group
+
+```bash [npm]
+npx dirsql
+```
+
+```bash [PyPI]
+uvx dirsql
 ```
 
 :::
 
-## What happens at startup
+```
+Running at localhost:7117
+```
 
-1. `dirsql` walks the directory tree
-2. Files matching each table's glob pattern are identified
-3. The `extract` function receives each matched file's path (relative to the scan root, or absolute when `root` is absolute) and returns rows
-4. Rows are inserted into an in-memory SQLite database
-5. SQL queries run against that database
+This time `dirsql` found your `.dirsql.toml` and served the `notes` table
+you defined instead of the default `files` table. Query it from the second
+terminal:
 
-The filesystem is always the source of truth. The database is rebuilt from files at startup.
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT author, _basename, _size FROM notes ORDER BY author, _basename"}' \
+  | jq
+```
 
-## Next steps
+```json
+[
+  {
+    "_basename": "ideas.md",
+    "_size": 52,
+    "author": "alice"
+  },
+  {
+    "_basename": "welcome.md",
+    "_size": 66,
+    "author": "alice"
+  },
+  {
+    "_basename": "reading-list.md",
+    "_size": 41,
+    "author": "bob"
+  }
+]
+```
+
+Every row now carries an `author` column extracted from its path — no
+extraction code, just a glob. And it is a real SQL column, so you can
+aggregate on it:
+
+```bash
+curl -s http://localhost:7117/query \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT author, COUNT(*) AS notes FROM notes GROUP BY author"}' \
+  | jq
+```
+
+```json
+[
+  {
+    "author": "alice",
+    "notes": 2
+  },
+  {
+    "author": "bob",
+    "notes": 1
+  }
+]
+```
 
-- [Defining Tables](./guide/tables.md) -- DDL, globs, and extract functions in detail
-- [Querying](./guide/querying.md) -- SQL queries and return format
-- [File Watching](./guide/watching.md) -- real-time change events
-- [Async API](./guide/async.md) -- async ready(), query(), and watch()
-- [Using `dirsql` from the CLI](./cli/) -- `dirsql` runs an HTTP server (`POST /query`, `GET /events` SSE)
-- [Collaboration with CRDTs](./guide/crdt.md) -- multi-writer document merging alongside `dirsql`
+That's the whole loop: files in a directory, a declarative table on top,
+SQL over HTTP.
+
+## Where to go next
+
+- [Configuration file](./reference/config.md) — the complete `.dirsql.toml`
+  reference: more tables, ignore patterns, persistence, hooks.
+- [CLI](./reference/cli.md) — flags like `--port` and `--config`, plus
+  `dirsql init`.
+- [HTTP API](./reference/http-api.md) — `POST /query` in full, plus
+  `GET /events`, a live stream of row changes as files change.
+- [SDK](./reference/sdk.md) — embed `dirsql` in a Python, Rust, or
+  TypeScript program instead of running the server.
+- Why is the database rebuilt from your files on every startup? See
+  [how `dirsql` thinks](./explanation.md).

package/docs/index.md

@@ -1,5 +1,4 @@
 ---
-canonical: https://thekevinscott.github.io/dirsql/
 layout: home
 hero:
   name: dirsql

package/docs/migrations.md

@@ -1,8 +1,5 @@
 ---
 title: Migrations
-canonical: https://thekevinscott.github.io/dirsql/migrations
 ---
 
-> Online: <https://thekevinscott.github.io/dirsql/migrations>
-
 <!--@include: ../MIGRATIONS.md-->

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dirsql",
-  "version": "0.3.65",
+  "version": "0.3.67",
   "description": "Ephemeral SQL index over a local directory",
   "license": "MIT",
   "repository": "https://github.com/thekevinscott/dirsql",
@@ -230,15 +230,15 @@
     ]
   },
   "optionalDependencies": {
-    "@dirsql/lib-linux-x64-gnu": "0.3.65",
-    "@dirsql/lib-linux-arm64-gnu": "0.3.65",
-    "@dirsql/lib-darwin-x64": "0.3.65",
-    "@dirsql/lib-darwin-arm64": "0.3.65",
-    "@dirsql/lib-win32-x64-msvc": "0.3.65",
-    "@dirsql/cli-linux-x64-gnu": "0.3.65",
-    "@dirsql/cli-linux-arm64-gnu": "0.3.65",
-    "@dirsql/cli-darwin-x64": "0.3.65",
-    "@dirsql/cli-darwin-arm64": "0.3.65",
-    "@dirsql/cli-win32-x64-msvc": "0.3.65"
+    "@dirsql/lib-linux-x64-gnu": "0.3.67",
+    "@dirsql/lib-linux-arm64-gnu": "0.3.67",
+    "@dirsql/lib-darwin-x64": "0.3.67",
+    "@dirsql/lib-darwin-arm64": "0.3.67",
+    "@dirsql/lib-win32-x64-msvc": "0.3.67",
+    "@dirsql/cli-linux-x64-gnu": "0.3.67",
+    "@dirsql/cli-linux-arm64-gnu": "0.3.67",
+    "@dirsql/cli-darwin-x64": "0.3.67",
+    "@dirsql/cli-darwin-arm64": "0.3.67",
+    "@dirsql/cli-win32-x64-msvc": "0.3.67"
   }
 }

package/docs/api/index.md

@@ -1,238 +0,0 @@
----
-canonical: https://thekevinscott.github.io/dirsql/api/
----
-
-# API Reference
-
-> Online: <https://thekevinscott.github.io/dirsql/api/>
-
-## DirSQL
-
-### Import
-
-::: code-group
-
-```python [Python]
-from dirsql import DirSQL
-```
-
-```rust [Rust]
-use dirsql::DirSQL;
-```
-
-```typescript [TypeScript]
-import { DirSQL } from 'dirsql';
-```
-
-:::
-
-### Constructor
-
-::: code-group
-
-```python [Python]
-DirSQL(
-    root: str | None = None,
-    *,
-    tables: list[Table] | None = None,
-    ignore: list[str] | None = None,
-    config: str | None = None,
-    extensions: list[dict] | None = None,  # [{ "path": str, "entrypoint"?: str }]
-)
-```
-
-```rust [Rust]
-DirSQL::builder()
-    .root(root)                 // optional
-    .tables(tables)             // optional; append with .table(t)
-    .ignore(patterns)           // optional
-    .config(config_toml_path)   // optional
-    .extensions(extensions)     // optional; Extension { path, entrypoint }
-    .build()                    // -> Result<DirSQL>
-```
-
-```typescript [TypeScript]
-new DirSQL(configPath: string)
-// or
-new DirSQL({
-    root?: string,
-    tables?: TableDef[],
-    ignore?: string[],
-    config?: string,
-    extensions?: ExtensionSpec[],  // [{ path: string, entrypoint?: string }]
-})
-```
-
-:::
-
-Creates an in-memory SQLite index over the given directory. At least one of `root` or `config` must be supplied.
-
-When both `root` and `config` are supplied -- or when `config` declares `[dirsql].root` -- the explicit `root` wins and a warning is emitted on stderr. A `[dirsql].root` declared in the config file is resolved relative to the config file's parent directory.
-
-In Python, the constructor starts scanning in a background thread and returns immediately. Call `await db.ready()` before querying. In Rust, `.build()` scans synchronously; use `.build_async()` (via `AsyncDirSQL`) for the tokio-driven equivalent. In TypeScript, scanning starts immediately and `db.ready` resolves when the scan finishes.
-
-**Parameters:**
-
-- `root` -- Path to the directory to index. Optional if `config` is supplied.
-- `tables` -- List of `Table` definitions. Each defines a SQLite table, a glob pattern, and an extract function.
-- `ignore` -- Optional list of glob patterns. Files matching any ignore pattern are skipped regardless of table globs.
-- `config` -- Optional path to a `.dirsql.toml` config file. Its `[[table]]` entries are appended to any programmatic `tables`; its `[dirsql].ignore` patterns are appended to any explicit `ignore`; its optional `[dirsql].root` supplies the root directory when `root` is not passed explicitly; its `[[dirsql.extension]]` entries are appended to any programmatic `extensions`.
-- `extensions` -- Optional SQLite extensions to load onto the connection at startup, before any table DDL (enable → load → disable, so the SQL `load_extension()` function is never left exposed). Each entry pairs a shared-library `path` with an optional `entrypoint` init-symbol override (Python: `{ "path", "entrypoint"? }` dicts; Rust: `Extension { path, entrypoint }`; TypeScript: `{ path, entrypoint? }` objects). A `path` is either a file path or a bare **package name**; a package name is resolved from the installed package (Python `importlib` / TypeScript `node_modules`, file-first, erroring on zero or multiple matches). This works on the Python/TypeScript constructor — for both the programmatic `extensions` list and a `config` file's `[[dirsql.extension]]` entries — and on the CLI; the Rust binary and SDK are file-path-only. Programmatic entries load first, then any `[[dirsql.extension]]` from `config`. See [Loading extensions](../cli/config.md#loading-extensions).
-
-### Methods
-
-#### `ready`
-
-::: code-group
-
-```python [Python]
-await db.ready() -> None
-```
-
-```rust [Rust]
-db.ready().await -> Result<()>
-```
-
-```typescript [TypeScript]
-await db.ready  // awaitable property
-```
-
-:::
-
-Wait for the initial scan to complete. Re-raises any exception from the scan. Safe to call multiple times.
-
-#### `query`
-
-::: code-group
-
-```python [Python]
-await db.query(sql: str) -> list[dict]
-```
-
-```rust [Rust]
-db.query(sql: &str) -> Result<Vec<HashMap<String, Value>>>
-```
-
-```typescript [TypeScript]
-await db.query(sql: string): Promise<Record<string, unknown>[]>
-```
-
-:::
-
-Execute a SQL query against the in-memory database. Returns results keyed by column name. Internal tracking columns (`_dirsql_file_path`, `_dirsql_row_index`) are excluded from results.
-
-#### `watch`
-
-::: code-group
-
-```python [Python]
-async for event in db.watch():  # AsyncIterator[RowEvent]
-    ...
-```
-
-```rust [Rust]
-let mut stream = db.watch();  // impl Stream<Item = RowEvent>
-while let Some(event) = stream.next().await { ... }
-```
-
-```typescript [TypeScript]
-for await (const event of db.watch()) {  // AsyncIterable<RowEvent>
-    ...
-}
-```
-
-:::
-
-Returns an async iterable of `RowEvent` objects. The file watcher starts automatically on first iteration. The iterator never terminates on its own.
-
----
-
-## Table
-
-### Import
-
-::: code-group
-
-```python [Python]
-from dirsql import Table
-```
-
-```rust [Rust]
-use dirsql::Table;
-```
-
-```typescript [TypeScript]
-import { Table } from 'dirsql';
-```
-
-:::
-
-### Constructor
-
-::: code-group
-
-```python [Python]
-Table(*, ddl: str, glob: str, extract: Callable[[str], list[dict]], strict: bool = False)
-```
-
-```rust [Rust]
-// Row = HashMap<String, Value>
-Table::new(ddl: &str, glob: &str, extract: fn(&str) -> Vec<HashMap<String, Value>>)
-```
-
-```typescript [TypeScript]
-new Table({ ddl: string, glob: string, extract: (path: string) => Record<string, unknown>[] })
-```
-
-:::
-
-Defines a mapping from files to SQLite table rows.
-
-**Parameters:**
-
-- `ddl` -- A `CREATE TABLE` statement. The table name is parsed from this DDL.
-- `glob` -- A glob pattern matched against file paths relative to the root directory.
-- `extract` -- A callable `(path) -> list[dict]`. Receives the path of the matched file -- relative to the scan root, or absolute when `root` is absolute. `dirsql` does not read file contents; a callback that needs the file body reads `path` itself. Returns a list of dicts/maps mapping column names to values. Return an empty list to skip a file.
-- `strict` -- Optional (default `False`). Controls row/schema validation. In the default relaxed mode, extra row keys are dropped and missing columns become `NULL`. When `True`, every row key must be a valid column identifier and any extra or missing key raises an error.
-
-**Attributes:**
-
-- `ddl` -- The DDL string (read-only).
-- `glob` -- The glob pattern (read-only).
-
----
-
-## RowEvent
-
-### Import
-
-::: code-group
-
-```python [Python]
-from dirsql import RowEvent
-```
-
-```rust [Rust]
-use dirsql::RowEvent;
-```
-
-```typescript [TypeScript]
-import type { RowEvent } from 'dirsql';
-```
-
-:::
-
-Emitted by the watch stream. Represents a change to a row in the database caused by a filesystem event.
-
-**Attributes:**
-
-| Attribute | Python | Rust | TypeScript |
-|-----------|--------|------|------------|
-| Table name | `table: str` | `table: String` | `table: string` |
-| Action | `action: str` | `action: Action` | `action: string` |
-| Current/new row | `row: dict \| None` | `row: Option<HashMap>` | `row?: Record` |
-| Previous row | `old_row: dict \| None` | `old_row: Option<HashMap>` | `oldRow?: Record` |
-| Error message | `error: str \| None` | `error: Option<String>` | `error?: string` |
-| File path | `file_path: str \| None` | `file_path: Option<String>` | `filePath?: string` |
-
-Action values: `"insert"`, `"update"`, `"delete"`, `"error"`.