dirsql 0.3.64 → 0.3.66

package/docs/guide/querying.md

@@ -1,221 +0,0 @@
----
-canonical: https://thekevinscott.github.io/dirsql/guide/querying
----
-
-# Querying
-
-> Online: <https://thekevinscott.github.io/dirsql/guide/querying>
-
-Once a `DirSQL` instance is created, the initial directory scan is complete and you can run SQL queries against the indexed data.
-
-::: tip From the CLI
-The `dirsql` HTTP server exposes the same query interface over [`POST /query`](../cli/http-api.md#post-query). Send `{"sql": "..."}` and get back the same JSON array of row objects. See the [CLI section](../cli/) for the full server setup.
-:::
-
-## Basic queries
-
-::: code-group
-
-```python [Python]
-# All rows from a table
-results = db.query("SELECT * FROM comments")
-
-# Filter with WHERE
-results = db.query("SELECT * FROM comments WHERE author = 'alice'")
-
-# Aggregations
-results = db.query("SELECT author, COUNT(*) as n FROM comments GROUP BY author")
-
-# JOINs across tables
-results = db.query("""
-    SELECT posts.title, authors.name
-    FROM posts
-    JOIN authors ON posts.author_id = authors.id
-""")
-```
-
-```rust [Rust]
-// All rows from a table
-let results = db.query("SELECT * FROM comments")?;
-
-// Filter with WHERE
-let results = db.query("SELECT * FROM comments WHERE author = 'alice'")?;
-
-// Aggregations
-let results = db.query("SELECT author, COUNT(*) as n FROM comments GROUP BY author")?;
-
-// JOINs across tables
-let results = db.query(
-    "SELECT posts.title, authors.name \
-     FROM posts JOIN authors ON posts.author_id = authors.id"
-)?;
-```
-
-```typescript [TypeScript]
-// All rows from a table
-const results = await db.query('SELECT * FROM comments');
-
-// Filter with WHERE
-const filtered = await db.query("SELECT * FROM comments WHERE author = 'alice'");
-
-// Aggregations
-const counts = await db.query('SELECT author, COUNT(*) as n FROM comments GROUP BY author');
-
-// JOINs across tables
-const joined = await db.query(`
-  SELECT posts.title, authors.name
-  FROM posts
-  JOIN authors ON posts.author_id = authors.id
-`);
-```
-
-:::
-
-Any valid SQLite **SELECT** works. The in-memory database supports the full SQLite dialect including subqueries, CTEs, window functions, and aggregate functions. See [Read-only queries](#read-only-queries) below for why write statements (`INSERT`, `UPDATE`, `DELETE`, `DROP`, etc.) are rejected.
-
-## Return format
-
-`query()` returns a list of dicts (Python), a `Vec<HashMap>` (Rust), or an array of objects (TypeScript). Each entry maps column names to values.
-
-::: code-group
-
-```python [Python]
-results = db.query("SELECT title, author FROM posts")
-# [
-#     {"title": "Hello World", "author": "alice"},
-#     {"title": "Second Post", "author": "bob"},
-# ]
-```
-
-```rust [Rust]
-let results = db.query("SELECT title, author FROM posts")?;
-// Vec<HashMap<String, Value>>
-// [{"title": "Hello World", "author": "alice"}, ...]
-```
-
-```typescript [TypeScript]
-const results = await db.query('SELECT title, author FROM posts');
-// [
-//   { title: 'Hello World', author: 'alice' },
-//   { title: 'Second Post', author: 'bob' },
-// ]
-```
-
-:::
-
-SQLite types map back to language types:
-
-| SQLite type | Python  | TypeScript | Rust             |
-|-------------|---------|------------|------------------|
-| TEXT        | `str`   | `string`   | `Value::Text`    |
-| INTEGER     | `int`   | `number`   | `Value::Integer` |
-| REAL        | `float` | `number`   | `Value::Real`    |
-| BLOB        | `bytes` | `Buffer`   | `Value::Blob`    |
-| NULL        | `None`  | `null`     | `Value::Null`    |
-
-## Internal columns
-
-`dirsql` adds internal tracking columns (`_dirsql_file_path`, `_dirsql_row_index`) to each table for file-change diffing. These columns are automatically excluded from `SELECT *` results, so day-to-day queries don't need to account for them.
-
-If you want to know which file a row came from, you can name the tracking columns explicitly in the projection:
-
-::: code-group
-
-```python [Python]
-rows = db.query("SELECT title, _dirsql_file_path FROM posts")
-# [{"title": "Hello World", "_dirsql_file_path": "posts/hello.json"}, ...]
-```
-
-```rust [Rust]
-let rows = db.query("SELECT title, _dirsql_file_path FROM posts")?;
-// [{"title": "Hello World", "_dirsql_file_path": "posts/hello.json"}, ...]
-```
-
-```typescript [TypeScript]
-const rows = await db.query('SELECT title, _dirsql_file_path FROM posts');
-// [{ title: 'Hello World', _dirsql_file_path: 'posts/hello.json' }, ...]
-```
-
-:::
-
-Tracking columns are only returned when named explicitly — `SELECT *` continues to exclude them.
-
-## Read-only queries
-
-`query()` accepts only read-only statements. Each statement is prepared on SQLite and then classified via `sqlite3_stmt_readonly`; anything SQLite itself flags as a write — `INSERT`, `UPDATE`, `DELETE`, `DROP`, `CREATE`, `ALTER`, `REPLACE`, `VACUUM`, `ANALYZE`, etc. — is rejected before any rows are produced.
-
-This keeps the in-memory index consistent with the on-disk files that back it. Mutations only happen through the watcher/indexer pipeline: to change data, edit the underlying file and let the watcher re-extract rows.
-
-::: code-group
-
-```python [Python]
-# Raises a RuntimeError; the index is unchanged.
-db.query("DELETE FROM posts")
-```
-
-```rust [Rust]
-// Returns DirSqlError::WriteForbidden; the index is unchanged.
-let err = db.query("DELETE FROM posts").unwrap_err();
-assert!(matches!(err, dirsql::DirSqlError::WriteForbidden));
-```
-
-```typescript [TypeScript]
-// Rejects with an Error whose message explains writes are not accepted.
-// `db.query` is async, so assert on the rejected promise.
-await expect(db.query('DELETE FROM posts')).rejects.toThrow(/read-only/i);
-```
-
-:::
-
-## Error handling
-
-Invalid SQL raises an exception:
-
-::: code-group
-
-```python [Python]
-try:
-    db.query("NOT VALID SQL")
-except Exception as e:
-    print(f"Query error: {e}")
-```
-
-```rust [Rust]
-match db.query("NOT VALID SQL") {
-    Ok(results) => println!("{:?}", results),
-    Err(e) => eprintln!("Query error: {}", e),
-}
-```
-
-```typescript [TypeScript]
-try {
-  await db.query('NOT VALID SQL');
-} catch (e) {
-  console.error(`Query error: ${e}`);
-}
-```
-
-:::
-
-## Empty results
-
-Queries that match no rows return an empty collection:
-
-::: code-group
-
-```python [Python]
-results = db.query("SELECT * FROM posts WHERE author = 'nobody'")
-assert results == []
-```
-
-```rust [Rust]
-let results = db.query("SELECT * FROM posts WHERE author = 'nobody'")?;
-assert!(results.is_empty());
-```
-
-```typescript [TypeScript]
-const results = await db.query("SELECT * FROM posts WHERE author = 'nobody'");
-console.assert(results.length === 0);
-```
-
-:::

package/docs/guide/tables.md

@@ -1,269 +0,0 @@
----
-canonical: https://thekevinscott.github.io/dirsql/guide/tables
----
-
-# Defining Tables
-
-> Online: <https://thekevinscott.github.io/dirsql/guide/tables>
-
-Each table in `dirsql` maps a set of files to rows in an in-memory SQLite table. A table definition has three parts: DDL, a glob pattern, and an extract function.
-
-## Table constructor
-
-::: code-group
-
-```python [Python]
-from dirsql import Table
-
-table = Table(
-    ddl="CREATE TABLE comments (id TEXT, body TEXT, author TEXT)",
-    glob="comments/**/index.jsonl",
-    extract=lambda path: [
-        {"id": "...", "body": "...", "author": "..."}
-    ],
-)
-```
-
-```rust [Rust]
-use dirsql::{Table, Value};
-use std::collections::HashMap;
-
-let table = Table::new(
-    "CREATE TABLE comments (id TEXT, body TEXT, author TEXT)",
-    "comments/**/index.jsonl",
-    |_path| {
-        let mut row: HashMap<String, Value> = HashMap::new();
-        row.insert("id".into(), Value::Text("...".into()));
-        row.insert("body".into(), Value::Text("...".into()));
-        row.insert("author".into(), Value::Text("...".into()));
-        vec![row]
-    },
-);
-```
-
-```typescript [TypeScript]
-import type { TableDef } from 'dirsql';
-
-const table: TableDef = {
-  ddl: 'CREATE TABLE comments (id TEXT, body TEXT, author TEXT)',
-  glob: 'comments/**/index.jsonl',
-  extract: (_path) => [
-    { id: '...', body: '...', author: '...' },
-  ],
-};
-```
-
-:::
-
-All three arguments are keyword-only (in Python). In Rust they are positional to `Table::new`. In TypeScript a table is a plain `TableDef` object literal — the TS SDK exports the `TableDef` type (not a class).
-
-### `ddl`
-
-A SQLite `CREATE TABLE` statement. This defines the schema of the table. `dirsql` executes this DDL directly against the in-memory database, so any valid SQLite column types and constraints work.
-
-```python
-# Simple text columns
-ddl="CREATE TABLE notes (title TEXT, body TEXT)"
-
-# Typed columns
-ddl="CREATE TABLE metrics (name TEXT, value REAL, count INTEGER)"
-
-# With constraints
-ddl="CREATE TABLE items (id TEXT PRIMARY KEY, name TEXT NOT NULL)"
-```
-
-The table name is parsed from the DDL and must resolve to a valid SQLite identifier.
-
-### `glob`
-
-A glob pattern that determines which files feed into this table. Matched relative to the root directory passed to `DirSQL`.
-
-```python
-glob="*.json"                  # JSON files in root only
-glob="**/*.json"               # JSON files at any depth
-glob="comments/**/index.jsonl" # JSONL files in comment subdirectories
-glob="data/*.csv"              # CSV files in data/
-```
-
-Glob syntax follows standard Unix globbing rules. `**` matches any number of directory levels.
-
-### `extract`
-
-A callable `(path: str) -> list[dict]` that converts a file into rows.
-
-- `path` is the path of the matched file, **relative to the scan root** (or absolute when the `root` passed to `DirSQL` is absolute)
-- Return a list of dicts, where each dict maps column names to values
-- Return an empty list to skip a file
-
-`dirsql` does not read file contents for you. If your extract needs the file
-body, read it inside the callback using `path`. Callbacks that derive columns
-only from the path (or that rely solely on the auto-injected filesystem-fact
-columns) never touch the file at all.
-
-```python
-import json
-
-# Single-object JSON files: one row per file
-def extract(path):
-    with open(path, encoding="utf-8") as f:
-        return [json.loads(f.read())]
-
-# JSONL files: one row per line
-def extract(path):
-    with open(path, encoding="utf-8") as f:
-        return [json.loads(line) for line in f]
-
-# Derive a value from the file path alone -- no file read
-import os
-extract = lambda path: [{"id": os.path.basename(os.path.dirname(path))}]
-
-# Conditionally skip files
-def extract(path):
-    with open(path, encoding="utf-8") as f:
-        data = json.loads(f.read())
-    if data.get("draft"):
-        return []
-    return [data]
-```
-
-## Multiple tables
-
-Pass multiple `Table` definitions to index different file types into separate tables:
-
-::: code-group
-
-```python [Python]
-from dirsql import DirSQL, Table
-import json
-
-db = DirSQL(
-    "./workspace",
-    tables=[
-        Table(
-            ddl="CREATE TABLE posts (title TEXT, author_id TEXT)",
-            glob="posts/*.json",
-            extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
-        ),
-        Table(
-            ddl="CREATE TABLE authors (id TEXT, name TEXT)",
-            glob="authors/*.json",
-            extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
-        ),
-    ],
-)
-```
-
-```rust [Rust]
-use dirsql::{DirSQL, Table, Value};
-use std::collections::HashMap;
-
-// See `row_from_json` in getting-started.md for a reusable helper.
-fn row_from_json(raw: &str) -> HashMap<String, Value> {
-    let v: serde_json::Value = serde_json::from_str(raw).unwrap();
-    let serde_json::Value::Object(obj) = v else { return HashMap::new() };
-    obj.into_iter()
-        .map(|(k, val)| {
-            let v = match val {
-                serde_json::Value::String(s) => Value::Text(s),
-                serde_json::Value::Number(n) => n
-                    .as_i64()
-                    .map(Value::Integer)
-                    .unwrap_or_else(|| Value::Real(n.as_f64().unwrap_or(0.0))),
-                serde_json::Value::Bool(b) => Value::Integer(b as i64),
-                serde_json::Value::Null => Value::Null,
-                other => Value::Text(other.to_string()),
-            };
-            (k, v)
-        })
-        .collect()
-}
-
-let db = DirSQL::new(
-    "./workspace",
-    vec![
-        Table::new(
-            "CREATE TABLE posts (title TEXT, author_id TEXT)",
-            "posts/*.json",
-            |path| vec![row_from_json(&std::fs::read_to_string(path).unwrap())],
-        ),
-        Table::new(
-            "CREATE TABLE authors (id TEXT, name TEXT)",
-            "authors/*.json",
-            |path| vec![row_from_json(&std::fs::read_to_string(path).unwrap())],
-        ),
-    ],
-)?;
-```
-
-```typescript [TypeScript]
-import { DirSQL, type TableDef } from 'dirsql';
-import { readFileSync } from 'node:fs';
-
-const tables: TableDef[] = [
-  {
-    ddl: 'CREATE TABLE posts (title TEXT, author_id TEXT)',
-    glob: 'posts/*.json',
-    extract: (path) => [JSON.parse(readFileSync(path, 'utf8'))],
-  },
-  {
-    ddl: 'CREATE TABLE authors (id TEXT, name TEXT)',
-    glob: 'authors/*.json',
-    extract: (path) => [JSON.parse(readFileSync(path, 'utf8'))],
-  },
-];
-
-const db = new DirSQL({ root: './workspace', tables });
-```
-
-:::
-
-Each table has its own glob and extract function. A file can only match one table (the first matching glob wins).
-
-## Ignore patterns
-
-Use the `ignore` parameter to exclude paths from all tables:
-
-::: code-group
-
-```python [Python]
-db = DirSQL(
-    "./workspace",
-    ignore=["**/node_modules/**", "**/.git/**"],
-    tables=[...],
-)
-```
-
-```rust [Rust]
-let db = DirSQL::with_ignore(
-    "./workspace",
-    vec![/* tables */],
-    vec!["**/node_modules/**", "**/.git/**"],
-)?;
-```
-
-```typescript [TypeScript]
-const db = new DirSQL({
-  root: './workspace',
-  tables: [/* tables */],
-  ignore: ['**/node_modules/**', '**/.git/**'],
-});
-```
-
-:::
-
-Ignore patterns are applied before glob matching. Any file matching an ignore pattern is skipped regardless of table globs.
-
-## Supported value types
-
-The extract function can return these types, which map to SQLite types:
-
-| SQLite type   | Python  | TypeScript              | Rust             |
-|---------------|---------|-------------------------|------------------|
-| TEXT          | `str`   | `string`                | `Value::Text`    |
-| INTEGER       | `int`   | integral `number`       | `Value::Integer` |
-| REAL          | `float` | fractional `number`     | `Value::Real`    |
-| INTEGER (0/1) | `bool`  | `boolean`               | `Value::Integer` |
-| BLOB          | `bytes` | `Buffer` / `Uint8Array` | `Value::Blob`    |
-| NULL          | `None`  | `null` / `undefined`    | `Value::Null`    |
-
-Any other type is converted to its string representation (`str()` in Python, string coercion in TypeScript).