dirsql 0.3.28__tar.gz → 0.3.29__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {dirsql-0.3.28 → dirsql-0.3.29}/Cargo.lock +1 -1
- dirsql-0.3.29/PKG-INFO +121 -0
- dirsql-0.3.29/README.md +102 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/Cargo.toml +1 -1
- dirsql-0.3.29/packages/python/README.md +102 -0
- dirsql-0.3.29/packages/rust/README.md +139 -0
- dirsql-0.3.28/PKG-INFO +0 -208
- dirsql-0.3.28/README.md +0 -189
- dirsql-0.3.28/packages/python/README.md +0 -189
- dirsql-0.3.28/packages/rust/README.md +0 -56
- {dirsql-0.3.28 → dirsql-0.3.29}/Cargo.toml +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/__init__.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/_async.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/_async_test.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/_dirsql.pyi +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/__init__.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/binary_path.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/binary_path_test.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/interpret/__init__.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/interpret/dispatch_extract.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/interpret/dispatch_extract_test.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/interpret/load_app.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/interpret/load_app_test.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/interpret/run.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/interpret/run_test.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/interpret/write_message.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/interpret/write_message_test.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/is_windows.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/is_windows_test.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/main.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/cli/main_test.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/py.typed +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/resolve_config.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/dirsql/resolve_config_test.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/.claude/CLAUDE.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/.vitepress/config.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/.vitepress/theme/index.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/.vitepress/theme/lang.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/AGENTS.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/api/index.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/cli/config.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/cli/http-api.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/cli/index.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/cli/init.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/cli/server.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/getting-started.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/guide/async.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/guide/crdt.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/guide/persistence.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/guide/querying.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/guide/tables.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/guide/watching.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/index.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/migrations.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/package.json +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/playwright.config.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/pnpm-lock.yaml +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/pnpm-workspace.yaml +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/tests/integration/home.spec.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/tests/integration/language-flag.spec.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/tests/unit/config.test.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/tests/unit/lang.test.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/docs/vitest.config.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/conftest.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/.claude/CLAUDE.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/.vitepress/config.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/.vitepress/theme/index.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/.vitepress/theme/lang.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/AGENTS.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/api/index.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/cli/config.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/cli/http-api.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/cli/index.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/cli/init.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/cli/server.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/getting-started.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/guide/async.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/guide/crdt.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/guide/persistence.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/guide/querying.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/guide/tables.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/guide/watching.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/index.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/migrations.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/package.json +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/playwright.config.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/pnpm-lock.yaml +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/pnpm-workspace.yaml +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/tests/integration/home.spec.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/tests/integration/language-flag.spec.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/tests/unit/config.test.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/tests/unit/lang.test.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/docs/vitest.config.ts +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/src/lib.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/__init__.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/conftest.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/e2e/__init__.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/__fixtures__/data/a/meta.json +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/__fixtures__/data/b/meta.json +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/__fixtures__/dirsql.config.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/__fixtures__/interpret/data/a/meta.json +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/__fixtures__/interpret/data/b/meta.json +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/__fixtures__/interpret/dirsql.config.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/__fixtures__/interpret/dirsql.config_no_app.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/__fixtures__/interpret/dirsql.config_raises.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/__init__.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/interpret_subprocess.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/test_async_dirsql.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/test_binding.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/test_dirsql.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/test_docs_examples.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/test_docs_gaps.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/test_from_config.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/test_interpret.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/test_native_config.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/test_persist.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/python/tests/integration/test_serialization.py +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/Cargo.toml +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/benches/db_bench.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/benches/differ_bench.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/benches/matcher_bench.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/benches/scanner_bench.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/api/index.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/cli/config.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/cli/http-api.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/cli/index.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/cli/init.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/cli/server.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/getting-started.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/guide/async.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/guide/crdt.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/guide/persistence.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/guide/querying.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/guide/tables.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/guide/watching.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/index.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/docs/migrations.md +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/bin/dirsql.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/cli/init.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/cli/mod.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/cli/native_config.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/cli/router.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/cli/serialize.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/cli/server.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/config.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/db.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/differ.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/lib.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/matcher.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/persist.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/scanner.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/src/watcher.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/async_sdk.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/cli_e2e.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/cli_integration.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/code_review_findings.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/config.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/docs_examples.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/docs_gaps.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/extensions.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/from_config.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/init_e2e.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/init_integration.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/persist.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/readonly_query.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/scanner.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/sdk.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/serialization.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/watch_relative_root.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/packages/rust/tests/watcher.rs +0 -0
- {dirsql-0.3.28 → dirsql-0.3.29}/pyproject.toml +0 -0
dirsql-0.3.29/PKG-INFO
ADDED
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: dirsql
|
|
3
|
+
Version: 0.3.29
|
|
4
|
+
Requires-Dist: pytest>=8 ; extra == 'dev'
|
|
5
|
+
Requires-Dist: pytest-describe>=2 ; extra == 'dev'
|
|
6
|
+
Requires-Dist: pytest-asyncio>=0.23 ; extra == 'dev'
|
|
7
|
+
Requires-Dist: pytest-cov>=5 ; extra == 'dev'
|
|
8
|
+
Requires-Dist: ruff>=0.4 ; extra == 'dev'
|
|
9
|
+
Requires-Dist: maturin>=1.0 ; extra == 'dev'
|
|
10
|
+
Requires-Dist: ty==0.0.42 ; extra == 'dev'
|
|
11
|
+
Provides-Extra: dev
|
|
12
|
+
Summary: Ephemeral SQL index over a local directory
|
|
13
|
+
Keywords: sql,filesystem,directory,sqlite,index
|
|
14
|
+
Author: Kevin Scott
|
|
15
|
+
License-Expression: MIT
|
|
16
|
+
Requires-Python: >=3.11
|
|
17
|
+
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
|
|
18
|
+
|
|
19
|
+
# `dirsql` (Python SDK)
|
|
20
|
+
|
|
21
|
+
Ephemeral SQL index over a local directory. `dirsql` watches a filesystem, ingests structured files into an in-memory SQLite database, and exposes a SQL query interface -- the filesystem is always the source of truth.
|
|
22
|
+
|
|
23
|
+
[Documentation](https://thekevinscott.github.io/dirsql/?lang=python)
|
|
24
|
+
|
|
25
|
+
Also available as [`dirsql` on crates.io](https://crates.io/crates/dirsql) and [`dirsql` on npm](https://www.npmjs.com/package/dirsql).
|
|
26
|
+
|
|
27
|
+
## Installation
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
pip install dirsql
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Requires Python >= 3.12. Ships as a native extension (Rust via PyO3); prebuilt binary wheels are provided for common platforms.
|
|
34
|
+
|
|
35
|
+
## Quick start
|
|
36
|
+
|
|
37
|
+
`DirSQL` is async by default: the constructor returns immediately, scanning runs in a background thread, and you `await db.ready()` before querying. Each table is a `(ddl, glob, extract)` triple: the DDL defines the SQLite schema, the glob selects files (relative to the root), and `extract` turns a matched file into a list of row dicts. `dirsql` does not read file contents -- if `extract` needs the file body it reads `path` itself; return an empty list to skip a file.
|
|
38
|
+
|
|
39
|
+
```python
|
|
40
|
+
import asyncio
|
|
41
|
+
import json
|
|
42
|
+
from dirsql import DirSQL, Table
|
|
43
|
+
|
|
44
|
+
async def main():
|
|
45
|
+
db = DirSQL(
|
|
46
|
+
"./my-blog",
|
|
47
|
+
tables=[
|
|
48
|
+
Table(
|
|
49
|
+
ddl="CREATE TABLE posts (title TEXT, author TEXT)",
|
|
50
|
+
glob="posts/*.json",
|
|
51
|
+
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
52
|
+
),
|
|
53
|
+
],
|
|
54
|
+
)
|
|
55
|
+
await db.ready()
|
|
56
|
+
|
|
57
|
+
posts = await db.query("SELECT * FROM posts WHERE author = 'alice'")
|
|
58
|
+
print(posts)
|
|
59
|
+
|
|
60
|
+
asyncio.run(main())
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
## Multiple tables and joins
|
|
64
|
+
|
|
65
|
+
```python
|
|
66
|
+
db = DirSQL(
|
|
67
|
+
"./my-blog",
|
|
68
|
+
tables=[
|
|
69
|
+
Table(
|
|
70
|
+
ddl="CREATE TABLE posts (title TEXT, author_id TEXT)",
|
|
71
|
+
glob="posts/*.json",
|
|
72
|
+
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
73
|
+
),
|
|
74
|
+
Table(
|
|
75
|
+
ddl="CREATE TABLE authors (id TEXT, name TEXT)",
|
|
76
|
+
glob="authors/*.json",
|
|
77
|
+
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
78
|
+
),
|
|
79
|
+
],
|
|
80
|
+
)
|
|
81
|
+
await db.ready()
|
|
82
|
+
|
|
83
|
+
results = await db.query("""
|
|
84
|
+
SELECT posts.title, authors.name
|
|
85
|
+
FROM posts JOIN authors ON posts.author_id = authors.id
|
|
86
|
+
""")
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
## Ignoring files
|
|
90
|
+
|
|
91
|
+
Pass `ignore` patterns to skip files during scanning and watching:
|
|
92
|
+
|
|
93
|
+
```python
|
|
94
|
+
db = DirSQL(
|
|
95
|
+
"./my-blog",
|
|
96
|
+
ignore=["**/drafts/**", "**/.git/**"],
|
|
97
|
+
tables=[...],
|
|
98
|
+
)
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
## Watching for changes
|
|
102
|
+
|
|
103
|
+
`db.watch()` returns an async iterator of row-level change events as files change on disk:
|
|
104
|
+
|
|
105
|
+
```python
|
|
106
|
+
async for event in db.watch():
|
|
107
|
+
print(f"{event.action} on {event.table}: {event.row}")
|
|
108
|
+
if event.action == "error":
|
|
109
|
+
print(f" error: {event.error}")
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Each event has `.action` (`"insert"`, `"update"`, `"delete"`, or `"error"`), `.table`, `.row` (the new row, or the deleted row on `delete`), `.old_row` (the previous row, on `update`), `.file_path`, and `.error` (on `error`).
|
|
113
|
+
|
|
114
|
+
## CLI
|
|
115
|
+
|
|
116
|
+
`pip install dirsql` also installs a `dirsql` console script that runs an HTTP server exposing the SDK over HTTP: `POST /query` for SQL and `GET /events` for a Server-Sent Events change stream. Run `dirsql` (or `uvx dirsql`) to start it. See the [CLI guide](https://thekevinscott.github.io/dirsql/cli/).
|
|
117
|
+
|
|
118
|
+
## License
|
|
119
|
+
|
|
120
|
+
MIT
|
|
121
|
+
|
dirsql-0.3.29/README.md
ADDED
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
# `dirsql` (Python SDK)
|
|
2
|
+
|
|
3
|
+
Ephemeral SQL index over a local directory. `dirsql` watches a filesystem, ingests structured files into an in-memory SQLite database, and exposes a SQL query interface -- the filesystem is always the source of truth.
|
|
4
|
+
|
|
5
|
+
[Documentation](https://thekevinscott.github.io/dirsql/?lang=python)
|
|
6
|
+
|
|
7
|
+
Also available as [`dirsql` on crates.io](https://crates.io/crates/dirsql) and [`dirsql` on npm](https://www.npmjs.com/package/dirsql).
|
|
8
|
+
|
|
9
|
+
## Installation
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
pip install dirsql
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Requires Python >= 3.12. Ships as a native extension (Rust via PyO3); prebuilt binary wheels are provided for common platforms.
|
|
16
|
+
|
|
17
|
+
## Quick start
|
|
18
|
+
|
|
19
|
+
`DirSQL` is async by default: the constructor returns immediately, scanning runs in a background thread, and you `await db.ready()` before querying. Each table is a `(ddl, glob, extract)` triple: the DDL defines the SQLite schema, the glob selects files (relative to the root), and `extract` turns a matched file into a list of row dicts. `dirsql` does not read file contents -- if `extract` needs the file body it reads `path` itself; return an empty list to skip a file.
|
|
20
|
+
|
|
21
|
+
```python
|
|
22
|
+
import asyncio
|
|
23
|
+
import json
|
|
24
|
+
from dirsql import DirSQL, Table
|
|
25
|
+
|
|
26
|
+
async def main():
|
|
27
|
+
db = DirSQL(
|
|
28
|
+
"./my-blog",
|
|
29
|
+
tables=[
|
|
30
|
+
Table(
|
|
31
|
+
ddl="CREATE TABLE posts (title TEXT, author TEXT)",
|
|
32
|
+
glob="posts/*.json",
|
|
33
|
+
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
34
|
+
),
|
|
35
|
+
],
|
|
36
|
+
)
|
|
37
|
+
await db.ready()
|
|
38
|
+
|
|
39
|
+
posts = await db.query("SELECT * FROM posts WHERE author = 'alice'")
|
|
40
|
+
print(posts)
|
|
41
|
+
|
|
42
|
+
asyncio.run(main())
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## Multiple tables and joins
|
|
46
|
+
|
|
47
|
+
```python
|
|
48
|
+
db = DirSQL(
|
|
49
|
+
"./my-blog",
|
|
50
|
+
tables=[
|
|
51
|
+
Table(
|
|
52
|
+
ddl="CREATE TABLE posts (title TEXT, author_id TEXT)",
|
|
53
|
+
glob="posts/*.json",
|
|
54
|
+
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
55
|
+
),
|
|
56
|
+
Table(
|
|
57
|
+
ddl="CREATE TABLE authors (id TEXT, name TEXT)",
|
|
58
|
+
glob="authors/*.json",
|
|
59
|
+
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
60
|
+
),
|
|
61
|
+
],
|
|
62
|
+
)
|
|
63
|
+
await db.ready()
|
|
64
|
+
|
|
65
|
+
results = await db.query("""
|
|
66
|
+
SELECT posts.title, authors.name
|
|
67
|
+
FROM posts JOIN authors ON posts.author_id = authors.id
|
|
68
|
+
""")
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
## Ignoring files
|
|
72
|
+
|
|
73
|
+
Pass `ignore` patterns to skip files during scanning and watching:
|
|
74
|
+
|
|
75
|
+
```python
|
|
76
|
+
db = DirSQL(
|
|
77
|
+
"./my-blog",
|
|
78
|
+
ignore=["**/drafts/**", "**/.git/**"],
|
|
79
|
+
tables=[...],
|
|
80
|
+
)
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
## Watching for changes
|
|
84
|
+
|
|
85
|
+
`db.watch()` returns an async iterator of row-level change events as files change on disk:
|
|
86
|
+
|
|
87
|
+
```python
|
|
88
|
+
async for event in db.watch():
|
|
89
|
+
print(f"{event.action} on {event.table}: {event.row}")
|
|
90
|
+
if event.action == "error":
|
|
91
|
+
print(f" error: {event.error}")
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
Each event has `.action` (`"insert"`, `"update"`, `"delete"`, or `"error"`), `.table`, `.row` (the new row, or the deleted row on `delete`), `.old_row` (the previous row, on `update`), `.file_path`, and `.error` (on `error`).
|
|
95
|
+
|
|
96
|
+
## CLI
|
|
97
|
+
|
|
98
|
+
`pip install dirsql` also installs a `dirsql` console script that runs an HTTP server exposing the SDK over HTTP: `POST /query` for SQL and `GET /events` for a Server-Sent Events change stream. Run `dirsql` (or `uvx dirsql`) to start it. See the [CLI guide](https://thekevinscott.github.io/dirsql/cli/).
|
|
99
|
+
|
|
100
|
+
## License
|
|
101
|
+
|
|
102
|
+
MIT
|
|
@@ -4,7 +4,7 @@ name = "dirsql-py-ext"
|
|
|
4
4
|
# pypi/maturin handler can rewrite it via `write-version` before
|
|
5
5
|
# `maturin build`. `pyproject.toml` declares `dynamic = ["version"]`
|
|
6
6
|
# and maturin reads this field. Mirrors `packages/rust/Cargo.toml`.
|
|
7
|
-
version = "0.3.
|
|
7
|
+
version = "0.3.29"
|
|
8
8
|
edition.workspace = true
|
|
9
9
|
publish = false
|
|
10
10
|
readme = "README.md"
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
# `dirsql` (Python SDK)
|
|
2
|
+
|
|
3
|
+
Ephemeral SQL index over a local directory. `dirsql` watches a filesystem, ingests structured files into an in-memory SQLite database, and exposes a SQL query interface -- the filesystem is always the source of truth.
|
|
4
|
+
|
|
5
|
+
[Documentation](https://thekevinscott.github.io/dirsql/?lang=python)
|
|
6
|
+
|
|
7
|
+
Also available as [`dirsql` on crates.io](https://crates.io/crates/dirsql) and [`dirsql` on npm](https://www.npmjs.com/package/dirsql).
|
|
8
|
+
|
|
9
|
+
## Installation
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
pip install dirsql
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Requires Python >= 3.12. Ships as a native extension (Rust via PyO3); prebuilt binary wheels are provided for common platforms.
|
|
16
|
+
|
|
17
|
+
## Quick start
|
|
18
|
+
|
|
19
|
+
`DirSQL` is async by default: the constructor returns immediately, scanning runs in a background thread, and you `await db.ready()` before querying. Each table is a `(ddl, glob, extract)` triple: the DDL defines the SQLite schema, the glob selects files (relative to the root), and `extract` turns a matched file into a list of row dicts. `dirsql` does not read file contents -- if `extract` needs the file body it reads `path` itself; return an empty list to skip a file.
|
|
20
|
+
|
|
21
|
+
```python
|
|
22
|
+
import asyncio
|
|
23
|
+
import json
|
|
24
|
+
from dirsql import DirSQL, Table
|
|
25
|
+
|
|
26
|
+
async def main():
|
|
27
|
+
db = DirSQL(
|
|
28
|
+
"./my-blog",
|
|
29
|
+
tables=[
|
|
30
|
+
Table(
|
|
31
|
+
ddl="CREATE TABLE posts (title TEXT, author TEXT)",
|
|
32
|
+
glob="posts/*.json",
|
|
33
|
+
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
34
|
+
),
|
|
35
|
+
],
|
|
36
|
+
)
|
|
37
|
+
await db.ready()
|
|
38
|
+
|
|
39
|
+
posts = await db.query("SELECT * FROM posts WHERE author = 'alice'")
|
|
40
|
+
print(posts)
|
|
41
|
+
|
|
42
|
+
asyncio.run(main())
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## Multiple tables and joins
|
|
46
|
+
|
|
47
|
+
```python
|
|
48
|
+
db = DirSQL(
|
|
49
|
+
"./my-blog",
|
|
50
|
+
tables=[
|
|
51
|
+
Table(
|
|
52
|
+
ddl="CREATE TABLE posts (title TEXT, author_id TEXT)",
|
|
53
|
+
glob="posts/*.json",
|
|
54
|
+
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
55
|
+
),
|
|
56
|
+
Table(
|
|
57
|
+
ddl="CREATE TABLE authors (id TEXT, name TEXT)",
|
|
58
|
+
glob="authors/*.json",
|
|
59
|
+
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
60
|
+
),
|
|
61
|
+
],
|
|
62
|
+
)
|
|
63
|
+
await db.ready()
|
|
64
|
+
|
|
65
|
+
results = await db.query("""
|
|
66
|
+
SELECT posts.title, authors.name
|
|
67
|
+
FROM posts JOIN authors ON posts.author_id = authors.id
|
|
68
|
+
""")
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
## Ignoring files
|
|
72
|
+
|
|
73
|
+
Pass `ignore` patterns to skip files during scanning and watching:
|
|
74
|
+
|
|
75
|
+
```python
|
|
76
|
+
db = DirSQL(
|
|
77
|
+
"./my-blog",
|
|
78
|
+
ignore=["**/drafts/**", "**/.git/**"],
|
|
79
|
+
tables=[...],
|
|
80
|
+
)
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
## Watching for changes
|
|
84
|
+
|
|
85
|
+
`db.watch()` returns an async iterator of row-level change events as files change on disk:
|
|
86
|
+
|
|
87
|
+
```python
|
|
88
|
+
async for event in db.watch():
|
|
89
|
+
print(f"{event.action} on {event.table}: {event.row}")
|
|
90
|
+
if event.action == "error":
|
|
91
|
+
print(f" error: {event.error}")
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
Each event has `.action` (`"insert"`, `"update"`, `"delete"`, or `"error"`), `.table`, `.row` (the new row, or the deleted row on `delete`), `.old_row` (the previous row, on `update`), `.file_path`, and `.error` (on `error`).
|
|
95
|
+
|
|
96
|
+
## CLI
|
|
97
|
+
|
|
98
|
+
`pip install dirsql` also installs a `dirsql` console script that runs an HTTP server exposing the SDK over HTTP: `POST /query` for SQL and `GET /events` for a Server-Sent Events change stream. Run `dirsql` (or `uvx dirsql`) to start it. See the [CLI guide](https://thekevinscott.github.io/dirsql/cli/).
|
|
99
|
+
|
|
100
|
+
## License
|
|
101
|
+
|
|
102
|
+
MIT
|
|
@@ -0,0 +1,139 @@
|
|
|
1
|
+
# `dirsql` (Rust SDK)
|
|
2
|
+
|
|
3
|
+
Ephemeral SQL index over a local directory. `dirsql` watches a filesystem, ingests structured files into an in-memory SQLite database, and exposes a SQL query interface -- the filesystem is always the source of truth.
|
|
4
|
+
|
|
5
|
+
[Documentation](https://thekevinscott.github.io/dirsql/?lang=rust)
|
|
6
|
+
|
|
7
|
+
Also available as [`dirsql` on PyPI](https://pypi.org/project/dirsql/) and [`dirsql` on npm](https://www.npmjs.com/package/dirsql).
|
|
8
|
+
|
|
9
|
+
## Installation
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
cargo add dirsql
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Quick start
|
|
16
|
+
|
|
17
|
+
`DirSQL::new` scans the directory synchronously and returns a ready instance. Each table is a `(ddl, glob, extract)` triple: the DDL defines the SQLite schema, the glob selects files (relative to the root), and `extract` turns a matched file into rows (`Vec<HashMap<String, Value>>`). `dirsql` does not read file contents -- the callback reads `path` itself; return an empty `Vec` to skip a file.
|
|
18
|
+
|
|
19
|
+
```rust
|
|
20
|
+
use dirsql::{DirSQL, Table, Value};
|
|
21
|
+
use std::collections::HashMap;
|
|
22
|
+
|
|
23
|
+
// Convert a JSON object string into a dirsql row. Reused by the examples below.
|
|
24
|
+
fn row_from_json(raw: &str) -> HashMap<String, Value> {
|
|
25
|
+
let v: serde_json::Value = serde_json::from_str(raw).unwrap();
|
|
26
|
+
let serde_json::Value::Object(obj) = v else { return HashMap::new() };
|
|
27
|
+
obj.into_iter()
|
|
28
|
+
.map(|(k, val)| {
|
|
29
|
+
let v = match val {
|
|
30
|
+
serde_json::Value::String(s) => Value::Text(s),
|
|
31
|
+
serde_json::Value::Number(n) => n
|
|
32
|
+
.as_i64()
|
|
33
|
+
.map(Value::Integer)
|
|
34
|
+
.unwrap_or_else(|| Value::Real(n.as_f64().unwrap_or(0.0))),
|
|
35
|
+
serde_json::Value::Bool(b) => Value::Integer(b as i64),
|
|
36
|
+
serde_json::Value::Null => Value::Null,
|
|
37
|
+
other => Value::Text(other.to_string()),
|
|
38
|
+
};
|
|
39
|
+
(k, v)
|
|
40
|
+
})
|
|
41
|
+
.collect()
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
let db = DirSQL::new(
|
|
45
|
+
"./my-blog",
|
|
46
|
+
vec![Table::new(
|
|
47
|
+
"CREATE TABLE posts (title TEXT, author TEXT)",
|
|
48
|
+
"posts/*.json",
|
|
49
|
+
|path| vec![row_from_json(&std::fs::read_to_string(path).unwrap())],
|
|
50
|
+
)],
|
|
51
|
+
)?;
|
|
52
|
+
|
|
53
|
+
let posts = db.query("SELECT * FROM posts WHERE author = 'alice'")?;
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Multiple tables and joins
|
|
57
|
+
|
|
58
|
+
```rust
|
|
59
|
+
let db = DirSQL::new(
|
|
60
|
+
"./my-blog",
|
|
61
|
+
vec![
|
|
62
|
+
Table::new(
|
|
63
|
+
"CREATE TABLE posts (title TEXT, author_id TEXT)",
|
|
64
|
+
"posts/*.json",
|
|
65
|
+
|path| vec![row_from_json(&std::fs::read_to_string(path).unwrap())],
|
|
66
|
+
),
|
|
67
|
+
Table::new(
|
|
68
|
+
"CREATE TABLE authors (id TEXT, name TEXT)",
|
|
69
|
+
"authors/*.json",
|
|
70
|
+
|path| vec![row_from_json(&std::fs::read_to_string(path).unwrap())],
|
|
71
|
+
),
|
|
72
|
+
],
|
|
73
|
+
)?;
|
|
74
|
+
|
|
75
|
+
let results = db.query(
|
|
76
|
+
"SELECT posts.title, authors.name \
|
|
77
|
+
FROM posts JOIN authors ON posts.author_id = authors.id",
|
|
78
|
+
)?;
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
## Ignoring files
|
|
82
|
+
|
|
83
|
+
Use `DirSQL::with_ignore` to skip files during scanning and watching:
|
|
84
|
+
|
|
85
|
+
```rust
|
|
86
|
+
let db = DirSQL::with_ignore(
|
|
87
|
+
"./my-blog",
|
|
88
|
+
vec![/* tables */],
|
|
89
|
+
vec!["**/drafts/**", "**/.git/**"],
|
|
90
|
+
)?;
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
## Watching for changes
|
|
94
|
+
|
|
95
|
+
`db.watch()` returns a stream of row-level change events as files change on disk. `.next()` comes from `StreamExt` in the `futures` crate (`cargo add futures`), driven inside an async runtime such as tokio:
|
|
96
|
+
|
|
97
|
+
```rust
|
|
98
|
+
use dirsql::RowEvent;
|
|
99
|
+
use futures::StreamExt;
|
|
100
|
+
|
|
101
|
+
let mut stream = db.watch()?;
|
|
102
|
+
while let Some(event) = stream.next().await {
|
|
103
|
+
match event {
|
|
104
|
+
RowEvent::Insert { table, row, file_path } => {
|
|
105
|
+
println!("insert on {table} ({file_path}): {row:?}")
|
|
106
|
+
}
|
|
107
|
+
RowEvent::Update { table, old_row, new_row, file_path } => {
|
|
108
|
+
println!("update on {table} ({file_path}): {old_row:?} -> {new_row:?}")
|
|
109
|
+
}
|
|
110
|
+
RowEvent::Delete { table, row, file_path } => {
|
|
111
|
+
println!("delete on {table} ({file_path}): {row:?}")
|
|
112
|
+
}
|
|
113
|
+
RowEvent::Error { table, file_path, error } => {
|
|
114
|
+
println!("error on {table:?} {file_path:?}: {error}")
|
|
115
|
+
}
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
## CLI
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
cargo install dirsql --features cli
|
|
124
|
+
dirsql
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
Running `dirsql` starts an HTTP server bound to `localhost:7117` that exposes the SDK over HTTP: `POST /query` for SQL and `GET /events` for a Server-Sent Events change stream. Override with `--host`, `--port`, `--config`. See the [CLI guide](https://thekevinscott.github.io/dirsql/cli/).
|
|
128
|
+
|
|
129
|
+
The `cli` feature is **opt-in** -- `cargo add dirsql` pulls no CLI dependencies. `cargo install dirsql` without `--features cli` silently installs nothing (`required-features` skips the bin target with no warning); always include the flag, or use `npx dirsql` / `uvx dirsql` for prebuilt binaries.
|
|
130
|
+
|
|
131
|
+
### Feature flags
|
|
132
|
+
|
|
133
|
+
| Feature | Default | Description |
|
|
134
|
+
|---|---|---|
|
|
135
|
+
| `cli` | no | Enables the `dirsql` binary and its dependencies. |
|
|
136
|
+
|
|
137
|
+
## License
|
|
138
|
+
|
|
139
|
+
MIT
|
dirsql-0.3.28/PKG-INFO
DELETED
|
@@ -1,208 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: dirsql
|
|
3
|
-
Version: 0.3.28
|
|
4
|
-
Requires-Dist: pytest>=8 ; extra == 'dev'
|
|
5
|
-
Requires-Dist: pytest-describe>=2 ; extra == 'dev'
|
|
6
|
-
Requires-Dist: pytest-asyncio>=0.23 ; extra == 'dev'
|
|
7
|
-
Requires-Dist: pytest-cov>=5 ; extra == 'dev'
|
|
8
|
-
Requires-Dist: ruff>=0.4 ; extra == 'dev'
|
|
9
|
-
Requires-Dist: maturin>=1.0 ; extra == 'dev'
|
|
10
|
-
Requires-Dist: ty==0.0.42 ; extra == 'dev'
|
|
11
|
-
Provides-Extra: dev
|
|
12
|
-
Summary: Ephemeral SQL index over a local directory
|
|
13
|
-
Keywords: sql,filesystem,directory,sqlite,index
|
|
14
|
-
Author: Kevin Scott
|
|
15
|
-
License-Expression: MIT
|
|
16
|
-
Requires-Python: >=3.11
|
|
17
|
-
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
|
|
18
|
-
|
|
19
|
-
# `dirsql` (Python SDK)
|
|
20
|
-
|
|
21
|
-
Ephemeral SQL index over a local directory. Watches a filesystem, ingests structured files into an in-memory SQLite database, and exposes a SQL query interface. The database is purely in-memory -- the filesystem is always the source of truth.
|
|
22
|
-
|
|
23
|
-
[Documentation](https://thekevinscott.github.io/dirsql/?lang=python)
|
|
24
|
-
|
|
25
|
-
## Installation
|
|
26
|
-
|
|
27
|
-
```bash
|
|
28
|
-
pip install dirsql
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
Requires Python >= 3.12. Ships as a native extension (Rust via PyO3) -- binary wheels are provided for common platforms.
|
|
32
|
-
|
|
33
|
-
Each wheel also bundles the `dirsql` HTTP-server CLI as a console script, so `pip install dirsql` also gives you a `dirsql` command on `$PATH`. See the [CLI guide](https://github.com/thekevinscott/dirsql/blob/main/docs/guide/cli.md).
|
|
34
|
-
|
|
35
|
-
## Publishing (maintainers)
|
|
36
|
-
|
|
37
|
-
Handled by `.github/workflows/publish.yml` (invoked from `minor-release.yml` / `patch-release.yml`). For each target triple the `build` job `cargo build`s the Rust CLI with `--features cli`, stages the binary into `dirsql/_binary/`, runs `maturin build` (which picks the binary up via the `[tool.maturin] include` rule in `pyproject.toml`), and the wheels + sdist are then trusted-published to PyPI.
|
|
38
|
-
|
|
39
|
-
## Quick Start
|
|
40
|
-
|
|
41
|
-
```python
|
|
42
|
-
import asyncio
|
|
43
|
-
import json
|
|
44
|
-
import os
|
|
45
|
-
import tempfile
|
|
46
|
-
from dirsql import DirSQL, Table
|
|
47
|
-
|
|
48
|
-
async def main():
|
|
49
|
-
# Create some data files
|
|
50
|
-
root = tempfile.mkdtemp()
|
|
51
|
-
os.makedirs(os.path.join(root, "comments", "abc"), exist_ok=True)
|
|
52
|
-
os.makedirs(os.path.join(root, "comments", "def"), exist_ok=True)
|
|
53
|
-
|
|
54
|
-
with open(os.path.join(root, "comments", "abc", "index.jsonl"), "w") as f:
|
|
55
|
-
f.write(json.dumps({"body": "looks good", "author": "alice"}) + "\n")
|
|
56
|
-
f.write(json.dumps({"body": "needs work", "author": "bob"}) + "\n")
|
|
57
|
-
|
|
58
|
-
with open(os.path.join(root, "comments", "def", "index.jsonl"), "w") as f:
|
|
59
|
-
f.write(json.dumps({"body": "agreed", "author": "carol"}) + "\n")
|
|
60
|
-
|
|
61
|
-
# Define a table: DDL, glob pattern, and an extract function
|
|
62
|
-
db = DirSQL(
|
|
63
|
-
root,
|
|
64
|
-
tables=[
|
|
65
|
-
Table(
|
|
66
|
-
ddl="CREATE TABLE comments (id TEXT, body TEXT, author TEXT)",
|
|
67
|
-
glob="comments/**/index.jsonl",
|
|
68
|
-
extract=lambda path: [
|
|
69
|
-
{
|
|
70
|
-
"id": os.path.basename(os.path.dirname(path)),
|
|
71
|
-
"body": row["body"],
|
|
72
|
-
"author": row["author"],
|
|
73
|
-
}
|
|
74
|
-
for line in open(path, encoding="utf-8").read().splitlines()
|
|
75
|
-
for row in [json.loads(line)]
|
|
76
|
-
],
|
|
77
|
-
),
|
|
78
|
-
],
|
|
79
|
-
)
|
|
80
|
-
await db.ready()
|
|
81
|
-
|
|
82
|
-
# Query with SQL
|
|
83
|
-
results = await db.query("SELECT * FROM comments WHERE author = 'alice'")
|
|
84
|
-
# [{"id": "abc", "body": "looks good", "author": "alice"}]
|
|
85
|
-
|
|
86
|
-
asyncio.run(main())
|
|
87
|
-
```
|
|
88
|
-
|
|
89
|
-
## Multiple Tables and Joins
|
|
90
|
-
|
|
91
|
-
```python
|
|
92
|
-
db = DirSQL(
|
|
93
|
-
root,
|
|
94
|
-
tables=[
|
|
95
|
-
Table(
|
|
96
|
-
ddl="CREATE TABLE posts (title TEXT, author_id TEXT)",
|
|
97
|
-
glob="posts/*.json",
|
|
98
|
-
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
99
|
-
),
|
|
100
|
-
Table(
|
|
101
|
-
ddl="CREATE TABLE authors (id TEXT, name TEXT)",
|
|
102
|
-
glob="authors/*.json",
|
|
103
|
-
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
104
|
-
),
|
|
105
|
-
],
|
|
106
|
-
)
|
|
107
|
-
await db.ready()
|
|
108
|
-
|
|
109
|
-
results = await db.query("""
|
|
110
|
-
SELECT posts.title, authors.name
|
|
111
|
-
FROM posts JOIN authors ON posts.author_id = authors.id
|
|
112
|
-
""")
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
## Ignoring Files
|
|
116
|
-
|
|
117
|
-
Pass `ignore` patterns to skip files during scanning and watching:
|
|
118
|
-
|
|
119
|
-
```python
|
|
120
|
-
db = DirSQL(
|
|
121
|
-
root,
|
|
122
|
-
ignore=["**/drafts/**", "**/.git/**"],
|
|
123
|
-
tables=[...],
|
|
124
|
-
)
|
|
125
|
-
```
|
|
126
|
-
|
|
127
|
-
## Watching for Changes
|
|
128
|
-
|
|
129
|
-
`DirSQL` is async by default. The `watch()` method returns an async iterator of row-level change events.
|
|
130
|
-
|
|
131
|
-
```python
|
|
132
|
-
import asyncio
|
|
133
|
-
import json
|
|
134
|
-
from dirsql import DirSQL, Table
|
|
135
|
-
|
|
136
|
-
async def main():
|
|
137
|
-
db = DirSQL(
|
|
138
|
-
"/path/to/data",
|
|
139
|
-
tables=[
|
|
140
|
-
Table(
|
|
141
|
-
ddl="CREATE TABLE items (name TEXT)",
|
|
142
|
-
glob="**/*.json",
|
|
143
|
-
extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
|
|
144
|
-
),
|
|
145
|
-
],
|
|
146
|
-
)
|
|
147
|
-
await db.ready()
|
|
148
|
-
|
|
149
|
-
# Query
|
|
150
|
-
results = await db.query("SELECT * FROM items")
|
|
151
|
-
|
|
152
|
-
# Watch for file changes (insert/update/delete/error events)
|
|
153
|
-
async for event in db.watch():
|
|
154
|
-
print(f"{event.action} on {event.table}: {event.row}")
|
|
155
|
-
if event.action == "error":
|
|
156
|
-
print(f" error: {event.error}")
|
|
157
|
-
|
|
158
|
-
asyncio.run(main())
|
|
159
|
-
```
|
|
160
|
-
|
|
161
|
-
## API Reference
|
|
162
|
-
|
|
163
|
-
### `Table(*, ddl, glob, extract)`
|
|
164
|
-
|
|
165
|
-
Defines how files map to a SQL table.
|
|
166
|
-
|
|
167
|
-
- **`ddl`** (`str`): A `CREATE TABLE` statement defining the schema.
|
|
168
|
-
- **`glob`** (`str`): A glob pattern matched against file paths relative to root.
|
|
169
|
-
- **`extract`** (`Callable[[str], list[dict]]`): A function receiving the matched file's absolute filesystem path and returning a list of row dicts. dirsql does not read file contents; a callback that needs the file body reads it itself (e.g. `open(path, encoding="utf-8").read()`). Each dict's keys must match the DDL column names.
|
|
170
|
-
|
|
171
|
-
### `DirSQL(root=None, *, tables=None, ignore=None, config=None)`
|
|
172
|
-
|
|
173
|
-
Creates an in-memory SQLite database indexed from the directory at `root`. The constructor is sync and returns immediately; scanning runs in a background thread.
|
|
174
|
-
|
|
175
|
-
At least one of `root` or `config` must be supplied. When both `root` and `config` are passed (or `config` declares `[dirsql].root`), the explicit `root` wins and a warning is emitted on stderr.
|
|
176
|
-
|
|
177
|
-
- **`root`** (`str | None`): Path to the directory to index. Optional when `config` supplies one.
|
|
178
|
-
- **`tables`** (`list[Table] | None`): Programmatic table definitions. Appended to any tables in the config file.
|
|
179
|
-
- **`ignore`** (`list[str] | None`): Glob patterns for paths to skip. Appended to any `[dirsql].ignore` patterns in the config file.
|
|
180
|
-
- **`config`** (`str | None`): Optional path to a `.dirsql.toml` file. Its `[[table]]` entries, `[dirsql].ignore`, and optional `[dirsql].root` are merged into the constructor's inputs.
|
|
181
|
-
|
|
182
|
-
#### `await DirSQL.ready()`
|
|
183
|
-
|
|
184
|
-
Wait for the initial scan to complete. Idempotent -- safe to call multiple times. Raises any exception that occurred during init.
|
|
185
|
-
|
|
186
|
-
#### `await DirSQL.query(sql) -> list[dict]`
|
|
187
|
-
|
|
188
|
-
Execute a SQL query. Returns a list of dicts keyed by column name. Internal tracking columns (`_dirsql_*`) are excluded from results.
|
|
189
|
-
|
|
190
|
-
#### `DirSQL.watch() -> AsyncIterator[RowEvent]`
|
|
191
|
-
|
|
192
|
-
Returns an async iterator that yields `RowEvent` objects as files change on disk. Starts the filesystem watcher on first iteration.
|
|
193
|
-
|
|
194
|
-
### `RowEvent`
|
|
195
|
-
|
|
196
|
-
Emitted by `watch()` when a file change produces row-level diffs.
|
|
197
|
-
|
|
198
|
-
- **`table`** (`str`): The affected table name.
|
|
199
|
-
- **`action`** (`str`): One of `"insert"`, `"update"`, `"delete"`, `"error"`.
|
|
200
|
-
- **`row`** (`dict | None`): The new row (for insert/update) or deleted row (for delete).
|
|
201
|
-
- **`old_row`** (`dict | None`): The previous row (for update only).
|
|
202
|
-
- **`error`** (`str | None`): Error message (for error events).
|
|
203
|
-
- **`file_path`** (`str | None`): The relative file path that triggered the event.
|
|
204
|
-
|
|
205
|
-
## License
|
|
206
|
-
|
|
207
|
-
MIT
|
|
208
|
-
|