arkiv 0.1.0__tar.gz

arkiv-0.1.0/.claude/settings.local.json

@@ -0,0 +1,10 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(pip install:*)",
+      "Bash(python -c:*)",
+      "Bash(python -m pytest tests/ -q)",
+      "Bash(python -m pytest:*)"
+    ]
+  }
+}

arkiv-0.1.0/.coveragerc

@@ -0,0 +1,10 @@
+[run]
+source = src/arkiv
+omit = src/arkiv/__pycache__/*
+
+[report]
+exclude_lines =
+    pragma: no cover
+    if __name__ == .__main__
+    if TYPE_CHECKING
+show_missing = true

arkiv-0.1.0/.gitignore

@@ -0,0 +1,20 @@
+.worktrees/
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+*.egg
+.eggs/
+*.so
+.coverage
+htmlcov/
+.mypy_cache/
+.pytest_cache/
+venv/
+.venv/
+*.db
+.idea/
+.vscode/
+*.swp
+*.swo

arkiv-0.1.0/CLAUDE.md

@@ -0,0 +1,143 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What arkiv Is
+
+Universal personal data format with JSONL canonical storage, SQLite query layer, and MCP server.
+
+- **JSONL** is the source of truth (human-readable, portable, mergeable)
+- **SQLite** is the derived query layer (efficient, standard SQL, JSON1 extension)
+- **MCP server** exposes 3 tools: `get_manifest()`, `get_schema()`, `sql_query()`
+
+## What arkiv Is NOT
+
+- Not a database -- it's a format with tooling
+- Not specific to personas -- longshade is one consumer, but arkiv is general-purpose
+- Not a replacement for SQL databases -- it's an interchange format that imports/exports to SQLite
+
+## Development Commands
+
+```bash
+pip install -e ".[dev]"
+pytest                                              # all tests
+pytest tests/test_database.py -v                    # one test file
+pytest tests/test_cli.py::TestCLI::test_export -v   # single test
+pytest --cov=src/arkiv --cov-report=term-missing    # coverage
+black src/ tests/
+flake8 src/ tests/
+mypy src/
+```
+
+CLI is invoked as `python -m arkiv.cli` (or `arkiv` if installed). Tests use `subprocess.run([sys.executable, "-m", "arkiv.cli", ...])`.
+
+## Architecture
+
+### Data Flow
+
+```
+README.md + schema.yaml + *.jsonl   (archive directory)
+        ↓ import_readme()
+    SQLite database                 (3 tables: records, _schema, _metadata)
+        ↓ export()
+README.md + schema.yaml + *.jsonl   (roundtrip-safe)
+```
+
+### Record Model (`record.py`)
+
+All fields optional: `mimetype`, `uri`, `content`, `timestamp`, `metadata`. Unknown fields on input are merged into `metadata`. Defined in `KNOWN_FIELDS`.
+
+### Schema System (`schema.py`)
+
+Two-tier schema: auto-discovered from data and curated from `schema.yaml`.
+
+- `discover_schema()` scans JSONL and produces `Dict[str, SchemaEntry]` with type, count, values/example
+- `merge_schema(auto, curated)` combines them: **live fields** (type, count) from auto, **stable fields** (description, curated values) from curated. Keys in curated but not data preserved with `count=0`
+- `CollectionSchema` bundles `record_count` + `metadata_keys`
+- `load_schema_yaml()` / `save_schema_yaml()` handle YAML I/O
+
+### Database (`database.py`)
+
+SQLite tables:
+- `records` -- (id, collection, mimetype, uri, content, timestamp, metadata JSON)
+- `_schema` -- (collection, key_path, type, count, sample_values, description)
+- `_metadata` -- (key TEXT PRIMARY KEY, value TEXT) — stores README frontmatter + body as KV pairs
+
+Key behaviors:
+- `import_jsonl()` uses **replace semantics** (deletes existing records for same collection before inserting)
+- `import_jsonl()` preserves existing schema descriptions across reimports via `_load_schema_descriptions()`
+- `import_readme()` parses README.md frontmatter, imports each JSONL from `contents` (resolves paths relative to README, so nested `collection/collection.jsonl` paths work), merges curated schema from sibling `schema.yaml`
+- `_save_schema_entries()` is the shared helper for writing to `_schema` table (used by both `import_jsonl` and `merge_curated_schema`)
+- `_store_readme_metadata()` / `_load_readme_metadata()` serialize README frontmatter + body to/from `_metadata` KV table
+- `export()` writes JSONL files + README.md + schema.yaml, preserving stored frontmatter metadata. Supports `nested` (per-collection subdirectories with own README + schema.yaml), `since`/`until` (temporal slicing with two-pass schema recomputation from filtered data), and schema-in-README injection via `render.py` sentinels
+- `get_readme()` is the public accessor for stored README metadata
+- `query()` is read-only (SELECT/WITH prefix check + sqlite3 authorizer)
+
+### README / Archive Identity (`readme.py`)
+
+`Readme` dataclass with `frontmatter: dict` and `body: str`. Frontmatter is `---`-delimited YAML. Known frontmatter keys by convention: `name`, `description`, `datetime`, `generator`, `contents`.
+
+### CLI (`cli.py`)
+
+Subcommands: `import`, `export`, `schema`, `query`, `info`, `detect`, `fix`, `mcp`. Export flags: `--nested` (per-collection subdirectories), `--since`/`--until` (temporal slicing). Import routing: `.md` → `import_readme()`, directory → looks for `README.md`, everything else → `import_jsonl()`. The CLI uses lazy imports (`from .database import Database` inside functions) to keep startup fast.
+
+### Schema Rendering (`render.py`)
+
+`render_schema_summary()` and `render_schema_detail()` produce markdown tables from `CollectionSchema`. `inject_schema_block()` handles sentinel-based injection into README bodies (`<!-- arkiv:schema:begin/end -->`). Used by `export()`.
+
+### Temporal Filtering (`timefilter.py`)
+
+`increment_iso_prefix()` handles ISO 8601 date arithmetic. `build_time_filter()` constructs SQL WHERE clauses for `--since`/`--until`. Used by `export()`.
+
+### MCP Server (`server.py`)
+
+`ArkivServer` wraps a read-only `Database`. `run_mcp_server()` uses FastMCP (requires `pip install arkiv[mcp]`). All metadata is derived from the DB — no external files needed at serve time.
+
+## Archive Format
+
+```
+archive/
+├── README.md           # Self-description (YAML frontmatter + markdown)
+├── schema.yaml         # Structured metadata schema (auto-generated, curatable)
+├── conversations.jsonl
+└── bookmarks.jsonl
+```
+
+With `--nested` export:
+
+```
+archive/
+├── README.md
+├── schema.yaml
+├── conversations/
+│   ├── README.md
+│   ├── schema.yaml
+│   └── conversations.jsonl
+└── bookmarks/
+    ├── README.md
+    ├── schema.yaml
+    └── bookmarks.jsonl
+```
+
+- **README.md** -- YAML frontmatter (`name`, `description`, `datetime`, `generator`, `contents`) + markdown body
+- **schema.yaml** -- per-collection metadata keys with type, count, values, description
+- **Merge-on-import** -- live fields (type, count) recomputed from data; stable fields (description, curated values) preserved
+
+A curated example archive lives at `examples/repos/` and is verified by `test_integration.py::test_example_archive_roundtrip`.
+
+## Key Principles
+
+1. All record fields optional -- permissive input, best-effort processing
+2. JSONL canonical, SQLite derived -- can always regenerate the DB from JSONL
+3. Standards-based -- MIME types, URIs, ISO 8601, SQL
+4. Document-oriented, not relational
+
+## Tech Stack
+
+- Python 3.8+, sqlite3 (stdlib), json (stdlib), pyyaml, MCP Python SDK
+
+## Related Projects
+
+- [longshade](../longshade/) -- Persona packaging convention (consumer of arkiv)
+- [memex](../memex/), [mtk](../mtk/), [btk](../btk/), [ptk](../ptk/), [ebk](../ebk/), [repoindex](../repoindex/), [chartfold](../chartfold/) -- Source toolkits (producers of arkiv JSONL)
+- [longecho](../longecho/) -- longecho compliance validator

arkiv-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Alex Towell
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

arkiv-0.1.0/PKG-INFO

@@ -0,0 +1,92 @@
+Metadata-Version: 2.4
+Name: arkiv
+Version: 0.1.0
+Summary: Universal personal data format. JSONL in, SQL out, MCP to LLMs.
+Project-URL: Homepage, https://github.com/queelius/arkiv
+Project-URL: Repository, https://github.com/queelius/arkiv
+Project-URL: Issues, https://github.com/queelius/arkiv/issues
+Author-email: Alexander Towell <lex@metafunctor.com>
+License: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Classifier: Topic :: System :: Archiving
+Requires-Python: >=3.8
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: black; extra == 'dev'
+Requires-Dist: flake8; extra == 'dev'
+Requires-Dist: mcp[cli]; extra == 'dev'
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Provides-Extra: mcp
+Requires-Dist: mcp[cli]; extra == 'mcp'
+Description-Content-Type: text/markdown
+
+# arkiv
+
+Universal personal data format. JSONL in, SQL out, MCP to LLMs.
+
+## The Format
+
+Every record is a JSON object. All fields optional.
+
+```jsonl
+{"mimetype": "text/plain", "content": "I think the key insight is...", "uri": "https://chatgpt.com/c/abc", "timestamp": "2023-05-14T10:30:00Z", "metadata": {"role": "user", "conversation_id": "abc"}}
+{"mimetype": "audio/wav", "uri": "file://media/podcast.wav", "timestamp": "2024-01-15", "metadata": {"transcript": "Welcome to...", "duration": 45.2}}
+{"mimetype": "image/jpeg", "uri": "file://media/photo.jpg", "metadata": {"caption": "My talk at MIT"}}
+```
+
+## The Stack
+
+```
+JSONL files (canonical, portable, human-readable)
+    ↓ arkiv import
+SQLite database (queryable, efficient, standard SQL)
+    ↓ arkiv mcp
+MCP server (3 tools → any LLM)
+```
+
+## Quick Start
+
+```bash
+pip install arkiv
+
+# Import JSONL to SQLite
+arkiv import conversations.jsonl --db archive.db
+
+# Query
+arkiv query archive.db "SELECT content FROM records WHERE metadata->>'role' = 'user' LIMIT 5"
+
+# Serve to LLMs via MCP
+arkiv mcp archive.db
+```
+
+## MCP Tools
+
+| Tool | Description |
+|------|-------------|
+| `get_manifest()` | What collections exist, their descriptions and schemas |
+| `get_schema(collection?)` | What metadata keys can be queried |
+| `sql_query(query)` | Run read-only SQL |
+
+## Why
+
+- Your data lives in silos (ChatGPT, email, bookmarks, photos, voice memos)
+- Source toolkits (memex, mtk, btk, ptk, ebk) export it as JSONL
+- arkiv gives you one format, one database, one query interface
+- Any LLM can query it via MCP
+- JSONL is human-readable and durable. SQLite is the most deployed database in history.
+
+## Spec
+
+See [SPEC.md](SPEC.md) for the full technical specification.

arkiv-0.1.0/README.md

@@ -0,0 +1,58 @@
+# arkiv
+
+Universal personal data format. JSONL in, SQL out, MCP to LLMs.
+
+## The Format
+
+Every record is a JSON object. All fields optional.
+
+```jsonl
+{"mimetype": "text/plain", "content": "I think the key insight is...", "uri": "https://chatgpt.com/c/abc", "timestamp": "2023-05-14T10:30:00Z", "metadata": {"role": "user", "conversation_id": "abc"}}
+{"mimetype": "audio/wav", "uri": "file://media/podcast.wav", "timestamp": "2024-01-15", "metadata": {"transcript": "Welcome to...", "duration": 45.2}}
+{"mimetype": "image/jpeg", "uri": "file://media/photo.jpg", "metadata": {"caption": "My talk at MIT"}}
+```
+
+## The Stack
+
+```
+JSONL files (canonical, portable, human-readable)
+    ↓ arkiv import
+SQLite database (queryable, efficient, standard SQL)
+    ↓ arkiv mcp
+MCP server (3 tools → any LLM)
+```
+
+## Quick Start
+
+```bash
+pip install arkiv
+
+# Import JSONL to SQLite
+arkiv import conversations.jsonl --db archive.db
+
+# Query
+arkiv query archive.db "SELECT content FROM records WHERE metadata->>'role' = 'user' LIMIT 5"
+
+# Serve to LLMs via MCP
+arkiv mcp archive.db
+```
+
+## MCP Tools
+
+| Tool | Description |
+|------|-------------|
+| `get_manifest()` | What collections exist, their descriptions and schemas |
+| `get_schema(collection?)` | What metadata keys can be queried |
+| `sql_query(query)` | Run read-only SQL |
+
+## Why
+
+- Your data lives in silos (ChatGPT, email, bookmarks, photos, voice memos)
+- Source toolkits (memex, mtk, btk, ptk, ebk) export it as JSONL
+- arkiv gives you one format, one database, one query interface
+- Any LLM can query it via MCP
+- JSONL is human-readable and durable. SQLite is the most deployed database in history.
+
+## Spec
+
+See [SPEC.md](SPEC.md) for the full technical specification.