chunkhound-index-compactor 0.2.0__tar.gz

chunkhound_index_compactor-0.2.0/.gitattributes

@@ -0,0 +1 @@
+tests/fixtures/shopware-cli-chunks.duckdb filter=lfs diff=lfs merge=lfs -text

chunkhound_index_compactor-0.2.0/.gitignore

@@ -0,0 +1,53 @@
+# Python bytecode
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Distribution / packaging
+*.egg-info/
+*.egg
+build/
+dist/
+.eggs/
+
+# Virtual environments
+.venv/
+venv/
+
+# Node (prettier toolchain)
+node_modules/
+
+# Tooling caches
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+
+# Coverage
+.coverage
+.coverage.*
+htmlcov/
+coverage.xml
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Compactor-produced artifacts (any location)
+*.compacted.duckdb
+*.bak
+
+# Superpowers plans and specs stay untracked
+docs/superpowers/plans/
+docs/superpowers/specs/
+
+# .claude is excluded except for the shared skills and project settings
+.claude/*
+!.claude/hook-contexts/
+!.claude/skills/
+!.claude/settings.json

chunkhound_index_compactor-0.2.0/.pre-commit-config.yaml

@@ -0,0 +1,38 @@
+# Mirrors the local checks documented in CONTRIBUTING.md §Local checks.
+# Install once per clone: `pre-commit install`.
+# Run against the whole tree on demand: `pre-commit run --all-files`.
+
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.15.13
+    hooks:
+      - id: ruff
+        # Default invocation is `ruff check` without --fix.
+        # Auto-fix is off by project policy; run `uv run ruff check --fix` manually.
+
+  - repo: https://github.com/crate-ci/typos
+    rev: v1.46.2
+    hooks:
+      - id: typos
+
+  - repo: https://github.com/rbubley/mirrors-prettier
+    rev: v3.8.3
+    hooks:
+      - id: prettier
+        args: [--check]
+
+  - repo: local
+    hooks:
+      - id: mypy
+        name: mypy
+        entry: uv run mypy src/
+        language: system
+        files: ^(src/.*\.py|pyproject\.toml)$
+        pass_filenames: false
+
+      - id: pytest
+        name: pytest
+        entry: uv run pytest
+        language: system
+        always_run: true
+        pass_filenames: false

chunkhound_index_compactor-0.2.0/.prettierignore

@@ -0,0 +1,17 @@
+node_modules/
+dist/
+.venv/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+
+src/
+tests/
+
+# Markdown is hand-formatted. Prettier mangles identifiers containing
+# underscores (treats them as emphasis) and pads tables past printWidth.
+**/*.md
+
+uv.lock
+package-lock.json
+LICENSE

chunkhound_index_compactor-0.2.0/.prettierrc.json

@@ -0,0 +1,7 @@
+{
+  "printWidth": 100,
+  "proseWrap": "preserve",
+  "tabWidth": 2,
+  "trailingComma": "all",
+  "endOfLine": "lf"
+}

chunkhound_index_compactor-0.2.0/.typos.toml

@@ -0,0 +1,13 @@
+[files]
+extend-exclude = [
+  "tests/fixtures/",
+  "dist/",
+  "node_modules/",
+  "uv.lock",
+  "package-lock.json",
+  "CHANGELOG.md",
+]
+
+[default.extend-words]
+# "unparseable" is a valid English variant; we use it in test names.
+unparseable = "unparseable"

chunkhound_index_compactor-0.2.0/AGENTS.md

@@ -0,0 +1,80 @@
+# AGENTS.md
+
+## Layout
+
+```
+chunkhound-index-compactor/
+├── pyproject.toml
+├── package.json                  # prettier dev dep (Node)
+├── .prettierrc.json
+├── .prettierignore
+├── .typos.toml
+├── .github/workflows/            # ci.yml, rolling.yml, release.yml
+├── README.md
+├── AGENTS.md
+├── CLAUDE.md                     # @AGENTS.md
+├── CONTRIBUTING.md               # dev tooling, CI, release process
+├── CHANGELOG.md
+├── LICENSE
+├── docs/
+│   ├── architecture.md           # pipeline, RAM asymmetry, recipe table, vss bundling, ChunkHound compat, refused inputs
+│   ├── benchmarks.md             # empirical baseline (1.25 TB ChunkHound index + fixture cross-check)
+│   └── out-of-scope.md           # refused shapes, dropped metadata, latent edges, rejected approaches + fix shapes
+├── src/chunkhound_index_compactor/
+│   ├── __init__.py               # public API re-exports
+│   ├── __main__.py               # python -m entry
+│   ├── cli.py                    # Typer app
+│   └── core.py                   # compaction logic
+└── tests/
+    ├── conftest.py               # fixtures: populated_db, bloated_db, hnsw_db, cosine_hnsw_db, shopware_cli_index
+    ├── fixtures/                 # committed real-world DB artifacts (provenance in conftest.py)
+    ├── test_core.py
+    ├── test_cli.py
+    ├── test_extensions.py
+    ├── test_rebuild.py
+    └── test_human_size.py
+```
+
+## Module → symbols
+
+| Module | Public | Private |
+|---|---|---|
+| `core.py` | `compact_database`, `restore_indexes`, `replace_with_compacted`, `human_size`, `CompactionResult`, `RestoreResult` | `_topological_order`, `_referenced_tables`, `_reject_unsupported_objects`, `_capture_hnsw_recipes`, `_write_recipe_table`, `_load_bundled_extension`, `_bundled_extension_path`, `_escape_sql_literal`, `_quote_identifier`, `RECIPE_TABLE` constant, regexes `_HNSW_RE`, `_HNSW_COLUMN_RE`, `_FK_REFERENCES_RE`, `_GENERATED_COLUMN_RE`, `_BARE_IDENTIFIER_RE` |
+| `cli.py` | `app` (Typer), `compact`, `restore` commands; `DefaultCommandGroup` routes bare args to `compact` | (none) |
+| `__main__.py` | `app()` invocation | (none) |
+| `__init__.py` | re-exports from `core` | (none) |
+
+## When to modify
+
+| Task | File / symbol |
+|---|---|
+| Rebuild SQL sequence | `core.py` → `compact_database()` |
+| FK ordering | `core.py` → `_topological_order()` / `_referenced_tables()` |
+| Front-gate refusal of unsupported source shapes | `core.py` → `_reject_unsupported_objects()` (schemas, views, user-defined types, generated columns, self-ref FKs) and `_capture_hnsw_recipes()` (expression HNSW columns) |
+| Cross-filesystem replace fallback | `core.py` → `replace_with_compacted()` (`shutil.move` on EXDEV) |
+| DuckDB spill location | `core.py` → `compact_database()` (architecture.md §Compaction pipeline) |
+| HNSW metric recovery / recipe table schema | `core.py` → `_capture_hnsw_recipes()` / `_write_recipe_table()` / `RECIPE_TABLE` |
+| Index restore | `core.py` → `restore_indexes()` |
+| Atomic replace / backup suffix | `core.py` → `replace_with_compacted()` |
+| CLI args / commands / output strings | `cli.py` (`DefaultCommandGroup`, `compact`, `restore`) |
+| Byte formatting | `core.py` → `human_size()` |
+| New public export | `core.py` + `__init__.py` `__all__` |
+| Pipeline narrative, design rationale, refused-input reasoning | `docs/architecture.md` (not here) |
+| Empirical baseline / scale numbers | `docs/benchmarks.md` (not here) |
+| Refused shapes, dropped metadata, latent edges, rejected approaches, and the fix shape per item | `docs/out-of-scope.md` (not here) |
+
+## Invariants enforced by code
+
+- HNSW metric must survive rebuild. Catalog DDL strips `WITH (...)`, so the metric is read from `pragma_hnsw_index_info()` in `_capture_hnsw_recipes`. (architecture.md §ChunkHound compatibility)
+- SQL DDL is built by string interpolation (no parameter binding); escape literals via `_escape_sql_literal`, wrap table and index names via `_quote_identifier`. (architecture.md §Compaction pipeline)
+- Public-API exceptions (`ValueError`, `FileNotFoundError`, `FileExistsError`, `RuntimeError`, `OSError`) enumerated at README §Library Usage; refused inputs reasoned at architecture.md §Not supported (and why).
+- Front-gate refusals run before `ATTACH dst`; on any failure after `ATTACH dst`, the partial target and its `.wal` are unlinked. (architecture.md §Compaction pipeline)
+- Reading the source never loads its HNSW into RAM; building the destination HNSW dominates peak RAM. `--skip-hnsw` is the small-RAM unlock; `restore` is a separate-machine step. (architecture.md §RAM cost asymmetry)
+
+## Build / verify
+
+- Setup, local-check commands, tooling configs, CI workflow details, and release process at `CONTRIBUTING.md` §Setup, §Local checks, §CI workflows, §Release process.
+
+## Runtime deps
+
+- Authoritative constraints at `pyproject.toml`. Load-bearing context: `duckdb` range matches `chunkhound` to stay file-format-compatible; `duckdb-extension-vss>=1.5.2` pins `duckdb==1.5.2` transitively. Python `>=3.10,<3.14`.

chunkhound_index_compactor-0.2.0/CHANGELOG.md

@@ -0,0 +1,54 @@
+# Changelog
+
+## [0.2.0] - 2026-05-21
+
+### Fail-hard
+
+- `compact_database` now refuses sources with user-defined types, generated columns, self-referential foreign keys, or HNSW indexes on non-bare-column expressions, raising `ValueError`. See architecture.md §Not supported (and why).
+- `replace_with_compacted` documents `OSError` as a raised exception (the move from `compacted` to `source` failing even via the `shutil.move` fallback).
+- `compact_database` and `restore_indexes` document `RuntimeError` for the case where the bundled `vss` extension binary cannot be located.
+
+### Fixed
+
+- The CLI surfaces `RuntimeError` from missing bundled `vss` and `OSError` from `replace_with_compacted` as clean `error:` lines instead of unhandled stack traces.
+- The `--skip-hnsw` note now prints after the `--replace` step and points at the path the artifact lives at after the whole CLI run (`result.source` with `--replace`, `result.target` without). Previously it pointed at the `.compacted` path that `--replace` renamed away.
+- `replace_with_compacted` falls back to `shutil.move` when the second rename fails with a cross-device error (EXDEV), so a cross-filesystem `--replace` (user-supplied target on another mount) now completes instead of crashing.
+- The failure-cleanup block in `compact_database` now also unlinks `<target>.wal` if a CHECKPOINT step left one behind.
+- The compact CLI surfaces spill-directory creation failures (`OSError`) as a clean `error:` line, and `compact_database` removes the spill directory when DuckDB never spills into it.
+- DDL identifier interpolation routes through a new `_quote_identifier()` helper that doubles any embedded `"`. The previous sites double-quoted identifiers without escaping.
+
+### Added
+
+- `--replace` with `--skip-hnsw` now prints a `warning:` line. The in-place file has no vector index until `restore` runs against it; the regression needs to be loud.
+
+### Changed
+
+- `compact_database` co-locates DuckDB spill with the target's filesystem (`temp_directory` beside the target). See architecture.md §Compaction pipeline.
+- README narrowed the "fully generic / works on any single-schema DuckDB file" claim to ChunkHound-shaped inputs; other shapes are refused at the front gate rather than rebuilt with silent loss. The published package description (`pyproject.toml`) dropped its "(ChunkHound index or otherwise)" parenthetical to match.
+- `docs/architecture.md` corrects the imprecise "drops HNSW on each write batch" claim to ChunkHound's actual `insert_embeddings_batch` 50-row threshold, removes the wrong COMMENT ON claim (see out-of-scope.md §Table and column comments for the actual drop behavior), and cites [duckdb/duckdb#16785](https://github.com/duckdb/duckdb/issues/16785) for the `COPY FROM DATABASE` FK race.
+- `docs/architecture.md` §Not supported collapsed to one-line pointers at `out-of-scope.md`, so per-case refusal reasoning lives on a single surface.
+- `docs/out-of-scope.md` promoted to the single per-topic catalog covering refused source shapes, silently-dropped metadata (HNSW tuning beyond `metric`, table and column comments), latent code edges (quoted referenced tables in `_FK_REFERENCES_RE`), and rejected alternative approaches. Each section owns both the why-not and the fix shape.
+
+## [0.1.1] - 2026-05-20
+
+### Packaging
+- README, `[project.urls]` (Homepage / Repository / Issues / Changelog), `authors`, `keywords`, and trove classifiers now ship in the published package metadata. The PyPI project page renders the README and links back to the GitHub repository; 0.1.0 shipped without any of these.
+
+## [0.1.0] - 2026-05-20
+
+### Added
+- Initial release of `chunkhound-index-compactor`.
+- `chunkhound-index-compactor` CLI (Typer-based) with a `compact` default command and a `restore` subcommand, routed via `DefaultCommandGroup` so `chunkhound-index-compactor SOURCE [TARGET]` still works.
+- `compact_database(source, target, *, skip_hnsw=False)`: rebuild a DuckDB database into a fresh file via a foreign-key-ordered streaming rebuild. Captures the source schema, recreates sequences/tables/indexes in a freshly-allocated file, computes a foreign-key-topological table order, and inserts one table at a time parent-before-child. This sidesteps the FK race that breaks `ATTACH` + `COPY FROM DATABASE` on large FK-bearing databases (e.g. ChunkHound indexes at scale) while still dropping orphaned blocks.
+- HNSW indexes are recreated with the metric recovered from `pragma_hnsw_index_info()`. The catalog DDL strips the `WITH (...)` clause, so a verbatim rebuild would silently reset a `cosine` index to the `l2sq` default and leave it dead (queries fall back to brute force).
+- `--skip-hnsw` flag / `skip_hnsw=True` parameter: rebuild without vector indexes (RAM-flat, smallest output) and record what was stripped in a `_compactor_hnsw_recipe` table inside the output.
+- `restore` CLI command / `restore_indexes()` function: rebuild the stripped HNSW indexes in place from the recipe table, idempotently, on a RAM-capable machine.
+- `replace_with_compacted()`: atomic swap with `.bak` backup.
+- `human_size()`: binary-prefix byte formatting.
+- `CompactionResult` and `RestoreResult` dataclasses.
+- `--replace` flag for in-place compaction with backup.
+- Bundled `vss.duckdb_extension` binary from `duckdb-extension-vss` is `LOAD`ed directly from disk when an HNSW index is present, so compaction of ChunkHound and other vector-search DuckDBs works offline out of the box.
+
+### Fail-hard
+- Sources with non-`main` schemas, views, or foreign-key cycles raise `ValueError` rather than silently dropping objects.
+- On any failure after the target file is created, the partial target is unlinked. A half-written multi-GB file is worse than nothing.

chunkhound_index_compactor-0.2.0/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

chunkhound_index_compactor-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,85 @@
+# Contributing
+
+## Setup
+
+```bash
+uv sync --extra dev      # Python toolchain (mypy, pytest, ruff, typos, pre-commit)
+npm install              # Node toolchain (prettier; one-time per clone)
+uv run pre-commit install  # wires the git pre-commit hook (one-time per clone)
+```
+
+Python 3.10 through 3.13 supported. macOS and Linux are tested.
+
+The pre-commit hook runs ruff, typos, prettier, mypy, and pytest on every commit. Bypass with `git commit --no-verify` when you need to ship a WIP commit; CI still runs the same checks.
+
+## Local checks
+
+```bash
+uv run pytest
+uv run ruff check src/ tests/
+uv run ruff format --check src/ tests/
+uv run mypy src/
+uv run typos
+npx prettier --check .
+```
+
+Apply prettier fixes with `npm run format:fix`. CI runs the same commands.
+
+### Tooling notes
+
+- **Ruff**: `E W F I B C4 UP ARG SIM PTH`, line length 100, `E501` ignored.
+- **MyPy**: strict; `tests.*` relaxed; `duckdb`, `duckdb_extension_vss` missing-imports ignored.
+- **Pytest**: discovers `tests/`. Fixtures bundle real ChunkHound DB samples (see `tests/conftest.py`).
+- **Typos**: config at `.typos.toml`.
+- **Prettier**: config at `.prettierrc.json` and `.prettierignore`. Markdown is excluded because prettier mangles identifiers containing underscores and bloats markdown tables past the 100-column line limit.
+
+## CI workflows
+
+Three workflows under `.github/workflows/`. All third-party actions are SHA-pinned with `# vX.Y.Z` comments so Renovate can update them later.
+
+### ci.yml
+
+Runs on every push to `main` and every PR targeting `main`. Six parallel jobs:
+
+| Job         | What                                                                                  |
+|-------------|---------------------------------------------------------------------------------------|
+| `lint`      | `ruff check` and `ruff format --check`                                                |
+| `typecheck` | `mypy src/`                                                                           |
+| `test`      | `pytest` with coverage on a Python 3.10 / 3.11 / 3.12 / 3.13 matrix (`ubuntu-latest`) |
+| `typos`     | `typos` against the repo                                                              |
+| `prettier`  | `prettier --check .`                                                                  |
+| `build`     | `uv build`; uploads wheel + sdist as a `dist` artifact (14-day retention)             |
+
+Coverage is reported but not gated. The `build` job runs independently of the others, so PR reviewers can download a wheel even when other jobs fail.
+
+### rolling.yml
+
+After `ci.yml` succeeds on `main`, deletes the existing `rolling` tag plus release and recreates them at the new HEAD SHA. Marked prerelease and not-latest so it never shadows a tagged release.
+
+Install the current main-branch build from the rolling asset:
+
+```bash
+uv tool install https://github.com/it-bens/chunkhound-index-compactor/releases/download/rolling/chunkhound_index_compactor-<version>-py3-none-any.whl
+```
+
+### release.yml
+
+Triggers on a `v*.*.*` tag push or manual `workflow_dispatch` against a tag ref.
+
+1. Verifies the ref is a tag.
+2. Verifies the tag (minus the `v` prefix) matches `pyproject.toml`'s `version`.
+3. Builds wheel + sdist.
+4. Publishes to PyPI via Trusted Publisher (OIDC; no PyPI token in the repo).
+5. Creates a GitHub Release with the wheel + sdist attached. Release notes are not auto-generated; edit them on GitHub afterward.
+
+## Release process
+
+To cut a release:
+
+1. Bump `version` in `pyproject.toml` (for example, `0.1.0` to `0.2.0`) and update `CHANGELOG.md`.
+2. Commit, push, wait for CI to pass on `main`.
+3. Tag and push: `git tag v0.2.0 && git push origin v0.2.0`.
+4. The release workflow publishes to PyPI and creates a GitHub Release.
+5. Open the Release on GitHub and write the release notes.
+
+To re-trigger from a tag manually (for example, after a transient PyPI failure): GitHub Actions, then Release, then Run workflow, then pick the tag from the ref dropdown.

chunkhound_index_compactor-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Martin Bens
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

chunkhound_index_compactor-0.2.0/PKG-INFO

@@ -0,0 +1,147 @@
+Metadata-Version: 2.4
+Name: chunkhound-index-compactor
+Version: 0.2.0
+Summary: Compact a bloated ChunkHound DuckDB index by rebuilding it into a fresh file
+Project-URL: Homepage, https://github.com/it-bens/chunkhound-index-compactor
+Project-URL: Repository, https://github.com/it-bens/chunkhound-index-compactor
+Project-URL: Issues, https://github.com/it-bens/chunkhound-index-compactor/issues
+Project-URL: Changelog, https://github.com/it-bens/chunkhound-index-compactor/blob/main/CHANGELOG.md
+Author-email: Martin Bens <martin.bens@it-bens.de>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: chunkhound,compact,database,duckdb,hnsw,vector-index
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: <3.14,>=3.10
+Requires-Dist: duckdb-extension-vss>=1.5.2
+Requires-Dist: duckdb<1.5.3.dev0,>=1.4.0
+Requires-Dist: typer>=0.25
+Provides-Extra: dev
+Requires-Dist: mypy>=2.1; extra == 'dev'
+Requires-Dist: pre-commit>=4.5; extra == 'dev'
+Requires-Dist: pytest-cov>=7.0; extra == 'dev'
+Requires-Dist: pytest>=9.0; extra == 'dev'
+Requires-Dist: ruff>=0.15; extra == 'dev'
+Requires-Dist: typos>=1.46; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# ChunkHound Index Compactor
+
+Compact a [DuckDB](https://duckdb.org) database by rebuilding it into a fresh file. The motivating and supported use case is shrinking a bloated [ChunkHound](https://github.com/chunkhound/chunkhound) index, whose drop-and-recreate HNSW churn (above its 50-row write-batch threshold) leaves large amounts of orphaned-but-counted blocks. The rebuild pipeline is structurally generic and works on other single-schema DuckDB files, but only ChunkHound-shaped inputs are promised: any shape outside that scope is refused at the front gate (see the Not Supported section below) rather than silently dropped or rebuilt with loss.
+
+That bloat comes back as ChunkHound keeps indexing, so compaction is periodic maintenance rather than a one-time cleanup.
+
+## ⚡ Quick Start
+
+```bash
+uvx chunkhound-index-compactor path/to/db.duckdb
+# writes path/to/db.duckdb.compacted
+
+uvx chunkhound-index-compactor path/to/db.duckdb --replace
+# swaps in the compacted copy and keeps the original at path/to/db.duckdb.bak
+
+uvx chunkhound-index-compactor path/to/db.duckdb --skip-hnsw
+# skips rebuilding vector indexes (RAM-flat, smallest output); restore them later
+uvx chunkhound-index-compactor restore path/to/db.duckdb.compacted
+```
+
+The source is opened read-only, but an active writer holds the file lock. Close any process writing to the database before running.
+
+## 🖥️ CLI Usage
+
+```
+$ chunkhound-index-compactor --help
+Usage: chunkhound-index-compactor [OPTIONS] COMMAND [ARGS]...
+
+Commands:
+  compact  Compact a DuckDB database by rebuilding it into a fresh file. (default)
+  restore  Rebuild HNSW vector indexes in a --skip-hnsw artifact, in place.
+```
+
+A bare invocation routes to `compact`, so `chunkhound-index-compactor SOURCE` still works:
+
+```
+chunkhound-index-compactor SOURCE [TARGET] [--replace] [--skip-hnsw]
+chunkhound-index-compactor restore DATABASE
+```
+
+| Argument / Option | Meaning                                                                           |
+|-------------------|-----------------------------------------------------------------------------------|
+| `SOURCE`          | Path to the existing DuckDB file (required)                                       |
+| `TARGET`          | Path for the compacted output [default: `<source>.compacted`]                     |
+| `--replace`       | After success, replace source with the compacted file (original → `<source>.bak`) |
+| `--skip-hnsw`     | Do not rebuild vector indexes; write a recipe table for later `restore`           |
+
+With `--skip-hnsw`, the output has no vector index and falls back to a brute-force scan (correct, just unaccelerated) until you run `restore`. Rebuilding the HNSW is the memory-dominant step, so `--skip-hnsw` lets you compact on a small machine and `restore` on a RAM-capable one. See [docs/benchmarks.md](docs/benchmarks.md) for peak-RAM numbers and [docs/architecture.md §RAM cost asymmetry](docs/architecture.md#ram-cost-asymmetry) for why.
+
+## 🐍 Library Usage
+
+```python
+from pathlib import Path
+from chunkhound_index_compactor import compact_database, restore_indexes, replace_with_compacted
+
+result = compact_database(Path("big.duckdb"), Path("small.duckdb"))
+print(f"{result.source_size} -> {result.target_size} ({result.delta_pct:+.1f}%)")
+
+# Small-RAM path: skip the vector index, restore it later on a bigger machine.
+compact_database(Path("big.duckdb"), Path("small.duckdb"), skip_hnsw=True)
+restored = restore_indexes(Path("small.duckdb"))
+print(f"restored: {restored.restored}")
+
+# Optional: swap in place with .bak backup
+backup = replace_with_compacted(result.source, result.target)
+```
+
+`compact_database()` raises:
+- `ValueError`: `target` resolves to the same path as `source`, the FK graph has a cycle, or the source has a shape refused at the front gate (see the Not Supported section below).
+- `FileNotFoundError`: `source` does not exist.
+- `FileExistsError`: `target` already exists.
+- `RuntimeError`: the bundled `vss` extension binary cannot be located (only reachable if the source contains an HNSW index).
+
+`restore_indexes()` raises:
+- `FileNotFoundError`: `database` does not exist.
+- `ValueError`: `database` has no `_compactor_hnsw_recipe` table (not a `--skip-hnsw` artifact).
+- `RuntimeError`: the bundled `vss` extension binary cannot be located.
+
+`replace_with_compacted()` raises:
+- `FileNotFoundError`: `source` or `compacted` is missing.
+- `FileExistsError`: `<source>.bak` already exists (it refuses to overwrite an existing backup).
+- `OSError`: the move from `compacted` to `source` fails even via the cross-filesystem fallback (`shutil.move`).
+
+## 🚫 Not Supported
+
+The tool fails hard rather than silently dropping anything it cannot reproduce.
+
+- Non-`main` schemas and views (raise `ValueError`).
+- User-defined types, generated columns, self-referential foreign keys, and HNSW indexes on non-bare-column expressions (raise `ValueError`).
+- Foreign-key cycles among tables (raise `ValueError`).
+- HNSW tuning parameters other than `metric` (`M`, `M0`, `ef_construction`, `ef_search`); they are not recoverable from a built index and are rebuilt at the `vss` defaults.
+- Table and column comments are not carried across the rebuild.
+
+See [docs/architecture.md](docs/architecture.md#not-supported-and-why) for the reasoning, and [docs/out-of-scope.md](docs/out-of-scope.md) for approaches considered and not pursued.
+
+## 🏗️ Development
+
+Setup, local checks, CI, and release process: [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## ⚖️ License
+
+MIT
+
+---
+
+> [!NOTE]
+> Yes, an AI wrote this README. And the code, the docs, the tests, and
+> the `.claude/skills` it now uses to write the next round. Yes, a human
+> told it to keep the emojis. The human has ADHD, which, as it turns
+> out, means his brain was already doing attention re-routing and
+> context-window thrashing before LLMs made it cool. They call him ...
+> LLMartin. The emojis are a feature.

chunkhound_index_compactor-0.2.0/README.md

@@ -0,0 +1,112 @@
+# ChunkHound Index Compactor
+
+Compact a [DuckDB](https://duckdb.org) database by rebuilding it into a fresh file. The motivating and supported use case is shrinking a bloated [ChunkHound](https://github.com/chunkhound/chunkhound) index, whose drop-and-recreate HNSW churn (above its 50-row write-batch threshold) leaves large amounts of orphaned-but-counted blocks. The rebuild pipeline is structurally generic and works on other single-schema DuckDB files, but only ChunkHound-shaped inputs are promised: any shape outside that scope is refused at the front gate (see the Not Supported section below) rather than silently dropped or rebuilt with loss.
+
+That bloat comes back as ChunkHound keeps indexing, so compaction is periodic maintenance rather than a one-time cleanup.
+
+## ⚡ Quick Start
+
+```bash
+uvx chunkhound-index-compactor path/to/db.duckdb
+# writes path/to/db.duckdb.compacted
+
+uvx chunkhound-index-compactor path/to/db.duckdb --replace
+# swaps in the compacted copy and keeps the original at path/to/db.duckdb.bak
+
+uvx chunkhound-index-compactor path/to/db.duckdb --skip-hnsw
+# skips rebuilding vector indexes (RAM-flat, smallest output); restore them later
+uvx chunkhound-index-compactor restore path/to/db.duckdb.compacted
+```
+
+The source is opened read-only, but an active writer holds the file lock. Close any process writing to the database before running.
+
+## 🖥️ CLI Usage
+
+```
+$ chunkhound-index-compactor --help
+Usage: chunkhound-index-compactor [OPTIONS] COMMAND [ARGS]...
+
+Commands:
+  compact  Compact a DuckDB database by rebuilding it into a fresh file. (default)
+  restore  Rebuild HNSW vector indexes in a --skip-hnsw artifact, in place.
+```
+
+A bare invocation routes to `compact`, so `chunkhound-index-compactor SOURCE` still works:
+
+```
+chunkhound-index-compactor SOURCE [TARGET] [--replace] [--skip-hnsw]
+chunkhound-index-compactor restore DATABASE
+```
+
+| Argument / Option | Meaning                                                                           |
+|-------------------|-----------------------------------------------------------------------------------|
+| `SOURCE`          | Path to the existing DuckDB file (required)                                       |
+| `TARGET`          | Path for the compacted output [default: `<source>.compacted`]                     |
+| `--replace`       | After success, replace source with the compacted file (original → `<source>.bak`) |
+| `--skip-hnsw`     | Do not rebuild vector indexes; write a recipe table for later `restore`           |
+
+With `--skip-hnsw`, the output has no vector index and falls back to a brute-force scan (correct, just unaccelerated) until you run `restore`. Rebuilding the HNSW is the memory-dominant step, so `--skip-hnsw` lets you compact on a small machine and `restore` on a RAM-capable one. See [docs/benchmarks.md](docs/benchmarks.md) for peak-RAM numbers and [docs/architecture.md §RAM cost asymmetry](docs/architecture.md#ram-cost-asymmetry) for why.
+
+## 🐍 Library Usage
+
+```python
+from pathlib import Path
+from chunkhound_index_compactor import compact_database, restore_indexes, replace_with_compacted
+
+result = compact_database(Path("big.duckdb"), Path("small.duckdb"))
+print(f"{result.source_size} -> {result.target_size} ({result.delta_pct:+.1f}%)")
+
+# Small-RAM path: skip the vector index, restore it later on a bigger machine.
+compact_database(Path("big.duckdb"), Path("small.duckdb"), skip_hnsw=True)
+restored = restore_indexes(Path("small.duckdb"))
+print(f"restored: {restored.restored}")
+
+# Optional: swap in place with .bak backup
+backup = replace_with_compacted(result.source, result.target)
+```
+
+`compact_database()` raises:
+- `ValueError`: `target` resolves to the same path as `source`, the FK graph has a cycle, or the source has a shape refused at the front gate (see the Not Supported section below).
+- `FileNotFoundError`: `source` does not exist.
+- `FileExistsError`: `target` already exists.
+- `RuntimeError`: the bundled `vss` extension binary cannot be located (only reachable if the source contains an HNSW index).
+
+`restore_indexes()` raises:
+- `FileNotFoundError`: `database` does not exist.
+- `ValueError`: `database` has no `_compactor_hnsw_recipe` table (not a `--skip-hnsw` artifact).
+- `RuntimeError`: the bundled `vss` extension binary cannot be located.
+
+`replace_with_compacted()` raises:
+- `FileNotFoundError`: `source` or `compacted` is missing.
+- `FileExistsError`: `<source>.bak` already exists (it refuses to overwrite an existing backup).
+- `OSError`: the move from `compacted` to `source` fails even via the cross-filesystem fallback (`shutil.move`).
+
+## 🚫 Not Supported
+
+The tool fails hard rather than silently dropping anything it cannot reproduce.
+
+- Non-`main` schemas and views (raise `ValueError`).
+- User-defined types, generated columns, self-referential foreign keys, and HNSW indexes on non-bare-column expressions (raise `ValueError`).
+- Foreign-key cycles among tables (raise `ValueError`).
+- HNSW tuning parameters other than `metric` (`M`, `M0`, `ef_construction`, `ef_search`); they are not recoverable from a built index and are rebuilt at the `vss` defaults.
+- Table and column comments are not carried across the rebuild.
+
+See [docs/architecture.md](docs/architecture.md#not-supported-and-why) for the reasoning, and [docs/out-of-scope.md](docs/out-of-scope.md) for approaches considered and not pursued.
+
+## 🏗️ Development
+
+Setup, local checks, CI, and release process: [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## ⚖️ License
+
+MIT
+
+---
+
+> [!NOTE]
+> Yes, an AI wrote this README. And the code, the docs, the tests, and
+> the `.claude/skills` it now uses to write the next round. Yes, a human
+> told it to keep the emojis. The human has ADHD, which, as it turns
+> out, means his brain was already doing attention re-routing and
+> context-window thrashing before LLMs made it cool. They call him ...
+> LLMartin. The emojis are a feature.