refindery 0.1.0__tar.gz

refindery-0.1.0/PKG-INFO

@@ -0,0 +1,233 @@
+Metadata-Version: 2.4
+Name: refindery
+Version: 0.1.0
+Summary: A local, single-machine retrieval engine over the web pages you read.
+Keywords: retrieval,search,rag,embeddings,mcp,browser-history
+Author: Harold Martin
+Author-email: Harold Martin <harold.martin@gmail.com>
+License-Expression: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Framework :: FastAPI
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Dist: aiosqlite>=0.22.1
+Requires-Dist: catsu>=0.1.8
+Requires-Dist: chonkie>=1.7.0
+Requires-Dist: duckdb>=1.5.4
+Requires-Dist: fastapi>=0.139.0
+Requires-Dist: fastapi-mcp>=0.4.0
+Requires-Dist: fastembed>=0.8.0
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: huey>=3.1.1
+Requires-Dist: inflect>=7.5.0
+Requires-Dist: lancedb>=0.34.0
+Requires-Dist: model2vec>=0.8.2
+Requires-Dist: numpy>=2.2
+Requires-Dist: opentelemetry-api>=1.43.0
+Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.43.0
+Requires-Dist: opentelemetry-instrumentation-fastapi>=0.64b0
+Requires-Dist: opentelemetry-sdk>=1.43.0
+Requires-Dist: prometheus-client>=0.25.0
+Requires-Dist: pyarrow>=24.0.0
+Requires-Dist: pydantic>=2.13.4
+Requires-Dist: pydantic-settings>=2.14.2
+Requires-Dist: pypdf>=6.14.2
+Requires-Dist: qdrant-client>=1.18.0
+Requires-Dist: rerankers>=0.10.0
+Requires-Dist: scikit-learn>=1.9.0
+Requires-Dist: scipy>=1.18.0
+Requires-Dist: spacy>=3.8.3,<4
+Requires-Dist: transformers>=5.13.0
+Requires-Dist: umap-learn>=0.5.12
+Requires-Dist: uuid6>=2025.0.1
+Requires-Dist: uvicorn>=0.51.0
+Requires-Dist: gliner>=0.2.24 ; extra == 'gliner'
+Requires-Dist: gliner-spacy>=0.0.11 ; extra == 'gliner'
+Requires-Dist: onnxruntime>=1.27.0 ; extra == 'gliner'
+Requires-Dist: torch>=2.13.0 ; extra == 'html'
+Requires-Dist: igraph>=1.0.0 ; extra == 'leiden'
+Requires-Dist: leidenalg>=0.12.0 ; extra == 'leiden'
+Requires-Dist: en-core-web-sm ; extra == 'ner'
+Requires-Python: >=3.13
+Project-URL: Homepage, https://github.com/hbmartin/refindery
+Project-URL: Repository, https://github.com/hbmartin/refindery
+Project-URL: Changelog, https://github.com/hbmartin/refindery/releases
+Provides-Extra: gliner
+Provides-Extra: html
+Provides-Extra: leiden
+Provides-Extra: ner
+Description-Content-Type: text/markdown
+
+# Refindery
+
+A local, single-machine retrieval engine over the web pages you read.
+
+Upstream capture systems (browser extensions, history readers) extract
+main-body text and POST it here. Refindery chunks, embeds, indexes, clusters,
+and extracts entities from that text, then serves hybrid retrieval over it via
+a local HTTP API and an MCP server.
+
+**This is a retrieval engine, not a Q&A system.** It returns ranked, grounded
+passages with provenance. Synthesis is the caller's job — typically an LLM
+agent (e.g. Claude via MCP) that treats Refindery as a tool. No generation
+appears on the query path.
+
+## Jobs to be done
+
+1. **Refind** — "I read something about X, take me back to it."
+2. **Synthesize** — "What have I learned about Y?" (agent-mediated; Refindery supplies the passages)
+3. **Resurface** — "What have I been reading a lot about?" (clusters, similarity)
+
+## Architecture
+
+Hexagonal / ports-and-adapters. A single non-blocking asyncio process hosts
+the FastAPI app, the MCP server, and the job queue consumer; CPU-bound work
+runs in a process pool.
+
+| Port              | Default adapter                | Alternatives                       |
+| ----------------- | ------------------------------ | ---------------------------------- |
+| `VectorStore`     | Qdrant (Docker)                | LanceDB (in-process, zero daemon)  |
+| `MetadataStore`   | SQLite (WAL)                   | Postgres (v2)                      |
+| `Embedder`        | Voyage (via catsu)             | Cohere, OpenAI, local              |
+| `Reranker`        | Cohere / Voyage (via rerankers)| local cross-encoders               |
+| `EntityExtractor` | spaCy + gazetteer              | GLiNER (extra), LLM                |
+| `ClusterEngine`   | UMAP + HDBSCAN                 | KMeans, Leiden (extra)             |
+
+Observability: OpenTelemetry traces (off by default), structured JSON logs,
+Prometheus `/metrics`, and a DuckDB append-only query log — the substrate for
+offline retrieval evals: `refindery eval score` computes nDCG/MRR/recall from
+logged queries + `/v1/feedback` labels, and `refindery eval replay` diffs two
+configurations (models or rerank on/off) over the same golden set.
+
+## Quickstart
+
+### Minimal profile (no Docker)
+
+```bash
+uv sync --extra ner
+export REFINDERY_AUTH_TOKEN="$(openssl rand -hex 24)"
+export REFINDERY_VECTOR_STORE=lancedb
+export VOYAGE_API_KEY=...           # or configure another embedding provider
+python -m refindery
+```
+
+### Docker profile (Qdrant, the default store)
+
+```bash
+uv sync --extra ner
+docker compose up -d qdrant
+export REFINDERY_AUTH_TOKEN="$(openssl rand -hex 24)"
+python -m refindery
+```
+
+### Fully containerized
+
+The multi-stage `Dockerfile` builds a slim image with the `ner` extra
+(no torch/gliner; add extras to the sync lines if you need them). Data
+lives on the `refindery_data` volume, model caches on `refindery_models`.
+
+```bash
+export REFINDERY_AUTH_TOKEN="$(openssl rand -hex 24)"
+export VOYAGE_API_KEY=...
+docker compose up -d --build
+curl -s http://127.0.0.1:8000/healthz
+```
+
+### Ingest and search
+
+```bash
+curl -s -X POST http://127.0.0.1:8000/v1/pages \
+  -H "Authorization: Bearer $REFINDERY_AUTH_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"url": "https://example.com/article", "title": "An Article",
+       "body_extracted": "Plain text main body content...",
+       "fetched_at": "2026-07-08T10:00:00Z", "source": "extension"}'
+
+curl -s -X POST http://127.0.0.1:8000/v1/search \
+  -H "Authorization: Bearer $REFINDERY_AUTH_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"query": "main body content"}'
+```
+
+### MCP
+
+The MCP server is served over streamable HTTP at `/mcp` (same bearer token):
+
+```bash
+claude mcp add --transport http refindery http://127.0.0.1:8000/mcp \
+  --header "Authorization: Bearer $REFINDERY_AUTH_TOKEN"
+```
+
+Read-only tools are exposed by default (`search`, `get_page`, `similar_to`,
+`list_clusters`, …). Mutating tools (`add_page`, `forget`) are opt-in via
+`REFINDERY_MCP__ENABLE_MUTATING_TOOLS=true`. That flag controls visibility
+only — authorization comes from the token's scopes on every transport.
+
+### Auth tokens and scopes
+
+`REFINDERY_AUTH_TOKEN` is a single full-access token. To hand each capture
+source or agent its own revocable token, configure named tokens with `read`
+or `write` scopes (write implies read; both forms can coexist):
+
+```bash
+export REFINDERY_AUTH_TOKENS='[
+  {"name": "chrome-capture", "token": "...", "scopes": ["write"]},
+  {"name": "agent",          "token": "...", "scopes": ["read"]}
+]'
+```
+
+Read-scoped tokens can search, browse, compare, and record feedback;
+mutating endpoints (`add_page`, `forget`, model management, …) return 403
+without the `write` scope.
+
+## HTTP API
+
+```
+POST   /v1/pages                      ingest (body_extracted XOR body_html; neither → fetch)
+GET    /v1/pages/{id}                 full body_text + metadata
+GET    /v1/pages/{id}/status          queued|indexing|indexed|failed|dead
+GET    /v1/pages/{id}/similar         ?mediation=vector|cluster|entity&k=
+GET    /v1/pages/{id}/entities
+POST   /v1/search                     hybrid dense+sparse, RRF, rerank, filters
+POST   /v1/compare                    A/B embedding models (Jaccard@k, RBO, Kendall's τ)
+GET    /v1/clusters                   ?include_tombstoned=false
+GET    /v1/clusters/{id}
+POST   /v1/clusters/recompute
+GET    /v1/entities/{id_or_form}
+POST   /v1/forget                     purge + blacklist atomically
+GET    /v1/blacklist
+DELETE /v1/blacklist/{id}
+POST   /v1/models                     register embedding model
+POST   /v1/models/{id}/backfill       dry-run estimate, then confirm
+POST   /v1/models/{id}/activate
+DELETE /v1/models/{id}
+POST   /v1/feedback                   { query_id, page_id, relevant }
+GET    /healthz  /readyz  /metrics
+```
+
+Bearer token is always required, even on loopback. The server binds to
+`127.0.0.1` by default.
+
+## Optional extras
+
+| Extra       | Enables                                        | Pulls in            |
+| ----------- | ---------------------------------------------- | ------------------- |
+| `html`      | `body_html` / fetched-HTML extraction (pulpie) | torch (~2 GB)       |
+| `gliner`    | GLiNER zero-shot NER                           | gliner, onnxruntime |
+| `ner`       | spaCy NER model                                | en_core_web_sm      |
+| `leiden`    | Leiden clustering                              | igraph, leidenalg   |
+
+Entity extraction is required at startup. The default extractor chain needs
+the `ner` extra unless you configure a healthy gazetteer, GLiNER, or LLM
+extractor.
+
+See [Architecture](docs/architecture.md) for the ports/adapters map, the
+ingest→index→cluster data flow, and the search pipeline. See
+[Operations](docs/operations.md) for alpha reset commands, query-log
+purging, job lease behavior, vector-store caveats, and accepted risks.
+
+## Development
+
+See [CONTRIBUTING.md](CONTRIBUTING.md). TL;DR: `uv sync --all-groups --extra ner`,
+then `uv run ruff format . && uv run ruff check . && uv run pytest && uv run ty check && uv run pyrefly check`.

refindery-0.1.0/README.md

@@ -0,0 +1,172 @@
+# Refindery
+
+A local, single-machine retrieval engine over the web pages you read.
+
+Upstream capture systems (browser extensions, history readers) extract
+main-body text and POST it here. Refindery chunks, embeds, indexes, clusters,
+and extracts entities from that text, then serves hybrid retrieval over it via
+a local HTTP API and an MCP server.
+
+**This is a retrieval engine, not a Q&A system.** It returns ranked, grounded
+passages with provenance. Synthesis is the caller's job — typically an LLM
+agent (e.g. Claude via MCP) that treats Refindery as a tool. No generation
+appears on the query path.
+
+## Jobs to be done
+
+1. **Refind** — "I read something about X, take me back to it."
+2. **Synthesize** — "What have I learned about Y?" (agent-mediated; Refindery supplies the passages)
+3. **Resurface** — "What have I been reading a lot about?" (clusters, similarity)
+
+## Architecture
+
+Hexagonal / ports-and-adapters. A single non-blocking asyncio process hosts
+the FastAPI app, the MCP server, and the job queue consumer; CPU-bound work
+runs in a process pool.
+
+| Port              | Default adapter                | Alternatives                       |
+| ----------------- | ------------------------------ | ---------------------------------- |
+| `VectorStore`     | Qdrant (Docker)                | LanceDB (in-process, zero daemon)  |
+| `MetadataStore`   | SQLite (WAL)                   | Postgres (v2)                      |
+| `Embedder`        | Voyage (via catsu)             | Cohere, OpenAI, local              |
+| `Reranker`        | Cohere / Voyage (via rerankers)| local cross-encoders               |
+| `EntityExtractor` | spaCy + gazetteer              | GLiNER (extra), LLM                |
+| `ClusterEngine`   | UMAP + HDBSCAN                 | KMeans, Leiden (extra)             |
+
+Observability: OpenTelemetry traces (off by default), structured JSON logs,
+Prometheus `/metrics`, and a DuckDB append-only query log — the substrate for
+offline retrieval evals: `refindery eval score` computes nDCG/MRR/recall from
+logged queries + `/v1/feedback` labels, and `refindery eval replay` diffs two
+configurations (models or rerank on/off) over the same golden set.
+
+## Quickstart
+
+### Minimal profile (no Docker)
+
+```bash
+uv sync --extra ner
+export REFINDERY_AUTH_TOKEN="$(openssl rand -hex 24)"
+export REFINDERY_VECTOR_STORE=lancedb
+export VOYAGE_API_KEY=...           # or configure another embedding provider
+python -m refindery
+```
+
+### Docker profile (Qdrant, the default store)
+
+```bash
+uv sync --extra ner
+docker compose up -d qdrant
+export REFINDERY_AUTH_TOKEN="$(openssl rand -hex 24)"
+python -m refindery
+```
+
+### Fully containerized
+
+The multi-stage `Dockerfile` builds a slim image with the `ner` extra
+(no torch/gliner; add extras to the sync lines if you need them). Data
+lives on the `refindery_data` volume, model caches on `refindery_models`.
+
+```bash
+export REFINDERY_AUTH_TOKEN="$(openssl rand -hex 24)"
+export VOYAGE_API_KEY=...
+docker compose up -d --build
+curl -s http://127.0.0.1:8000/healthz
+```
+
+### Ingest and search
+
+```bash
+curl -s -X POST http://127.0.0.1:8000/v1/pages \
+  -H "Authorization: Bearer $REFINDERY_AUTH_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"url": "https://example.com/article", "title": "An Article",
+       "body_extracted": "Plain text main body content...",
+       "fetched_at": "2026-07-08T10:00:00Z", "source": "extension"}'
+
+curl -s -X POST http://127.0.0.1:8000/v1/search \
+  -H "Authorization: Bearer $REFINDERY_AUTH_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"query": "main body content"}'
+```
+
+### MCP
+
+The MCP server is served over streamable HTTP at `/mcp` (same bearer token):
+
+```bash
+claude mcp add --transport http refindery http://127.0.0.1:8000/mcp \
+  --header "Authorization: Bearer $REFINDERY_AUTH_TOKEN"
+```
+
+Read-only tools are exposed by default (`search`, `get_page`, `similar_to`,
+`list_clusters`, …). Mutating tools (`add_page`, `forget`) are opt-in via
+`REFINDERY_MCP__ENABLE_MUTATING_TOOLS=true`. That flag controls visibility
+only — authorization comes from the token's scopes on every transport.
+
+### Auth tokens and scopes
+
+`REFINDERY_AUTH_TOKEN` is a single full-access token. To hand each capture
+source or agent its own revocable token, configure named tokens with `read`
+or `write` scopes (write implies read; both forms can coexist):
+
+```bash
+export REFINDERY_AUTH_TOKENS='[
+  {"name": "chrome-capture", "token": "...", "scopes": ["write"]},
+  {"name": "agent",          "token": "...", "scopes": ["read"]}
+]'
+```
+
+Read-scoped tokens can search, browse, compare, and record feedback;
+mutating endpoints (`add_page`, `forget`, model management, …) return 403
+without the `write` scope.
+
+## HTTP API
+
+```
+POST   /v1/pages                      ingest (body_extracted XOR body_html; neither → fetch)
+GET    /v1/pages/{id}                 full body_text + metadata
+GET    /v1/pages/{id}/status          queued|indexing|indexed|failed|dead
+GET    /v1/pages/{id}/similar         ?mediation=vector|cluster|entity&k=
+GET    /v1/pages/{id}/entities
+POST   /v1/search                     hybrid dense+sparse, RRF, rerank, filters
+POST   /v1/compare                    A/B embedding models (Jaccard@k, RBO, Kendall's τ)
+GET    /v1/clusters                   ?include_tombstoned=false
+GET    /v1/clusters/{id}
+POST   /v1/clusters/recompute
+GET    /v1/entities/{id_or_form}
+POST   /v1/forget                     purge + blacklist atomically
+GET    /v1/blacklist
+DELETE /v1/blacklist/{id}
+POST   /v1/models                     register embedding model
+POST   /v1/models/{id}/backfill       dry-run estimate, then confirm
+POST   /v1/models/{id}/activate
+DELETE /v1/models/{id}
+POST   /v1/feedback                   { query_id, page_id, relevant }
+GET    /healthz  /readyz  /metrics
+```
+
+Bearer token is always required, even on loopback. The server binds to
+`127.0.0.1` by default.
+
+## Optional extras
+
+| Extra       | Enables                                        | Pulls in            |
+| ----------- | ---------------------------------------------- | ------------------- |
+| `html`      | `body_html` / fetched-HTML extraction (pulpie) | torch (~2 GB)       |
+| `gliner`    | GLiNER zero-shot NER                           | gliner, onnxruntime |
+| `ner`       | spaCy NER model                                | en_core_web_sm      |
+| `leiden`    | Leiden clustering                              | igraph, leidenalg   |
+
+Entity extraction is required at startup. The default extractor chain needs
+the `ner` extra unless you configure a healthy gazetteer, GLiNER, or LLM
+extractor.
+
+See [Architecture](docs/architecture.md) for the ports/adapters map, the
+ingest→index→cluster data flow, and the search pipeline. See
+[Operations](docs/operations.md) for alpha reset commands, query-log
+purging, job lease behavior, vector-store caveats, and accepted risks.
+
+## Development
+
+See [CONTRIBUTING.md](CONTRIBUTING.md). TL;DR: `uv sync --all-groups --extra ner`,
+then `uv run ruff format . && uv run ruff check . && uv run pytest && uv run ty check && uv run pyrefly check`.

refindery-0.1.0/pyproject.toml

@@ -0,0 +1,153 @@
+[project]
+name = "refindery"
+version = "0.1.0"
+description = "A local, single-machine retrieval engine over the web pages you read."
+readme = "README.md"
+requires-python = ">=3.13"
+authors = [
+    { name = "Harold Martin", email = "harold.martin@gmail.com" },
+]
+license = "MIT"
+keywords = ["retrieval", "search", "rag", "embeddings", "mcp", "browser-history"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Programming Language :: Python :: 3.13",
+    "Framework :: FastAPI",
+    "Intended Audience :: Developers",
+    "Topic :: Text Processing :: Indexing",
+]
+dependencies = [
+    "aiosqlite>=0.22.1",
+    "catsu>=0.1.8",
+    "chonkie>=1.7.0",
+    "duckdb>=1.5.4",
+    "fastapi>=0.139.0",
+    "fastapi-mcp>=0.4.0",
+    "fastembed>=0.8.0",
+    "httpx>=0.28.1",
+    "huey>=3.1.1",
+    "inflect>=7.5.0",
+    "lancedb>=0.34.0",
+    "model2vec>=0.8.2",
+    "numpy>=2.2",
+    "opentelemetry-api>=1.43.0",
+    "opentelemetry-exporter-otlp-proto-http>=1.43.0",
+    "opentelemetry-instrumentation-fastapi>=0.64b0",
+    "opentelemetry-sdk>=1.43.0",
+    "prometheus-client>=0.25.0",
+    "pyarrow>=24.0.0",
+    "pydantic>=2.13.4",
+    "pydantic-settings>=2.14.2",
+    "pypdf>=6.14.2",
+    "qdrant-client>=1.18.0",
+    "rerankers>=0.10.0",
+    "scikit-learn>=1.9.0",
+    "scipy>=1.18.0",
+    "spacy>=3.8.3,<4",
+    "transformers>=5.13.0",
+    "umap-learn>=0.5.12",
+    "uuid6>=2025.0.1",
+    "uvicorn>=0.51.0",
+]
+
+[project.scripts]
+refindery = "refindery.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/hbmartin/refindery"
+Repository = "https://github.com/hbmartin/refindery"
+Changelog = "https://github.com/hbmartin/refindery/releases"
+
+[project.optional-dependencies]
+html = [
+    "torch>=2.13.0",
+]
+gliner = [
+    "gliner>=0.2.24",
+    "gliner-spacy>=0.0.11",
+    "onnxruntime>=1.27.0",
+]
+leiden = [
+    "igraph>=1.0.0",
+    "leidenalg>=0.12.0",
+]
+ner = [
+    "en-core-web-sm",
+]
+
+[dependency-groups]
+dev = [
+    "deptry>=0.24.0",
+    "lizard>=1.23.0",
+    "pyrefly>=1.1.1",
+    "pyroma>=5.0",
+    "pytest>=9.1.1",
+    "pytest-asyncio>=1.0.0",
+    "pytest-cov>=7.1.0",
+    "pytest-httpx>=0.36.2",
+    "ruff>=0.15.20",
+    "testcontainers>=4.14.2",
+    "ty>=0.0.57",
+]
+
+[tool.uv]
+# numba is not imported directly; the floor keeps the resolver from walking
+# back to ancient sdists instead of downgrading numpy (numba needs numpy<2.5).
+constraint-dependencies = ["numba>=0.61"]
+
+[tool.uv.sources]
+en-core-web-sm = { url = "https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl" }
+
+[tool.deptry.per_rule_ignores]
+DEP002 = [
+    # Optional extras, imported lazily (deptry cannot see conditional imports):
+    "torch", "gliner", "gliner-spacy", "onnxruntime", "en-core-web-sm",
+    "igraph", "leidenalg",
+    # Lazy imports inside functions/adapters:
+    "scikit-learn", "umap-learn", "spacy", "model2vec",
+    "opentelemetry-exporter-otlp-proto-http",
+    "opentelemetry-instrumentation-fastapi",
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+asyncio_mode = "auto"
+markers = [
+    "qdrant: requires a running Qdrant instance (QDRANT_URL or Docker)",
+    "slow: slow tests (model downloads, JIT warmup)",
+    "external: hits external APIs; requires credentials",
+]
+addopts = "-m 'not external'"
+
+[tool.coverage.run]
+source = ["src/refindery"]
+
+[tool.coverage.report]
+# Floor set just below measured coverage (85% local with qdrant tests
+# skipped) to stop silent rot; applies to any full --cov run, local or CI.
+fail_under = 84
+
+[tool.pyrefly]
+search-path = ["src", "."]
+
+[tool.ruff]
+# Exclude a variety of commonly ignored directories.
+exclude = [".bzr", ".direnv", ".eggs", ".git", ".git-rewrite", ".hg", ".ipynb_checkpoints", ".mypy_cache", ".nox", ".pants.d", ".pyenv", ".pytest_cache", ".pytype", ".ruff_cache", ".svn", ".tox", ".venv", ".vscode", "__pypackages__", "_build", "buck-out", "build", "dist", "node_modules", "setup.py", "site-packages", "venv"]
+
+line-length = 88
+indent-width = 4
+target-version = "py313"
+
+[tool.ruff.lint]
+select = ["ALL"]
+ignore = ["COM812", "D105", "D107", "D203", "D212", "D213", "EM101", "PLR2004", "T201"]
+
+[tool.ruff.lint.pylint]
+max-args = 10
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**/*.py" = ["ANN001", "ANN002", "ANN003", "ANN201", "ARG001", "ARG005", "D101", "D102", "D103", "D104", "DTZ003", "ERA001", "F403", "F405", "FBT002", "G004", "N802", "PLC0415", "PLR2004", "PLW1510", "PT011", "PT013", "PTH110", "PTH118", "PTH119", "PTH122", "PTH123", "PTH207", "S101", "S102", "S311", "S603", "S607", "SIM115"]
+
+[build-system]
+requires = ["uv_build>=0.11.28,<0.12.0"]
+build-backend = "uv_build"

refindery-0.1.0/src/refindery/__init__.py

@@ -0,0 +1,3 @@
+"""Refindery: a local, single-machine retrieval engine over the web pages you read."""
+
+__version__ = "0.1.0"

refindery-0.1.0/src/refindery/__main__.py

@@ -0,0 +1,6 @@
+"""Entry point: ``python -m refindery`` dispatches to the CLI (default: serve)."""
+
+from refindery.cli import main
+
+if __name__ == "__main__":
+    main()

refindery-0.1.0/src/refindery/adapters/__init__.py

@@ -0,0 +1 @@
+"""Adapters: concrete implementations of the application ports."""

refindery-0.1.0/src/refindery/adapters/chunking/__init__.py

@@ -0,0 +1 @@
+"""Canonical chunking adapter (chonkie)."""

refindery-0.1.0/src/refindery/adapters/chunking/chonkie_chunker.py

@@ -0,0 +1,73 @@
+"""Sentence-aware chunking via chonkie with a hard token ceiling.
+
+Chunking is canonical and model-independent: one chunking, every embedding
+model embeds the same spans. The cl100k tokenizer is the canonical counter.
+chonkie targets ``chunk_size`` but can exceed it on pathological input, so a
+post-pass re-splits anything above the hard max (which model registration
+guarantees is within every registered model's budget).
+"""
+
+from chonkie import SentenceChunker, TokenChunker
+
+from refindery.domain.ids import PageId, new_chunk_id
+from refindery.domain.models import Chunk
+
+_TOKENIZER = "cl100k_base"
+
+
+class ChonkieChunker:
+    """Chunker port implementation over chonkie's SentenceChunker."""
+
+    def __init__(
+        self,
+        *,
+        target_tokens: int = 448,
+        overlap_tokens: int = 64,
+        hard_max_tokens: int = 512,
+    ) -> None:
+        self._hard_max = hard_max_tokens
+        self._sentence = SentenceChunker(
+            tokenizer=_TOKENIZER,
+            chunk_size=target_tokens,
+            chunk_overlap=overlap_tokens,
+        )
+        self._splitter = TokenChunker(
+            tokenizer=_TOKENIZER,
+            chunk_size=hard_max_tokens,
+            chunk_overlap=0,
+        )
+
+    def chunk(self, *, page_id: PageId, text: str) -> list[Chunk]:
+        """Split ``text`` into ordered chunks with char offsets and token counts."""
+        if not text.strip():
+            return []
+        spans: list[tuple[str, int, int, int]] = []
+        for piece in self._sentence.chunk(text):
+            if piece.token_count <= self._hard_max:
+                spans.append(
+                    (piece.text, piece.start_index, piece.end_index, piece.token_count)
+                )
+                continue
+            spans.extend(
+                (
+                    sub.text,
+                    piece.start_index + sub.start_index,
+                    piece.start_index + sub.end_index,
+                    sub.token_count,
+                )
+                for sub in self._splitter.chunk(piece.text)
+            )
+        return [
+            Chunk(
+                id=new_chunk_id(),
+                page_id=page_id,
+                ordinal=ordinal,
+                text=chunk_text,
+                token_count=token_count,
+                char_start=char_start,
+                char_end=char_end,
+            )
+            for ordinal, (chunk_text, char_start, char_end, token_count) in enumerate(
+                spans
+            )
+        ]

refindery-0.1.0/src/refindery/adapters/clock.py

@@ -0,0 +1,11 @@
+"""System clock adapter."""
+
+from datetime import UTC, datetime
+
+
+class SystemClock:
+    """Wall clock returning tz-aware UTC datetimes."""
+
+    def now(self) -> datetime:
+        """Return the current UTC time."""
+        return datetime.now(tz=UTC)

refindery-0.1.0/src/refindery/adapters/cluster/__init__.py

@@ -0,0 +1 @@
+"""Clustering engine adapters (UMAP/PCA + HDBSCAN/KMeans in a process pool)."""