biblicus 0.1.1__tar.gz

biblicus-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Biblicus Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

biblicus-0.1.1/MANIFEST.in

@@ -0,0 +1,19 @@
+include README.md
+include LICENSE
+include pyproject.toml
+
+recursive-include src *.py
+recursive-include docs *.rst *.md *.py
+recursive-include features *.feature *.py
+recursive-include scripts *.py
+recursive-include datasets *.json
+
+prune corpora
+prune reports
+prune docs/_build
+
+global-exclude *.pyc
+global-exclude *.pyo
+global-exclude __pycache__/*
+global-exclude .DS_Store
+global-exclude .coverage

biblicus-0.1.1/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.4
+Name: biblicus
+Version: 0.1.1
+Summary: Command line interface and Python library for corpus ingestion, retrieval, and evaluation.
+License: MIT
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.0
+Requires-Dist: PyYAML>=6.0
+Provides-Extra: dev
+Requires-Dist: behave>=1.2.6; extra == "dev"
+Requires-Dist: coverage[toml]>=7.0; extra == "dev"
+Requires-Dist: sphinx>=7.0; extra == "dev"
+Requires-Dist: myst-parser>=2.0; extra == "dev"
+Requires-Dist: ruff>=0.4.0; extra == "dev"
+Requires-Dist: black>=24.0; extra == "dev"
+Requires-Dist: python-semantic-release>=9.0.0; extra == "dev"
+Dynamic: license-file
+
+# Biblicus
+
+Make your documents usable by your assistant, then decide later how you will search and retrieve them.
+
+If you are building an assistant in Python, you probably have material you want it to use: notes, documents, web pages, and reference files. A common approach is retrieval augmented generation, where a system retrieves relevant material and uses it as evidence when generating a response.
+
+The first practical problem is not retrieval. It is collection and care. You need a stable place to put raw items, you need a small amount of metadata so you can find them again, and you need a way to evolve your retrieval approach over time without rewriting ingestion.
+
+This library gives you a corpus, which is a normal folder on disk. It stores each ingested item as a file, with optional metadata stored next to it. You can open and inspect the raw files directly. Any derived catalog or index can be rebuilt from the raw corpus.
+
+It integrates with LangChain, Tactus, Pydantic AI, and the agent development kit. Use it from Python or from the command line interface.
+
+See [retrieval augmented generation overview] for a short introduction to the idea.
+
+## The framework
+
+The framework is a small, explicit vocabulary that appears in code, specifications, and documentation. If you learn these words, the rest of the system becomes predictable.
+
+- Corpus is the folder that holds raw items and their metadata.
+- Item is the raw bytes of a document or other artifact, plus its source.
+- Catalog is the rebuildable index of the corpus.
+- Evidence is what retrieval returns, ready to be turned into context for a large language model.
+- Run is a recorded retrieval build for a corpus.
+- Backend is a pluggable retrieval implementation.
+- Recipe is a named configuration for a backend.
+- Pipeline stage is a distinct retrieval step such as retrieve, rerank, and filter.
+
+## Practical value
+
+- You can ingest raw material once, then try many retrieval approaches over time.
+- You can keep raw files readable and portable, without locking your data inside a database.
+- You can evaluate retrieval runs against shared datasets and compare backends using the same corpus.
+
+## Typical flow
+
+- Initialize a corpus folder.
+- Ingest items from file paths, web addresses, or text input.
+- Reindex to refresh the catalog after edits.
+- Build a retrieval run with a backend.
+- Query the run to collect evidence and evaluate it with datasets.
+
+## Install
+
+This repository is a working Python package. Install it into a virtual environment from the repository root.
+
+```
+python3 -m pip install -e .
+```
+
+After the first release, you can install it from Python Package Index.
+
+```
+python3 -m pip install biblicus
+```
+
+## Quick start
+
+```
+biblicus init corpora/example
+biblicus ingest --corpus corpora/example notes/example.txt
+echo "A short note" | biblicus ingest --corpus corpora/example --stdin --title "First note"
+biblicus list --corpus corpora/example
+biblicus build --corpus corpora/example --backend scan
+biblicus query --corpus corpora/example --query "note"
+```
+
+## Python usage
+
+From Python, the same flow is available through the Corpus class and backend interfaces. The public surface area is small on purpose.
+
+- Create a corpus with `Corpus.init` or open one with `Corpus.open`.
+- Ingest notes with `Corpus.ingest_note`.
+- Ingest files or web addresses with `Corpus.ingest_source`.
+- List items with `Corpus.list_items`.
+- Build a retrieval run with `get_backend` and `backend.build_run`.
+- Query a run with `backend.query`.
+- Evaluate with `evaluate_run`.
+
+## How it fits into an assistant
+
+In an assistant system, retrieval usually produces context for a model call. This library treats evidence as the primary output so you can decide how to use it.
+
+- Use a corpus as the source of truth for raw items.
+- Use a backend run to build any derived artifacts needed for retrieval.
+- Use queries to obtain evidence objects.
+- Convert evidence into the format your framework expects, such as message content, tool output, or citations.
+
+## Learn more
+
+The documents below are written to be read in order.
+
+- [Architecture][architecture]
+- [Backends][backends]
+
+## Metadata and catalog
+
+Raw items are stored as files in the corpus raw directory. Metadata can live in a Markdown front matter block or a sidecar file with the suffix `.biblicus.yml`. The catalog lives in `.biblicus/catalog.json` and can be rebuilt at any time with `biblicus reindex`.
+
+## Corpus layout
+
+```
+corpus/
+  raw/
+    item.bin
+    item.bin.biblicus.yml
+  .biblicus/
+    config.json
+    catalog.json
+    runs/
+      run-id.json
+```
+
+## Retrieval backends
+
+Two backends are included.
+
+- `scan` is a minimal baseline that scans raw items directly.
+- `sqlite-full-text-search` is a practical baseline that builds a full text search index in Sqlite.
+
+## Integration corpus and evaluation dataset
+
+Use `scripts/download_wikipedia.py` to download a small integration corpus from Wikipedia when running tests or demos. The repository does not include that content.
+
+The dataset file `datasets/wikipedia_mini.json` provides a small evaluation set that matches the integration corpus.
+
+## Tests and coverage
+
+```
+python3 scripts/test.py
+```
+
+## Releases
+
+Releases are automated from the main branch using semantic versioning and conventional commit messages.
+
+The release pipeline publishes a GitHub release and uploads the package to Python Package Index when continuous integration succeeds.
+
+Publishing uses a Python Package Index token stored in the GitHub secret named PYPI_TOKEN.
+
+## Documentation
+
+Reference documentation is generated from Sphinx style docstrings. Build the documentation with the command below.
+
+```
+sphinx-build -b html docs docs/_build
+```
+
+## License
+
+License terms are in `LICENSE`.
+
+[retrieval augmented generation overview]: https://en.wikipedia.org/wiki/Retrieval-augmented_generation
+[architecture]: docs/ARCHITECTURE.md
+[backends]: docs/BACKENDS.md

biblicus-0.1.1/README.md

@@ -0,0 +1,154 @@
+# Biblicus
+
+Make your documents usable by your assistant, then decide later how you will search and retrieve them.
+
+If you are building an assistant in Python, you probably have material you want it to use: notes, documents, web pages, and reference files. A common approach is retrieval augmented generation, where a system retrieves relevant material and uses it as evidence when generating a response.
+
+The first practical problem is not retrieval. It is collection and care. You need a stable place to put raw items, you need a small amount of metadata so you can find them again, and you need a way to evolve your retrieval approach over time without rewriting ingestion.
+
+This library gives you a corpus, which is a normal folder on disk. It stores each ingested item as a file, with optional metadata stored next to it. You can open and inspect the raw files directly. Any derived catalog or index can be rebuilt from the raw corpus.
+
+It integrates with LangChain, Tactus, Pydantic AI, and the agent development kit. Use it from Python or from the command line interface.
+
+See [retrieval augmented generation overview] for a short introduction to the idea.
+
+## The framework
+
+The framework is a small, explicit vocabulary that appears in code, specifications, and documentation. If you learn these words, the rest of the system becomes predictable.
+
+- Corpus is the folder that holds raw items and their metadata.
+- Item is the raw bytes of a document or other artifact, plus its source.
+- Catalog is the rebuildable index of the corpus.
+- Evidence is what retrieval returns, ready to be turned into context for a large language model.
+- Run is a recorded retrieval build for a corpus.
+- Backend is a pluggable retrieval implementation.
+- Recipe is a named configuration for a backend.
+- Pipeline stage is a distinct retrieval step such as retrieve, rerank, and filter.
+
+## Practical value
+
+- You can ingest raw material once, then try many retrieval approaches over time.
+- You can keep raw files readable and portable, without locking your data inside a database.
+- You can evaluate retrieval runs against shared datasets and compare backends using the same corpus.
+
+## Typical flow
+
+- Initialize a corpus folder.
+- Ingest items from file paths, web addresses, or text input.
+- Reindex to refresh the catalog after edits.
+- Build a retrieval run with a backend.
+- Query the run to collect evidence and evaluate it with datasets.
+
+## Install
+
+This repository is a working Python package. Install it into a virtual environment from the repository root.
+
+```
+python3 -m pip install -e .
+```
+
+After the first release, you can install it from Python Package Index.
+
+```
+python3 -m pip install biblicus
+```
+
+## Quick start
+
+```
+biblicus init corpora/example
+biblicus ingest --corpus corpora/example notes/example.txt
+echo "A short note" | biblicus ingest --corpus corpora/example --stdin --title "First note"
+biblicus list --corpus corpora/example
+biblicus build --corpus corpora/example --backend scan
+biblicus query --corpus corpora/example --query "note"
+```
+
+## Python usage
+
+From Python, the same flow is available through the Corpus class and backend interfaces. The public surface area is small on purpose.
+
+- Create a corpus with `Corpus.init` or open one with `Corpus.open`.
+- Ingest notes with `Corpus.ingest_note`.
+- Ingest files or web addresses with `Corpus.ingest_source`.
+- List items with `Corpus.list_items`.
+- Build a retrieval run with `get_backend` and `backend.build_run`.
+- Query a run with `backend.query`.
+- Evaluate with `evaluate_run`.
+
+## How it fits into an assistant
+
+In an assistant system, retrieval usually produces context for a model call. This library treats evidence as the primary output so you can decide how to use it.
+
+- Use a corpus as the source of truth for raw items.
+- Use a backend run to build any derived artifacts needed for retrieval.
+- Use queries to obtain evidence objects.
+- Convert evidence into the format your framework expects, such as message content, tool output, or citations.
+
+## Learn more
+
+The documents below are written to be read in order.
+
+- [Architecture][architecture]
+- [Backends][backends]
+
+## Metadata and catalog
+
+Raw items are stored as files in the corpus raw directory. Metadata can live in a Markdown front matter block or a sidecar file with the suffix `.biblicus.yml`. The catalog lives in `.biblicus/catalog.json` and can be rebuilt at any time with `biblicus reindex`.
+
+## Corpus layout
+
+```
+corpus/
+  raw/
+    item.bin
+    item.bin.biblicus.yml
+  .biblicus/
+    config.json
+    catalog.json
+    runs/
+      run-id.json
+```
+
+## Retrieval backends
+
+Two backends are included.
+
+- `scan` is a minimal baseline that scans raw items directly.
+- `sqlite-full-text-search` is a practical baseline that builds a full text search index in Sqlite.
+
+## Integration corpus and evaluation dataset
+
+Use `scripts/download_wikipedia.py` to download a small integration corpus from Wikipedia when running tests or demos. The repository does not include that content.
+
+The dataset file `datasets/wikipedia_mini.json` provides a small evaluation set that matches the integration corpus.
+
+## Tests and coverage
+
+```
+python3 scripts/test.py
+```
+
+## Releases
+
+Releases are automated from the main branch using semantic versioning and conventional commit messages.
+
+The release pipeline publishes a GitHub release and uploads the package to Python Package Index when continuous integration succeeds.
+
+Publishing uses a Python Package Index token stored in the GitHub secret named PYPI_TOKEN.
+
+## Documentation
+
+Reference documentation is generated from Sphinx style docstrings. Build the documentation with the command below.
+
+```
+sphinx-build -b html docs docs/_build
+```
+
+## License
+
+License terms are in `LICENSE`.
+
+[retrieval augmented generation overview]: https://en.wikipedia.org/wiki/Retrieval-augmented_generation
+[architecture]: docs/ARCHITECTURE.md
+[backends]: docs/BACKENDS.md

biblicus-0.1.1/datasets/wikipedia_mini.json

@@ -0,0 +1,37 @@
+{
+  "schema_version": 1,
+  "name": "wikipedia-mini",
+  "description": "Small evaluation set aligned with the Wikipedia integration corpus.",
+  "queries": [
+    {
+      "query_id": "q1",
+      "query_text": "mathematician who worked on the Analytical Engine",
+      "expected_source_uri": "https://en.wikipedia.org/wiki/Ada_Lovelace",
+      "kind": "gold"
+    },
+    {
+      "query_id": "q2",
+      "query_text": "pioneer of computer programming in the US Navy",
+      "expected_source_uri": "https://en.wikipedia.org/wiki/Grace_Hopper",
+      "kind": "gold"
+    },
+    {
+      "query_id": "q3",
+      "query_text": "proposed the Church-Turing thesis",
+      "expected_source_uri": "https://en.wikipedia.org/wiki/Alan_Turing",
+      "kind": "gold"
+    },
+    {
+      "query_id": "q4",
+      "query_text": "invented information theory",
+      "expected_source_uri": "https://en.wikipedia.org/wiki/Claude_Shannon",
+      "kind": "gold"
+    },
+    {
+      "query_id": "q5",
+      "query_text": "coined the term artificial intelligence",
+      "expected_source_uri": "https://en.wikipedia.org/wiki/John_McCarthy_(computer_scientist)",
+      "kind": "synthetic"
+    }
+  ]
+}

biblicus-0.1.1/docs/ARCHITECTURE.md

@@ -0,0 +1,179 @@
+# Biblicus Architecture
+
+Biblicus is a command line interface first **corpus** manager for ingesting, curating, and evaluating corpora used by assistant systems. The early goal is to make it easy to add raw, unstructured content, while keeping the system structured enough to support reproducible experiments.
+
+## How we design
+
+Design starts from strict behavior-driven development:
+- The authoritative description of behavior lives in `features/*.feature`.
+- All changes should follow specification-first behavior-driven development: failing scenario, implementation, passing scenario, then refactor.
+- Behavior-driven development scenarios are not an afterthought: they are how we keep the domain vocabulary consistent and the platform comparable across backends and recipes.
+- **Specification completeness** is mandatory: if behavior exists, it must be specified. Ambiguous or untestable behavior should be removed or turned into an explicit error.
+
+## Domain-specific cognitive framework
+
+The domain-specific cognitive framework is the set of **stable nouns**, **verbs**, and **invariants** that make Biblicus pleasant to use over time.
+We prefer a small set of universal concepts with strict semantics over a large set of ad-hoc flags.
+
+We also treat **Pydantic models** as the canonical way to codify and validate these constructs at boundaries.
+
+## The Python developer mental model
+
+If this system is pleasant to use, a Python developer should be able to describe intent with the core nouns:
+
+- I have a **corpus** at this path or uniform resource identifier.
+- I ingest an **item** with optional **metadata**.
+- I rebuild the derived **index** after edits.
+- I run a **recipe** against the same corpus.
+- I query and receive **evidence**.
+
+Anything that does not map cleanly to these nouns is either a derived helper or a backend-specific implementation detail that should not leak.
+
+## Relationship to agent frameworks
+
+Biblicus is designed to integrate with agent frameworks through explicit tools and clear application programming interfaces. Tactus is one target environment with strong isolation requirements.
+
+- **Tools and toolsets**, including the Model Context Protocol, are the primary capability boundary.
+- **Sandboxing and brokered or secretless execution** are primary deployment modes.
+- **Durability and evaluations** are central: invariants via specifications, quality via evaluations.
+
+## Core concepts
+
+### Concepts
+
+- **Corpus**: a named, mutable collection rooted at a path or uniform resource identifier. In version zero it is typically a local folder containing raw files plus a `.biblicus/` directory for minimal metadata.
+- **Item**: the unit of ingestion in a corpus: raw bytes of any modality, including text, images, Portable Document Format documents, audio, and video, plus optional metadata and provenance.
+- **Knowledge base backend**: an implementation that can ingest and retrieve from a corpus, such as scan, full text search, vector retrieval, or hybrid retrieval, exposed to procedures through retrieval primitives.
+- **Retrieval recipe**: a named configuration bundle for a backend, such as chunking rules, embedding model and version, hybrid weights, reranker choice, and filters. This is what we benchmark and compare.
+- **Recipe manifest**: a reproducibility record describing the backend and recipe parameters, plus any referenced materializations and build runs.
+- **Materialization**: an optional, persisted representation derived from raw content for a given recipe and backend, such as chunks, embeddings, or indexes. Some backends intentionally have none and operate on demand.
+- **Evidence**: structured retrieval output from backend queries. Evidence includes spans, scores, and provenance used by downstream retrieval augmented generation procedures.
+- **Pipeline stage / editorial layer**: a structured step that transforms, filters, extracts, or curates content, such as raw, curated, and published, or extract text from Portable Document Format documents.
+
+## Design principles
+
+- **Primitives + derived constructs**: keep the protocol surface small and composable; ship higher-level helpers and example procedures on top.
+- **Minimal opinion raw store**: raw ingestion should work for a folder of files with optional lightweight tagging.
+- **Reproducibility by default**: comparisons require manifests (even when there are no persisted materializations).
+- **Mutability is real**: corpora are edited, pruned, and reorganized; re-indexing must be a core workflow.
+- **Separation of concerns**: retrieval returns evidence; retrieval-augmented generation patterns live in Tactus procedures (not inside the knowledge base backend).
+- **Deployment flexibility**: same interface across local/offline, brokered external services, and hybrid environments.
+- **Evidence is the primary output**: every retrieval returns structured evidence; everything else is a derived helper.
+
+## Locked decisions (version zero)
+
+These are explicit, opinionated policies encoded into the project:
+
+- **Evidence schema strictness**: moderate-to-strong schema. Evidence must include stable identifiers, provenance, and retrieval scores; richer fields (spans, stage, recipe and run identifiers) are expected.
+- **Retrieval stages**: multi-stage is explicit (retrieve, rerank, then filter). Pipelines are expressed through evidence metadata rather than hard-coded backends.
+- **Corpus versioning**: snapshot or reindex runs are versioned; full directed acyclic graph lineage is deferred.
+- **Evaluation datasets**: mixed human-labeled and synthetic questions; human-labeled for truth, synthetic for scale.
+- **Baseline retriever**: hybrid is the strategic target, but the first reference backend is deterministic lexical.
+- **Context budgeting**: evidence selection is governed by budgets (token, unit, and per-source limits), not a fixed count.
+
+## Evidence schema (version zero)
+
+Evidence is the canonical output of retrieval. Required fields:
+
+- `item_id`, `source_uri`, `media_type`
+- `score` and `rank`
+- `text` (or `content_ref` when non-text)
+- `stage` (for example, `scan`, `full-text-search`, `rerank`)
+- `recipe_id` / `run_id` (for reproducibility)
+- Optional: `span_start`, `span_end`, `hash`
+
+## Architectural policies version zero
+
+### Integration boundary
+
+- Biblicus can integrate with Tactus as a **Model Context Protocol toolset**, for example with tool names such as `knowledge_base_ingest`, `knowledge_base_query`, and `knowledge_base_stats`.
+- We will **not** add a knowledge base or retrieval augmented generation language primitive in version zero. Revisit only if we need semantics that tools cannot express cleanly, such as enforceable policy boundaries, runtime managed durability, caching hooks, or guaranteed instrumentation.
+
+### Interface packaging
+
+- The knowledge base interface is a **small protocol and reference implementation**, including tool schemas and a reference Model Context Protocol server. We will not build a full managed service in version zero.
+
+### Corpus identity and layout
+
+- Corpora are identified by a **uniform resource identifier**; simple strings and paths normalize to canonical `file://...`.
+- The raw corpus is the source of truth and must support:
+  - a plain folder of arbitrary files
+  - optional Markdown + Yet Another Markup Language front matter for lightweight tagging
+  - sidecar metadata for any file type (for example, `file.pdf.biblicus.yml`)
+- Raw items are written with **usable file extensions** whenever possible (based on `media_type`) so the corpus remains easy to browse and recover with ordinary operating system tools.
+
+### Mutability and editorial workflow
+
+- Corpora are **mutable**. Re-indexing and refresh are primary operations.
+- Filtering, pruning, and curation are primary needs; we may model this as a **multi-layer editorial pipeline** such as raw, curated, then published.
+
+### Pipeline stages
+
+- Text extraction (Portable Document Format, office documents, or image optical character recognition) is a **pipeline stage**, not part of raw ingestion.
+
+### Backend hosting modes (all supported)
+
+Biblicus must support all three backend hosting modes behind the same interface, and ship at least one reference example of each:
+
+- **In-process plugin**: simplest local minimum viable product and deterministic testing.
+- **Out-of-process local daemon**: isolates dependencies and supports warm indexes for heavier systems.
+- **Remote service**: production deployments, multi-tenant separation, and managed infrastructure.
+
+Backend hosting mode is a primary benchmark dimension (cold start, warm start, latency, throughput, cost, operational complexity).
+
+### Security / sandbox topology (all supported)
+
+Biblicus must support all three deployment topologies, selected as appropriate per environment and backend:
+
+- **In-sandbox**: the knowledge base runs inside the Tactus sandbox container (local, offline, simplest wiring).
+- **Brokered or external**: the knowledge base runs outside the sandbox and is accessed via tools (aligns with secretless or brokered execution).
+- **Hybrid**: mix modes across environments (for example, local development in-sandbox; production external).
+
+The interface stays the same; topology is configuration.
+
+### Query semantics
+
+- `knowledge_base_query` returns **evidence objects** as the low-level, composable building block.
+- Biblicus may ship higher-level convenience helpers built on top of evidence (for example, a prompt-ready context pack formatter), but those helpers remain derived and swappable.
+
+### Reproducibility
+
+- Biblicus always records a **recipe manifest** for reproducibility.
+- When a backend produces persisted materializations, Biblicus treats them as **versioned build runs** identified by `run_id` (rather than overwriting in place by default).
+- Manifests exist even for just-in-time backends (materializations may be empty).
+- Full directed acyclic graph lineage is not included in version zero; revisit only if needed.
+- Future (optional): define **shared materialization formats** (canonical chunk and embedding stores) so multiple backends can reuse intermediates when it makes sense; keep it opt-in.
+
+### Evaluation
+
+- Evaluate **both** knowledge base level behavior and end-to-end procedure behavior using **shared datasets**:
+  - **Knowledge base level**: retrieval metrics and system properties (for example, recall and mean reciprocal rank, latency, index size, and cost).
+  - **Procedure-level (Tactus)**: end-to-end success, policy compliance, and quality metrics across real inputs.
+
+### Catalog stance
+
+- The corpus catalog is **file-based** (committable, portable, backend-agnostic) so any backend/tool can consume it without requiring a database engine.
+- Canonical version zero format is a single JavaScript Object Notation file at `.biblicus/catalog.json`, written atomically (temporary file and rename) on updates.
+- The catalog includes `latest_run_id` and run manifests are stored at `.biblicus/runs/<run_id>.json`.
+- If this ever becomes a bottleneck at very large scales, we will **change the specification** (bump `schema_version`) rather than introduce multiple “supported” catalog storage modes.
+
+## Near-term deliverables
+
+1. Define Biblicus version zero knowledge base tool schemas (Model Context Protocol) for:
+   - `knowledge_base_ingest` (upsert documents)
+   - `knowledge_base_query` (retrieve evidence)
+   - `knowledge_base_get` and `knowledge_base_list` (basic management)
+   - `knowledge_base_stats` (latency, counts, sizes)
+2. Implement reference backend examples for each hosting mode:
+   - **In-process plugin**: a naive local backend (for example, metadata registry and lexical baseline) for determinism and tests
+   - **Local daemon**: a vector backend (Qdrant or Postgres with pgvector) for real use
+   - **Remote service**: the same vector backend configured against a remote endpoint
+3. Implement one reference Tactus procedure showing a basic retrieval-augmented generation pattern using the toolset.
+4. Add a small evaluation dataset and run `tactus eval` against multiple retrieval configs.
+
+## Open questions
+
+- **Editorial pipeline model**: do layers live as directory views, metadata flags, or both?
+- **Chunking strategy**: semantic vs fixed-size, and how to compare fairly across corpora.
+- **Re-ranking tradeoffs**: quality versus cost and latency, and when to use cross-encoders.
+- **Context synthesis**: raw snippets vs summary-based packs, and how to evaluate hallucination risk.

biblicus-0.1.1/docs/BACKENDS.md

@@ -0,0 +1,36 @@
+# Adding a Retrieval Backend
+
+Backends are pluggable engines that implement a small, stable interface.
+The goal is to make new retrieval ideas easy to test without reshaping the corpus.
+
+## Backend contract
+
+Backends implement two operations:
+
+- **Build run**: create a `RetrievalRun` manifest (and optional artifacts).
+- **Query**: return structured `Evidence` objects under a `QueryBudget`.
+
+## Implementation checklist
+
+1. **Define a Pydantic configuration model** for your backend recipe.
+2. **Implement `RetrievalBackend`**:
+   - `build_run(corpus, recipe_name, config)`
+   - `query(corpus, run, query_text, budget)`
+3. **Emit `Evidence`** with required fields:
+   - `item_id`, `source_uri`, `media_type`, `score`, `rank`, `stage`, `recipe_id`, `run_id`
+   - `text` **or** `content_ref`
+4. **Register the backend** in `biblicus.backends.available_backends`.
+5. **Add behavior-driven development specifications** before implementation and make them pass with 100% coverage.
+
+## Design notes
+
+- Treat **runs** as immutable manifests with reproducible parameters.
+- If your backend needs artifacts, store them under `.biblicus/runs/` and record paths in `artifact_paths`.
+- Keep **text extraction** in explicit pipeline stages, not in backend ingestion.
+
+## Examples
+
+See:
+
+- `biblicus.backends.scan.ScanBackend` (minimal baseline)
+- `biblicus.backends.sqlite_full_text_search.SqliteFullTextSearchBackend` (practical local backend)

biblicus-0.1.1/docs/api.rst

@@ -0,0 +1,32 @@
+Application Programming Interface Reference
+===========================================
+
+Core
+----
+
+.. automodule:: biblicus.corpus
+   :members:
+   :undoc-members:
+
+.. automodule:: biblicus.models
+   :members:
+   :undoc-members:
+
+.. automodule:: biblicus.retrieval
+   :members:
+   :undoc-members:
+
+.. automodule:: biblicus.evaluation
+   :members:
+   :undoc-members:
+
+Backends
+--------
+
+.. automodule:: biblicus.backends.scan
+   :members:
+   :undoc-members:
+
+.. automodule:: biblicus.backends.sqlite_full_text_search
+   :members:
+   :undoc-members: