biblicus 0.15.1__tar.gz → 0.16.0__tar.gz

{biblicus-0.15.1/src/biblicus.egg-info → biblicus-0.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: biblicus
-Version: 0.15.1
+Version: 0.16.0
 Summary: Command line interface and Python library for corpus ingestion, retrieval, and evaluation.
 License: MIT
 Requires-Python: >=3.9
@@ -11,6 +11,7 @@ Requires-Dist: PyYAML>=6.0
 Requires-Dist: pypdf>=4.0
 Requires-Dist: Jinja2>=3.1
 Requires-Dist: dotyaml>=0.1.3
+Requires-Dist: numpy>=1.24
 Provides-Extra: dev
 Requires-Dist: behave>=1.2.6; extra == "dev"
 Requires-Dist: coverage[toml]>=7.0; extra == "dev"

{biblicus-0.15.1 → biblicus-0.16.0}/docs/BACKENDS.md

@@ -61,4 +61,4 @@ See:
 
 - `biblicus.backends.scan.ScanBackend` (minimal baseline)
 - `biblicus.backends.sqlite_full_text_search.SqliteFullTextSearchBackend` (practical local backend)
-- `biblicus.backends.vector.VectorBackend` (term-frequency vector baseline)
+- `biblicus.backends.vector.VectorBackend` (term-frequency vector baseline; `tf-vector`)

biblicus-0.16.0/docs/CHUNKING.md

@@ -0,0 +1,69 @@
+# Chunking
+
+Embedding retrieval depends on chunking.
+
+Most corpora contain documents that are too long to retrieve effectively as single “whole-document” units. Biblicus
+therefore treats chunking as part of the retrieval indexing contract: embeddings are computed over chunks, and retrieval
+returns evidence with chunk boundaries so you can trace results back to the original item text.
+
+## Chunkers are pluggable
+
+Chunking is a pluggable interface selected by identifier in a retrieval recipe:
+
+- `chunker_id`
+- `chunker_config` (Pydantic validated; `extra="forbid"`)
+
+There are no hidden fallbacks. If you select a chunker without providing required configuration, Biblicus fails with a
+user-facing error that explains what is missing.
+
+## Chunk identifiers and provenance
+
+Chunks must be addressable and reproducible:
+
+- Each chunk has a stable `chunk_id`.
+- Each chunk references a parent `item_id`.
+- Each chunk records boundaries (for example `start_char` and `end_char`) and optional metadata.
+
+Evidence produced by embedding retrieval references chunk provenance so downstream tooling can reconstruct context packs
+without guessing.
+
+## Built-in chunking strategies
+
+Biblicus provides multiple built-in chunking strategies so you can compare tradeoffs explicitly:
+
+### Fixed character window
+
+Split text into windows of a fixed character length with optional overlap.
+
+Typical parameters:
+
+- `window_characters`
+- `overlap_characters`
+
+### Paragraph chunking
+
+Split text into paragraphs (blank-line delimited), with optional joining/splitting rules to avoid chunks that are too
+small or too large.
+
+Typical parameters:
+
+- `max_characters`
+- `join_short_paragraphs`
+
+### Fixed token window
+
+Split text into windows of a fixed token length with optional overlap.
+
+Token-based chunking depends on a tokenizer interface (see next section) so Biblicus can support multiple tokenization
+strategies without locking into a single library.
+
+## Tokenization is pluggable too
+
+Token-based chunkers depend on a second pluggable interface selected by identifier:
+
+- `tokenizer_id`
+- `tokenizer_config`
+
+This keeps the surface configurable while avoiding implicit dependencies. If a token-based chunker is selected without
+a tokenizer implementation configured, Biblicus fails fast with explicit guidance.
+

biblicus-0.16.0/docs/EMBEDDING_RETRIEVAL.md

@@ -0,0 +1,57 @@
+# Embedding Retrieval
+
+Embedding retrieval turns a large collection of text into a reusable index that can be queried efficiently.
+
+Biblicus supports embedding retrieval through backends that:
+
+1) chunk extracted text,
+2) compute embeddings for each chunk,
+3) build an index under the corpus as run artifacts, and
+4) return evidence with chunk provenance on query.
+
+## Concepts
+
+- **Chunking**: the unit of embedding and retrieval. See `docs/CHUNKING.md`.
+- **Embedding provider**: a pluggable implementation that turns text into vectors.
+- **Embedding index backend**: a retrieval backend that materializes vectors and supports similarity search.
+
+## A local, textbook embedding index
+
+Biblicus provides two embedding index backends that avoid external services:
+
+- `embedding-index-inmemory` for small demos with safety caps
+- `embedding-index-file` for a file-backed, memory-mapped exact index
+
+Both use exact cosine similarity. This is intentionally easy to validate and compare.
+
+## Build and query
+
+Embedding retrieval is a run-based workflow:
+
+1) ingest items
+2) extract text (or select an existing extraction run)
+3) build an embedding retrieval run (which materializes artifacts under the corpus)
+4) query the run and inspect evidence
+
+Example build:
+
+```
+python -m biblicus build --corpus corpora/example --backend embedding-index-file
+```
+
+Example query:
+
+```
+python -m biblicus query --corpus corpora/example --run embedding-index-file:RUN_ID --query "meaningful phrase"
+```
+
+## Evidence and provenance
+
+Evidence returned by embedding retrieval includes:
+
+- the parent `item_id` and `source_uri`
+- a retrieval `score` and `rank`
+- chunk provenance (boundaries and identifiers)
+
+This allows downstream tooling (including context pack formatting) to remain evidence-first and reproducible.
+

biblicus-0.16.0/docs/PR_FAQ_EMBEDDING_RETRIEVAL.md

@@ -0,0 +1,105 @@
+# PR-FAQ (Draft): Embedding Retrieval and Chunking
+
+## Press release
+
+Today we are adding true embedding-based retrieval to Biblicus, including a pluggable chunking surface and a
+textbook-simple local index.
+
+Biblicus already supports deterministic lexical retrieval and hybrid retrieval wiring. This feature adds retrieval
+backends that build a reusable embedding index under the corpus, so retrieval is fast, repeatable, and evaluatable.
+Chunking is treated as part of the indexing contract (not an afterthought): embeddings are computed over chunks, and
+retrieval returns evidence with item-level provenance plus chunk boundaries.
+
+## FAQ
+
+### What problem does this solve?
+
+- Provide a real embedding retrieval backend with an explicit build/query lifecycle.
+- Make chunking a first-class, fully configurable pipeline stage for embedding retrieval.
+- Establish a stable surface for swapping embedding providers without rewriting backends.
+
+### What will users be able to do?
+
+- Build a retrieval run that materializes an embedding index under a corpus.
+- Query that run and receive evidence-first outputs with stable identifiers and provenance.
+- Choose a chunking strategy (and its configuration) per recipe, and compare results across recipes.
+
+### Why do we need chunking now?
+
+Whole-document embeddings are misleading for most real corpora. Chunking is the practical unit of retrieval and must
+be part of the indexing contract so that:
+
+- embeddings are computed over well-defined text spans,
+- evidence can cite exact spans,
+- hybrid retrieval and evaluation can compare like-for-like.
+
+### How is chunking configured?
+
+Chunking is a pluggable interface selected by identifier in the retrieval recipe:
+
+- `chunker_id`
+- `chunker_config` (Pydantic validated; `extra="forbid"`)
+
+No fallbacks: if a recipe selects a chunker but its implementation or required configuration is missing, Biblicus
+fails fast with a user-facing error that explains exactly what to install/configure.
+
+### What chunkers are provided in the initial implementation?
+
+We provide multiple built-in chunkers from day 1, each with its own configuration model:
+
+- Fixed character window chunking (size + overlap).
+- Paragraph chunking (blank-line delimited, with configurable joining/splitting behavior).
+- Fixed token window chunking, via a tokenizer interface (see below).
+
+### How is token-based chunking handled without locking in one tokenizer library?
+
+Token-based chunking uses a separate pluggable interface (for example `Tokenizer` or `TokenCounter`), selected by id.
+
+This keeps the chunker configurable while avoiding hidden dependencies. If a token-based chunker is selected without a
+configured tokenizer implementation, Biblicus fails with explicit guidance.
+
+### What embedding index backends are included?
+
+Two concrete backends are introduced to avoid external vector stores while still being “real” retrievers:
+
+1) In-memory exact index
+- Intended for small corpora and demos.
+- Enforces a hard safety cap (for example maximum vectors or bytes).
+
+2) File-backed exact index (NumPy-backed)
+- Writes an embedding matrix and id mapping as run artifacts under the corpus.
+- Queries by memory-mapping and scanning in batches so memory usage is bounded.
+
+Both backends use exact cosine similarity. This is intentionally “textbook” behavior that is easy to validate and
+compare. Approximate nearest neighbor indexes are explicitly out of scope for this slice.
+
+### How do embeddings get generated?
+
+Embeddings are generated by a pluggable `EmbeddingProvider` interface (an abstract base class), selected by Pydantic
+configuration in the recipe. Concrete implementations can wrap OpenAI, Bedrock, or other providers via Biblicus’s AI
+provider wiring.
+
+### How does this fit the “derived artifacts live under the corpus” rule?
+
+Embedding retrieval is implemented as a run + artifacts:
+
+- `build_run(...)` materializes the embedding index and records artifact paths in a run manifest.
+- `query(...)` loads artifacts from the run and returns evidence with stable chunk provenance.
+
+### How do we test this in CI?
+
+- Behavior specs cover all success and failure behaviors and enforce 100% coverage.
+- CI fetches and caches WikiText-2 raw (`wikitext-2-raw-v1`) to provide “real-ish” text at scale without committing the
+  dataset into the repository.
+
+### Dependencies
+
+- `numpy` is required for the file-backed index backend (pip-only, no system services).
+- `datasets` is used only for CI fixture fetching and is installed in CI.
+
+## Non-goals (this slice)
+
+- External vector stores (Qdrant, pgvector, Pinecone, etc.).
+- ANN indexing (FAISS/HNSW tuning).
+- Backwards-compatible aliasing for renamed backend ids.
+

{biblicus-0.15.1 → biblicus-0.16.0}/docs/RETRIEVAL.md

@@ -54,7 +54,8 @@ Start with the simplest backend that answers your question:
 
 - `scan` for tiny corpora or sanity checks.
 - `sqlite-full-text-search` for a practical lexical baseline.
-- `vector` when you want deterministic term-frequency similarity without external dependencies.
+- `tf-vector` when you want deterministic term-frequency similarity without external dependencies.
+- `embedding-index-file` when you want embedding retrieval with a local, file-backed index.
 
 You can compare them with the same dataset and budget using the retrieval evaluation workflow.
 

biblicus-0.16.0/docs/backends/embedding-index-file.md

@@ -0,0 +1,34 @@
+# Embedding index (file-backed)
+
+This backend builds an embedding index under a corpus and queries it using exact cosine similarity.
+
+It is intended for larger corpora where you want a local, pip-installable workflow that does not depend on an external
+vector database.
+
+## Backend ID
+
+`embedding-index-file`
+
+## What it builds
+
+This backend builds a retrieval run that materializes run artifacts under the corpus, for example:
+
+- an embedding matrix stored as a NumPy array on disk
+- an id mapping from chunk identifiers to embedding row offsets
+- chunk records (text + boundaries + provenance)
+
+Queries memory-map the embedding matrix and scan in batches so memory usage stays bounded, even when the index is larger
+than available RAM.
+
+## Chunking
+
+Embeddings are computed over chunks. Chunking is configured per recipe by selecting a chunker and its configuration.
+
+Chunking is part of the index contract: evidence references chunk boundaries so you can trace retrieval outputs back to
+the original item text.
+
+## Dependencies
+
+- Requires `numpy`.
+- Requires an embedding provider configuration.
+

biblicus-0.16.0/docs/backends/embedding-index-inmemory.md

@@ -0,0 +1,34 @@
+# Embedding index (in-memory)
+
+This backend builds an embedding index in memory and queries it using exact cosine similarity.
+
+It is intended for textbook demos and small corpora where you want a “real” embedding retrieval loop without running an
+external vector database.
+
+## Backend ID
+
+`embedding-index-inmemory`
+
+## What it builds
+
+This backend builds a retrieval run that materializes:
+
+- chunk records (text + boundaries + provenance)
+- embedding vectors for each chunk
+
+All of this lives in memory while the process is running. For safety, the backend enforces explicit caps so a build does
+not accidentally consume unbounded memory.
+
+## Chunking
+
+Embeddings are computed over chunks. Chunking is configured per recipe by selecting a chunker and its configuration.
+
+Chunking is part of the index contract: evidence references chunk boundaries so you can trace retrieval outputs back to
+the original item text.
+
+## Dependencies
+
+- Requires an embedding provider configuration.
+
+This backend does not require a database or server.
+

{biblicus-0.15.1 → biblicus-0.16.0}/docs/backends/index.md

@@ -8,7 +8,9 @@ Biblicus provides pluggable retrieval backends that implement different search a
 
 scan
 sqlite-full-text-search
-vector
+tf-vector
+embedding-index-inmemory
+embedding-index-file
 ```
 
 ## Available Backends
@@ -33,16 +35,36 @@ Production-ready full-text search using SQLite FTS5 with BM25 ranking.
 - **Index**: SQLite database with FTS5 virtual tables
 - **Speed**: Fast with persistent index
 
-### [vector](vector.md)
+### [tf-vector](tf-vector.md)
 
-Deterministic term-frequency vector retrieval with cosine similarity.
+Deterministic term-frequency vector retrieval (vector space model baseline) with cosine similarity.
 
-- **Backend ID**: `vector`
+- **Backend ID**: `tf-vector`
 - **Installation**: Included by default
 - **Best for**: Semantic-style baselines without embeddings
 - **Index**: None (scans and scores at query time)
 - **Speed**: Moderate for small corpora
 
+### [embedding-index-inmemory](embedding-index-inmemory.md)
+
+Embedding-based retrieval with an in-memory exact cosine similarity index.
+
+- **Backend ID**: `embedding-index-inmemory`
+- **Installation**: Requires `numpy` and an embedding provider configuration
+- **Best for**: Textbook demos and small corpora
+- **Index**: In-memory embedding matrix
+- **Speed**: Fast for small corpora; bounded by safety caps
+
+### [embedding-index-file](embedding-index-file.md)
+
+Embedding-based retrieval with a file-backed exact cosine similarity index.
+
+- **Backend ID**: `embedding-index-file`
+- **Installation**: Requires `numpy` and an embedding provider configuration
+- **Best for**: Larger corpora without running an external vector database
+- **Index**: Memory-mapped embedding matrix + id mapping under the corpus
+- **Speed**: Exact scan; bounded memory via batching
+
 ## Quick Start
 
 ### Installation
@@ -115,7 +137,9 @@ See `docs/RETRIEVAL_EVALUATION.md` for evaluation workflows and dataset formats.
 | Production applications | [sqlite-full-text-search](sqlite-full-text-search.md) | Fast queries with BM25 ranking |
 | Large corpora (>10,000 items) | [sqlite-full-text-search](sqlite-full-text-search.md) | Essential for performance |
 | Baseline comparisons | [scan](scan.md) | Simple reference implementation |
-| Term-frequency vector baseline | [vector](vector.md) | Deterministic cosine similarity |
+| Term-frequency vector baseline | [tf-vector](tf-vector.md) | Deterministic cosine similarity |
+| Embedding retrieval (in-memory) | [embedding-index-inmemory](embedding-index-inmemory.md) | Exact cosine similarity |
+| Embedding retrieval (file-backed) | [embedding-index-file](embedding-index-file.md) | Exact cosine similarity, memory-mapped |
 
 ## Reproducibility checklist
 

biblicus-0.16.0/docs/backends/tf-vector.md

@@ -0,0 +1,59 @@
+# TF Vector backend
+
+The TF Vector backend implements a deterministic vector space model baseline using term-frequency vectors and cosine
+similarity. It builds no persistent index and scores items at query time. This makes it useful as a lightweight
+“vector-style” baseline without dense embeddings or external services.
+
+## When to use it
+
+- You want a minimal baseline to compare against lexical search.
+- You want deterministic, inspectable similarity scoring.
+- You are teaching retrieval concepts and want a small, runnable backend.
+
+## Backend ID
+
+`tf-vector`
+
+## How it works
+
+1) Tokenize the query and each item into lowercase word tokens.
+2) Build term-frequency vectors.
+3) Compute cosine similarity between the query vector and each item vector.
+4) Return evidence ranked by similarity score.
+
+## Configuration
+
+The backend accepts these configuration fields:
+
+- `snippet_characters`: maximum characters to include in evidence snippets.
+- `extraction_run`: optional extraction run reference (`extractor_id:run_id`).
+
+Example recipe:
+
+```yaml
+snippet_characters: 320
+extraction_run: pipeline:RUN_ID
+```
+
+## Build a run
+
+```
+python -m biblicus build --corpus corpora/example --backend tf-vector --config extraction_run=pipeline:RUN_ID
+```
+
+This backend does not create artifacts beyond the run manifest.
+
+## Query a run
+
+```
+python -m biblicus query --corpus corpora/example --run tf-vector:RUN_ID --query "semantic match"
+```
+
+The evidence results include a `stage` value of `tf-vector` and similarity scores for each match.
+
+## What it is not
+
+- This backend does not compute dense embeddings.
+- It does not use approximate nearest neighbor indexing.
+- It does not depend on external services.
+

{biblicus-0.15.1 → biblicus-0.16.0}/docs/conf.py

@@ -25,7 +25,8 @@ extensions = [
 templates_path = ["_templates"]
 exclude_patterns = ["_build"]
 autodoc_typehints = "description"
-html_theme = "sphinx_rtd_theme"
+html_theme = "biblicus_rtd"
+html_theme_path = [Path(__file__).resolve().parent / "_themes"]
 
 html_theme_options = {
     "prev_next_buttons_location": "bottom",

{biblicus-0.15.1 → biblicus-0.16.0}/features/cli_parsing.feature

@@ -20,6 +20,32 @@ Feature: Command-line interface parsing
       | mode=fast |
     Then the parsed config value "mode" is string "fast"
 
+  Scenario: Config pairs parse integers
+    When I parse config pairs:
+      | pair        |
+      | max_docs=12 |
+    Then the parsed config value "max_docs" is int 12
+
+  Scenario: Config pairs parse JSON objects
+    When I parse config pairs:
+      | pair                      |
+      | vectorizer={"stop":["a"]} |
+    Then the parsed config value "vectorizer" has JSON key "stop" list item "a"
+
+  Scenario: Config pairs reject missing equals
+    When I attempt to parse config pairs:
+      | pair     |
+      | justkey |
+    Then a config parsing error is raised
+    And the config parsing error mentions "key=value"
+
+  Scenario: Config pairs reject invalid JSON
+    When I attempt to parse config pairs:
+      | pair         |
+      | config={bad |
+    Then a config parsing error is raised
+    And the config parsing error mentions "valid JSON"
+
   Scenario: Step specs reject empty strings
     When I attempt to parse an empty step spec
     Then a step spec parsing error is raised