biblicus 0.10.0__tar.gz → 0.12.0__tar.gz

{biblicus-0.10.0/src/biblicus.egg-info → biblicus-0.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: biblicus
-Version: 0.10.0
+Version: 0.12.0
 Summary: Command line interface and Python library for corpus ingestion, retrieval, and evaluation.
 License: MIT
 Requires-Python: >=3.9
@@ -493,6 +493,12 @@ Two backends are included.
 
 For detailed documentation including configuration options, performance characteristics, and usage examples, see the [Backend Reference][backend-reference].
 
+## Retrieval documentation
+
+For the retrieval pipeline overview and run artifacts, see `docs/RETRIEVAL.md`. For retrieval quality upgrades
+(tuned lexical baseline, reranking, hybrid retrieval), see `docs/RETRIEVAL_QUALITY.md`. For evaluation workflows
+and dataset formats, see `docs/RETRIEVAL_EVALUATION.md`.
+
 ## Extraction backends
 
 These extractors are built in. Optional ones require extra dependencies. See [text extraction documentation][text-extraction] for details.

{biblicus-0.10.0 → biblicus-0.12.0}/README.md

@@ -447,6 +447,12 @@ Two backends are included.
 
 For detailed documentation including configuration options, performance characteristics, and usage examples, see the [Backend Reference][backend-reference].
 
+## Retrieval documentation
+
+For the retrieval pipeline overview and run artifacts, see `docs/RETRIEVAL.md`. For retrieval quality upgrades
+(tuned lexical baseline, reranking, hybrid retrieval), see `docs/RETRIEVAL_QUALITY.md`. For evaluation workflows
+and dataset formats, see `docs/RETRIEVAL_EVALUATION.md`.
+
 ## Extraction backends
 
 These extractors are built in. Optional ones require extra dependencies. See [text extraction documentation][text-extraction] for details.

{biblicus-0.10.0 → biblicus-0.12.0}/docs/ARCHITECTURE.md

@@ -88,11 +88,11 @@ Evidence is the canonical output of retrieval. Required fields:
 ### Integration boundary
 
 - Biblicus can integrate with Tactus as a **Model Context Protocol toolset**, for example with tool names such as `knowledge_base_ingest`, `knowledge_base_query`, and `knowledge_base_stats`.
-- We will **not** add a knowledge base or retrieval augmented generation language primitive in version zero. Revisit only if we need semantics that tools cannot express cleanly, such as enforceable policy boundaries, runtime managed durability, caching hooks, or guaranteed instrumentation.
+- We do **not** add a knowledge base or retrieval augmented generation language primitive in version zero. Revisit only if we need semantics that tools cannot express cleanly, such as enforceable policy boundaries, runtime managed durability, caching hooks, or guaranteed instrumentation.
 
 ### Interface packaging
 
-- The knowledge base interface is a **small protocol and reference implementation**, including tool schemas and a reference Model Context Protocol server. We will not build a full managed service in version zero.
+- The knowledge base interface is a **small protocol and reference implementation**, including tool schemas and a reference Model Context Protocol server. We do not build a full managed service in version zero.
 
 ### Corpus identity and layout
 
@@ -143,7 +143,7 @@ The interface stays the same; topology is configuration.
 - When a backend produces persisted materializations, Biblicus treats them as **versioned build runs** identified by `run_id` (rather than overwriting in place by default).
 - Manifests exist even for just-in-time backends (materializations may be empty).
 - Full directed acyclic graph lineage is not included in version zero; revisit only if needed.
-- Future (optional): define **shared materialization formats** (canonical chunk and embedding stores) so multiple backends can reuse intermediates when it makes sense; keep it opt-in.
+- Optional: define **shared materialization formats** (canonical chunk and embedding stores) so multiple backends can reuse intermediates when it makes sense; keep it opt-in.
 
 ### Evaluation
 
@@ -156,7 +156,7 @@ The interface stays the same; topology is configuration.
 - The corpus catalog is **file-based** (committable, portable, backend-agnostic) so any backend/tool can consume it without requiring a database engine.
 - Canonical version zero format is a single JavaScript Object Notation file at `.biblicus/catalog.json`, written atomically (temporary file and rename) on updates.
 - The catalog includes `latest_run_id` and run manifests are stored at `.biblicus/runs/<run_id>.json`.
-- If this ever becomes a bottleneck at very large scales, we will **change the specification** (bump `schema_version`) rather than introduce multiple “supported” catalog storage modes.
+- If this becomes a bottleneck at very large scales, we **change the specification** (bump `schema_version`) rather than introduce multiple “supported” catalog storage modes.
 
 ## Near-term deliverables
 

{biblicus-0.10.0 → biblicus-0.12.0}/docs/CONTEXT_PACK.md

@@ -23,13 +23,49 @@ context_pack = build_context_pack(result, policy=policy)
 print(context_pack.text)
 ```
 
+## Policy surfaces
+
+Context pack policies make ordering and formatting explicit.
+
+### Ordering
+
+Use `ordering` to control how evidence blocks are arranged before joining:
+
+- `rank`: use the evidence rank as provided by retrieval.
+- `score`: sort by score (descending) and then item identifier.
+- `source`: group by source uniform resource identifier, then sort by score.
+
+### Metadata inclusion
+
+Set `include_metadata=True` to prepend metadata to each block. Metadata includes:
+
+- `item_id`
+- `source_uri`
+- `score`
+- `stage`
+
+### Character budgets
+
+Character budgets drop trailing blocks until the context pack fits the specified limit. This keeps context shaping
+deterministic without relying on a tokenizer.
+
+In Python:
+
+```python
+from biblicus.context import CharacterBudget, ContextPackPolicy, fit_context_pack_to_character_budget
+
+policy = ContextPackPolicy(join_with="\n\n", ordering="score", include_metadata=True)
+fitted = fit_context_pack_to_character_budget(context_pack, policy=policy, character_budget=CharacterBudget(max_characters=500))
+print(fitted.text)
+```
+
 ## Command-line interface
 
 The command-line interface can build a context pack from a retrieval result by reading JavaScript Object Notation from standard input.
 
 ```bash
 biblicus query --corpus corpora/example --query "primary button style preference" \\
-  | biblicus context-pack build
+  | biblicus context-pack build --ordering score --include-metadata --max-characters 500
 ```
 
 ## What context pack building does

{biblicus-0.10.0 → biblicus-0.12.0}/docs/CORPUS_DESIGN.md

@@ -216,7 +216,7 @@ Version zero locked this as policy. A prune workflow was not implemented yet.
 
 Goal: retain derived artifacts from multiple implementations side by side so a user can compare results and switch between implementations without losing work.
 
-This decision applies to extraction plugins and retrieval backends, and to any future plugin type that produces derived artifacts.
+This decision applies to extraction plugins and retrieval backends, and to any plugin type that produces derived artifacts.
 
 Option A: store artifacts under the corpus, partitioned by plugin type
 
@@ -369,7 +369,7 @@ Version zero implemented option A by writing structured log entries for hook exe
 
 ## Outcomes and remaining questions
 
-The hook protocol and hook logging policy above were implemented in version zero. This section records what was implemented, plus the questions that remain for future iterations.
+The hook protocol and hook logging policy above were implemented in version zero. This section records what was implemented and the open questions tracked for later iterations.
 
 ### Hook contexts implemented in version zero
 

{biblicus-0.10.0 → biblicus-0.12.0}/docs/DEMOS.md

@@ -6,7 +6,7 @@ For the ordered plan of what to build next, see `docs/ROADMAP.md`.
 
 ## Diagram of the current system and the next layers
 
-Blue boxes are implemented now. Purple boxes are planned next layers that we can build and compare.
+Blue boxes are implemented now. Purple boxes are layers not implemented yet that we can build and compare.
 
 ```mermaid
 %%{init: {"flowchart": {"useMaxWidth": true, "nodeSpacing": 18, "rankSpacing": 22}}}%%
@@ -233,7 +233,7 @@ python3 -m biblicus extract build --corpus corpora/demo \\
   --step select-text
 ```
 
-Copy the `run_id` from the JavaScript Object Notation output. You will use it as `EXTRACTION_RUN_ID` in the next command.
+Copy the `run_id` from the JavaScript Object Notation output. Use it as `EXTRACTION_RUN_ID` in the next command.
 
 ```
 python3 -m biblicus build --corpus corpora/demo --backend sqlite-full-text-search \\
@@ -251,7 +251,7 @@ python3 scripts/download_pdf_samples.py --corpus corpora/pdf_samples --force
 python3 -m biblicus extract build --corpus corpora/pdf_samples --step pdf-text
 ```
 
-Copy the `run_id` from the JavaScript Object Notation output. You will use it as `PDF_EXTRACTION_RUN_ID` in the next command.
+Copy the `run_id` from the JavaScript Object Notation output. Use it as `PDF_EXTRACTION_RUN_ID` in the next command.
 
 ```
 python3 -m biblicus build --corpus corpora/pdf_samples --backend sqlite-full-text-search --config extraction_run=pipeline:PDF_EXTRACTION_RUN_ID --config chunk_size=200 --config chunk_overlap=50 --config snippet_characters=120

{biblicus-0.10.0 → biblicus-0.12.0}/docs/FEATURE_INDEX.md

@@ -204,6 +204,7 @@ Documentation:
 Behavior specifications:
 
 - `features/context_pack.feature`
+- `features/context_pack_policies.feature`
 - `features/token_budget.feature`
 
 Primary implementation:

biblicus-0.12.0/docs/RETRIEVAL.md

@@ -0,0 +1,47 @@
+# Retrieval
+
+Biblicus treats retrieval as a reproducible, explicit pipeline stage that transforms a corpus into structured evidence.
+Retrieval is separated from extraction and context shaping so each can be evaluated independently and swapped without
+rewriting ingestion.
+
+## Retrieval concepts
+
+- **Backend**: a pluggable retrieval implementation that can build and query runs.
+- **Run**: a recorded retrieval build for a corpus and extraction run.
+- **Evidence**: structured output containing identifiers, provenance, and scores.
+- **Stage**: explicit steps such as retrieve, rerank, and filter.
+
+## How retrieval runs work
+
+1) Ingest raw items into a corpus.
+2) Build an extraction run to produce text artifacts.
+3) Build a retrieval run with a backend, referencing the extraction run.
+4) Query the run to return evidence.
+
+Retrieval runs are stored under:
+
+```
+.biblicus/runs/retrieval/<backend_id>/<run_id>/
+```
+
+## Backends
+
+See `docs/backends/index.md` for backend selection and configuration.
+
+## Evaluation
+
+Retrieval runs are evaluated against datasets with explicit budgets. See `docs/RETRIEVAL_EVALUATION.md` for the
+dataset format and workflow, `docs/FEATURE_INDEX.md` for the behavior specifications, and `docs/CONTEXT_PACK.md` for
+how evidence feeds into context packs.
+
+## Why the separation matters
+
+Keeping extraction and retrieval distinct makes it possible to:
+
+- Reuse the same extracted artifacts across many retrieval backends.
+- Compare backends against the same corpus and dataset inputs.
+- Record and audit retrieval decisions without mixing in prompting or context formatting.
+
+## Retrieval quality
+
+For retrieval quality upgrades, see `docs/RETRIEVAL_QUALITY.md`.

biblicus-0.12.0/docs/RETRIEVAL_EVALUATION.md

@@ -0,0 +1,74 @@
+# Retrieval evaluation
+
+Biblicus evaluates retrieval runs against deterministic datasets so quality comparisons are repeatable across backends
+and corpora. Evaluations keep the evidence-first model intact by reporting per-query evidence alongside summary
+metrics.
+
+## Dataset format
+
+Retrieval datasets are stored as JavaScript Object Notation files with a strict schema:
+
+```json
+{
+  "schema_version": 1,
+  "name": "example-dataset",
+  "description": "Small hand-labeled dataset for smoke tests.",
+  "queries": [
+    {
+      "query_id": "q-001",
+      "query_text": "alpha",
+      "expected_item_id": "item-id-123",
+      "kind": "gold"
+    }
+  ]
+}
+```
+
+Each query includes either an `expected_item_id` or an `expected_source_uri`. The `kind` field records whether the
+query is hand-labeled (`gold`) or synthetic.
+
+## Running an evaluation
+
+Use the command-line interface to evaluate a retrieval run against a dataset:
+
+```bash
+biblicus eval --corpus corpora/example --run <run_id> --dataset datasets/retrieval.json \
+  --max-total-items 5 --max-total-characters 2000 --max-items-per-source 5
+```
+
+If `--run` is omitted, the latest retrieval run is used. Evaluations are deterministic for the same corpus, run, and
+budget.
+
+## Output
+
+The evaluation output includes:
+
+- Dataset metadata (name, description, query count).
+- Run metadata (backend ID, run ID, evaluation timestamp).
+- Metrics (hit rate, precision-at-k, mean reciprocal rank).
+- System diagnostics (latency percentiles and index size).
+
+The output is JavaScript Object Notation suitable for downstream reporting.
+
+## Python usage
+
+```python
+from pathlib import Path
+
+from biblicus.corpus import Corpus
+from biblicus.evaluation import evaluate_run, load_dataset
+from biblicus.models import QueryBudget
+
+corpus = Corpus.open("corpora/example")
+run = corpus.load_run("<run_id>")
+dataset = load_dataset(Path("datasets/retrieval.json"))
+budget = QueryBudget(max_total_items=5, max_total_characters=2000, max_items_per_source=5)
+result = evaluate_run(corpus=corpus, run=run, dataset=dataset, budget=budget)
+print(result.model_dump_json(indent=2))
+```
+
+## Design notes
+
+- Evaluation is reproducible by construction: the run manifest, dataset, and budget fully determine the results.
+- The evaluation workflow expects retrieval stages to remain explicit in the run artifacts.
+- Reports are portable, so comparisons across backends and corpora are straightforward.

biblicus-0.12.0/docs/RETRIEVAL_QUALITY.md

@@ -0,0 +1,42 @@
+# Retrieval quality upgrades
+
+This document describes the retrieval quality upgrades available in Biblicus. It is a reference for how retrieval
+quality is expressed in runs and how to interpret the signals in artifacts and evidence.
+
+## Goals
+
+- Improve relevance without losing determinism or reproducibility.
+- Keep retrieval stages explicit and visible in run artifacts.
+- Preserve the evidence-first output model.
+
+## Available upgrades
+
+### 1) Tuned lexical baseline
+
+- BM25-style scoring with configurable parameters.
+- N-gram range controls.
+- Stop word strategy per backend.
+- Field weighting (for example: title, body, metadata).
+
+### 2) Reranking stage
+
+- Optional rerank step that re-scores top-N candidates.
+- Deterministic scoring keeps rerank behavior reproducible.
+
+### 3) Hybrid retrieval
+
+- Combine lexical and embedding signals.
+- Expose fusion weights in the recipe schema.
+- Emit stage-level scores and weights in evidence metadata.
+
+## Evaluation guidance
+
+- Measure accuracy-at-k and compare against the same datasets.
+- Run artifacts capture each stage and configuration for auditability.
+- Deterministic settings remain available as the default baseline.
+
+## Non-goals
+
+- Automated hyperparameter tuning.
+- Hidden fallback stages that obscure retrieval behavior.
+- UI-driven tuning in this phase.

{biblicus-0.10.0 → biblicus-0.12.0}/docs/ROADMAP.md

@@ -17,34 +17,27 @@ If you are looking for what already exists, start with:
 - Raw corpus items remain readable, portable files.
 - Derived artifacts are stored under the corpus and can coexist for multiple implementations.
 
-## Next: retrieval evaluation and datasets
+## Completed foundations
 
-Goal: make evaluation results easier to interpret and compare.
+These are the capability slices that already exist and have end-to-end behavior specifications.
 
-Deliverables:
-
-- A dataset authoring workflow that supports small hand-labeled sets and larger synthetic sets.
-- A report that includes per-query diagnostics and a clear summary.
-
-Acceptance checks:
+### Retrieval evaluation and datasets
 
-- Dataset formats are versioned when they change.
-- Reports remain deterministic for the same inputs.
+- Dataset authoring workflow for small hand-labeled sets and larger synthetic sets.
+- Evaluation reports with per-query diagnostics and summary metrics.
+- Versioned dataset formats and deterministic reports for stable inputs.
 
-## Next: context pack policy surfaces
+### Retrieval quality upgrades
 
-Goal: make context shaping policies easier to evaluate and swap.
+- Tuned lexical baseline with BM25, n-gram range controls, and stop word policies.
+- Reranking stage for top-N candidates with explicit stage metadata.
+- Hybrid retrieval with explicit fusion weights and stage-level scores.
 
-Deliverables:
-
-- A clear set of context pack policy variants (formatting, ordering, metadata inclusion).
-- Token budget strategies that can use a real tokenizer.
-- Documentation that explains where context shaping fits in the pipeline.
-
-Acceptance checks:
+### Context pack policy surfaces
 
-- Behavior specifications cover policy selection and budgeting behaviors.
-- Example outputs show how context packs differ across policies.
+- Policy variants for formatting, ordering, and metadata inclusion.
+- Token and character budget strategies with explicit selectors.
+- Documentation and examples that show how policy choices change outputs.
 
 ## Next: extraction evaluation harness
 
@@ -67,7 +60,7 @@ Goal: provide lightweight analysis utilities that summarize corpus themes and gu
 
 Deliverables:
 
-- Basic data profiling reports (counts, media types, size distributions, tag coverage).
+- Basic corpus profiling with deterministic metrics for raw items and extracted text.
 - Hidden Markov modeling analysis for sequence-driven corpora.
 - A way to compare analysis outputs across corpora or corpus snapshots.
 

{biblicus-0.10.0 → biblicus-0.12.0}/docs/extractors/text-document/pass-through.md

@@ -120,12 +120,12 @@ title: My Document
 tags: [note, draft]
 ---
 
-This is the body content that will be extracted.
+This is the body content that is extracted.
 ```
 
 Output text:
 ```
-This is the body content that will be extracted.
+This is the body content that is extracted.
 ```
 
 ### Mixed Format Pipeline
@@ -185,7 +185,7 @@ Non-text items are silently skipped (returns `None`). This allows the extractor
 
 ### Encoding Errors
 
-UTF-8 decoding errors will cause per-item failures recorded in `errored_items` but won't halt the entire extraction run.
+UTF-8 decoding errors cause per-item failures recorded in `errored_items` but do not halt the entire extraction run.
 
 ### Missing Files
 

{biblicus-0.10.0 → biblicus-0.12.0}/docs/extractors/text-document/unstructured.md

@@ -78,7 +78,7 @@ class UnstructuredExtractorConfig(BaseModel):
 
 ### Configuration Options
 
-This extractor currently accepts no configuration. Future versions may expose Unstructured library options.
+This extractor currently accepts no configuration. Optional extensions may expose Unstructured library options.
 
 ## Usage
 

{biblicus-0.10.0 → biblicus-0.12.0}/docs/index.rst

@@ -15,6 +15,9 @@ Contents
    KNOWLEDGE_BASE
    BACKENDS
    backends/index
+   RETRIEVAL
+   RETRIEVAL_QUALITY
+   RETRIEVAL_EVALUATION
    CONTEXT_PACK
    ANALYSIS
    PROFILING

{biblicus-0.10.0 → biblicus-0.12.0}/features/context_pack_cli.feature

@@ -23,6 +23,31 @@ Feature: Context pack command-line interface
       one two three
       """
 
+  Scenario: Context pack build can include metadata
+    Given a retrieval result exists with sourced evidence:
+      | source_uri | score | text  |
+      | source-a   | 10.0  | alpha |
+    When I run "context-pack build" joining with "\n\n" ordering "score" and including metadata
+    Then the context pack build output text equals:
+      """
+      item_id: item-1
+      source_uri: source-a
+      score: 10.0
+      stage: scan
+      alpha
+      """
+
+  Scenario: Context pack build can fit to a character budget
+    Given a retrieval result exists with evidence text:
+      | text |
+      | alpha |
+      | beta |
+    When I run "context-pack build" joining with "\n\n" and character budget 6
+    Then the context pack build output text equals:
+      """
+      alpha
+      """
+
   Scenario: Context pack build fails without retrieval result on standard input
     When I run "context-pack build" with empty standard input
     Then the command fails with exit code 2

biblicus-0.12.0/features/context_pack_policies.feature

@@ -0,0 +1,92 @@
+Feature: Context pack policies
+  Context pack policies control evidence ordering, metadata inclusion, and budgets.
+
+  Scenario: Score ordering sorts evidence by score
+    Given a retrieval result exists with scored evidence:
+      | score | text  |
+      | 1.0   | beta  |
+      | 5.0   | alpha |
+    When I build a context pack from that retrieval result with policy:
+      | key              | value |
+      | join_with        | \n\n |
+      | ordering         | score |
+      | include_metadata | false |
+    Then the context pack text equals:
+      """
+      alpha
+
+      beta
+      """
+
+  Scenario: Source ordering groups evidence by source
+    Given a retrieval result exists with sourced evidence:
+      | source_uri | score | text  |
+      | source-b   | 1.0   | beta  |
+      | source-a   | 2.0   | alpha |
+      | source-a   | 1.0   | delta |
+    When I build a context pack from that retrieval result with policy:
+      | key              | value |
+      | join_with        | \n\n |
+      | ordering         | source |
+      | include_metadata | false |
+    Then the context pack text equals:
+      """
+      alpha
+
+      delta
+
+      beta
+      """
+
+  Scenario: Metadata inclusion prepends block metadata
+    Given a retrieval result exists with sourced evidence:
+      | source_uri | score | text  |
+      | source-a   | 10.0  | alpha |
+    When I build a context pack from that retrieval result with policy:
+      | key              | value |
+      | join_with        | \n\n |
+      | ordering         | rank |
+      | include_metadata | true |
+    Then the context pack text equals:
+      """
+      item_id: item-1
+      source_uri: source-a
+      score: 10.0
+      stage: scan
+      alpha
+      """
+
+  Scenario: Character budgets drop trailing blocks
+    Given a retrieval result exists with evidence text:
+      | text |
+      | alpha |
+      | beta |
+    When I build a context pack from that retrieval result with policy:
+      | key              | value |
+      | join_with        | \n\n |
+      | ordering         | rank |
+      | include_metadata | false |
+    And I fit the context pack to a character budget of 6 characters
+    Then the context pack text equals:
+      """
+      alpha
+      """
+
+  Scenario: Character budgets can produce empty context packs
+    Given a retrieval result exists with evidence text:
+      | text |
+      | alpha |
+    When I build a context pack from that retrieval result with policy:
+      | key              | value |
+      | join_with        | \n\n |
+      | ordering         | rank |
+      | include_metadata | false |
+    And I fit the context pack to a character budget of 1 characters
+    Then the context pack text is empty
+
+  Scenario: Unknown ordering raises a policy error
+    Given a retrieval result exists with evidence text:
+      | text |
+      | alpha |
+    When I attempt to build a context pack with invalid ordering "mystery"
+    Then the context pack ordering error mentions "Unknown context pack ordering"