biblicus 0.13.0__tar.gz → 0.15.0__tar.gz

{biblicus-0.13.0/src/biblicus.egg-info → biblicus-0.15.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: biblicus
-Version: 0.13.0
+Version: 0.15.0
 Summary: Command line interface and Python library for corpus ingestion, retrieval, and evaluation.
 License: MIT
 Requires-Python: >=3.9
@@ -9,6 +9,8 @@ License-File: LICENSE
 Requires-Dist: pydantic>=2.0
 Requires-Dist: PyYAML>=6.0
 Requires-Dist: pypdf>=4.0
+Requires-Dist: Jinja2>=3.1
+Requires-Dist: dotyaml>=0.1.3
 Provides-Extra: dev
 Requires-Dist: behave>=1.2.6; extra == "dev"
 Requires-Dist: coverage[toml]>=7.0; extra == "dev"
@@ -18,6 +20,9 @@ Requires-Dist: sphinx_rtd_theme>=2.0; extra == "dev"
 Requires-Dist: ruff>=0.4.0; extra == "dev"
 Requires-Dist: black>=24.0; extra == "dev"
 Requires-Dist: python-semantic-release>=9.0.0; extra == "dev"
+Provides-Extra: dspy
+Requires-Dist: dspy>=2.5; extra == "dspy"
+Requires-Dist: litellm>=1.0; extra == "dspy"
 Provides-Extra: openai
 Requires-Dist: openai>=1.0; extra == "openai"
 Provides-Extra: unstructured
@@ -40,6 +45,8 @@ Provides-Extra: docling-mlx
 Requires-Dist: docling[mlx-vlm]>=2.0.0; extra == "docling-mlx"
 Provides-Extra: topic-modeling
 Requires-Dist: bertopic>=0.15.0; extra == "topic-modeling"
+Provides-Extra: markov-analysis
+Requires-Dist: hmmlearn>=0.3.0; extra == "markov-analysis"
 Provides-Extra: datasets
 Requires-Dist: datasets>=2.18.0; extra == "datasets"
 Dynamic: license-file
@@ -56,12 +63,20 @@ If you are building an assistant in Python, you probably have material you want
 
 The first practical problem is not retrieval. It is collection and care. You need a stable place to put raw items, you need a small amount of metadata so you can find them again, and you need a way to evolve your retrieval approach over time without rewriting ingestion.
 
-This library gives you a corpus, which is a normal folder on disk. It stores each ingested item as a file, with optional metadata stored next to it. You can open and inspect the raw files directly. Any derived catalog or index can be rebuilt from the raw corpus.
+Biblicus gives you a normal folder on disk to manage. In Biblicus documentation, that managed folder is called a *corpus* (plural: *corpora*). It stores each ingested item as a file, with optional metadata stored next to it. You can open and inspect the raw files directly. Any derived catalog or index can be rebuilt from the raw files.
 
 It can be used alongside LangGraph, Tactus, Pydantic AI, any agent framework, or your own setup. Use it from Python or from the command line interface.
 
 See [retrieval augmented generation overview] for a short introduction to the idea.
 
+## Analysis highlights
+
+- `biblicus analyze markov` learns a directed, weighted state transition graph over segmented text.
+- YAML recipes support cascading composition plus dotted `--config key=value` overrides.
+- Text extract splits long texts with an LLM by inserting XML tags in-place for structured spans.
+- See `docs/MARKOV_ANALYSIS.md` for Markov analysis details and runnable demos.
+- See `docs/TEXT_EXTRACT.md` for the text extract utility and examples.
+
 ## Start with a knowledge base
 
 If you just want to hand a folder to your assistant and move on, use the high-level knowledge base interface. The folder can be nothing more than a handful of plain text files. You are not choosing a retrieval strategy yet. You are just collecting.
@@ -106,7 +121,7 @@ Think in three stages.
 
 If you learn a few project words, the rest of the system becomes predictable.
 
-- Corpus is the folder that holds raw items and their metadata.
+- Corpus is the managed folder that holds raw items and their metadata.
 - Item is the raw bytes plus optional metadata and source information.
 - Catalog is the rebuildable index of the corpus.
 - Extraction run is a recorded extraction build that produces text artifacts.
@@ -161,28 +176,28 @@ sequenceDiagram
 This repository is a working Python package. Install it into a virtual environment from the repository root.
 
 ```
-python3 -m pip install -e .
+python -m pip install -e .
 ```
 
 After the first release, you can install it from Python Package Index.
 
 ```
-python3 -m pip install biblicus
+python -m pip install biblicus
 ```
 
 ### Optional extras
 
 Some extractors are optional so the base install stays small.
 
-- Optical character recognition for images: `python3 -m pip install "biblicus[ocr]"`
-- Advanced optical character recognition with PaddleOCR: `python3 -m pip install "biblicus[paddleocr]"`
-- Document understanding with Docling VLM: `python3 -m pip install "biblicus[docling]"`
-- Document understanding with Docling VLM and MLX acceleration: `python3 -m pip install "biblicus[docling-mlx]"`
-- Speech to text transcription with OpenAI: `python3 -m pip install "biblicus[openai]"` (requires an OpenAI API key in `~/.biblicus/config.yml` or `./.biblicus/config.yml`)
-- Speech to text transcription with Deepgram: `python3 -m pip install "biblicus[deepgram]"` (requires a Deepgram API key in `~/.biblicus/config.yml` or `./.biblicus/config.yml`)
-- Broad document parsing fallback: `python3 -m pip install "biblicus[unstructured]"`
-- MarkItDown document conversion (requires Python 3.10 or higher): `python3 -m pip install "biblicus[markitdown]"`
-- Topic modeling analysis with BERTopic: `python3 -m pip install "biblicus[topic-modeling]"`
+- Optical character recognition for images: `python -m pip install "biblicus[ocr]"`
+- Advanced optical character recognition with PaddleOCR: `python -m pip install "biblicus[paddleocr]"`
+- Document understanding with Docling VLM: `python -m pip install "biblicus[docling]"`
+- Document understanding with Docling VLM and MLX acceleration: `python -m pip install "biblicus[docling-mlx]"`
+- Speech to text transcription with OpenAI: `python -m pip install "biblicus[openai]"` (requires an OpenAI API key in `~/.biblicus/config.yml` or `./.biblicus/config.yml`)
+- Speech to text transcription with Deepgram: `python -m pip install "biblicus[deepgram]"` (requires a Deepgram API key in `~/.biblicus/config.yml` or `./.biblicus/config.yml`)
+- Broad document parsing fallback: `python -m pip install "biblicus[unstructured]"`
+- MarkItDown document conversion (requires Python 3.10 or higher): `python -m pip install "biblicus[markitdown]"`
+- Topic modeling analysis with BERTopic: `python -m pip install "biblicus[topic-modeling]"`
 
 ## Quick start
 
@@ -200,16 +215,49 @@ biblicus build --corpus corpora/example --backend scan
 biblicus query --corpus corpora/example --query "note"
 ```
 
-If you want to turn a website section into corpus items, crawl a root web address while restricting the crawl to an allowed prefix:
+## Web Ingestion
+
+Biblicus supports ingesting content directly from the web using two approaches.
+
+### Ingest from URLs
+
+Ingest individual documents or web pages from URLs. The `ingest` command automatically detects content types including PDF, HTML, Markdown, images, and audio:
 
+```bash
+# Ingest a document from a URL
+biblicus ingest https://example.com/document.pdf --tags "research"
+
+# Ingest a web page
+biblicus ingest https://example.com/article.html --tags "article"
+
+# Ingest with a corpus path specified
+biblicus ingest --corpus corpora/example https://docs.example.com/guide.md --tags "documentation"
 ```
-biblicus crawl --corpus corpora/example \\
-  --root-url https://example.com/docs/index.html \\
-  --allowed-prefix https://example.com/docs/ \\
-  --max-items 50 \\
-  --tag crawled
+
+### Crawl Websites
+
+Crawl entire website sections with automatic link discovery. The crawler follows links within the allowed prefix and stores discovered content:
+
+```bash
+# Crawl a documentation site
+biblicus crawl \
+  --corpus corpora/example \
+  --root-url https://docs.example.com/ \
+  --allowed-prefix https://docs.example.com/ \
+  --max-items 100 \
+  --tags "documentation"
+
+# Crawl a specific blog category
+biblicus crawl \
+  --corpus corpora/example \
+  --root-url https://blog.example.com/category/tutorials/ \
+  --allowed-prefix https://blog.example.com/category/tutorials/ \
+  --max-items 50 \
+  --tags "tutorials,blog"
 ```
 
+The `--allowed-prefix` parameter restricts the crawler to only follow links that start with the specified URL prefix, preventing it from crawling outside the intended scope. The crawler respects `.biblicusignore` rules and stores items under `raw/imports/crawl/` in your corpus.
+
 ## End-to-end example: lower-level control
 
 The command-line interface returns JavaScript Object Notation by default. This makes it easy to use Biblicus in scripts and to treat retrieval as a deterministic, testable step.
@@ -498,7 +546,8 @@ For detailed documentation including configuration options, performance characte
 
 For the retrieval pipeline overview and run artifacts, see `docs/RETRIEVAL.md`. For retrieval quality upgrades
 (tuned lexical baseline, reranking, hybrid retrieval), see `docs/RETRIEVAL_QUALITY.md`. For evaluation workflows
-and dataset formats, see `docs/RETRIEVAL_EVALUATION.md`.
+and dataset formats, see `docs/RETRIEVAL_EVALUATION.md`. For a runnable walkthrough, use the retrieval evaluation lab
+script (`scripts/retrieval_evaluation_lab.py`).
 
 ## Extraction backends
 
@@ -539,6 +588,21 @@ For detailed documentation on all extractors, see the [Extractor Reference][extr
 For extraction evaluation workflows, dataset formats, and report interpretation, see
 `docs/EXTRACTION_EVALUATION.md`.
 
+## Text extract utility
+
+Text extract is a reusable analysis utility that lets a model insert XML tags into a long text without re-emitting the
+entire document. It returns structured spans and the marked-up text, and it is used as a segmentation option in Markov
+analysis.
+
+See `docs/TEXT_EXTRACT.md` for the utility API and examples, and `docs/MARKOV_ANALYSIS.md` for the Markov integration.
+
+## Text slice utility
+
+Text slice is a reusable analysis utility that lets a model insert `<slice/>` markers into a long text without
+re-emitting the entire document. It returns ordered slices and the marked-up text for auditing and reuse.
+
+See `docs/TEXT_SLICE.md` for the utility API and examples.
+
 ## Topic modeling analysis
 
 Biblicus can run analysis pipelines on extracted text without changing the raw corpus. Profiling and topic modeling
@@ -593,7 +657,7 @@ AG News integration runs require `biblicus[datasets]` in addition to `biblicus[t
 For a repeatable, real-world integration run that downloads AG News and executes topic modeling, use:
 
 ```
-python3 scripts/topic_modeling_integration.py --corpus corpora/ag_news_demo --force
+python scripts/topic_modeling_integration.py --corpus corpora/ag_news_demo --force
 ```
 
 See `docs/TOPIC_MODELING.md` for parameter examples and per-topic output behavior.
@@ -607,13 +671,13 @@ Use `scripts/download_pdf_samples.py` to download a small Portable Document Form
 ## Tests and coverage
 
 ```
-python3 scripts/test.py
+python scripts/test.py
 ```
 
 To include integration scenarios that download public test data at runtime, run this command.
 
 ```
-python3 scripts/test.py --integration
+python scripts/test.py --integration
 ```
 
 ## Releases
@@ -631,13 +695,13 @@ Reference documentation is generated from Sphinx style docstrings.
 Install development dependencies:
 
 ```
-python3 -m pip install -e ".[dev]"
+python -m pip install -e ".[dev]"
 ```
 
 Build the documentation:
 
 ```
-python3 -m sphinx -b html docs docs/_build/html
+python -m sphinx -b html docs docs/_build/html
 ```
 
 ## License

{biblicus-0.13.0 → biblicus-0.15.0}/README.md

@@ -10,12 +10,20 @@ If you are building an assistant in Python, you probably have material you want
 
 The first practical problem is not retrieval. It is collection and care. You need a stable place to put raw items, you need a small amount of metadata so you can find them again, and you need a way to evolve your retrieval approach over time without rewriting ingestion.
 
-This library gives you a corpus, which is a normal folder on disk. It stores each ingested item as a file, with optional metadata stored next to it. You can open and inspect the raw files directly. Any derived catalog or index can be rebuilt from the raw corpus.
+Biblicus gives you a normal folder on disk to manage. In Biblicus documentation, that managed folder is called a *corpus* (plural: *corpora*). It stores each ingested item as a file, with optional metadata stored next to it. You can open and inspect the raw files directly. Any derived catalog or index can be rebuilt from the raw files.
 
 It can be used alongside LangGraph, Tactus, Pydantic AI, any agent framework, or your own setup. Use it from Python or from the command line interface.
 
 See [retrieval augmented generation overview] for a short introduction to the idea.
 
+## Analysis highlights
+
+- `biblicus analyze markov` learns a directed, weighted state transition graph over segmented text.
+- YAML recipes support cascading composition plus dotted `--config key=value` overrides.
+- Text extract splits long texts with an LLM by inserting XML tags in-place for structured spans.
+- See `docs/MARKOV_ANALYSIS.md` for Markov analysis details and runnable demos.
+- See `docs/TEXT_EXTRACT.md` for the text extract utility and examples.
+
 ## Start with a knowledge base
 
 If you just want to hand a folder to your assistant and move on, use the high-level knowledge base interface. The folder can be nothing more than a handful of plain text files. You are not choosing a retrieval strategy yet. You are just collecting.
@@ -60,7 +68,7 @@ Think in three stages.
 
 If you learn a few project words, the rest of the system becomes predictable.
 
-- Corpus is the folder that holds raw items and their metadata.
+- Corpus is the managed folder that holds raw items and their metadata.
 - Item is the raw bytes plus optional metadata and source information.
 - Catalog is the rebuildable index of the corpus.
 - Extraction run is a recorded extraction build that produces text artifacts.
@@ -115,28 +123,28 @@ sequenceDiagram
 This repository is a working Python package. Install it into a virtual environment from the repository root.
 
 ```
-python3 -m pip install -e .
+python -m pip install -e .
 ```
 
 After the first release, you can install it from Python Package Index.
 
 ```
-python3 -m pip install biblicus
+python -m pip install biblicus
 ```
 
 ### Optional extras
 
 Some extractors are optional so the base install stays small.
 
-- Optical character recognition for images: `python3 -m pip install "biblicus[ocr]"`
-- Advanced optical character recognition with PaddleOCR: `python3 -m pip install "biblicus[paddleocr]"`
-- Document understanding with Docling VLM: `python3 -m pip install "biblicus[docling]"`
-- Document understanding with Docling VLM and MLX acceleration: `python3 -m pip install "biblicus[docling-mlx]"`
-- Speech to text transcription with OpenAI: `python3 -m pip install "biblicus[openai]"` (requires an OpenAI API key in `~/.biblicus/config.yml` or `./.biblicus/config.yml`)
-- Speech to text transcription with Deepgram: `python3 -m pip install "biblicus[deepgram]"` (requires a Deepgram API key in `~/.biblicus/config.yml` or `./.biblicus/config.yml`)
-- Broad document parsing fallback: `python3 -m pip install "biblicus[unstructured]"`
-- MarkItDown document conversion (requires Python 3.10 or higher): `python3 -m pip install "biblicus[markitdown]"`
-- Topic modeling analysis with BERTopic: `python3 -m pip install "biblicus[topic-modeling]"`
+- Optical character recognition for images: `python -m pip install "biblicus[ocr]"`
+- Advanced optical character recognition with PaddleOCR: `python -m pip install "biblicus[paddleocr]"`
+- Document understanding with Docling VLM: `python -m pip install "biblicus[docling]"`
+- Document understanding with Docling VLM and MLX acceleration: `python -m pip install "biblicus[docling-mlx]"`
+- Speech to text transcription with OpenAI: `python -m pip install "biblicus[openai]"` (requires an OpenAI API key in `~/.biblicus/config.yml` or `./.biblicus/config.yml`)
+- Speech to text transcription with Deepgram: `python -m pip install "biblicus[deepgram]"` (requires a Deepgram API key in `~/.biblicus/config.yml` or `./.biblicus/config.yml`)
+- Broad document parsing fallback: `python -m pip install "biblicus[unstructured]"`
+- MarkItDown document conversion (requires Python 3.10 or higher): `python -m pip install "biblicus[markitdown]"`
+- Topic modeling analysis with BERTopic: `python -m pip install "biblicus[topic-modeling]"`
 
 ## Quick start
 
@@ -154,16 +162,49 @@ biblicus build --corpus corpora/example --backend scan
 biblicus query --corpus corpora/example --query "note"
 ```
 
-If you want to turn a website section into corpus items, crawl a root web address while restricting the crawl to an allowed prefix:
+## Web Ingestion
+
+Biblicus supports ingesting content directly from the web using two approaches.
+
+### Ingest from URLs
+
+Ingest individual documents or web pages from URLs. The `ingest` command automatically detects content types including PDF, HTML, Markdown, images, and audio:
 
+```bash
+# Ingest a document from a URL
+biblicus ingest https://example.com/document.pdf --tags "research"
+
+# Ingest a web page
+biblicus ingest https://example.com/article.html --tags "article"
+
+# Ingest with a corpus path specified
+biblicus ingest --corpus corpora/example https://docs.example.com/guide.md --tags "documentation"
 ```
-biblicus crawl --corpus corpora/example \\
-  --root-url https://example.com/docs/index.html \\
-  --allowed-prefix https://example.com/docs/ \\
-  --max-items 50 \\
-  --tag crawled
+
+### Crawl Websites
+
+Crawl entire website sections with automatic link discovery. The crawler follows links within the allowed prefix and stores discovered content:
+
+```bash
+# Crawl a documentation site
+biblicus crawl \
+  --corpus corpora/example \
+  --root-url https://docs.example.com/ \
+  --allowed-prefix https://docs.example.com/ \
+  --max-items 100 \
+  --tags "documentation"
+
+# Crawl a specific blog category
+biblicus crawl \
+  --corpus corpora/example \
+  --root-url https://blog.example.com/category/tutorials/ \
+  --allowed-prefix https://blog.example.com/category/tutorials/ \
+  --max-items 50 \
+  --tags "tutorials,blog"
 ```
 
+The `--allowed-prefix` parameter restricts the crawler to only follow links that start with the specified URL prefix, preventing it from crawling outside the intended scope. The crawler respects `.biblicusignore` rules and stores items under `raw/imports/crawl/` in your corpus.
+
 ## End-to-end example: lower-level control
 
 The command-line interface returns JavaScript Object Notation by default. This makes it easy to use Biblicus in scripts and to treat retrieval as a deterministic, testable step.
@@ -452,7 +493,8 @@ For detailed documentation including configuration options, performance characte
 
 For the retrieval pipeline overview and run artifacts, see `docs/RETRIEVAL.md`. For retrieval quality upgrades
 (tuned lexical baseline, reranking, hybrid retrieval), see `docs/RETRIEVAL_QUALITY.md`. For evaluation workflows
-and dataset formats, see `docs/RETRIEVAL_EVALUATION.md`.
+and dataset formats, see `docs/RETRIEVAL_EVALUATION.md`. For a runnable walkthrough, use the retrieval evaluation lab
+script (`scripts/retrieval_evaluation_lab.py`).
 
 ## Extraction backends
 
@@ -493,6 +535,21 @@ For detailed documentation on all extractors, see the [Extractor Reference][extr
 For extraction evaluation workflows, dataset formats, and report interpretation, see
 `docs/EXTRACTION_EVALUATION.md`.
 
+## Text extract utility
+
+Text extract is a reusable analysis utility that lets a model insert XML tags into a long text without re-emitting the
+entire document. It returns structured spans and the marked-up text, and it is used as a segmentation option in Markov
+analysis.
+
+See `docs/TEXT_EXTRACT.md` for the utility API and examples, and `docs/MARKOV_ANALYSIS.md` for the Markov integration.
+
+## Text slice utility
+
+Text slice is a reusable analysis utility that lets a model insert `<slice/>` markers into a long text without
+re-emitting the entire document. It returns ordered slices and the marked-up text for auditing and reuse.
+
+See `docs/TEXT_SLICE.md` for the utility API and examples.
+
 ## Topic modeling analysis
 
 Biblicus can run analysis pipelines on extracted text without changing the raw corpus. Profiling and topic modeling
@@ -547,7 +604,7 @@ AG News integration runs require `biblicus[datasets]` in addition to `biblicus[t
 For a repeatable, real-world integration run that downloads AG News and executes topic modeling, use:
 
 ```
-python3 scripts/topic_modeling_integration.py --corpus corpora/ag_news_demo --force
+python scripts/topic_modeling_integration.py --corpus corpora/ag_news_demo --force
 ```
 
 See `docs/TOPIC_MODELING.md` for parameter examples and per-topic output behavior.
@@ -561,13 +618,13 @@ Use `scripts/download_pdf_samples.py` to download a small Portable Document Form
 ## Tests and coverage
 
 ```
-python3 scripts/test.py
+python scripts/test.py
 ```
 
 To include integration scenarios that download public test data at runtime, run this command.
 
 ```
-python3 scripts/test.py --integration
+python scripts/test.py --integration
 ```
 
 ## Releases
@@ -585,13 +642,13 @@ Reference documentation is generated from Sphinx style docstrings.
 Install development dependencies:
 
 ```
-python3 -m pip install -e ".[dev]"
+python -m pip install -e ".[dev]"
 ```
 
 Build the documentation:
 
 ```
-python3 -m sphinx -b html docs docs/_build/html
+python -m sphinx -b html docs docs/_build/html
 ```
 
 ## License

biblicus-0.15.0/datasets/retrieval_lab/labels.json

@@ -0,0 +1,25 @@
+{
+  "schema_version": 1,
+  "name": "retrieval-evaluation-lab",
+  "description": "Bundled labels for the retrieval evaluation lab.",
+  "queries": [
+    {
+      "query_id": "q1",
+      "query_text": "alpha unique",
+      "expected_filename": "alpha.txt",
+      "kind": "gold"
+    },
+    {
+      "query_id": "q2",
+      "query_text": "beta unique",
+      "expected_filename": "beta.txt",
+      "kind": "gold"
+    },
+    {
+      "query_id": "q3",
+      "query_text": "gamma unique",
+      "expected_filename": "gamma.txt",
+      "kind": "gold"
+    }
+  ]
+}

biblicus-0.15.0/docs/ANALYSIS.md

@@ -0,0 +1,143 @@
+# Corpus analysis
+
+Biblicus supports analysis backends that run on extracted text artifacts without changing the raw corpus. Analysis is a
+pluggable phase that reads an extraction run, produces structured output, and stores artifacts under the corpus runs
+folder. Each analysis backend declares its own configuration schema and output contract, and all schemas are validated
+strictly.
+
+## How analysis runs work
+
+- Analysis runs are tied to a corpus state via the extraction run reference.
+- The analysis output is written under `.biblicus/runs/analysis/<analysis-id>/<run_id>/`.
+- Analysis is reproducible when you supply the same extraction run and corpus catalog state.
+- Analysis configuration is stored as a recipe manifest in the run metadata.
+
+If you omit the extraction run, Biblicus uses the most recent extraction run and emits a reproducibility warning. For
+repeatable analysis runs, always pass the extraction run reference explicitly.
+
+## Analysis run artifacts
+
+Every analysis run records a manifest alongside the output:
+
+```
+.biblicus/runs/analysis/<analysis-id>/<run_id>/
+  manifest.json
+  output.json
+```
+
+The manifest captures the recipe, extraction run reference, and catalog timestamp so results can be reproduced and
+compared later.
+
+## Inspecting output
+
+Analysis outputs are JSON documents. You can view them directly:
+
+```
+cat corpora/example/.biblicus/runs/analysis/profiling/RUN_ID/output.json
+```
+
+Each analysis backend defines its own `report` payload. The run metadata is consistent across backends.
+
+## Comparing analysis runs
+
+When you compare analysis results, record:
+
+- Corpus path and catalog timestamp.
+- Extraction run reference.
+- Analysis recipe name and configuration.
+- Analysis run identifier and output path.
+
+These make it possible to rerun the analysis and explain differences.
+
+## Pluggable analysis backends
+
+Analysis backends implement the `CorpusAnalysisBackend` interface and are registered under `biblicus.analysis`.
+A backend receives the corpus, a recipe name, a configuration mapping, and an extraction run reference. It returns a
+Pydantic model that is serialized to JavaScript Object Notation for storage.
+
+## Choosing an analysis backend
+
+Start with profiling when you need fast, deterministic baselines. Use topic modeling when you want thematic clustering
+and exploratory labels. Use Markov analysis when you want state-transition structure over sequences of segments.
+Combine multiple backends for a clear view of corpus composition, themes, and state dynamics.
+
+## Recipe files
+
+Analysis recipes are optional JavaScript Object Notation or YAML files that capture configuration in a repeatable way.
+They are useful for sharing experiments and keeping runs reproducible.
+
+Recipes support cascading composition. When a command accepts `--recipe`, you can pass multiple recipe files. Biblicus
+merges them in order, where later recipes override earlier recipes via a deep merge. You can then apply `--config`
+overrides on top of the composed view.
+
+Minimal profiling recipe:
+
+```
+schema_version: 1
+```
+
+Minimal topic modeling recipe:
+
+```
+schema_version: 1
+text_source:
+  sample_size: 500
+bertopic_analysis:
+  parameters:
+    nr_topics: 8
+```
+
+Minimal Markov analysis recipe:
+
+```
+schema_version: 1
+model:
+  family: gaussian
+  n_states: 8
+segmentation:
+  method: sentence
+observations:
+  encoder: tfidf
+```
+
+## Topic modeling
+
+Topic modeling is the first analysis backend. It uses BERTopic to cluster extracted text, produces per-topic evidence,
+and optionally labels topics using an LLM. See `docs/TOPIC_MODELING.md` for detailed configuration and examples.
+
+The integration demo script is a working reference you can use as a starting point:
+
+```
+python scripts/topic_modeling_integration.py --corpus corpora/ag_news_demo --force
+```
+
+The command prints the analysis run identifier and the output path. Open the resulting `output.json` to inspect per-topic
+labels, keywords, and document examples.
+
+## Markov analysis
+
+Markov analysis learns a directed, weighted state transition graph over sequences of text segments. The output includes
+per-state exemplars, per-item decoded paths, and optional GraphViz exports. See `docs/MARKOV_ANALYSIS.md` for detailed
+configuration and examples.
+
+Text extract is available as a segmentation strategy for long texts. It inserts XML tags in-place using a virtual file
+editing loop, then extracts spans without requiring the model to re-emit the full transcript.
+
+## Profiling analysis
+
+Profiling is the baseline analysis backend. It summarizes corpus composition and extraction coverage using
+deterministic counts and distribution metrics. See `docs/PROFILING.md` for the full reference and working demo.
+
+### Minimal profiling run
+
+```
+python -m biblicus analyze profile --corpus corpora/example --extraction-run pipeline:RUN_ID
+```
+
+The command writes an analysis run directory and prints the run identifier.
+
+Run profiling from the CLI:
+
+```
+biblicus analyze profile --corpus corpora/example --extraction-run pipeline:RUN_ID
+```

biblicus-0.15.0/docs/ARCHITECTURE.md

@@ -0,0 +1,46 @@
+# Biblicus Architecture
+
+Biblicus sits between raw, unstructured data and the moment you need reliable answers from it.
+It is built for teams who receive large, messy corpora and must extract usable signals without
+losing provenance or reproducibility. Retrieval-augmented generation is one use case, but the
+system is broader than chatbots: it supports any pipeline that needs structured insight from
+unstructured data.
+
+At a high level the system does five things:
+
+1. **Ingests** raw content into a corpus with minimal friction.
+2. **Extracts** text from diverse media (documents, images, audio).
+3. **Transforms** and annotates text with reusable LLM utilities.
+4. **Retrieves** evidence through explicit, reproducible stages.
+5. **Evaluates** results so improvements are measurable, not anecdotal.
+
+The guiding idea is that every retrieval produces **evidence**: structured outputs with scores
+and provenance that can be inspected, audited, and reused. Context packs, summaries, and downstream
+generation are all derived from that evidence.
+
+## Why it exists
+
+Real-world AI work often starts with a folder full of files, not a clean database. Biblicus is the
+toolkit that turns those files into a manageable, testable system. It supports workflows like:
+
+- Indexing large collections of emails and making them searchable while protecting sensitive data.
+- Processing discovery dumps of scanned PDFs with OCR and extracting evidence for analysis.
+- Turning policy or rules documents into a controlled knowledge base for assistants.
+
+## How it fits into AI systems
+
+Biblicus integrates with agent frameworks through explicit tool interfaces. It does not hide
+retrieval inside the model. Instead, it provides repeatable pipelines that expose *what* was
+retrieved and *why*, so models can use evidence directly and safely.
+
+## Where to go next
+
+- Start with **CORPUS** and **EXTRACTION** to understand how raw content is ingested.
+- Move to **RETRIEVAL** and **RETRIEVAL_EVALUATION** to see how evidence is produced and tested.
+- Explore **TOPIC_MODELING** and **MARKOV_ANALYSIS** if you need higher-level analysis tools.
+- See **TEXT_UTILITIES** for reusable, AI-assisted text transformations.
+
+## Detailed architecture and policies
+
+For a deep, internal reference (including design policies and architectural constraints), see
+`ARCHITECTURE_DETAIL.md`.