pandion 0.1.0__tar.gz

pandion-0.1.0/.gitignore

@@ -0,0 +1,49 @@
+.remember/
+
+# Python
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+.coverage
+htmlcov/
+*.egg-info/
+dist/
+build/
+
+# Playground data — regenerable, sometimes large
+artifacts/
+sources/
+runs/
+graphs/
+qa_sets/
+
+# Secrets
+.env
+.env.*
+!.env.example
+
+# Node (phase 5)
+node_modules/
+web/dist/
+
+# Editors / OS
+.vscode/
+.idea/
+.DS_Store
+Thumbs.db
+
+# Third-party agent skills (installed via `npx skills add`, not project source)
+.agents/
+.claude/skills/
+
+# Browser-automation screenshots and traces
+.playwright-mcp/
+
+# Agent working notes, and the private plans, specs and design system (.superpowers/docs/)
+.superpowers/
+# Local launch-video output and the skill installer lock file
+brag-output*/
+skills-lock.json
+# Agent worktrees
+.claude/worktrees/

pandion-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nadeem Khan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pandion-0.1.0/PKG-INFO

@@ -0,0 +1,118 @@
+Metadata-Version: 2.5
+Name: pandion
+Version: 0.1.0
+Summary: Retrieval pipelines you can inspect, step by step
+Project-URL: Homepage, https://rag.codewithnk.com
+Project-URL: Documentation, https://github.com/nadeem4/rag-playground/blob/main/docs/pandion.md
+Project-URL: Source, https://github.com/nadeem4/rag-playground/tree/main/packages/pandion
+Project-URL: Issues, https://github.com/nadeem4/rag-playground/issues
+Author: Nadeem Khan
+License-Expression: MIT
+License-File: LICENSE
+Keywords: chunking,embeddings,pdf,rag,reranking,retrieval,search
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: <3.14,>=3.13
+Requires-Dist: pydantic>=2.9
+Requires-Dist: pyyaml>=6.0.3
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=1.7.0; extra == 'anthropic'
+Provides-Extra: docling
+Requires-Dist: docling>=2.129.0; extra == 'docling'
+Requires-Dist: onnxruntime>=1.30.0; extra == 'docling'
+Provides-Extra: lancedb
+Requires-Dist: lancedb<0.39,>=0.38; extra == 'lancedb'
+Requires-Dist: pyarrow>=25.0.1; extra == 'lancedb'
+Provides-Extra: local
+Requires-Dist: sentence-transformers>=6.1.0; extra == 'local'
+Requires-Dist: torch>=2.14.0; extra == 'local'
+Provides-Extra: openai
+Requires-Dist: openai>=3.17.0; extra == 'openai'
+Provides-Extra: pdfium
+Requires-Dist: pypdfium2>=5.13.0; extra == 'pdfium'
+Description-Content-Type: text/markdown
+
+# Pandion
+
+Retrieval pipelines you can inspect, step by step.
+
+Pandion parses your documents, cleans and chunks them, embeds and stores them, then
+searches, reranks and answers questions. Every step is a strategy you can swap, and every
+step tells you what it did. Each result is cached under a hash of its settings, so the
+same files with the same settings are never processed twice.
+
+Pandion is named after the osprey, which spots a fish under the water and dives to
+retrieve it. It is the engine of [RAG Playground](https://rag.codewithnk.com), a web app
+that shows every step of a pipeline on a PDF.
+
+## Install
+
+Pandion needs Python 3.13.
+
+```bash
+pip install "pandion[pdfium,lancedb,local]"
+```
+
+The base install is small. Parsers, stores and models are extras, and a step whose extra
+is missing tells you which one to install:
+
+| Extra | What it brings | Needed by |
+|---|---|---|
+| `pdfium` | pypdfium2 | the fast `pdfium` parser |
+| `docling` | Docling and its OCR engine | the `docling` parser, for layout, tables and scans |
+| `lancedb` | LanceDB and pyarrow | the index, and the `dense`, `bm25` and `hybrid_rrf` search |
+| `local` | sentence-transformers and torch | the local embedders, and the `mmr` and `cross_encoder` rerankers |
+| `anthropic` | the Anthropic SDK | `llm_rewrite`, `llm_rerank` and `chat` with Claude |
+| `openai` | the OpenAI SDK | the same steps with OpenAI, OpenRouter or a compatible server |
+
+`pandion install pipeline.yaml` installs the extras a pipeline file needs.
+
+## Quick start
+
+```python
+from pandion import Pipeline
+
+p = Pipeline(
+    path="./my-index",
+    parse="pdfium",
+    chunk={"use": "recursive_character", "chunk_size": 400, "chunk_overlap": 40},
+    embed="bge-small-en-v1.5",
+    store="lancedb",
+    search="hybrid_rrf",
+    answer="search",
+)
+p.index.run("report.pdf")
+print(p.ask("What does the report recommend?").text)
+```
+
+The embedder downloads once, on first use. After that, a second `p.index.run` on the
+same file finds every step already stored and reuses it.
+
+## What you can do
+
+- **Run one step on its own**, with no pipeline and no cache:
+  `pandion.parse(pdf, use="pdfium")`, `pandion.clean(...)`, `pandion.chunk(...)`.
+- **Write a pipeline three ways**: as a YAML file (`Pipeline.from_file("pipeline.yaml")`),
+  as a dict per step, or with classes such as `from pandion.parse import Docling`. The
+  three give the same pipeline and the same cache.
+- **Build and update an index**: `p.index.run(files)` indexes them, and
+  `p.index.sync(folder)` adds, changes and removes files to match a folder.
+- **Search**: `p.retrieve(question)` runs the rewrite, search and rerank steps and returns
+  each step's result as well as the hits.
+- **Ask**: `p.ask(question)` returns the pieces found, or an answer from a chat model
+  that cites them.
+
+## Learn more
+
+- [The guide](https://github.com/nadeem4/rag-playground/blob/main/docs/pandion.md): every
+  step, setting and error.
+- [Examples](https://github.com/nadeem4/rag-playground/tree/main/examples/pandion): index,
+  sync, search, ask, and the three ways to write a pipeline.
+- [RAG Playground](https://rag.codewithnk.com): try the same pipelines in the browser.
+
+## License
+
+MIT. See [LICENSE](https://github.com/nadeem4/rag-playground/blob/main/LICENSE).

pandion-0.1.0/README.md

@@ -0,0 +1,81 @@
+# Pandion
+
+Retrieval pipelines you can inspect, step by step.
+
+Pandion parses your documents, cleans and chunks them, embeds and stores them, then
+searches, reranks and answers questions. Every step is a strategy you can swap, and every
+step tells you what it did. Each result is cached under a hash of its settings, so the
+same files with the same settings are never processed twice.
+
+Pandion is named after the osprey, which spots a fish under the water and dives to
+retrieve it. It is the engine of [RAG Playground](https://rag.codewithnk.com), a web app
+that shows every step of a pipeline on a PDF.
+
+## Install
+
+Pandion needs Python 3.13.
+
+```bash
+pip install "pandion[pdfium,lancedb,local]"
+```
+
+The base install is small. Parsers, stores and models are extras, and a step whose extra
+is missing tells you which one to install:
+
+| Extra | What it brings | Needed by |
+|---|---|---|
+| `pdfium` | pypdfium2 | the fast `pdfium` parser |
+| `docling` | Docling and its OCR engine | the `docling` parser, for layout, tables and scans |
+| `lancedb` | LanceDB and pyarrow | the index, and the `dense`, `bm25` and `hybrid_rrf` search |
+| `local` | sentence-transformers and torch | the local embedders, and the `mmr` and `cross_encoder` rerankers |
+| `anthropic` | the Anthropic SDK | `llm_rewrite`, `llm_rerank` and `chat` with Claude |
+| `openai` | the OpenAI SDK | the same steps with OpenAI, OpenRouter or a compatible server |
+
+`pandion install pipeline.yaml` installs the extras a pipeline file needs.
+
+## Quick start
+
+```python
+from pandion import Pipeline
+
+p = Pipeline(
+    path="./my-index",
+    parse="pdfium",
+    chunk={"use": "recursive_character", "chunk_size": 400, "chunk_overlap": 40},
+    embed="bge-small-en-v1.5",
+    store="lancedb",
+    search="hybrid_rrf",
+    answer="search",
+)
+p.index.run("report.pdf")
+print(p.ask("What does the report recommend?").text)
+```
+
+The embedder downloads once, on first use. After that, a second `p.index.run` on the
+same file finds every step already stored and reuses it.
+
+## What you can do
+
+- **Run one step on its own**, with no pipeline and no cache:
+  `pandion.parse(pdf, use="pdfium")`, `pandion.clean(...)`, `pandion.chunk(...)`.
+- **Write a pipeline three ways**: as a YAML file (`Pipeline.from_file("pipeline.yaml")`),
+  as a dict per step, or with classes such as `from pandion.parse import Docling`. The
+  three give the same pipeline and the same cache.
+- **Build and update an index**: `p.index.run(files)` indexes them, and
+  `p.index.sync(folder)` adds, changes and removes files to match a folder.
+- **Search**: `p.retrieve(question)` runs the rewrite, search and rerank steps and returns
+  each step's result as well as the hits.
+- **Ask**: `p.ask(question)` returns the pieces found, or an answer from a chat model
+  that cites them.
+
+## Learn more
+
+- [The guide](https://github.com/nadeem4/rag-playground/blob/main/docs/pandion.md): every
+  step, setting and error.
+- [Examples](https://github.com/nadeem4/rag-playground/tree/main/examples/pandion): index,
+  sync, search, ask, and the three ways to write a pipeline.
+- [RAG Playground](https://rag.codewithnk.com): try the same pipelines in the browser.
+
+## License
+
+MIT. See [LICENSE](https://github.com/nadeem4/rag-playground/blob/main/LICENSE).

pandion-0.1.0/pyproject.toml

@@ -0,0 +1,54 @@
+[project]
+name = "pandion"
+version = "0.1.0"
+description = "Retrieval pipelines you can inspect, step by step"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Nadeem Khan" }]
+requires-python = ">=3.13,<3.14"
+keywords = ["retrieval", "rag", "search", "embeddings", "pdf", "chunking", "reranking"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Text Processing :: Indexing",
+]
+dependencies = ["pydantic>=2.9", "pyyaml>=6.0.3"]
+
+# Parsers, models and stores are extras, so the base install stays small.
+# A step whose extra is missing says which one to install.
+[project.optional-dependencies]
+pdfium = ["pypdfium2>=5.13.0"]
+docling = [
+    "docling>=2.129.0",
+    # Docling's OCR engine. RapidOCR's ONNX checkpoints ship inside its wheel,
+    # so OCR downloads nothing; without onnxruntime Docling falls back to
+    # RapidOCR's torch backend, which fetches .pth files into site-packages.
+    "onnxruntime>=1.30.0",
+]
+# Local models: the embedders, the cross-encoder and mmr. On Linux the root
+# pyproject.toml takes torch from the CPU-only index.
+local = ["sentence-transformers>=6.1.0", "torch>=2.14.0"]
+lancedb = ["lancedb>=0.38,<0.39", "pyarrow>=25.0.1"]
+# Chat models, for llm_rewrite: OpenAI, OpenRouter and any OpenAI-compatible
+# server through the OpenAI SDK, and Claude through the Anthropic SDK.
+openai = ["openai>=3.17.0"]
+anthropic = ["anthropic>=1.7.0"]
+
+[project.urls]
+Homepage = "https://rag.codewithnk.com"
+Documentation = "https://github.com/nadeem4/rag-playground/blob/main/docs/pandion.md"
+Source = "https://github.com/nadeem4/rag-playground/tree/main/packages/pandion"
+Issues = "https://github.com/nadeem4/rag-playground/issues"
+
+[project.scripts]
+pandion = "pandion.cli:main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/pandion"]

pandion-0.1.0/src/pandion/__init__.py

@@ -0,0 +1,45 @@
+"""Pandion: indexing and retrieval pipelines you can inspect stage by stage.
+
+The engine lives in plain modules (`pandion.graph`, `pandion.executor`,
+`pandion.storage` and the rest). A name that starts with an underscore is
+private; the RAG Playground app may import every other name.
+
+`pandion.parse`, `pandion.clean`, `pandion.chunk` and `pandion.rewrite` run one
+stage when called, and hold that stage's step classes. `pandion.retrieve`,
+`pandion.rerank` and `pandion.answer` hold the step classes of the rest.
+"""
+
+from pandion.models import ModelSettings, preload_embedder, preload_reranker
+from pandion.pipeline import (
+    Answer,
+    IndexPipeline,
+    Pipeline,
+    QueryPipeline,
+    Retrieval,
+    RetrievalPipeline,
+)
+from pandion.results import StageResult
+from pandion import answer, chunk, clean, parse, rerank, retrieve, rewrite
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "Answer",
+    "IndexPipeline",
+    "ModelSettings",
+    "Pipeline",
+    "QueryPipeline",
+    "Retrieval",
+    "RetrievalPipeline",
+    "StageResult",
+    "answer",
+    "chunk",
+    "clean",
+    "parse",
+    "preload_embedder",
+    "preload_reranker",
+    "rerank",
+    "retrieve",
+    "rewrite",
+    "__version__",
+]

pandion-0.1.0/src/pandion/answer.py

@@ -0,0 +1,16 @@
+"""The answer steps, as classes for a pipeline written in code.
+
+`Search(top_k=5)` shows the pieces retrieval found, as they are. `Chat(model=
+"claude-haiku-4-5")` asks a chat model for an answer whose claims cite those
+pieces. Each is the same as its block, for example `{"use": "search",
+"top_k": 5}` (`pandion.config.step_class`). Answering runs through a
+pipeline: `p.ask(question)`.
+"""
+
+from pandion.config import step_class
+from pandion.ports import Stage
+
+Search = step_class(Stage.USE_CASE, "search", __name__)
+Chat = step_class(Stage.USE_CASE, "chat", __name__)
+
+__all__ = ["Chat", "Search"]

pandion-0.1.0/src/pandion/artifacts.py

@@ -0,0 +1,44 @@
+"""Artifact types and the immutable artifact value.
+
+An artifact is what a transform produces: an immutable value identified by a
+recipe hash (see `pandion.ids`). The payload itself lives on disk and is loaded
+lazily; `meta` is small and always in memory.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from enum import StrEnum
+from typing import Any
+
+
+class ArtifactType(StrEnum):
+    """The value types that flow between pipeline stages.
+
+    A StrEnum rather than a Literal so that `inputs = {"file": ArtifactType.RAW_FILE}`
+    typechecks in plugins without an explicit annotation, and so it serializes to
+    JSON as a plain string.
+    """
+
+    RAW_FILE = "raw_file"
+    QUERY = "query"
+    PARSED_DOC = "parsed_doc"
+    CHUNK_SET = "chunk_set"
+    INDEX = "index"
+    RETRIEVAL_RESULT = "retrieval_result"
+    OUTPUT = "output"
+    QA_SET = "qa_set"
+
+
+@dataclass(frozen=True)
+class Artifact:
+    """A committed pipeline value.
+
+    `id` is a *recipe* hash, not a content hash — it is computed from the
+    transform and its inputs before the payload exists. That is sound only for
+    deterministic transforms; see `Transform.deterministic`.
+    """
+
+    id: str
+    type: ArtifactType
+    meta: dict[str, Any] = field(default_factory=dict)

pandion-0.1.0/src/pandion/chunk.py

@@ -0,0 +1,23 @@
+"""The chunkers: call this module to chunk one document, or use a class in a pipeline.
+
+`pandion.chunk(doc, use="recursive_character", chunk_size=400)` cuts one parsed
+document into pieces (`pandion.stages.chunk`). `RecursiveCharacter(chunk_size=400)`
+is the same settings for a pipeline written in code (`pandion.config.step_class`).
+"""
+
+import sys
+
+from pandion.config import step_class
+from pandion.ports import Stage
+from pandion.stages import StageModule
+from pandion.stages import chunk as run
+
+RecursiveCharacter = step_class(Stage.CHUNK, "recursive_character", __name__)
+LayoutBlocks = step_class(Stage.CHUNK, "layout_blocks", __name__)
+MarkdownHeader = step_class(Stage.CHUNK, "markdown_header", __name__)
+TokenBased = step_class(Stage.CHUNK, "token_based", __name__)
+SentenceWindow = step_class(Stage.CHUNK, "sentence_window", __name__)
+
+__all__ = ["LayoutBlocks", "MarkdownHeader", "RecursiveCharacter", "SentenceWindow", "TokenBased"]
+
+sys.modules[__name__].__class__ = StageModule

pandion-0.1.0/src/pandion/clean.py

@@ -0,0 +1,21 @@
+"""The cleaners: call this module to clean one document, or use a class in a pipeline.
+
+`pandion.clean(doc, steps=[{"use": "dedupe_blocks"}])` cleans one parsed
+document (`pandion.stages.clean`). `DedupeBlocks()` is the same step for a
+pipeline written in code (`pandion.config.step_class`).
+"""
+
+import sys
+
+from pandion.config import step_class
+from pandion.ports import Stage
+from pandion.stages import StageModule
+from pandion.stages import clean as run
+
+HeaderFooterStrip = step_class(Stage.CLEAN, "header_footer_strip", __name__)
+DedupeBlocks = step_class(Stage.CLEAN, "dedupe_blocks", __name__)
+DropMatching = step_class(Stage.CLEAN, "drop_matching", __name__)
+
+__all__ = ["DedupeBlocks", "DropMatching", "HeaderFooterStrip"]
+
+sys.modules[__name__].__class__ = StageModule

pandion-0.1.0/src/pandion/cli.py

@@ -0,0 +1,87 @@
+"""The `pandion` command.
+
+`pandion install pipeline.yaml` reads the file, works out which extras its
+steps need, and installs them with pip into the Python that runs it.
+`--print` only prints the command. Nothing is ever installed while a pipeline
+runs.
+"""
+
+from __future__ import annotations
+
+import argparse
+import subprocess
+import sys
+
+from pandion.config import ConfigError, PipelineConfig, load
+
+#: The extra each strategy or embedder needs. A name that is not here needs
+#: only the base install.
+EXTRAS = {
+    "pdfium": "pdfium",
+    "docling": "docling",
+    "lancedb": "lancedb",
+    "qwen3-embedding-0.6b": "local",
+    "bge-small-en-v1.5": "local",
+    "mmr": "local",
+    "cross_encoder": "local",
+}
+
+#: Steps that call a chat model, by stage; their extra depends on the model's provider.
+CHAT_STEPS = {"llm_rewrite": "query", "llm_rerank": "rerank", "chat": "use_case"}
+
+
+def extras_for(config: PipelineConfig) -> list[str]:
+    """The extras the pipeline's steps need, sorted."""
+    index = config.index
+    names = [index.parse[0], *(use for use, _ in index.clean), index.chunk[0]]
+    names += ["lancedb", index.store["embedder"]]
+    extras = {EXTRAS[name] for name in names if name in EXTRAS}
+    retrieve = config.retrieve
+    if retrieve is not None:
+        steps = [*([retrieve.rewrite] if retrieve.rewrite else []), retrieve.search, *retrieve.rerank]
+        if config.answer is not None:
+            steps.append(config.answer)
+        for use, settings in steps:
+            if use in CHAT_STEPS:
+                extras.add(_chat_extra(use, settings))
+            elif use in EXTRAS:
+                extras.add(EXTRAS[use])
+    return sorted(extras)
+
+
+def _chat_extra(use: str, settings: dict) -> str:
+    """Claude models need the Anthropic SDK; every other model the OpenAI SDK."""
+    from pandion.llm import CHAT_MODELS
+    from pandion.plugins import discover
+    from pandion.ports import Stage
+    from pandion.registry import registry
+
+    discover()
+    model = registry.get(Stage(CHAT_STEPS[use]), use).config_model(**settings).model
+    return "anthropic" if CHAT_MODELS[model].provider == "anthropic" else "openai"
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="pandion")
+    commands = parser.add_subparsers(dest="command", required=True)
+    install = commands.add_parser("install", help="install the extras a pipeline file needs")
+    install.add_argument("config", help="the pipeline file, for example pipeline.yaml")
+    install.add_argument("--print", action="store_true", help="only print the pip command")
+    args = parser.parse_args(argv)
+
+    try:
+        config = load(args.config)
+    except (ConfigError, OSError) as exc:
+        print(f"pandion: {exc}", file=sys.stderr)
+        return 2
+    extras = extras_for(config)
+    package = f"pandion[{','.join(extras)}]" if extras else "pandion"
+    command = [sys.executable, "-m", "pip", "install", package]
+    if args.print:
+        print(f'"{sys.executable}" -m pip install "{package}"')
+        return 0
+    return subprocess.call(command)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())