right-rag 0.1.0__tar.gz

right_rag-0.1.0/.claude/skills/right-rag/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: right-rag
+description: Work on right-rag domain-structured RAG repositories. Use when the user asks to inspect, extend, run, debug, query, or evaluate a right-rag project; create or modify document loaders, unit parsers, chunk parsers, savers, retrievers, answer generators, evaluators, reporters, pipelines, or tests; design domain-specific RAG behavior without coupling the framework to vector, keyword, SQL, graph, or any specific search method.
+---
+
+# Right RAG
+
+## Core Model
+
+Treat right-rag as a domain-structured RAG template, not a plain text chunker.
+Keep the framework generic and put domain logic in `spec/` plugins.
+
+The pipeline shape is:
+
+- Index: document loader -> unit parser -> chunk parser -> unit/chunk savers -> artifacts.
+- Query: one user query -> retrievers read artifacts -> answer generator combines retrieved values.
+- Evaluate: eval cases -> query pipeline -> evaluator -> reporter.
+
+Do not assume retrieval means vector search, keyword search, BM25, SQL, or graph traversal. A saver may store however it wants, and its produced artifact should expose the retrieval/use behavior needed by retrievers.
+
+## Before Editing
+
+Inspect local state first:
+
+```powershell
+git status --short
+uv sync
+python -m compileall src/right_rag
+python -m unittest discover -s tests
+uv run python -c "from right_rag.registry import build_registry; r=build_registry(); print(r.document_loaders, r.unit_parsers, r.chunk_parsers, r.unit_savers, r.chunk_savers)"
+```
+
+Respect dirty worktrees. Do not revert user changes or generated outputs unless explicitly asked.
+
+## Extension Points
+
+Prefer adding or editing plugins under `spec/` before changing core files:
+
+- `spec/document_loaders/`: load source documents.
+- `spec/unit_parsers/`: parse domain units and return `ParsedUnit`.
+- `spec/chunk_parsers/`: chunk from semantic `Unit` objects.
+- `spec/unit_savers/`: persist units and return unit artifacts.
+- `spec/chunk_savers/`: persist chunks and return chunk artifacts.
+- `spec/retrievers/`: read all artifacts and decide how to retrieve.
+- `spec/answer_generators/`: generate an answer from retrieved values.
+- `spec/evaluators/`: score query outputs.
+- `spec/reporters/`: write evaluation reports.
+
+Keep contracts small. Add a new abstraction only when it removes real coupling or is required for third-party plugins.
+
+## Parser Rules
+
+Model the author's real document structure:
+
+- laws: act -> chapter -> section -> clause -> subclause
+- contracts: agreement -> article -> clause -> schedule item
+- policies: policy -> section -> rule -> exception
+- manuals: manual -> chapter -> procedure -> step
+- academic papers: paper -> section -> subsection -> paragraph/table/figure note
+
+`UnitParser.parse_document()` returns `Iterable[ParsedUnit]`, not `Unit`. The core assigns ids, document ids, parent ids, raw text, source spans, and change replay.
+
+Do not add catch-all production parsers. `parser_can_handle_document()` should be narrow, and parser acceptance should not overlap. Reject unsupported formats visibly instead of creating one huge document unit.
+
+## Chunking
+
+Chunk inside one `Unit`; every chunk must trace back to its unit. Keep chunks semantically complete: definition, rule, exception, method, condition, table row, or procedure step.
+
+Carry citation and filtering attributes when available: source path, page, heading path, section number, role, unit type, effective date, jurisdiction, party, method, dataset, or citation.
+
+## Artifacts And Retrieval
+
+Savers define how stored data can be used later. A unit saver creates a unit artifact; a chunk saver creates a chunk artifact. Retrievers receive all artifacts, usually grouped by saver name, and decide how to combine them.
+
+Design artifacts by capability, not storage brand:
+
+- A vector-backed artifact may expose semantic search.
+- A SQL-backed artifact may expose filter/query methods.
+- A graph-backed artifact may expose traversal.
+- A custom artifact may expose any domain-specific lookup.
+
+Retrievers should not need to know saver implementation details. They should call artifact methods or protocol capabilities and produce retrieved value items for answer generation.
+
+## Query Parameters
+
+Keep user-facing query simple: one query string plus optional parameters. Broadcast generic parameters to all retrievers. Allow retriever-specific overrides only when explicitly namespaced by the CLI or request format.
+
+Do not hard-code parameter names like `top_k` in core logic. Let each retriever declare or parse the parameters it supports so command help and validation can follow the selected retrievers.
+
+## Evaluation
+
+Provide a default evaluator only as smoke/regression coverage. Real projects should customize `spec/evaluators/` for their domain.
+
+Useful default checks:
+
+- expected retrieved ids are present
+- answer contains required text
+- retrieved evidence contains required text
+- forbidden text is absent
+- simple hit/recall/overlap metrics
+
+Reports should be generated by reporters under `spec/reporters/`, commonly Markdown for humans and JSON for automation.
+
+## Useful Commands
+
+```powershell
+right-rag index
+right-rag query "your question"
+right-rag evaluate eval\cases.example.json --output output\evaluation\report.md
+python -m compileall src/right_rag
+python -m unittest discover -s tests
+```

right_rag-0.1.0/.codex/skills/right-rag/SKILL.md

@@ -0,0 +1,128 @@
+---
+name: right-rag
+description: Work on right-rag domain-structured RAG repositories. Use when Codex needs to inspect, extend, run, debug, query, or evaluate a right-rag project; create or modify document loaders, unit parsers, chunk parsers, savers, retrievers, answer generators, evaluators, reporters, pipelines, or tests; design domain-specific RAG behavior without coupling the framework to vector, keyword, SQL, graph, or any specific search method.
+---
+
+# Right RAG
+
+## Core Model
+
+Treat right-rag as a domain-structured RAG template, not a plain text chunker.
+Keep the framework generic and put domain logic in `spec/` plugins.
+
+The default shape is:
+
+- Index: document loader -> unit parser -> chunk parser -> unit/chunk savers -> artifacts.
+- Query: one user query -> retrievers read artifacts -> answer generator combines retrieved values.
+- Evaluate: eval cases -> query pipeline -> evaluator -> reporter.
+
+Do not assume retrieval means vector search, keyword search, BM25, SQL, or graph traversal. A saver may store however it wants, and its produced artifact should expose the retrieval/use behavior needed by retrievers.
+
+## Before Editing
+
+In a right-rag repo, first inspect local structure and current state:
+
+```powershell
+uv sync
+python -m compileall src/right_rag
+python -m unittest discover -s tests
+uv run python -c "from right_rag.registry import build_registry; r=build_registry(); print(r.document_loaders, r.unit_parsers, r.chunk_parsers, r.unit_savers, r.chunk_savers)"
+```
+
+Respect dirty worktrees. Do not revert user changes or generated outputs unless explicitly asked.
+
+## Extension Points
+
+Prefer adding or editing plugins under `spec/` before changing core files:
+
+- `spec/document_loaders/`: load source documents.
+- `spec/unit_parsers/`: parse domain units and return `ParsedUnit`.
+- `spec/chunk_parsers/`: chunk from semantic `Unit` objects.
+- `spec/unit_savers/`: persist units and return unit artifacts.
+- `spec/chunk_savers/`: persist chunks and return chunk artifacts.
+- `spec/retrievers/`: read all artifacts and decide how to retrieve.
+- `spec/answer_generators/`: generate an answer from retrieved values.
+- `spec/evaluators/`: score query outputs.
+- `spec/reporters/`: write evaluation reports.
+
+Keep contracts small. Add a new abstraction only when it removes real coupling or is required for third-party plugins.
+
+## Parser Rules
+
+Model the author's real document structure. Examples:
+
+- laws: act -> chapter -> section -> clause -> subclause
+- contracts: agreement -> article -> clause -> schedule item
+- policies: policy -> section -> rule -> exception
+- manuals: manual -> chapter -> procedure -> step
+- academic papers: paper -> section -> subsection -> paragraph/table/figure note
+
+`UnitParser.parse_document()` returns `Iterable[ParsedUnit]`, not `Unit`. The core assigns ids, document ids, parent ids, raw text, source spans, and change replay.
+
+Do not add catch-all production parsers. `parser_can_handle_document()` should be narrow, and parser acceptance should not overlap. Reject unsupported formats visibly instead of creating one huge document unit.
+
+## Chunking
+
+Chunk inside one `Unit`; every chunk must trace back to its unit. Keep chunks semantically complete: definition, rule, exception, method, condition, table row, or procedure step.
+
+Carry citation and filtering attributes when available: source path, page, heading path, section number, role, unit type, effective date, jurisdiction, party, method, dataset, or citation.
+
+## Artifacts And Retrieval
+
+Savers define how stored data can be used later. A unit saver creates a unit artifact; a chunk saver creates a chunk artifact. Retrievers receive all artifacts, usually grouped by saver name, and decide how to combine them.
+
+Design artifacts by capability, not storage brand:
+
+- A vector-backed artifact may expose semantic search.
+- A SQL-backed artifact may expose filter/query methods.
+- A graph-backed artifact may expose traversal.
+- A custom artifact may expose any domain-specific lookup.
+
+Retrievers should not need to know the implementation details of a saver. They should call artifact methods or protocol capabilities and produce retrieved value items for answer generation.
+
+## Query Parameters
+
+Keep user-facing query simple: one query string plus optional parameters. Broadcast generic parameters to all retrievers. Allow retriever-specific overrides only when explicitly namespaced by the CLI or request format.
+
+Do not hard-code parameter names like `top_k` in core logic. Let each retriever declare or parse the parameters it supports so command help and validation can follow the selected retrievers.
+
+## Evaluation
+
+Provide a default evaluator only as smoke/regression coverage. Make it explicit that real projects should customize `spec/evaluators/` for their domain.
+
+Useful default checks:
+
+- expected retrieved ids are present
+- answer contains required text
+- retrieved evidence contains required text
+- forbidden text is absent
+- simple hit/recall/overlap metrics
+
+Reports should be generated by reporters under `spec/reporters/`, commonly Markdown for humans and JSON for automation.
+
+## Useful Commands
+
+Run index:
+
+```powershell
+right-rag index
+```
+
+Run query:
+
+```powershell
+right-rag query "your question"
+```
+
+Run evaluation:
+
+```powershell
+right-rag evaluate eval\cases.example.json --output output\evaluation\report.md
+```
+
+Final verification after changes:
+
+```powershell
+python -m compileall src/right_rag
+python -m unittest discover -s tests
+```

right_rag-0.1.0/.codex/skills/right-rag/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Right RAG"
+  short_description: "Work on right-rag RAG pipelines"
+  default_prompt: "Use $right-rag to inspect, extend, run, query, or evaluate a right-rag repository."

right_rag-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,34 @@
+name: CI
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Install uv
+        run: python -m pip install uv
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Compile package
+        run: uv run python -m compileall src/right_rag
+
+      - name: Run tests
+        run: uv run python -m unittest discover -s tests
+
+      - name: Check CLI
+        run: uv run right-rag --help
+
+      - name: Build package
+        run: uv build

right_rag-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,37 @@
+name: Publish
+
+on:
+  push:
+    tags: ["v*"]
+
+permissions:
+  contents: read
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Install uv
+        run: python -m pip install uv
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Run tests
+        run: uv run python -m unittest discover -s tests
+
+      - name: Build package
+        run: uv build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

right_rag-0.1.0/.gitignore

@@ -0,0 +1,19 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+
+# Local source documents
+data/documents/*
+
+# Pipeline state
+.right-rag-state.json
+
+# Generated pipeline output
+output/

right_rag-0.1.0/.python-version

@@ -0,0 +1 @@
+3.13

right_rag-0.1.0/AGENTS.md

@@ -0,0 +1,222 @@
+# AGENTS.md
+
+Guide for AI agents working in this repository.
+
+## Setup First
+
+Before changing code, make sure the project is runnable:
+
+```powershell
+uv sync
+python -m compileall src/right_rag
+python -m unittest discover -s tests
+uv run python -c "from right_rag.registry import build_registry; r=build_registry(); print(r.document_loaders, r.unit_parsers, r.chunk_parsers, r.unit_savers, r.chunk_savers)"
+```
+
+Source documents go in `data/documents/`. Pipeline output goes to `output/`.
+
+## How This Repo Works
+
+`right-rag` keeps the pipeline core separate from task-specific logic in
+`spec/`.
+
+Core files:
+
+- `src/right_rag/interfaces.py`: plugin contracts.
+- `src/right_rag/model.py`: `Document`, `ParsedUnit`, `Unit`, `Chunk`, change replay helpers.
+- `src/right_rag/registry.py`: auto-discovers plugins under `spec/`.
+- `src/right_rag/pipeline.py`: orchestration, incremental state, parser output normalization.
+- `src/right_rag/pipeline_state.py`: JSON serialization and stable hashing.
+- `tests/`: stdlib `unittest` suite for core behavior and default plugins.
+
+Prefer editing or adding plugins in `spec/` before changing the pipeline core.
+
+## Adapt Plugins To The Documents
+
+`right-rag` is a domain-structured RAG pipeline, not a plain text chunker. The
+default unit/chunk parsers are smoke-test tools only. Never use them for
+production data. Real work belongs in domain parsers under `spec/`.
+
+Read the user's actual document type before editing. Choose domain units from
+the document's real structure:
+
+- laws: act -> chapter -> section -> clause -> subclause
+- contracts: agreement -> article -> clause -> schedule item
+- policies: policy -> section -> rule -> exception
+- manuals: manual -> chapter -> procedure -> step
+- academic papers: paper -> section -> subsection -> paragraph/table/figure note
+
+Plugin folders:
+
+- `spec/document_loaders/`: `DocumentLoader`
+- `spec/unit_parsers/`: `UnitParser` returning `ParsedUnit`
+- `spec/chunk_parsers/`: `ChunkParser`
+- `spec/unit_savers/`: `UnitSaver`
+- `spec/chunk_savers/`: `ChunkSaver`
+
+Use the default loader unless the source is not a normal local document folder
+or its output is not enough. For PDFs, the default loader uses `pypdfium2` text
+extraction first to avoid Docling OCR/preprocess memory errors; non-PDF files
+still go through Docling. Scanned PDFs without a text layer need a custom OCR
+loader.
+
+## How To Design Parsers
+
+1. Survey the documents and count the real formats.
+2. Create one parser per stable document format.
+3. Identify structural markers: `หมวด`, `มาตรา`, `ข้อ`, `Section`, `Chapter`,
+   definitions, penalties, duties, prohibitions, transitory provisions, tables,
+   schedules, and appendices.
+4. Make `UnitParser` return a `ParsedUnit` tree.
+5. Make `ChunkParser` chunk from semantic `Unit` objects, not raw text.
+
+Each parser must implement `parser_can_handle_document()` narrowly. Multiple
+parsers are fine when the corpus has multiple formats, but their acceptance
+rules should not overlap. Do not add catch-all parsers. Do not fallback to
+`ParsedUnit(topic="document", fulltext=document.text)` or one huge chunk when
+structure is missing. Reject/report the reason instead.
+
+## Unit Parser Contract
+
+`UnitParser.parse_document()` returns `Iterable[ParsedUnit]`, not `Unit`.
+
+A `ParsedUnit` contains only what the parser should know:
+
+```python
+ParsedUnit(
+    topic="Clause 1",
+    fulltext="...",
+    temporal_topic="Procurement Rules 2569 Clause 1",
+    source_start=120,
+    source_end=260,
+    attributes={"page_start": 4, "page_end": 5, "clause": "1"},
+    metadata={"heading_path": ["Part 1", "Clause 1"]},
+    subunits=[
+        ParsedUnit(
+            topic="Subclause 1",
+            temporal_topic="Procurement Rules 2569 Clause 1 Subclause 1",
+            fulltext="...",
+        )
+    ],
+)
+```
+
+The library handles:
+
+- `Unit.id`
+- `document_id`
+- `parent_id`
+- copying `source_start` and `source_end` to `Unit`
+- computing `Unit.raw_text` from `Document.text[source_start:source_end]`
+- recursive subunit conversion
+- `AddUnit`, `EditUnit`, `DeleteUnit`
+- attribute changes
+- parent `AddSubUnit`, `EditSubUnit`, `DeleteSubUnit`
+- current-stage replay from changes
+
+Identity uses `topic` plus the parent path. Metadata is stored only and is not
+used for processing decisions. Child ids include the parent path, so repeated
+topics under different parents do not collide.
+
+## Unit Parser Best Practices
+
+- Preserve the author's hierarchy. Do not flatten meaningful sections.
+- Keep `topic` stable and normalized.
+- Use `temporal_topic` only when the parser can identify version/date context.
+- Put hierarchy in `ParsedUnit.subunits`.
+- Set `source_start` and `source_end` to character offsets in `Document.text`.
+- Keep `fulltext` faithful: no summaries, rewrites, translations, or invented
+  text.
+- Store provenance in `attributes` or `metadata`: page, heading path, section
+  number, table id, figure id, citation, source filename.
+- Mark heuristic extraction in metadata instead of pretending it is certain.
+- Do not create `Unit` inside a unit parser.
+
+## Chunk Parser Best Practices
+
+Chunk parsers should make retrieval accurate and citable.
+
+- Chunk inside one `Unit`; do not mix unrelated units.
+- Include parent context in chunk text or attributes: `temporal_topic`, section
+  number, citation, and source path.
+- Keep chunks semantically complete: definition, rule, exception, method,
+  condition, table row, or procedure step.
+- Do not split controlling language from its exception or limitation.
+- Use attributes for filters: document type, jurisdiction, effective date,
+  section number, party, topic, method, dataset, citation.
+- Include `parser_name`, `unit_type`, `hierarchy`, and `role` when available.
+- Keep source text grounded. Add labels only when they improve citation without
+  changing meaning.
+- Prefer fewer useful chunks over many tiny fragments.
+
+## Anti-Patterns
+
+- One parser accepts every document.
+- `parser_can_handle_document()` returns `True` for any non-empty text.
+- Fallback to one document-sized unit or chunk.
+- Chunking by text length before parsing structure.
+- Using metadata instead of `ParsedUnit.subunits` for hierarchy.
+- Assuming every document has the same format.
+
+## Production Checklist
+
+- Each parser documents what it accepts.
+- Parser acceptance does not overlap.
+- Documents with no parser are visible and reviewed.
+- `unit_type` covers the domain.
+- Domain roles are represented: definition, penalty, duty, prohibition,
+  transitory provision, or equivalents.
+- Chunk attributes carry `parser_name`, `unit_type`, `hierarchy`, and `role`.
+- No default parser and no catch-all parser is selected for production.
+
+## Academic And Practical Accuracy
+
+For scholarly, legal, policy, and technical RAG, accuracy means retrieval can be
+audited back to source evidence.
+
+- Preserve source file, page, section, heading path, citation, table, and figure
+  references when available.
+- Separate extraction from interpretation. Parsers extract; downstream systems
+  summarize or reason.
+- Preserve version context: date, edition, amendment, jurisdiction, policy
+  version, publication year.
+- Make boundaries match the argument or rule in the document.
+- Test with real domain questions and inspect saved unit/chunk JSON.
+- Use parser coverage in `.right-rag-state.json` as the development loop after
+  a full pipeline run.
+
+## Parser Coverage
+
+After `uv run right-rag index`, each unit parser state records:
+
+- `document_chars`
+- `covered_chars`
+- `uncovered_chars`
+- `coverage_percent`
+- `units_total`
+- `units_with_source_span`
+- merged `source_ranges`
+
+Coverage is computed from all active units and subunits with source spans. Low
+coverage means the parser missed source text. Very high coverage with very few
+units may mean the units are too broad.
+
+## Rules
+
+- Keep changes small and local.
+- Prefer editing `spec/` over core files.
+- Use stdlib before adding dependencies.
+- Add the smallest runnable check for non-trivial parser logic.
+- Do not commit generated `output/`, `.right-rag-state.json`, or local source
+  documents.
+
+## Checks
+
+```powershell
+python -m compileall src/right_rag
+python -m unittest discover -s tests
+uv run python -c "from right_rag.registry import build_registry; r=build_registry(); print(r.document_loaders, r.unit_parsers, r.chunk_parsers, r.unit_savers, r.chunk_savers)"
+```
+
+The test suite covers model conversion and change replay, parser coverage,
+state serialization, registry discovery, and default plugins.