pg-raggraph 0.3.0a2__tar.gz

pg_raggraph-0.3.0a2/.gitignore

@@ -0,0 +1,31 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+.eggs/
+*.egg
+.venv/
+venv/
+.env
+.pytest_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+skill-output/
+.autonomy/
+.claude/
+.worktrees/
+*.db
+
+# Downloaded benchmark corpora — too large for git, regenerated by scripts
+benchmarks/hotpotqa/
+benchmarks/postgres-docs/
+benchmarks/scotus/
+benchmarks/kg-rag-eval/
+benchmarks/*.json
+benchmarks/*.log
+benchmarks/age-bakeoff/corpora/msgraph-work/
+benchmarks/python-versioned-docs/_tmp/
+benchmarks/medical-hrt/_tmp/

pg_raggraph-0.3.0a2/CHANGELOG.md

@@ -0,0 +1,210 @@
+# Changelog
+
+## 0.3.0a2 — 2026-05-02 (pre-public-push hardening)
+
+Second prod-ready audit pass on top of `0.3.0a1`, ahead of the public-repo
+flip + first real PyPI release. Five PRs closed (PR-301..PR-305) plus a
+fix for an evolution-tracking bug surfaced during test hardening.
+
+### Security
+
+- **PR-301 — Bearer auth uses constant-time compare.** `pgrg serve`'s
+  optional `PGRG_SERVER_API_KEY` Bearer middleware now uses
+  `secrets.compare_digest` instead of `!=`. The previous comparison
+  short-circuited on the first differing byte and leaked the key
+  length + prefix via response timing — bypassable by a network
+  attacker over thousands of probes. Added regression-lock test that
+  spies on `secrets.compare_digest` and asserts the auth path actually
+  invokes it (catches a future revert to `==` directly).
+- **PR-303 — Defense-in-depth security headers.** New middleware
+  attaches `Content-Security-Policy`, `X-Content-Type-Options: nosniff`,
+  `X-Frame-Options: DENY`, and `Referrer-Policy: no-referrer` to every
+  response — including the 401/403 short-circuits from the auth
+  middleware. CSP allows `https://unpkg.com` for the bundled
+  `vis-network` UI; tighten to `'self'` once the JS is bundled locally.
+- **PR-304 — MCP `pgrg_ingest` enforces the file-extension allowlist.**
+  Hoisted the canonical extension list to `pg_raggraph.INGEST_ALLOWED_EXTS`
+  so the FastAPI `/ingest` endpoint, the MCP `pgrg_ingest` tool, and
+  the library's directory walker all share one source. An LLM agent
+  that asks the MCP server to ingest a `.exe`, `.so`, `.tar`, etc.
+  now gets a structured `{"error": "unsupported_extension", ...}`
+  response — no garbage entities polluting the knowledge graph.
+
+### Packaging / DX
+
+- **PR-302 — `[project.urls]` table on PyPI.** `pyproject.toml` now
+  declares Homepage, Repository, Issues, Changelog, and Documentation
+  URLs. The PyPI project page sidebar surfaces these — without them
+  the listing was barebones, a permanent first-impression tax.
+- **PR-305 — Reranker actionable ImportError.** When fastembed's
+  cross-encoder submodule isn't available, `FastEmbedReranker._load()`
+  now raises `ImportError` with a `pip install --upgrade 'fastembed>=0.4'`
+  hint instead of letting a bare `ModuleNotFoundError` propagate. Matches
+  the pattern used for the chunkshop integration in `chunking.py`.
+
+### Bug fixes
+
+- **datetime metadata no longer crashes ingest.** `rag.ingest(metadata={...})`
+  with `effective_from` / `effective_to` `datetime` values previously
+  failed with `TypeError: Object of type datetime is not JSON serializable`
+  in the `documents.metadata` JSONB path (and similarly for chunk
+  metadata in the chunkshop pre-chunked path). Added a `_json_default`
+  helper that serializes datetimes as ISO 8601 strings — queryable from
+  JSONB via `metadata->>'effective_from'`. Fixed 5 evolution-tier1
+  integration tests that were silently failing on `main` since the
+  `22d83f7` documents-metadata-persistence change.
+
+### Tests
+
+- 13 new tests in `tests/integration/test_error_paths.py` covering
+  PR-301..PR-304 (Bearer auth contract, security-header presence on
+  success + auth-failure paths, MCP `pgrg_ingest` extension rejection
+  and partial-state guard, public `INGEST_ALLOWED_EXTS` import).
+- New `tests/unit/test_reranker.py` (2 tests) covering PR-305.
+- All 204 tests pass on the full suite.
+
+### CI / hygiene
+
+- Cleared the lint+format failures that had been silently red on `main`
+  for several pushes. `ruff check .` and `ruff format --check .` are
+  now both green. Two new excludes added: `benchmarks/sales-crm-demo/`
+  (cookbook demo with its own SQL-heavy conventions, matches the prior
+  `benchmarks/age-bakeoff/` precedent) and `docs/cookbook/samples/*.py`
+  (documentation/demo scripts). One auto-fixed import sort in
+  `chunking.py`; one trimmed docstring example in `__init__.py`.
+
+## 0.3.0a1 — 2026-04-28 (post-audit hardening)
+
+First public PyPI release. Polish + hardening pass on top of `0.3.0a0`. No public-API changes; all 23 production-readiness items from the prod-ready audit closed (22 fixed, 1 false positive). Library + server ready for external use.
+
+### Real-world Tier 1 benchmarks
+
+- **`benchmarks/python-versioned-docs/`** — 12 Python docs (3.10/3.11/3.12), 1364 chunks, 15 hand-written gold questions. **13/13 perfect `version_filter` purity** (100%). Closes the "Tier 1 only synthetic-fixture-tested" gap.
+- **`benchmarks/medical-hrt/`** — 48 PubMed HRT/CV abstracts, 7 epistemically-retracted (modeling WHI 2002 supersession of the prior consensus), 15 gold questions. **5/5 retraction_aware + 5/5 time_travel = 15/15 perfect.** First real-world demonstration of `retracted_behavior="hide"` + `as_of`.
+- 3-part dev-rel blog series in [`docs/blog/`](docs/blog/) walking through both paths from a fresh `git clone`.
+- [`docs/USE-CASES.md`](docs/USE-CASES.md) — decision matrix for classic GraphRAG vs evolving knowledge.
+
+### Server hardening (PR-103, PR-104, PR-205, PR-208)
+
+- **`/graph` pagination** — default `LIMIT 500`, `?limit=N` (max 5000), `?limit=all` for tiny corpora. No more browser OOM on real-corpus visualization.
+- **`/ingest` hardening** — `PGRG_SERVER_MAX_UPLOAD_MB` cap (default 100 MB → 413), extension allowlist (→ 415), filename sanitization (path-traversal-safe, → 400 on empty), temp-file cleanup wrapped in `try/finally` so leaks are impossible.
+- **Optional Bearer auth** — `PGRG_SERVER_API_KEY` env enables auth middleware. Server logs a startup WARN when the env is unset so the unauthenticated state is loud, not silent. `/health` and `/ready` always probe-friendly.
+- **Origin allowlist** — `PGRG_SERVER_ALLOWED_ORIGINS` (comma-separated). When unset, only loopback Origins accepted on POST/PUT/DELETE/PATCH; non-browser clients (curl, requests) without Origin headers still work.
+- **`/ready` endpoint** — distinct from `/health`. Verifies DB connectivity AND `pgrg_meta.schema_version >= SCHEMA_VERSION`. Returns 503 with a structured payload on `db_unreachable` / `schema_pending_migration`.
+- **`/query` default mode** — was `hybrid` (the slowest mode); now `smart` (matches `/ask`).
+
+### Library hardening (PR-203, PR-206, PR-209, PR-210, PR-211, PR-215, PR-216)
+
+- `pg_raggraph.__version__` now resolved via `importlib.metadata.version("pg-raggraph")` so it always matches the installed metadata (no more "0.3.0" string drift from `0.3.0a0` in pyproject).
+- `PGRGConfig` refuses to start with the default Postgres credentials when `PGRG_ENV=production` (raises `RuntimeError`); logs a one-time WARN otherwise.
+- `tune_scoring_weights()` gains a `max_grid_size` parameter (default 50) — refuses to run grids exceeding the cap before any LLM call. Cost-safety guard.
+- `rag.request_shutdown()` for graceful drain of long-running ingest. SIGTERM/SIGINT handlers can wire it; in-flight per-doc transactions finish, queued files become no-ops counted as `skipped`. Re-running `ingest()` resumes via content-hash dedup.
+- `PGRG_LOG_FORMAT=json` — stdlib-only structured logging on stderr (no extra dep). Activates only when no handlers are pre-attached to the `pg_raggraph` logger.
+- `os.nice()` no longer mutates process priority on `PGRGConfig` import. New `apply_nice_level()` method called from `ingest()` where CPU-yield was actually wanted.
+- `ingest_profile` and `extraction_prompt` typed as `Literal[...]` — typos via env now raise `ValidationError` at init instead of silently defaulting.
+
+### Renamed: `skimr_spacy` → `lede_spacy`
+
+The Tier-2 fact-extractor enum value was renamed to match the package's PyPI name (shipped as `lede` + `lede-spacy` 2026-04-28). Active surfaces updated: `PGRGConfig.fact_extractor` Literal, schema comment, user-guide, cookbook. Released migration `002_evolution_tracking.sql` and dated audit-trail specs under `docs/superpowers/` left untouched per project policy.
+
+### CI
+
+- **CI fixed.** `.github/workflows/test.yml` was running `pytest tests/integration/ tests/test_e2e.py -v`, but `tests/test_e2e.py` was moved to `tests/integration/test_e2e.py` in the alpha merge — pytest exited 5 (no tests at the explicit path) on every push since 2026-04-27. Removed the stale path.
+- `benchmarks/age-bakeoff` excluded from root ruff config — separate sub-project with its own `pyproject.toml` and lint posture.
+- All 195 tests passing across `tests/unit/` and `tests/integration/` on Python 3.12 and 3.13.
+
+### Tests
+
+- New `tests/integration/test_error_paths.py` (15 tests) — asserts specific exception types or behaviors on bad DSN, naive `as_of`, oversize `/ingest`, path-traversal filenames, `tune_scoring_weights` cost guard, namespace allowlist, etc.
+- Latency-test thresholds widened from `< 200 ms` to `< 1500 ms` — these were flaking under cold-start CI / contended dev machines despite no real perf regression. Tight perf gating belongs in the dedicated bake-off harness, not in user-journey tests.
+- `test_07_bus_factor` xfail removed — replaced the empirically-flaky directional assertion (`hybrid_score >= naive_score`) with a property-style check (both modes return ≥ 1 expected keyword).
+
+### Docs
+
+- README rewritten with layered structure (what / why / how → weeds).
+- New [`docs/EVOLUTION-API-QUICKREF.md`](docs/EVOLUTION-API-QUICKREF.md) — common assumptions vs reality for the Tier 1 API (which kwargs are per-query vs config-only, how to read evolution columns, `as_of` × `retracted_at` semantics).
+- `docs/user-guide.md` gains "Schema migrations", "Concurrency / sizing", "Logging", and "Graceful shutdown" subsections.
+- README quickstart switched from `pip install pg-raggraph` (not yet on PyPI) to a clone-based install that actually works.
+- `pgrg serve` now carries an explicit "deploy behind auth, do not expose publicly" banner in README + user-guide.
+
+### Dependency / supply-chain
+
+- `pip-audit --skip-editable` is clean: zero CVEs in any direct or transitive dependency.
+
+## 0.3.0-alpha — 2026-04-25
+
+### Added
+
+- **Evolving-knowledge RAG, Tier 1 (Structural).** Opt-in evolution tracking
+  that respects document effective-dates, retractions, and supersession at
+  the document level. Opt in via `PGRGConfig(evolution_tier="structural")`
+  or env `PGRG_EVOLUTION_TIER=structural`.
+- `rag.ingest(metadata={...})` now accepts `effective_from`, `effective_to`,
+  `retracted`, `retracted_at`, `retraction_reason`, `version_label`,
+  `supersedes_document_id`. Per-ingest scope (applies to every file in the
+  call).
+- `rag.query()` new kwargs: `as_of=datetime(...)` time-travel filter,
+  `version_filter="..."` version restriction, `evolution_aware=False`
+  per-call override to force classic retrieval.
+- `rag.tune_scoring_weights(namespace, gold, grid, ...)` grid-search
+  utility for per-corpus weight tuning. Writes the best cell back to
+  `rag.config`.
+- Schema: three new tables (`facts`, `fact_edges`, `document_versions`)
+  and four new columns on `documents` via migration
+  `002_evolution_tracking.sql`. All additive; fact-level tables stay empty
+  at Tier 1.
+- Behavior modes: `retracted_behavior` ∈ {hide, flag, surface_both};
+  `supersession_behavior` ∈ {hide, prefer_new, surface_both}.
+
+### Changed
+
+- `PGRGConfig` gains 15+ fields for evolution tracking. Defaults leave
+  Tier 0 behavior unchanged.
+- Retrieval SQL templates (`naive`, `local`, `global`) are now built
+  per-query from the config rather than stored as string constants. When
+  `evolution_tier="off"`, the generated SQL is semantically identical to
+  the prior version.
+
+### Deferred to future tiers
+
+- Fact-level extraction (Tier 2).
+- LLM-inferred fact edges and contradiction detection (Tier 3).
+- Async slow-path fact-edge inference (Tier 3).
+
+See `docs/cookbook/evolution-tracking.md` for the quickstart.
+
+## 2026-04-20 — `chunk_strategy="hierarchy"` opt-in chunker
+
+### Added
+
+- **`chunk_strategy="hierarchy"`** — heading-prefixed chunker ported from the AGE bake-off (`benchmarks/age-bakeoff/src/age_bakeoff/chunker.py:_split_hierarchy`). Each section body is prefixed with its markdown heading so pgvector embeds `heading+body` as one unit. When a document has no headings, the body is prefixed with a derived title (first H1, else source filename). No token-budget split — sections over `chunk_max_tokens` are passed through unchanged and get truncated at embed time, mirroring the benchmarked behavior byte-for-byte.
+- **When to use it:** corpora with concrete, per-doc disambiguating titles — SCOTUS-style case names ("Miranda v. Arizona"), article titles, product names. On the SCOTUS corpus this cleared DC-003 by 2.5× across all six retrieval modes (`benchmarks/age-bakeoff/results/REPORT-VERDICT.md` §6).
+- **When NOT to use it:** corpora with format-string titles that repeat across docs — meeting updates ("Weekly sync: …"), ticket prefixes, templated status reports. The acme replication on that shape regressed −1 to −2 questions per retrieval mode and tripled hallucinations (`benchmarks/age-bakeoff/results/ACME-HIER-REPLICATION.md`).
+- Default `chunk_strategy` remains `"auto"`. This is an opt-in config, not a behavior change for existing users.
+
+## 2026-04-17 — AGE Bake-Off Benchmark (v0.3.1)
+
+### Added
+
+- **AGE vs pg-raggraph bake-off benchmark** (`benchmarks/age-bakeoff/`) — reproducible head-to-head comparison on two corpora (Acme Labs + SCOTUS) measuring retrieval latency, answer quality (LLM judge), and fact recall across 60 gold-labeled questions.
+- **Benchmark results:**
+  - pg-raggraph retrieval is **1.4x faster on Acme** (33ms vs 47ms p50) and **47x faster on SCOTUS** (60ms vs 2,863ms p50)
+  - Answer quality roughly comparable; AGE slightly better on Acme (zero hallucinations vs 3), tied on SCOTUS
+  - Full pipeline: shared chunker, engine adapters, runner, fact-recall scorer, LLM judge (gpt-4.1-mini), deterministic report generator
+- **`docs/why-not-apache-age.md`** — user-facing guide distilled from the research doc, now updated with measured bake-off numbers replacing cited third-party benchmarks
+- **70 passing tests** for the benchmark suite (all mocked, no external API calls in test suite)
+- **Docker stack** with both engines side-by-side (pgvector/pg16 on 5434, AGE+pgvector on 5435)
+- **CLI** (`age-bakeoff ingest|run|judge|report`) for one-command reproduction
+
+### Fixed
+
+- Entity INSERT uses `ON CONFLICT` for corpora with duplicate entity names (SCOTUS has duplicate case names)
+- `relationship_chunks` linking scoped to relevant chunks only (was O(R*C) = 3.5M INSERTs for SCOTUS; now O(R*matches))
+- Chunker `_split_plain` hard-splits oversized paragraphs (was silently emitting chunks > MAX_CHARS)
+- `BakeoffConfig` strips/rejects whitespace-only `OPENAI_API_KEY`
+
+### Infrastructure
+
+- Postgres REL_16_5 executor+planner slice cloned via sparse-checkout (116 .c files) for the code corpus (pg-src questions written, extraction pipeline ready, run deferred to next session)
+- Acme seed data: 42 entities, 103 relationships, 160 documents mirrored from graphrag-demo
+- SCOTUS seed data: 416 entities, 4,397 relationships, 772 documents mirrored from graphrag-demo

pg_raggraph-0.3.0a2/CODE_OF_CONDUCT.md

@@ -0,0 +1,33 @@
+# Code of Conduct
+
+This project adopts the **Contributor Covenant v2.1** as its code of conduct. The canonical text is maintained at:
+
+- <https://www.contributor-covenant.org/version/2/1/code_of_conduct/>
+
+All contributors, maintainers, and participants in project spaces (GitHub issues, pull requests, discussions, any associated chat channels) are expected to follow it.
+
+## Scope
+
+This code applies within all project spaces and also applies when an individual is officially representing the project in public spaces.
+
+## Reporting
+
+Instances of behavior that violates the Contributor Covenant may be reported to the project maintainer at **matt@theyonk.com**. All reports will be reviewed and investigated promptly and fairly. Reporter identity is kept confidential.
+
+## Enforcement
+
+The maintainer is responsible for clarifying and enforcing the standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior they deem inappropriate, threatening, offensive, or harmful.
+
+Enforcement may include:
+
+1. A private warning with an explanation of what needs to change and why.
+2. A public warning on the relevant issue or PR.
+3. A temporary or permanent ban from project spaces.
+
+The severity of the response is guided by the Contributor Covenant's enforcement guidelines at:
+
+- <https://www.contributor-covenant.org/version/2/1/code_of_conduct/#enforcement-guidelines>
+
+## Attribution
+
+This Code of Conduct is adopted from the [Contributor Covenant](https://www.contributor-covenant.org/), version 2.1, available at the URL above. The Contributor Covenant is licensed under CC BY 4.0.

pg_raggraph-0.3.0a2/CONTRIBUTING.md

@@ -0,0 +1,68 @@
+# Contributing to pg-raggraph
+
+Thanks for considering a contribution. This is a small, focused library and we want to keep it that way — clear code, honest benchmarks, and a low barrier to reading the whole thing in a sitting.
+
+## What kinds of changes we welcome
+
+- **Bug reports and fixes.** File an issue first for anything non-obvious so we can agree on the shape of the fix.
+- **New chunkers, embedders, or retrieval modes** — ship them behind a config flag, with a test, and a benchmark-table entry showing the tradeoff.
+- **Documentation fixes and clarifications.** Always welcome.
+- **Benchmark extensions.** New corpora, new question sets, new metrics. Evidence is the point.
+
+## What to check before opening a PR
+
+- Tests pass: `uv run pytest`
+- Lint clean: `uv run ruff check . && uv run ruff format --check .`
+- New behavior has a test. New config knobs are documented in `docs/user-guide.md` and the `README.md` config table.
+- Benchmark numbers in the PR description cite a raw result file, not a summary paragraph.
+
+## Local development setup
+
+```bash
+# 1. Clone and enter the repo
+git clone https://github.com/<you>/pg-raggraph.git
+cd pg-raggraph
+
+# 2. Install dependencies (Python 3.12+ required)
+uv sync --all-extras
+
+# 3. Start PostgreSQL with pgvector + pg_trgm
+docker compose up -d
+
+# 4. Copy env examples and fill in keys
+cp .env.example .env
+cp benchmarks/age-bakeoff/.env.example benchmarks/age-bakeoff/.env
+# edit both files — see README for the full config table
+
+# 5. Run the test suite
+uv run pytest                    # all tests (needs DB up)
+uv run pytest tests/unit/        # just unit (no DB needed)
+uv run pytest tests/integration/ # integration (needs DB up)
+```
+
+## Code style
+
+- **Python 3.12+, async-first.** All database operations use `asyncpg` / `psycopg` async.
+- **Ruff for lint + format.** We match `pyproject.toml` settings; don't reformat with a different tool.
+- **Small, focused PRs.** One logical change per PR. Bug fix ≠ refactor ≠ feature; split them.
+- **Comments are rare.** Name things well enough that most comments are unnecessary. When a comment is warranted, explain *why*, not *what*.
+- **Tests tell the story.** If the behavior isn't obvious from a test, the test is unclear.
+
+## Commit and PR expectations
+
+- Commit messages follow the existing repo style — see `git log --oneline -20` for examples. A one-line summary with a short imperative verb (`feat:`, `fix:`, `docs:`, `test:`) followed by a body explaining *why* the change matters.
+- PR titles should finish the sentence "This PR ...". Bodies should explain the user-facing effect and link any relevant issue.
+- PRs that change benchmark or library defaults must include before/after numbers from a real run.
+- Co-authored-by lines are welcome.
+
+## When in doubt
+
+Open a draft PR early or start a discussion issue. Aligning on approach before you've written a lot of code saves everyone time.
+
+## Code of conduct
+
+This project follows the [Contributor Covenant](CODE_OF_CONDUCT.md). Please read it before opening an issue or PR.
+
+## Security
+
+See [SECURITY.md](SECURITY.md) for how to report vulnerabilities.

pg_raggraph-0.3.0a2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 yonk-tools
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.