docugraph 0.1.1__tar.gz → 0.2.1__tar.gz

docugraph-0.2.1/.github/workflows/bench.yml

@@ -0,0 +1,79 @@
+# The retrieval benchmark, run against the real embedder, weekly and on demand.
+#
+# Fast CI proves the harness with a deterministic fake embedder
+# (tests/integration/test_bench_runner.py) and never downloads weights. What
+# that leaves unexercised is the thing the benchmark exists for: the numbers
+# on the shipped model. This job runs `docugraph bench run` on the real
+# bge-small-en-v1.5 (~130 MB, cached between runs) and holds the result
+# against the committed baseline with `docugraph bench compare`, which exits 1
+# when nDCG@10 -- overall or in any single failure-mode category -- drops by
+# more than the tolerance.
+#
+# The baseline under benchmarks/retrieval/baselines/ is committed, per
+# embedding model, and regenerated deliberately: when the dataset or the
+# retrieval stack changes on purpose, a human reruns the benchmark, reads the
+# delta table, and commits the new baseline with the change that moved it.
+# Until a baseline exists for the model, this job fails -- a comparison
+# against nothing proving nothing would be green that means "checked nothing"
+# (the live-reranker header has the longer form of this argument).
+#
+# Reranking arms are not measured here: the cross-encoder is ~2.3 GB and has
+# its own opt-in workflow; run `docugraph bench run --rerank` locally when a
+# reranker change needs numbers.
+name: Retrieval benchmark
+
+on:
+  # Manual: Actions -> Retrieval benchmark -> Run workflow.
+  workflow_dispatch:
+  schedule:
+    # Weekly, Monday 06:17 UTC. An off-hour minute, deliberately: scheduled
+    # workflows on :00 queue behind everyone else's.
+    - cron: "17 6 * * 1"
+
+env:
+  FASTEMBED_CACHE_PATH: /home/runner/.cache/fastembed
+
+jobs:
+  bench:
+    name: bench run · compare to baseline
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Sync dependencies
+        run: uv sync --all-extras
+
+      - name: Cache embedding model
+        uses: actions/cache@v4
+        with:
+          path: /home/runner/.cache/fastembed
+          key: fastembed-bge-small-en-v1.5
+
+      - name: Run benchmark
+        run: uv run docugraph bench run --json bench-report.json
+
+      - name: Compare against committed baseline
+        run: |
+          baseline="benchmarks/retrieval/baselines/bge-small-en-v1.5.json"
+          if [ ! -f "$baseline" ]; then
+            echo "::error::No committed baseline at $baseline. Run 'docugraph bench run --json' locally and commit the report as the baseline; a compare against nothing is not a gate."
+            exit 1
+          fi
+          uv run docugraph bench compare "$baseline" bench-report.json
+
+      - name: Upload report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: bench-report
+          path: bench-report.json
+          if-no-files-found: ignore

{docugraph-0.1.1 → docugraph-0.2.1}/.github/workflows/ci.yml

@@ -163,9 +163,14 @@ jobs:
       - name: Assert the handshake
         run: |
           python3 - <<'PY'
-          import json, pathlib, sys
-
-          EXPECTED = "0.1.1"  # the version this repo publishes; pinned deliberately
+          import json, pathlib, sys, tomllib
+
+          # Read the expected version from pyproject.toml rather than pinning a
+          # literal. pyproject is the single source of the version -- the
+          # package, the CLI and this handshake all derive from it -- so a
+          # hardcoded constant only goes stale at the next release, which is
+          # exactly how it broke: it stayed "0.1.1" into the 0.2.0 release.
+          EXPECTED = tomllib.loads(pathlib.Path("pyproject.toml").read_text())["project"]["version"]
           raw = pathlib.Path("/tmp/handshake.jsonl").read_text()
 
           # stdout carries the JSON-RPC stream and nothing else. A stray line

docugraph-0.2.1/CHANGELOG.md

@@ -0,0 +1,117 @@
+# Changelog
+
+Notable changes to docugraph. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
+versions follow [SemVer](https://semver.org/spec/v2.0.0.html).
+
+docugraph is pre-1.0: a minor release may change a tool's behaviour or a setting's meaning, and a
+patch release will not. Breaking changes are listed as breaking either way, because a reader of this
+file is entitled to know which of their own setup stops working. The supported interfaces are the MCP
+tools, the CLI commands and the configuration settings; the Python modules behind them are not a
+public API.
+
+## [Unreleased]
+
+## [0.2.1] - 2026-11-07
+
+### Security
+
+- **`crawl_url` and `index_git` now judge model-supplied URLs before acting.** Pages must be
+  `http`/`https`; git remotes must be `https`, `ssh`, or `user@host:path`. A value beginning with `-`
+  is refused as `option_like` rather than passed to git. A host is refused unless it resolves to a
+  global address (`private_address` covers loopback, link-local, `0.0.0.0` and the metadata
+  address), unless `crawler.allow_private_networks` is on. `crawler.allowed_domains` narrows what may
+  be reached. A refusal is a tool error whose text is `Refused: <reason> — <detail>`.
+- **Redirects are enforced hop by hop.** Chromium follows a redirect without re-entering a Playwright
+  route handler, so each `Location` is resolved and judged before the next request is made. A page
+  whose final URL is refused is dropped and reported (`Dropped N page(s) by policy:`) and never
+  indexed.
+- **Service workers cannot bypass the route handler.** The page-context hook removes
+  `navigator.serviceWorker` in every frame before page scripts run. Websockets are closed. Routing is
+  registered on the browser context, so popups, frames and workers are judged by the same handler.
+- **crawl4ai's anti-bot replay cannot escape the guards.** Its retry navigates through the same hooked
+  context and is routed; its browser-less `fallback_fetch_function` would run outside every guard, so
+  the retry and the fallback are pinned off in both crawl paths.
+- **An existing clone is updated only from the origin that was judged.** When the clone destination
+  already exists, its `origin` must equal the requested URL and pass the same policy; a mismatch is
+  refused as `path_escape`. Pull and the fetch fallback carry the same protocol and redirect lockdown
+  as the clone.
+- **`index_git` cannot write outside the repository directory.** The name — or the one derived from
+  the URL — must be a single path component, and the resolved destination must stay inside the
+  configured repos directory. An escaping name is refused as `path_escape`, not renamed.
+- **Git runs hardened under the policy.** `--` precedes the URL and path, refs are validated against
+  `git-check-ref-format`, and clones carry `protocol.file.allow=never`, `protocol.ext.allow=never`
+  and `http.followRedirects=false`.
+- **A site crawl is capped at 50 pages per MCP tool call**, and `max_pages` below 1 or not an integer
+  is an error rather than a crawl.
+- **Graphiti telemetry is off by default.** `GRAPHITI_TELEMETRY_ENABLED` is set to `false` when
+  absent; an explicit value is respected.
+
+### Fixed
+
+- `docugraph index git` now exits non-zero with `Error: Refused: …` when a repository name or ref is
+  refused. Previously a refused path reached the generic `RuntimeError` branch, printed a note, and
+  exited 0.
+
+### Changed
+
+- `CrawlerConfig` gains `allow_private_networks` (default `false`) and `allowed_domains` (default
+  empty).
+- `crawl_site` now counts pages it **attempts**, not pages that produced a document, so a page that
+  fails to scrape still consumes the budget. This changes the CLI too: `docugraph crawl --max-pages`
+  now limits attempted pages.
+- Under a URL policy the crawl cache is not read, so a page cached by an earlier unpoliced crawl is
+  never served without its final URL being judged. Cached documents are still written.
+
+## [0.2.0] - 2026-10-23
+
+### Added
+
+- A retrieval-quality benchmark with a committed corpus and a regression gate. It scores
+  vector-only, BM25-only, LanceDB-native hybrid, RRF and weighted arms on the same queries and fails
+  when nDCG@10 drops past the tolerance.
+
+### Fixed
+
+- The full-text index was never built, so keyword search silently returned nothing; it is now built
+  and BM25 is real. The hybrid path records when it falls back to the vector leg.
+
+## [0.1.1] - 2026-10-08
+
+Maintenance release — no changes to the package's code or behaviour.
+
+### Fixed
+
+- The release workflow publishes the container image,
+  `ghcr.io/cj-coding-apps/docugraph:v0.1.1` (and `:latest`). 0.1.0 has no image: the workflow built
+  the tag from the repository name with its original capitalisation, which GHCR rejects.
+
+### Added
+
+- `pyproject.toml` declares project URLs, so the PyPI page links to the repository, issues and
+  releases.
+
+## [0.1.0] - 2026-10-08
+
+First public release, on PyPI.
+
+### Added
+
+- Documentation RAG for coding agents: crawl or index documentation, build a documentation knowledge
+  graph and a repository/branch-scoped agent memory, and serve all of it to MCP clients over stdio.
+  The surface is nine tools (`crawl_url`, `index_git`, `search_docs`, `hybrid_search`, `graph_add`,
+  `graph_query`, `memory_store`, `memory_recall`, `get_stats`) and a CLI for out-of-band index
+  bootstrapping.
+- Embeddings run in-process on ONNX: `embeddings.provider: auto` resolves to fastembed and never
+  consults the environment, so documents are not sent anywhere to be embedded unless a cloud provider
+  is named.
+- Optional extras: `cloud` (OpenAI, Anthropic and Cohere SDKs), `ollama` (local LLM) and `dev`
+  (test and lint tooling).
+
+### Notes
+
+- Python 3.11–3.13 on Linux. `kuzu` is pinned `==0.11.3`, the final release upstream, and is treated
+  as a migration boundary rather than a floor.
+- URL crawling needs Playwright's Chromium (`playwright install chromium`).
+
+[Unreleased]: https://github.com/CJ-coding-apps/docugraph/compare/v0.2.1...HEAD
+[0.2.1]: https://github.com/CJ-coding-apps/docugraph/compare/v0.2.0...v0.2.1

{docugraph-0.1.1 → docugraph-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: docugraph
-Version: 0.1.1
+Version: 0.2.1
 Summary: Documentation RAG for coding agents: vector search, knowledge graph, agent memory, and an MCP server — torch-free local embeddings via fastembed
 Project-URL: Homepage, https://github.com/CJ-coding-apps/docugraph
 Project-URL: Repository, https://github.com/CJ-coding-apps/docugraph
@@ -78,6 +78,7 @@ Code and other MCP clients. The CLI is for out-of-band index bootstrapping
 - **Agent memory**: persistent, repository/branch-scoped key-value store
 - **Optional reranking**: local multilingual cross-encoder (BAAI/bge-reranker-v2-m3), opt-in
 - **Torch-free**: default embeddings and reranking run on ONNX — no PyTorch in the dependency tree
+- **Measured retrieval**: a committed adversarial benchmark scores every search mode per failure category, with a regression gate (`docugraph bench`)
 
 ## Quick start
 
@@ -123,6 +124,16 @@ playwright install chromium
 This is ~190 MB and is cached, so it is a one-off. `index local`, `index git`,
 `search`, `memory` and `stats` do not need it — only fetching a URL does.
 
+**URLs that an agent supplies are treated as untrusted.** `crawl_url` and
+`index_git` refuse, before fetching or cloning, a value that is not `http(s)`
+(or, for git, `https`/`ssh`), that begins with `-`, that is not a public
+address, that leaves an allowlist you set, or that would write outside the
+repository directory. Refusals are one line — `Refused: private_address — …` —
+and appear in the tool result with `isError`. A site crawl is capped at 50 pages
+per call. Set `crawler.allow_private_networks: true` to crawl an intranet, and
+`crawler.allowed_domains` to narrow what may be reached; see
+[docs/configuration.md](docs/configuration.md#url-policy).
+
 ### MCP integration with Claude Code (primary surface)
 
 The MCP server is a console script, so where it lives depends on how you
@@ -285,8 +296,13 @@ crawler:
   max_concurrent: 5
   rate_limit: 2.0
   respect_robots: true
+  allow_private_networks: false   # opt in to intranet/loopback targets
+  allowed_domains: []             # empty = any public host
 ```
 
+Graphiti telemetry is **off** unless you set `GRAPHITI_TELEMETRY_ENABLED`
+yourself; an explicit value is never overridden.
+
 `provider: auto` resolves to **fastembed** (local ONNX) — nothing about your
 environment changes that, so documents are not sent anywhere for embedding
 unless you name a cloud provider. Set `provider: openai` (or `cohere`) to send
@@ -298,6 +314,27 @@ overridden by environment variable, e.g. `DOCUGRAPH_EMBEDDINGS__MODEL` or
 See [docs/configuration.md](https://github.com/CJ-coding-apps/docugraph/blob/main/docs/configuration.md) for the full reference,
 including the embedding-model consistency guard and the graph LLM requirement.
 
+## Network activity
+
+DocuGraph runs locally, but these connections are possible — each only in the situation named:
+
+- **Model downloads.** `fastembed` (the default embedder) downloads its ONNX model from Hugging
+  Face on first use, and the reranker (about 2 GB) when you first enable it. Afterwards the model is
+  read from the local cache. Nothing about your documents is sent.
+- **Crawl and clone targets.** `crawl_url` fetches the pages you point it at, and `index_git`
+  clones the repository URL you give it. The MCP tools treat those values as untrusted: pages must be
+  `http`/`https` and resolve to a global address, git remotes must pass the same check, and
+  `crawler.allow_private_networks` is the opt-in for an intranet or a local docs server.
+- **A named LLM or embedding provider.** Only if you name one: `embeddings.provider: openai` (or
+  `cohere`) sends the text being embedded there, and the graph tools send prompts to the configured
+  LLM — OpenAI, Anthropic, or Ollama.
+- **Ollama**, when configured — to your Ollama server, localhost by default.
+- **Graphiti telemetry**, to PostHog, **off unless you opt in** with
+  `GRAPHITI_TELEMETRY_ENABLED=true`.
+
+CLI crawls and clones are user-directed: they are exempt from the scheme, DNS and address rules,
+but the repository-name containment and the git hardening apply to every caller.
+
 ## The documentation knowledge graph
 
 DocuGraph builds a temporal entity/relationship graph over your docs (Kuzu,
@@ -308,6 +345,47 @@ required**. However, Graphiti performs entity extraction with an **LLM**, so
 OpenAI/Anthropic key). Without one, the graph tools degrade gracefully with a
 clear message; vector and keyword search are unaffected.
 
+## Retrieval benchmark
+
+Retrieval quality is measured, not asserted. `benchmarks/retrieval/` commits a
+hand-authored documentation corpus for a fictional SDK plus graded relevance
+judgments, built adversarially around seven failure modes: exact-token lookups
+(error codes, parameter names), paraphrase queries with no lexical overlap,
+keyword-stuffed distractor pages, near-sibling semantic distractors, version-
+pinned questions, acronym/synonym gaps, and "X without Y" constraint queries.
+Each query records the trap it sets in a `design_note`, and a manifest pins the
+dataset by hash — editing it is a deliberate act that invalidates baselines.
+The corpus holds 70 documents and 56 queries, eight per failure mode; the
+keyword-trap distractors are built to cover many *distinct* query terms rather
+than repeat one (BM25 saturates a repeated term, so a one-token trap does not
+bind), and the semantic-distractor queries share few exact tokens with their
+target so that only meaning — not lexical overlap — can recover them.
+
+```bash
+# Score every arm (vector-only, BM25-only, LanceDB-native hybrid, RRF, weighted)
+# against a temporary store built from the committed corpus:
+docugraph bench run --json report.json
+
+# Hold a report against a committed baseline; exits 1 when nDCG@10 drops by
+# more than the tolerance, overall or in any single category:
+docugraph bench compare benchmarks/retrieval/baselines/bge-small-en-v1.5.json report.json
+```
+
+Reported metrics are recall@1/5/10, MRR@10, and nDCG@10 (graded, linear gain),
+macro-averaged overall and per category. The per-category table is the point:
+a fusion change that wins on average while losing exact-code lookup shows up
+as the regression it is. Baselines are per embedding model and live in
+`benchmarks/retrieval/baselines/`; the `Retrieval benchmark` workflow re-runs
+the suite weekly against the committed baseline.
+
+Honest scope: the corpus is small and purpose-built, so scores are comparable
+across docugraph changes — not with BEIR or any external leaderboard. The
+first thing this benchmark caught predates it: before v0.2.0 no FTS index was
+ever created, so "keyword" search silently returned vector results and
+RRF fused the vector ranking with itself. The benchmark's `fts` arm and the
+hybrid arm's refuse-to-score-on-fallback check exist so that class of silent
+degradation cannot come back unnoticed.
+
 ### KuzuDB version
 
 `kuzu` is pinned to `==0.11.3` in `pyproject.toml` (reached through

{docugraph-0.1.1 → docugraph-0.2.1}/README.md

@@ -16,6 +16,7 @@ Code and other MCP clients. The CLI is for out-of-band index bootstrapping
 - **Agent memory**: persistent, repository/branch-scoped key-value store
 - **Optional reranking**: local multilingual cross-encoder (BAAI/bge-reranker-v2-m3), opt-in
 - **Torch-free**: default embeddings and reranking run on ONNX — no PyTorch in the dependency tree
+- **Measured retrieval**: a committed adversarial benchmark scores every search mode per failure category, with a regression gate (`docugraph bench`)
 
 ## Quick start
 
@@ -61,6 +62,16 @@ playwright install chromium
 This is ~190 MB and is cached, so it is a one-off. `index local`, `index git`,
 `search`, `memory` and `stats` do not need it — only fetching a URL does.
 
+**URLs that an agent supplies are treated as untrusted.** `crawl_url` and
+`index_git` refuse, before fetching or cloning, a value that is not `http(s)`
+(or, for git, `https`/`ssh`), that begins with `-`, that is not a public
+address, that leaves an allowlist you set, or that would write outside the
+repository directory. Refusals are one line — `Refused: private_address — …` —
+and appear in the tool result with `isError`. A site crawl is capped at 50 pages
+per call. Set `crawler.allow_private_networks: true` to crawl an intranet, and
+`crawler.allowed_domains` to narrow what may be reached; see
+[docs/configuration.md](docs/configuration.md#url-policy).
+
 ### MCP integration with Claude Code (primary surface)
 
 The MCP server is a console script, so where it lives depends on how you
@@ -223,8 +234,13 @@ crawler:
   max_concurrent: 5
   rate_limit: 2.0
   respect_robots: true
+  allow_private_networks: false   # opt in to intranet/loopback targets
+  allowed_domains: []             # empty = any public host
 ```
 
+Graphiti telemetry is **off** unless you set `GRAPHITI_TELEMETRY_ENABLED`
+yourself; an explicit value is never overridden.
+
 `provider: auto` resolves to **fastembed** (local ONNX) — nothing about your
 environment changes that, so documents are not sent anywhere for embedding
 unless you name a cloud provider. Set `provider: openai` (or `cohere`) to send
@@ -236,6 +252,27 @@ overridden by environment variable, e.g. `DOCUGRAPH_EMBEDDINGS__MODEL` or
 See [docs/configuration.md](https://github.com/CJ-coding-apps/docugraph/blob/main/docs/configuration.md) for the full reference,
 including the embedding-model consistency guard and the graph LLM requirement.
 
+## Network activity
+
+DocuGraph runs locally, but these connections are possible — each only in the situation named:
+
+- **Model downloads.** `fastembed` (the default embedder) downloads its ONNX model from Hugging
+  Face on first use, and the reranker (about 2 GB) when you first enable it. Afterwards the model is
+  read from the local cache. Nothing about your documents is sent.
+- **Crawl and clone targets.** `crawl_url` fetches the pages you point it at, and `index_git`
+  clones the repository URL you give it. The MCP tools treat those values as untrusted: pages must be
+  `http`/`https` and resolve to a global address, git remotes must pass the same check, and
+  `crawler.allow_private_networks` is the opt-in for an intranet or a local docs server.
+- **A named LLM or embedding provider.** Only if you name one: `embeddings.provider: openai` (or
+  `cohere`) sends the text being embedded there, and the graph tools send prompts to the configured
+  LLM — OpenAI, Anthropic, or Ollama.
+- **Ollama**, when configured — to your Ollama server, localhost by default.
+- **Graphiti telemetry**, to PostHog, **off unless you opt in** with
+  `GRAPHITI_TELEMETRY_ENABLED=true`.
+
+CLI crawls and clones are user-directed: they are exempt from the scheme, DNS and address rules,
+but the repository-name containment and the git hardening apply to every caller.
+
 ## The documentation knowledge graph
 
 DocuGraph builds a temporal entity/relationship graph over your docs (Kuzu,
@@ -246,6 +283,47 @@ required**. However, Graphiti performs entity extraction with an **LLM**, so
 OpenAI/Anthropic key). Without one, the graph tools degrade gracefully with a
 clear message; vector and keyword search are unaffected.
 
+## Retrieval benchmark
+
+Retrieval quality is measured, not asserted. `benchmarks/retrieval/` commits a
+hand-authored documentation corpus for a fictional SDK plus graded relevance
+judgments, built adversarially around seven failure modes: exact-token lookups
+(error codes, parameter names), paraphrase queries with no lexical overlap,
+keyword-stuffed distractor pages, near-sibling semantic distractors, version-
+pinned questions, acronym/synonym gaps, and "X without Y" constraint queries.
+Each query records the trap it sets in a `design_note`, and a manifest pins the
+dataset by hash — editing it is a deliberate act that invalidates baselines.
+The corpus holds 70 documents and 56 queries, eight per failure mode; the
+keyword-trap distractors are built to cover many *distinct* query terms rather
+than repeat one (BM25 saturates a repeated term, so a one-token trap does not
+bind), and the semantic-distractor queries share few exact tokens with their
+target so that only meaning — not lexical overlap — can recover them.
+
+```bash
+# Score every arm (vector-only, BM25-only, LanceDB-native hybrid, RRF, weighted)
+# against a temporary store built from the committed corpus:
+docugraph bench run --json report.json
+
+# Hold a report against a committed baseline; exits 1 when nDCG@10 drops by
+# more than the tolerance, overall or in any single category:
+docugraph bench compare benchmarks/retrieval/baselines/bge-small-en-v1.5.json report.json
+```
+
+Reported metrics are recall@1/5/10, MRR@10, and nDCG@10 (graded, linear gain),
+macro-averaged overall and per category. The per-category table is the point:
+a fusion change that wins on average while losing exact-code lookup shows up
+as the regression it is. Baselines are per embedding model and live in
+`benchmarks/retrieval/baselines/`; the `Retrieval benchmark` workflow re-runs
+the suite weekly against the committed baseline.
+
+Honest scope: the corpus is small and purpose-built, so scores are comparable
+across docugraph changes — not with BEIR or any external leaderboard. The
+first thing this benchmark caught predates it: before v0.2.0 no FTS index was
+ever created, so "keyword" search silently returned vector results and
+RRF fused the vector ranking with itself. The benchmark's `fts` arm and the
+hybrid arm's refuse-to-score-on-fallback check exist so that class of silent
+degradation cannot come back unnoticed.
+
 ### KuzuDB version
 
 `kuzu` is pinned to `==0.11.3` in `pyproject.toml` (reached through