docpull 4.1.0__tar.gz → 4.3.0__tar.gz

{docpull-4.1.0/src/docpull.egg-info → docpull-4.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpull
-Version: 4.1.0
+Version: 4.3.0
 Summary: Pull documentation from the web and convert to clean markdown
 Author-email: Zachary Roth <support@raintree.technology>
 Maintainer-email: Raintree Technology <support@raintree.technology>
@@ -64,6 +64,8 @@ Requires-Dist: python-multipart>=0.0.27; extra == "mcp"
 Requires-Dist: starlette>=1.0.1; extra == "mcp"
 Provides-Extra: parallel
 Requires-Dist: parallel-web>=1.0.1; extra == "parallel"
+Provides-Extra: observability
+Requires-Dist: raindrop-ai>=0.0.50; extra == "observability"
 Provides-Extra: llm
 Requires-Dist: tiktoken>=0.7.0; extra == "llm"
 Provides-Extra: all
@@ -76,6 +78,7 @@ Requires-Dist: pyjwt>=2.13.0; extra == "all"
 Requires-Dist: python-multipart>=0.0.27; extra == "all"
 Requires-Dist: starlette>=1.0.1; extra == "all"
 Requires-Dist: parallel-web>=1.0.1; extra == "all"
+Requires-Dist: raindrop-ai>=0.0.50; extra == "all"
 Provides-Extra: dev
 Requires-Dist: pytest>=7.0.0; extra == "dev"
 Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
@@ -93,11 +96,12 @@ Dynamic: license-file
 
 # docpull
 
-**Security-hardened, browser-free crawler that turns static documentation sites into clean, AI-ready Markdown — fast.**
+**Security-hardened, browser-free web scraper and crawler that turns server-rendered pages into clean, AI-ready Markdown — fast.**
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![PyPI version](https://badge.fury.io/py/docpull.svg)](https://badge.fury.io/py/docpull)
-[![Downloads](https://pepy.tech/badge/docpull)](https://pepy.tech/project/docpull)
+[![PyPI downloads](https://img.shields.io/pepy/dt/docpull?label=downloads)](https://pepy.tech/project/docpull)
+[![GitHub stars](https://img.shields.io/github/stars/raintree-technology/docpull?style=social)](https://github.com/raintree-technology/docpull/stargazers)
 [![License: MIT](https://img.shields.io/github/license/raintree-technology/docpull)](https://github.com/raintree-technology/docpull/blob/main/LICENSE)
 
 <p align="center">
@@ -106,15 +110,37 @@ Dynamic: license-file
   </a>
 </p>
 
-docpull uses async HTTP (not Playwright) to fetch server-rendered pages,
-extracts main content, and writes clean Markdown with source-URL frontmatter —
-in seconds, with a small install footprint. It won't render JavaScript, but for
-the large class of docs that don't need it (API references, Python/Go stdlib,
-most dev-tool docs, OpenAPI specs, Next.js and Docusaurus builds), it is a
-fast, auditable, sandbox-friendly way to pipe documentation into an LLM context,
-a RAG index, or an offline archive. SSRF, XXE, DNS-rebinding, and
-CRLF-injection protections are on by default — a necessity when an AI agent
-is choosing the URLs.
+## Download History
+
+<a href="https://pepy.tech/project/docpull">
+  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
+</a>
+
+## Star History
+
+<a href="https://star-history.com/#raintree-technology/docpull&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+  </picture>
+</a>
+
+docpull is a web scraper for static and server-rendered sites, with
+documentation crawling as its sharpest default workflow. It uses async HTTP (not
+Playwright) to fetch pages, discover links, extract main content, and write clean
+Markdown with source-URL frontmatter — in seconds, with a small install
+footprint. It won't render JavaScript, but for the large class of sites that
+don't need it (API references, vendor docs, Python/Go stdlib, blogs, OpenAPI
+specs, Next.js and Docusaurus builds), it is a fast, auditable,
+sandbox-friendly way to pipe web content into an LLM context, a RAG index, or an
+offline archive. SSRF, XXE, DNS-rebinding, and CRLF-injection protections are on
+by default — a necessity when an AI agent is choosing the URLs.
+
+docpull is intentionally not a general browser-automation scraper. See
+[`docs/scraping-boundary.md`](docs/scraping-boundary.md) for the exact product
+boundary and when to use Scrapy, Crawlee, hosted extraction APIs, or trafilatura
+directly.
 
 ## Install
 
@@ -125,22 +151,27 @@ pip install docpull
 pip install 'docpull[llm]'           # tiktoken for token-accurate chunking
 pip install 'docpull[trafilatura]'   # alternative extractor for noisy pages
 pip install 'docpull[mcp]'           # run as an MCP server for AI agents
+pip install 'docpull[parallel]'      # Parallel API context packs
+pip install 'docpull[observability]' # Raindrop benchmark tracing
 pip install 'docpull[all]'           # everything above
 ```
 
 ## Quick start
 
 ```bash
-# Crawl and save Markdown
+# Scrape a page graph and save Markdown
 docpull https://docs.example.com
 
-# One page, no crawl — the fast path for agents
+# Scrape one page, no crawl — the fast path for agents
 docpull https://docs.example.com/guide --single
 
 # LLM-ready NDJSON with 4k-token chunks streamed to stdout
 docpull https://docs.example.com --profile llm --stream | jq .
 
-# Mirror a site for offline use
+# Open Knowledge Format bundle for agent/wiki interoperability
+docpull https://docs.example.com --format okf
+
+# Mirror scraped content for offline use
 docpull https://docs.example.com --profile mirror --cache
 ```
 
@@ -154,8 +185,13 @@ content directly from framework data feeds:
 | Next.js   | Parses `__NEXT_DATA__` JSON |
 | Mintlify  | `__NEXT_DATA__` with Mintlify tagging |
 | OpenAPI   | Renders `openapi.json` / `swagger.json` into Markdown |
-| Docusaurus| Detected and tagged; generic extractor produces Markdown |
-| Sphinx    | Detected from generator metadata / Read the Docs hosts and tagged; generic extractor produces Markdown |
+| Docusaurus| Extracts static article regions and tags framework metadata |
+| Sphinx    | Extracts static body/document regions and tags framework metadata |
+| MkDocs / Material | Extracts static content regions and tags framework metadata |
+| VitePress / VuePress | Extracts static doc regions and tags framework metadata |
+| Astro Starlight | Extracts static markdown content regions and tags framework metadata |
+| GitBook / ReadMe.io | Extracts static article/content regions and tags framework metadata |
+| Redoc / Scalar | Extracts static API reference regions; OpenAPI JSON is still preferred when available |
 
 JS-only SPAs with no server-rendered content are detected and skipped with a
 clear reason (or, with `--strict-js-required`, reported as an error so agents
@@ -175,6 +211,30 @@ can route elsewhere).
 
 ## Python API
 
+Scraper-facing API:
+
+```python
+import asyncio
+from docpull import scrape_one, Scraper
+
+async def main():
+    page = await scrape_one("https://docs.python.org/3/library/asyncio.html")
+    print(page.title, page.source_type)
+    print(page.text[:500])
+
+    scraper = Scraper()
+    run = await scraper.scrape_site(
+        "https://docs.example.com",
+        output_format="ndjson",
+        max_pages=100,
+    )
+    print(run.stats.pages_fetched, run.manifest_path)
+
+asyncio.run(main())
+```
+
+Core fetcher API:
+
 ```python
 from docpull import fetch_one
 
@@ -219,6 +279,7 @@ async def tool_call(url: str) -> str:
 ```bash
 docpull https://site.com --profile rag      # Default. Dedup, rich metadata.
 docpull https://site.com --profile llm      # NDJSON + chunks + metadata; JS-only pages skip unless --strict-js-required is passed.
+docpull https://site.com --profile okf      # OKF bundle with generated index.md files.
 docpull https://site.com --profile mirror   # Full archive, polite, cached, hierarchical paths.
 docpull https://site.com --profile quick    # Sampling: 50 pages, depth 2.
 ```
@@ -274,7 +335,32 @@ value and reports the key source (`env`, `project_env`, `user_config`, or
 `missing`). It does not make a live Parallel call or prove the key is valid. Use
 `--json` when an agent or CI job needs machine-readable configuration status.
 
-The pack contains:
+## Optional live providers
+
+Parallel, Tavily, and Exa are equal optional live providers for benchmark and
+provider context-pack workflows. You can configure zero, one, two, or all three
+API keys. docpull looks for keys in this order: environment variables, project
+`.env.local`, then `~/.config/docpull/secrets.env`. Missing providers are skipped
+instead of failing provider-neutral runs.
+
+```bash
+docpull providers auth --json
+docpull providers init parallel
+docpull providers init tavily
+docpull providers init exa
+
+docpull providers context-pack "Compare AI web-search APIs for agents" \
+  --provider auto \
+  --query "AI web search API" \
+  --include-domain docs.parallel.ai \
+  --output-dir ./packs/provider-comparison
+```
+
+Use `--provider all` to request all three and record skipped-provider metadata
+for keys or optional SDKs that are not configured. Use repeated `--provider`
+values to run a specific subset.
+
+A Parallel pack contains:
 
 - `AGENT_CONTEXT.md` — agent load plan with source order, pack signals, warnings, errors, and artifact map.
 - `documents.ndjson` — chunked records for agents and RAG pipelines.
@@ -475,10 +561,42 @@ NDJSON (one record per page or chunk):
 {"document_id": "doc_...", "chunk_id": "chunk_...", "url": "...", "title": "...", "content": "...", "hash": "...", "token_count": 842, "chunk_index": 0}
 ```
 
+Open Knowledge Format:
+
+```bash
+docpull https://docs.example.com --format okf
+```
+
+OKF output is an opt-in Markdown bundle. Scraped pages are written as concept
+documents with OKF frontmatter (`type`, `title`, `description`, `resource`,
+`tags`, `timestamp` when source metadata exists) and docpull keeps `source` as a
+compatibility extension. Generated `index.md` files are reserved for directory
+listings, so URL landing pages use safe concept filenames such as `_root.md` or
+`_page.md` instead of occupying `index.md`. The root `index.md` declares
+`okf_version: "0.1"`; nested indexes stay plain listing files.
+
+Because OKF treats every non-reserved `.md` file in the tree as a concept, write
+OKF output to a clean or dedicated directory rather than mixing it with unrelated
+Markdown files.
+
 Every output format also writes `corpus.manifest.json` next to the generated
 documents. The manifest records the run identity, output format, stable
-`document_id` / `chunk_id` values, content hashes, relative output paths, and
-chunk counts so regenerated corpora can be diffed and cited by agents.
+`document_id` / `chunk_id` values, content hashes, file-backed relative output
+paths (or the stdout marker for streamed NDJSON), and chunk counts so
+regenerated corpora can be diffed and cited by agents. See
+[`docs/corpus-manifest.md`](docs/corpus-manifest.md) for the schema and
+stability contract.
+
+SQLite output includes an FTS5 index for local retrieval:
+
+```python
+from pathlib import Path
+from docpull import search_sqlite_documents
+
+hits = search_sqlite_documents(Path("docs/documents.db"), "rate limit")
+for hit in hits:
+    print(hit.url, hit.snippet)
+```
 
 ## Security
 
@@ -499,9 +617,9 @@ Run `docpull --help` for the full list. Highlights:
 
 ```
 Core:
-  --profile {rag,mirror,quick,llm}
+  --profile {rag,mirror,quick,llm,okf}
   --single                Fetch one URL (no crawl)
-  --format {markdown,json,ndjson,sqlite}
+  --format {markdown,json,ndjson,sqlite,okf}
   --stream                Stream NDJSON to stdout
 
 LLM / chunking:
@@ -547,6 +665,77 @@ Reproduce with `make benchmark` (requires `aiohttp`; runs the gated
 benchmark in `tests/benchmarks/` and prints progress plus a JSON line you can
 pipe into trend tooling).
 
+For real-site and live-provider context-pack benchmarks, use the first-class
+benchmark harness:
+
+```bash
+docpull benchmark quick
+docpull benchmark quick --provider auto --max-estimated-cost 0.05
+docpull benchmark quick --provider all --max-estimated-cost 0.10
+docpull benchmark quick --target-set provider-matrix --provider all \
+  --max-pages 8 --max-depth 1 --max-search-results 5 --extract-limit 2 \
+  --max-estimated-cost 0.10
+docpull benchmark article .bench/runs/<run>/benchmark.report.json
+```
+
+`docpull benchmark quick` writes `benchmark.report.json` and
+`benchmark.summary.md` under `.bench/runs/<timestamp>/` by default. The core
+cases measure an LLM-profile crawl plus a cached rerun. Passing `--provider auto`
+runs every locally ready provider. Passing `--provider all` requests Parallel,
+Tavily, and Exa, then records missing keys or optional SDKs in
+`skipped_providers` without failing the benchmark. Live Parallel calls still use
+the local `--max-estimated-cost` guard before any work starts.
+
+Use `--target-set tool-docs` to run a cross-pull matrix across the Parallel,
+Exa, Tavily, Raindrop, and DocPull documentation sites. Use `--target-set provider-matrix`
+to add three low-cap adversarial public targets for JS-heavy docs, noisy
+archived navigation, and freshness-sensitive pricing. Matrix runs skip the
+cached core pass by default so the headline grid stays provider x target; pass
+`--cached-pass` to force cache measurement across the matrix.
+
+The compatibility flags `--parallel`, `--tavily`, and `--exa` still work as
+provider aliases. Tavily Search + Extract and Exa Search-with-contents are
+normalized into the same local
+`documents.ndjson`, `corpus.manifest.json`, `sources.md`, provider `*.pack.json`,
+`pack.score.json`, and `source.scores.json` artifacts as the core and Parallel
+cases. The report keeps the legacy pack score and adds weighted benchmark
+sub-scores for coverage, cleanliness, source fidelity, freshness, and density.
+Keys can be exported as `PARALLEL_API_KEY`, `TAVILY_API_KEY`, and `EXA_API_KEY`
+or stored together in `~/.config/docpull/secrets.env`; docpull does not write
+keys into benchmark artifacts.
+
+Tavily usage is credit-based, so set `TAVILY_CREDIT_USD` or pass
+`--tavily-credit-usd` to convert credits into estimated dollars for cost
+comparisons. Without that value, Tavily credits are still recorded in
+`cost_units` but excluded from normalized USD totals.
+
+To trace the benchmark in Raindrop for a publishable observability/eval loop,
+install the optional SDK and pass `--trace raindrop`:
+
+```bash
+pip install 'docpull[parallel,observability]'
+export PARALLEL_API_KEY="<your-parallel-api-key>"
+export TAVILY_API_KEY="<your-tavily-api-key>"
+export TAVILY_CREDIT_USD="<account-credit-value>"
+export EXA_API_KEY="<your-exa-api-key>"
+export RAINDROP_WRITE_KEY="<your-raindrop-write-key>"
+docpull benchmark quick --target-set provider-matrix --provider all --trace raindrop \
+  --max-pages 8 --max-depth 1 --max-search-results 5 --extract-limit 2 \
+  --max-estimated-cost 0.10
+docpull benchmark article .bench/runs/<run>/benchmark.report.json
+```
+
+Raindrop traces are metadata-only by default: docpull records timings, counts,
+scores, costs, selected URLs, provider, workflow, target, prompt, settings, and
+artifact paths, but does not send scraped page content unless a future caller
+explicitly adds that behavior. `RAINDROP_WRITE_KEY` can be exported or stored in
+the same `~/.config/docpull/secrets.env` file as the provider keys.
+
+When Raindrop tracing is enabled, DocPull also emits Raindrop signals for
+attention-worthy benchmark cells: failed cases, low scores, slow cases,
+high-cost cells, and score-dimension warnings. The benchmark report stores the
+Raindrop event id and signal counts so traced runs can be audited later.
+
 ## Troubleshooting
 
 ```bash

{docpull-4.1.0 → docpull-4.3.0}/README.md

@@ -1,10 +1,11 @@
 # docpull
 
-**Security-hardened, browser-free crawler that turns static documentation sites into clean, AI-ready Markdown — fast.**
+**Security-hardened, browser-free web scraper and crawler that turns server-rendered pages into clean, AI-ready Markdown — fast.**
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![PyPI version](https://badge.fury.io/py/docpull.svg)](https://badge.fury.io/py/docpull)
-[![Downloads](https://pepy.tech/badge/docpull)](https://pepy.tech/project/docpull)
+[![PyPI downloads](https://img.shields.io/pepy/dt/docpull?label=downloads)](https://pepy.tech/project/docpull)
+[![GitHub stars](https://img.shields.io/github/stars/raintree-technology/docpull?style=social)](https://github.com/raintree-technology/docpull/stargazers)
 [![License: MIT](https://img.shields.io/github/license/raintree-technology/docpull)](https://github.com/raintree-technology/docpull/blob/main/LICENSE)
 
 <p align="center">
@@ -13,15 +14,37 @@
   </a>
 </p>
 
-docpull uses async HTTP (not Playwright) to fetch server-rendered pages,
-extracts main content, and writes clean Markdown with source-URL frontmatter —
-in seconds, with a small install footprint. It won't render JavaScript, but for
-the large class of docs that don't need it (API references, Python/Go stdlib,
-most dev-tool docs, OpenAPI specs, Next.js and Docusaurus builds), it is a
-fast, auditable, sandbox-friendly way to pipe documentation into an LLM context,
-a RAG index, or an offline archive. SSRF, XXE, DNS-rebinding, and
-CRLF-injection protections are on by default — a necessity when an AI agent
-is choosing the URLs.
+## Download History
+
+<a href="https://pepy.tech/project/docpull">
+  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
+</a>
+
+## Star History
+
+<a href="https://star-history.com/#raintree-technology/docpull&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+  </picture>
+</a>
+
+docpull is a web scraper for static and server-rendered sites, with
+documentation crawling as its sharpest default workflow. It uses async HTTP (not
+Playwright) to fetch pages, discover links, extract main content, and write clean
+Markdown with source-URL frontmatter — in seconds, with a small install
+footprint. It won't render JavaScript, but for the large class of sites that
+don't need it (API references, vendor docs, Python/Go stdlib, blogs, OpenAPI
+specs, Next.js and Docusaurus builds), it is a fast, auditable,
+sandbox-friendly way to pipe web content into an LLM context, a RAG index, or an
+offline archive. SSRF, XXE, DNS-rebinding, and CRLF-injection protections are on
+by default — a necessity when an AI agent is choosing the URLs.
+
+docpull is intentionally not a general browser-automation scraper. See
+[`docs/scraping-boundary.md`](docs/scraping-boundary.md) for the exact product
+boundary and when to use Scrapy, Crawlee, hosted extraction APIs, or trafilatura
+directly.
 
 ## Install
 
@@ -32,22 +55,27 @@ pip install docpull
 pip install 'docpull[llm]'           # tiktoken for token-accurate chunking
 pip install 'docpull[trafilatura]'   # alternative extractor for noisy pages
 pip install 'docpull[mcp]'           # run as an MCP server for AI agents
+pip install 'docpull[parallel]'      # Parallel API context packs
+pip install 'docpull[observability]' # Raindrop benchmark tracing
 pip install 'docpull[all]'           # everything above
 ```
 
 ## Quick start
 
 ```bash
-# Crawl and save Markdown
+# Scrape a page graph and save Markdown
 docpull https://docs.example.com
 
-# One page, no crawl — the fast path for agents
+# Scrape one page, no crawl — the fast path for agents
 docpull https://docs.example.com/guide --single
 
 # LLM-ready NDJSON with 4k-token chunks streamed to stdout
 docpull https://docs.example.com --profile llm --stream | jq .
 
-# Mirror a site for offline use
+# Open Knowledge Format bundle for agent/wiki interoperability
+docpull https://docs.example.com --format okf
+
+# Mirror scraped content for offline use
 docpull https://docs.example.com --profile mirror --cache
 ```
 
@@ -61,8 +89,13 @@ content directly from framework data feeds:
 | Next.js   | Parses `__NEXT_DATA__` JSON |
 | Mintlify  | `__NEXT_DATA__` with Mintlify tagging |
 | OpenAPI   | Renders `openapi.json` / `swagger.json` into Markdown |
-| Docusaurus| Detected and tagged; generic extractor produces Markdown |
-| Sphinx    | Detected from generator metadata / Read the Docs hosts and tagged; generic extractor produces Markdown |
+| Docusaurus| Extracts static article regions and tags framework metadata |
+| Sphinx    | Extracts static body/document regions and tags framework metadata |
+| MkDocs / Material | Extracts static content regions and tags framework metadata |
+| VitePress / VuePress | Extracts static doc regions and tags framework metadata |
+| Astro Starlight | Extracts static markdown content regions and tags framework metadata |
+| GitBook / ReadMe.io | Extracts static article/content regions and tags framework metadata |
+| Redoc / Scalar | Extracts static API reference regions; OpenAPI JSON is still preferred when available |
 
 JS-only SPAs with no server-rendered content are detected and skipped with a
 clear reason (or, with `--strict-js-required`, reported as an error so agents
@@ -82,6 +115,30 @@ can route elsewhere).
 
 ## Python API
 
+Scraper-facing API:
+
+```python
+import asyncio
+from docpull import scrape_one, Scraper
+
+async def main():
+    page = await scrape_one("https://docs.python.org/3/library/asyncio.html")
+    print(page.title, page.source_type)
+    print(page.text[:500])
+
+    scraper = Scraper()
+    run = await scraper.scrape_site(
+        "https://docs.example.com",
+        output_format="ndjson",
+        max_pages=100,
+    )
+    print(run.stats.pages_fetched, run.manifest_path)
+
+asyncio.run(main())
+```
+
+Core fetcher API:
+
 ```python
 from docpull import fetch_one
 
@@ -126,6 +183,7 @@ async def tool_call(url: str) -> str:
 ```bash
 docpull https://site.com --profile rag      # Default. Dedup, rich metadata.
 docpull https://site.com --profile llm      # NDJSON + chunks + metadata; JS-only pages skip unless --strict-js-required is passed.
+docpull https://site.com --profile okf      # OKF bundle with generated index.md files.
 docpull https://site.com --profile mirror   # Full archive, polite, cached, hierarchical paths.
 docpull https://site.com --profile quick    # Sampling: 50 pages, depth 2.
 ```
@@ -181,7 +239,32 @@ value and reports the key source (`env`, `project_env`, `user_config`, or
 `missing`). It does not make a live Parallel call or prove the key is valid. Use
 `--json` when an agent or CI job needs machine-readable configuration status.
 
-The pack contains:
+## Optional live providers
+
+Parallel, Tavily, and Exa are equal optional live providers for benchmark and
+provider context-pack workflows. You can configure zero, one, two, or all three
+API keys. docpull looks for keys in this order: environment variables, project
+`.env.local`, then `~/.config/docpull/secrets.env`. Missing providers are skipped
+instead of failing provider-neutral runs.
+
+```bash
+docpull providers auth --json
+docpull providers init parallel
+docpull providers init tavily
+docpull providers init exa
+
+docpull providers context-pack "Compare AI web-search APIs for agents" \
+  --provider auto \
+  --query "AI web search API" \
+  --include-domain docs.parallel.ai \
+  --output-dir ./packs/provider-comparison
+```
+
+Use `--provider all` to request all three and record skipped-provider metadata
+for keys or optional SDKs that are not configured. Use repeated `--provider`
+values to run a specific subset.
+
+A Parallel pack contains:
 
 - `AGENT_CONTEXT.md` — agent load plan with source order, pack signals, warnings, errors, and artifact map.
 - `documents.ndjson` — chunked records for agents and RAG pipelines.
@@ -382,10 +465,42 @@ NDJSON (one record per page or chunk):
 {"document_id": "doc_...", "chunk_id": "chunk_...", "url": "...", "title": "...", "content": "...", "hash": "...", "token_count": 842, "chunk_index": 0}
 ```
 
+Open Knowledge Format:
+
+```bash
+docpull https://docs.example.com --format okf
+```
+
+OKF output is an opt-in Markdown bundle. Scraped pages are written as concept
+documents with OKF frontmatter (`type`, `title`, `description`, `resource`,
+`tags`, `timestamp` when source metadata exists) and docpull keeps `source` as a
+compatibility extension. Generated `index.md` files are reserved for directory
+listings, so URL landing pages use safe concept filenames such as `_root.md` or
+`_page.md` instead of occupying `index.md`. The root `index.md` declares
+`okf_version: "0.1"`; nested indexes stay plain listing files.
+
+Because OKF treats every non-reserved `.md` file in the tree as a concept, write
+OKF output to a clean or dedicated directory rather than mixing it with unrelated
+Markdown files.
+
 Every output format also writes `corpus.manifest.json` next to the generated
 documents. The manifest records the run identity, output format, stable
-`document_id` / `chunk_id` values, content hashes, relative output paths, and
-chunk counts so regenerated corpora can be diffed and cited by agents.
+`document_id` / `chunk_id` values, content hashes, file-backed relative output
+paths (or the stdout marker for streamed NDJSON), and chunk counts so
+regenerated corpora can be diffed and cited by agents. See
+[`docs/corpus-manifest.md`](docs/corpus-manifest.md) for the schema and
+stability contract.
+
+SQLite output includes an FTS5 index for local retrieval:
+
+```python
+from pathlib import Path
+from docpull import search_sqlite_documents
+
+hits = search_sqlite_documents(Path("docs/documents.db"), "rate limit")
+for hit in hits:
+    print(hit.url, hit.snippet)
+```
 
 ## Security
 
@@ -406,9 +521,9 @@ Run `docpull --help` for the full list. Highlights:
 
 ```
 Core:
-  --profile {rag,mirror,quick,llm}
+  --profile {rag,mirror,quick,llm,okf}
   --single                Fetch one URL (no crawl)
-  --format {markdown,json,ndjson,sqlite}
+  --format {markdown,json,ndjson,sqlite,okf}
   --stream                Stream NDJSON to stdout
 
 LLM / chunking:
@@ -454,6 +569,77 @@ Reproduce with `make benchmark` (requires `aiohttp`; runs the gated
 benchmark in `tests/benchmarks/` and prints progress plus a JSON line you can
 pipe into trend tooling).
 
+For real-site and live-provider context-pack benchmarks, use the first-class
+benchmark harness:
+
+```bash
+docpull benchmark quick
+docpull benchmark quick --provider auto --max-estimated-cost 0.05
+docpull benchmark quick --provider all --max-estimated-cost 0.10
+docpull benchmark quick --target-set provider-matrix --provider all \
+  --max-pages 8 --max-depth 1 --max-search-results 5 --extract-limit 2 \
+  --max-estimated-cost 0.10
+docpull benchmark article .bench/runs/<run>/benchmark.report.json
+```
+
+`docpull benchmark quick` writes `benchmark.report.json` and
+`benchmark.summary.md` under `.bench/runs/<timestamp>/` by default. The core
+cases measure an LLM-profile crawl plus a cached rerun. Passing `--provider auto`
+runs every locally ready provider. Passing `--provider all` requests Parallel,
+Tavily, and Exa, then records missing keys or optional SDKs in
+`skipped_providers` without failing the benchmark. Live Parallel calls still use
+the local `--max-estimated-cost` guard before any work starts.
+
+Use `--target-set tool-docs` to run a cross-pull matrix across the Parallel,
+Exa, Tavily, Raindrop, and DocPull documentation sites. Use `--target-set provider-matrix`
+to add three low-cap adversarial public targets for JS-heavy docs, noisy
+archived navigation, and freshness-sensitive pricing. Matrix runs skip the
+cached core pass by default so the headline grid stays provider x target; pass
+`--cached-pass` to force cache measurement across the matrix.
+
+The compatibility flags `--parallel`, `--tavily`, and `--exa` still work as
+provider aliases. Tavily Search + Extract and Exa Search-with-contents are
+normalized into the same local
+`documents.ndjson`, `corpus.manifest.json`, `sources.md`, provider `*.pack.json`,
+`pack.score.json`, and `source.scores.json` artifacts as the core and Parallel
+cases. The report keeps the legacy pack score and adds weighted benchmark
+sub-scores for coverage, cleanliness, source fidelity, freshness, and density.
+Keys can be exported as `PARALLEL_API_KEY`, `TAVILY_API_KEY`, and `EXA_API_KEY`
+or stored together in `~/.config/docpull/secrets.env`; docpull does not write
+keys into benchmark artifacts.
+
+Tavily usage is credit-based, so set `TAVILY_CREDIT_USD` or pass
+`--tavily-credit-usd` to convert credits into estimated dollars for cost
+comparisons. Without that value, Tavily credits are still recorded in
+`cost_units` but excluded from normalized USD totals.
+
+To trace the benchmark in Raindrop for a publishable observability/eval loop,
+install the optional SDK and pass `--trace raindrop`:
+
+```bash
+pip install 'docpull[parallel,observability]'
+export PARALLEL_API_KEY="<your-parallel-api-key>"
+export TAVILY_API_KEY="<your-tavily-api-key>"
+export TAVILY_CREDIT_USD="<account-credit-value>"
+export EXA_API_KEY="<your-exa-api-key>"
+export RAINDROP_WRITE_KEY="<your-raindrop-write-key>"
+docpull benchmark quick --target-set provider-matrix --provider all --trace raindrop \
+  --max-pages 8 --max-depth 1 --max-search-results 5 --extract-limit 2 \
+  --max-estimated-cost 0.10
+docpull benchmark article .bench/runs/<run>/benchmark.report.json
+```
+
+Raindrop traces are metadata-only by default: docpull records timings, counts,
+scores, costs, selected URLs, provider, workflow, target, prompt, settings, and
+artifact paths, but does not send scraped page content unless a future caller
+explicitly adds that behavior. `RAINDROP_WRITE_KEY` can be exported or stored in
+the same `~/.config/docpull/secrets.env` file as the provider keys.
+
+When Raindrop tracing is enabled, DocPull also emits Raindrop signals for
+attention-worthy benchmark cells: failed cases, low scores, slow cases,
+high-cost cells, and score-dimension warnings. The benchmark report stores the
+Raindrop event id and signal counts so traced runs can be audited later.
+
 ## Troubleshooting
 
 ```bash