docpull 5.5.1__tar.gz → 6.0.1__tar.gz

{docpull-5.5.1/src/docpull.egg-info → docpull-6.0.1}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: docpull
-Version: 5.5.1
-Summary: Keep AI agents synced with changing public docs and web sources
+Version: 6.0.1
+Summary: Declare, sync, diff, and lock context dependencies for AI agents
 Author-email: Zachary Roth <support@raintree.technology>
 Maintainer-email: Raintree Technology <support@raintree.technology>
 License-Expression: MIT
@@ -15,7 +15,7 @@ Project-URL: MCP Plugin, https://github.com/raintree-technology/docpull/tree/mai
 Project-URL: Comparison Guide, https://github.com/raintree-technology/docpull/blob/main/docs/alternatives.md
 Project-URL: Download Stats, https://pepy.tech/project/docpull
 Project-URL: Changelog, https://github.com/raintree-technology/docpull/blob/main/docs/CHANGELOG.md
-Keywords: python,markdown,documentation,web,website,web-extraction,web-to-markdown,web-scraping,web-crawler,crawler,content-extraction,developer-tools,cli,rag,llm,mcp,ai-agents,context-engineering,source-packs,research,claude,ai-training-data
+Keywords: python,markdown,documentation,web,website,web-extraction,web-to-markdown,source-ingestion,provenance,context-ci,content-extraction,developer-tools,cli,rag,llm,mcp,ai-agents,context-engineering,context-dependencies,context-packs,source-packs,research,claude,ai-training-data
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Information Technology
@@ -68,12 +68,17 @@ Requires-Dist: python-multipart>=0.0.27; extra == "mcp"
 Requires-Dist: starlette>=1.3.1; extra == "mcp"
 Provides-Extra: serve
 Requires-Dist: uvicorn>=0.30.0; extra == "serve"
-Provides-Extra: parallel
-Requires-Dist: parallel-web>=1.0.1; extra == "parallel"
-Provides-Extra: observability
-Requires-Dist: raindrop-ai>=0.0.50; extra == "observability"
 Provides-Extra: parquet
 Requires-Dist: pyarrow>=16.0.0; extra == "parquet"
+Provides-Extra: markitdown
+Requires-Dist: markitdown>=0.1.0; extra == "markitdown"
+Provides-Extra: unstructured
+Requires-Dist: unstructured[all-docs]>=0.16.0; extra == "unstructured"
+Provides-Extra: parse
+Requires-Dist: markitdown>=0.1.0; extra == "parse"
+Requires-Dist: unstructured[all-docs]>=0.16.0; extra == "parse"
+Provides-Extra: presidio
+Requires-Dist: presidio-analyzer>=2.2.0; extra == "presidio"
 Provides-Extra: e2b
 Requires-Dist: e2b>=2.0.0; extra == "e2b"
 Provides-Extra: llm
@@ -90,9 +95,10 @@ Requires-Dist: pyjwt>=2.13.0; extra == "all"
 Requires-Dist: python-multipart>=0.0.27; extra == "all"
 Requires-Dist: starlette>=1.3.1; extra == "all"
 Requires-Dist: uvicorn>=0.30.0; extra == "all"
-Requires-Dist: parallel-web>=1.0.1; extra == "all"
-Requires-Dist: raindrop-ai>=0.0.50; extra == "all"
 Requires-Dist: pyarrow>=16.0.0; extra == "all"
+Requires-Dist: markitdown>=0.1.0; extra == "all"
+Requires-Dist: unstructured[all-docs]>=0.16.0; extra == "all"
+Requires-Dist: presidio-analyzer>=2.2.0; extra == "all"
 Requires-Dist: e2b>=2.0.0; extra == "all"
 Provides-Extra: dev
 Requires-Dist: pytest>=7.0.0; extra == "dev"
@@ -112,7 +118,7 @@ Dynamic: license-file
 
 # docpull
 
-**Keep AI agents synced with changing public docs. Browser-free by default.**
+**Context dependencies for AI agents. Browser-free by default.**
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![PyPI version](https://img.shields.io/pypi/v/docpull.svg?label=package)](https://pypi.org/project/docpull/)
@@ -120,10 +126,38 @@ Dynamic: license-file
 [![GitHub stars](https://img.shields.io/github/stars/raintree-technology/docpull?style=social)](https://github.com/raintree-technology/docpull/stargazers)
 [![License: MIT](https://img.shields.io/github/license/raintree-technology/docpull)](https://github.com/raintree-technology/docpull/blob/main/LICENSE)
 
-DocPull turns public docs and web sources into refreshable, cited, agent-ready
-context packs. Define source sets once, sync them over time, diff what changed,
-and export fresh context for Cursor, Claude, Codex, OpenAI, LlamaIndex,
-LangChain, MCP clients, and RAG pipelines.
+<!-- mcp-name: io.github.raintree-technology/docpull -->
+
+DocPull is a local-first dependency manager for AI context. Define the public
+web sources an agent depends on, sync them into cited context packs, diff what
+changed, and export reproducible context for Cursor, Claude, OpenAI,
+LlamaIndex, LangChain, MCP clients, and RAG pipelines.
+
+The core workflow is a `docpull.yaml` plus a `.docpull/context.lock.json`,
+similar in spirit to code dependency manifests and lockfiles:
+
+```bash
+docpull init my-agent-context
+docpull add stripe react postgres
+docpull install
+docpull deps
+docpull sync
+docpull diff
+docpull export context-pack --target openai
+```
+
+Bundled aliases such as `stripe`, `react`, `postgres`, `openai`, and
+`apple-hig` expand to normal HTTPS sources in `docpull.yaml`. Runs stay
+reproducible through the lockfile: source URLs, discovered URLs, content hashes,
+run IDs, aliases, and export metadata are recorded without storing secrets.
+Use `docpull sources list` to inspect the bundled alias catalog and
+`docpull install` to validate or recreate the local dependency lock.
+Use `docpull deps` to see the current dependency, lockfile, latest run, and
+export status.
+
+Projects can also track typed known-source specs such as `pypi:requests`,
+`rfc:9110`, `wiki:Web_scraping`, or a local dataset path. Those sources sync
+through their typed lanes and do not use discovery.
 
 The original `docpull URL ...` workflow still works: fetch public or explicitly
 authorized static/server-rendered web pages and write clean Markdown, NDJSON,
@@ -131,14 +165,15 @@ SQLite, or OKF outputs. Project mode adds the persistent evidence lifecycle on
 top: sources, runs, diffs, exports, evals, accounting, and local auditability.
 
 DocPull is local-first: direct fetching, sitemap/link discovery, extraction,
-indexing, pack intelligence, and local `agent-browser` rendering can run with no
-provider account and no required API spend. Tavily, Exa, Parallel, and cloud
-renderers are optional escalation paths when local and open-source routes are
-not enough.
+indexing, pack intelligence, and opt-in `agent-browser` rendering can run with
+no external account and no required API spend. Cloud rendering is explicit and
+budget-guarded.
 
-DocPull exposes the same core workflows through CLI, Python SDK, and MCP, with
-each surface optimized for its user. The [Surface Contract](docs/surface-contract.md)
-defines how those surfaces align and where they intentionally differ.
+DocPull aligns core workflows across CLI, Python SDK, and MCP, with each surface
+optimized for its user. The [Surface Contract](docs/surface-contract.md) defines
+how those surfaces align and where they intentionally differ.
+For the context dependency workflow, see
+[Context Dependencies](docs/context-dependencies.md).
 
 Web-source ingestion is the core workflow. Documentation is one high-value
 lane, not the product boundary. It works best on static or server-rendered
@@ -146,9 +181,9 @@ pages such as blogs, API references, OpenAPI specs, changelogs, vendor pages,
 product pages, filings, docs sites, and other pages where the useful content is
 available in HTML or embedded page data.
 
-docpull is browser-free by default. JS-only pages are skipped with a clear
-reason unless you explicitly opt into the local `agent-browser` renderer. See
-[Scraping Boundary](docs/scraping-boundary.md) and
+DocPull is browser-free by default. JS-only pages are skipped with a clear
+reason unless you explicitly opt into a local renderer. See
+[Web Source Boundary](docs/scraping-boundary.md) and
 [Alternatives](docs/alternatives.md) for the full boundary.
 
 ## Install
@@ -161,12 +196,50 @@ pip install docpull
 
 ```bash
 docpull init stripe-docs
-docpull add https://docs.stripe.com
+docpull add stripe
+docpull install
 docpull sync
+docpull deps
 docpull diff
 docpull export context-pack --target cursor
 ```
 
+## Context CI
+
+Use Context CI when an agent loop depends on current, cited context and a
+missing or stale source should fail the build:
+
+```bash
+docpull ci --prepare
+```
+
+`docpull ci` runs locally against either a project root or a standalone pack. It
+checks the project lockfile, pack score, pack audit, coverage confidence,
+citation coverage, eval-grade sidecars, evidence basis quality, rights
+metadata, and optional context predictions. It writes `context-ci.report.json` and `CONTEXT_CI.md`;
+the command exits non-zero when hard gates fail.
+
+Minimal GitHub Actions job:
+
+```yaml
+name: Context CI
+on: [pull_request]
+jobs:
+  context:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - run: pip install docpull
+      - run: docpull ci --prepare
+```
+
+For the full workflow, see [Context CI](docs/context-ci.md). The durable
+artifact shape is documented in
+[Context Pack Contract v3](docs/context-pack-contract-v3.md).
+
 Example diff after a later sync:
 
 ```text
@@ -185,29 +258,64 @@ Changed pages:
 0 paid/cloud routes used
 ```
 
+## Context Pack Contract
+
+DocPull writes three explicit layers of artifacts:
+
+| Layer | Purpose | Contract check |
+| --- | --- | --- |
+| Raw extraction | Fetched documents, chunks, routes, and source index sidecars | `docpull pack validate PACK --level raw` |
+| Agent-ready pack | Raw evidence plus citation index, coverage, score, audit, and lock sidecars | `docpull pack validate PACK --level agent` |
+| Eval-grade pack | Agent pack plus rights, provenance, basis/eval artifacts, and pack card | `docpull pack validate PACK --level eval` |
+
+Core ingestion paths write into the same v3 contract:
+
+```bash
+docpull https://docs.example.com -o packs/docs
+docpull parse ./handbook.pdf -o packs/handbook --backend auto
+docpull openapi-pack ./openapi.json -o packs/api
+docpull feed-pack https://example.com/news -o packs/news
+docpull paper-pack arxiv:1706.03762 -o packs/papers
+docpull repo-pack psf/requests -o packs/repo --cache
+docpull package-pack pypi:requests -o packs/package
+docpull standards-pack rfc:9110 -o packs/standard
+docpull dataset-pack ./metrics.csv -o packs/dataset
+docpull transcript-pack ./meeting.vtt -o packs/transcript
+docpull wiki-pack wiki:Web_scraping -o packs/wiki
+docpull pack prepare packs/docs --eval-grade
+docpull pack validate packs/docs --level eval
+docpull export packs/docs --format openai-vector-jsonl -o exports/openai.jsonl
+docpull export packs/docs --format cursor-rules -o .cursor/rules --skill-name docs
+```
+
+Use `docpull ci --prepare` to validate a project or standalone pack in CI.
+
 Install optional extras as needed:
 
 ```bash
 pip install 'docpull[llm]'           # tiktoken for token-accurate chunking
 pip install 'docpull[trafilatura]'   # alternative extractor for noisy pages
+pip install 'docpull[parse]'         # MarkItDown + Unstructured local document parsers
+pip install 'docpull[presidio]'      # optional Presidio PII detection for redaction
 pip install 'docpull[mcp]'           # stdio MCP server
 pip install 'docpull[serve]'         # local pack JSON server runner
-pip install 'docpull[parallel]'      # Parallel context packs
-pip install 'docpull[observability]' # Raindrop benchmark tracing
+pip install 'docpull[parquet]'       # optional Parquet export support
 pip install 'docpull[e2b]'           # E2B cloud sandbox renderer SDK
-pip install 'docpull[all]'           # all optional extras
 ```
 
+Prefer installing the extras needed for the current lane instead of a broad
+bundle. The base install remains useful without API keys or paid services.
+
 Browser rendering is an explicit external extension, not part of the base
 install. Install an `agent-browser` compatible CLI separately, put it on
 `PATH`, or set `DOCPULL_AGENT_BROWSER_BIN=/path/to/agent-browser`. Verify the
 runtime with `docpull render --check`. Render targets must use HTTPS except for
 localhost/loopback HTTP during local testing, and DocPull keeps renderer action
-permissions locked down to HTML retrieval only. Because the current
-`agent-browser` CLI contract cannot enforce redirect, subresource, or
-connect-time DNS allow-lists, network browser rendering fails closed unless the
-operator sets `DOCPULL_RENDER_TRUSTED_BROWSER_TARGETS=1` for trusted targets.
-For localhost/loopback HTTP tests, set `DOCPULL_RENDER_ALLOW_LOCAL_TARGETS=1`.
+permissions locked down to HTML retrieval only. Because browser rendering
+cannot fully enforce redirect, subresource, or connect-time DNS allow-lists,
+network browser rendering fails closed unless the operator sets
+`DOCPULL_RENDER_TRUSTED_BROWSER_TARGETS=1` for trusted targets. For
+localhost/loopback HTTP tests, set `DOCPULL_RENDER_ALLOW_LOCAL_TARGETS=1`.
 
 For stronger isolation, cloud runtimes are available explicitly:
 `docpull render URL --runtime vercel` uses the Vercel Sandbox CLI and Vercel
@@ -218,56 +326,43 @@ set a local per-render budget guard, and use `--cloud-agent-browser-install skip
 with a prebuilt sandbox/template that already includes `agent-browser`. For E2B,
 pass `--template` or set `DOCPULL_E2B_TEMPLATE` to use that prebuilt environment.
 
+For release acceptance, run the opt-in real-data smoke harness. The default path
+uses public free sources and local tooling; `--include-cloud` also attempts the
+keyed/cloud render lanes when the local environment is configured for them.
+The strict scorecard also requires synchronized generated metadata and a clean
+`git status --short` before tagging.
+
+```bash
+python scripts/release_a_plus_check.py --strict
+python scripts/real_feature_smoke.py --json --full-mcp --strict-ci --auth-matrix --monitor-soak-minutes 10
+python scripts/real_feature_smoke.py --include-cloud --json
+```
+
 ## Free-First Budgets
 
-Use `--budget 0` when a run must not make paid-capable provider or cloud calls:
+Use `--budget 0` when a run must not make paid-capable cloud calls:
 
 ```bash
 docpull https://docs.example.com --budget 0 -o ./docs/example
-docpull discover scan https://docs.example.com -o ./packs/discovery
 DOCPULL_RENDER_TRUSTED_BROWSER_TARGETS=1 docpull render https://example.com/app --runtime local --budget 0
-docpull providers context-pack "Find official docs" --provider all --dry-run --budget 0 --json
-docpull benchmark quick --zero-dollar --target-set zero-dollar --provider all
 ```
 
 Under a zero budget, local cache, direct HTTP, sitemap/static-link discovery,
 local extraction, local indexing, pack analysis, monitors, and local
-`agent-browser` rendering for trusted targets remain allowed. Live Tavily, Exa,
-Parallel, Vercel Sandbox, and E2B calls are blocked before execution. Runs
-involving a budget or paid-capable route write `run.accounting.json` with
-non-secret route, cost, HTTP/cache, browser, and blocked-action metadata.
-
-Use `docpull discover scan URL` to build a provider-free discovery pack from
-open site hints: `llms.txt`, RSS/Atom feeds, OpenAPI specs, sitemap indexes,
-and public GitHub docs trees. It writes the same `candidate_sources.ndjson`
-contract as provider imports and URL/sitemap files, so the next step is still
-`docpull discover select` or `docpull discover fetch`.
-
-When a zero-dollar benchmark or local run is partial, DocPull reports the
-lowest-friction escalation path before spending money: trusted-target local
-`--render fallback` first, BYOK providers next, and cloud rendering only when
-local rendering or infrastructure is the blocker. Benchmark reports include
-suggested commands, estimated paid request counts, and estimated paid cost
-guards before any provider or cloud call is made.
-
-The `zero-dollar` benchmark target set is the Phase 2 measurement matrix. It
-keeps the existing docs/provider targets and adds JS-heavy docs, pricing,
-filings, feeds, sitemaps, and search-to-evidence tasks. The report classifies
-each target as `complete_for_0`, `complete_with_local_browser`, `partial_for_0`,
-`requires_provider`, `requires_cloud_browser`, or `blocked_by_policy`.
-
-## Open Source And Hosted Boundary
-
-The open-source package owns local fetching, local rendering adapters,
-provider-free discovery, extraction, indexing, packs, diffs, monitors, MCP,
-BYOK providers, budget policy, accounting, and benchmarks.
-
-A hosted DocPull product, if offered, should sell managed execution: always-on
-schedules, browser/proxy infrastructure, persistent auth profiles, queues,
-alerts, dashboards, collaboration, retention, SSO, audit logs, SLAs, and
-bundled provider billing. The hosted boundary does not change the OSS default:
-no hidden paid calls, no CAPTCHA bypass, no stealth scraping, and no claim of a
-proprietary web-scale index.
+browser rendering for trusted targets remain allowed. Vercel Sandbox and E2B
+rendering are blocked before execution. Runs involving a budget or paid-capable
+route write `run.accounting.json` with non-secret route, cost, HTTP/cache,
+browser, and blocked-action metadata.
+
+## Release Boundary
+
+The open-source package owns local fetching, explicit rendering adapters,
+source aliases, v3 pack contracts, validation, preparation, exports, Context
+CI, monitors, MCP, budget policy, and accounting.
+
+This release does not include a hosted scheduler, browser/proxy service,
+accounts, marketplace, proprietary web index, CAPTCHA bypass, stealth scraping,
+or hidden paid calls.
 
 ## Persistent Projects
 
@@ -304,15 +399,20 @@ docpull review
 # Create a versioned context-pack release
 docpull release context-pack --target cursor --tag stripe-docs-v1
 
-# Generate eval cases from changed or latest documents
-docpull eval-set --limit 25
-
 # One-command project sync, diff, and export for one source
 docpull watch https://docs.stripe.com --export cursor --alert changes
 ```
 
+Ad hoc `docpull watch` projects are bounded to one page and one level of depth
+by default. Use explicit bounds when the watch should cover more:
+
+```bash
+docpull watch https://docs.stripe.com --export cursor --max-pages 10 --max-depth 2
+```
+
 `docpull diff` is hash-based and deterministic locally. Optional BYOK semantic
-summaries are advisory and skip cleanly when no model key is configured.
+summaries are advisory and skip cleanly when no model key is configured. Each
+diff also writes local semantic categories to `semantic.diff.json`.
 Use `docpull add URL --discover` or `docpull sync --update-discovery` to
 refresh and persist discovered source URLs in `docpull.yaml`; sync then uses
 that stored URL set for repeatable exact refreshes.
@@ -332,14 +432,6 @@ sources:
       policy: explicit-private
 ```
 
-The commercial control-plane shape is API-led but not API-only: the local CLI
-remains the trust-building engine, while hosted DocPull manages projects, sync
-jobs, diffs, exports, releases, and signed webhooks through `/v1` JSON
-endpoints. The hosted ASGI MVP lives in `docpull.hosted`; `docpull remote ...`
-stores hosted API connection metadata and calls the same project lifecycle
-remotely. Remote bearer tokens are sent only to HTTPS API URLs by default;
-`--allow-insecure-local-http` is limited to localhost/loopback development.
-
 The launch screenshot for this flow lives at
 [`docs/launch-assets/docpull-project-diff-demo.png`](docs/launch-assets/docpull-project-diff-demo.png).
 
@@ -409,39 +501,16 @@ docpull https://sdk.vercel.ai \
   --skill-description "Vercel AI SDK source reference"
 ```
 
-Local-first parity workflows mirror common hosted search/extract/crawl/research
-API shapes while writing auditable files instead of relying on a hosted index:
-
-```bash
-# Normalize candidate URLs without fetching content
-docpull map urls ./urls.txt -o ./packs/map
-
-# Extract known URLs into a local pack
-docpull extract-pack ./urls.txt -o ./packs/extract
-
-# Select mapped candidates and fetch them
-docpull crawl-pack ./packs/map --select top:10 -o ./packs/crawl
-
-# Answer/research from an existing local pack with lifecycle artifacts
-docpull research-pack ./packs/crawl \
-  --objective "Summarize auth and webhook behavior" \
-  --schema ./output.schema.json
-
-# Build a cited entity/list pack from existing evidence
-docpull entities-pack ./packs/crawl --limit 100
-```
-
 More examples live in [CLI Recipes](docs/examples/README.md).
 
-With an explicit `--skill-agent`, docpull stores the scraped corpus under
+With an explicit `--skill-agent`, docpull stores the fetched corpus under
 `.docpull/skills/<name>/references` and creates agent-specific wrappers that
 point at that corpus. `--skill-agent claude` writes a Claude Code skill under
-`.claude/skills/<name>/`, `--skill-agent codex` writes a Codex skill under
-`.agents/skills/<name>/` with `agents/openai.yaml`, and `--skill-agent cursor`
-writes a Cursor project rule at `.cursor/rules/<name>.mdc`. Use
-`--skill-agent all` to create all three. If you pass `--output-dir`, docpull
-stages the generated corpus there; explicit `--skill-agent` targets still write
-their active agent wrappers.
+`.claude/skills/<name>/`, and `--skill-agent cursor` writes a Cursor project
+rule at `.cursor/rules/<name>.mdc`. Use `--skill-agent all` to create every
+supported wrapper. If you pass `--output-dir`, docpull stages the generated
+corpus there; explicit `--skill-agent` targets still write their active agent
+wrappers.
 
 Use docpull when you need to:
 
@@ -478,6 +547,15 @@ special handling for common web, documentation, and API surfaces.
 | Static HTML / SSR pages | Extracts article, main, or document regions |
 | Next.js / Mintlify | Parses static HTML and `__NEXT_DATA__` when available |
 | OpenAPI / Swagger | Renders specs into Markdown |
+| OpenAPI pack | `docpull openapi-pack` emits endpoint/schema records with v3 sidecars |
+| RSS / Atom / JSON Feed | `docpull feed-pack` emits item-level records, dates, and listing sidecars |
+| Research papers | `docpull paper-pack` emits paper metadata, abstracts, optional local/arXiv PDF full text, and references |
+| Public GitHub repos | `docpull repo-pack` emits repo metadata, README/docs/examples/changelog files, manifests, and releases |
+| npm / PyPI packages | `docpull package-pack` emits registry metadata, README/description, versions, license, dependencies, and install commands |
+| Standards | `docpull standards-pack` emits RFC, IETF, W3C, and WHATWG metadata plus section-level records |
+| Local datasets | `docpull dataset-pack` emits bounded schema, exact row counts where streamable, column, null-count, and sample summaries |
+| Transcripts | `docpull transcript-pack` emits timestamped segment records from VTT, SRT, text, JSON, or direct transcript URLs |
+| Wikimedia / Wikipedia | `docpull wiki-pack` emits MediaWiki REST page metadata, license/revision metadata, and section-level records |
 | Docusaurus / Sphinx / MkDocs | Extracts static article or document regions |
 | VitePress / VuePress / Astro Starlight | Extracts static content regions |
 | GitBook / ReadMe.io | Extracts available article or content regions |
@@ -497,6 +575,23 @@ errors instead of normal skips.
 | OKF | Portable Open Knowledge Format bundles with indexes and manifests |
 | Archive / mirror | Cached offline source snapshots |
 
+All file-backed outputs now write the DocPull output contract v3 raw sidecars:
+`corpus.manifest.json`, `sources.md`, and `acquisition.routes.json`. Use
+`docpull pack validate <pack-dir> --level raw|agent|eval` to check whether a
+pack is raw extraction output, agent-ready context, or eval-grade context.
+
+Local files can enter the same contract with the document parse lane:
+
+```bash
+docpull parse ./handbook.pdf -o ./packs/handbook --backend auto
+docpull parse ./handbook.docx -o ./packs/handbook --prepare --eval-grade
+```
+
+`--backend auto` reads plain text/Markdown directly and uses optional
+MarkItDown or Unstructured parsers for complex office/PDF files when installed.
+Install `docpull[markitdown]`, `docpull[unstructured]`, or `docpull[parse]`
+for those backends.
+
 Every file-backed run writes `corpus.manifest.json` with stable document IDs,
 chunk IDs, hashes, output paths, and chunk counts. See
 [Corpus Manifest](docs/corpus-manifest.md).
@@ -524,12 +619,17 @@ enabled. It is not the right tool for:
 - Pages behind CAPTCHA or bot challenges.
 - Workflows that require clicking, scrolling, or browser state.
 
-For those cases, use browser automation, such as Playwright, then pass rendered
-HTML or exported content into your pipeline. For simple public JS-rendered
-pages, `docpull render` and `--render fallback` provide an explicit local
-fallback without changing the default crawler behavior. The fallback requires
-the optional external `agent-browser` backend and
-`DOCPULL_RENDER_TRUSTED_BROWSER_TARGETS=1` for trusted network targets.
+For those cases, use full browser automation outside DocPull, then pass
+rendered HTML or exported content into your pipeline. For simple public
+JS-rendered pages, use `docpull render --runtime local` or fetch with
+`--render fallback` for an explicit `agent-browser` fallback without changing
+the default fetch behavior. DocPull does not claim complete browser coverage
+unless rendering is explicitly enabled and available.
+
+Use `--extractor ensemble` when a crawl should score multiple local extraction
+candidates and keep the strongest Markdown. The ensemble always includes the
+built-in generic extractor and adds trafilatura when `docpull[trafilatura]` is
+installed.
 
 ## How It Compares
 
@@ -599,41 +699,28 @@ part of the package release contract.
 
 ## Advanced Workflows
 
-- `docpull[parallel]` can discover, extract, enrich, score, diff, and archive
-  live web sources with your own Parallel API key. See
-  [Parallel Integration](docs/parallel.md).
 - Local pack intelligence can build citation maps, extract cited entities,
-  search pack records, write provider-free research briefs, build cited source
-  graphs, or prepare the full sidecar bundle with
+  search pack records, build cited source graphs, prepare the full sidecar
+  bundle, and write eval-grade rights/provenance artifacts with
   `docpull pack citations`, `docpull pack entities`, `docpull pack search`,
-  `docpull pack brief`, `docpull graph build`, `docpull graph query`, and
-  `docpull pack prepare`.
-- Local-first expansion commands add policy files, discovery packs, refresh
-  reports, audits, cited answers, exports, a localhost pack server, explicit
-  rendering, authenticated-source checks, and cron-friendly monitors:
-  `docpull policy`, `docpull discover`, `docpull refresh`,
-  `docpull pack audit`, `docpull answer-pack`, `docpull export`,
-  `docpull serve`, `docpull share`, `docpull render`, `docpull auth check`, and
-  `docpull monitor`.
+  `docpull pack brief`, `docpull graph build`, `docpull graph query`,
+  and `docpull pack prepare --eval-grade`.
+- Release commands add policy files, refresh reports, audits, exports, a
+  localhost pack server, explicit rendering, authenticated-source checks, and
+  cron-friendly monitors:
+  `docpull policy`, `docpull refresh`,
+  `docpull parse`, `docpull openapi-pack`, `docpull feed-pack`,
+  `docpull paper-pack`, `docpull repo-pack`, `docpull package-pack`,
+  `docpull standards-pack`, `docpull dataset-pack`, `docpull transcript-pack`,
+  `docpull wiki-pack`,
+  `docpull pack validate`,
+  `docpull pack audit`, `docpull export`,
+  `docpull serve`, `docpull share`, `docpull render`, `docpull auth check`,
+  and `docpull monitor`.
 - `docpull export` writes local files for OpenAI vector JSONL, LangChain,
   LlamaIndex, DSPy, Sheets CSV/TSV, n8n workflow JSON, Vercel AI SDK JSON,
   CrewAI JSON, warehouse NDJSON, optional Parquet via `docpull[parquet]`, and
-  Codex/Claude/Cursor agent references.
-- Optional provider workflows can use Parallel, Tavily, and Exa when configured.
-  Tavily and Exa are available through `docpull providers ...` and first-class
-  aliases such as `docpull tavily context-pack`, `docpull exa context-pack`,
-  `docpull exa extract-pack`, and `docpull tavily map-pack`. Use
-  `docpull providers capabilities` to see the shared baseline and provider-only
-  extensions. For agent or CI logs, use
-  `docpull providers auth --json --require-ready --redact-paths` for offline
-  local readiness, then `docpull providers probe --json --require-verified
-  --redact-paths` when explicit live key validation is intended. Successful
-  provider context-pack runs are post-processed into the same local pack
-  intelligence artifacts.
-  See [CLI Recipes](docs/examples/README.md#parallel-context-pack).
-- SEC filing evidence packs use rule profiles such as
-  [vendor-dependency-rules.yml](docs/examples/vendor-dependency-rules.yml).
-
+  agent reference bundles.
 ## Security Defaults
 
 - HTTPS-only fetching with robots.txt compliance.
@@ -659,11 +746,10 @@ docpull URL --preview-urls
 ## Documentation
 
 - [CLI Recipes](docs/examples/README.md) - common commands and advanced workflows.
-- [Scraping Boundary](docs/scraping-boundary.md) - what docpull does and does not fetch.
+- [Web Source Boundary](docs/scraping-boundary.md) - what docpull does and does not fetch.
 - [Alternatives](docs/alternatives.md) - when to use browser automation or hosted extraction.
 - [Corpus Manifest](docs/corpus-manifest.md) - stable IDs, hashes, and source maps.
 - [Surface Contract](docs/surface-contract.md) - how the CLI, Python SDK/API, and MCP surfaces align.
-- [Parallel Integration](docs/parallel.md) - live-source context pack workflows.
 - [Changelog](docs/CHANGELOG.md) - release history.
 
 ## Links