docpull 4.3.0__tar.gz → 4.4.0__tar.gz

{docpull-4.3.0 → docpull-4.4.0}/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2025 Raintree Technology
+Copyright (c) 2025-2026 Raintree Technology
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

{docpull-4.3.0/src/docpull.egg-info → docpull-4.4.0}/PKG-INFO

@@ -1,26 +1,28 @@
 Metadata-Version: 2.4
 Name: docpull
-Version: 4.3.0
-Summary: Pull documentation from the web and convert to clean markdown
+Version: 4.4.0
+Summary: Convert public web pages into clean Markdown for AI agents and RAG
 Author-email: Zachary Roth <support@raintree.technology>
 Maintainer-email: Raintree Technology <support@raintree.technology>
 License-Expression: MIT
-Project-URL: Homepage, https://github.com/raintree-technology/docpull
-Project-URL: Documentation, https://github.com/raintree-technology/docpull#readme
+Project-URL: Homepage, https://docpull.raintree.technology
+Project-URL: Documentation, https://docpull.raintree.technology
 Project-URL: Repository, https://github.com/raintree-technology/docpull
 Project-URL: Source Code, https://github.com/raintree-technology/docpull
 Project-URL: Bug Tracker, https://github.com/raintree-technology/docpull/issues
 Project-URL: Releases, https://github.com/raintree-technology/docpull/releases
-Keywords: python,markdown,documentation,web-scraping,developer-tools,claude,ai-training-data
+Project-URL: MCP Plugin, https://github.com/raintree-technology/docpull/tree/main/plugin
+Project-URL: Comparison Guide, https://github.com/raintree-technology/docpull/blob/main/docs/alternatives.md
+Project-URL: Download Stats, https://pepy.tech/project/docpull
+Project-URL: Changelog, https://github.com/raintree-technology/docpull/blob/main/docs/CHANGELOG.md
+Keywords: python,markdown,documentation,web-extraction,web-scraping,web-crawler,crawler,developer-tools,cli,rag,llm,mcp,ai-agents,context-engineering,claude,ai-training-data
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Information Technology
 Classifier: Intended Audience :: Science/Research
 Classifier: Intended Audience :: Education
 Classifier: Environment :: Console
-Classifier: Topic :: Documentation
 Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
-Classifier: Topic :: Software Development :: Documentation
 Classifier: Topic :: Text Processing :: Markup :: HTML
 Classifier: Topic :: Text Processing :: Markup :: Markdown
 Classifier: Topic :: Utilities
@@ -50,7 +52,7 @@ Requires-Dist: pyyaml>=6.0
 Requires-Dist: pydantic>=2.0
 Requires-Dist: urllib3>=2.7.0
 Provides-Extra: proxy
-Requires-Dist: aiohttp-socks>=0.8.0; extra == "proxy"
+Requires-Dist: aiohttp-socks>=0.11.0; extra == "proxy"
 Provides-Extra: normalize
 Requires-Dist: url-normalize>=1.4.0; extra == "normalize"
 Provides-Extra: trafilatura
@@ -69,7 +71,7 @@ Requires-Dist: raindrop-ai>=0.0.50; extra == "observability"
 Provides-Extra: llm
 Requires-Dist: tiktoken>=0.7.0; extra == "llm"
 Provides-Extra: all
-Requires-Dist: aiohttp-socks>=0.8.0; extra == "all"
+Requires-Dist: aiohttp-socks>=0.11.0; extra == "all"
 Requires-Dist: url-normalize>=1.4.0; extra == "all"
 Requires-Dist: trafilatura>=1.12.0; extra == "all"
 Requires-Dist: tiktoken>=0.7.0; extra == "all"
@@ -96,7 +98,7 @@ Dynamic: license-file
 
 # docpull
 
-**Security-hardened, browser-free web scraper and crawler that turns server-rendered pages into clean, AI-ready Markdown — fast.**
+**Public web to agent-ready Markdown. Fast, local, browser-free.**
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![PyPI version](https://badge.fury.io/py/docpull.svg)](https://badge.fury.io/py/docpull)
@@ -110,37 +112,35 @@ Dynamic: license-file
   </a>
 </p>
 
-## Download History
+docpull is a Python CLI, SDK, and MCP server for pulling public static and
+server-rendered web pages into clean Markdown, NDJSON, OKF bundles, SQLite, or
+local archives. Web-source extraction is the core workflow; docs are one
+high-value lane alongside blogs, API references, vendor pages, OpenAPI specs,
+and other server-rendered web content. It uses async HTTP instead
+of Playwright, preserves source metadata, and is built for agent-selected URLs
+with SSRF, XXE, DNS-rebinding, and CRLF-injection protections enabled by
+default.
 
-<a href="https://pepy.tech/project/docpull">
-  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
-</a>
+Use docpull when you need to:
 
-## Star History
+- Load public web pages, docs, API references, or blogs into an LLM context.
+- Build inspectable source corpora for RAG/search pipelines.
+- Give a coding agent a local MCP tool for fetching, caching, grepping, and
+  reading docs.
+- Mirror public web content for offline work while keeping source attribution.
 
-<a href="https://star-history.com/#raintree-technology/docpull&Date">
-  <picture>
-    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
-    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
-    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
-  </picture>
-</a>
+docpull intentionally does not render JavaScript. For JS-only apps, browser
+automation, paid extraction APIs, or general web crawling, see
+[`docs/scraping-boundary.md`](docs/scraping-boundary.md) and
+[`docs/alternatives.md`](docs/alternatives.md).
+
+## Try it in 60 seconds
 
-docpull is a web scraper for static and server-rendered sites, with
-documentation crawling as its sharpest default workflow. It uses async HTTP (not
-Playwright) to fetch pages, discover links, extract main content, and write clean
-Markdown with source-URL frontmatter — in seconds, with a small install
-footprint. It won't render JavaScript, but for the large class of sites that
-don't need it (API references, vendor docs, Python/Go stdlib, blogs, OpenAPI
-specs, Next.js and Docusaurus builds), it is a fast, auditable,
-sandbox-friendly way to pipe web content into an LLM context, a RAG index, or an
-offline archive. SSRF, XXE, DNS-rebinding, and CRLF-injection protections are on
-by default — a necessity when an AI agent is choosing the URLs.
-
-docpull is intentionally not a general browser-automation scraper. See
-[`docs/scraping-boundary.md`](docs/scraping-boundary.md) for the exact product
-boundary and when to use Scrapy, Crawlee, hosted extraction APIs, or trafilatura
-directly.
+```bash
+pip install docpull
+docpull https://docs.python.org/3/library/asyncio.html --single
+docpull https://docs.python.org/3/library/asyncio.html --profile llm --stream | jq .
+```
 
 ## Install
 
@@ -169,12 +169,33 @@ docpull https://docs.example.com/guide --single
 docpull https://docs.example.com --profile llm --stream | jq .
 
 # Open Knowledge Format bundle for agent/wiki interoperability
-docpull https://docs.example.com --format okf
+docpull https://example.com --format okf -o ./site-okf
+
+# SEC filing evidence pack with citation anchors
+docpull evidence-pack filings.ndjson \
+  --profile sec-filing \
+  --rules docs/examples/vendor-dependency-rules.yml \
+  --sec-user-agent "YourOrg your-email@example.com" \
+  --output-dir packs/dla-vendors
 
 # Mirror scraped content for offline use
 docpull https://docs.example.com --profile mirror --cache
 ```
 
+## Project traction
+
+<a href="https://pepy.tech/project/docpull">
+  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
+</a>
+
+<a href="https://star-history.com/#raintree-technology/docpull&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+  </picture>
+</a>
+
 ## Framework-aware extraction
 
 docpull inspects each page before running the generic extractor and can pull
@@ -280,15 +301,37 @@ async def tool_call(url: str) -> str:
 docpull https://site.com --profile rag      # Default. Dedup, rich metadata.
 docpull https://site.com --profile llm      # NDJSON + chunks + metadata; JS-only pages skip unless --strict-js-required is passed.
 docpull https://site.com --profile okf      # OKF bundle with generated index.md files.
+docpull https://site.com --profile sec-filing # EDGAR-friendly NDJSON chunks, trafilatura, Inline XBRL cleanup.
 docpull https://site.com --profile mirror   # Full archive, polite, cached, hierarchical paths.
 docpull https://site.com --profile quick    # Sampling: 50 pages, depth 2.
 ```
 
+## SEC filing evidence packs
+
+`docpull evidence-pack` turns an NDJSON list of filings into a source pack for
+government/vendor-dependency research. Each input row needs a filing URL under
+`primary_document_url` or `url`; filing fields such as `cik`,
+`accession_number`, `form`, `filing_date`, and `issuer_name` are carried into
+the emitted records.
+
+Use `docs/examples/vendor-dependency-rules.yml` as a starting rule profile for
+government customer, customer concentration, segment revenue, and related-party
+signals. Rules can use literal strings or regex entries with optional
+confidence values.
+
+The pack writes `evidence.pack.json`, `documents.ndjson`, `evidence.ndjson`,
+`diagnostics.ndjson`, `sources.md`, `corpus.manifest.json`,
+`EVIDENCE_CONTEXT.md`, and `AGENT_CONTEXT.md`. Evidence hits include source URL,
+chunk id, section heading when available, quote, surrounding context, source
+document hash, confidence, and extraction method. Diagnostics flag no readable
+content, high XBRL noise, degraded tables, missing evidence categories, and
+source hash changes since the last run.
+
 ## Parallel context packs
 
 `docpull[parallel]` adds an optional source-discovery and research layer on top
 of Parallel web intelligence APIs. Use the core crawler when you already know
-the docs URL. Use Parallel packs when an agent needs current web sources found,
+the target URL. Use Parallel packs when an agent needs current web sources found,
 extracted, researched, organized, and loaded as durable local context.
 
 Parallel handles live Search, Extract, Task, Entity Search, FindAll, TaskGroup,
@@ -511,7 +554,7 @@ Write:
 - `parallel_context_pack(...)` — build or dry-run a Parallel Search + Extract pack without shelling out.
 - `parallel_api_pack(source, kind?, output_dir?)` — build a pack from `llms.txt` or OpenAPI.
 - `add_source(name, url, description?, category?, max_pages?, force?)` — register a user alias (HTTPS-only, atomic write to `sources.yaml`).
-- `remove_source(name, delete_cache?)` — drop a user alias and (optionally) its cached docs.
+- `remove_source(name, delete_cache?)` — drop a user alias and (optionally) its cached Markdown.
 
 All schema-backed tools return `structuredContent` validated against an
 `outputSchema` for clients that prefer typed output. `fetch_url` intentionally
@@ -564,7 +607,22 @@ NDJSON (one record per page or chunk):
 Open Knowledge Format:
 
 ```bash
-docpull https://docs.example.com --format okf
+docpull https://example.com --format okf -o ./site-okf
+
+# Equivalent profile form
+docpull https://example.com --profile okf -o ./site-okf
+```
+
+A generated OKF bundle has a normal Markdown tree:
+
+```text
+site-okf/
+  index.md
+  corpus.manifest.json
+  _root.md
+  docs/
+    index.md
+    getting-started.md
 ```
 
 OKF output is an opt-in Markdown bundle. Scraped pages are written as concept
@@ -610,6 +668,8 @@ for hit in hits:
 When running with `--proxy`, DNS pinning is delegated to the proxy. Pass
 `--require-pinned-dns` to refuse this configuration and keep the connector-
 level SSRF guarantees in effect.
+HTTP and HTTPS proxies work with the base install; SOCKS proxies require
+`pip install "docpull[proxy]"`.
 
 ## Options
 
@@ -617,7 +677,7 @@ Run `docpull --help` for the full list. Highlights:
 
 ```
 Core:
-  --profile {rag,mirror,quick,llm,okf}
+  --profile {rag,mirror,quick,llm,okf,sec-filing}
   --single                Fetch one URL (no crawl)
   --format {markdown,json,ndjson,sqlite,okf}
   --stream                Stream NDJSON to stdout

{docpull-4.3.0 → docpull-4.4.0}/README.md

@@ -1,6 +1,6 @@
 # docpull
 
-**Security-hardened, browser-free web scraper and crawler that turns server-rendered pages into clean, AI-ready Markdown — fast.**
+**Public web to agent-ready Markdown. Fast, local, browser-free.**
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![PyPI version](https://badge.fury.io/py/docpull.svg)](https://badge.fury.io/py/docpull)
@@ -14,37 +14,35 @@
   </a>
 </p>
 
-## Download History
+docpull is a Python CLI, SDK, and MCP server for pulling public static and
+server-rendered web pages into clean Markdown, NDJSON, OKF bundles, SQLite, or
+local archives. Web-source extraction is the core workflow; docs are one
+high-value lane alongside blogs, API references, vendor pages, OpenAPI specs,
+and other server-rendered web content. It uses async HTTP instead
+of Playwright, preserves source metadata, and is built for agent-selected URLs
+with SSRF, XXE, DNS-rebinding, and CRLF-injection protections enabled by
+default.
 
-<a href="https://pepy.tech/project/docpull">
-  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
-</a>
+Use docpull when you need to:
 
-## Star History
+- Load public web pages, docs, API references, or blogs into an LLM context.
+- Build inspectable source corpora for RAG/search pipelines.
+- Give a coding agent a local MCP tool for fetching, caching, grepping, and
+  reading docs.
+- Mirror public web content for offline work while keeping source attribution.
 
-<a href="https://star-history.com/#raintree-technology/docpull&Date">
-  <picture>
-    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
-    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
-    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
-  </picture>
-</a>
+docpull intentionally does not render JavaScript. For JS-only apps, browser
+automation, paid extraction APIs, or general web crawling, see
+[`docs/scraping-boundary.md`](docs/scraping-boundary.md) and
+[`docs/alternatives.md`](docs/alternatives.md).
+
+## Try it in 60 seconds
 
-docpull is a web scraper for static and server-rendered sites, with
-documentation crawling as its sharpest default workflow. It uses async HTTP (not
-Playwright) to fetch pages, discover links, extract main content, and write clean
-Markdown with source-URL frontmatter — in seconds, with a small install
-footprint. It won't render JavaScript, but for the large class of sites that
-don't need it (API references, vendor docs, Python/Go stdlib, blogs, OpenAPI
-specs, Next.js and Docusaurus builds), it is a fast, auditable,
-sandbox-friendly way to pipe web content into an LLM context, a RAG index, or an
-offline archive. SSRF, XXE, DNS-rebinding, and CRLF-injection protections are on
-by default — a necessity when an AI agent is choosing the URLs.
-
-docpull is intentionally not a general browser-automation scraper. See
-[`docs/scraping-boundary.md`](docs/scraping-boundary.md) for the exact product
-boundary and when to use Scrapy, Crawlee, hosted extraction APIs, or trafilatura
-directly.
+```bash
+pip install docpull
+docpull https://docs.python.org/3/library/asyncio.html --single
+docpull https://docs.python.org/3/library/asyncio.html --profile llm --stream | jq .
+```
 
 ## Install
 
@@ -73,12 +71,33 @@ docpull https://docs.example.com/guide --single
 docpull https://docs.example.com --profile llm --stream | jq .
 
 # Open Knowledge Format bundle for agent/wiki interoperability
-docpull https://docs.example.com --format okf
+docpull https://example.com --format okf -o ./site-okf
+
+# SEC filing evidence pack with citation anchors
+docpull evidence-pack filings.ndjson \
+  --profile sec-filing \
+  --rules docs/examples/vendor-dependency-rules.yml \
+  --sec-user-agent "YourOrg your-email@example.com" \
+  --output-dir packs/dla-vendors
 
 # Mirror scraped content for offline use
 docpull https://docs.example.com --profile mirror --cache
 ```
 
+## Project traction
+
+<a href="https://pepy.tech/project/docpull">
+  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
+</a>
+
+<a href="https://star-history.com/#raintree-technology/docpull&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+  </picture>
+</a>
+
 ## Framework-aware extraction
 
 docpull inspects each page before running the generic extractor and can pull
@@ -184,15 +203,37 @@ async def tool_call(url: str) -> str:
 docpull https://site.com --profile rag      # Default. Dedup, rich metadata.
 docpull https://site.com --profile llm      # NDJSON + chunks + metadata; JS-only pages skip unless --strict-js-required is passed.
 docpull https://site.com --profile okf      # OKF bundle with generated index.md files.
+docpull https://site.com --profile sec-filing # EDGAR-friendly NDJSON chunks, trafilatura, Inline XBRL cleanup.
 docpull https://site.com --profile mirror   # Full archive, polite, cached, hierarchical paths.
 docpull https://site.com --profile quick    # Sampling: 50 pages, depth 2.
 ```
 
+## SEC filing evidence packs
+
+`docpull evidence-pack` turns an NDJSON list of filings into a source pack for
+government/vendor-dependency research. Each input row needs a filing URL under
+`primary_document_url` or `url`; filing fields such as `cik`,
+`accession_number`, `form`, `filing_date`, and `issuer_name` are carried into
+the emitted records.
+
+Use `docs/examples/vendor-dependency-rules.yml` as a starting rule profile for
+government customer, customer concentration, segment revenue, and related-party
+signals. Rules can use literal strings or regex entries with optional
+confidence values.
+
+The pack writes `evidence.pack.json`, `documents.ndjson`, `evidence.ndjson`,
+`diagnostics.ndjson`, `sources.md`, `corpus.manifest.json`,
+`EVIDENCE_CONTEXT.md`, and `AGENT_CONTEXT.md`. Evidence hits include source URL,
+chunk id, section heading when available, quote, surrounding context, source
+document hash, confidence, and extraction method. Diagnostics flag no readable
+content, high XBRL noise, degraded tables, missing evidence categories, and
+source hash changes since the last run.
+
 ## Parallel context packs
 
 `docpull[parallel]` adds an optional source-discovery and research layer on top
 of Parallel web intelligence APIs. Use the core crawler when you already know
-the docs URL. Use Parallel packs when an agent needs current web sources found,
+the target URL. Use Parallel packs when an agent needs current web sources found,
 extracted, researched, organized, and loaded as durable local context.
 
 Parallel handles live Search, Extract, Task, Entity Search, FindAll, TaskGroup,
@@ -415,7 +456,7 @@ Write:
 - `parallel_context_pack(...)` — build or dry-run a Parallel Search + Extract pack without shelling out.
 - `parallel_api_pack(source, kind?, output_dir?)` — build a pack from `llms.txt` or OpenAPI.
 - `add_source(name, url, description?, category?, max_pages?, force?)` — register a user alias (HTTPS-only, atomic write to `sources.yaml`).
-- `remove_source(name, delete_cache?)` — drop a user alias and (optionally) its cached docs.
+- `remove_source(name, delete_cache?)` — drop a user alias and (optionally) its cached Markdown.
 
 All schema-backed tools return `structuredContent` validated against an
 `outputSchema` for clients that prefer typed output. `fetch_url` intentionally
@@ -468,7 +509,22 @@ NDJSON (one record per page or chunk):
 Open Knowledge Format:
 
 ```bash
-docpull https://docs.example.com --format okf
+docpull https://example.com --format okf -o ./site-okf
+
+# Equivalent profile form
+docpull https://example.com --profile okf -o ./site-okf
+```
+
+A generated OKF bundle has a normal Markdown tree:
+
+```text
+site-okf/
+  index.md
+  corpus.manifest.json
+  _root.md
+  docs/
+    index.md
+    getting-started.md
 ```
 
 OKF output is an opt-in Markdown bundle. Scraped pages are written as concept
@@ -514,6 +570,8 @@ for hit in hits:
 When running with `--proxy`, DNS pinning is delegated to the proxy. Pass
 `--require-pinned-dns` to refuse this configuration and keep the connector-
 level SSRF guarantees in effect.
+HTTP and HTTPS proxies work with the base install; SOCKS proxies require
+`pip install "docpull[proxy]"`.
 
 ## Options
 
@@ -521,7 +579,7 @@ Run `docpull --help` for the full list. Highlights:
 
 ```
 Core:
-  --profile {rag,mirror,quick,llm,okf}
+  --profile {rag,mirror,quick,llm,okf,sec-filing}
   --single                Fetch one URL (no crawl)
   --format {markdown,json,ndjson,sqlite,okf}
   --stream                Stream NDJSON to stdout

{docpull-4.3.0 → docpull-4.4.0}/pyproject.toml

@@ -4,9 +4,9 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "docpull"
-version = "4.3.0"
+version = "4.4.0"
 dynamic = []
-description = "Pull documentation from the web and convert to clean markdown"
+description = "Convert public web pages into clean Markdown for AI agents and RAG"
 readme = {file = "README.md", content-type = "text/markdown"}
 requires-python = ">=3.10"
 license = "MIT"
@@ -18,8 +18,22 @@ maintainers = [
     {name = "Raintree Technology", email = "support@raintree.technology"}
 ]
 keywords = [
-    "python", "markdown", "documentation", "web-scraping", "developer-tools",
-    "claude", "ai-training-data"
+    "python",
+    "markdown",
+    "documentation",
+    "web-extraction",
+    "web-scraping",
+    "web-crawler",
+    "crawler",
+    "developer-tools",
+    "cli",
+    "rag",
+    "llm",
+    "mcp",
+    "ai-agents",
+    "context-engineering",
+    "claude",
+    "ai-training-data",
 ]
 classifiers = [
     # Development Status
@@ -35,9 +49,7 @@ classifiers = [
     "Environment :: Console",
 
     # Topic
-    "Topic :: Documentation",
     "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
-    "Topic :: Software Development :: Documentation",
     "Topic :: Text Processing :: Markup :: HTML",
     "Topic :: Text Processing :: Markup :: Markdown",
     "Topic :: Utilities",
@@ -67,17 +79,17 @@ dependencies = [
     "defusedxml>=0.7.1",
     "extruct>=0.15.0",
     "aiohttp>=3.14.0",  # 3.14.0 fixes CVE-2026-34993 and CVE-2026-47265
-    "idna>=3.15",
+    "idna>=3.15",  # Security floor for transitive URL/IDNA handling; Dependabot raises as needed
     "regex>=2024.11.6",
     "rich>=13.0.0",
     "pyyaml>=6.0",
     "pydantic>=2.0",
-    "urllib3>=2.7.0",
+    "urllib3>=2.7.0",  # Security floor for transitive HTTP clients; Dependabot raises as needed
 ]
 
 [project.optional-dependencies]
 proxy = [
-    "aiohttp-socks>=0.8.0",
+    "aiohttp-socks>=0.11.0",
 ]
 normalize = [
     "url-normalize>=1.4.0",
@@ -104,7 +116,7 @@ llm = [
     "tiktoken>=0.7.0",
 ]
 all = [
-    "aiohttp-socks>=0.8.0",
+    "aiohttp-socks>=0.11.0",
     "url-normalize>=1.4.0",
     "trafilatura>=1.12.0",
     "tiktoken>=0.7.0",
@@ -134,12 +146,16 @@ dev = [
 docpull = "docpull.cli:main"
 
 [project.urls]
-Homepage = "https://github.com/raintree-technology/docpull"
-Documentation = "https://github.com/raintree-technology/docpull#readme"
+Homepage = "https://docpull.raintree.technology"
+Documentation = "https://docpull.raintree.technology"
 Repository = "https://github.com/raintree-technology/docpull"
 "Source Code" = "https://github.com/raintree-technology/docpull"
 "Bug Tracker" = "https://github.com/raintree-technology/docpull/issues"
 "Releases" = "https://github.com/raintree-technology/docpull/releases"
+"MCP Plugin" = "https://github.com/raintree-technology/docpull/tree/main/plugin"
+"Comparison Guide" = "https://github.com/raintree-technology/docpull/blob/main/docs/alternatives.md"
+"Download Stats" = "https://pepy.tech/project/docpull"
+"Changelog" = "https://github.com/raintree-technology/docpull/blob/main/docs/CHANGELOG.md"
 
 [tool.setuptools.packages.find]
 where = ["src"]

{docpull-4.3.0 → docpull-4.4.0}/src/docpull/__init__.py

@@ -1,5 +1,5 @@
 """
-docpull - Fetch and convert static/server-rendered documentation to markdown.
+docpull - Fetch and convert static/server-rendered web content to markdown.
 
 Usage:
     from docpull import Fetcher, DocpullConfig, ProfileName
@@ -14,7 +14,7 @@ Usage:
             print(event)
 """
 
-__version__ = "4.3.0"
+__version__ = "4.4.0"
 
 from .cache import CacheManager, StreamingDeduplicator
 from .conversion.chunking import Chunk, TokenCounter, chunk_markdown

{docpull-4.3.0 → docpull-4.4.0}/src/docpull/benchmark.py

@@ -237,7 +237,7 @@ def create_benchmark_parser() -> argparse.ArgumentParser:
         "quick",
         help="Run a small real-site benchmark and write JSON/Markdown reports",
     )
-    quick.add_argument("--target-url", default=DEFAULT_TARGET_URL, help="Docs URL to crawl")
+    quick.add_argument("--target-url", default=DEFAULT_TARGET_URL, help="Target URL to crawl")
     quick.add_argument(
         "--target-set",
         choices=TARGET_SET_CHOICES,

{docpull-4.3.0 → docpull-4.4.0}/src/docpull/cli.py

@@ -48,7 +48,7 @@ def create_parser() -> argparse.ArgumentParser:
     """Create argument parser for CLI."""
     parser = argparse.ArgumentParser(
         prog="docpull",
-        description="Fetch and convert static/server-rendered documentation to markdown",
+        description="Fetch and convert static/server-rendered web content to markdown",
         formatter_class=argparse.RawDescriptionHelpFormatter,
         epilog="""
 Examples:
@@ -69,7 +69,7 @@ Examples:
     parser.add_argument(
         "url",
         nargs="?",
-        help="URL to fetch documentation from",
+        help="URL to fetch content from",
     )
 
     parser.add_argument(
@@ -87,9 +87,12 @@ Examples:
     parser.add_argument(
         "--profile",
         "-p",
-        choices=["rag", "mirror", "quick", "llm", "okf"],
+        choices=["rag", "mirror", "quick", "llm", "okf", "sec-filing"],
         default="rag",
-        help="Preset profile (default: rag). 'llm' streams chunked NDJSON; 'okf' writes an OKF bundle.",
+        help=(
+            "Preset profile (default: rag). 'llm' streams chunked NDJSON; "
+            "'okf' writes an OKF bundle; 'sec-filing' tunes extraction for EDGAR filings."
+        ),
     )
 
     parser.add_argument(
@@ -257,7 +260,7 @@ Examples:
         "--proxy",
         type=str,
         metavar="URL",
-        help="Proxy URL",
+        help="HTTP, HTTPS, or SOCKS proxy URL (SOCKS requires docpull[proxy])",
     )
     network_group.add_argument(
         "--user-agent",
@@ -384,6 +387,7 @@ def run_fetcher(args: argparse.Namespace) -> int:
         "quick": ProfileName.QUICK,
         "llm": ProfileName.LLM,
         "okf": ProfileName.OKF,
+        "sec-filing": ProfileName.SEC_FILING,
     }
     profile = profile_map.get(args.profile, ProfileName.RAG)
 
@@ -683,6 +687,10 @@ def main(argv: list[str] | None = None) -> int:
         from .pack_tools import run_pack_cli
 
         return run_pack_cli(raw_argv[1:])
+    if raw_argv and raw_argv[0] == "evidence-pack":
+        from .evidence_pack import run_evidence_pack_cli
+
+        return run_evidence_pack_cli(raw_argv[1:])
     if raw_argv and raw_argv[0] == "benchmark":
         from .benchmark import run_benchmark_cli