docpull 4.3.0__tar.gz → 4.3.1__tar.gz

{docpull-4.3.0/src/docpull.egg-info → docpull-4.3.1}/PKG-INFO

@@ -1,17 +1,21 @@
 Metadata-Version: 2.4
 Name: docpull
-Version: 4.3.0
-Summary: Pull documentation from the web and convert to clean markdown
+Version: 4.3.1
+Summary: Convert public web pages into clean Markdown for AI agents and RAG
 Author-email: Zachary Roth <support@raintree.technology>
 Maintainer-email: Raintree Technology <support@raintree.technology>
 License-Expression: MIT
-Project-URL: Homepage, https://github.com/raintree-technology/docpull
-Project-URL: Documentation, https://github.com/raintree-technology/docpull#readme
+Project-URL: Homepage, https://docpull.raintree.technology
+Project-URL: Documentation, https://docpull.raintree.technology
 Project-URL: Repository, https://github.com/raintree-technology/docpull
 Project-URL: Source Code, https://github.com/raintree-technology/docpull
 Project-URL: Bug Tracker, https://github.com/raintree-technology/docpull/issues
 Project-URL: Releases, https://github.com/raintree-technology/docpull/releases
-Keywords: python,markdown,documentation,web-scraping,developer-tools,claude,ai-training-data
+Project-URL: MCP Plugin, https://github.com/raintree-technology/docpull/tree/main/plugin
+Project-URL: Comparison Guide, https://github.com/raintree-technology/docpull/blob/main/docs/alternatives.md
+Project-URL: Download Stats, https://pepy.tech/project/docpull
+Project-URL: Changelog, https://github.com/raintree-technology/docpull/blob/main/docs/CHANGELOG.md
+Keywords: python,markdown,documentation,web-scraping,web-crawler,crawler,developer-tools,cli,rag,llm,mcp,ai-agents,claude,ai-training-data
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Information Technology
@@ -96,7 +100,7 @@ Dynamic: license-file
 
 # docpull
 
-**Security-hardened, browser-free web scraper and crawler that turns server-rendered pages into clean, AI-ready Markdown — fast.**
+**Public web to agent-ready Markdown. Fast, local, browser-free.**
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![PyPI version](https://badge.fury.io/py/docpull.svg)](https://badge.fury.io/py/docpull)
@@ -110,37 +114,35 @@ Dynamic: license-file
   </a>
 </p>
 
-## Download History
+docpull is a Python CLI, SDK, and MCP server for pulling public static and
+server-rendered web pages into clean Markdown, NDJSON, OKF bundles, SQLite, or
+local archives. Documentation crawling is its sharpest default workflow, but
+the product is broader than docs: blogs, API references, vendor pages, OpenAPI
+specs, and other server-rendered web content all fit. It uses async HTTP instead
+of Playwright, preserves source metadata, and is built for agent-selected URLs
+with SSRF, XXE, DNS-rebinding, and CRLF-injection protections enabled by
+default.
 
-<a href="https://pepy.tech/project/docpull">
-  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
-</a>
+Use docpull when you need to:
 
-## Star History
+- Load public web pages, docs, API references, or blogs into an LLM context.
+- Build inspectable source corpora for RAG/search pipelines.
+- Give a coding agent a local MCP tool for fetching, caching, grepping, and
+  reading docs.
+- Mirror public web content for offline work while keeping source attribution.
 
-<a href="https://star-history.com/#raintree-technology/docpull&Date">
-  <picture>
-    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
-    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
-    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
-  </picture>
-</a>
+docpull intentionally does not render JavaScript. For JS-only apps, browser
+automation, paid extraction APIs, or general web crawling, see
+[`docs/scraping-boundary.md`](docs/scraping-boundary.md) and
+[`docs/alternatives.md`](docs/alternatives.md).
 
-docpull is a web scraper for static and server-rendered sites, with
-documentation crawling as its sharpest default workflow. It uses async HTTP (not
-Playwright) to fetch pages, discover links, extract main content, and write clean
-Markdown with source-URL frontmatter — in seconds, with a small install
-footprint. It won't render JavaScript, but for the large class of sites that
-don't need it (API references, vendor docs, Python/Go stdlib, blogs, OpenAPI
-specs, Next.js and Docusaurus builds), it is a fast, auditable,
-sandbox-friendly way to pipe web content into an LLM context, a RAG index, or an
-offline archive. SSRF, XXE, DNS-rebinding, and CRLF-injection protections are on
-by default — a necessity when an AI agent is choosing the URLs.
-
-docpull is intentionally not a general browser-automation scraper. See
-[`docs/scraping-boundary.md`](docs/scraping-boundary.md) for the exact product
-boundary and when to use Scrapy, Crawlee, hosted extraction APIs, or trafilatura
-directly.
+## Try it in 60 seconds
+
+```bash
+pip install docpull
+docpull https://docs.python.org/3/library/asyncio.html --single
+docpull https://docs.python.org/3/library/asyncio.html --profile llm --stream | jq .
+```
 
 ## Install
 
@@ -169,12 +171,26 @@ docpull https://docs.example.com/guide --single
 docpull https://docs.example.com --profile llm --stream | jq .
 
 # Open Knowledge Format bundle for agent/wiki interoperability
-docpull https://docs.example.com --format okf
+docpull https://example.com --format okf -o ./site-okf
 
 # Mirror scraped content for offline use
 docpull https://docs.example.com --profile mirror --cache
 ```
 
+## Project traction
+
+<a href="https://pepy.tech/project/docpull">
+  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
+</a>
+
+<a href="https://star-history.com/#raintree-technology/docpull&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+  </picture>
+</a>
+
 ## Framework-aware extraction
 
 docpull inspects each page before running the generic extractor and can pull
@@ -564,7 +580,22 @@ NDJSON (one record per page or chunk):
 Open Knowledge Format:
 
 ```bash
-docpull https://docs.example.com --format okf
+docpull https://example.com --format okf -o ./site-okf
+
+# Equivalent profile form
+docpull https://example.com --profile okf -o ./site-okf
+```
+
+A generated OKF bundle has a normal Markdown tree:
+
+```text
+site-okf/
+  index.md
+  corpus.manifest.json
+  _root.md
+  docs/
+    index.md
+    getting-started.md
 ```
 
 OKF output is an opt-in Markdown bundle. Scraped pages are written as concept

{docpull-4.3.0 → docpull-4.3.1}/README.md

@@ -1,6 +1,6 @@
 # docpull
 
-**Security-hardened, browser-free web scraper and crawler that turns server-rendered pages into clean, AI-ready Markdown — fast.**
+**Public web to agent-ready Markdown. Fast, local, browser-free.**
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![PyPI version](https://badge.fury.io/py/docpull.svg)](https://badge.fury.io/py/docpull)
@@ -14,37 +14,35 @@
   </a>
 </p>
 
-## Download History
+docpull is a Python CLI, SDK, and MCP server for pulling public static and
+server-rendered web pages into clean Markdown, NDJSON, OKF bundles, SQLite, or
+local archives. Documentation crawling is its sharpest default workflow, but
+the product is broader than docs: blogs, API references, vendor pages, OpenAPI
+specs, and other server-rendered web content all fit. It uses async HTTP instead
+of Playwright, preserves source metadata, and is built for agent-selected URLs
+with SSRF, XXE, DNS-rebinding, and CRLF-injection protections enabled by
+default.
 
-<a href="https://pepy.tech/project/docpull">
-  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
-</a>
+Use docpull when you need to:
 
-## Star History
+- Load public web pages, docs, API references, or blogs into an LLM context.
+- Build inspectable source corpora for RAG/search pipelines.
+- Give a coding agent a local MCP tool for fetching, caching, grepping, and
+  reading docs.
+- Mirror public web content for offline work while keeping source attribution.
 
-<a href="https://star-history.com/#raintree-technology/docpull&Date">
-  <picture>
-    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
-    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
-    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
-  </picture>
-</a>
+docpull intentionally does not render JavaScript. For JS-only apps, browser
+automation, paid extraction APIs, or general web crawling, see
+[`docs/scraping-boundary.md`](docs/scraping-boundary.md) and
+[`docs/alternatives.md`](docs/alternatives.md).
 
-docpull is a web scraper for static and server-rendered sites, with
-documentation crawling as its sharpest default workflow. It uses async HTTP (not
-Playwright) to fetch pages, discover links, extract main content, and write clean
-Markdown with source-URL frontmatter — in seconds, with a small install
-footprint. It won't render JavaScript, but for the large class of sites that
-don't need it (API references, vendor docs, Python/Go stdlib, blogs, OpenAPI
-specs, Next.js and Docusaurus builds), it is a fast, auditable,
-sandbox-friendly way to pipe web content into an LLM context, a RAG index, or an
-offline archive. SSRF, XXE, DNS-rebinding, and CRLF-injection protections are on
-by default — a necessity when an AI agent is choosing the URLs.
-
-docpull is intentionally not a general browser-automation scraper. See
-[`docs/scraping-boundary.md`](docs/scraping-boundary.md) for the exact product
-boundary and when to use Scrapy, Crawlee, hosted extraction APIs, or trafilatura
-directly.
+## Try it in 60 seconds
+
+```bash
+pip install docpull
+docpull https://docs.python.org/3/library/asyncio.html --single
+docpull https://docs.python.org/3/library/asyncio.html --profile llm --stream | jq .
+```
 
 ## Install
 
@@ -73,12 +71,26 @@ docpull https://docs.example.com/guide --single
 docpull https://docs.example.com --profile llm --stream | jq .
 
 # Open Knowledge Format bundle for agent/wiki interoperability
-docpull https://docs.example.com --format okf
+docpull https://example.com --format okf -o ./site-okf
 
 # Mirror scraped content for offline use
 docpull https://docs.example.com --profile mirror --cache
 ```
 
+## Project traction
+
+<a href="https://pepy.tech/project/docpull">
+  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
+</a>
+
+<a href="https://star-history.com/#raintree-technology/docpull&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+  </picture>
+</a>
+
 ## Framework-aware extraction
 
 docpull inspects each page before running the generic extractor and can pull
@@ -468,7 +480,22 @@ NDJSON (one record per page or chunk):
 Open Knowledge Format:
 
 ```bash
-docpull https://docs.example.com --format okf
+docpull https://example.com --format okf -o ./site-okf
+
+# Equivalent profile form
+docpull https://example.com --profile okf -o ./site-okf
+```
+
+A generated OKF bundle has a normal Markdown tree:
+
+```text
+site-okf/
+  index.md
+  corpus.manifest.json
+  _root.md
+  docs/
+    index.md
+    getting-started.md
 ```
 
 OKF output is an opt-in Markdown bundle. Scraped pages are written as concept

{docpull-4.3.0 → docpull-4.3.1}/pyproject.toml

@@ -4,9 +4,9 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "docpull"
-version = "4.3.0"
+version = "4.3.1"
 dynamic = []
-description = "Pull documentation from the web and convert to clean markdown"
+description = "Convert public web pages into clean Markdown for AI agents and RAG"
 readme = {file = "README.md", content-type = "text/markdown"}
 requires-python = ">=3.10"
 license = "MIT"
@@ -18,8 +18,20 @@ maintainers = [
     {name = "Raintree Technology", email = "support@raintree.technology"}
 ]
 keywords = [
-    "python", "markdown", "documentation", "web-scraping", "developer-tools",
-    "claude", "ai-training-data"
+    "python",
+    "markdown",
+    "documentation",
+    "web-scraping",
+    "web-crawler",
+    "crawler",
+    "developer-tools",
+    "cli",
+    "rag",
+    "llm",
+    "mcp",
+    "ai-agents",
+    "claude",
+    "ai-training-data",
 ]
 classifiers = [
     # Development Status
@@ -134,12 +146,16 @@ dev = [
 docpull = "docpull.cli:main"
 
 [project.urls]
-Homepage = "https://github.com/raintree-technology/docpull"
-Documentation = "https://github.com/raintree-technology/docpull#readme"
+Homepage = "https://docpull.raintree.technology"
+Documentation = "https://docpull.raintree.technology"
 Repository = "https://github.com/raintree-technology/docpull"
 "Source Code" = "https://github.com/raintree-technology/docpull"
 "Bug Tracker" = "https://github.com/raintree-technology/docpull/issues"
 "Releases" = "https://github.com/raintree-technology/docpull/releases"
+"MCP Plugin" = "https://github.com/raintree-technology/docpull/tree/main/plugin"
+"Comparison Guide" = "https://github.com/raintree-technology/docpull/blob/main/docs/alternatives.md"
+"Download Stats" = "https://pepy.tech/project/docpull"
+"Changelog" = "https://github.com/raintree-technology/docpull/blob/main/docs/CHANGELOG.md"
 
 [tool.setuptools.packages.find]
 where = ["src"]

{docpull-4.3.0 → docpull-4.3.1}/src/docpull/__init__.py

@@ -14,7 +14,7 @@ Usage:
             print(event)
 """
 
-__version__ = "4.3.0"
+__version__ = "4.3.1"
 
 from .cache import CacheManager, StreamingDeduplicator
 from .conversion.chunking import Chunk, TokenCounter, chunk_markdown

{docpull-4.3.0 → docpull-4.3.1/src/docpull.egg-info}/PKG-INFO

@@ -1,17 +1,21 @@
 Metadata-Version: 2.4
 Name: docpull
-Version: 4.3.0
-Summary: Pull documentation from the web and convert to clean markdown
+Version: 4.3.1
+Summary: Convert public web pages into clean Markdown for AI agents and RAG
 Author-email: Zachary Roth <support@raintree.technology>
 Maintainer-email: Raintree Technology <support@raintree.technology>
 License-Expression: MIT
-Project-URL: Homepage, https://github.com/raintree-technology/docpull
-Project-URL: Documentation, https://github.com/raintree-technology/docpull#readme
+Project-URL: Homepage, https://docpull.raintree.technology
+Project-URL: Documentation, https://docpull.raintree.technology
 Project-URL: Repository, https://github.com/raintree-technology/docpull
 Project-URL: Source Code, https://github.com/raintree-technology/docpull
 Project-URL: Bug Tracker, https://github.com/raintree-technology/docpull/issues
 Project-URL: Releases, https://github.com/raintree-technology/docpull/releases
-Keywords: python,markdown,documentation,web-scraping,developer-tools,claude,ai-training-data
+Project-URL: MCP Plugin, https://github.com/raintree-technology/docpull/tree/main/plugin
+Project-URL: Comparison Guide, https://github.com/raintree-technology/docpull/blob/main/docs/alternatives.md
+Project-URL: Download Stats, https://pepy.tech/project/docpull
+Project-URL: Changelog, https://github.com/raintree-technology/docpull/blob/main/docs/CHANGELOG.md
+Keywords: python,markdown,documentation,web-scraping,web-crawler,crawler,developer-tools,cli,rag,llm,mcp,ai-agents,claude,ai-training-data
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Information Technology
@@ -96,7 +100,7 @@ Dynamic: license-file
 
 # docpull
 
-**Security-hardened, browser-free web scraper and crawler that turns server-rendered pages into clean, AI-ready Markdown — fast.**
+**Public web to agent-ready Markdown. Fast, local, browser-free.**
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![PyPI version](https://badge.fury.io/py/docpull.svg)](https://badge.fury.io/py/docpull)
@@ -110,37 +114,35 @@ Dynamic: license-file
   </a>
 </p>
 
-## Download History
+docpull is a Python CLI, SDK, and MCP server for pulling public static and
+server-rendered web pages into clean Markdown, NDJSON, OKF bundles, SQLite, or
+local archives. Documentation crawling is its sharpest default workflow, but
+the product is broader than docs: blogs, API references, vendor pages, OpenAPI
+specs, and other server-rendered web content all fit. It uses async HTTP instead
+of Playwright, preserves source metadata, and is built for agent-selected URLs
+with SSRF, XXE, DNS-rebinding, and CRLF-injection protections enabled by
+default.
 
-<a href="https://pepy.tech/project/docpull">
-  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
-</a>
+Use docpull when you need to:
 
-## Star History
+- Load public web pages, docs, API references, or blogs into an LLM context.
+- Build inspectable source corpora for RAG/search pipelines.
+- Give a coding agent a local MCP tool for fetching, caching, grepping, and
+  reading docs.
+- Mirror public web content for offline work while keeping source attribution.
 
-<a href="https://star-history.com/#raintree-technology/docpull&Date">
-  <picture>
-    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
-    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
-    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
-  </picture>
-</a>
+docpull intentionally does not render JavaScript. For JS-only apps, browser
+automation, paid extraction APIs, or general web crawling, see
+[`docs/scraping-boundary.md`](docs/scraping-boundary.md) and
+[`docs/alternatives.md`](docs/alternatives.md).
 
-docpull is a web scraper for static and server-rendered sites, with
-documentation crawling as its sharpest default workflow. It uses async HTTP (not
-Playwright) to fetch pages, discover links, extract main content, and write clean
-Markdown with source-URL frontmatter — in seconds, with a small install
-footprint. It won't render JavaScript, but for the large class of sites that
-don't need it (API references, vendor docs, Python/Go stdlib, blogs, OpenAPI
-specs, Next.js and Docusaurus builds), it is a fast, auditable,
-sandbox-friendly way to pipe web content into an LLM context, a RAG index, or an
-offline archive. SSRF, XXE, DNS-rebinding, and CRLF-injection protections are on
-by default — a necessity when an AI agent is choosing the URLs.
-
-docpull is intentionally not a general browser-automation scraper. See
-[`docs/scraping-boundary.md`](docs/scraping-boundary.md) for the exact product
-boundary and when to use Scrapy, Crawlee, hosted extraction APIs, or trafilatura
-directly.
+## Try it in 60 seconds
+
+```bash
+pip install docpull
+docpull https://docs.python.org/3/library/asyncio.html --single
+docpull https://docs.python.org/3/library/asyncio.html --profile llm --stream | jq .
+```
 
 ## Install
 
@@ -169,12 +171,26 @@ docpull https://docs.example.com/guide --single
 docpull https://docs.example.com --profile llm --stream | jq .
 
 # Open Knowledge Format bundle for agent/wiki interoperability
-docpull https://docs.example.com --format okf
+docpull https://example.com --format okf -o ./site-okf
 
 # Mirror scraped content for offline use
 docpull https://docs.example.com --profile mirror --cache
 ```
 
+## Project traction
+
+<a href="https://pepy.tech/project/docpull">
+  <img alt="Cumulative PyPI download history chart for docpull" src="docs/cumulative-downloads-history.svg" />
+</a>
+
+<a href="https://star-history.com/#raintree-technology/docpull&Date">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date&theme=dark" />
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+    <img alt="Star history chart for raintree-technology/docpull" src="https://api.star-history.com/svg?repos=raintree-technology/docpull&type=Date" />
+  </picture>
+</a>
+
 ## Framework-aware extraction
 
 docpull inspects each page before running the generic extractor and can pull
@@ -564,7 +580,22 @@ NDJSON (one record per page or chunk):
 Open Knowledge Format:
 
 ```bash
-docpull https://docs.example.com --format okf
+docpull https://example.com --format okf -o ./site-okf
+
+# Equivalent profile form
+docpull https://example.com --profile okf -o ./site-okf
+```
+
+A generated OKF bundle has a normal Markdown tree:
+
+```text
+site-okf/
+  index.md
+  corpus.manifest.json
+  _root.md
+  docs/
+    index.md
+    getting-started.md
 ```
 
 OKF output is an opt-in Markdown bundle. Scraped pages are written as concept