docpull 1.0.1__py3-none-any.whl

docpull-1.0.1.dist-info/METADATA

@@ -0,0 +1,440 @@
+Metadata-Version: 2.4
+Name: docpull
+Version: 1.0.1
+Summary: Pull documentation from the web and convert to clean markdown
+Author-email: Zachary Roth <support@raintree.technology>
+Maintainer-email: Raintree Technology <support@raintree.technology>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/raintree-technology/docpull
+Project-URL: Documentation, https://github.com/raintree-technology/docpull#readme
+Project-URL: Repository, https://github.com/raintree-technology/docpull
+Project-URL: Source Code, https://github.com/raintree-technology/docpull
+Project-URL: Bug Tracker, https://github.com/raintree-technology/docpull/issues
+Project-URL: Changelog, https://github.com/raintree-technology/docpull/blob/main/CHANGELOG.md
+Keywords: python,markdown,documentation,web-scraping,developer-tools,claude,ai-training-data
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Education
+Classifier: Environment :: Console
+Classifier: Topic :: Documentation
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Documentation
+Classifier: Topic :: Text Processing :: Markup :: HTML
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Classifier: Topic :: Utilities
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.31.0
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: html2text>=2020.1.16
+Requires-Dist: defusedxml>=0.7.1
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: rich>=13.0.0
+Provides-Extra: yaml
+Requires-Dist: pyyaml>=6.0; extra == "yaml"
+Provides-Extra: js
+Requires-Dist: playwright>=1.40.0; extra == "js"
+Provides-Extra: all
+Requires-Dist: pyyaml>=6.0; extra == "all"
+Requires-Dist: playwright>=1.40.0; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: bandit>=1.7.0; extra == "dev"
+Requires-Dist: pip-audit>=2.0.0; extra == "dev"
+Requires-Dist: types-requests>=2.31.0; extra == "dev"
+Requires-Dist: types-beautifulsoup4>=4.12.0; extra == "dev"
+Requires-Dist: types-aiohttp>=3.9.0; extra == "dev"
+Dynamic: license-file
+
+# docpull
+
+**Pull documentation from ANY website and convert to clean, AI-ready markdown.**
+
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![PyPI version](https://badge.fury.io/py/docpull.svg)](https://badge.fury.io/py/docpull)
+[![License: MIT](https://img.shields.io/github/license/raintree-technology/docpull)](https://github.com/raintree-technology/docpull/blob/main/LICENSE)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Type checked: mypy](https://img.shields.io/badge/type%20checked-mypy-blue.svg)](http://mypy-lang.org/)
+[![Security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg)](https://github.com/PyCQA/bandit)
+
+## Why docpull?
+
+Unlike wget or httrack that dump messy HTML, **docpull extracts clean markdown** perfect for:
+- Training AI models / RAG systems
+- Building knowledge bases
+- Creating searchable documentation archives
+- Offline documentation reading
+
+**Production-ready**: Full type safety (mypy), security scanning (Bandit), zero linting issues (Ruff), comprehensive test coverage, and no known vulnerabilities.
+
+## Features
+
+- **Universal**: Scrape ANY documentation site - not limited to predefined sources
+- **Smart extraction**: Auto-detects main content, removes navigation/ads
+- **Blazing fast**: Async/parallel fetching (10x faster than sync)
+- **JavaScript support**: Handles JS-heavy sites with Playwright
+- **Progress bars**: Beautiful real-time progress with Rich
+- **Sitemap support**: Auto-discovers pages via sitemap.xml
+- **Link crawling**: Optionally follows links to discover all pages
+- **Secure**: Rate limiting, content validation, timeout controls
+- **Clean output**: Markdown with YAML frontmatter
+- **Configurable**: Control depth, page limits, concurrency
+- **Resumable**: Skip already-fetched files
+
+## Quick Start
+
+```bash
+# Install
+pip install docpull
+
+# Scrape ANY documentation site
+docpull https://aptos.dev
+docpull https://docs.anthropic.com
+docpull https://go.dev/doc
+
+# Use optimized profiles for popular sites
+docpull stripe
+docpull nextjs react
+
+# Control scraping behavior
+docpull https://newsite.com/docs --max-pages 100 --max-concurrent 20
+```
+
+## Installation
+
+```bash
+# Basic installation
+pip install docpull
+
+# With YAML config support
+pip install docpull[yaml]
+
+# With JavaScript rendering (for JS-heavy sites)
+pip install docpull[js]
+python -m playwright install chromium
+
+# Everything
+pip install docpull[all]
+python -m playwright install chromium
+```
+
+## Usage
+
+### Scrape Any URL
+
+The primary way to use docpull is by providing any documentation URL:
+
+```bash
+# Single site
+docpull https://aptos.dev
+
+# Multiple sites
+docpull https://aptos.dev https://docs.soliditylang.org
+
+# Control crawling
+docpull https://docs.example.com \
+  --max-pages 200 \
+  --max-depth 4 \
+  --rate-limit 1.0
+```
+
+### Use Optimized Profiles
+
+For popular documentation sites, use shortcut names for optimized scraping:
+
+```bash
+# Single profile
+docpull stripe
+
+# Multiple profiles
+docpull stripe plaid nextjs
+
+# Mix profiles and URLs
+docpull stripe https://newsite.com/docs
+```
+
+### JavaScript Rendering
+
+For sites that require JavaScript to render content:
+
+```bash
+# Enable JS rendering with Playwright
+docpull https://js-heavy-site.com --js
+
+# Combine with other options
+docpull https://site.com --js --max-pages 50 --max-concurrent 5
+```
+
+**Note:** JS rendering is slower but handles modern SPAs and dynamically-loaded content.
+
+### Available Profiles
+
+| Profile | Site | Optimizations |
+|---------|------|---------------|
+| `stripe` | docs.stripe.com | Filters changelog, focused on API docs |
+| `nextjs` | nextjs.org | Excludes blog/showcase, docs only |
+| `react` | react.dev | Learn & reference sections only |
+| `plaid` | plaid.com | API + guides, excludes marketing |
+| `tailwind` | tailwindcss.com | Documentation only |
+| `bun` | bun.sh | Runtime documentation |
+| `d3` | d3js.org | Data visualization docs |
+| `turborepo` | turbo.build | Monorepo tooling docs |
+
+### Python API
+
+```python
+from docpull import GenericAsyncFetcher
+
+# Scrape any URL (async/parallel)
+fetcher = GenericAsyncFetcher(
+    url_or_profile="https://aptos.dev",
+    output_dir="./docs",
+    max_pages=100,
+    max_concurrent=20,
+    use_js=False,  # Set to True for JS rendering
+)
+fetcher.fetch()
+
+# Or use a profile
+fetcher = GenericAsyncFetcher(
+    url_or_profile="stripe",
+    output_dir="./docs",
+)
+fetcher.fetch()
+```
+
+### Advanced Options
+
+```bash
+# Limit pages and depth
+docpull https://docs.example.com --max-pages 50 --max-depth 2
+
+# Control concurrent requests (default: 10)
+docpull https://site.com --max-concurrent 20
+
+# Enable JavaScript rendering
+docpull https://site.com --js
+
+# Custom output directory
+docpull stripe --output-dir ./my-docs
+
+# Adjust rate limiting
+docpull https://site.com --rate-limit 2.0
+
+# Re-fetch existing files
+docpull stripe --no-skip-existing
+
+# Verbose logging
+docpull https://site.com --verbose
+
+# Disable progress bars
+docpull https://site.com --no-progress
+
+# Dry run (see what would be fetched)
+docpull https://site.com --dry-run
+```
+
+## Performance
+
+**Async/Parallel Fetching** makes docpull **10x faster** than traditional sync scrapers:
+
+| Pages | Sync (old) | Async (new) | Speedup |
+|-------|-----------|-------------|---------|
+| 5 | ~5.0s | ~1.8s | 2.8x faster |
+| 50 | ~50s | ~6s | 8.3x faster |
+| 500 | ~500s | ~45s | 11x faster |
+
+With `--max-concurrent 20`, even faster for large sites!
+
+## Output Format
+
+Each page is saved as markdown with YAML frontmatter:
+
+```markdown
+---
+url: https://stripe.com/docs/payments
+fetched: 2025-11-13
+---
+
+# Payment Intents
+
+Your clean documentation content here...
+```
+
+Files are organized by URL structure:
+
+```
+docs/
+├── stripe/
+│   ├── api/
+│   │   ├── charges.md
+│   │   └── customers.md
+│   └── payments/
+│       └── payment-intents.md
+└── aptos_dev/
+    ├── guides/
+    │   └── getting-started.md
+    └── reference/
+        └── api.md
+```
+
+## How It Works
+
+1. **Discovery**: Tries sitemap.xml first, falls back to link crawling
+2. **Filtering**: Applies URL patterns to focus on documentation
+3. **Extraction**: Removes nav/footer/ads, extracts main content
+4. **Conversion**: Converts HTML to clean markdown
+5. **Organization**: Saves with structure that mirrors the site
+6. **Async Magic**: Fetches multiple pages concurrently with rate limiting
+
+## Configuration File
+
+Create `config.yaml` for complex setups:
+
+```yaml
+output_dir: ./docs
+rate_limit: 0.5
+skip_existing: true
+log_level: INFO
+
+sources:
+  - stripe
+  - nextjs
+  - react
+```
+
+Run with:
+```bash
+docpull --config config.yaml
+```
+
+## Creating Custom Profiles
+
+You can create optimized profiles for your frequently-scraped sites:
+
+```python
+from docpull.profiles.base import SiteProfile
+
+MY_PROFILE = SiteProfile(
+    name="mysite",
+    domains={"docs.mysite.com"},
+    sitemap_url="https://docs.mysite.com/sitemap.xml",
+    base_url="https://docs.mysite.com/",
+    include_patterns=["/docs/", "/api/"],
+    exclude_patterns=["/blog/"],
+    output_subdir="mysite",
+    rate_limit=0.5,
+)
+```
+
+## Security
+
+docpull is designed with security in mind:
+
+- **HTTPS-only** by default
+- **Private IP blocking** (no localhost, 192.168.x.x, etc.)
+- **Content size limits** (50MB max per page)
+- **Timeout controls** (30s per request)
+- **Rate limiting** (async-safe, prevents DoS)
+- **Concurrent connection limits** (prevents overwhelming servers)
+- **Content-type validation** (only fetches HTML/XML)
+- **Playwright sandboxing** (when using --js)
+
+See [SECURITY.md](SECURITY.md) for detailed security information.
+
+## Comparison with Alternatives
+
+| Tool | Output | Works on any site? | Clean extraction? | Speed | JS Support |
+|------|--------|-------------------|-------------------|-------|------------|
+| **docpull** | Clean markdown | Yes | Yes | Fast (async) | Optional |
+| wget | Raw HTML | Yes | No | Slow (sync) | No |
+| httrack | Raw HTML | Yes | No | Slow (sync) | No |
+| Site-specific | Varies | No | Varies | Varies | No |
+
+## Troubleshooting
+
+### Site requires JavaScript
+
+```bash
+# Install Playwright support
+pip install docpull[js]
+python -m playwright install chromium
+
+# Use --js flag
+docpull https://site.com --js
+```
+
+### Too slow / rate limited
+
+```bash
+# Reduce concurrent requests
+docpull https://site.com --max-concurrent 5 --rate-limit 2.0
+```
+
+### Memory issues on large sites
+
+```bash
+# Limit pages fetched
+docpull https://site.com --max-pages 1000
+```
+
+## Contributing
+
+Contributions welcome! To add:
+- **New site profiles**: Create a profile in `docpull/profiles/`
+- **Better extraction**: Improve content detection in `fetchers/base.py`
+- **Performance improvements**: Optimize async fetching
+- **Bug reports**: Use the [issue tracker](https://github.com/raintree-technology/docpull/issues)
+
+### Development Setup
+
+```bash
+# Clone and install
+git clone https://github.com/raintree-technology/docpull
+cd docpull
+pip install -e ".[dev]"
+
+# Run all quality checks (as per CI)
+black --check .           # Code formatting
+ruff check .              # Linting
+mypy docpull              # Type checking
+bandit -r docpull         # Security scanning
+pip-audit                 # Dependency vulnerabilities
+pytest --cov=docpull -v   # Tests with coverage
+```
+
+All PRs must pass these checks before merging.
+
+## Documentation
+
+- [Changelog](CHANGELOG.md)
+- [Security Policy](SECURITY.md)
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details
+
+## Links
+
+- [PyPI](https://pypi.org/project/docpull/)
+- [GitHub](https://github.com/raintree-technology/docpull)
+- [Issues](https://github.com/raintree-technology/docpull/issues)

docpull-1.0.1.dist-info/RECORD

@@ -0,0 +1,38 @@
+docpull/__init__.py,sha256=GxP9o5rz9t6wxmo6Gu7t4mRrAsUyI-1qHWVTjGROin0,835
+docpull/__main__.py,sha256=4JMK66Wj4uLZTKbF-sT3LAxOsr6buig77PmOkJCRRxw,83
+docpull/cli.py,sha256=WmgN7Vwv7ggmiVzHjFYff-w1ZV09lnkKPSw9dEkZdNQ,12157
+docpull/config.py,sha256=9CzuNFYbDUX1o2SG25_dxcC6Z0JFLPC0KWJxeqE3O6k,6264
+docpull/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+docpull/fetchers/__init__.py,sha256=GU2_3i9Qles7fEKpCd29N471HTvuLjVGReylOY5oHuE,572
+docpull/fetchers/async_fetcher.py,sha256=YxenNx88bP1SwhgfPbm1iNPxZxX-RPYnyOHD9D7iXuc,11213
+docpull/fetchers/base.py,sha256=1cSIvNwVuhqZ_zwCYAXFZ01HEvU8K2gX3iIliR2ELE8,15691
+docpull/fetchers/bun.py,sha256=Wp3HGwnbDoI1OCIEUXuZyavrt2J0-rmUryjIcYD9Bx0,1782
+docpull/fetchers/d3.py,sha256=CGVEL6QyXz22cNhCcLU05dcR-_uvgLZCfJE49ivm8eM,7148
+docpull/fetchers/generic.py,sha256=2MglTVvrdaYJBpHMovFVcoo4mtEhaqBxr-f2Oz2ThPA,9373
+docpull/fetchers/generic_async.py,sha256=xX84RhsnKUon3MUJGKFYc8RNuq-hY-o4sL7Jskx4zZo,11125
+docpull/fetchers/nextjs.py,sha256=3ozWLbAFYUtVwvPSbRcPuar5kQVA9sHgo4Qfn4BqhaI,1729
+docpull/fetchers/parallel_base.py,sha256=Eb1qZvHKub_hL5cIcAbFoGmR17EEZ2mQeo2P6mFgnjE,3090
+docpull/fetchers/plaid.py,sha256=ARoo60g9h0xV_LfSoa2bOSve6dIgRB3y43dSxSmCcfQ,3223
+docpull/fetchers/react.py,sha256=UwUkFzz8oNT9tW3xLLZ1B8dVhHLiJUQoQhBkx26wcNE,1804
+docpull/fetchers/stripe.py,sha256=ezmmL41E4x6GgNYaUy_GSPCm5-jvsyWS4aZfigZxbkA,2080
+docpull/fetchers/tailwind.py,sha256=76Bj-gCFDHw_BNIlx16CsKD1PWBuH3eusa-fNwHnfEI,1882
+docpull/fetchers/turborepo.py,sha256=4hRntjjne5ZrAf2eZavFW7aZ8jGvJkZGsuShSosM5KM,1816
+docpull/profiles/__init__.py,sha256=TCE13-hgoSkqOP8lf1uoafZ_R9E6osgGTnJ29huQdsU,1563
+docpull/profiles/base.py,sha256=6LTgxnddR1vsygPLvTM_Gyqua6mU0gM210vgg0i5b4k,1933
+docpull/profiles/bun.py,sha256=RFvMgmWNrxIhi3Xlhk5wzLPzhBum7_IpHhoUZRE7cVI,315
+docpull/profiles/d3.py,sha256=6UBcRtZ5Y_9XCd1LPKSgxrvqvgzPEN_KMTHpGbglZt0,478
+docpull/profiles/nextjs.py,sha256=y6uPhxD4K16FHCfWoKmANliTnE2XRIMWRNkBS_l_h4s,406
+docpull/profiles/plaid.py,sha256=FfE4cG7Iph_-n6VR5vckD-il55vJZZpOATboN89_9L0,514
+docpull/profiles/react.py,sha256=nQ9YwMa3fiOAFVwOiGWOLwL_L_dMSc7_QMoE7LoJB4E,412
+docpull/profiles/stripe.py,sha256=0ZoAIKltAvPE0oIr6vLNUugmqgqFaHIFaDBnwbEAbMA,413
+docpull/profiles/tailwind.py,sha256=tjdQE_wmVis0tkORYFE1Y_8QNAACRT8_xcAkTLmnvUk,366
+docpull/profiles/turborepo.py,sha256=GS-DKYMC9JqFWy2R_mq5T3wkpdCd5b_48Pij9vtQX4Q,381
+docpull/utils/__init__.py,sha256=KmZIu_MLGAYn9RaZ_p-FRW7JAEktqqHu3vAJnp6x3S4,238
+docpull/utils/file_utils.py,sha256=ofwqmWh5bOdURmZ4D5Jntab-eMJmzbhPpaGYdffnQPs,2657
+docpull/utils/logging_config.py,sha256=0l4U4KsnLYJ8qD7fMDbTOe6_bij-q-gUAMDgymS0cnM,1731
+docpull-1.0.1.dist-info/licenses/LICENSE,sha256=PaRVpLb_4aICC6aK-Mmx_-UWm9hVPmG3GCc8TlVFePI,1076
+docpull-1.0.1.dist-info/METADATA,sha256=eiC0K4JKO9w04E7US_RAK3C-be7kJgE5X7xsouq4UIA,12834
+docpull-1.0.1.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+docpull-1.0.1.dist-info/entry_points.txt,sha256=2XpUlOneuBhB00fqOCfzoywf56nYM1FRyKW6HVCsIC4,45
+docpull-1.0.1.dist-info/top_level.txt,sha256=Ya0YBKQEKxPLBYU5wNsPtsdgdV0WGM_0J3Y_75MI9e4,8
+docpull-1.0.1.dist-info/RECORD,,

docpull-1.0.1.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (80.9.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

docpull-1.0.1.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+docpull = docpull.cli:main

docpull-1.0.1.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Raintree Technology
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

docpull-1.0.1.dist-info/top_level.txt

@@ -0,0 +1 @@
+docpull