kabigon 0.14.2__py3-none-any.whl

kabigon-0.14.2.dist-info/METADATA

@@ -0,0 +1,319 @@
+Metadata-Version: 2.4
+Name: kabigon
+Version: 0.14.2
+Author-email: narumi <toucans-cutouts0f@icloud.com>
+License-File: LICENSE
+Requires-Python: >=3.12
+Requires-Dist: firecrawl-py>=2.4.1
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: loguru>=0.7.3
+Requires-Dist: markdownify>=0.14.1
+Requires-Dist: openai-whisper>=20250625
+Requires-Dist: playwright>=1.52.0
+Requires-Dist: pypdf>=5.3.0
+Requires-Dist: rich>=13.9.4
+Requires-Dist: typer>=0.15.3
+Requires-Dist: youtube-transcript-api>=1.2.2
+Requires-Dist: yt-dlp>=2025.4.30
+Description-Content-Type: text/markdown
+
+# kabigon
+
+[![PyPI version](https://badge.fury.io/py/kabigon.svg)](https://badge.fury.io/py/kabigon)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![codecov](https://codecov.io/gh/narumiruna/kabigon/branch/main/graph/badge.svg)](https://codecov.io/gh/narumiruna/kabigon)
+
+A URL content loader library that extracts content from various sources (YouTube, Instagram Reels, Twitter/X, Reddit, Truth Social, GitHub files, PDFs, web pages) and converts them to text/markdown format.
+
+## Features
+
+✨ **Multi-Platform Support**: YouTube, Twitter/X, Truth Social, Reddit, Instagram Reels, PTT, GitHub files, PDFs, and generic web pages
+
+🔄 **Async-First Design**: Built with async/await for efficient parallel processing
+
+🎯 **Smart Fallback**: Automatically tries multiple extraction strategies until one succeeds
+
+🚀 **Simple API**: Single-line usage with sensible defaults, or full control with custom loader chains
+
+🔌 **Extensible**: Easy to add new loaders for additional platforms
+
+## Table of Contents
+
+- [Installation](#installation)
+- [Usage](#usage)
+  - [CLI](#cli)
+  - [Python API - Sync](#python-api---sync)
+  - [Python API - Async](#python-api---async)
+- [Supported Sources](#supported-sources)
+- [Examples](#examples)
+- [Troubleshooting](#troubleshooting)
+- [Development](#development)
+- [License](#license)
+
+## Installation
+
+```shell
+pip install kabigon
+playwright install chromium
+```
+
+## Usage
+
+### CLI
+
+```shell
+kabigon <url>
+
+# Examples
+kabigon https://www.youtube.com/watch?v=dQw4w9WgXcQ
+kabigon https://x.com/elonmusk/status/123456789
+kabigon https://truthsocial.com/@realDonaldTrump/posts/123456
+kabigon https://reddit.com/r/python/comments/xyz/...
+kabigon https://github.com/anthropics/claude-code/blob/main/plugins/ralph-wiggum/README.md
+kabigon https://example.com/document.pdf
+```
+
+### Python API - Sync
+
+```python
+import kabigon
+
+url = "https://www.google.com.tw"
+
+# Simplest usage - automatically uses the best loader
+content = kabigon.load_url_sync(url)
+print(content)
+
+# Or use specific loader
+content = kabigon.PlaywrightLoader().load_sync(url)
+print(content)
+
+# With multiple loaders (tries each in order)
+loader = kabigon.Compose([
+    kabigon.TwitterLoader(),
+    kabigon.TruthSocialLoader(),
+    kabigon.YoutubeLoader(),
+    kabigon.RedditLoader(),
+    kabigon.PDFLoader(),
+    kabigon.PlaywrightLoader(),  # Fallback for generic URLs
+])
+content = loader.load_sync(url)
+print(content)
+```
+
+### Python API - Async
+
+```python
+import asyncio
+import kabigon
+
+async def main():
+    url = "https://www.google.com.tw"
+
+    # Simplest usage - automatically uses the best loader
+    content = await kabigon.load_url(url)
+    print(content)
+
+    # Or use specific loader
+    loader = kabigon.PlaywrightLoader()
+    content = await loader.load(url)
+    print(content)
+
+    # Batch processing multiple URLs in parallel
+    urls = [
+        "https://x.com/user1/status/123",
+        "https://truthsocial.com/@user/posts/456",
+        "https://youtube.com/watch?v=abc",
+        "https://reddit.com/r/python/comments/xyz",
+    ]
+
+    loader = kabigon.Compose([
+        kabigon.TwitterLoader(),
+        kabigon.TruthSocialLoader(),
+        kabigon.YoutubeLoader(),
+        kabigon.RedditLoader(),
+        kabigon.PlaywrightLoader(),
+    ])
+
+    # Parallel processing with automatic loader selection
+    results = await asyncio.gather(*[kabigon.load_url(url) for url in urls])
+    for url, content in zip(urls, results):
+        print(f"{url}: {len(content)} chars")
+
+asyncio.run(main())
+```
+
+### API Comparison
+
+| Usage | Simplest | Custom Loader Chain |
+|-------|----------|---------------------|
+| **Sync** | `kabigon.load_url_sync(url)` | `loader.load_sync(url)` |
+| **Async** | `await kabigon.load_url(url)` | `await loader.load(url)` |
+| **Batch Async** | `await asyncio.gather(*[kabigon.load_url(url) for url in urls])` | `await asyncio.gather(*[loader.load(url) for url in urls])` |
+
+## Supported Sources
+
+| Source | Loader | Description |
+|--------|--------|-------------|
+| YouTube | `YoutubeLoader` | Extracts video transcripts |
+| YouTube | `YoutubeYtdlpLoader` | Audio transcription via yt-dlp + Whisper |
+| Twitter/X | `TwitterLoader` | Extracts tweet content |
+| Truth Social | `TruthSocialLoader` | Extracts Truth Social posts |
+| Reddit | `RedditLoader` | Extracts Reddit posts and comments |
+| Instagram Reels | `ReelLoader` | Audio transcription + metadata |
+| GitHub | `GitHubLoader` | Fetches GitHub web pages and file content (supports repo URLs + `github.com/.../blob/...`) |
+| PDF | `PDFLoader` | Extracts text from PDF files (URL or local) |
+| PTT | `PttLoader` | Taiwan PTT forum posts |
+| Generic Web | `PlaywrightLoader` | Browser-based scraping for any website |
+| Generic Web | `HttpxLoader` | Simple HTTP requests with markdown conversion |
+
+## Examples
+
+See the [`examples/`](examples/) directory for more usage examples:
+
+- [`simple_usage.py`](examples/simple_usage.py) - Basic single-line usage
+- [`async_usage.py`](examples/async_usage.py) - Async usage and parallel batch processing
+- [`twitter.py`](examples/twitter.py) - Twitter/X post extraction
+- [`truthsocial.py`](examples/truthsocial.py) - Truth Social post extraction
+- [`read_reddit.py`](examples/read_reddit.py) - Reddit post and comments extraction
+- [`ptt.py`](examples/ptt.py) - PTT forum post extraction
+- [`fetch_billgertz_tweet.py`](examples/fetch_billgertz_tweet.py) - Real-world Twitter scraping example
+
+## Troubleshooting
+
+### Playwright browser not installed
+
+**Error**: `Executable doesn't exist at /path/to/chromium`
+
+**Solution**: Install Playwright browsers after installing kabigon:
+```bash
+playwright install chromium
+```
+
+### FFmpeg not found (for audio transcription)
+
+**Error**: `ffmpeg not found`
+
+**Solution**: Install FFmpeg for your platform:
+```bash
+# Ubuntu/Debian
+sudo apt-get install ffmpeg
+
+# macOS
+brew install ffmpeg
+
+# Windows
+# Download from https://ffmpeg.org/download.html
+```
+
+Or set custom FFmpeg path:
+```bash
+export FFMPEG_PATH=/path/to/ffmpeg
+```
+
+### Timeout errors
+
+**Error**: `Timeout 30000ms exceeded`
+
+**Solution**: Increase timeout for slow-loading pages:
+```python
+# Increase timeout to 60 seconds
+loader = kabigon.PlaywrightLoader(timeout=60_000)
+content = loader.load_sync(url)
+```
+
+### CAPTCHA or rate limiting
+
+Some websites may show CAPTCHAs or block automated access. For Reddit, kabigon automatically uses `old.reddit.com` to avoid CAPTCHAs. For other sites, you may need to:
+
+- Add delays between requests
+- Use a custom user agent
+- Implement retry logic with exponential backoff
+
+### Logging and debugging
+
+Enable debug logging to see what's happening:
+```bash
+export LOGURU_LEVEL=DEBUG
+kabigon <url>
+```
+
+Or in Python:
+```python
+import os
+os.environ["LOGURU_LEVEL"] = "DEBUG"
+import kabigon
+```
+
+## Development
+
+### Setup
+
+```bash
+# Clone the repository
+git clone https://github.com/narumiruna/kabigon.git
+cd kabigon
+
+# Install dependencies with uv
+uv sync
+
+# Install Playwright browsers
+playwright install chromium
+```
+
+### Testing
+
+```bash
+# Run all tests with coverage
+uv run pytest -v -s --cov=src tests
+
+# Run specific test file
+uv run pytest -v -s tests/loaders/test_youtube.py
+```
+
+Current test coverage: **69%** (37 tests passing)
+
+### Linting and Type Checking
+
+```bash
+# Run linter
+uv run ruff check .
+
+# Run type checker
+uv run ty check .
+
+# Auto-fix linting issues
+uv run ruff check --fix .
+
+# Format code
+uv run ruff format .
+```
+
+### Building and Publishing
+
+```bash
+# Build wheel
+uv build -f wheel
+
+# Publish to PyPI
+uv publish
+```
+
+### Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
+
+When adding a new loader:
+1. Create a new file in `src/kabigon/loaders/`
+2. Inherit from the `Loader` base class
+3. Implement `async def load(url: str) -> str`
+4. Add domain validation
+5. Add tests in `tests/loaders/`
+6. Update documentation
+
+See [`CLAUDE.md`](CLAUDE.md) for detailed development guidelines.
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.

kabigon-0.14.2.dist-info/RECORD

@@ -0,0 +1,28 @@
+kabigon/__init__.py,sha256=dkHmFZImqh0b4YnoDrD3bhjknyO5r_0LTg7CIcmq0zI,1070
+kabigon/api.py,sha256=9xLM_A9L0V5Ipxpb921lusPO4NcNrBMJxdYNMLPPi5A,2029
+kabigon/cli.py,sha256=-fl8KhW6k2bLREVponkvZ82a3Q_D0St8oE2dn032EZ8,188
+kabigon/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+kabigon/core/__init__.py,sha256=ZJNvmTRPUAqFxWn4QuVizsbbP9pFNGupl8rt67wNFrg,536
+kabigon/core/exception.py,sha256=OZFAyU0CMX9eA2rKBrKM1lTRWR3KFOm6m1_FgWl79pw,1267
+kabigon/core/loader.py,sha256=DXKRzmd8Dl4u-UgPHVcn4V37nPfeMehpL4O7cBzOibU,220
+kabigon/loaders/__init__.py,sha256=ZmcXKE9RxDnX_q5eg4UnzlBG_i1KjUFYJNsIvn49B8Q,779
+kabigon/loaders/compose.py,sha256=37F5C_HS1YeMUmtTFc4VzQCbFszXdq8OBymXzqmP5PU,971
+kabigon/loaders/firecrawl.py,sha256=SHA_X1qKkb9R2nxe32NqD5dS3-3lHHCb266N0-CgbZM,804
+kabigon/loaders/github.py,sha256=quoSHikGRaIpbKI_9z7PtSbh96-eG3ry1iJyE96J1sY,5217
+kabigon/loaders/httpx.py,sha256=RoR2YIZErSq9JMeM8L6Xk7i0HfURT-2YCkSvE4uQl4o,499
+kabigon/loaders/pdf.py,sha256=jxzo8DTB6YBi3k4U3Ub7ngkd7__5PUZB0kut6j9c1Ak,1484
+kabigon/loaders/playwright.py,sha256=3n8gfOoIxHs8tn6vBtPMCzDY7u1J7d7qLXIlRWE28G0,1220
+kabigon/loaders/ptt.py,sha256=BLbsaJfnnGRVKr8PJpybTWn49jXXvH2pzQFt-AKO5wA,824
+kabigon/loaders/reddit.py,sha256=R5hBDftJ7bc4xtK8EBbXrxkul9kX3olm1k7ALKxVmtA,2244
+kabigon/loaders/reel.py,sha256=8U-MDag31lTnF9siptLS2hv1uAh_P4EF92o4XsTfmmY,692
+kabigon/loaders/truthsocial.py,sha256=HnIlci_p7lWwnkDt4xaRHdt-x0aeLXjahwxzEB7toNM,1929
+kabigon/loaders/twitter.py,sha256=deQRQphE6WWVe5YE37YhDZsRyDnGgeZmHRB5TgD1DY4,3955
+kabigon/loaders/utils.py,sha256=eNTLtHLSB2erDac2HH3jWemgfr8Ou_ozwVb8h9BD-4g,922
+kabigon/loaders/youtube.py,sha256=cmzhw8aCsmh5LYrtmWpKtCl06OU6KUS3pHa6tpVHgFg,4611
+kabigon/loaders/youtube_ytdlp.py,sha256=EsTkWWwhxFPmv7wtzZsVcWQoC89-CZ7qLHo9cpG3sdQ,342
+kabigon/loaders/ytdlp.py,sha256=De4myywRYPmUEz5ZVUzPCLb2T9vFLPvS9NWZqsHkLog,1696
+kabigon-0.14.2.dist-info/METADATA,sha256=qerhecLBMs4Jyji4SuRXLFIA0VkttTs_H_ypHL2hgRc,8801
+kabigon-0.14.2.dist-info/WHEEL,sha256=WLgqFyCfm_KASv4WHyYy0P3pM_m7J5L9k2skdKLirC8,87
+kabigon-0.14.2.dist-info/entry_points.txt,sha256=O3FYAO9w-NQvlGMJrBvtrnGHSK2QkUnQBTa30YXRbVE,45
+kabigon-0.14.2.dist-info/licenses/LICENSE,sha256=H2T3_RTgmcngMeC7p_SXT3GwBLkd2DaNgAZuxulcfiA,1066
+kabigon-0.14.2.dist-info/RECORD,,

kabigon-0.14.2.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.28.0
+Root-Is-Purelib: true
+Tag: py3-none-any

kabigon-0.14.2.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+kabigon = kabigon.cli:main

kabigon-0.14.2.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 なるみ
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.