reddipy 1.0.0__tar.gz

reddipy-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Muhammad Zaid
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

reddipy-1.0.0/PKG-INFO

@@ -0,0 +1,645 @@
+Metadata-Version: 2.4
+Name: reddipy
+Version: 1.0.0
+Summary: Scrape Reddit posts, comments, and user activity — no API key needed. Python SDK + CLI tool.
+Author: Muhammad Zaid
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/zaidkx37/reddipy
+Project-URL: Repository, https://github.com/zaidkx37/reddipy
+Project-URL: Issues, https://github.com/zaidkx37/reddipy/issues
+Keywords: reddit,scraper,reddit-scraper,reddit-api,posts,comments,subreddit,user-activity,data-extraction,web-scraping,cli,sdk,no-api-key,json,csv,export
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Environment :: Console
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.28.0
+Requires-Dist: rich>=13.0
+Provides-Extra: socks
+Requires-Dist: requests[socks]; extra == "socks"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Dynamic: license-file
+
+<h1 align="center">Reddipy</h1>
+
+<p align="center">
+  <strong>Scrape Reddit posts, comments, and user activity. Python SDK + CLI.</strong>
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/reddipy/"><img src="https://img.shields.io/pypi/v/reddipy.svg?style=flat-square&color=blue" alt="PyPI version"></a>
+  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10%20|%203.11%20|%203.12%20|%203.13-blue.svg?style=flat-square" alt="Python versions"></a>
+  <a href="https://github.com/zaidkx37/reddipy/actions"><img src="https://img.shields.io/github/actions/workflow/status/zaidkx37/reddipy/ci.yml?style=flat-square&label=CI" alt="CI"></a>
+  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/license-MIT-green.svg?style=flat-square" alt="License: MIT"></a>
+  <a href="https://pypi.org/project/reddipy/"><img src="https://img.shields.io/pypi/dm/reddipy?style=flat-square&color=orange&label=downloads" alt="Downloads"></a>
+</p>
+
+<p align="center">
+  Built on Reddit's public JSON endpoints. Two interfaces: <b>Python SDK</b> and <b>CLI</b>.
+</p>
+
+---
+
+## Features
+
+- **Search** Reddit by keyword with sort, time, and subreddit filters
+- **Scrape posts** with full comment trees, scores, and metadata
+- **Subreddit browsing** with hot, new, top, rising, and controversial sorting
+- **User profiles** with post/comment activity filtering
+- **Export** to JSON, CSV, or rich console tables
+- **Auto-pagination** to collect beyond the 100-per-request limit
+- **Proxy rotation** with single proxy, proxy file, and dead proxy cooldown
+- **Rate limiting** with configurable delay and exponential backoff
+- **Realistic headers** matching a real Chrome browser fingerprint
+- **Lightweight** with only `requests` and `rich` as dependencies
+
+---
+
+## Installation
+
+Recommended to **[install using pip](https://pypi.org/project/reddipy/)**
+
+```bash
+pip install reddipy
+```
+
+For SOCKS5 proxy support:
+
+```bash
+pip install "reddipy[socks]"
+```
+
+**Requirements:** Python 3.10+
+
+## Authentication
+
+Reddipy requires a `reddit_session` cookie from your browser. This is the only cookie needed.
+
+### Step 1: Get the Cookie
+
+1. Open [reddit.com](https://www.reddit.com) in your browser and log in
+2. Open DevTools (`F12`) > **Application** tab > **Cookies** > `https://www.reddit.com`
+3. Find the cookie named `reddit_session`
+4. Copy its **Value** (a long JWT string starting with `eyJ...`)
+
+### Step 2: Configure (One-Time)
+
+**SDK:**
+
+```python
+from reddipy import Reddit
+
+Reddit.configure(cookie="eyJhbGciOiJS...")
+```
+
+**CLI:**
+
+```bash
+reddipy configure "eyJhbGciOiJS..."
+```
+
+The cookie is saved to `~/.reddipy/config.json`. You only need to do this once. All future SDK and CLI calls will use it automatically.
+
+### Auth Management
+
+```python
+Reddit.configure(cookie="...")   # save cookie
+Reddit.is_configured()           # check if cookie exists (returns True/False)
+Reddit.logout()                  # remove stored cookie
+```
+
+```bash
+reddipy configure "cookie"      # save
+reddipy logout                  # remove
+```
+
+### Override Stored Cookie
+
+If you need to use a different cookie for a single session:
+
+```python
+reddit = Reddit(cookie="different_cookie_here")
+```
+
+```bash
+reddipy search "python" --cookie "different_cookie_here"
+```
+
+---
+
+## Python SDK
+
+### Initialize
+
+```python
+from reddipy import Reddit
+
+reddit = Reddit()
+```
+
+All methods below assume you have already configured authentication.
+
+### Search
+
+Search Reddit for posts matching a keyword.
+
+```python
+# Basic search
+posts = reddit.search("machine learning", limit=10)
+for post in posts:
+    print(f"[{post.score}] {post.title}")
+    print(f"  r/{post.subreddit} by {post.author}")
+```
+
+**Search within a specific subreddit:**
+
+```python
+posts = reddit.search("async", subreddit="python", limit=10)
+```
+
+**All parameters:**
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `query` | str | required | Search keywords |
+| `limit` | int | 25 | Max results (up to ~1000) |
+| `sort` | str | "relevance" | relevance, hot, new, top, comments |
+| `time_filter` | str | "all" | day, week, month, year, all |
+| `subreddit` | str | None | Scope search to a subreddit |
+
+**Returns:** `list[Post]`
+
+### Scrape a Post
+
+Extract a single post with its full comment tree.
+
+```python
+post, comments = reddit.post("https://www.reddit.com/r/python/comments/abc123/title/")
+
+# Post data
+print(post.title)
+print(post.author)
+print(post.score)
+print(post.selftext)       # post body
+print(post.flair)           # post flair (or None)
+print(post.num_comments)
+print(post.upvote_ratio)
+
+# Comments
+print(f"{len(comments)} top-level comments loaded")
+
+for comment in comments[:5]:
+    print(f"[{comment.score}] {comment.author}: {comment.body[:80]}")
+    print(f"  depth: {comment.depth}, is_submitter: {comment.is_submitter}")
+```
+
+Accepts any Reddit post URL format:
+
+```python
+# All of these work
+reddit.post("https://www.reddit.com/r/python/comments/abc123/title/")
+reddit.post("https://old.reddit.com/r/python/comments/abc123/")
+reddit.post("https://reddit.com/r/python/comments/abc123/title")
+reddit.post("/r/python/comments/abc123/title/")   # bare permalink
+```
+
+**Returns:** `tuple[Post, list[Comment]]`
+
+### Subreddit Posts
+
+Fetch posts from a subreddit with sort and time filters.
+
+```python
+# Hot posts (default)
+posts = reddit.subreddit("python")
+
+# Top posts this week
+posts = reddit.subreddit("python", sort="top", time_filter="week", limit=50)
+
+# New posts
+posts = reddit.subreddit("datascience", sort="new", limit=20)
+
+# Rising posts
+posts = reddit.subreddit("technology", sort="rising", limit=10)
+
+for post in posts:
+    print(f"[{post.score:>5}] {post.title}")
+    print(f"        {post.num_comments} comments | r/{post.subreddit}")
+```
+
+**All parameters:**
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `name` | str | required | Subreddit name (without `r/`) |
+| `sort` | str | "hot" | hot, new, top, rising, controversial |
+| `time_filter` | str | "all" | day, week, month, year, all (for top/controversial) |
+| `limit` | int | 25 | Max posts |
+
+**Returns:** `list[Post]`
+
+### User Activity
+
+Fetch a Reddit user's recent posts and comments.
+
+```python
+# All activity
+activity = reddit.user("spez", limit=10)
+
+for item in activity:
+    print(f"[{item.type}] r/{item.subreddit} ({item.score} pts)")
+    if item.type == "post":
+        print(f"  {item.title}")
+    else:
+        print(f"  {item.body[:80]}")
+```
+
+**Filter by activity type:**
+
+```python
+# Posts only
+posts = reddit.user("spez", limit=10, activity_type="posts")
+
+# Comments only
+comments = reddit.user("spez", limit=10, activity_type="comments")
+
+# Both (default)
+all_activity = reddit.user("spez", limit=10, activity_type="all")
+```
+
+**All parameters:**
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `username` | str | required | Reddit username (without `u/`) |
+| `limit` | int | 25 | Max items |
+| `sort` | str | "new" | new, hot, top |
+| `activity_type` | str | "all" | posts, comments, all |
+
+**Returns:** `list[UserActivity]`
+
+### Export Data
+
+Save scraped data to JSON, CSV, or display as a formatted table.
+
+**JSON:**
+
+```python
+from reddipy.exporters import JSONExporter
+
+posts = reddit.search("python", limit=20)
+JSONExporter().export(posts, "results.json")
+
+# Or print to stdout (no file path)
+JSONExporter().export(posts)
+```
+
+**CSV:**
+
+```python
+from reddipy.exporters import CSVExporter
+
+posts = reddit.subreddit("python", sort="top", limit=50)
+CSVExporter().export(posts, "posts.csv")
+```
+
+**Console table (rich formatted):**
+
+```python
+from reddipy.exporters import ConsoleExporter
+
+posts = reddit.search("python", limit=10)
+ConsoleExporter().export(posts)
+```
+
+**Exporting comments:**
+
+```python
+post, comments = reddit.post("https://reddit.com/r/python/comments/abc123/title/")
+
+# Comments to JSON (includes post and comment data)
+import json
+combined = {
+    "post": post.to_dict(),
+    "comments": [c.to_dict() for c in comments]
+}
+with open("post_data.json", "w") as f:
+    json.dump(combined, f, indent=2)
+
+# Comments to CSV (flattened, with depth/parent_id columns)
+CSVExporter().export(comments, "comments.csv")
+```
+
+### Proxy Support (SDK)
+
+Route requests through proxies to distribute load or bypass IP restrictions.
+
+**Single proxy:**
+
+```python
+reddit = Reddit(proxy="http://1.2.3.4:8080")
+```
+
+**Multiple proxies with random rotation:**
+
+```python
+reddit = Reddit(proxy_file="proxies.txt")
+```
+
+Each request picks a random proxy from the file. Failed proxies are automatically skipped for 60 seconds.
+
+**proxies.txt format:**
+
+```
+http://1.2.3.4:8080
+http://5.6.7.8:3128
+socks5://9.10.11.12:1080
+```
+
+SOCKS5 requires: `pip install "reddipy[socks]"`
+
+### Data Models
+
+Every scraped item is a Python dataclass with typed fields. All models have a `.to_dict()` method for serialization.
+
+**Post**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `id` | str | Reddit post ID |
+| `title` | str | Post title |
+| `author` | str | Author username |
+| `subreddit` | str | Subreddit name |
+| `score` | int | Net upvotes |
+| `upvote_ratio` | float | Upvote percentage (0.0 - 1.0) |
+| `num_comments` | int | Total comment count |
+| `created_utc` | datetime | Post creation time (UTC) |
+| `selftext` | str | Post body text (empty for link posts) |
+| `url` | str | Post URL or linked URL |
+| `permalink` | str | Reddit permalink |
+| `flair` | str or None | Post flair text |
+
+**Comment**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `id` | str | Comment ID |
+| `author` | str | Author username |
+| `body` | str | Comment text |
+| `score` | int | Net upvotes |
+| `created_utc` | datetime | Comment creation time (UTC) |
+| `parent_id` | str | Parent post/comment ID |
+| `depth` | int | Nesting depth (0 = top-level) |
+| `is_submitter` | bool | True if comment author is the post author |
+| `replies` | list[Comment] | Nested reply comments |
+
+**UserActivity**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `type` | str | "post" or "comment" |
+| `id` | str | Item ID |
+| `author` | str | Username |
+| `subreddit` | str | Subreddit name |
+| `score` | int | Net upvotes |
+| `created_utc` | datetime | Creation time (UTC) |
+| `title` | str or None | Post title (None for comments) |
+| `body` | str or None | Comment body (None for posts) |
+| `permalink` | str | Reddit permalink |
+
+**Serialization:**
+
+```python
+post_dict = post.to_dict()
+# {"id": "abc123", "title": "...", "created_utc": "2026-07-03T12:00:00+00:00", ...}
+
+# datetime fields become ISO 8601 strings in to_dict()
+# Nested comment replies are recursively serialized
+```
+
+### Error Handling
+
+Reddipy raises specific exceptions you can catch:
+
+```python
+from reddipy.exceptions import (
+    ReddipyError,           # base exception (catch-all)
+    SubredditNotFoundError,  # subreddit is private, banned, or doesn't exist
+    PostNotFoundError,       # post URL is invalid or deleted
+    UserNotFoundError,       # user is suspended or doesn't exist
+    RateLimitError,          # rate limited after 3 retries with backoff
+    NetworkError,            # connection failure or timeout
+    ProxyError,              # all proxies are dead
+)
+
+try:
+    posts = reddit.subreddit("nonexistent_sub_12345")
+except SubredditNotFoundError:
+    print("Subreddit doesn't exist")
+except RateLimitError:
+    print("Rate limited, try again later or reduce --delay")
+except NetworkError:
+    print("Connection failed")
+```
+
+### Advanced Configuration
+
+```python
+reddit = Reddit(
+    cookie="override_cookie",          # override stored cookie
+    delay=1.0,                         # seconds between requests (default: 2.0)
+    proxy="http://1.2.3.4:8080",       # single proxy
+    proxy_file="proxies.txt",          # proxy list with random rotation
+    user_agent="Custom Agent/1.0",     # custom User-Agent string
+    verbose=True,                      # print progress to stderr
+)
+```
+
+---
+
+## CLI Usage
+
+All CLI commands use stored authentication by default. Run `reddipy configure` first.
+
+### Search (CLI)
+
+```bash
+# Basic search
+reddipy search "python"
+
+# With filters
+reddipy search "machine learning" --limit 20 --sort top --time week
+
+# Search within a subreddit
+reddipy search "flask" --subreddit python --limit 10
+reddipy search "hooks" -s react --sort new
+```
+
+### Post (CLI)
+
+```bash
+# Scrape a post with all comments
+reddipy post "https://www.reddit.com/r/python/comments/abc123/title/"
+
+# Export to JSON
+reddipy post "https://reddit.com/r/python/comments/abc123/title/" --export json -o post.json
+```
+
+### Subreddit (CLI)
+
+```bash
+# Hot posts (default)
+reddipy subreddit python
+
+# Top posts this month
+reddipy subreddit datascience --sort top --time month --limit 50
+
+# New posts
+reddipy subreddit technology --sort new --limit 20
+
+# Rising
+reddipy subreddit programming --sort rising
+```
+
+### User (CLI)
+
+```bash
+# All activity
+reddipy user spez --limit 10
+
+# Posts only
+reddipy user spez --type posts --limit 20
+
+# Comments only, sorted by top
+reddipy user spez --type comments --sort top
+```
+
+### Export (CLI)
+
+```bash
+# JSON file
+reddipy search "python" --export json -o results.json
+
+# CSV file
+reddipy subreddit python --export csv -o posts.csv
+
+# Console table (default, no flag needed)
+reddipy search "python" --limit 5
+
+# Auto-generated filename (omit -o)
+reddipy search "python" --export json
+# Creates: reddipy_search_20260703_143000.json
+```
+
+### Proxy Support (CLI)
+
+```bash
+# Single proxy
+reddipy search "python" --proxy http://1.2.3.4:8080
+
+# Multiple proxies with random rotation
+reddipy search "python" --proxy-file proxies.txt
+```
+
+**proxies.txt** (one proxy per line):
+
+```
+http://1.2.3.4:8080
+http://5.6.7.8:3128
+socks5://9.10.11.12:1080
+```
+
+SOCKS5 requires: `pip install "reddipy[socks]"`
+
+### All CLI Flags
+
+**Common flags (available on all commands):**
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--limit` | Max number of results | 25 |
+| `--sort` | Sort order | varies by command |
+| `--time` / `-t` | Time filter (day/week/month/year/all) | all |
+| `--export` | Output format (json/csv/console) | console |
+| `--output` / `-o` | Output file path | auto-generated |
+| `--cookie` | Override stored reddit_session cookie | stored config |
+| `--cookie-file` | Path to JSON file with cookies | - |
+| `--proxy` | Single proxy URL | - |
+| `--proxy-file` | Path to proxy list file | - |
+| `--delay` | Seconds between requests | 2.0 |
+| `--user-agent` | Custom User-Agent header | Chrome 149 |
+| `--verbose` / `-v` | Show progress and retry info | false |
+
+**Command-specific flags:**
+
+| Command | Flag | Description | Default |
+|---------|------|-------------|---------|
+| `search` | `--subreddit` / `-s` | Scope search to a subreddit | all of Reddit |
+| `user` | `--type` | Filter activity (posts/comments/all) | all |
+
+**Exit codes:**
+
+| Code | Meaning |
+|------|---------|
+| 0 | Success |
+| 1 | General error (invalid args, unexpected failure) |
+| 2 | Network error (connection failed, rate limited, proxies dead) |
+| 3 | Not found (subreddit/post/user doesn't exist) |
+
+---
+
+## Rate Limiting
+
+Reddipy has built-in protections to avoid getting blocked:
+
+- **Request delay:** 2-second gap between requests (configurable with `--delay` or `delay=` parameter)
+- **Cooldown on 429:** Automatic exponential backoff (2s, 4s, 8s) when rate-limited by Reddit
+- **Max retries:** 3 attempts with increasing wait times before raising an error
+- **Dead proxy cooldown:** Failed proxies are skipped for 60 seconds before being retried
+- **Realistic headers:** All requests include 14 browser-standard headers (Accept, Sec-CH-UA, Sec-Fetch-*, etc.) matching a real Chrome browser fingerprint
+
+**Tips for staying safe:**
+
+- Keep `delay` at 2.0 or higher for normal use
+- Use `--verbose` to see when retries and cooldowns happen
+- For large scrapes (500+ items), use proxies to distribute load
+- Don't run multiple instances on the same account simultaneously
+
+## Project Structure
+
+```
+reddipy/
+    __init__.py       # Package exports: Reddit, Post, Comment, UserActivity
+    sdk.py            # Python SDK (Reddit class)
+    cli.py            # CLI entry point (argparse subcommands)
+    client.py         # HTTP client (headers, rate limiting, pagination, proxies)
+    config.py         # Persistent auth (~/.reddipy/config.json)
+    models.py         # Post, Comment, UserActivity dataclasses
+    exceptions.py     # ReddipyError, SubredditNotFoundError, etc.
+    utils.py          # URL parsing, timestamp conversion, filename generation
+    scrapers/
+        base.py       # BaseScraper abstract class
+        search.py     # SearchScraper
+        post.py       # PostScraper
+        subreddit.py  # SubredditScraper
+        user.py       # UserScraper
+    exporters/
+        base.py       # BaseExporter abstract class
+        json_exporter.py
+        csv_exporter.py
+        console.py    # Rich formatted tables
+```
+
+## License
+
+MIT