reddrix 0.1.0__tar.gz

reddrix-0.1.0/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Omar Abdelhamid
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

reddrix-0.1.0/PKG-INFO

@@ -0,0 +1,341 @@
+Metadata-Version: 2.4
+Name: reddrix
+Version: 0.1.0
+Summary: Reddrix is an open-source, blazing fast Reddit Scraper for Python (No API Key)
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: camoufox[geoip]
+Requires-Dist: whiskeysour
+Requires-Dist: curl-cffi
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: summary
+
+<p align="center">
+    <img src="https://raw.githubusercontent.com/Coding-Doctor-Omar/reddrix/main/images/reddrix.png" width="350" alt="reddrix logo">
+</p>
+
+<p align="center">
+    <a href="https://ko-fi.com/CodingDoctorOmar">
+        <img src="https://ko-fi.com/img/githubbutton_sm.svg" alt="Support Me on Ko-fi">
+    </a>
+</p>
+
+# Reddrix &mdash; An open-source, blazing fast Reddit Scraper for Python (No API Key)
+
+<p align="center"><h3>A powerful, async-first Reddit scraper (with built-in proxy support) for posts and comments with detailed metrics. No API key required. Scrape subreddits, profiles, and specific posts, or search Reddit site-wide by keywords or domains. Collect rich, structured data including authors, scores, timestamps, flairs, comment threads, and more (30+ fields).</h3></p>
+
+> If this repo is useful to you or saved your time, consider starring ⭐ it to support me and help other find it and have an easier time scraping Reddit ❤️.
+
+> Need a hosted version, check [Reddit Scraper Pro](https://apify.com/coding-doctor-omar/Reddit-Scraper-Pro) on Apify.
+
+<p align="center">
+    <a href="#installation">Installation</a> • <a href="#quick-start">Quick Start</a> • <a href="#output-dataset">Output Dataset</a> • 
+    <a href="#features">Features</a> • <a href="#scraper-options">Scraper Options</a>
+</p>
+
+## Installation
+
+```bash
+pip install reddrix
+
+reddrix install # Optional for installing Camoufox if you want to use Reddrix with Camoufox
+```
+
+## Quick start
+
+**Using Async Context Manager:**
+
+```python
+from reddrix import Reddrix
+import asyncio
+
+async def main():
+    async with Reddrix(
+        mode="standard",
+        subreddit_urls=["https://www.reddit.com/r/webscraping"],
+        sort="new",
+        include_comments=True,
+        max_results=500,
+        max_posts_per_collection=500,
+        verbose=True
+    ) as reddrix:
+        async for item in reddrix.get_items():
+            print(item["submissionType"], item["submissionUrl"])
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+**Without the context manager:**
+
+```python
+from reddrix import Reddrix
+import asyncio
+
+async def main():
+    reddrix = Reddrix(
+        mode="standard",
+        subreddit_urls=["https://www.reddit.com/r/webscraping"],
+        sort="new",
+        include_comments=True,
+        max_results=500,
+        max_posts_per_collection=500,
+        proxies=["http://user:pass@hostname:port", "http://user2:pass2@hostname2:port2"], # Built-in automatic proxy rotation
+        verbose=True
+    )
+    await reddrix.start()
+    
+    async for item in reddrix.get_items():
+        print(item["submissionType"], item["submissionUrl"])
+    
+    await reddrix.stop()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+**Scraping Posts and Comments Based on Keywords:**
+
+```python
+from reddrix import Reddrix
+import asyncio
+
+async def main():
+    async with Reddrix(
+        mode="keyword_search",
+        sort="new", # The sort option is ignored for this mode
+        query="how to scrape reddit",
+        include_comments=True,
+        max_results=500,
+        max_posts_per_collection=500,
+        verbose=True
+    ) as reddrix:
+        async for item in reddrix.get_items():
+            print(item["submissionType"], item["submissionUrl"])
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+**Scraping Posts and Comments Based on Domains:**
+
+```python
+from reddrix import Reddrix
+import asyncio
+
+async def main():
+    async with Reddrix(
+        mode="domain_search",
+        sort="new",
+        domains=["github.com", "apify.com"],
+        include_comments=True,
+        max_results=500,
+        max_posts_per_collection=500,
+        verbose=True
+    ) as reddrix:
+        async for item in reddrix.get_items():
+            print(item["submissionType"], item["submissionUrl"])
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+
+
+**Using Reddrix as a standalone Reddit http client for your custom scraping setup:**
+
+```python
+from reddrix import Reddrix
+import asyncio
+
+async def main():
+    async with Reddrix() as reddrix:
+        json_data = await reddrix.get("https://www.reddit.com/r/webscraping.json")
+        html_content = await reddrix.get(
+            url="https://www.reddit.com/search?q=how+to+scrape+reddit",
+            expected_data_type="HTML/XML" # You must specify the data type if you want to fetch html or xml.
+        )
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+**Using Reddrix with Camoufox:**
+
+```bash
+pip install reddrix
+reddrix install # Installs the camoufox browser
+```
+
+```python
+from reddrix import Reddrix
+import asyncio
+
+async def main():
+    async with Reddrix(
+        mode="standard",
+        http_client="camoufox", # By default, the http_client is "curl_cffi", but you can choose Camoufox.
+        subreddit_urls=["https://www.reddit.com/r/webscraping"],
+        sort="new",
+        include_comments=True,
+        max_results=500,
+        max_posts_per_collection=500,
+        verbose=True
+    ) as reddrix:
+        async for item in reddrix.get_items():
+            print(item["submissionType"], item["submissionUrl"])
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+This option is included in anticipation of future aggressive anti-scraping measures from Reddit. Even when Camoufox is used as the http client, the scraper uses it intelligently. It still utilizes the same JSON endpoints that curl_cffi uses via `page.evaluate()` with JS fetch. This results in nearly the same scraping speed, but with more stealth. The only caveat is more resource usage and slower scraper initialization speed.
+
+## Output Dataset
+
+If you use the preset scraping setup from `reddrix.get_items()`, the output dataset is a Python dictionary.
+
+**Example Reddit Post Output:**
+
+```python
+{
+  "submissionId": "1w1gc8p",
+  "submissionIdPrefixed": "t3_1w1gc8p",
+  "submissionType": "post",
+  "submissionUrl": "https://www.reddit.com/r/Sat/comments/1w1gc8p/is_this_good_improvement_for_10_hours_of/",
+  "submittedAt": "2026-08-29T07:53:42+00:00",
+  "parentId": None,
+  "parentPostId": None,
+  "parentPostUrl": None,
+  "postType": "image",
+  "isCrosspost": False,
+  "crosspostParentId": None,
+  "crosspostParentUrl": None,
+  "title": "Is this good improvement for 10 hours of dedicated study + need help for rw section",
+  "bodyText": "math significantly improved but reading writing is too tough for me (international student)\n\nplease help, i have to give the sat in two months, will reading novels improve my rw score?",
+  "isEdited": False,
+  "author": "stcer",
+  "authorFlairText": None,
+  "authorFlairImageUrl": None,
+  "depth": None,
+  "postFlair": None,
+  "isNSFW": False,
+  "subreddit": "r/Sat",
+  "score": 32,
+  "upvoteRatio": 0.98,
+  "numComments": 29,
+  "numCrossposts": 0,
+  "externalUrl": "https://i.redd.it/cjhezz5bs9mh1.jpeg",
+  "thumbnailUrl": None,
+  "videoUrl": None,
+  "imageUrl": "https://i.redd.it/cjhezz5bs9mh1.jpeg",
+  "galleryImagesUrls": None
+}
+```
+
+**Example Comment Output:**
+
+```python
+{
+  "submissionId": "p6lwobb",
+  "submissionIdPrefixed": "t1_p6lwobb",
+  "submissionType": "comment",
+  "submissionUrl": "https://www.reddit.com/r/Sat/comments/1w1gc8p/is_this_good_improvement_for_10_hours_of/p6lwobb/",
+  "submittedAt": "2026-08-29T13:41:52+00:00",
+  "parentId": "t3_1w1gc8p",
+  "parentPostId": "t3_1w1gc8p",
+  "parentPostUrl": "https://www.reddit.com/r/Sat/comments/1w1gc8p",
+  "postType": None,
+  "isCrosspost": False,
+  "crosspostParentId": None,
+  "crosspostParentUrl": None,
+  "title": None,
+  "bodyText": "How did you do that in 10 hours? Can you please give me the sources that you used? I also need help in rw section and I’ll take the sat in 2 weeks",
+  "isEdited": False,
+  "author": "AnalysisStill4932",
+  "authorFlairText": None,
+  "authorFlairImageUrl": None,
+  "depth": 0,
+  "postFlair": None,
+  "isNSFW": False,
+  "subreddit": "r/Sat",
+  "score": 9,
+  "upvoteRatio": None,
+  "numComments": None,
+  "numCrossposts": None,
+  "externalUrl": None,
+  "thumbnailUrl": None,
+  "videoUrl": None,
+  "imageUrl": None,
+  "galleryImagesUrls": None
+}
+```
+**Output Fields Explanation:**
+
+1. `submissionId` &mdash; [`string`] The ID of the post or comment.
+2. `submissionIdPrefixed` &mdash; [`string`] The prefixed ID of the post or comment.
+3. `submissionType` &mdash; [`string`] Whether the submission is a `post` or `comment`.
+4. `submissionUrl` &mdash; [`string`] The URL for the submission.
+5. `submittedAt` &mdash; [`string`] The date and time the post or comment was submitted in UTC timezone and ISO format.
+6. `parentId` &mdash; [`string`, `None`] The ID of the direct parent post or comment.
+7. `parentPostId` &mdash; [`string`, `None`] The ID of the parent post. For posts, this field will always be `None` since posts have no parent.
+8. `parentPostUrl` &mdash; [`string`, `None`] The URL of the parent post. For posts, this field will always be `None` since posts have no parent.
+9. `postType` &mdash; [`string`, `None`] The post type. Can be any of `text`, `poll`, `link`, `image`, `gallery`, or `video`. For comments, this field will always be `None`.
+10. `isCrosspost` &mdash; [`boolean`] Whether the post is a crosspost.
+11. `crosspostParentId` &mdash; [`string`, `None`] The ID of the parent of the crosspost (for posts that are not crossposts, will be `None`).
+12. `crosspostParentUrl` &mdash; [`string`, `None`] The URL of the parent of the crosspost (for posts that are not crossposts, will be `None`).
+13. `title` &mdash; [`string`, `None`] The post title (will always be `None` for comments).
+14. `bodyText` &mdash; [`string`, `None`] The main text content of a post or comment.
+15. `isEdited` &mdash; [`boolean`] Whether the post or comment is edited.
+16. `author` &mdash; [`string`] The author of the post or comment.
+17. `authorFlairText` &mdash; [`string`, `None`] The text content of the author flair or a text version of it.
+18. `authorFlairImageUrl` &mdash; [`string`, `None`] The URL for the image in an author flair.
+19. `depth` &mdash; [`integer`, `None`] The comment depth (top-level comments have a depth of `0` and posts have a depth of `None`).
+20. `postFlair` &mdash; [`string`, `None`] The post flair (will be `None` for comments).
+21. `isNSFW` &mdash; [`boolean`] Whether the post or comment is NSFW.
+22. `subreddit` &mdash; [`string`, `None`] The subreddit in which the post or comment was submitted.
+23. `score` &mdash; [`integer`, `None`] The vote score of the post or comment.
+24. `upvoteRatio` &mdash; [`float`, `None`] The ratio of upvotes to total number of votes of a post (always `None` for comments).
+25. `numComments` &mdash; [`integer`, `None`] The number of comments in a post (always `None` for comments).
+26. `numCrossposts` &mdash; [`integer`, `None`] The number of crossposts for a post (always `None` for comments).
+27. `externalUrl` &mdash; [`string`, `None`] The external URL in a post. For comments, this field will be `None` because external URLs will be included in the `bodyText` anyway.
+28. `thumbnailUrl` &mdash; [`string`, `None`] The thumbnail image URL for `video`, `image`, and `gallery` post types (always `None` for any other submission).
+29. `videoUrl` &mdash; [`string`, `None`] The URL for the video in a post (always `None` for any submission other than `video` posts).
+30. `imageUrl` &mdash; [`string`, `None`] The URL for the image in a post (always `None` for any submission other than `image` posts).
+31. `galleryImagesUrls` &mdash; [`list`] A list of URLs for all images in a gallery (always empty for any submission other than `gallery` posts).
+
+## Features
+
+- 🚀 **Async-First:** High-performance asynchronous scraping with `asyncio` and an async iterator for streaming results.
+- 🔑 **No API Key Required:** Scrape Reddit without requiring an API key.
+- 🔍 **Multiple Search Modes:** Scrape subreddits, user profiles, and specific posts, or search Reddit by keywords and domains.
+- 💬 **Comprehensive Data Extraction:** Collect posts and comments with 30+ structured fields, including authors, scores, timestamps, flairs, media URLs, and comment thread metadata.
+- 🌐 **Flexible HTTP Clients:** Choose between `curl_cffi` for lightweight HTTP requests and `camoufox` for intelligent, browser-based fetching.
+- 🛡️ **Resilient Scraping:** Built-in retries, cookie renewal, and configurable proxies.
+- ⚙️ **Fine-Grained Control:** Configure sorting, time frames, result limits, comment depth, and comment counts.
+- 🧩 **Standalone HTTP Client:** Use Reddrix's asynchronous `get()` method in your own custom scraping workflows.
+- 🐍 **Pythonic API:** Manage scraper resources with an async context manager or explicit `start()` and `stop()` methods.
+
+## Scraper Options
+
+1. `mode` (string) &mdash; One of `standard`, `keyword_search`, and `domain_search`. Default is `standard`.
+2. `http_client` (string) &mdash; One of `curl_cffi` and `camoufox`. Default is `curl_cffi`.
+3. `max_results` (integer) &mdash; The maximum total number of items (posts and comments combined) to return across all sources. Default is `999999`.
+4. `max_posts_per_collection` (integer) &mdash; The maximum number of posts to scrape from each collection. A collection is a single subreddit, profile, keyword search, or domain. Default is `999999`.
+5. `sort` (string) &mdash; One of `default`, `new`, and `hot`. Applies to subreddits, profiles, and `domain_search`. This option is ignored in `keyword_search` mode, which always uses Reddit's relevance sorting. Default is `default`.
+6. `time_frame` (string) &mdash; One of `all_time`, `past_hour`, `today`, `past_week`, `past_month`, and `past_year`. Used in `standard` mode for subreddits and profiles, and ignored in `keyword_search` and `domain_search` modes. Default is `all_time`.
+7. `subreddit_urls` (list of strings) &mdash; Subreddit URLs to scrape, e.g. `["https://www.reddit.com/r/webscraping"]`. Used in `standard` mode only. URLs that are not valid subreddit URLs are skipped. Default is `[]`.
+8. `post_urls` (list of strings) &mdash; Specific post URLs to scrape, e.g. `["https://www.reddit.com/r/webscraping/comments/abc123/post_title/"]`. Used in `standard` mode only. URLs that are not valid post URLs are skipped. Default is `[]`.
+9. `profile_urls` (list of strings) &mdash; User profile URLs to scrape the submitted posts of, e.g. `["https://www.reddit.com/user/username"]`. Both `/user/` and `/u/` URLs are supported. Used in `standard` mode only. URLs that are not valid profile URLs are skipped. Default is `[]`.
+10. `domains` (list of strings) &mdash; Domains to search for in Reddit posts that link to them, e.g. `["github.com", "apify.com"]`. Used in `domain_search` mode only. Default is `[]`.
+11. `query` (string) &mdash; The search keywords to look up site-wide on Reddit. Used in `keyword_search` mode only. Default is `None`.
+12. `include_comments` (boolean) &mdash; Whether to scrape the comments of each post in addition to the post itself. Comments are returned as separate items with `submissionType` set to `comment`. Default is `False`.
+13. `max_comments_depth` (integer) &mdash; The maximum comment depth to scrape. Top-level comments have a depth of `0`, their replies have a depth of `1`, and so on. Only applies if `include_comments` is `True`. Default is `10`.
+14. `max_comments_per_post` (integer) &mdash; The maximum total number of comments (top-level comments and replies) to scrape per post. Only applies if `include_comments` is `True`. Default is `999999`.
+15. `max_top_comments_per_post` (integer) &mdash; The maximum number of top-level comments (depth `0`) to scrape per post. Only applies if `include_comments` is `True`. Default is `999999`.
+16. `proxies` (list of strings) &mdash; A list of proxy URLs in the format `scheme://user:pass@hostname:port`. Reddrix rotates through them automatically, picking an unused proxy first and reusing proxies only after the whole list has been used. Default is `[]` (no proxy).
+17. `verbose` (boolean) &mdash; Whether to print timestamped log messages (initialization, retries, etc.) to the console. Default is `False`.