Scweet 5.2__tar.gz → 5.3__tar.gz

{scweet-5.2 → scweet-5.3}/CHANGELOG.md

@@ -2,6 +2,21 @@
 
 All notable changes to this project are documented in this file.
 
+## [5.3.0] - 2026-04-14
+
+### Added
+
+- **`manifest_scrape_on_init=` constructor shorthand** — pass `manifest_scrape_on_init=True` directly to `Scweet()` without needing `ScweetConfig`. Overrides the config value (same pattern as `proxy=`).
+- **`--manifest-scrape-on-init` CLI flag** — scrape fresh GraphQL query IDs from X's `main.js` bundle on startup from the command line.
+- **`save_name=` parameter** on `search`, `get_profile_tweets`, `get_followers`, `get_following`, and `get_user_info` — set a custom base filename for saved output (without extension).
+
+### Changed
+
+- Documentation now marks `ct0` as optional in cookies examples — Scweet bootstraps it automatically from `auth_token`.
+- Manifest scrape docs updated to reflect the new top-level constructor arg and CLI flag.
+
+---
+
 ## [5.2.0] - 2026-04-03
 
 ### Added

{scweet-5.2 → scweet-5.3}/DOCUMENTATION.md

@@ -30,11 +30,13 @@ Create a `cookies.json` file:
 [
   {
     "username": "your_account",
-    "cookies": { "auth_token": "...", "ct0": "..." }
+    "cookies": { "auth_token": "..." }
   }
 ]
 ```
 
+`ct0` is optional — Scweet bootstraps it automatically from `auth_token`. You can include it if you have it:
+
 ```python
 s = Scweet(cookies_file="cookies.json")
 ```
@@ -43,8 +45,8 @@ For multiple accounts with per-account proxies (enables concurrent scraping and
 
 ```json
 [
-  { "username": "account1", "cookies": { "auth_token": "...", "ct0": "..." }, "proxy": "http://user1:pass1@host1:port1" },
-  { "username": "account2", "cookies": { "auth_token": "...", "ct0": "..." }, "proxy": "http://user2:pass2@host2:port2" }
+  { "username": "account1", "cookies": { "auth_token": "..." }, "proxy": "http://user1:pass1@host1:port1" },
+  { "username": "account2", "cookies": { "auth_token": "..." }, "proxy": "http://user2:pass2@host2:port2" }
 ]
 ```
 
@@ -211,6 +213,7 @@ tweets = s.search(
 | `resume` | `bool` | `False` | Resume from last checkpoint |
 | `save` | `bool` | `False` | Save results to disk |
 | `save_format` | `str` | config value | `"csv"`, `"json"`, or `"both"` |
+| `save_name` | `str` | auto-generated | Base filename for saved output (without extension) |
 
 ### Async variant
 
@@ -245,6 +248,7 @@ tweets = s.get_profile_tweets(
 | `resume` | `bool` | `False` | Resume from last checkpoint |
 | `save` | `bool` | `False` | Save results to disk |
 | `save_format` | `str` | config value | `"csv"`, `"json"`, or `"both"` |
+| `save_name` | `str` | auto-generated | Base filename for saved output (without extension) |
 
 Async: `await s.aget_profile_tweets(["elonmusk"], limit=100)`
 
@@ -269,6 +273,7 @@ users = s.get_following(["OpenAI"], limit=500)
 | `raw_json` | `bool` | `False` | Include full Twitter user payload under `raw` key |
 | `save` | `bool` | `False` | Save results to disk |
 | `save_format` | `str` | config value | `"csv"`, `"json"`, or `"both"` |
+| `save_name` | `str` | auto-generated | Base filename for saved output (without extension) |
 
 ### raw_json option
 
@@ -297,6 +302,7 @@ profiles = s.get_user_info(["elonmusk", "OpenAI"])
 | `users` | `list[str]` | *required* | Usernames or profile URLs |
 | `save` | `bool` | `False` | Save results to disk |
 | `save_format` | `str` | config value | `"csv"`, `"json"`, or `"both"` |
+| `save_name` | `str` | auto-generated | Base filename for saved output (without extension) |
 
 Async: `await s.aget_user_info(["elonmusk"])`
 
@@ -485,14 +491,17 @@ s = Scweet(
 Twitter/X rotates GraphQL query IDs periodically. When IDs go stale, requests return 404. Scweet ships with default IDs that work at release time, but you can auto-fetch fresh ones:
 
 ```python
-s = Scweet(
-    cookies_file="cookies.json",
-    config=ScweetConfig(manifest_scrape_on_init=True),
-)
+s = Scweet(cookies_file="cookies.json", manifest_scrape_on_init=True)
 ```
 
 This fetches the current `main.js` bundle from X on init and extracts the latest query IDs. It adds a few seconds to startup but ensures your requests use current IDs.
 
+From the CLI:
+
+```bash
+scweet --auth-token TOKEN --manifest-scrape-on-init search "query" --limit 100
+```
+
 ---
 
 ## Account Management (ScweetDB)
@@ -629,7 +638,7 @@ The sync methods (`search`, `get_followers`, etc.) wrap their async counterparts
 
 ## Error Handling
 
-All Scweet methods raise exceptions on failure — there is no silent empty-result mode. Wrap calls in try/except to handle errors explicitly:
+All Scweet methods raise exceptions on failure. Wrap calls in try/except to handle errors explicitly:
 
 ```python
 from Scweet import Scweet, AccountPoolExhausted, RateLimitError, AuthError, NetworkError, RunFailed
@@ -699,7 +708,7 @@ from Scweet import (
 
 **`RunFailed` / `NetworkError`**
 - Check your internet connection and proxy configuration
-- X may have rotated GraphQL query IDs — enable `manifest_scrape_on_init=True` to auto-fetch fresh ones
+- X may have rotated GraphQL query IDs — pass `manifest_scrape_on_init=True` to `Scweet()` (or `--manifest-scrape-on-init` in the CLI) to auto-fetch fresh ones
 - 404 errors in logs mean stale query IDs (transient) — not bad auth
 
 ---
@@ -731,6 +740,7 @@ scweet [auth options] [config options] <subcommand> [subcommand options]
 |------|-------------|
 | `--proxy PROXY` | Proxy URL or JSON string |
 | `--concurrency N` | Worker concurrency (default: `5`) |
+| `--manifest-scrape-on-init` | Scrape fresh query IDs from X's main.js on startup |
 
 ### Output options
 

scweet-5.3/PKG-INFO

@@ -0,0 +1,314 @@
+Metadata-Version: 2.4
+Name: Scweet
+Version: 5.3
+Summary: Scrape tweets, profiles, followers and more from Twitter/X — no API key needed
+Home-page: https://github.com/Altimis/Scweet
+Download-URL: https://github.com/Altimis/Scweet/archive/v5.3.tar.gz
+Author: Yassine AIT JEDDI
+Author-email: aitjeddiyassine@gmail.com
+License: MIT
+Keywords: twitter,x,scraper,twitter-scraper,x-scraper,tweets,tweet-search,twitter-search,followers,following,twitter-api-alternative,graphql,web-scraping,social-media,data-collection,python
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: python-dotenv
+Requires-Dist: sqlmodel>=0.0.22
+Requires-Dist: SQLAlchemy>=2.0
+Requires-Dist: curl_cffi
+Requires-Dist: XClientTransaction>=1.0.2
+Requires-Dist: pydantic>=2.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: download-url
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+<h1 align="center">Scweet</h1>
+
+<p align="center">
+  <strong>Scrape Twitter / X without the official API.</strong>
+  <br>
+  Hosted on Apify, or run locally with Python and CLI.
+</p>
+
+<p align="center">
+  <a href="https://apify.com/altimis/scweet?fpr=a40q9">
+    <img alt="Run on Apify" src="https://img.shields.io/badge/Run%20on-Apify-246DFF?logo=apify&logoColor=white">
+  </a>
+  <a href="#python-quickstart">
+    <img alt="Python Quickstart" src="https://img.shields.io/badge/Python-Quickstart-3776AB?logo=python&logoColor=white">
+  </a>
+  <a href="#cli">
+    <img alt="CLI Quickstart" src="https://img.shields.io/badge/CLI-Quickstart-111111?logo=gnu-bash&logoColor=white">
+  </a>
+  <a href="#documentation">
+    <img alt="Documentation" src="https://img.shields.io/badge/Docs-Full%20Reference-0A66C2">
+  </a>
+</p>
+
+<p align="center">
+  <a href="https://github.com/Altimis/Scweet/actions/workflows/tests.yml">
+    <img alt="Tests" src="https://github.com/Altimis/Scweet/actions/workflows/tests.yml/badge.svg">
+  </a>
+  <a href="https://pypi.org/project/scweet/">
+    <img alt="PyPI Version" src="https://img.shields.io/pypi/v/scweet.svg">
+  </a>
+  <a href="https://pepy.tech/projects/scweet">
+    <img alt="PyPI Downloads" src="https://static.pepy.tech/badge/scweet/month">
+  </a>
+  <a href="https://github.com/Altimis/scweet/blob/main/LICENSE">
+    <img alt="License" src="https://img.shields.io/github/license/Altimis/scweet">
+  </a>
+  <a href="https://apify.com/altimis/scweet">
+    <img alt="Scweet Actor Status" src="https://apify.com/actor-badge?actor=altimis/scweet">
+  </a>
+</p>
+
+
+---
+<p align="center">
+<strong>Scrape tweets, profiles, followers and more from Twitter/X. Uses X's own web GraphQL API, authenticated with your browser cookies. No official API key or developer account needed.<strong>
+</p>
+
+---
+
+**What you can scrape:**
+- [**Tweets**](DOCUMENTATION.md#search-api) — by keyword, hashtag, user, date range, engagement filters, language, location
+- [**Profile timelines**](DOCUMENTATION.md#profile-tweets) — a user's full tweet history
+- [**Followers / Following**](DOCUMENTATION.md#followers--following) — full account lists at scale
+- [**User profiles**](DOCUMENTATION.md#user-info) — bio, follower count, verification status, and more
+
+---
+
+## Hosted — no setup needed
+
+> [Run Scweet on Apify](https://apify.com/altimis/scweet?fpr=a40q9&fp_sid=jeb97) with no code, no cookies, and no account management. Free tier included.
+
+> Running Scweet locally at any real volume usually means dedicated accounts and proxies. You can use a single proxy, a rotating proxy, or assign one per account. If you need a provider, [Webshare](https://www.webshare.io/?referral_code=kdgjcc09945q) works well. Disclosure: affiliate link.
+
+---
+
+## Quickstart
+
+**1. Install**
+
+```bash
+pip install -U Scweet
+```
+
+**2. Get your `auth_token`**
+
+Log into [x.com](https://x.com) → DevTools `F12` → **Application** → **Cookies** → `https://x.com` → copy the `auth_token` value.
+
+Scweet auto-bootstraps the `ct0` CSRF token from `auth_token` alone — or use the [`cookies.json` format](#multiple-accounts--proxies) for multiple accounts at once.
+
+**3. Scrape**
+
+```python
+from Scweet import Scweet
+
+# Credentials are stored in scweet_state.db automatically on first run
+s = Scweet(auth_token="YOUR_AUTH_TOKEN", proxy="http://user:pass@host:port")
+
+# Search tweets — save to CSV (save_format="json" or "both" also works)
+tweets = s.search("bitcoin", since="2025-01-01", limit=500, save=True)
+
+# Reuse provisioned accounts on subsequent runs — no credentials needed
+s = Scweet(db_path="scweet_state.db")
+tweets = s.search("ethereum", limit=500, save=True)
+```
+
+> Always set `limit` — without it, scraping continues until your account's daily cap is hit.
+
+> All methods have async variants: `asearch()`, `aget_profile_tweets()`, `aget_followers()`, etc.
+
+---
+
+## Common tasks
+
+### Search tweets
+
+```python
+tweets = s.search(
+    "AI agents",
+    since="2026-01-01",
+    from_users=["elonmusk"],
+    min_likes=50,
+    limit=200,
+    save=True,
+)
+```
+
+### Profile timeline
+
+```python
+tweets = s.get_profile_tweets(["elonmusk"], limit=200)
+```
+
+### Followers / Following
+
+```python
+followers = s.get_followers(["elonmusk"], limit=1000)
+following = s.get_following(["elonmusk"], limit=1000)
+```
+
+### User profiles
+
+```python
+profiles = s.get_user_info(["githubstatus", "elonmusk"])
+```
+
+For the full list of supported search operators, see [twitter-advanced-search](https://github.com/igorbrigadir/twitter-advanced-search).
+
+---
+
+## CLI
+
+```bash
+# Search with proxy, save to CSV
+scweet --auth-token YOUR_AUTH_TOKEN --proxy http://user:pass@host:port \
+  search "bitcoin" --since 2025-01-01 --limit 500 --save
+
+# Followers, saved as JSON
+scweet --auth-token YOUR_AUTH_TOKEN \
+  followers elonmusk --limit 1000 --save --save-format json
+```
+
+For multi-account runs, use `--cookies-file cookies.json`.
+
+---
+
+## Multiple accounts & proxies
+
+For higher throughput and reduced ban risk, use multiple dedicated accounts with per-account proxies.
+
+**`cookies.json` format:**
+
+```json
+[
+  { "username": "acct1", "cookies": { "auth_token": "..." }, "proxy": "http://user1:pass1@host1:port1" },
+  { "username": "acct2", "cookies": { "auth_token": "..." }, "proxy": "http://user2:pass2@host2:port2" }
+]
+```
+
+```python
+s = Scweet(cookies_file="cookies.json")  # proxies are read from the file, one per account
+```
+
+> **Never use your personal Twitter/X account for scraping.** Use dedicated accounts only.
+
+---
+
+## Why Scweet?
+
+| | twint | snscrape | twscrape | **Scweet** |
+|---|---|---|---|---|
+| **Works in 2026** | ❌ unmaintained | ❌ broken | ✅ | ✅ |
+| Cookie / token auth | ❌ | ❌ | ✅ | ✅ |
+| Multi-account pooling | ❌ | ❌ | ✅ | ✅ |
+| Proxy support | ❌ | ❌ | ✅ | ✅ |
+| Resume interrupted scrapes | ❌ | ❌ | ❌ | ✅ |
+| Built-in CSV / JSON output | ✅ | ✅ | ❌ | ✅ |
+| Sync + async API | ❌ | ❌ | Async only | ✅ both |
+| Hosted, no-setup option | ❌ | ❌ | ❌ | ✅ Apify |
+| Active maintenance | ❌ | ❌ | ⚠️ | ✅ |
+
+[twint](https://github.com/twintproject/twint) has been unmaintained since 2023. [snscrape](https://github.com/JustAnotherArchivist/snscrape) broke after X's backend changes. [twscrape](https://github.com/vladkens/twscrape) is the closest active alternative — worth knowing, but async-only, no built-in file output, and no resume support.
+
+---
+
+## Limitations
+
+- Only publicly visible content is accessible — private/protected accounts are not supported
+- Relies on undocumented X web endpoints; breakage is always possible after platform changes. Scweet can self-heal by scraping fresh query IDs and feature flags from X's `main.js` bundle at startup — pass `manifest_scrape_on_init=True` to enable:
+  ```python
+  s = Scweet(auth_token="...", manifest_scrape_on_init=True)
+  ```
+- Cookies can expire and may need periodic refreshing
+- A single account typically handles hundreds to a few thousand tweets per day before hitting rate limits; multi-account pooling scales proportionally
+- Proxies and dedicated accounts are strongly recommended for anything beyond light testing
+- Username/password login is not supported. X's anti-automation defenses make programmatic login unreliable and likely to trigger account locks. Scweet authenticates with browser cookies (`auth_token`) extracted from an active session instead
+
+---
+
+## Documentation
+
+Full API reference, config options, structured search filters, async patterns, resume, proxies, and troubleshooting:
+
+- [Account setup](DOCUMENTATION.md#account-setup)
+- [Search API](DOCUMENTATION.md#search-api)
+- [Profile tweets](DOCUMENTATION.md#profile-tweets)
+- [Followers / Following](DOCUMENTATION.md#followers--following)
+- [User info](DOCUMENTATION.md#user-info)
+- [Saving results](DOCUMENTATION.md#saving-results)
+- [Resume interrupted searches](DOCUMENTATION.md#resume-interrupted-searches)
+- [Controlling limits](DOCUMENTATION.md#controlling-limits)
+- [Configuration reference](DOCUMENTATION.md#configuration-reference)
+- [Account management (ScweetDB)](DOCUMENTATION.md#account-management-scweetdb)
+- [Async usage](DOCUMENTATION.md#async-usage)
+- [Error handling](DOCUMENTATION.md#error-handling)
+- [CLI reference](DOCUMENTATION.md#cli)
+
+For advanced Twitter search operators, see [twitter-advanced-search](https://github.com/igorbrigadir/twitter-advanced-search).
+
+---
+
+<details>
+<summary><strong>FAQ</strong></summary>
+
+<br>
+
+**Does it work without an official Twitter API key?**
+Yes. Scweet calls X's internal GraphQL API — the same one the web app uses. No developer account or API key required.
+
+**Is it a replacement for twint or snscrape?**
+Yes. Both are broken as of 2024–2025. Scweet uses a different, currently-working approach: cookies + GraphQL instead of legacy unauthenticated endpoints.
+
+**How many tweets can I scrape?**
+A single account typically handles hundreds to a few thousand tweets per day before hitting rate limits. Multi-account pooling scales this proportionally. The hosted Apify actor manages accounts and rate limits automatically.
+
+**Will my account get banned?**
+Never use your personal account — use dedicated accounts only. To reduce risk further: use **multiple accounts** (distributes load) and pair each with a **proxy** (prevents all requests from one IP).
+
+**Does it work for private accounts?**
+No. Only publicly visible content is accessible.
+
+**Does it still work in 2025 / 2026?**
+Yes — last verified working in March 2026 against X's current GraphQL API.
+
+</details>
+
+---
+
+## Community
+
+Have a question or want to share what you built with Scweet?
+Open a thread in [**GitHub Discussions**](https://github.com/Altimis/Scweet/discussions).
+
+**Found it useful? [Star the repo](https://github.com/Altimis/Scweet/stargazers)**
+
+---
+
+## Contributing
+
+Bug reports, feature suggestions, and PRs are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md).
+
+---
+
+*MIT License*