xthread-agent 3.2.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,6 @@
1
+ import sys
2
+
3
+ from xthread_agent import main
4
+
5
+ if __name__ == "__main__":
6
+ sys.exit(main())
@@ -0,0 +1,571 @@
1
+ Metadata-Version: 2.4
2
+ Name: xthread-agent
3
+ Version: 3.2.0
4
+ Summary: Agentic X/Twitter thread content & media harvester for AI agents — no login, no API keys, no browser. MCP wrapper included.
5
+ Author: Bilal140202
6
+ License: MIT License
7
+
8
+ Copyright (c) 2026
9
+
10
+ Permission is hereby granted, free of charge, to any person obtaining a copy
11
+ of this software and associated documentation files (the "Software"), to deal
12
+ in the Software without restriction, including without limitation the rights
13
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
14
+ copies of the Software, and to permit persons to whom the Software is
15
+ furnished to do so, subject to the following conditions:
16
+
17
+ The above copyright notice and this permission notice shall be included in all
18
+ copies or substantial portions of the Software.
19
+
20
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
21
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
22
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
23
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
24
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
25
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
26
+ SOFTWARE.
27
+
28
+ Project-URL: Homepage, https://bilal140202.github.io/xthread-agent/
29
+ Project-URL: Documentation, https://github.com/Bilal140202/xthread-agent#readme
30
+ Project-URL: Repository, https://github.com/Bilal140202/xthread-agent
31
+ Project-URL: Issues, https://github.com/Bilal140202/xthread-agent/issues
32
+ Project-URL: Changelog, https://github.com/Bilal140202/xthread-agent/blob/main/RELEASE_NOTES.md
33
+ Keywords: twitter,x,thread,threads,agent,ai-agents,mcp,model-context-protocol,scraper,media-downloader,archiver,cli,harvester
34
+ Classifier: Development Status :: 5 - Production/Stable
35
+ Classifier: Environment :: Console
36
+ Classifier: Intended Audience :: Developers
37
+ Classifier: License :: OSI Approved :: MIT License
38
+ Classifier: Operating System :: OS Independent
39
+ Classifier: Programming Language :: Python :: 3
40
+ Classifier: Programming Language :: Python :: 3.9
41
+ Classifier: Programming Language :: Python :: 3.10
42
+ Classifier: Programming Language :: Python :: 3.11
43
+ Classifier: Programming Language :: Python :: 3.12
44
+ Classifier: Programming Language :: Python :: 3.13
45
+ Classifier: Topic :: Communications
46
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
47
+ Classifier: Topic :: System :: Archiving
48
+ Classifier: Topic :: Utilities
49
+ Requires-Python: >=3.9
50
+ Description-Content-Type: text/markdown
51
+ License-File: LICENSE
52
+ Dynamic: license-file
53
+
54
+ <div align="center">
55
+
56
+ <img src="https://raw.githubusercontent.com/Bilal140202/xthread-agent/main/assets/readme/banner.svg" width="820" alt="xthread-agent — public X/Twitter threads as machine-readable JSON, no login required">
57
+
58
+ **No-login access to public X/Twitter threads — built for AI agents.**
59
+
60
+ Give it any public status URL; it returns the reconstructed thread (posts,
61
+ authors, timestamps, quoted posts), every photo and video as files on disk,
62
+ and a machine-readable manifest. No login. No API keys. No browser.
63
+ Deterministic, stdlib-only Python.
64
+
65
+ [![CI](https://github.com/Bilal140202/xthread-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/Bilal140202/xthread-agent/actions/workflows/ci.yml)
66
+ [![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-8A63D2?style=flat-square)](https://www.python.org/downloads/)
67
+ [![License: MIT](https://img.shields.io/badge/license-MIT-8A63D2?style=flat-square)](LICENSE)
68
+ [![dependencies: stdlib only](https://img.shields.io/badge/dependencies-stdlib%20only-2da44e?style=flat-square)](#requirements)
69
+ [![tests: 135 passing](https://img.shields.io/badge/tests-135%20passing-2da44e?style=flat-square)](#testing)
70
+ [![Docs](https://img.shields.io/badge/docs-bilal140202.github.io%2Fxthread--agent-8A63D2?style=flat-square)](https://bilal140202.github.io/xthread-agent/)
71
+ [![MCP](https://img.shields.io/badge/MCP-stdio%20server-5b45b1?style=flat-square)](#mcp-server-model-context-protocol)
72
+ [![GitHub stars](https://img.shields.io/github/stars/Bilal140202/xthread-agent?style=social)](https://github.com/Bilal140202/xthread-agent/stargazers)
73
+
74
+ ```bash
75
+ python3 xthread-agent.py "https://x.com/<user>/status/<status_id>" --out media/
76
+ ```
77
+
78
+ [Website](https://bilal140202.github.io/xthread-agent/) ·
79
+ [Why it exists](#why-this-exists) ·
80
+ [For AI agents](#for-ai-agents) ·
81
+ [Endpoint matrix](docs/endpoint-matrix.md) ·
82
+ [Releases](https://github.com/Bilal140202/xthread-agent/releases)
83
+
84
+ </div>
85
+
86
+ ---
87
+
88
+ ## Why this exists
89
+
90
+ X locked down its public GraphQL endpoints. As of 2026, the standard toolbox is
91
+ broken in specific, well-defined ways — this tool routes around all of them
92
+ (see the [endpoint matrix](docs/endpoint-matrix.md) for the full autopsy):
93
+
94
+ | Approach | Status in 2026 |
95
+ |---|---|
96
+ | `gallery-dl` guest-token + `TweetResultByRestId` | ❌ Dead — guest token activates, GraphQL returns empty payload |
97
+ | `yt-dlp` on status URLs | ❌ Dead — same GraphQL wall |
98
+ | **Nitter** (all public instances) | ❌ Dead — timeouts / HTTP 451 |
99
+ | Direct `x.com` scrape (headless or curl) | ❌ Blocked — empty shell for datacenter IPs |
100
+ | `api.vxtwitter.com` | ✅ **Alive again** (re-verified 2026-09-24) — used as the **fallback decoder slot** |
101
+ | sotwe.com / twstalker.com mirrors | ❌ Blocked — Cloudflare 403 |
102
+ | `syndication.twitter.com` timeline endpoint | ⚠️ Rate-limited (429) — unusable for thread walking |
103
+ | `cdn.syndication.twimg.com/tweet-result` | ✅ Works — single tweets only, no traversal |
104
+ | `api.fxtwitter.com/status/<id>` | ✅ **Works** — full tweet JSON incl. multi-video "amplify" media |
105
+ | **unrollnow.com/status/<id>** | ✅ **Works** — public thread walk; ⚠️ its pages also embed *recommendations*, which this tool filters out |
106
+
107
+ **xthread-agent = Thread Walker (UnrollNow → ThreadReaderApp fallback) +
108
+ Metadata Decoder (FixTweet, with vxtwitter fallback) + Thread Reconstructor
109
+ (`replying_to_status` chain) + Media Fetcher (twimg CDN) — dual-homed where
110
+ it matters, honest everywhere.**
111
+
112
+ ---
113
+
114
+ ## What you get
115
+
116
+ <p align="center">
117
+ <img src="https://raw.githubusercontent.com/Bilal140202/xthread-agent/main/assets/readme/pipeline.svg" width="860"
118
+ alt="Five-stage pipeline: normalize any input form; walk the thread (UnrollNow with ThreadReaderApp fallback); decode each post (FixTweet with vxtwitter fallback); reconstruct the true self-reply chain and filter recommendations; deliver thread_manifest.json plus verified media from the twimg CDN.">
119
+ </p>
120
+
121
+ The deliverable is a versioned envelope — `thread_manifest.json` — validated
122
+ against a bundled [draft-07 JSON Schema](schema/thread-result.schema.json),
123
+ next to the downloaded media. Illustrative excerpt (the schema is the
124
+ contract):
125
+
126
+ ```json
127
+ {
128
+ "schema_version": "3.0",
129
+ "source": { "tool": "xthread-agent", "version": "3.2.0", "generated_at": "…" },
130
+ "request": { "input": "https://x.com/jack/status/20", "status_id": "20",
131
+ "canonical_url": "https://x.com/i/web/status/20" },
132
+ "status": "ok",
133
+ "thread": {
134
+ "root_status_id": "20", "tweet_count": 1,
135
+ "walker_slot": "unrollnow",
136
+ "chain_reconstructed": true, "degraded_to_root_only": false
137
+ },
138
+ "posts": [
139
+ {
140
+ "id": "20",
141
+ "url": "https://x.com/i/web/status/20",
142
+ "text": "just setting up my twttr",
143
+ "created_at_iso": "2006-03-21T20:50:14.000Z",
144
+ "author": { "screen_name": "jack", "name": "jack", "followers": "…" },
145
+ "metrics": { "likes": "…", "retweets": "…", "views": "…" },
146
+ "media": {
147
+ "photos": [ { "url": "…pbs.twimg.com/…", "file": "20_p1.jpg", "downloaded": true } ],
148
+ "videos": []
149
+ },
150
+ "thread_position": 1
151
+ }
152
+ ],
153
+ "errors": [],
154
+ "metadata": { "duration_sec": "…" }
155
+ }
156
+ ```
157
+
158
+ `status` is `ok` (posts, no errors), `partial` (posts but something degraded —
159
+ see `errors[]`), or `empty` (nothing harvested; fail-closed). Every degraded
160
+ path names itself with a stable error code instead of guessing.
161
+
162
+ ### Numbers — verified, not promised
163
+
164
+ | Verified | Value |
165
+ |---|---|
166
+ | Live endpoints re-verified | 2026-09-24, 3/3 — walker, both decoders, CDN ([matrix](docs/endpoint-matrix.md)) |
167
+ | Redundancy | 2 walker slots + 2 decoder slots — dual-homed discovery and decode |
168
+ | Input handling | 24 URL forms accepted, 10 rejected with stable codes (incl. t.co one-hop expansion) |
169
+ | Offline tests | 135 in ~1 s — no network, synthetic fixtures only |
170
+ | CI matrix | Python 3.9 – 3.13, every push |
171
+ | Runtime dependencies | 0 — Python stdlib only |
172
+ | Largest live-verified transfer | 167 MB 4K MP4 + poster — atomic write, `Content-Length`-verified |
173
+ | Media actually verified live | JPEG photo (1455×980), 4K MP4, poster frames, real manifests |
174
+
175
+ ---
176
+
177
+ ## Architecture
178
+
179
+ ```
180
+ status URL
181
+ │
182
+ ▼
183
+ ┌───────────────────────────────┐
184
+ │ 0. NORMALIZE + VALIDATE │ proper URL parsing (x.com / twitter.com,
185
+ │ → bare status id │ mobile/www hosts, /photo /video suffixes,
186
+ └───────────────┬───────────────┘ t.co shortlinks resolved one hop)
187
+ ▼
188
+ ┌───────────────────────────────┐
189
+ │ 1. THREAD WALK (2 slots) │ GET unrollnow.com/status/<root_id>
190
+ │ regex-extract every │ → ordered, deduped candidate IDs
191
+ │ candidate ID │ (ThreadReaderApp fallback slot;
192
+ │ │ root ALWAYS kept; capped at 50;
193
+ │ │ slot named in walker_slot)
194
+ └───────────────┬───────────────┘
195
+ ▼
196
+ ┌───────────────────────────────┐
197
+ │ 2. METADATA DECODE │ GET api.fxtwitter.com/status/<id>
198
+ │ (3 retries, backoff) │ → text, author, stats, media[]
199
+ │ 404s = media IDs, skipped │ fallback slot: api.vxtwitter.com
200
+ └───────────────┬───────────────┘
201
+ ▼
202
+ ┌───────────────────────────────┐
203
+ │ 3. CHAIN RECONSTRUCTION │ true self-reply chain from
204
+ │ walk UP to thread start, │ replying_to_status — unrelated
205
+ │ walk DOWN through replies │ same-author recommendations are
206
+ └───────────────┬───────────────┘ excluded, not harvested
207
+ ▼
208
+ ┌───────────────────────────────┐
209
+ │ 4. CDN DOWNLOAD │ video.twimg.com/…mp4 (best variant;
210
+ │ atomic (.part + rename) │ m3u8-only videos fall back to the
211
+ │ resumable, size-verified │ highest-bitrate mp4 in formats[])
212
+ └───────────────┬───────────────┘
213
+ ▼
214
+ ┌───────────────────────────────┐
215
+ │ 5. thread_manifest.json │ enveloped result: source, request,
216
+ │ (schema_version 3.0) │ thread stats, posts[], errors[],
217
+ │ │ metadata — UTF-8, atomic write
218
+ └───────────────────────────────┘
219
+ ```
220
+
221
+ ### Key research findings baked into the code
222
+
223
+ - **UnrollNow pages embed recommendations, not just the conversation.**
224
+ Same-author tweets that do not reply to the root appear alongside thread
225
+ members; the walker's output is treated strictly as *candidates*. Chain
226
+ membership is decided by `replying_to_status` from the decoder — never by
227
+ page order. (v2.0.0 harvested recommendations as thread members; v3 does not.)
228
+ - **The root is always harvested**, even for short legacy IDs the walker's
229
+ regex cannot see (v2.0.0 silently dropped the root in that case).
230
+ - **A 404/451 from the decoder is a filter signal, not an error.** It arrives
231
+ both as a 200 body with `code: 404` and as a real HTTP status — both are
232
+ handled as instant, retry-free filters.
233
+ - **FixTweet decodes multi-video "amplify" tweets**, and its `formats[]`
234
+ array carries mp4 variants with bitrates — so even videos whose primary URL
235
+ is HLS-only can usually be downloaded as mp4 by variant selection.
236
+ - **`video.twimg.com` and `pbs.twimg.com` need no auth** once you have the
237
+ URL. All authentication burden sits in front of *discovery*, not *delivery*.
238
+ - Media downloads are restricted to `*.twimg.com` over https, verified against
239
+ `Content-Length`, streamed, and atomically renamed — a truncated transfer or
240
+ a compromised decoder payload cannot corrupt local files.
241
+
242
+ ---
243
+
244
+ ## Quickstart
245
+
246
+ ```bash
247
+ # stdlib only — Python 3.9+, nothing to install
248
+ python3 xthread-agent.py "https://x.com/<user>/status/<id>"
249
+ ```
250
+
251
+ Output layout:
252
+
253
+ ```
254
+ media/
255
+ ├── <tweetid>_v1.mp4 # videos (best-quality mp4)
256
+ ├── <tweetid>_v1_poster.jpg # poster frames
257
+ ├── <tweetid>_p1.jpg # photos
258
+ └── thread_manifest.json # everything, mapped (envelope, schema 3.0)
259
+ ```
260
+
261
+ ### CLI reference
262
+
263
+ ```bash
264
+ python3 xthread-agent.py <status_url_or_id> [--out DIR] [--no-download]
265
+ [--json] [--quiet] [--version]
266
+ ```
267
+
268
+ | Flag | Purpose |
269
+ |---|---|
270
+ | `--out DIR` | Output directory (default `x_thread_media`) |
271
+ | `--no-download` | Manifest only — resolve, decode, reconstruct; skip media |
272
+ | `--json` | Machine-readable summary on **stdout** (all logs stay on stderr) |
273
+ | `--quiet` | Suppress log lines |
274
+ | `--version` | Print version |
275
+
276
+ Exit codes: `0` = at least one post harvested, `1` = nothing harvested / error,
277
+ `2` = usage error.
278
+
279
+ ### JSON output (for AI agents)
280
+
281
+ ```bash
282
+ python3 xthread-agent.py "https://x.com/<user>/status/<id>" --json --quiet
283
+ ```
284
+
285
+ ```json
286
+ {
287
+ "ok": true,
288
+ "status": "ok",
289
+ "root_id": "…",
290
+ "canonical_url": "https://x.com/i/web/status/…",
291
+ "tweets": 4,
292
+ "videos": 11,
293
+ "photos": 2,
294
+ "downloaded": 13,
295
+ "failed_downloads": 0,
296
+ "out_dir": "media",
297
+ "manifest_path": "media/thread_manifest.json",
298
+ "errors": 0,
299
+ "duration_sec": 86.3
300
+ }
301
+ ```
302
+
303
+ `status` is `ok` (posts, no errors), `partial` (posts but something degraded —
304
+ see `errors`), or `empty` (nothing harvested). The full detail — posts in
305
+ thread order, authors, timestamps, quoted posts, media URLs, local file paths,
306
+ per-stage errors — lives in `thread_manifest.json`
307
+ ([JSON Schema](schema/thread-result.schema.json)).
308
+
309
+ New in v3.2: `thread.walker_slot` names the discovery slot that served the
310
+ walk (`unrollnow`, `threadreaderapp`, or `none` when degraded to root-only).
311
+
312
+ ---
313
+
314
+ ## For AI agents
315
+
316
+ ### The 3-command contract
317
+
318
+ ```bash
319
+ python3 xthread-agent.py "<status_url>" --json --quiet # run
320
+ cat <out>/thread_manifest.json # inspect
321
+ ```
322
+
323
+ Exit code `0` = at least one post harvested. `1` = nothing. `2` = usage error.
324
+ Logs are always on **stderr**; `--json` results are always on **stdout**, so
325
+ the two can be piped safely.
326
+
327
+ `agent.md` is the complete operating manual — an AI agent reading only that
328
+ file can run this tool end-to-end without asking a human a single question.
329
+ `agents.md` defines the role prompts each internal stage must conform to.
330
+ `demo.py` is a minimal runnable example of programmatic consumption.
331
+
332
+ ### MCP server (Model Context Protocol)
333
+
334
+ For MCP-compatible agent hosts (Claude Desktop, Zed, custom hosts),
335
+ `mcp_server.py` exposes the harvester as tools over the standard stdio
336
+ transport — still stdlib-only, still no login:
337
+
338
+ ```bash
339
+ python3 mcp_server.py # speaks MCP on stdin/stdout; logs on stderr
340
+ ```
341
+
342
+ | Tool | What it does |
343
+ |---|---|
344
+ | `extract_thread` | Full harvest: thread reconstruction + media downloads; returns the envelope |
345
+ | `lookup_status` | Metadata-only (`--no-download` equivalent): text, authors, timestamps, media URLs |
346
+ | `read_manifest` | Returns an existing `thread_manifest.json` verbatim (refuses any other filename) |
347
+ | `get_schema` | Returns the JSON Schema for the envelope contract |
348
+
349
+ The server never reimplements the pipeline — each tool call shells out to
350
+ `xthread-agent.py` as a subprocess with a hard timeout, so the CLI contract,
351
+ schema, and politeness rules stay the single source of truth. Register it in
352
+ your MCP client config as a stdio command, e.g.
353
+ `{"command": "python3", "args": ["/path/to/mcp_server.py"]}`.
354
+
355
+ ---
356
+
357
+ ## Documentation
358
+
359
+ | File | Purpose |
360
+ |---|---|
361
+ | [`agent.md`](agent.md) | Agent entry point — how to run, the JSON contract, decision tree |
362
+ | [`agents.md`](agents.md) | Role prompts for the internal agent roster (the spec) |
363
+ | [`mcp_server.py`](mcp_server.py) | MCP wrapper — exposes the harvester as MCP tools over stdio (stdlib-only) |
364
+ | [`demo.py`](demo.py) | Minimal end-to-end consumption example |
365
+ | [`schema/thread-result.schema.json`](schema/thread-result.schema.json) | JSON Schema (draft-07) for the manifest envelope |
366
+ | [`docs/research-blog.md`](docs/research-blog.md) | Research chronicle: the X lockdown and the bypass architecture |
367
+ | [`docs/endpoint-matrix.md`](docs/endpoint-matrix.md) | Living reference: every endpoint, its status, its failure signature |
368
+ | [`PROJECT_CONTEXT.md`](PROJECT_CONTEXT.md) | Why this exists, design decisions, fragile parts — for future maintainers |
369
+ | [`RELEASE_NOTES.md`](RELEASE_NOTES.md) | Version history |
370
+ | [`PUBLISHING.md`](PUBLISHING.md) | The exact PyPI publishing checklist (Trusted Publishing) |
371
+ | [`CONTRIBUTING.md`](CONTRIBUTING.md) | How to contribute |
372
+ | [Portfolio site](https://bilal140202.github.io/xthread-agent/) | This project's GitHub Pages home (source: `site/`) |
373
+
374
+ ---
375
+
376
+ ## Constraints (non-negotiable)
377
+
378
+ 1. **No login. No cookies. No OAuth. No browser.** Public content only.
379
+ 2. **No GUI. No interactive prompts.** 100% non-interactive CLI.
380
+ 3. **No LLM at runtime.** Deterministic state machine.
381
+ 4. **stdlib only.** Single file, no pip installs, Python 3.9+.
382
+ 5. **Logs on stderr, data on stdout.** Always pipe-safe.
383
+ 6. **Files stay under the output directory.** Media URLs are restricted to
384
+ X's CDN hosts; remote IDs are validated before use in filenames.
385
+
386
+ ## Requirements
387
+
388
+ - Python **3.9+** (the tool itself). Tests also run on stdlib `unittest`.
389
+ - Outbound HTTPS to `unrollnow.com`, `threadreaderapp.com` (fallback walker),
390
+ `api.fxtwitter.com`, `api.vxtwitter.com` (fallback only), `t.co` (only for
391
+ shortlink inputs), `video.twimg.com`, `pbs.twimg.com`.
392
+
393
+ ## Install
394
+
395
+ ```bash
396
+ # zero-install: curl one file and run it (works today)
397
+ python3 xthread-agent.py "https://x.com/<user>/status/<id>" --json --quiet
398
+
399
+ # from PyPI (console script + module, same single-file core)
400
+ pip install xthread-agent
401
+ xthread-agent "https://x.com/<user>/status/<id>" --json --quiet
402
+ ```
403
+
404
+ > **PyPI status:** the package is built, `twine check`-passed, and smoke-tested
405
+ > locally; publication is pending one-time Trusted-Publisher configuration on
406
+ > PyPI (owner action — [PUBLISHING.md](PUBLISHING.md) §1 has the exact steps).
407
+ > Until then, the zero-install path above works with no installation at all.
408
+
409
+ The PyPI wheel carries `xthread_agent/__init__.py`, a byte-identical copy of
410
+ `xthread-agent.py` enforced by a drift-guard test — the single-file design
411
+ constraint (PROJECT_CONTEXT.md §5.1) is intact.
412
+
413
+ ---
414
+
415
+ ## Testing
416
+
417
+ ```bash
418
+ python3 -m unittest discover -s tests -p "test_*.py" -v # 135 offline tests, ~1s
419
+ ```
420
+
421
+ The suite covers URL normalization (including t.co expansion), both walker
422
+ slots, both decoders (including the vxtwitter fallback and HTTP
423
+ 404/451/429 paths), chain reconstruction, payload mapping, atomic downloads,
424
+ the envelope contract, CLI behavior, the MCP wrapper (protocol framing,
425
+ tools, error paths), and package/source sync — all against synthetic
426
+ fixtures (no network). CI (`.github/workflows/ci.yml`) runs the same suite on
427
+ Python 3.9–3.13 on every push and PR; live endpoints are deliberately never
428
+ probed from CI. For a real end-to-end run, use `demo.py` against any public
429
+ status URL of your choice; keep the request rate polite and test against
430
+ content you control where possible.
431
+
432
+ ---
433
+
434
+ ## Media
435
+
436
+ - **Videos**: downloaded as mp4 at the best available quality (the decoder's
437
+ primary URL, or the highest-bitrate mp4 variant when the primary is HLS).
438
+ Video metadata includes duration, dimensions, format, and the full variant
439
+ list. Videos that genuinely have no mp4 variant are reported with
440
+ `downloadable: false` and a `reason` (`hls_only`) — never silently dropped.
441
+ - **Photos**: downloaded at source resolution with alt text and dimensions.
442
+ - **Posters**: every downloaded video gets its poster frame.
443
+ - **Quoted posts**: recorded with author, text, timestamp, and media *URLs*
444
+ (quoted media is not downloaded — it belongs to the quoted post, and this
445
+ keeps runs polite and output directories honest).
446
+
447
+ ## No-login architecture
448
+
449
+ "No login" here means: the tool never presents credentials, cookies, session
450
+ tokens, or a browser fingerprint, and it never touches `x.com` itself. It
451
+ reads three *public* surfaces that any visitor can reach without
452
+ authentication: an unrolling service (thread candidates), two open
453
+ link-decoder workers (per-tweet JSON), and X's own media CDN (bytes). This
454
+ works because X's authentication wall guards *discovery* APIs, while the CDN
455
+ serves whatever URL a decoder already resolved. It does **not** mean the
456
+ access is officially supported by X or guaranteed to last — see
457
+ [Limitations](#limitations) and the [endpoint matrix](docs/endpoint-matrix.md).
458
+
459
+ ## Limitations (be honest, we are)
460
+
461
+ - **Dependency on free third-party services.** If UnrollNow or FixTweet change
462
+ or gate datacenter IPs, functionality degrades (root-only harvest, or
463
+ `empty`). The fallback decoder slot mitigates but does not eliminate this.
464
+ - **Retweet URLs resolve to the original post.** For a retweet URL the
465
+ decoder returns the original tweet's payload, so `posts[0].id` is the
466
+ *original* post ID while `request.status_id` / `thread.root_status_id`
467
+ stay the requested ID. The harvested content is exactly what that URL
468
+ publicly shows (the retweeted post).
469
+ - **Linear self-reply chains only.** Threads where the author branches into
470
+ multiple replies get the first-seen branch; cross-author reply trees are out
471
+ of scope by design.
472
+ - **Deleted/protected/age-restricted content** fails closed (`status: empty`)
473
+ — nothing here bypasses access controls.
474
+ - **HLS-only videos** (rare) are reported but not downloaded.
475
+ - **A slow-drip CDN transfer** is cut off at the transfer deadline; the retry
476
+ restarts the file rather than resuming by byte range.
477
+ - **UnrollNow outages** degrade the walk to root-only; the manifest records
478
+ this (`degraded_to_root_only: true`) so callers can tell.
479
+ - **One run at a time per output directory.** Concurrent runs can interleave.
480
+ - **"Publicly accessible" ≠ "officially supported."** Every technique here
481
+ depends on surfaces X does not officially expose to tools.
482
+
483
+ ## Legal / platform considerations
484
+
485
+ This tool reaches only publicly accessible content and depends on third-party
486
+ public services. Using it may be subject to — and is **your** responsibility
487
+ under — X's Terms of Service, the terms of the third-party services involved,
488
+ applicable copyright law, data-protection law, and any other rules that apply
489
+ to you or your jurisdiction. Nothing in this repository grants any license to
490
+ the content harvested: posts and media belong to their authors and rights
491
+ holders. Do not use the tool to harass, dox, or invade privacy; do not
492
+ republish harvested media commercially; archive responsibly and credit
493
+ creators. If your use case requires guaranteed, sanctioned access, use the
494
+ official X API.
495
+
496
+ ## Ethics & disclaimer
497
+
498
+ For personal archiving and research. Respect creators: whoever curated the
499
+ thread you harvest added value; the underlying media belongs to its original
500
+ rights holders. Don't repost harvested media commercially. Don't use this tool
501
+ to invade anyone's privacy — it only reaches public content that any visitor
502
+ can see.
503
+
504
+ ---
505
+
506
+ ## FAQ
507
+
508
+ **Does it really need no login?**
509
+ It never presents credentials, cookies, session tokens, or a browser
510
+ fingerprint, and never touches `x.com` itself. It reads three public surfaces:
511
+ an unrolling service, two open link-decoder workers, and X's own media CDN.
512
+ That works because X's auth wall guards *discovery* APIs while the CDN serves
513
+ bytes to anyone holding a resolved URL. It does not mean X sanctions or
514
+ guarantees this access.
515
+
516
+ **Is this allowed by X's Terms of Service?**
517
+ Reaching publicly accessible content through third-party surfaces may be
518
+ subject to X's Terms of Service, the third parties' terms, copyright and
519
+ data-protection law — your responsibility, your jurisdiction. The tool reaches
520
+ only what any visitor can see, fails closed on protected content, and grants
521
+ no license to harvested media. If you need guaranteed, sanctioned access, use
522
+ the official X API.
523
+
524
+ **What happens when an upstream service dies?**
525
+ Nothing explodes: discovery is dual-homed (UnrollNow → ThreadReaderApp),
526
+ decoding has a fallback slot (vxtwitter), and every failure lands in the
527
+ envelope's `errors[]` with a stable code. Worst case is a root-only harvest or
528
+ an honest `status: empty`. Slots are designed to be replaced behind their
529
+ contracts.
530
+
531
+ **Why does a retweet URL resolve to the original post?**
532
+ The decoder returns the original tweet's payload for a retweet URL, so the
533
+ harvested content is exactly what that URL publicly shows (the retweeted
534
+ post). `posts[0].id` is the original post ID; `request.status_id` and
535
+ `thread.root_status_id` stay the ID you asked about, so nothing is hidden.
536
+
537
+ **Why stdlib-only Python?**
538
+ Deployment story: copy one file into a bare sandbox and run it. No pip, no
539
+ node, no ffmpeg — anywhere Python 3.9+ exists, the agent works. The PyPI
540
+ package exists for convenience and is byte-identical to the single file,
541
+ enforced by a drift-guard test.
542
+
543
+ ---
544
+
545
+ ## Star history
546
+
547
+ <p align="center">
548
+ <picture>
549
+ <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=Bilal140202/xthread-agent&type=Date&theme=dark" />
550
+ <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=Bilal140202/xthread-agent&type=Date" />
551
+ <img alt="Star history chart for Bilal140202/xthread-agent" src="https://api.star-history.com/svg?repos=Bilal140202/xthread-agent&type=Date">
552
+ </picture>
553
+ </p>
554
+
555
+ ## License
556
+
557
+ MIT. See [`LICENSE`](LICENSE).
558
+
559
+ ## Acknowledgments
560
+
561
+ - [FixTweet / FxTwitter](https://github.com/FixTweet/FxTwitter) — the public worker that decodes tweet payloads
562
+ - [vxtwitter](https://github.com/dylanpdx/BetterTwitFix) — the fallback decoder slot
563
+ - [UnrollNow](https://unrollnow.com) — public thread unrolling service
564
+ - [ytagent](https://github.com/Bilal140202/ytagent) — sibling project; the "document which doors remain open" philosophy started there
565
+
566
+ ## Links
567
+
568
+ - **Website:** https://bilal140202.github.io/xthread-agent/
569
+ - **GitHub:** https://github.com/Bilal140202/xthread-agent
570
+ - **Issues:** https://github.com/Bilal140202/xthread-agent/issues
571
+ - **Releases:** https://github.com/Bilal140202/xthread-agent/releases
@@ -0,0 +1,8 @@
1
+ xthread_agent/__init__.py,sha256=ESEDPYEYoHHbzcYCojR44p_JvbWi5W_bxc1-aC_APgo,43034
2
+ xthread_agent/__main__.py,sha256=TvsQ1m3nq2YxibfseUaFJGUQylsDUoINo1kOlEnVw_g,92
3
+ xthread_agent-3.2.0.dist-info/licenses/LICENSE,sha256=ESYyLizI0WWtxMeS7rGVcX3ivMezm-HOd5WdeOh-9oU,1056
4
+ xthread_agent-3.2.0.dist-info/METADATA,sha256=W2wtiS8BmWBhjZMu5vdLVSoGKcoxUY0BZZ2U1MTZbI0,27626
5
+ xthread_agent-3.2.0.dist-info/WHEEL,sha256=YVMoNqKzERt-wjUZwJ33xBGAwnFl-4cqbYkTtWa4itE,91
6
+ xthread_agent-3.2.0.dist-info/entry_points.txt,sha256=vbNsqxdzOAkLk_sek6luMIPOomatbudfxMvf6smmXT8,53
7
+ xthread_agent-3.2.0.dist-info/top_level.txt,sha256=hdBREv29lv3EeQ8n0f9Vcrt3EDuw04l3PSgr5i6ZUho,14
8
+ xthread_agent-3.2.0.dist-info/RECORD,,
@@ -0,0 +1,5 @@
1
+ Wheel-Version: 1.0
2
+ Generator: setuptools (84.0.0)
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
5
+
@@ -0,0 +1,2 @@
1
+ [console_scripts]
2
+ xthread-agent = xthread_agent:main
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1 @@
1
+ xthread_agent