pyXcom 0.6.0__tar.gz → 0.7.0__tar.gz

{pyxcom-0.6.0 → pyxcom-0.7.0}/CITATION.cff

@@ -6,12 +6,12 @@ authors:
   - family-names: Wang
     given-names: Haocheng
     affiliation: Communication University of China
-version: 0.6.0
-date-released: 2026-09-30
+version: 0.7.0
+date-released: 2026-10-01
 license: MIT
 repository-code: "https://github.com/haochengw372-hash/pyXcom"
 url: "https://github.com/haochengw372-hash/pyXcom"
-abstract: "A Python package for collecting public X profiles, authored posts and replies, keyword matches, and bounded multi-level conversations with linked tables and resumable collection."
+abstract: "A Python package for collecting public X posts, native search results, reply branches, quotations, repost activity and observed follower/following networks, with linked tables, raw-response provenance and resumable snapshots."
 keywords:
   - computational communication
   - social media

{pyxcom-0.6.0 → pyxcom-0.7.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.5
 Name: pyXcom
-Version: 0.6.0
-Summary: Collect public X profiles, posts, replies, and search results with your existing browser session
+Version: 0.7.0
+Summary: Collect public X posts, search results, conversations, reposts, and observed follow networks with your browser session
 Project-URL: Homepage, https://github.com/haochengw372-hash/pyXcom
 Project-URL: Repository, https://github.com/haochengw372-hash/pyXcom
 Author: Haocheng Wang
@@ -20,7 +20,9 @@ Description-Content-Type: text/markdown
 
 **使用前请自行打开 Chrome 并登录 X。pyXcom 不会打开或控制浏览器，也不会替你登录。**
 
-Collect public X user profiles, main posts, authored replies and keyword matches using your existing Chrome or Edge session. Outputs are three linked CSV tables: `users.csv`, `posts.csv` and `comments.csv`.
+Collect public X user profiles, posts, replies, search results and observed network relationships using your existing Chrome or Edge session. Post datasets contain three linked CSV tables: `users.csv`, `posts.csv` and `comments.csv`; network datasets have separate relationship and snapshot tables.
+
+Version 0.7.0 adds native search, quote discovery, repost activity and follower/following snapshots. These capabilities have passed offline tests and bounded authenticated live endpoint tests. Collection remains limited to results returned by X; the tests do not establish complete historical or network coverage.
 
 The interface follows [PykTok](https://github.com/dfreelon/pyktok)'s approachable get/save convention, with explicit parameters and resumable datasets. pyXcom is an independent implementation; it does not depend on PykTok or twikit.
 
@@ -81,18 +83,23 @@ Methods belong to `XClient`. Every method lists its accepted parameters explicit
 | Multiple post IDs | `get_posts(post_ids)` | — | — |
 | Authored main posts | `get_user_posts(handle, ...)` | `iter_user_posts(handle, ...)` | `save_user_posts(handle, output_dir, ...)` |
 | Authored replies | `get_user_replies(handle, ...)` | `iter_user_replies(handle, ...)` | `save_user_replies(handle, output_dir, ...)` |
-| Replies below a main post | `get_post_comments(post_id_or_url, ...)` | `iter_post_comments(post_id_or_url, ...)` | `save_post_comments(post_id_or_url, output_dir, ...)` |
+| Replies below a post or reply | `get_post_comments(post_id_or_url, ...)` | `iter_post_comments(post_id_or_url, ...)` | `save_post_comments(post_id_or_url, output_dir, ...)` |
 | Keyword matches | `get_search_posts(keyword, ...)` | `iter_search_posts(keyword, ...)` | `save_search_posts(keyword, output_dir, ...)` |
+| Native cross-user query | `get_search_query(query, ...)` | `iter_search_query(query, ...)` | `save_search_query(query, output_dir, ...)` |
+| Quotes of a post | `get_post_quotes(post_id_or_url, ...)` | `iter_post_quotes(post_id_or_url, ...)` | `save_post_quotes(post_id_or_url, output_dir, ...)` |
+| Account repost activity | `get_user_reposts(handle, ...)` | `iter_user_reposts(handle, ...)` | `save_user_reposts(handle, output_dir, ...)` |
+| Followers / following | `get_followers(user_id, ...)` / `get_following(user_id, ...)` | `iter_followers(user_id, ...)` / `iter_following(user_id, ...)` | `save_followers(user_id, output_dir, ...)` / `save_following(user_id, output_dir, ...)` |
+| Reposter user list | `get_post_reposters(post_id_or_url, ...)` | `iter_post_reposters(post_id_or_url, ...)` | `save_post_reposters(post_id_or_url, output_dir, ...)` |
 | Main posts + replies | — | — | `save_user_activity(handle, output_dir, ...)` |
 | Multiple accounts | — | — | `save_users_activity(handles, output_dir, ...)` |
 
-- `get_*` returns a `Profile`, `Post`, or `list[Post]`; it does not write a dataset.
+- `get_*` returns a `Profile`, `Post`, `list[Post]`, or network `list[Profile]`; it does not write a dataset.
 - `iter_*` returns an iterator. Requests happen as it is consumed.
-- `save_*` writes the standard dataset and returns `CollectionResult` with `output_dir`, `post_count`, `pages_fetched`, `complete`, and `reason`. `post_count` counts **all collected observations**, including replies; use the output manifest for separate table counts.
+- `save_*` returns `CollectionResult` with `output_dir`, `post_count`, `pages_fetched`, `complete`, and `reason`. For post datasets `post_count` counts unique records, including replies; for network datasets this compatibility field counts observed users. Consult each manifest for table counts and coverage.
 - Parameters: `handle` for one account, `handles` for several, `keyword` for search, `output_dir` for saved datasets, and `since`/`until` for dates. Options are keyword-only.
 - Dates are UTC: `since` inclusive, `until` exclusive. `max_pages` and `limit` must be positive integers or `None`. Saved timeline/mirror `limit` is a page-boundary stopping threshold, so a final page can exceed it; iterator limits are exact.
 
-See [API reference](docs/api.md) for signatures, return types and compatibility names.
+See [API reference](https://github.com/haochengw372-hash/pyXcom/blob/main/docs/api.md) for signatures, return types and compatibility names.
 
 ### Search one account
 
@@ -110,6 +117,40 @@ with XClient(profile="Default") as client:
 
 Direct account search scans main posts and authored replies, then matches a literal word/phrase without case sensitivity. Matching posts do not represent deduplicated real-world events. Cross-user search requires an explicit `mirror_base`; it sends search terms to that mirror and retrieves discovered post details from X. No mirror is enabled by default.
 
+### Native search, quotes and repost activity
+
+`*_search_query` submits X search syntax to the native Latest search endpoint; it does not require or use a mirror. `*_search_posts` retains its account/literal behavior above. Use a separate output directory for each query or source:
+
+```python
+with XClient(profile="Default") as client:
+    result = client.save_search_query(
+        '("usage reset" OR "rate limit reset") (OpenAI OR Codex)',
+        "output/reset_search", since="2025-09-29", until="2026-09-30",
+        max_pages=2, limit=100,
+    )
+    print(result.complete, result.reason)
+    client.save_post_quotes("1973931546550894681", "output/reset_quotes", max_pages=2)
+    client.save_user_reposts("thsottiaux", "output/tibo_reposts", max_pages=2)
+```
+
+Quote discovery uses native search followed by a check of `quoted_post_id`; zero matches do not establish zero quotes. Repost activity is collected from visible account timeline wrappers, preserving the repost ID and its action timestamp separately from the original post ID/time. A flattened timeline that only returns originals cannot establish action times and reports `repost_activity_unavailable` with an incomplete result. It is not a complete historical repost archive. Search sorting, visibility and pagination remain controlled by X.
+
+### Follow networks and reposter lists
+
+Resolve a handle with `get_user(handle).id`, then pass that stable numeric ID to follower/following methods:
+
+```python
+with XClient(profile="Default") as client:
+    user_id = client.get_user("thsottiaux").id
+    client.save_followers(user_id, "output/tibo_network", max_pages=2, snapshot_id="wave-1")
+    client.save_following(user_id, "output/tibo_network", max_pages=2, snapshot_id="wave-1")
+    client.save_post_reposters("1973931546550894681", "output/reset_reposters", max_pages=2)
+```
+
+Follow edges point from follower to followed account. `follow_edges.csv`, `reposters.csv`, `user_snapshots.csv` and `network_manifest.json` preserve observed relationships, user profiles and coverage per snapshot. A reposter list identifies who visibly reposted a post, but its `action_time_utc` is unknown. Use account repost activity when the wrapper supplies an action time; never substitute the original post time.
+
+Network saves resume a recoverable unfinished snapshot. Once completed or stopped by a terminal bad cursor, calling again without `snapshot_id` creates a new observation; explicitly reusing a completed ID returns that saved snapshot. Memory/iterator methods raise endpoint, parse or rate-limit errors and retain coverage in `last_network_collection`; hitting a budget returns the observed prefix. Saves return an incomplete result and preserve checkpoints on source failures. Network and post collections require separate output directories to protect their different `users.csv` exports. Follow observations cannot establish when a relationship began or reconstruct a historical follow network. Keep collection dates in the analysis. See [examples/research_networks.py](examples/research_networks.py) for a small collection recipe; importing it does not collect data.
+
 ### Collect several accounts
 
 ```python
@@ -124,7 +165,7 @@ with XClient(profile="Default") as client:
 
 Repeat to resume. `rounds=0, wait_on_rate_limit=True` keeps running and waits for rate windows when needed. `progress=callback` receives status dictionaries.
 
-## Collect a main post and nested comments
+## Collect a post or reply and nested comments
 
 ```python
 with XClient(profile="Default") as client:
@@ -144,7 +185,7 @@ pyxcom post-comments 1973931546550894681 --profile Default \
   --output-dir output/tibo_conversation
 ```
 
-The root belongs in `posts.csv`, replies from **all visible authors** in `comments.csv`, and author profiles in `users.csv`. The root must be a main post. Level 1 replies directly to it; level 2 replies to those comments. Quoted content, unrelated recommendations and comments beyond the requested depth are excluded. Profile details absent from the response remain unavailable.
+The seed can be a main post or a reply. Level 1 replies directly to that seed; level 2 replies to those comments. Main seeds belong in `posts.csv`; reply seeds remain replies in `comments.csv`. `seed_post_id` and `seed_relative_depth` describe this branch; `root_post_id`, `parent_post_id` and `depth` retain actual conversation relationships and do not relabel a reply as a platform root. Ancestors returned as context are stored in `context_posts.csv`, not counted as collected descendants. `since`/`until` filter descendant timestamps; traversal retains the ancestry needed to reach in-window descendants. Quoted content, unrelated recommendations and comments beyond the requested depth are excluded. Profile details absent from the response remain unavailable.
 
 `max_comments` is an exact total comment cap, excluding the root. `max_pages` caps conversation requests per call. Repeat the same root/depth/output to resume saved branches and cursors; raise the comment cap to continue a capped dataset. Pending page records are retained, so stopping midway through a page does not skip them. Changing depth requires a separate dataset. Python accepts `None` for unlimited comment/page budgets; depth must be positive.
 
@@ -169,12 +210,16 @@ output/tibo_year/
 
 Replies to oneself remain comments. A quote within a reply retains its quote link in `comments.csv`. Direct replies are depth 1; replies to those are depth 2. Missing ancestry leaves `depth` empty with a `depth_status`; `parent_in_dataset` and `root_in_dataset` indicate observed relationships. Do not interpret missing parents as first-level replies.
 
-`user_replies` means replies **authored by the selected account**. Use `*_post_comments` to collect replies below a particular main post from all visible authors, including nested replies. Neither endpoint establishes full conversation coverage.
+`user_replies` means replies **authored by the selected account**. Use `*_post_comments` to collect replies below a particular post or reply from all visible authors, including nested replies. Neither endpoint establishes full conversation coverage.
 
 CSV conventions: UTF-8 with BOM, snake_case columns, string IDs (import as text in Excel), JSON arrays for list fields, empty values for unavailable scalars, UTC timestamp fields. Users without a saved profile have `profile_available=false`. Engagement counts are snapshots; unknown values are not replaced with zero.
 
 If source records include pure reposts, an additional `interactions.csv` preserves them. The current authored-account collector does not establish complete repost activity. `manifest.json` records schema/layout versions, table counts, collection scope, relationship coverage and hashes. `.pyxcom/` must be retained for resuming; the three CSV files can be shared independently for analysis.
 
+Saved collection pages also retain platform response JSON under `.pyxcom/raw/` (network pages under their snapshot state) and query provenance in `.pyxcom/collection_log.jsonl`. Repeated post observations keep returned engagement metrics in `metric_snapshots.csv`, with source observations in `.pyxcom/observations.jsonl` and metric history in `.pyxcom/metric_snapshots.jsonl`. Unreturned counts remain unavailable. These are observations at retrieval time, not historical metrics at publication. Credentials are not included in the response archive.
+
+When source records contain reply, quote or repost relationships, `post_edges.csv` exports their directions, source/target post and user IDs, action and observation times, and target availability/resolution flags. An unavailable target is retained; its user ID stays empty if unresolved. Known target authors can appear in `users.csv` as stubs with `profile_available=false`. Context-only records do not create source edges. Relationship, context and metric exports appear when observations provide the corresponding data.
+
 ## Command line
 
 CLI task names correspond to the Python methods:
@@ -189,10 +234,19 @@ pyxcom search-posts reset --handle thsottiaux --profile Default \
   --since 2025-09-29 --until 2026-09-30 --output-dir output/tibo_reset
 pyxcom users-activity --handles OpenAI AnthropicAI --profile Default \
   --since 2025-09-29 --until 2026-09-30 --output-dir output/ai_year
+pyxcom search-query '("usage reset" OR "rate limit reset") Codex' --profile Default \
+  --since 2025-09-29 --until 2026-09-30 --max-pages 2 --output-dir output/reset_search
+pyxcom post-quotes 1973931546550894681 --max-pages 2 --output-dir output/reset_quotes
+pyxcom user-reposts thsottiaux --max-pages 2 --output-dir output/tibo_reposts
+pyxcom user-followers USER_ID --max-pages 2 --snapshot-id wave-1 --output-dir output/network
+pyxcom user-following USER_ID --max-pages 2 --snapshot-id wave-1 --output-dir output/network
+pyxcom post-reposters 1973931546550894681 --max-pages 2 --output-dir output/reposters
 ```
 
 Add `--proxy URL` if your network requires one. Use `pyxcom COMMAND --help` for task options. Individual `user`, `post`, and `raw-post` commands accept `--output FILE` for a JSON file.
 
+`post-comments --max-comments all --max-pages all` removes comment/page caps; `--max-depth` still requires a positive integer (default 2). New discovery/network commands also accept `all` for `--max-pages` and `--limit`. Unlimited budgets do not remove visibility or rate limits.
+
 ## Export, migration and verification
 
 These operations work offline, without browser credentials:
@@ -226,17 +280,17 @@ X endpoints may change or impose limits. pyXcom discovers current GraphQL query
 
 ## License
 
-pyXcom is released under the [MIT License](LICENSE). Copyright © 2026 Haocheng Wang.
+pyXcom is released under the [MIT License](https://github.com/haochengw372-hash/pyXcom/blob/main/LICENSE). Copyright © 2026 Haocheng Wang.
 
 ## Citation
 
 **Author:** Haocheng Wang, Communication University of China.
 
-If you use pyXcom in a paper, thesis, dataset, or other research output, please cite the software and report the version used. GitHub's **Cite this repository** menu reads the machine-readable [CITATION.cff](CITATION.cff).
+If you use pyXcom in a paper, thesis, dataset, or other research output, please cite the software and report the version used. GitHub's **Cite this repository** menu reads the machine-readable [CITATION.cff](https://github.com/haochengw372-hash/pyXcom/blob/main/CITATION.cff).
 
 **Suggested reference**
 
-Wang, H. (2026). *pyXcom: Structured and auditable X data collection for communication research* (Version 0.6.0) [Computer software]. https://github.com/haochengw372-hash/pyXcom
+Wang, H. (2026). *pyXcom: Structured and auditable X data collection for communication research* (Version 0.7.0) [Computer software]. https://github.com/haochengw372-hash/pyXcom
 
 **BibTeX**
 
@@ -245,9 +299,15 @@ Wang, H. (2026). *pyXcom: Structured and auditable X data collection for communi
   author  = {Wang, Haocheng},
   title   = {{pyXcom}: Structured and Auditable X Data Collection for Communication Research},
   year    = {2026},
-  version = {0.6.0},
+  version = {0.7.0},
   url     = {https://github.com/haochengw372-hash/pyXcom}
 }
 ```
 
 For reproducible reporting, also describe the collection dates, account or keyword scope, comment-depth limits, package version, and coverage/stop reasons recorded in the output manifest. No DOI or published-paper citation is currently assigned; this reference cites the software itself. Citation is appreciated and does not add a condition to the MIT license.
+
+For longitudinal profile comparisons use `user_snapshots.csv`; `users.csv` consolidates known profile fields. Canonical post tables keep the first saved observation for each ID; later returned text and metrics remain in observation archives and metric snapshots.
+
+Offline development checks: `PYTHONPATH=src python -m pytest -q` and `ruff check src tests examples`. The source distribution includes tests; testing tools are development-only, not runtime dependencies.
+
+Live validation: current X SearchTimeline/Followers use POST read queries; Retweeters is discovered from public lazy-loaded TweetActivity assets. If a search source returns posts outside explicit dates, they are excluded locally and `manifest.json.date_scope` records the mismatch. An empty bounded result with this warning does not establish historical absence. Native reposts generate only repost edges; embedded quote/reply context is not attributed as another action by the reposter.

{pyxcom-0.6.0 → pyxcom-0.7.0}/README.md

@@ -2,7 +2,9 @@
 
 **使用前请自行打开 Chrome 并登录 X。pyXcom 不会打开或控制浏览器，也不会替你登录。**
 
-Collect public X user profiles, main posts, authored replies and keyword matches using your existing Chrome or Edge session. Outputs are three linked CSV tables: `users.csv`, `posts.csv` and `comments.csv`.
+Collect public X user profiles, posts, replies, search results and observed network relationships using your existing Chrome or Edge session. Post datasets contain three linked CSV tables: `users.csv`, `posts.csv` and `comments.csv`; network datasets have separate relationship and snapshot tables.
+
+Version 0.7.0 adds native search, quote discovery, repost activity and follower/following snapshots. These capabilities have passed offline tests and bounded authenticated live endpoint tests. Collection remains limited to results returned by X; the tests do not establish complete historical or network coverage.
 
 The interface follows [PykTok](https://github.com/dfreelon/pyktok)'s approachable get/save convention, with explicit parameters and resumable datasets. pyXcom is an independent implementation; it does not depend on PykTok or twikit.
 
@@ -63,18 +65,23 @@ Methods belong to `XClient`. Every method lists its accepted parameters explicit
 | Multiple post IDs | `get_posts(post_ids)` | — | — |
 | Authored main posts | `get_user_posts(handle, ...)` | `iter_user_posts(handle, ...)` | `save_user_posts(handle, output_dir, ...)` |
 | Authored replies | `get_user_replies(handle, ...)` | `iter_user_replies(handle, ...)` | `save_user_replies(handle, output_dir, ...)` |
-| Replies below a main post | `get_post_comments(post_id_or_url, ...)` | `iter_post_comments(post_id_or_url, ...)` | `save_post_comments(post_id_or_url, output_dir, ...)` |
+| Replies below a post or reply | `get_post_comments(post_id_or_url, ...)` | `iter_post_comments(post_id_or_url, ...)` | `save_post_comments(post_id_or_url, output_dir, ...)` |
 | Keyword matches | `get_search_posts(keyword, ...)` | `iter_search_posts(keyword, ...)` | `save_search_posts(keyword, output_dir, ...)` |
+| Native cross-user query | `get_search_query(query, ...)` | `iter_search_query(query, ...)` | `save_search_query(query, output_dir, ...)` |
+| Quotes of a post | `get_post_quotes(post_id_or_url, ...)` | `iter_post_quotes(post_id_or_url, ...)` | `save_post_quotes(post_id_or_url, output_dir, ...)` |
+| Account repost activity | `get_user_reposts(handle, ...)` | `iter_user_reposts(handle, ...)` | `save_user_reposts(handle, output_dir, ...)` |
+| Followers / following | `get_followers(user_id, ...)` / `get_following(user_id, ...)` | `iter_followers(user_id, ...)` / `iter_following(user_id, ...)` | `save_followers(user_id, output_dir, ...)` / `save_following(user_id, output_dir, ...)` |
+| Reposter user list | `get_post_reposters(post_id_or_url, ...)` | `iter_post_reposters(post_id_or_url, ...)` | `save_post_reposters(post_id_or_url, output_dir, ...)` |
 | Main posts + replies | — | — | `save_user_activity(handle, output_dir, ...)` |
 | Multiple accounts | — | — | `save_users_activity(handles, output_dir, ...)` |
 
-- `get_*` returns a `Profile`, `Post`, or `list[Post]`; it does not write a dataset.
+- `get_*` returns a `Profile`, `Post`, `list[Post]`, or network `list[Profile]`; it does not write a dataset.
 - `iter_*` returns an iterator. Requests happen as it is consumed.
-- `save_*` writes the standard dataset and returns `CollectionResult` with `output_dir`, `post_count`, `pages_fetched`, `complete`, and `reason`. `post_count` counts **all collected observations**, including replies; use the output manifest for separate table counts.
+- `save_*` returns `CollectionResult` with `output_dir`, `post_count`, `pages_fetched`, `complete`, and `reason`. For post datasets `post_count` counts unique records, including replies; for network datasets this compatibility field counts observed users. Consult each manifest for table counts and coverage.
 - Parameters: `handle` for one account, `handles` for several, `keyword` for search, `output_dir` for saved datasets, and `since`/`until` for dates. Options are keyword-only.
 - Dates are UTC: `since` inclusive, `until` exclusive. `max_pages` and `limit` must be positive integers or `None`. Saved timeline/mirror `limit` is a page-boundary stopping threshold, so a final page can exceed it; iterator limits are exact.
 
-See [API reference](docs/api.md) for signatures, return types and compatibility names.
+See [API reference](https://github.com/haochengw372-hash/pyXcom/blob/main/docs/api.md) for signatures, return types and compatibility names.
 
 ### Search one account
 
@@ -92,6 +99,40 @@ with XClient(profile="Default") as client:
 
 Direct account search scans main posts and authored replies, then matches a literal word/phrase without case sensitivity. Matching posts do not represent deduplicated real-world events. Cross-user search requires an explicit `mirror_base`; it sends search terms to that mirror and retrieves discovered post details from X. No mirror is enabled by default.
 
+### Native search, quotes and repost activity
+
+`*_search_query` submits X search syntax to the native Latest search endpoint; it does not require or use a mirror. `*_search_posts` retains its account/literal behavior above. Use a separate output directory for each query or source:
+
+```python
+with XClient(profile="Default") as client:
+    result = client.save_search_query(
+        '("usage reset" OR "rate limit reset") (OpenAI OR Codex)',
+        "output/reset_search", since="2025-09-29", until="2026-09-30",
+        max_pages=2, limit=100,
+    )
+    print(result.complete, result.reason)
+    client.save_post_quotes("1973931546550894681", "output/reset_quotes", max_pages=2)
+    client.save_user_reposts("thsottiaux", "output/tibo_reposts", max_pages=2)
+```
+
+Quote discovery uses native search followed by a check of `quoted_post_id`; zero matches do not establish zero quotes. Repost activity is collected from visible account timeline wrappers, preserving the repost ID and its action timestamp separately from the original post ID/time. A flattened timeline that only returns originals cannot establish action times and reports `repost_activity_unavailable` with an incomplete result. It is not a complete historical repost archive. Search sorting, visibility and pagination remain controlled by X.
+
+### Follow networks and reposter lists
+
+Resolve a handle with `get_user(handle).id`, then pass that stable numeric ID to follower/following methods:
+
+```python
+with XClient(profile="Default") as client:
+    user_id = client.get_user("thsottiaux").id
+    client.save_followers(user_id, "output/tibo_network", max_pages=2, snapshot_id="wave-1")
+    client.save_following(user_id, "output/tibo_network", max_pages=2, snapshot_id="wave-1")
+    client.save_post_reposters("1973931546550894681", "output/reset_reposters", max_pages=2)
+```
+
+Follow edges point from follower to followed account. `follow_edges.csv`, `reposters.csv`, `user_snapshots.csv` and `network_manifest.json` preserve observed relationships, user profiles and coverage per snapshot. A reposter list identifies who visibly reposted a post, but its `action_time_utc` is unknown. Use account repost activity when the wrapper supplies an action time; never substitute the original post time.
+
+Network saves resume a recoverable unfinished snapshot. Once completed or stopped by a terminal bad cursor, calling again without `snapshot_id` creates a new observation; explicitly reusing a completed ID returns that saved snapshot. Memory/iterator methods raise endpoint, parse or rate-limit errors and retain coverage in `last_network_collection`; hitting a budget returns the observed prefix. Saves return an incomplete result and preserve checkpoints on source failures. Network and post collections require separate output directories to protect their different `users.csv` exports. Follow observations cannot establish when a relationship began or reconstruct a historical follow network. Keep collection dates in the analysis. See [examples/research_networks.py](examples/research_networks.py) for a small collection recipe; importing it does not collect data.
+
 ### Collect several accounts
 
 ```python
@@ -106,7 +147,7 @@ with XClient(profile="Default") as client:
 
 Repeat to resume. `rounds=0, wait_on_rate_limit=True` keeps running and waits for rate windows when needed. `progress=callback` receives status dictionaries.
 
-## Collect a main post and nested comments
+## Collect a post or reply and nested comments
 
 ```python
 with XClient(profile="Default") as client:
@@ -126,7 +167,7 @@ pyxcom post-comments 1973931546550894681 --profile Default \
   --output-dir output/tibo_conversation
 ```
 
-The root belongs in `posts.csv`, replies from **all visible authors** in `comments.csv`, and author profiles in `users.csv`. The root must be a main post. Level 1 replies directly to it; level 2 replies to those comments. Quoted content, unrelated recommendations and comments beyond the requested depth are excluded. Profile details absent from the response remain unavailable.
+The seed can be a main post or a reply. Level 1 replies directly to that seed; level 2 replies to those comments. Main seeds belong in `posts.csv`; reply seeds remain replies in `comments.csv`. `seed_post_id` and `seed_relative_depth` describe this branch; `root_post_id`, `parent_post_id` and `depth` retain actual conversation relationships and do not relabel a reply as a platform root. Ancestors returned as context are stored in `context_posts.csv`, not counted as collected descendants. `since`/`until` filter descendant timestamps; traversal retains the ancestry needed to reach in-window descendants. Quoted content, unrelated recommendations and comments beyond the requested depth are excluded. Profile details absent from the response remain unavailable.
 
 `max_comments` is an exact total comment cap, excluding the root. `max_pages` caps conversation requests per call. Repeat the same root/depth/output to resume saved branches and cursors; raise the comment cap to continue a capped dataset. Pending page records are retained, so stopping midway through a page does not skip them. Changing depth requires a separate dataset. Python accepts `None` for unlimited comment/page budgets; depth must be positive.
 
@@ -151,12 +192,16 @@ output/tibo_year/
 
 Replies to oneself remain comments. A quote within a reply retains its quote link in `comments.csv`. Direct replies are depth 1; replies to those are depth 2. Missing ancestry leaves `depth` empty with a `depth_status`; `parent_in_dataset` and `root_in_dataset` indicate observed relationships. Do not interpret missing parents as first-level replies.
 
-`user_replies` means replies **authored by the selected account**. Use `*_post_comments` to collect replies below a particular main post from all visible authors, including nested replies. Neither endpoint establishes full conversation coverage.
+`user_replies` means replies **authored by the selected account**. Use `*_post_comments` to collect replies below a particular post or reply from all visible authors, including nested replies. Neither endpoint establishes full conversation coverage.
 
 CSV conventions: UTF-8 with BOM, snake_case columns, string IDs (import as text in Excel), JSON arrays for list fields, empty values for unavailable scalars, UTC timestamp fields. Users without a saved profile have `profile_available=false`. Engagement counts are snapshots; unknown values are not replaced with zero.
 
 If source records include pure reposts, an additional `interactions.csv` preserves them. The current authored-account collector does not establish complete repost activity. `manifest.json` records schema/layout versions, table counts, collection scope, relationship coverage and hashes. `.pyxcom/` must be retained for resuming; the three CSV files can be shared independently for analysis.
 
+Saved collection pages also retain platform response JSON under `.pyxcom/raw/` (network pages under their snapshot state) and query provenance in `.pyxcom/collection_log.jsonl`. Repeated post observations keep returned engagement metrics in `metric_snapshots.csv`, with source observations in `.pyxcom/observations.jsonl` and metric history in `.pyxcom/metric_snapshots.jsonl`. Unreturned counts remain unavailable. These are observations at retrieval time, not historical metrics at publication. Credentials are not included in the response archive.
+
+When source records contain reply, quote or repost relationships, `post_edges.csv` exports their directions, source/target post and user IDs, action and observation times, and target availability/resolution flags. An unavailable target is retained; its user ID stays empty if unresolved. Known target authors can appear in `users.csv` as stubs with `profile_available=false`. Context-only records do not create source edges. Relationship, context and metric exports appear when observations provide the corresponding data.
+
 ## Command line
 
 CLI task names correspond to the Python methods:
@@ -171,10 +216,19 @@ pyxcom search-posts reset --handle thsottiaux --profile Default \
   --since 2025-09-29 --until 2026-09-30 --output-dir output/tibo_reset
 pyxcom users-activity --handles OpenAI AnthropicAI --profile Default \
   --since 2025-09-29 --until 2026-09-30 --output-dir output/ai_year
+pyxcom search-query '("usage reset" OR "rate limit reset") Codex' --profile Default \
+  --since 2025-09-29 --until 2026-09-30 --max-pages 2 --output-dir output/reset_search
+pyxcom post-quotes 1973931546550894681 --max-pages 2 --output-dir output/reset_quotes
+pyxcom user-reposts thsottiaux --max-pages 2 --output-dir output/tibo_reposts
+pyxcom user-followers USER_ID --max-pages 2 --snapshot-id wave-1 --output-dir output/network
+pyxcom user-following USER_ID --max-pages 2 --snapshot-id wave-1 --output-dir output/network
+pyxcom post-reposters 1973931546550894681 --max-pages 2 --output-dir output/reposters
 ```
 
 Add `--proxy URL` if your network requires one. Use `pyxcom COMMAND --help` for task options. Individual `user`, `post`, and `raw-post` commands accept `--output FILE` for a JSON file.
 
+`post-comments --max-comments all --max-pages all` removes comment/page caps; `--max-depth` still requires a positive integer (default 2). New discovery/network commands also accept `all` for `--max-pages` and `--limit`. Unlimited budgets do not remove visibility or rate limits.
+
 ## Export, migration and verification
 
 These operations work offline, without browser credentials:
@@ -208,17 +262,17 @@ X endpoints may change or impose limits. pyXcom discovers current GraphQL query
 
 ## License
 
-pyXcom is released under the [MIT License](LICENSE). Copyright © 2026 Haocheng Wang.
+pyXcom is released under the [MIT License](https://github.com/haochengw372-hash/pyXcom/blob/main/LICENSE). Copyright © 2026 Haocheng Wang.
 
 ## Citation
 
 **Author:** Haocheng Wang, Communication University of China.
 
-If you use pyXcom in a paper, thesis, dataset, or other research output, please cite the software and report the version used. GitHub's **Cite this repository** menu reads the machine-readable [CITATION.cff](CITATION.cff).
+If you use pyXcom in a paper, thesis, dataset, or other research output, please cite the software and report the version used. GitHub's **Cite this repository** menu reads the machine-readable [CITATION.cff](https://github.com/haochengw372-hash/pyXcom/blob/main/CITATION.cff).
 
 **Suggested reference**
 
-Wang, H. (2026). *pyXcom: Structured and auditable X data collection for communication research* (Version 0.6.0) [Computer software]. https://github.com/haochengw372-hash/pyXcom
+Wang, H. (2026). *pyXcom: Structured and auditable X data collection for communication research* (Version 0.7.0) [Computer software]. https://github.com/haochengw372-hash/pyXcom
 
 **BibTeX**
 
@@ -227,9 +281,15 @@ Wang, H. (2026). *pyXcom: Structured and auditable X data collection for communi
   author  = {Wang, Haocheng},
   title   = {{pyXcom}: Structured and Auditable X Data Collection for Communication Research},
   year    = {2026},
-  version = {0.6.0},
+  version = {0.7.0},
   url     = {https://github.com/haochengw372-hash/pyXcom}
 }
 ```
 
 For reproducible reporting, also describe the collection dates, account or keyword scope, comment-depth limits, package version, and coverage/stop reasons recorded in the output manifest. No DOI or published-paper citation is currently assigned; this reference cites the software itself. Citation is appreciated and does not add a condition to the MIT license.
+
+For longitudinal profile comparisons use `user_snapshots.csv`; `users.csv` consolidates known profile fields. Canonical post tables keep the first saved observation for each ID; later returned text and metrics remain in observation archives and metric snapshots.
+
+Offline development checks: `PYTHONPATH=src python -m pytest -q` and `ruff check src tests examples`. The source distribution includes tests; testing tools are development-only, not runtime dependencies.
+
+Live validation: current X SearchTimeline/Followers use POST read queries; Retweeters is discovered from public lazy-loaded TweetActivity assets. If a search source returns posts outside explicit dates, they are excluded locally and `manifest.json.date_scope` records the mismatch. An empty bounded result with this warning does not establish historical absence. Native reposts generate only repost edges; embedded quote/reply context is not attributed as another action by the reposter.

pyxcom-0.7.0/docs/api.md

@@ -0,0 +1,172 @@
+# Public API — pyXcom 0.7.0
+
+The native discovery and network methods have passed offline tests and bounded authenticated live endpoint tests. These tests verify the exercised acquisition paths, not complete historical or network coverage.
+
+Import `XClient`, `Profile`, `Post`, `CollectionResult`, `PyXcomError`, `AuthenticationError`, `APIError`, and `RateLimitError` from `pyxcom`.
+
+## Client
+
+```python
+XClient(*, browser="chrome", profile=None, cookie_db=None, proxy=None,
+        mirror_base=None, delay=1.0, timeout=30)
+```
+
+Use as a context manager to close connections. `profile` selects the local browser profile; it is not an X username. Manually sign in first. The client never launches a browser.
+
+## Lookup methods
+
+```python
+get_user(handle: str) -> Profile
+get_post(post_id_or_url: str) -> Post
+get_posts(post_ids: list[str]) -> list[Post]
+get_raw_post(post_id_or_url: str) -> dict
+get_raw_user_timeline(handle: str, *, timeline="posts", cursor=None) -> dict
+```
+
+`get_posts` can return fewer items when X does not return every requested ID. Raw methods return the platform response for advanced use and have no stable field schema.
+
+## Account content
+
+```python
+iter_user_posts(handle, *, timeline="posts", since=None, until=None,
+                max_pages=None, limit=None) -> Iterator[Post]
+get_user_posts(handle, *, timeline="posts", since=None, until=None,
+               max_pages=None, limit=None) -> list[Post]
+save_user_posts(handle, output_dir, *, timeline="posts", since=None, until=None,
+                max_pages=None, limit=None) -> CollectionResult
+
+iter_user_replies(handle, *, since=None, until=None,
+                  max_pages=None, limit=None) -> Iterator[Post]
+get_user_replies(handle, *, since=None, until=None,
+                 max_pages=None, limit=None) -> list[Post]
+save_user_replies(handle, output_dir, *, since=None, until=None,
+                  max_pages=None, limit=None) -> CollectionResult
+
+save_user_activity(handle, output_dir, *, since=None, until=None,
+                   max_pages=None) -> CollectionResult
+save_users_activity(handles, output_dir, *, since, until, pages_per_round=5,
+                    rounds=1, wait_on_rate_limit=False, progress=None) -> CollectionResult
+```
+
+`handle: str`; `handles: list[str]`; `output_dir: str | Path`. Dates are `str | None` in YYYY-MM-DD UTC. `max_pages` and `limit` are positive `int | None`. `progress` is an optional callable taking a dictionary. `get`/`iter` start a fresh traversal; `save` resumes the same dataset query. `limit` on a saved timeline/mirror search is checked after a page; it may overshoot. Account activity and multi-account saves intentionally have no total-record `limit`.
+
+`user_posts` defaults to original/quote main posts. `user_replies` returns authored replies, including self-replies. Neither is an API to retrieve a main post's entire received comment tree. `timeline` is a compatibility option for selecting the source stream.
+
+## Comments received by a post or reply
+
+```python
+iter_post_comments(post_id_or_url: str, *, max_depth: int = 2,
+                   max_comments: int | None = 100,
+                   max_pages: int | None = 20,
+                   since: str | None = None, until: str | None = None) -> Iterator[Post]
+get_post_comments(post_id_or_url: str, *, max_depth: int = 2,
+                  max_comments: int | None = 100,
+                  max_pages: int | None = 20,
+                  since: str | None = None, until: str | None = None) -> list[Post]
+save_post_comments(post_id_or_url: str, output_dir: str | Path, *,
+                   max_depth: int = 2, max_comments: int | None = 100,
+                   max_pages: int | None = 20,
+                   since: str | None = None, until: str | None = None) -> CollectionResult
+```
+
+These methods expand descendants of the seed from all visible authors, including when the seed itself is a reply. They differ from `*_user_replies`, which fetch only one account's authored replies. `get` and `iter` return descendants only; `save` preserves the seed in its proper main/comment table, profiles in `users.csv`, comments in `comments.csv`, and private traversal state for resume. Ancestor context is retained in `context_posts.csv` and is not counted as descendants.
+
+Traversal depth is relative to the seed (direct replies=1), recorded as `seed_post_id` and `seed_relative_depth`. Actual platform `root_post_id`, `parent_post_id` and `depth` remain separate; missing ancestors can leave actual `depth` unresolved. UTC `since`/`until` filter returned descendants while preserving traversal context. Comment cap is exact and cumulative per dataset; page cap applies to each call. A rate-limited save preserves its queue with `complete=false`; memory/iterator methods raise `RateLimitError`. Network/parse failures save a partial checkpoint and raise. Repeating a save with larger comment/page budgets resumes it; changing seed/depth/date scope requires a separate dataset. Completion only means visible traversal within the requested depth ended, not an exhaustive historical thread. CLI comment/page caps accept `all` for Python `None`; depth must remain a positive integer.
+
+## Keyword search
+
+```python
+iter_search_posts(keyword, *, handle=None, since=None, until=None,
+                  max_pages=None, limit=None) -> Iterator[Post]
+get_search_posts(keyword, *, handle=None, since=None, until=None,
+                 max_pages=None, limit=None) -> list[Post]
+save_search_posts(keyword, output_dir, *, handle=None, since=None, until=None,
+                  max_pages=None, limit=None) -> CollectionResult
+```
+
+`keyword: str`, `handle: str | None`. Direct search requires a handle and matches literal text without case sensitivity across that account's main posts and replies. Cross-account discovery requires constructor `mirror_base`. `max_pages` applies per stream for direct search.
+
+## Native discovery
+
+```python
+iter_search_query(query, *, since=None, until=None, max_pages=None, limit=None) -> Iterator[Post]
+get_search_query(query, *, since=None, until=None, max_pages=None, limit=None) -> list[Post]
+save_search_query(query, output_dir, *, since=None, until=None,
+                  max_pages=None, limit=None) -> CollectionResult
+
+iter_post_quotes(post_id_or_url, *, since=None, until=None,
+                 max_pages=None, limit=None) -> Iterator[Post]
+get_post_quotes(post_id_or_url, *, since=None, until=None,
+                max_pages=None, limit=None) -> list[Post]
+save_post_quotes(post_id_or_url, output_dir, *, since=None, until=None,
+                 max_pages=None, limit=None) -> CollectionResult
+
+iter_user_reposts(handle, *, since=None, until=None,
+                  max_pages=None, limit=None) -> Iterator[Post]
+get_user_reposts(handle, *, since=None, until=None,
+                 max_pages=None, limit=None) -> list[Post]
+save_user_reposts(handle, output_dir, *, since=None, until=None,
+                  max_pages=None, limit=None) -> CollectionResult
+```
+
+All options are explicit and keyword-only. Dates are inclusive/exclusive UTC dates; budgets are positive integers or `None`. Native search uses X Latest, preserves query operators, and does not use the mirror. `*_post_quotes` discovers candidates using `quoted_tweet_id:` and verifies the returned target ID. Its result is a visible search sample, not the complete quote population. `*_user_reposts` parses visible repost wrappers in the account timeline, preserving the action ID/time and original target ID/time; it cannot infer missing reposts. If X returns flattened originals instead of wrappers, action times cannot be recovered; the save reports incomplete `repost_activity_unavailable` rather than substituting original publication times.
+
+Saved discovery queries retain original/effective query provenance, raw response pages and resumable cursors. Save scopes cannot be changed within the same directory. Use `complete` and `reason` to report caps, source end, rate limits or stalled pagination. A visible source end does not establish historical exhaustiveness.
+
+## Network lists and snapshots
+
+```python
+iter_followers(user_id, *, max_pages=None, limit=None) -> Iterator[Profile]
+get_followers(user_id, *, max_pages=None, limit=None) -> list[Profile]
+save_followers(user_id, output_dir, *, max_pages=None, limit=None,
+               snapshot_id=None) -> CollectionResult
+
+iter_following(user_id, *, max_pages=None, limit=None) -> Iterator[Profile]
+get_following(user_id, *, max_pages=None, limit=None) -> list[Profile]
+save_following(user_id, output_dir, *, max_pages=None, limit=None,
+               snapshot_id=None) -> CollectionResult
+
+iter_post_reposters(post_id_or_url, *, max_pages=None, limit=None) -> Iterator[Profile]
+get_post_reposters(post_id_or_url, *, max_pages=None, limit=None) -> list[Profile]
+save_post_reposters(post_id_or_url, output_dir, *, max_pages=None, limit=None,
+                    snapshot_id=None) -> CollectionResult
+```
+
+`user_id` is a numeric ID string, resolved with `get_user(handle).id`. Network methods have no historical date filter. Saves write `follow_edges.csv` or `reposters.csv`, `users.csv`, `user_snapshots.csv`, and `network_manifest.json`. Network and post collections must use separate output directories to protect their different table layouts. `CollectionResult.post_count` is the observed user count for these methods. Follow edges point follower → followed account. Reposter edges point reposter → original author with `target_post_id`; `action_time_utc` remains unknown because the user list does not return repost activity times. Discovery/network CLI budgets accept `all` for Python `None`.
+
+`snapshot_id` accepts 1–128 letters, digits, underscores or hyphens. Without it, a save resumes a recoverable unfinished snapshot for that source, or creates a new snapshot after completion or a terminal bad cursor. Reusing an explicit completed ID returns that observation. History and raw pages remain under `.pyxcom/networks/`; the network manifest records completion and stop reasons per snapshot. `get`/`iter` raise `APIError`, `ParseError` or `RateLimitError` on source failures and set `last_network_collection`; budget stops return the prefix. Saves return incomplete status with resumable checkpoints for recoverable failures. Snapshot time is the time of observation, not relationship creation time. Present-day follower/following lists cannot backfill historical follow networks.
+
+## Observation archives
+
+Saved post collections retain page responses under `.pyxcom/raw/` and collection provenance under `.pyxcom/collection_log.jsonl`; network responses are kept under each snapshot state. Re-observing a post preserves returned engagement-count history in `metric_snapshots.csv` and `.pyxcom/metric_snapshots.jsonl`, with observations retained in `.pyxcom/observations.jsonl`, while public post tables remain deduplicated by ID. Unknown counts remain unavailable, not zero. A repeated observation does not reconstruct an earlier historical metric. Raw archives contain platform payloads, not request cookies or authorization headers.
+
+`post_edges.csv` exports observed reply, quote and repost edges when present. Columns are `source_post_id`, `target_post_id`, `source_user_id`, `target_user_id`, `edge_type`, `action_time_utc`, `observed_at_utc`, `target_post_available`, `target_author_resolved` and `observation_role`. Edges point from the acting user/post to the target. Unavailable targets remain in the table; unresolved target user IDs stay empty with `target_author_resolved=false`. Known but otherwise unobserved target authors have stub user rows (`profile_available=false`). Context-only records do not generate source edges. Relationship, context and metric exports are conditional on corresponding observations being present.
+
+## Offline dataset functions
+
+```python
+export_tables(output_dir: str | Path) -> dict
+validate_tables(output_dir: str | Path) -> dict
+validate_collection(output_dir: str | Path) -> dict
+validate_network_collection(output_dir: str | Path) -> dict
+finalize_collection(output_dir: str | Path) -> CollectionResult
+schema_summary(output_dir: str | Path) -> dict
+write_schema_report(output_dir: str | Path) -> dict
+apply_role_schema(output_dir: str | Path) -> dict
+```
+
+Export/finalization migrate legacy files with backups and regenerate public tables. Validation never fetches X data. `validate_collection` checks internal observations and public tables; `validate_tables` checks normalized relationships, source/file hashes and counts. Reports describe missing ancestry explicitly. `schema_summary` describes the internal standard Post/Profile fields, not only the normalized table columns.
+
+Post export/finalization/schema helpers operate on post datasets. `validate_collection` automatically recognizes network datasets; `validate_network_collection` explicitly verifies network snapshot hashes, directions and provenance. To resume or regenerate network exports, repeat the corresponding network `save_*` call.
+
+## Backward-compatible names
+
+| Previous API | Preferred API |
+| --- | --- |
+| `iter_search(..., user=...)` | `iter_search_posts(..., handle=...)` |
+| `get_search(..., user=...)` | `get_search_posts(..., handle=...)` |
+| `save_search(..., user=...)` | `save_search_posts(..., handle=...)` |
+| `save_accounts(...)` | `save_users_activity(...)` |
+| `*_user_posts(..., timeline="replies")` | `*_user_replies(...)` |
+
+No removal is scheduled. These names call the same implementation; no separate authentication flow or duplicate scraper is introduced.

{pyxcom-0.6.0 → pyxcom-0.7.0}/docs/publishing.md

@@ -36,18 +36,18 @@ production releases.
    `twine check`, installs the wheel into a clean virtual environment, runs the
    unit tests, and checks the CLI. The separate publish job downloads those exact
    artifacts and uploads them with an OIDC identity.
-4. Install the TestPyPI release in a new environment. For version `0.6.0`:
+4. Install the TestPyPI release in a new environment. For version `0.7.0`:
 
    ```bash
-   python3 -m venv /tmp/pyxcom-testpypi-060
-   /tmp/pyxcom-testpypi-060/bin/python -m pip install \
+   python3 -m venv /tmp/pyxcom-testpypi-070
+   /tmp/pyxcom-testpypi-070/bin/python -m pip install \
      'browser-cookie3>=0.19,<1' 'beautifulsoup4>=4.12,<5' 'httpx>=0.27,<1'
-   /tmp/pyxcom-testpypi-060/bin/python -m pip install \
-     --index-url https://test.pypi.org/simple/ --no-deps 'pyXcom==0.6.0'
-   /tmp/pyxcom-testpypi-060/bin/python -m pip check
-   /tmp/pyxcom-testpypi-060/bin/python -c \
+   /tmp/pyxcom-testpypi-070/bin/python -m pip install \
+     --index-url https://test.pypi.org/simple/ --no-deps 'pyXcom==0.7.0'
+   /tmp/pyxcom-testpypi-070/bin/python -m pip check
+   /tmp/pyxcom-testpypi-070/bin/python -c \
      'from importlib.metadata import version; from pyxcom import XClient; print(version("pyXcom"))'
-   /tmp/pyxcom-testpypi-060/bin/pyxcom --help
+   /tmp/pyxcom-testpypi-070/bin/pyxcom --help
    ```
 
    Dependencies come from the normal PyPI index, while the package under test
@@ -65,11 +65,11 @@ After the production jobs succeed, verify the public index in another new virtua
 environment:
 
 ```bash
-python3 -m venv /tmp/pyxcom-pypi-060
-/tmp/pyxcom-pypi-060/bin/python -m pip install --index-url https://pypi.org/simple/ 'pyXcom==0.6.0'
-/tmp/pyxcom-pypi-060/bin/python -m pip check
-/tmp/pyxcom-pypi-060/bin/python -c 'from importlib.metadata import version; from pyxcom import XClient; print(version("pyXcom"))'
-/tmp/pyxcom-pypi-060/bin/pyxcom --help
+python3 -m venv /tmp/pyxcom-pypi-070
+/tmp/pyxcom-pypi-070/bin/python -m pip install --index-url https://pypi.org/simple/ 'pyXcom==0.7.0'
+/tmp/pyxcom-pypi-070/bin/python -m pip check
+/tmp/pyxcom-pypi-070/bin/python -c 'from importlib.metadata import version; from pyxcom import XClient; print(version("pyXcom"))'
+/tmp/pyxcom-pypi-070/bin/pyxcom --help
 ```
 
 The workflow is manually dispatched; pull requests and branch pushes do not