pyXcom 1.2.0__tar.gz → 1.3.0__tar.gz

{pyxcom-1.2.0 → pyxcom-1.3.0}/CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## 1.3.0 — 2026-10-13
+
+### Added
+
+- `XClient.save_comment_ancestry()` verifies missing comment parents through bounded,
+  deduplicated exact-ID lookups, using the source observer when bound.
+- Write a separate ancestry supplement with verified seed-relative/platform depths,
+  explicit unresolved statuses, context-only parents and source/output hashes.
+- Preserve original sampling roles, files, cursors and partial status; stop further
+  lookups on rate limits or request errors without rotating accounts.
+
+No new runtime dependencies or changes to existing comment collection defaults.
+
+## 1.2.1 — 2026-10-13
+
+### Fixed
+
+- Retain comment cursors paused by the empty-page guard and support explicit
+  `retry_stalled=True` through the saver, CLI and pool resume.
+- Recover older discarded cursors only from hash-verified archives bound to the
+  last committed focal/cursor request, including after an ordinary resume.
+- Keep historical pagination warnings while reporting still-stalled and
+  unrecoverable branches separately; bounded retries may remain partial.
+
 ## 1.2.0 — 2026-10-12
 
 ### Added

{pyxcom-1.2.0 → pyxcom-1.3.0}/CITATION.cff

@@ -6,8 +6,8 @@ authors:
   - family-names: Wang
     given-names: Haocheng
     affiliation: Communication University of China
-version: 1.2.0
-date-released: 2026-10-12
+version: 1.3.0
+date-released: 2026-10-13
 license: Apache-2.0
 repository-code: "https://github.com/haochengw372-hash/pyXcom"
 url: "https://github.com/haochengw372-hash/pyXcom"

{pyxcom-1.2.0 → pyxcom-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pyXcom
-Version: 1.2.0
+Version: 1.3.0
 Summary: Collect public X posts, search results, conversations, reposts, and observed follow networks with your browser session
 Project-URL: Homepage, https://github.com/haochengw372-hash/pyXcom
 Project-URL: Repository, https://github.com/haochengw372-hash/pyXcom
@@ -193,6 +193,24 @@ Repeat the same seed, depth, dates and output directory to resume. Raise the com
 
 For a main-post seed, the root enters `posts.csv` and descendants enter `comments.csv`. A reply seed remains in `comments.csv`. `seed_post_id` and `seed_relative_depth` describe the requested branch; platform root, parent and depth fields describe the retained ancestry. Unknown ancestry stays unresolved. `get_post_comments()` and `iter_post_comments()` return descendants without writing a dataset.
 
+When a saved tree contains pending replies with missing parents, you can verify
+their exact parent chains in a separate supplement:
+
+```python
+with XClient(profile="Default") as client:
+    report = client.save_comment_ancestry(
+        "data/conversation", "data/conversation_ancestry", lookup_limit=20,
+    )
+```
+
+This reads a verified `post_comments` collection and looks up missing parent IDs
+using the same observer when the source has a pool identity binding. It writes
+`resolution.csv`, context-only `ancestor_posts.jsonl`, the original pending
+records, raw responses and a hash manifest into a new directory. Source files,
+sampling roles and completeness remain unchanged. The limit counts lookup calls;
+identity checks and transport retries are additional HTTP attempts. See
+[comment ancestry](docs/comment-ancestry.md) for field definitions and unresolved cases.
+
 ### Collect several accounts
 
 One client can collect several target accounts using the same authenticated browser session:
@@ -411,7 +429,7 @@ These are offline development checks. They do not collect new X data.
 
 Created by **Haocheng Wang, Communication University of China**.
 
-[PykTok](https://github.com/dfreelon/pyktok) inspired the approachable get/save naming and task-by-task examples. pyXcom is an independent implementation and does not depend on PykTok or twikit.
+[PykTok](https://github.com/dfreelon/pyktok) inspired the approachable get/save naming and task-by-task examples. 
 
 ## Citation
 

{pyxcom-1.2.0 → pyxcom-1.3.0}/README.md

@@ -174,6 +174,24 @@ Repeat the same seed, depth, dates and output directory to resume. Raise the com
 
 For a main-post seed, the root enters `posts.csv` and descendants enter `comments.csv`. A reply seed remains in `comments.csv`. `seed_post_id` and `seed_relative_depth` describe the requested branch; platform root, parent and depth fields describe the retained ancestry. Unknown ancestry stays unresolved. `get_post_comments()` and `iter_post_comments()` return descendants without writing a dataset.
 
+When a saved tree contains pending replies with missing parents, you can verify
+their exact parent chains in a separate supplement:
+
+```python
+with XClient(profile="Default") as client:
+    report = client.save_comment_ancestry(
+        "data/conversation", "data/conversation_ancestry", lookup_limit=20,
+    )
+```
+
+This reads a verified `post_comments` collection and looks up missing parent IDs
+using the same observer when the source has a pool identity binding. It writes
+`resolution.csv`, context-only `ancestor_posts.jsonl`, the original pending
+records, raw responses and a hash manifest into a new directory. Source files,
+sampling roles and completeness remain unchanged. The limit counts lookup calls;
+identity checks and transport retries are additional HTTP attempts. See
+[comment ancestry](docs/comment-ancestry.md) for field definitions and unresolved cases.
+
 ### Collect several accounts
 
 One client can collect several target accounts using the same authenticated browser session:
@@ -392,7 +410,7 @@ These are offline development checks. They do not collect new X data.
 
 Created by **Haocheng Wang, Communication University of China**.
 
-[PykTok](https://github.com/dfreelon/pyktok) inspired the approachable get/save naming and task-by-task examples. pyXcom is an independent implementation and does not depend on PykTok or twikit.
+[PykTok](https://github.com/dfreelon/pyktok) inspired the approachable get/save naming and task-by-task examples. 
 
 ## Citation
 

{pyxcom-1.2.0 → pyxcom-1.3.0}/docs/api.md

@@ -1,4 +1,4 @@
-# Public API — pyXcom 1.2.0
+# Public API — pyXcom 1.2.1
 
 The native discovery and network methods have passed offline tests and bounded authenticated live endpoint tests. These tests verify the exercised acquisition paths, not complete historical or network coverage.
 
@@ -81,6 +81,42 @@ Traversal depth is relative to the seed (direct replies=1), recorded as `seed_po
 
 For offline validation of `post_comments`, UTC date scope applies to analysis descendants, while the seed and explicitly marked context can lie outside it. These retained records still participate in identifier, relationship and artifact-integrity checks. The exemption does not apply to other collection types; a descendant missing `observation_role` is treated as analysis for this date check.
 
+### Retry a paused comment branch
+
+`save_post_comments(..., retry_stalled=True)` explicitly requeues cursors saved
+by the empty-page guard. Ordinary resume keeps these branches paused. Repeated
+cursor warnings and unresolved ancestors are still reported as partial.
+
+For older collections, an empty-page cursor is recovered only from a hash-checked
+raw response whose focal ID and request cursor match the last committed request.
+Missing or mismatched evidence remains unresolved; no cursor or tree depth is guessed.
+
+```python
+result = client.save_post_comments(
+    post_id, output_dir, max_depth=3, max_comments=100,
+    max_pages=2, retry_stalled=True,
+)
+```
+
+### Verify missing comment ancestors
+
+```python
+save_comment_ancestry(source_dir: str | Path, output_dir: str | Path, *,
+                      lookup_limit: int = 20) -> dict
+```
+
+Reads a verified saved `post_comments` source and writes a new, separate ancestry
+supplement. Source scopes and sampling roles are immutable. Exact-parent lookups
+are bounded and deduplicated; known source observers must match the current login.
+`lookup_limit=0` makes no network calls. A resolved chain establishes descent from
+the requested seed; it does not establish exhaustive collection. See
+[comment ancestry](comment-ancestry.md) for outputs, depth definitions and failure handling.
+
+The manifest distinguishes queued requests, stalled requests, active warnings,
+and historical warnings. A retry can still return a partial result when the
+platform keeps returning empty pages. CLI: `pyxcom post-comments ID --output-dir
+DIR --retry-stalled`. Pool: `resume_job(job_id, retry_stalled=True)`.
+
 ## Keyword search
 
 ```python

pyxcom-1.3.0/docs/comment-ancestry.md

@@ -0,0 +1,94 @@
+# Verifying missing comment ancestors
+
+X can return a child reply while omitting its parent. A saved `post_comments`
+collection keeps such records in its private pending state, rather than assigning
+them a guessed depth. `XClient.save_comment_ancestry()` verifies those parent chains
+with bounded single-post lookups and writes a separate supplement.
+
+```python
+from pyxcom import XClient
+
+# Sign in manually before running. Select the original collection's observer.
+with XClient(profile="Default") as client:
+    report = client.save_comment_ancestry(
+        "data/conversation", "data/conversation_ancestry", lookup_limit=20,
+    )
+print(report["resolved_records"], report["unresolved_records"])
+```
+
+The source must pass retained-record and checkpoint integrity verification and
+must be a `post_comments` collection. Both scopes are locked; overlapping paths,
+existing destinations and symlink source contents are rejected. Source files,
+cursors, roles, counts and partial status remain unchanged.
+
+## Lookup rules
+
+Only a record's explicit `in_reply_to_id` is requested. Each missing ID is looked
+up once per invocation, including failed lookups, and shared ancestors reuse the
+result. The returned ID and conversation must agree with the source. A chain is
+resolved only when it reaches the exact requested seed, without a cycle. Other
+branches of a reply-seeded conversation are not admitted.
+
+Before the first lookup, the client verifies its current identity. If the source
+has a pool observer binding, that identity must match; a mismatch raises before
+parent content is requested. Older unbound sources remain explicitly unbound in
+the manifest. No observer is inferred retroactively and no account rotation occurs.
+
+`lookup_limit` is a nonnegative integer limiting single-post lookup calls, not
+HTTP attempts. Identity checks, route discovery and the existing transport's
+retries can add HTTP attempts. Zero makes no network calls. Calls respect client
+delay; a rate limit or request error stops further parent lookups in that run.
+
+## Outputs
+
+| File | Contents |
+| --- | --- |
+| `resolution.csv` | One row per original pending record, verified links, depth and scope flag |
+| `ancestor_posts.jsonl` | Newly fetched parents, all `observation_role="context"` |
+| `pending_posts.jsonl` | Copy of the original pending records |
+| `raw/<parent_id>.json` | Credential-free successful lookup payloads, including unavailable results |
+| `manifest.json` | Source and supplement hashes, caps, counts, observer checks and source completeness |
+
+`seed_post_id` is the requested collection seed; `root_post_id` is the platform
+conversation root. `chain_ids` lists verified retained IDs from the pending record
+back toward the seed; an unresolved chain does not include the unreturned parent.
+`seed_relative_depth` counts links to the seed and is blank until that chain is
+resolved. `depth` counts links to the platform root and also requires retained
+evidence between a reply seed and that root; otherwise it stays blank. This method
+does not make extra requests above a reply seed just to fill platform depth.
+
+`within_requested_bounds` checks the pending record's verified seed-relative depth
+and the original UTC date bounds (inclusive `since`, exclusive `until`). It is blank
+for unresolved chains. A true flag is not automatic sample admission: supplement
+records never alter the original `posts.csv` or `comments.csv`.
+
+Statuses include `resolved`, `lookup_budget`, `parent_not_returned`, `rate_limited`,
+`request_error`, `identity_conflict`, `conversation_conflict`, `outside_seed_branch`,
+`cycle` and `protected_parent`. A successful response without a parseable parent
+is `parent_not_returned`; that does not establish deletion, protection or absence
+on the platform. Explicitly protected parent content is not retained or archived.
+
+Each invocation uses a new output directory. There is no supplement resume or
+automatic merging into the source. Output is published after traversal; a process
+termination before publication can lose that run's in-memory supplement. Retain
+the original source and compare dated supplements explicitly. The manifest's
+`collection_complete` and `source_collection_reason` copy the original collection's
+status, even if every pending chain has now been resolved.
+
+## Recorded validation
+
+On October 13, 2026, two retained personal-agent discussion trees contained three
+pending replies. With their original authenticated observers, three parent lookup
+calls resolved two chains in the first tree; the second tree's parent did not
+return a parseable record. Source file inventories and hashes were unchanged.
+
+The resolved replies had depths two and four. The source depth bound was three,
+so only the depth-two record passed the depth/date scope flag. This illustrates why
+resolving ancestry and admitting a reply to a research sample are separate steps.
+Both original collections remained partial. These are recorded results, not a
+coverage benchmark; later requests may return different content.
+
+Eighteen focused offline regression tests cover exact chains, shared parents,
+reply-seed boundaries, cycles, budgets, source integrity, identity mismatch,
+malformed responses and protected content. The complete suite contained 454
+passing tests at this release.

{pyxcom-1.2.0 → pyxcom-1.3.0}/docs/pool.md

@@ -83,7 +83,7 @@ X's quota implementation. Requests already in flight cannot be recalled.
 
 Reopen the same `state_dir` with the same login IDs and call `run` again. Re-adding
 an identical job is harmless; changing its query, scope, or observer is rejected.
-Use `resume_job(job_id)` for a paused job. Explicit discovery-stall retries use
+Use `resume_job(job_id)` for a paused job. Explicit discovery or comment empty-page retries use
 `resume_job(job_id, retry_stalled=True)` where the underlying saver supports it.
 
 Adopting existing output requires both a pinned `login_id` and

{pyxcom-1.2.0 → pyxcom-1.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pyXcom"
-version = "1.2.0"
+version = "1.3.0"
 description = "Collect public X posts, search results, conversations, reposts, and observed follow networks with your browser session"
 readme = "README.md"
 license = "Apache-2.0"

{pyxcom-1.2.0 → pyxcom-1.3.0}/src/pyxcom/__init__.py

@@ -50,4 +50,4 @@ __all__ = [
     "validate_collection",
     "validate_network_collection",
 ]
-__version__ = "1.2.0"
+__version__ = "1.3.0"

pyxcom-1.3.0/src/pyxcom/ancestry.py

@@ -0,0 +1,286 @@
+"""Copy-only, bounded verification of missing comment ancestors."""
+
+import csv
+import json
+import time
+from contextlib import ExitStack
+from dataclasses import replace
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from ._persistence import append_jsonl, atomic_json, read_jsonl
+from ._pool_locks import ScopeLock
+from .errors import IntegrityError, ParseError, PyXcomError, RateLimitError
+from .integrity import assess_recovery
+from .models import Post
+from .parse import single_post
+from .recovery import _files
+from .transport import now_utc
+
+if TYPE_CHECKING:
+    from .client import XClient
+
+
+def _directory(value: str | Path) -> Path:
+    path = Path(value).expanduser()
+    if path.is_symlink():
+        raise ValueError("Ancestry directory must not be a symbolic link")
+    return path.resolve()
+
+
+def _mapping(value) -> dict:
+    return value if isinstance(value, dict) else {}
+
+
+def _protected(payload: dict) -> bool:
+    result = _mapping(
+        _mapping(_mapping(payload.get("data")).get("tweetResult")).get("result")
+    )
+    tweet = _mapping(result.get("tweet", result))
+    user = _mapping(
+        _mapping(_mapping(tweet.get("core")).get("user_results")).get("result")
+    )
+    return any(
+        _mapping(user.get(key)).get("protected") is True
+        for key in ("privacy", "legacy")
+    )
+
+
+def save_comment_ancestry(
+    client: "XClient",
+    source_dir: str | Path,
+    output_dir: str | Path,
+    *,
+    lookup_limit: int = 20,
+) -> dict:
+    """Verify exact parent chains without admitting records to the original sample.
+
+    The cap counts single-post lookup calls, not transport retry attempts. A new
+    supplement is required for every run; source files are read-only throughout.
+    """
+    if (
+        isinstance(lookup_limit, bool)
+        or not isinstance(lookup_limit, int)
+        or lookup_limit < 0
+    ):
+        raise ValueError("lookup_limit must be a nonnegative integer")
+    source, target = _directory(source_dir), _directory(output_dir)
+    if source == target or source in target.parents or target in source.parents:
+        raise ValueError("Source and supplement scopes must not overlap")
+    with ExitStack() as stack:
+        for scope in sorted((source, target)):
+            stack.enter_context(ScopeLock(scope))
+        if target.exists():
+            raise FileExistsError("Ancestry supplement already exists")
+        assessment = assess_recovery(source)
+        if not assessment["allowed"] or assessment["record_integrity"] != "verified":
+            raise IntegrityError(
+                "Source collection did not pass integrity verification"
+            )
+        query = assessment["query"]
+        if query.get("kind") != "post_comments":
+            raise ValueError(
+                "Ancestry verification requires a post_comments collection"
+            )
+        state = json.loads((source / ".pyxcom/state.json").read_text(encoding="utf-8"))
+        conversation = state["conversation"]
+        pending = conversation.get("pending", {})
+        posts = {p["id"]: Post(**p) for p in read_jsonl(source / ".pyxcom/posts.jsonl")}
+        posts.update({key: Post(**value) for key, value in pending.items()})
+        root = str(query["root_post_id"])
+        conversation_id = str(conversation.get("conversation_id", root))
+        owner_path = source / ".pyxcom/pool-owner.json"
+        owner = (
+            json.loads(owner_path.read_text(encoding="utf-8"))
+            if owner_path.exists()
+            else {}
+        )
+        expected_observer = owner.get("observer_id")
+        verified_observer = None
+        fetched: dict[str, Post] = {}
+        failures: dict[str, str] = {}
+        raw_responses: dict[str, dict] = {}
+        calls = 0
+        stop_reason = None
+
+        def lookup(identifier: str) -> tuple[Post | None, str | None]:
+            nonlocal calls, stop_reason, verified_observer
+            if identifier in failures:
+                return None, failures[identifier]
+            if stop_reason:
+                return None, stop_reason
+            if calls >= lookup_limit:
+                return None, "lookup_budget"
+            if verified_observer is None:
+                verified_observer = str(client.check_login_identity())
+                if expected_observer is not None and verified_observer != str(
+                    expected_observer
+                ):
+                    raise ValueError(
+                        "Ancestry lookup observer differs from source observer"
+                    )
+            if calls:
+                time.sleep(client.delay)
+            calls += 1
+            payload = None
+            try:
+                payload = client.get_raw_post(identifier)
+                if _protected(payload):
+                    failures[identifier] = "protected_parent"
+                    return None, failures[identifier]
+                raw_responses[identifier] = payload
+                try:
+                    post = single_post(payload, captured_at_utc=now_utc())
+                except (ParseError, ValueError, TypeError, KeyError, AttributeError):
+                    failures[identifier] = "parent_not_returned"
+                    return None, failures[identifier]
+                if post.id != identifier:
+                    failures[identifier] = "identity_conflict"
+                elif post.conversation_id != conversation_id or post.reposted_post_id:
+                    failures[identifier] = "conversation_conflict"
+                else:
+                    post = replace(
+                        post, observation_role="context", discovery_source="x_ancestor"
+                    )
+                    posts[identifier] = fetched[identifier] = post
+                    return post, None
+            except ParseError:
+                stop_reason = failures[identifier] = "request_error"
+            except RateLimitError:
+                stop_reason = failures[identifier] = "rate_limited"
+            except PyXcomError:
+                stop_reason = failures[identifier] = "request_error"
+            return None, failures[identifier]
+
+        def platform_depth(seed_depth: int | None) -> int | None:
+            if seed_depth is None:
+                return None
+            current = posts[root]
+            seen = {root}
+            offset = 0
+            while current.id != conversation_id:
+                parent = posts.get(current.in_reply_to_id or "")
+                if (
+                    parent is None
+                    or parent.id in seen
+                    or parent.conversation_id != conversation_id
+                ):
+                    return None
+                seen.add(parent.id)
+                offset += 1
+                current = parent
+            return seed_depth + offset
+
+        rows = []
+        for identifier, value in pending.items():
+            post = posts[identifier]
+            chain = [identifier]
+            current = post
+            status = "resolved"
+            while current.id != root:
+                if (
+                    current.conversation_id != conversation_id
+                    or current.reposted_post_id
+                ):
+                    status = "conversation_conflict"
+                    break
+                parent_id = current.in_reply_to_id
+                if parent_id is None or (
+                    parent_id == conversation_id and conversation_id != root
+                ):
+                    status = "outside_seed_branch"
+                    break
+                if parent_id in chain:
+                    status = "cycle"
+                    break
+                parent = posts.get(parent_id)
+                if parent is None:
+                    parent, failure = lookup(parent_id)
+                    if parent is None:
+                        status = failure or "parent_not_returned"
+                        break
+                if parent.conversation_id != conversation_id or parent.reposted_post_id:
+                    status = "conversation_conflict"
+                    break
+                chain.append(parent_id)
+                current = parent
+            seed_depth = len(chain) - 1 if status == "resolved" else None
+            within = None
+            if seed_depth is not None:
+                day = post.created_at_utc[:10]
+                within = (
+                    seed_depth <= query["max_depth"]
+                    and (not query.get("since") or day >= query["since"])
+                    and (not query.get("until") or day < query["until"])
+                )
+            rows.append(
+                {
+                    "post_id": identifier,
+                    "root_post_id": conversation_id,
+                    "seed_post_id": root,
+                    "parent_post_id": value.get("in_reply_to_id"),
+                    "ancestry_status": status,
+                    "depth": platform_depth(seed_depth),
+                    "seed_relative_depth": seed_depth,
+                    "chain_ids": json.dumps(chain),
+                    "within_requested_bounds": within,
+                }
+            )
+        if _files(source) != assessment["input_sha256"]:
+            raise IntegrityError("Source changed during ancestry verification")
+        target.mkdir(parents=True, exist_ok=False)
+        (target / "raw").mkdir()
+        for name, records in (
+            ("ancestor_posts.jsonl", (p.to_dict() for p in fetched.values())),
+            ("pending_posts.jsonl", pending.values()),
+        ):
+            append_jsonl(target / name, records)
+        with (target / "resolution.csv").open(
+            "w", encoding="utf-8-sig", newline=""
+        ) as stream:
+            writer = csv.DictWriter(
+                stream,
+                fieldnames=[
+                    "post_id",
+                    "root_post_id",
+                    "seed_post_id",
+                    "parent_post_id",
+                    "ancestry_status",
+                    "depth",
+                    "seed_relative_depth",
+                    "chain_ids",
+                    "within_requested_bounds",
+                ],
+            )
+            writer.writeheader()
+            writer.writerows(rows)
+        for identifier, payload in raw_responses.items():
+            atomic_json(target / "raw" / f"{identifier}.json", payload)
+        counts = {
+            status: sum(r["ancestry_status"] == status for r in rows)
+            for status in sorted({r["ancestry_status"] for r in rows})
+        }
+        from . import __version__
+
+        report = {
+            "schema": "pyxcom.comment-ancestry/1",
+            "package_version": __version__,
+            "captured_at_utc": now_utc(),
+            "query": query,
+            "source_sha256": assessment["input_sha256"],
+            "lookup_limit": lookup_limit,
+            "lookup_calls": calls,
+            "stop_reason": stop_reason,
+            "identity_verified": verified_observer is not None,
+            "source_observer_bound": expected_observer is not None
+            and verified_observer is not None,
+            "status_counts": counts,
+            "resolved_records": counts.get("resolved", 0),
+            "unresolved_records": len(rows) - counts.get("resolved", 0),
+            "context_records": len(fetched),
+            "collection_complete": assessment["complete"],
+            "source_collection_reason": assessment["reason"],
+            "files_sha256": _files(target),
+        }
+        atomic_json(target / "manifest.json", report)
+        return report

{pyxcom-1.2.0 → pyxcom-1.3.0}/src/pyxcom/cli.py

@@ -251,6 +251,7 @@ def build_parser() -> argparse.ArgumentParser:
         help="Collect replies below a post or reply, including nested replies",
     )
     comments.add_argument("post_id_or_url")
+    comments.add_argument("--retry-stalled", action="store_true", help="Explicitly retry saved comment cursors paused by empty-page guard")
     comments.add_argument("--max-depth", type=_depth, default=2)
     comments.add_argument("--max-comments", type=_budget, default=100)
     comments.add_argument("--max-pages", type=_budget, default=20)
@@ -576,6 +577,7 @@ def main(argv: list[str] | None = None) -> int:
                     max_pages=args.max_pages,
                     since=args.since,
                     until=args.until,
+                    **({"retry_stalled": True} if args.retry_stalled else {}),
                 )
                 print(json.dumps(result.to_dict(), ensure_ascii=False))
             elif args.command in ("search-query", "post-quotes", "user-reposts"):

{pyxcom-1.2.0 → pyxcom-1.3.0}/src/pyxcom/client.py

@@ -957,6 +957,24 @@ class XClient(DiscoveryMixin, NetworkMixin):
             )
         )
 
+    def save_comment_ancestry(
+        self,
+        source_dir: str | Path,
+        output_dir: str | Path,
+        *,
+        lookup_limit: int = 20,
+    ) -> dict:
+        """Save a separate, bounded supplement verifying missing comment parents.
+
+        Original collection files and sampling roles are preserved. The cap counts
+        single-post lookup calls; transport retries can make more HTTP attempts.
+        """
+        from .ancestry import save_comment_ancestry
+
+        return save_comment_ancestry(
+            self, source_dir, output_dir, lookup_limit=lookup_limit
+        )
+
     @guarded_collection
     def save_post_comments(
         self,
@@ -968,9 +986,10 @@ class XClient(DiscoveryMixin, NetworkMixin):
         max_pages: int | None = 20,
         since: str | None = None,
         until: str | None = None,
+        retry_stalled: bool = False,
     ) -> CollectionResult:
         """Save a root post, its reply tree and observed author profiles; resume safely."""
-        from .comments import CommentTraversal
+        from .comments import CommentTraversal, restore_archived_stalls
 
         root_id = _post_id(post_id_or_url)
         # Validate bounds before creating or migrating output.
@@ -982,6 +1001,7 @@ class XClient(DiscoveryMixin, NetworkMixin):
             max_pages=max_pages,
             since=since,
             until=until,
+            retry_stalled=retry_stalled,
         )
         store = PostStore(
             output_dir,
@@ -994,6 +1014,8 @@ class XClient(DiscoveryMixin, NetworkMixin):
         )
         if store.complete:
             return store.finish(complete=True, reason=store.state["reason"])
+        if retry_stalled:
+            restore_archived_stalls(store)
         traversal = CommentTraversal(
             self,
             root_id,
@@ -1004,6 +1026,7 @@ class XClient(DiscoveryMixin, NetworkMixin):
             until=until,
             records=store._posts,
             state=store.state.setdefault("conversation", {}),
+            retry_stalled=retry_stalled,
         )
 
         def save_profiles():