pyXcom 1.2.1__tar.gz → 1.3.1__tar.gz

{pyxcom-1.2.1 → pyxcom-1.3.1}/CHANGELOG.md

@@ -1,5 +1,36 @@
 # Changelog
 
+## 1.3.1 — 2026-10-13
+
+### Fixed
+
+- Distinguish malformed or unparseable parent responses (`response_unparsed`)
+  from recognized empty results and tombstones (`parent_not_returned`).
+- Honor inherited source observer bindings and reject malformed markers by reusing
+  the package's ownership reader; check ownership again before publication.
+- Reject supplement destinations owned by another pool before network requests,
+  including inherited ownership, and check again before publication.
+- Publish ancestry output as one staged generation. Ordinary write failures leave
+  no partial final directory blocking retries.
+
+### Added
+
+- `blocked_post_id` identifies the actual point of failure along a parent chain,
+  while preserving the original `parent_post_id`.
+
+## 1.3.0 — 2026-10-13
+
+### Added
+
+- `XClient.save_comment_ancestry()` verifies missing comment parents through bounded,
+  deduplicated exact-ID lookups, using the source observer when bound.
+- Write a separate ancestry supplement with verified seed-relative/platform depths,
+  explicit unresolved statuses, context-only parents and source/output hashes.
+- Preserve original sampling roles, files, cursors and partial status; stop further
+  lookups on rate limits or request errors without rotating accounts.
+
+No new runtime dependencies or changes to existing comment collection defaults.
+
 ## 1.2.1 — 2026-10-13
 
 ### Fixed

{pyxcom-1.2.1 → pyxcom-1.3.1}/CITATION.cff

@@ -6,7 +6,7 @@ authors:
   - family-names: Wang
     given-names: Haocheng
     affiliation: Communication University of China
-version: 1.2.1
+version: 1.3.1
 date-released: 2026-10-13
 license: Apache-2.0
 repository-code: "https://github.com/haochengw372-hash/pyXcom"

{pyxcom-1.2.1 → pyxcom-1.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pyXcom
-Version: 1.2.1
+Version: 1.3.1
 Summary: Collect public X posts, search results, conversations, reposts, and observed follow networks with your browser session
 Project-URL: Homepage, https://github.com/haochengw372-hash/pyXcom
 Project-URL: Repository, https://github.com/haochengw372-hash/pyXcom
@@ -193,6 +193,24 @@ Repeat the same seed, depth, dates and output directory to resume. Raise the com
 
 For a main-post seed, the root enters `posts.csv` and descendants enter `comments.csv`. A reply seed remains in `comments.csv`. `seed_post_id` and `seed_relative_depth` describe the requested branch; platform root, parent and depth fields describe the retained ancestry. Unknown ancestry stays unresolved. `get_post_comments()` and `iter_post_comments()` return descendants without writing a dataset.
 
+When a saved tree contains pending replies with missing parents, you can verify
+their exact parent chains in a separate supplement:
+
+```python
+with XClient(profile="Default") as client:
+    report = client.save_comment_ancestry(
+        "data/conversation", "data/conversation_ancestry", lookup_limit=20,
+    )
+```
+
+This reads a verified `post_comments` collection and looks up missing parent IDs
+using the same observer when the source has a pool identity binding. It writes
+`resolution.csv`, context-only `ancestor_posts.jsonl`, the original pending
+records, raw responses and a hash manifest into a new directory. Source files,
+sampling roles and completeness remain unchanged. The limit counts lookup calls;
+identity checks and transport retries are additional HTTP attempts. See
+[comment ancestry](docs/comment-ancestry.md) for field definitions and unresolved cases.
+
 ### Collect several accounts
 
 One client can collect several target accounts using the same authenticated browser session:

{pyxcom-1.2.1 → pyxcom-1.3.1}/README.md

@@ -174,6 +174,24 @@ Repeat the same seed, depth, dates and output directory to resume. Raise the com
 
 For a main-post seed, the root enters `posts.csv` and descendants enter `comments.csv`. A reply seed remains in `comments.csv`. `seed_post_id` and `seed_relative_depth` describe the requested branch; platform root, parent and depth fields describe the retained ancestry. Unknown ancestry stays unresolved. `get_post_comments()` and `iter_post_comments()` return descendants without writing a dataset.
 
+When a saved tree contains pending replies with missing parents, you can verify
+their exact parent chains in a separate supplement:
+
+```python
+with XClient(profile="Default") as client:
+    report = client.save_comment_ancestry(
+        "data/conversation", "data/conversation_ancestry", lookup_limit=20,
+    )
+```
+
+This reads a verified `post_comments` collection and looks up missing parent IDs
+using the same observer when the source has a pool identity binding. It writes
+`resolution.csv`, context-only `ancestor_posts.jsonl`, the original pending
+records, raw responses and a hash manifest into a new directory. Source files,
+sampling roles and completeness remain unchanged. The limit counts lookup calls;
+identity checks and transport retries are additional HTTP attempts. See
+[comment ancestry](docs/comment-ancestry.md) for field definitions and unresolved cases.
+
 ### Collect several accounts
 
 One client can collect several target accounts using the same authenticated browser session:

{pyxcom-1.2.1 → pyxcom-1.3.1}/docs/api.md

@@ -98,6 +98,20 @@ result = client.save_post_comments(
 )
 ```
 
+### Verify missing comment ancestors
+
+```python
+save_comment_ancestry(source_dir: str | Path, output_dir: str | Path, *,
+                      lookup_limit: int = 20) -> dict
+```
+
+Reads a verified saved `post_comments` source and writes a new, separate ancestry
+supplement. Source scopes and sampling roles are immutable. Exact-parent lookups
+are bounded and deduplicated; known source observers must match the current login.
+`lookup_limit=0` makes no network calls. A resolved chain establishes descent from
+the requested seed; it does not establish exhaustive collection. See
+[comment ancestry](comment-ancestry.md) for outputs, depth definitions and failure handling.
+
 The manifest distinguishes queued requests, stalled requests, active warnings,
 and historical warnings. A retry can still return a partial result when the
 platform keeps returning empty pages. CLI: `pyxcom post-comments ID --output-dir

pyxcom-1.3.1/docs/comment-ancestry.md

@@ -0,0 +1,112 @@
+# Verifying missing comment ancestors
+
+X can return a child reply while omitting its parent. A saved `post_comments`
+collection keeps such records in its private pending state, rather than assigning
+them a guessed depth. `XClient.save_comment_ancestry()` verifies those parent chains
+with bounded single-post lookups and writes a separate supplement.
+
+```python
+from pyxcom import XClient
+
+# Sign in manually before running. Select the original collection's observer.
+with XClient(profile="Default") as client:
+    report = client.save_comment_ancestry(
+        "data/conversation", "data/conversation_ancestry", lookup_limit=20,
+    )
+print(report["resolved_records"], report["unresolved_records"])
+```
+
+The source must pass retained-record and checkpoint integrity verification and
+must be a `post_comments` collection. Both scopes are locked; overlapping paths,
+existing destinations and symlink source contents are rejected. Source files,
+cursors, roles, counts and partial status remain unchanged.
+Destinations inside another pool's owned scope are rejected before network requests
+and checked again before publication; an idle pool still owns its persisted scope.
+
+## Lookup rules
+
+Only a record's explicit `in_reply_to_id` is requested. Each missing ID is looked
+up once per invocation, including failed lookups, and shared ancestors reuse the
+result. The returned ID and conversation must agree with the source. A chain is
+resolved only when it reaches the exact requested seed, without a cycle. Other
+branches of a reply-seeded conversation are not admitted.
+
+Before the first lookup, the client verifies its current identity. If the source
+has a pool observer binding, including one inherited from an ancestor directory,
+that identity must match; a mismatch raises before parent content is requested.
+Malformed ownership markers are rejected before identity or parent requests, and
+the marker is checked again before publication. Older unbound sources remain explicitly unbound in
+the manifest. No observer is inferred retroactively and no account rotation occurs.
+
+`lookup_limit` is a nonnegative integer limiting single-post lookup calls, not
+HTTP attempts. Identity checks, route discovery and the existing transport's
+retries can add HTTP attempts. Zero makes no network calls. Calls respect client
+delay; a rate limit or request error stops further parent lookups in that run.
+
+## Outputs
+
+| File | Contents |
+| --- | --- |
+| `resolution.csv` | One row per original pending record, verified links, depth and scope flag |
+| `ancestor_posts.jsonl` | Newly fetched parents, all `observation_role="context"` |
+| `pending_posts.jsonl` | Copy of the original pending records |
+| `raw/<parent_id>.json` | Credential-free successful lookup payloads, including unavailable results |
+| `manifest.json` | Source and supplement hashes, caps, counts, observer checks and source completeness |
+
+`seed_post_id` is the requested collection seed; `root_post_id` is the platform
+conversation root. `chain_ids` lists verified retained IDs from the pending record
+back toward the seed; an unresolved chain does not include the unreturned parent.
+`blocked_post_id` identifies the first post where that chain could not be verified,
+including a deeper missing parent after the original parent was fetched. It is
+blank for resolved chains; `parent_post_id` always preserves the original reply link.
+`seed_relative_depth` counts links to the seed and is blank until that chain is
+resolved. `depth` counts links to the platform root and also requires retained
+evidence between a reply seed and that root; otherwise it stays blank. This method
+does not make extra requests above a reply seed just to fill platform depth.
+
+`within_requested_bounds` checks the pending record's verified seed-relative depth
+and the original UTC date bounds (inclusive `since`, exclusive `until`). It is blank
+for unresolved chains. A true flag is not automatic sample admission: supplement
+records never alter the original `posts.csv` or `comments.csv`.
+
+Statuses include `resolved`, `lookup_budget`, `parent_not_returned`, `rate_limited`,
+`request_error`, `identity_conflict`, `conversation_conflict`, `outside_seed_branch`,
+`cycle`, `response_unparsed` and `protected_parent`. A recognized empty result or
+tombstone is `parent_not_returned`; that does not establish deletion, protection
+or absence on the platform. A changed response shape or invalid returned post is
+`response_unparsed`, with raw evidence retained for diagnosis. Explicitly protected
+parent content is not retained or archived.
+
+Each invocation uses a new output directory. There is no supplement resume or
+automatic merging into the source. After traversal, all files are written in a
+temporary sibling directory, then renamed to the requested destination together.
+Ordinary write failures remove staging and leave the destination free for retry.
+A process termination before publication can lose that run's in-memory supplement;
+an abrupt kill can also leave an unpublished hidden staging directory. Retain
+the original source and compare dated supplements explicitly. The manifest's
+`collection_complete` and `source_collection_reason` copy the original collection's
+status, even if every pending chain has now been resolved.
+
+## Recorded validation
+
+On October 13, 2026, two retained personal-agent discussion trees contained three
+pending replies. With their original authenticated observers, three parent lookup
+calls resolved two chains in the first tree; the second tree's parent did not
+return a parseable record. Source file inventories and hashes were unchanged.
+
+The resolved replies had depths two and four. The source depth bound was three,
+so only the depth-two record passed the depth/date scope flag. This illustrates why
+resolving ancestry and admitting a reply to a research sample are separate steps.
+Both original collections remained partial. These are recorded results, not a
+coverage benchmark; later requests may return different content.
+
+Eighteen focused offline regression tests cover exact chains, shared parents,
+reply-seed boundaries, cycles, budgets, source integrity, identity mismatch,
+malformed responses and protected content. The complete suite contained 454
+passing tests at this release.
+
+The 1.3.1 review expanded this to 25 focused tests and 461 complete-suite tests.
+It also replayed the three previously archived lookup responses offline, preserving
+the two resolved chains and one empty parent result. No new network collection
+was used for that replay. Injected JSONL, raw-response and manifest write failures
+each left no final directory and allowed a retry to the same destination.

{pyxcom-1.2.1 → pyxcom-1.3.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pyXcom"
-version = "1.2.1"
+version = "1.3.1"
 description = "Collect public X posts, search results, conversations, reposts, and observed follow networks with your browser session"
 readme = "README.md"
 license = "Apache-2.0"

{pyxcom-1.2.1 → pyxcom-1.3.1}/src/pyxcom/__init__.py

@@ -50,4 +50,4 @@ __all__ = [
     "validate_collection",
     "validate_network_collection",
 ]
-__version__ = "1.2.1"
+__version__ = "1.3.1"

pyxcom-1.3.1/src/pyxcom/ancestry.py

@@ -0,0 +1,333 @@
+"""Copy-only, bounded verification of missing comment ancestors."""
+
+import csv
+import json
+import time
+from contextlib import ExitStack
+from dataclasses import replace
+from pathlib import Path
+from tempfile import TemporaryDirectory
+from typing import TYPE_CHECKING
+
+from ._persistence import append_jsonl, atomic_json, atomic_text, read_jsonl
+from ._pool_locks import ScopeLock
+from .errors import IntegrityError, ParseError, PyXcomError, RateLimitError
+from .integrity import assess_recovery
+from .models import Post
+from .parse import single_post
+from .provenance import _owner, assert_collection_owner
+from .recovery import _files
+from .transport import now_utc
+
+if TYPE_CHECKING:
+    from .client import XClient
+
+
+def _directory(value: str | Path) -> Path:
+    path = Path(value).expanduser()
+    if path.is_symlink():
+        raise ValueError("Ancestry directory must not be a symbolic link")
+    return path.resolve()
+
+
+def _mapping(value) -> dict:
+    return value if isinstance(value, dict) else {}
+
+
+def _protected(payload: dict) -> bool:
+    result = _mapping(
+        _mapping(_mapping(payload.get("data")).get("tweetResult")).get("result")
+    )
+    tweet = _mapping(result.get("tweet", result))
+    user = _mapping(
+        _mapping(_mapping(tweet.get("core")).get("user_results")).get("result")
+    )
+    return any(
+        _mapping(user.get(key)).get("protected") is True
+        for key in ("privacy", "legacy")
+    )
+
+
+def _parent_not_returned(payload: dict) -> bool:
+    """Recognize explicit empty slots/tombstones, not unknown response shapes."""
+    data = payload.get("data")
+    if payload.get("errors") or not isinstance(data, dict) or "tweetResult" not in data:
+        return False
+    slot = data["tweetResult"]
+    if slot is None:
+        return True
+    if not isinstance(slot, dict):
+        return False
+    result = slot.get("result")
+    return (
+        result is None
+        or result == {}
+        or (isinstance(result, dict) and result.get("__typename") == "TweetTombstone")
+    )
+
+
+def _publish_supplement(
+    target: Path,
+    pending: dict,
+    fetched: dict[str, Post],
+    rows: list[dict],
+    raw_responses: dict[str, dict],
+    report: dict,
+) -> dict:
+    """Publish a complete generation; ordinary write failures leave no final target."""
+    assert_collection_owner(target)
+    target.parent.mkdir(parents=True, exist_ok=True)
+    with TemporaryDirectory(prefix=".pyxcom-ancestry-", dir=target.parent) as temporary:
+        staged = Path(temporary)
+        (staged / "raw").mkdir()
+        for name, records in (
+            ("ancestor_posts.jsonl", (p.to_dict() for p in fetched.values())),
+            ("pending_posts.jsonl", pending.values()),
+        ):
+            append_jsonl(staged / name, records)
+        with atomic_text(
+            staged / "resolution.csv", encoding="utf-8-sig", newline=""
+        ) as stream:
+            writer = csv.DictWriter(
+                stream,
+                fieldnames=[
+                    "post_id",
+                    "root_post_id",
+                    "seed_post_id",
+                    "parent_post_id",
+                    "ancestry_status",
+                    "depth",
+                    "seed_relative_depth",
+                    "chain_ids",
+                    "within_requested_bounds",
+                    "blocked_post_id",
+                ],
+            )
+            writer.writeheader()
+            writer.writerows(rows)
+        for identifier, payload in raw_responses.items():
+            atomic_json(staged / "raw" / f"{identifier}.json", payload)
+        report["files_sha256"] = _files(staged)
+        atomic_json(staged / "manifest.json", report)
+        if target.exists():
+            raise FileExistsError("Ancestry supplement already exists")
+        assert_collection_owner(target)
+        staged.rename(target)
+    return report
+
+
+def save_comment_ancestry(
+    client: "XClient",
+    source_dir: str | Path,
+    output_dir: str | Path,
+    *,
+    lookup_limit: int = 20,
+) -> dict:
+    """Verify exact parent chains without admitting records to the original sample.
+
+    The cap counts single-post lookup calls, not transport retry attempts. A new
+    supplement is required for every run; source files are read-only throughout.
+    """
+    if (
+        isinstance(lookup_limit, bool)
+        or not isinstance(lookup_limit, int)
+        or lookup_limit < 0
+    ):
+        raise ValueError("lookup_limit must be a nonnegative integer")
+    source, target = _directory(source_dir), _directory(output_dir)
+    if source == target or source in target.parents or target in source.parents:
+        raise ValueError("Source and supplement scopes must not overlap")
+    with ExitStack() as stack:
+        for scope in sorted((source, target)):
+            stack.enter_context(ScopeLock(scope))
+        if target.exists():
+            raise FileExistsError("Ancestry supplement already exists")
+        assert_collection_owner(target)
+        assessment = assess_recovery(source)
+        if not assessment["allowed"] or assessment["record_integrity"] != "verified":
+            raise IntegrityError(
+                "Source collection did not pass integrity verification"
+            )
+        query = assessment["query"]
+        if query.get("kind") != "post_comments":
+            raise ValueError(
+                "Ancestry verification requires a post_comments collection"
+            )
+        state = json.loads((source / ".pyxcom/state.json").read_text(encoding="utf-8"))
+        conversation = state["conversation"]
+        pending = conversation.get("pending", {})
+        posts = {p["id"]: Post(**p) for p in read_jsonl(source / ".pyxcom/posts.jsonl")}
+        posts.update({key: Post(**value) for key, value in pending.items()})
+        root = str(query["root_post_id"])
+        conversation_id = str(conversation.get("conversation_id", root))
+        owner = _owner(source)
+        expected_observer = (owner or {}).get("observer_id")
+        verified_observer = None
+        fetched: dict[str, Post] = {}
+        failures: dict[str, str] = {}
+        raw_responses: dict[str, dict] = {}
+        calls = 0
+        stop_reason = None
+
+        def lookup(identifier: str) -> tuple[Post | None, str | None]:
+            nonlocal calls, stop_reason, verified_observer
+            if identifier in failures:
+                return None, failures[identifier]
+            if stop_reason:
+                return None, stop_reason
+            if calls >= lookup_limit:
+                return None, "lookup_budget"
+            if verified_observer is None:
+                verified_observer = str(client.check_login_identity())
+                if expected_observer is not None and verified_observer != str(
+                    expected_observer
+                ):
+                    raise ValueError(
+                        "Ancestry lookup observer differs from source observer"
+                    )
+            if calls:
+                time.sleep(client.delay)
+            calls += 1
+            payload = None
+            try:
+                payload = client.get_raw_post(identifier)
+                if _protected(payload):
+                    failures[identifier] = "protected_parent"
+                    return None, failures[identifier]
+                raw_responses[identifier] = payload
+                try:
+                    post = single_post(payload, captured_at_utc=now_utc())
+                except (ParseError, ValueError, TypeError, KeyError, AttributeError):
+                    failures[identifier] = (
+                        "parent_not_returned"
+                        if _parent_not_returned(payload)
+                        else "response_unparsed"
+                    )
+                    return None, failures[identifier]
+                if post.id != identifier:
+                    failures[identifier] = "identity_conflict"
+                elif post.conversation_id != conversation_id or post.reposted_post_id:
+                    failures[identifier] = "conversation_conflict"
+                else:
+                    post = replace(
+                        post, observation_role="context", discovery_source="x_ancestor"
+                    )
+                    posts[identifier] = fetched[identifier] = post
+                    return post, None
+            except ParseError:
+                stop_reason = failures[identifier] = "request_error"
+            except RateLimitError:
+                stop_reason = failures[identifier] = "rate_limited"
+            except PyXcomError:
+                stop_reason = failures[identifier] = "request_error"
+            return None, failures[identifier]
+
+        def platform_depth(seed_depth: int | None) -> int | None:
+            if seed_depth is None:
+                return None
+            current = posts[root]
+            seen = {root}
+            offset = 0
+            while current.id != conversation_id:
+                parent = posts.get(current.in_reply_to_id or "")
+                if (
+                    parent is None
+                    or parent.id in seen
+                    or parent.conversation_id != conversation_id
+                ):
+                    return None
+                seen.add(parent.id)
+                offset += 1
+                current = parent
+            return seed_depth + offset
+
+        rows = []
+        for identifier, value in pending.items():
+            post = posts[identifier]
+            chain = [identifier]
+            current = post
+            status = "resolved"
+            blocked_id = None
+            while current.id != root:
+                blocked_id = current.id
+                if (
+                    current.conversation_id != conversation_id
+                    or current.reposted_post_id
+                ):
+                    status = "conversation_conflict"
+                    break
+                parent_id = current.in_reply_to_id
+                blocked_id = parent_id or current.id
+                if parent_id is None or (
+                    parent_id == conversation_id and conversation_id != root
+                ):
+                    status = "outside_seed_branch"
+                    break
+                if parent_id in chain:
+                    status = "cycle"
+                    break
+                parent = posts.get(parent_id)
+                if parent is None:
+                    parent, failure = lookup(parent_id)
+                    if parent is None:
+                        status = failure or "parent_not_returned"
+                        break
+                if parent.conversation_id != conversation_id or parent.reposted_post_id:
+                    status = "conversation_conflict"
+                    break
+                chain.append(parent_id)
+                current = parent
+                blocked_id = None
+            seed_depth = len(chain) - 1 if status == "resolved" else None
+            within = None
+            if seed_depth is not None:
+                day = post.created_at_utc[:10]
+                within = (
+                    seed_depth <= query["max_depth"]
+                    and (not query.get("since") or day >= query["since"])
+                    and (not query.get("until") or day < query["until"])
+                )
+            rows.append(
+                {
+                    "post_id": identifier,
+                    "root_post_id": conversation_id,
+                    "seed_post_id": root,
+                    "parent_post_id": value.get("in_reply_to_id"),
+                    "ancestry_status": status,
+                    "depth": platform_depth(seed_depth),
+                    "seed_relative_depth": seed_depth,
+                    "chain_ids": json.dumps(chain),
+                    "within_requested_bounds": within,
+                    "blocked_post_id": blocked_id,
+                }
+            )
+        if _files(source) != assessment["input_sha256"] or _owner(source) != owner:
+            raise IntegrityError("Source changed during ancestry verification")
+        counts = {
+            status: sum(r["ancestry_status"] == status for r in rows)
+            for status in sorted({r["ancestry_status"] for r in rows})
+        }
+        from . import __version__
+
+        report = {
+            "schema": "pyxcom.comment-ancestry/1",
+            "package_version": __version__,
+            "captured_at_utc": now_utc(),
+            "query": query,
+            "source_sha256": assessment["input_sha256"],
+            "lookup_limit": lookup_limit,
+            "lookup_calls": calls,
+            "stop_reason": stop_reason,
+            "identity_verified": verified_observer is not None,
+            "source_observer_bound": expected_observer is not None
+            and verified_observer is not None,
+            "status_counts": counts,
+            "resolved_records": counts.get("resolved", 0),
+            "unresolved_records": len(rows) - counts.get("resolved", 0),
+            "context_records": len(fetched),
+            "collection_complete": assessment["complete"],
+            "source_collection_reason": assessment["reason"],
+        }
+        return _publish_supplement(
+            target, pending, fetched, rows, raw_responses, report
+        )

{pyxcom-1.2.1 → pyxcom-1.3.1}/src/pyxcom/client.py

@@ -957,6 +957,24 @@ class XClient(DiscoveryMixin, NetworkMixin):
             )
         )
 
+    def save_comment_ancestry(
+        self,
+        source_dir: str | Path,
+        output_dir: str | Path,
+        *,
+        lookup_limit: int = 20,
+    ) -> dict:
+        """Save a separate, bounded supplement verifying missing comment parents.
+
+        Original collection files and sampling roles are preserved. The cap counts
+        single-post lookup calls; transport retries can make more HTTP attempts.
+        """
+        from .ancestry import save_comment_ancestry
+
+        return save_comment_ancestry(
+            self, source_dir, output_dir, lookup_limit=lookup_limit
+        )
+
     @guarded_collection
     def save_post_comments(
         self,