pyXcom 1.3.0__tar.gz → 1.3.1__tar.gz

{pyxcom-1.3.0 → pyxcom-1.3.1}/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## 1.3.1 — 2026-10-13
+
+### Fixed
+
+- Distinguish malformed or unparseable parent responses (`response_unparsed`)
+  from recognized empty results and tombstones (`parent_not_returned`).
+- Honor inherited source observer bindings and reject malformed markers by reusing
+  the package's ownership reader; check ownership again before publication.
+- Reject supplement destinations owned by another pool before network requests,
+  including inherited ownership, and check again before publication.
+- Publish ancestry output as one staged generation. Ordinary write failures leave
+  no partial final directory blocking retries.
+
+### Added
+
+- `blocked_post_id` identifies the actual point of failure along a parent chain,
+  while preserving the original `parent_post_id`.
+
 ## 1.3.0 — 2026-10-13
 
 ### Added

{pyxcom-1.3.0 → pyxcom-1.3.1}/CITATION.cff

@@ -6,7 +6,7 @@ authors:
   - family-names: Wang
     given-names: Haocheng
     affiliation: Communication University of China
-version: 1.3.0
+version: 1.3.1
 date-released: 2026-10-13
 license: Apache-2.0
 repository-code: "https://github.com/haochengw372-hash/pyXcom"

{pyxcom-1.3.0 → pyxcom-1.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pyXcom
-Version: 1.3.0
+Version: 1.3.1
 Summary: Collect public X posts, search results, conversations, reposts, and observed follow networks with your browser session
 Project-URL: Homepage, https://github.com/haochengw372-hash/pyXcom
 Project-URL: Repository, https://github.com/haochengw372-hash/pyXcom

{pyxcom-1.3.0 → pyxcom-1.3.1}/docs/comment-ancestry.md

@@ -20,6 +20,8 @@ The source must pass retained-record and checkpoint integrity verification and
 must be a `post_comments` collection. Both scopes are locked; overlapping paths,
 existing destinations and symlink source contents are rejected. Source files,
 cursors, roles, counts and partial status remain unchanged.
+Destinations inside another pool's owned scope are rejected before network requests
+and checked again before publication; an idle pool still owns its persisted scope.
 
 ## Lookup rules
 
@@ -30,8 +32,10 @@ resolved only when it reaches the exact requested seed, without a cycle. Other
 branches of a reply-seeded conversation are not admitted.
 
 Before the first lookup, the client verifies its current identity. If the source
-has a pool observer binding, that identity must match; a mismatch raises before
-parent content is requested. Older unbound sources remain explicitly unbound in
+has a pool observer binding, including one inherited from an ancestor directory,
+that identity must match; a mismatch raises before parent content is requested.
+Malformed ownership markers are rejected before identity or parent requests, and
+the marker is checked again before publication. Older unbound sources remain explicitly unbound in
 the manifest. No observer is inferred retroactively and no account rotation occurs.
 
 `lookup_limit` is a nonnegative integer limiting single-post lookup calls, not
@@ -52,6 +56,9 @@ delay; a rate limit or request error stops further parent lookups in that run.
 `seed_post_id` is the requested collection seed; `root_post_id` is the platform
 conversation root. `chain_ids` lists verified retained IDs from the pending record
 back toward the seed; an unresolved chain does not include the unreturned parent.
+`blocked_post_id` identifies the first post where that chain could not be verified,
+including a deeper missing parent after the original parent was fetched. It is
+blank for resolved chains; `parent_post_id` always preserves the original reply link.
 `seed_relative_depth` counts links to the seed and is blank until that chain is
 resolved. `depth` counts links to the platform root and also requires retained
 evidence between a reply seed and that root; otherwise it stays blank. This method
@@ -64,13 +71,18 @@ records never alter the original `posts.csv` or `comments.csv`.
 
 Statuses include `resolved`, `lookup_budget`, `parent_not_returned`, `rate_limited`,
 `request_error`, `identity_conflict`, `conversation_conflict`, `outside_seed_branch`,
-`cycle` and `protected_parent`. A successful response without a parseable parent
-is `parent_not_returned`; that does not establish deletion, protection or absence
-on the platform. Explicitly protected parent content is not retained or archived.
+`cycle`, `response_unparsed` and `protected_parent`. A recognized empty result or
+tombstone is `parent_not_returned`; that does not establish deletion, protection
+or absence on the platform. A changed response shape or invalid returned post is
+`response_unparsed`, with raw evidence retained for diagnosis. Explicitly protected
+parent content is not retained or archived.
 
 Each invocation uses a new output directory. There is no supplement resume or
-automatic merging into the source. Output is published after traversal; a process
-termination before publication can lose that run's in-memory supplement. Retain
+automatic merging into the source. After traversal, all files are written in a
+temporary sibling directory, then renamed to the requested destination together.
+Ordinary write failures remove staging and leave the destination free for retry.
+A process termination before publication can lose that run's in-memory supplement;
+an abrupt kill can also leave an unpublished hidden staging directory. Retain
 the original source and compare dated supplements explicitly. The manifest's
 `collection_complete` and `source_collection_reason` copy the original collection's
 status, even if every pending chain has now been resolved.
@@ -92,3 +104,9 @@ Eighteen focused offline regression tests cover exact chains, shared parents,
 reply-seed boundaries, cycles, budgets, source integrity, identity mismatch,
 malformed responses and protected content. The complete suite contained 454
 passing tests at this release.
+
+The 1.3.1 review expanded this to 25 focused tests and 461 complete-suite tests.
+It also replayed the three previously archived lookup responses offline, preserving
+the two resolved chains and one empty parent result. No new network collection
+was used for that replay. Injected JSONL, raw-response and manifest write failures
+each left no final directory and allowed a retry to the same destination.

{pyxcom-1.3.0 → pyxcom-1.3.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pyXcom"
-version = "1.3.0"
+version = "1.3.1"
 description = "Collect public X posts, search results, conversations, reposts, and observed follow networks with your browser session"
 readme = "README.md"
 license = "Apache-2.0"

{pyxcom-1.3.0 → pyxcom-1.3.1}/src/pyxcom/__init__.py

@@ -50,4 +50,4 @@ __all__ = [
     "validate_collection",
     "validate_network_collection",
 ]
-__version__ = "1.3.0"
+__version__ = "1.3.1"

{pyxcom-1.3.0 → pyxcom-1.3.1}/src/pyxcom/ancestry.py

@@ -6,14 +6,16 @@ import time
 from contextlib import ExitStack
 from dataclasses import replace
 from pathlib import Path
+from tempfile import TemporaryDirectory
 from typing import TYPE_CHECKING
 
-from ._persistence import append_jsonl, atomic_json, read_jsonl
+from ._persistence import append_jsonl, atomic_json, atomic_text, read_jsonl
 from ._pool_locks import ScopeLock
 from .errors import IntegrityError, ParseError, PyXcomError, RateLimitError
 from .integrity import assess_recovery
 from .models import Post
 from .parse import single_post
+from .provenance import _owner, assert_collection_owner
 from .recovery import _files
 from .transport import now_utc
 
@@ -46,6 +48,74 @@ def _protected(payload: dict) -> bool:
     )
 
 
+def _parent_not_returned(payload: dict) -> bool:
+    """Recognize explicit empty slots/tombstones, not unknown response shapes."""
+    data = payload.get("data")
+    if payload.get("errors") or not isinstance(data, dict) or "tweetResult" not in data:
+        return False
+    slot = data["tweetResult"]
+    if slot is None:
+        return True
+    if not isinstance(slot, dict):
+        return False
+    result = slot.get("result")
+    return (
+        result is None
+        or result == {}
+        or (isinstance(result, dict) and result.get("__typename") == "TweetTombstone")
+    )
+
+
+def _publish_supplement(
+    target: Path,
+    pending: dict,
+    fetched: dict[str, Post],
+    rows: list[dict],
+    raw_responses: dict[str, dict],
+    report: dict,
+) -> dict:
+    """Publish a complete generation; ordinary write failures leave no final target."""
+    assert_collection_owner(target)
+    target.parent.mkdir(parents=True, exist_ok=True)
+    with TemporaryDirectory(prefix=".pyxcom-ancestry-", dir=target.parent) as temporary:
+        staged = Path(temporary)
+        (staged / "raw").mkdir()
+        for name, records in (
+            ("ancestor_posts.jsonl", (p.to_dict() for p in fetched.values())),
+            ("pending_posts.jsonl", pending.values()),
+        ):
+            append_jsonl(staged / name, records)
+        with atomic_text(
+            staged / "resolution.csv", encoding="utf-8-sig", newline=""
+        ) as stream:
+            writer = csv.DictWriter(
+                stream,
+                fieldnames=[
+                    "post_id",
+                    "root_post_id",
+                    "seed_post_id",
+                    "parent_post_id",
+                    "ancestry_status",
+                    "depth",
+                    "seed_relative_depth",
+                    "chain_ids",
+                    "within_requested_bounds",
+                    "blocked_post_id",
+                ],
+            )
+            writer.writeheader()
+            writer.writerows(rows)
+        for identifier, payload in raw_responses.items():
+            atomic_json(staged / "raw" / f"{identifier}.json", payload)
+        report["files_sha256"] = _files(staged)
+        atomic_json(staged / "manifest.json", report)
+        if target.exists():
+            raise FileExistsError("Ancestry supplement already exists")
+        assert_collection_owner(target)
+        staged.rename(target)
+    return report
+
+
 def save_comment_ancestry(
     client: "XClient",
     source_dir: str | Path,
@@ -72,6 +142,7 @@ def save_comment_ancestry(
             stack.enter_context(ScopeLock(scope))
         if target.exists():
             raise FileExistsError("Ancestry supplement already exists")
+        assert_collection_owner(target)
         assessment = assess_recovery(source)
         if not assessment["allowed"] or assessment["record_integrity"] != "verified":
             raise IntegrityError(
@@ -89,13 +160,8 @@ def save_comment_ancestry(
         posts.update({key: Post(**value) for key, value in pending.items()})
         root = str(query["root_post_id"])
         conversation_id = str(conversation.get("conversation_id", root))
-        owner_path = source / ".pyxcom/pool-owner.json"
-        owner = (
-            json.loads(owner_path.read_text(encoding="utf-8"))
-            if owner_path.exists()
-            else {}
-        )
-        expected_observer = owner.get("observer_id")
+        owner = _owner(source)
+        expected_observer = (owner or {}).get("observer_id")
         verified_observer = None
         fetched: dict[str, Post] = {}
         failures: dict[str, str] = {}
@@ -132,7 +198,11 @@ def save_comment_ancestry(
                 try:
                     post = single_post(payload, captured_at_utc=now_utc())
                 except (ParseError, ValueError, TypeError, KeyError, AttributeError):
-                    failures[identifier] = "parent_not_returned"
+                    failures[identifier] = (
+                        "parent_not_returned"
+                        if _parent_not_returned(payload)
+                        else "response_unparsed"
+                    )
                     return None, failures[identifier]
                 if post.id != identifier:
                     failures[identifier] = "identity_conflict"
@@ -177,7 +247,9 @@ def save_comment_ancestry(
             chain = [identifier]
             current = post
             status = "resolved"
+            blocked_id = None
             while current.id != root:
+                blocked_id = current.id
                 if (
                     current.conversation_id != conversation_id
                     or current.reposted_post_id
@@ -185,6 +257,7 @@ def save_comment_ancestry(
                     status = "conversation_conflict"
                     break
                 parent_id = current.in_reply_to_id
+                blocked_id = parent_id or current.id
                 if parent_id is None or (
                     parent_id == conversation_id and conversation_id != root
                 ):
@@ -204,6 +277,7 @@ def save_comment_ancestry(
                     break
                 chain.append(parent_id)
                 current = parent
+                blocked_id = None
             seed_depth = len(chain) - 1 if status == "resolved" else None
             within = None
             if seed_depth is not None:
@@ -224,38 +298,11 @@ def save_comment_ancestry(
                     "seed_relative_depth": seed_depth,
                     "chain_ids": json.dumps(chain),
                     "within_requested_bounds": within,
+                    "blocked_post_id": blocked_id,
                 }
             )
-        if _files(source) != assessment["input_sha256"]:
+        if _files(source) != assessment["input_sha256"] or _owner(source) != owner:
             raise IntegrityError("Source changed during ancestry verification")
-        target.mkdir(parents=True, exist_ok=False)
-        (target / "raw").mkdir()
-        for name, records in (
-            ("ancestor_posts.jsonl", (p.to_dict() for p in fetched.values())),
-            ("pending_posts.jsonl", pending.values()),
-        ):
-            append_jsonl(target / name, records)
-        with (target / "resolution.csv").open(
-            "w", encoding="utf-8-sig", newline=""
-        ) as stream:
-            writer = csv.DictWriter(
-                stream,
-                fieldnames=[
-                    "post_id",
-                    "root_post_id",
-                    "seed_post_id",
-                    "parent_post_id",
-                    "ancestry_status",
-                    "depth",
-                    "seed_relative_depth",
-                    "chain_ids",
-                    "within_requested_bounds",
-                ],
-            )
-            writer.writeheader()
-            writer.writerows(rows)
-        for identifier, payload in raw_responses.items():
-            atomic_json(target / "raw" / f"{identifier}.json", payload)
         counts = {
             status: sum(r["ancestry_status"] == status for r in rows)
             for status in sorted({r["ancestry_status"] for r in rows})
@@ -280,7 +327,7 @@ def save_comment_ancestry(
             "context_records": len(fetched),
             "collection_complete": assessment["complete"],
             "source_collection_reason": assessment["reason"],
-            "files_sha256": _files(target),
         }
-        atomic_json(target / "manifest.json", report)
-        return report
+        return _publish_supplement(
+            target, pending, fetched, rows, raw_responses, report
+        )

{pyxcom-1.3.0 → pyxcom-1.3.1}/tests/test_comment_ancestry.py

@@ -3,7 +3,7 @@ import json
 import tempfile
 import unittest
 from pathlib import Path
-from unittest.mock import Mock
+from unittest.mock import Mock, patch
 
 from test_comments import ROOT, A, B, C, client, page, tweet
 from pyxcom import IntegrityError, RateLimitError
@@ -14,6 +14,15 @@ def lookup_payload(post):
     return {"data": {"tweetResult": {"result": post}}}
 
 
+def owner_marker(observer_id, source):
+    return {
+        "observer_id": observer_id,
+        "login_id": "test_login",
+        "job_id": "test_job",
+        "pool_state_dir": str(source.parent / "pool"),
+    }
+
+
 class AncestryTests(unittest.TestCase):
     def make_source(self, tmp, posts):
         source = Path(tmp) / "source"
@@ -117,13 +126,53 @@ class AncestryTests(unittest.TestCase):
         with tempfile.TemporaryDirectory() as tmp:
             c, source = self.make_source(tmp, [tweet(B, A)])
             (source / ".pyxcom/pool-owner.json").write_text(
-                json.dumps({"observer_id": "200"})
+                json.dumps(owner_marker("200", source))
             )
             c.check_login_identity = Mock(return_value="100")
             c.get_raw_post = Mock()
             with self.assertRaises(ValueError):
                 c.save_comment_ancestry(source, Path(tmp) / "resolved")
             c.get_raw_post.assert_not_called()
+            c.check_login_identity.assert_called_once()
+
+    def test_inherited_observer_mismatch_rejects_before_lookup(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            c, source = self.make_source(Path(tmp) / "owned", [tweet(B, A)])
+            marker = source.parent / ".pyxcom/pool-owner.json"
+            marker.parent.mkdir()
+            marker.write_text(json.dumps(owner_marker("200", source)))
+            c.check_login_identity = Mock(return_value="100")
+            c.get_raw_post = Mock()
+            with self.assertRaises(ValueError):
+                c.save_comment_ancestry(source, Path(tmp) / "resolved")
+            c.get_raw_post.assert_not_called()
+            self.assertFalse((Path(tmp) / "resolved").exists())
+            c.check_login_identity.assert_called_once()
+
+    def test_other_pool_owned_destination_rejects_before_network(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            c, source = self.make_source(tmp, [tweet(B, A)])
+            other = Path(tmp) / "other_pool_output"
+            marker = other / ".pyxcom/pool-owner.json"
+            marker.parent.mkdir(parents=True)
+            marker.write_text(json.dumps(owner_marker("200", other)))
+            before = self.files(other)
+            c.check_login_identity = Mock(return_value="100")
+            c.get_raw_post = Mock(return_value=lookup_payload(tweet(A, ROOT)))
+            with self.assertRaises(IntegrityError):
+                c.save_comment_ancestry(source, other / "supplement")
+            c.check_login_identity.assert_not_called()
+            self.assertEqual(self.files(other), before)
+
+    def test_malformed_owner_marker_rejects_before_identity_check(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            c, source = self.make_source(tmp, [tweet(B, A)])
+            (source / ".pyxcom/pool-owner.json").write_text('{"observer_id": null}')
+            c.check_login_identity = Mock(return_value="100")
+            c.get_raw_post = Mock(return_value={"data": {"tweetResult": {}}})
+            with self.assertRaises(IntegrityError):
+                c.save_comment_ancestry(source, Path(tmp) / "resolved")
+            c.check_login_identity.assert_not_called()
 
     def test_shared_missing_parent_is_fetched_once(self):
         with tempfile.TemporaryDirectory() as tmp:
@@ -222,9 +271,86 @@ class AncestryTests(unittest.TestCase):
             target = Path(tmp) / "resolved"
             report = c.save_comment_ancestry(source, target)
             self.assertEqual(report["resolved_records"], 1)
-            self.assertEqual(report["status_counts"]["parent_not_returned"], 1)
+            self.assertEqual(report["status_counts"]["response_unparsed"], 1)
             self.assertEqual(report["context_records"], 1)
 
+    def test_invalid_returned_parent_is_parse_failure_not_absence(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            c, source = self.make_source(tmp, [tweet(B, A)])
+            parent = tweet(A, ROOT)
+            del parent["legacy"]["created_at"]
+            c.check_login_identity = Mock(return_value="100")
+            c.get_raw_post = Mock(return_value=lookup_payload(parent))
+            target = Path(tmp) / "resolved"
+            report = c.save_comment_ancestry(source, target)
+            self.assertEqual(
+                self.rows(target)[0]["ancestry_status"], "response_unparsed"
+            )
+            self.assertEqual(report["context_records"], 0)
+            self.assertTrue((target / "raw" / f"{A}.json").exists())
+
+    def test_tombstone_is_unreturned_not_unparsed(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            c, source = self.make_source(tmp, [tweet(B, A)])
+            c.check_login_identity = Mock(return_value="100")
+            c.get_raw_post = Mock(
+                return_value=lookup_payload({"__typename": "TweetTombstone"})
+            )
+            target = Path(tmp) / "resolved"
+            c.save_comment_ancestry(source, target)
+            self.assertEqual(
+                self.rows(target)[0]["ancestry_status"], "parent_not_returned"
+            )
+
+    def test_deeper_blocked_id_is_distinct_from_original_parent(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            c, source = self.make_source(tmp, [tweet(C, B)])
+            c.check_login_identity = Mock(return_value="100")
+            c.get_raw_post = Mock(
+                side_effect=[lookup_payload(tweet(B, A)), {"data": {"tweetResult": {}}}]
+            )
+            target = Path(tmp) / "resolved"
+            c.save_comment_ancestry(source, target)
+            row = self.rows(target)[0]
+            self.assertEqual(row["parent_post_id"], B)
+            self.assertEqual(row["blocked_post_id"], A)
+            self.assertEqual(row["ancestry_status"], "parent_not_returned")
+            self.assertEqual(json.loads(row["chain_ids"]), [C, B])
+
+    def test_publication_failure_leaves_no_partial_target_and_can_retry(self):
+        from pyxcom._persistence import atomic_json as write_json
+
+        for phase in ("jsonl", "raw", "manifest"):
+            with self.subTest(phase=phase), tempfile.TemporaryDirectory() as tmp:
+                c, source = self.make_source(tmp, [tweet(B, A)])
+                c.check_login_identity = Mock(return_value="100")
+                c.get_raw_post = Mock(return_value=lookup_payload(tweet(A, ROOT)))
+                target = Path(tmp) / "resolved"
+                before = self.files(source)
+
+                def fail_write(path, value):
+                    if (phase == "raw" and path.parent.name == "raw") or (
+                        phase == "manifest" and path.name == "manifest.json"
+                    ):
+                        raise OSError("injected write failure")
+                    write_json(path, value)
+
+                failure = (
+                    patch(
+                        "pyxcom.ancestry.append_jsonl",
+                        side_effect=OSError("injected write failure"),
+                    )
+                    if phase == "jsonl"
+                    else patch("pyxcom.ancestry.atomic_json", side_effect=fail_write)
+                )
+                with failure, self.assertRaises(OSError):
+                    c.save_comment_ancestry(source, target)
+                self.assertFalse(target.exists())
+                self.assertEqual(list(target.parent.glob(".pyxcom-ancestry-*")), [])
+                self.assertEqual(self.files(source), before)
+                report = c.save_comment_ancestry(source, target)
+                self.assertEqual(report["resolved_records"], 1)
+
     def test_paths_and_corruption_reject_before_network(self):
         with tempfile.TemporaryDirectory() as tmp:
             c, source = self.make_source(tmp, [tweet(B, A)])