matrx-graph 0.2.331__tar.gz → 0.2.332__tar.gz

{matrx_graph-0.2.331 → matrx_graph-0.2.332}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: matrx-graph
-Version: 0.2.331
+Version: 0.2.332
 Summary: Channel-based, super-step workflow orchestration engine for the Matrx ecosystem
 Project-URL: Homepage, https://github.com/AI-Matrix-Engine/aidream-current
 Project-URL: Repository, https://github.com/AI-Matrix-Engine/aidream-current

{matrx_graph-0.2.331 → matrx_graph-0.2.332}/matrx_graph/executor/scheduler.py

@@ -2181,6 +2181,11 @@ class Scheduler:
         try:
             while open_members:
                 task, result, result_step = await done_queue.get()
+                if task.cancelled() and _job_lease().lost():
+                    # A member stopped because this copy lost its job lease (``JobLeaseLost`` is a cancellation):
+                    # the whole copy stops here, never routing or checkpointing on (lane K1b, B2).
+                    lease = _job_lease().current()
+                    raise _job_lease().JobLeaseLost(f"job lease lost: {lease.why if lease else ''}")
                 if id(result) in member_order and result_step == step:
                     # Kept in frontier order by insertion — O(log n) search per finished member, never a re-sort.
                     bisect.insort(finished_members, result, key=lambda r: member_order[id(r)])
@@ -6490,3 +6495,10 @@ def _settle_default_on_decision_gate(
     logger.info("decision gate settled by its default %r → %s", raw, shaped["answer"])
     value["answer"] = shaped["answer"]
     return value
+
+
+def _job_lease():
+    """``matrx_graph.workers.job_lease``, imported on use (the workers package imports the executor)."""
+    from matrx_graph.workers import job_lease
+
+    return job_lease

{matrx_graph-0.2.331 → matrx_graph-0.2.332}/matrx_graph/workers/FEATURE.md

@@ -17,7 +17,7 @@ and `workflow.job`'s post-reorg baseline schema on 2026-08-09.
 - **Jobs run in parallel, bounded, each in its own task (2026-09-17).** `WorkerRunner` holds an `asyncio.Semaphore(max_concurrent)`: it keeps claiming while slots are free, waits on the *clock* (`poll_interval`) only when the queue is empty, and waits on a *slot* when every one is busy. Each claimed job runs as its own `asyncio.Task` with its own lease renewer, and `complete`/`fail`/cancel semantics are per job and unchanged. **`max_concurrent` defaults to 1** — a host that has not thought about the bound keeps exactly the behavior it had; aidream passes `workflow_worker.max_concurrent_jobs` (a `platform.feature_knob` row, currently 8, read once at worker start). Why: the loop was strictly serial, so ONE slow job held every other workflow on the platform. Measured in production 2026-09-17, five simultaneous Cloud Browser activations (each a workflow job that waits on a container coming up) came up as a 5.5-second-per-run ladder, and for those 22 seconds nothing else in the queue moved; at the 50-run fleet ceiling that is a ~4.5-minute tail. **Per-job isolation is structural, not conventional**: a job task is created from the claim loop's context, so `contextvars` it sets (an ambient `AppContext`, an ORM connection) belong to that task alone — they cannot reach a sibling job and cannot outlive the job in the loop. Under the old serial `await` they did: aidream's `_run_one` installs an ambient AppContext at the tail of every job, and that context stayed installed in the claim loop afterwards. Proof: `tests/test_workers.py` — the wall-clock test, the peak-concurrency test, the neighbour-isolation test, the two drain tests, and the context-isolation test (each demonstrated failing against the serial loop before the change).
 - **Stopping is a bounded drain.** `stop()` takes no new claims; `run_forever()` then waits out the jobs already running for `stop_grace_seconds` (aidream passes its share of THE DEPLOY WINDOW) and cancels whatever is left. A canceled job fails retryable and its run resumes from its last checkpoint — the same outcome a serial worker's single job always had when the container was replaced.
 - **A worker only claims what it can run (2026-10-12, `capability.py`).** Every server — deployed fleet and developer laptops — shares one database, so a job may belong to a graph whose node types this build has not shipped (live: a local build's `brand.setup.land_logo` run was resumed by production and errored "Unknown node type"). `enqueue()` / `enqueue_claimed()` stamp `workflow.job.metadata.required_node_types` (canonical spellings, removed types excluded) from the run's stored definition; `claim_one(capabilities=...)` adds `COALESCE(j.metadata->'required_node_types','[]') <@ caps` (`matrx_orm.JsonbContainedBy`), so an unstamped legacy job stays claimable by anyone. `WorkerRunner` and `run_single_job` pass `worker_capabilities()` (the default registry) on every claim. Belt two: a host job runner that finds a gap at run time raises `JobNeedsCapability`; the runner calls `defer_for_capability()` (un-claim, attempt refunded, requirement stamped, immediately visible) — never `fail()`, never `complete()`, never a run-status write. The run says what it waits for in `workflow.run.metadata._awaiting_capability` (`missing`, `message`, `since`, `checked_at`, `checked_by`) plus one `workflow.run_log` line (`run_awaiting_capability`); every incapable worker refreshes it via `stamp_runs_awaiting_capability()` and `checked_at` is a lifecycle-watchdog liveness signal, so a waiting run is not force-failed as stranded while a fleet is watching it. The first capable worker clears the marker. Proof: `tests/test_capability_aware_claim.py` (incl. the predicate evaluated in live Postgres; fails against the pre-change queue/runner).
-- **A copy that loses its lease stops itself (lane K1, 2026-10-21).** A renewal that answers "not ours" (reaped — including a FALSE "dead" after a database outage longer than the lease, live job c1d89868 — or settled elsewhere) marks the copy's `job_lease.JobLease` lost and cancels its task; the host skips its terminal write (the copy that holds the job owns the run); many-item work calls `job_lease.check()` before each item (it also reads the job row, at most every tenth of a renewal beat). The reaper does not try to tell an outage from a death: a false "dead" is harmless by construction. Proof: aidream `aidream/services/tests/test_a_reaped_run_never_waits_on_itself.py`.
+- **A copy that loses its lease stops itself (lane K1, 2026-10-21).** A renewal that answers "not ours" (reaped — including a FALSE "dead" after a database outage longer than the lease, live job c1d89868 — or settled elsewhere) marks the copy's `job_lease.JobLease` lost and cancels its task; the host skips its terminal write (the copy that holds the job owns the run); many-item work calls `job_lease.check()` before each item (it also reads the job row, at most every tenth of a renewal beat). The reaper does not try to tell an outage from a death: a false "dead" is harmless by construction. Proof: aidream `aidream/services/tests/test_a_reaped_run_never_waits_on_itself.py`. **Lane K1b:** whoever notices the loss first (the renewal or the work's own `check`) cancels the copy's owner task, and run-ahead re-raises a member that ended cancelled on a lost lease (it used to swallow it and route on). A job the reaper only DEAD-LETTERED (a false "dead" on its last attempt) is taken back by the live copy (`queue.take_back_false_dead_letter`, compare-and-set) and finished — stopping would strand the run, since nothing claims a dead-lettered job; a requeued or re-claimed job still stops the old copy. A renewal landing after `complete()` is not a loss (`JobLease.settling`). Each copy's `JobLease.generation` (the job's attempt) fences what it holds (import slots). A platform-wide outage longer than the lease plus grace cancels EVERY in-flight job at once and they all restart from their checkpoints together: the same spend per job as before, but as one burst. Proof: `tests/test_a_lost_lease_never_strands_or_lingers.py`.
 - **Recovery starts before work.** `WorkerRunner.run_forever()` calls `reap_expired_leases()` before its first claim. The host's recurring reaper remains a backstop.
 - **A transient claim outage never kills the worker.** A database timeout or
   connection failure while claiming `workflow.job` is captured durably and

{matrx_graph-0.2.331 → matrx_graph-0.2.332}/matrx_graph/workers/job_lease.py

@@ -41,15 +41,31 @@ class JobLease:
     #: fewest seconds between two reads of the job row by ``check`` (a many-item run checks before every item); the
     #: runner sets it from its lease (a tenth of the renewal beat), and the renewal's own answer is used in between
     look_every: float = 0.0
+    #: the task running this copy of the job: whoever notices the loss first (the renewal, or the work's own
+    #: ``check``) cancels it, so the WHOLE copy stops at once (lane K1b, B2: run-ahead swallowed a ``JobLeaseLost``
+    #: raised inside one node, and the copy kept checkpointing and routing until the next renewal)
+    owner: asyncio.Task | None = field(default=None, repr=False)
+    #: the copy finished its work and is settling the job (``complete``): a renewal that lands after the settle
+    #: commits answers "not ours", and that is not a loss (K1b, M1)
+    settling: bool = False
+    #: per-copy fencing token for anything this copy holds (an import slot): a newer copy's is always greater
+    generation: float = field(default_factory=time.time)
+    visibility_timeout: int = 600
     lost: bool = False
     why: str = ""
     _looked: float = field(default=0.0, repr=False)
 
     def mark_lost(self, why: str) -> None:
+        if self.settling:
+            log.info("job %s: settled while a renewal was in flight; not a lost lease", self.job_id[:8])
+            return
         if not self.lost:
             log.error("job %s (run %s) lost its lease (%s): this copy stops and writes nothing more",
                       self.job_id[:8], self.run_id[:8], why)
         self.lost, self.why = True, why
+        owner = self.owner
+        if owner is not None and not owner.done() and owner is not _running_task():
+            owner.cancel(f"job {self.job_id[:8]} lease lost: {why}")
 
     async def still_held(self) -> bool | None:
         """The job row's answer: True while this claim holds it, False once it does not, None when unreadable."""
@@ -66,7 +82,28 @@ class JobLease:
         get = row.get if isinstance(row, dict) else lambda k: getattr(row, k, None)
         if get("claimed_at") is None:
             return False
-        return self.claimed_by is None or get("claimed_by") == self.claimed_by
+        # every claim stamps a token: a token that is not this copy's (or one where this copy had none) is another
+        # claim (K1b, M4: a token-less copy used to read every later claim as its own)
+        return get("claimed_by") == self.claimed_by
+
+    async def not_ours(self, why: str) -> bool:
+        """The queue says this copy no longer holds its job. When the reaper only DEAD-LETTERED it (a false "dead" on
+        the last attempt), no other copy can ever own the run: take the job back and keep going, so the run is
+        finished and its terminal status written (K1b, B1). Otherwise mark the lease lost (which stops the copy).
+        Returns True when the copy keeps the job."""
+        from matrx_graph.workers.queue import take_back_false_dead_letter
+
+        if self.settling:
+            return True
+        try:
+            kept = await take_back_false_dead_letter(
+                self.job_id, claimed_by=self.claimed_by, visibility_timeout_seconds=self.visibility_timeout)
+        except Exception:  # noqa: BLE001 — unknown: the copy stops (the safe side for a run another copy may own)
+            log.warning("job %s: could not try to take back a dead-lettered job", self.job_id[:8], exc_info=True)
+            kept = False
+        if not kept:
+            self.mark_lost(why)
+        return kept
 
     async def check(self) -> None:
         """Raise :class:`JobLeaseLost` when this copy no longer holds its job; read the row at most every
@@ -74,11 +111,18 @@ class JobLease:
         if not self.lost and time.monotonic() - self._looked >= self.look_every:
             self._looked = time.monotonic()
             if await self.still_held() is False:
-                self.mark_lost("the job row is claimed by another copy or was requeued")
+                await self.not_ours("the job row is claimed by another copy or was requeued")
         if self.lost:
             raise JobLeaseLost(f"job {self.job_id[:8]} lease lost: {self.why}")
 
 
+def _running_task() -> asyncio.Task | None:
+    try:
+        return asyncio.current_task()
+    except RuntimeError:
+        return None
+
+
 _current: ContextVar[JobLease | None] = ContextVar("matrx_graph_job_lease", default=None)
 
 

{matrx_graph-0.2.331 → matrx_graph-0.2.332}/matrx_graph/workers/queue.py

@@ -707,6 +707,62 @@ async def renew_lease(
     return result.rows_affected > 0
 
 
+async def take_back_false_dead_letter(
+    job_id: str,
+    *,
+    claimed_by: str | None,
+    visibility_timeout_seconds: int = 600,
+) -> bool:
+    """A live copy whose job the reaper DEAD-LETTERED (a false "dead": an outage outlasted the lease on the job's last
+    attempt) takes the job back, so it finishes the run and writes its terminal status (lane K1b, B1, 2026-10-21).
+
+    Without this the live copy read "not ours", cancelled itself and skipped its terminal write, and no other copy would
+    ever claim a dead-lettered job: the run stayed non-terminal (stranded) for every job kind. A dead-lettered job has
+    no other owner, so taking it back can never run two copies. Compare-and-set: only a row that is still unclaimed,
+    still dead-lettered BY THE REAPER (``last_error.type == 'lease_expired'``) and still parked in the far future
+    (an operator's retry moves ``visible_at`` back) is taken; anything else answers False and the copy stops.
+    """
+    from matrx_graph.db._registry import get_model
+
+    model = get_model("WfJob")
+    try:
+        rows = await model.filter(id=str(UUID(job_id))).all(use_cache=False)
+    except Exception:  # noqa: BLE001 — unreadable: not taken back (the caller decides on the renewal's answer)
+        log.warning("could not read job %s to see whether the reaper dead-lettered it", job_id[:8], exc_info=True)
+        return False
+    if not rows:
+        return False
+    row = rows[0]
+    get = row.get if isinstance(row, dict) else lambda k: getattr(row, k, None)
+    note = _parse_json(get("last_error")) or {}
+    if get("claimed_at") is not None or not isinstance(note, dict):
+        return False
+    if note.get("queue_state") != "dead_letter" or note.get("type") != "lease_expired":
+        return False
+    now = datetime.now(UTC)
+    taken = (
+        await model.filter(
+            id=str(UUID(job_id)),
+            claimed_at__isnull=True,
+            visible_at__gt=now + timedelta(days=_FAR_FUTURE_DAYS // 2),
+        ).update(
+            claimed_at=now,
+            claimed_by=claimed_by,
+            visible_at=now + timedelta(seconds=visibility_timeout_seconds),
+            last_error={
+                "queue_state": "in_progress",
+                "type": "false_dead_taken_back",
+                "message": "the reaper dead-lettered a job its worker was still running (an outage outlasted the "
+                "lease); the live copy took it back to finish the run",
+            },
+        )
+    ).updated_rows
+    if taken:
+        log.warning("job %s: the reaper dead-lettered it while this worker still ran it; taken back to finish the run",
+                    job_id[:8])
+    return bool(taken)
+
+
 async def reap_expired_leases(*, grace_seconds: int = 120) -> int:
     """Requeue jobs whose lease expired ≥ ``grace_seconds`` ago (audit P1-14).
 
@@ -731,7 +787,9 @@ async def reap_expired_leases(*, grace_seconds: int = 120) -> int:
     new copy at once, so the run never waits on its own earlier copy.
 
     Jobs with attempts left are requeued (claim_one re-claims and bumps
-    ``attempt``); exhausted jobs are dead-lettered with the same far-future
+    ``attempt``); exhausted jobs are dead-lettered (a live copy whose job was
+    falsely dead-lettered takes it back, ``take_back_false_dead_letter``, lane
+    K1b, so its run is never stranded) with the same far-future
     visible_at shape ``fail()`` uses. Returns how many rows were touched.
     """
     from matrx_graph.db._registry import get_model

{matrx_graph-0.2.331 → matrx_graph-0.2.332}/matrx_graph/workers/runner.py

@@ -328,6 +328,10 @@ class WorkerRunner:
             run_id=job.run_id,
             claimed_by=job.claimed_by,
             look_every=max(self._visibility_timeout // 3, 10) / 10,
+            owner=asyncio.current_task(),
+            visibility_timeout=self._visibility_timeout,
+            # the job's attempt: every re-claim bumps it, so a later copy's fencing token is always greater (K1b, B5)
+            generation=float(job.attempt or 0),
         )
         renewer = asyncio.create_task(
             self._renew_loop(job.job_id, claimed_by=job.claimed_by, lease=lease, owner=asyncio.current_task())
@@ -336,6 +340,7 @@ class WorkerRunner:
             app = await self._app_factory(job)
             with job_lease.holding(lease):
                 await self._run_job(job, app)
+            lease.settling = True  # a renewal answering "not ours" after this settles is not a loss (K1b, M1)
             await complete(job.job_id, claimed_by=job.claimed_by)
         except JobNeedsCapability as need:
             await hand_back_for_capability(job, need)
@@ -452,13 +457,19 @@ class WorkerRunner:
                 # slot and the restarted copy waited on itself (run 8749d414). Another copy may own the run now, so
                 # this one stops: the lease is marked lost (many-item work checks it before each item), the job
                 # task is cancelled, and the host skips its terminal write.
+                # Lane K1b, B1: when the reaper only DEAD-LETTERED the job (a false "dead" on its last attempt), no
+                # other copy can ever own the run, so this copy takes the job back and finishes it (stopping would
+                # strand the run: nothing would ever write its terminal status).
+                if lease is not None:
+                    if lease.settling:
+                        return
+                    if await lease.not_ours("the queue reaped or settled the job"):
+                        continue
                 log.error(
                     "job %s lease is no longer ours (reaped or deleted) — "
                     "this copy stops; the current holder owns the run",
                     job_id,
                 )
-                if lease is not None:
-                    lease.mark_lost("the queue reaped or settled the job")
                 if owner is not None and not owner.done():
                     owner.cancel()
                 return

{matrx_graph-0.2.331 → matrx_graph-0.2.332}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "matrx-graph"
-version = "0.2.331"
+version = "0.2.332"
 description = "Channel-based, super-step workflow orchestration engine for the Matrx ecosystem"
 readme = "README.md"
 license = { text = "MIT" }

matrx_graph-0.2.332/tests/test_a_lost_lease_never_strands_or_lingers.py

@@ -0,0 +1,191 @@
+"""Lane K1b (2026-10-21): a copy that is told "not ours" never strands its run, and a copy that really lost its job
+stops at once — whoever notices first.
+
+- B1: on a job's LAST attempt the reaper dead-letters (never requeues) an expired lease. After a database outage that
+  outlasted the lease, the live copy then read "not ours", cancelled itself and skipped its terminal write — and no
+  copy ever claims a dead-lettered job, so the run was stranded. Now the live copy takes the job back and finishes.
+  A job the reaper REQUEUED (another copy may run it) still stops the old copy.
+- B2: the work's own ``check`` (it reads the job row more often than the renewal) raised ``JobLeaseLost`` inside one
+  node; run-ahead swallowed it and the copy kept checkpointing and routing until the next renewal. Now marking the
+  lease lost cancels the copy's owner task at once.
+- M1: a renewal that lands just after ``complete()`` commits is not a lost lease.
+- M4: a copy with no claim token read every later claim as its own.
+
+The job row lives in an in-memory stand-in of ``WfJob`` that answers the queue's real queries.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import types
+from datetime import UTC, datetime, timedelta
+from types import SimpleNamespace
+
+import pytest
+
+from matrx_graph.db import _registry
+from matrx_graph.workers import job_lease
+from matrx_graph.workers import queue as queue_mod
+from matrx_graph.workers import runner as runner_mod
+
+JOB = "c1d89868-ef16-4d9b-82e6-dd5279775886"
+RUN = "8749d414-0000-4c39-9a5e-00000000k1bb"
+
+
+class _Jobs:
+    """One wf_job row, filtered and updated the way the ORM does for the lookups the queue uses."""
+
+    def __init__(self, **row):
+        self.row = {"id": JOB, "attempt": 3, "max_attempts": 3, **row}
+
+    def _match(self, filters):
+        for key, want in filters.items():
+            name, _, op = key.partition("__")
+            have = self.row.get(name)
+            if op == "isnull":
+                if (have is None) != bool(want):
+                    return False
+            elif op == "gt":
+                if have is None or not have > want:
+                    return False
+            elif op == "lt":
+                if have is None or not have < want:
+                    return False
+            elif have != want:
+                return False
+        return True
+
+    def filter(self, *_compare, **filters):
+        jobs = self
+
+        class _Q:
+            async def all(self, use_cache=True):  # noqa: ARG002
+                return [dict(jobs.row)] if jobs._match(filters) else []
+
+            async def update(self, **updates):
+                if not jobs._match(filters):
+                    return SimpleNamespace(updated_rows=[])
+                jobs.row.update(updates)
+                return SimpleNamespace(updated_rows=[dict(jobs.row)])
+
+        return _Q()
+
+    async def update_where(self, filters, **updates):
+        if not self._match(filters):
+            return SimpleNamespace(rows_affected=0)
+        self.row.update(updates)
+        return SimpleNamespace(rows_affected=1)
+
+
+def _dead_lettered_by_the_reaper() -> dict:
+    return {"claimed_at": None, "claimed_by": None, "visible_at": datetime.now(UTC) + timedelta(days=36_500),
+            "last_error": {"queue_state": "dead_letter", "type": "lease_expired", "retry_exhausted": True}}
+
+
+def _requeued_by_the_reaper() -> dict:
+    return {"claimed_at": None, "claimed_by": None, "visible_at": datetime.now(UTC),
+            "last_error": {"queue_state": "pending", "type": "lease_expired"}}
+
+
+@pytest.fixture
+def jobs(monkeypatch):
+    store = _Jobs(claimed_at=datetime.now(UTC), claimed_by="worker-1:A", visible_at=datetime.now(UTC) + timedelta(minutes=10),
+                  last_error=None)
+    monkeypatch.setattr(_registry, "get_model", lambda _key: store)
+
+    async def capture_error(*_a, **_k):
+        return None
+
+    real_sleep = asyncio.sleep
+    monkeypatch.setattr(runner_mod, "capture_error", capture_error)
+    monkeypatch.setattr(runner_mod, "asyncio", types.SimpleNamespace(sleep=lambda s: real_sleep(0.02),
+                                                                       Task=asyncio.Task, CancelledError=asyncio.CancelledError))
+    return store
+
+
+def _copy(jobs: _Jobs, *, work_seconds: float, reaper_turns_row_into: dict):
+    """One copy of the run as the worker runs it: the runner's real renewal loop beside the work; part way the reaper
+    rewrites the row (the outage outlasted the lease). Returns (outcome, lease)."""
+
+    async def main():
+        out: dict = {}
+        me = asyncio.current_task()
+        lease = job_lease.JobLease(job_id=JOB, run_id=RUN, claimed_by="worker-1:A", owner=me, visibility_timeout=600)
+        stub = types.SimpleNamespace(_visibility_timeout=30)
+        renewer = asyncio.create_task(runner_mod.WorkerRunner._renew_loop(stub, JOB, claimed_by="worker-1:A", lease=lease, owner=me))
+        try:
+            await asyncio.sleep(0.03)
+            jobs.row.update(reaper_turns_row_into)
+            await asyncio.sleep(work_seconds)
+            await job_lease.JobLease.check(lease)
+            out["outcome"] = "finished"
+        except asyncio.CancelledError:
+            out["outcome"] = "cancelled"
+        finally:
+            renewer.cancel()
+        return out["outcome"], lease
+
+    return asyncio.run(main())
+
+
+def test_a_false_dead_letter_on_the_last_attempt_is_taken_back_and_the_run_finishes(jobs):
+    outcome, lease = _copy(jobs, work_seconds=0.2, reaper_turns_row_into=_dead_lettered_by_the_reaper())
+    assert outcome == "finished", "the live copy stopped on a dead-lettered job: nothing would ever end its run"
+    assert not lease.lost
+    assert jobs.row["claimed_by"] == "worker-1:A" and jobs.row["claimed_at"] is not None, "the job was not taken back"
+    assert jobs.row["last_error"]["type"] == "false_dead_taken_back"
+
+
+def test_a_requeued_job_still_stops_the_old_copy(jobs):
+    outcome, lease = _copy(jobs, work_seconds=0.2, reaper_turns_row_into=_requeued_by_the_reaper())
+    assert outcome == "cancelled", "a copy whose job was requeued (another copy may run it) kept going"
+    assert lease.lost
+
+
+def test_a_job_another_worker_claimed_is_never_taken_back(jobs):
+    claimed = {"claimed_at": datetime.now(UTC), "claimed_by": "worker-2:B", "visible_at": datetime.now(UTC) + timedelta(minutes=10),
+               "last_error": None}
+    outcome, lease = _copy(jobs, work_seconds=0.2, reaper_turns_row_into=claimed)
+    assert outcome == "cancelled" and lease.lost
+    assert jobs.row["claimed_by"] == "worker-2:B"
+
+
+def test_the_work_noticing_the_loss_first_stops_the_whole_copy_at_once(jobs):
+    """B2: the node's own check raises inside a child task (run-ahead never reads its result); the copy's owner task
+    must be cancelled right then, not at the next renewal."""
+
+    async def main():
+        me = asyncio.current_task()
+        lease = job_lease.JobLease(job_id=JOB, run_id=RUN, claimed_by="worker-1:A", owner=me)
+        jobs.row.update(_requeued_by_the_reaper())
+        with job_lease.holding(lease):
+            node = asyncio.create_task(job_lease.check())  # a node in run-ahead: nobody awaits its result
+        try:
+            await asyncio.sleep(5)  # the copy would go on to checkpoint and route
+            return "kept going", node
+        except asyncio.CancelledError:
+            return "stopped", node
+
+    outcome, node = asyncio.run(main())
+    assert outcome == "stopped"
+    assert node.done()
+
+
+def test_a_renewal_landing_after_the_settle_is_not_a_loss(jobs):
+    async def main():
+        me = asyncio.current_task()
+        lease = job_lease.JobLease(job_id=JOB, run_id=RUN, claimed_by="worker-1:A", owner=me)
+        lease.settling = True
+        jobs.row.update({"claimed_at": None, "claimed_by": None, "last_error": {"queue_state": "succeeded"}})
+        kept = await lease.not_ours("the queue reaped or settled the job")
+        await asyncio.sleep(0)
+        return kept, lease
+
+    kept, lease = asyncio.run(main())
+    assert kept and not lease.lost
+
+
+def test_a_copy_without_a_claim_token_sees_another_claim(jobs):
+    jobs.row.update({"claimed_at": datetime.now(UTC), "claimed_by": "worker-2:B"})
+    lease = job_lease.JobLease(job_id=JOB, run_id=RUN, claimed_by=None)
+    assert asyncio.run(lease.still_held()) is False