matrx-graph 0.2.331__py3-none-any.whl → 0.2.332__py3-none-any.whl

matrx_graph/executor/scheduler.py

@@ -2181,6 +2181,11 @@ class Scheduler:
         try:
             while open_members:
                 task, result, result_step = await done_queue.get()
+                if task.cancelled() and _job_lease().lost():
+                    # A member stopped because this copy lost its job lease (``JobLeaseLost`` is a cancellation):
+                    # the whole copy stops here, never routing or checkpointing on (lane K1b, B2).
+                    lease = _job_lease().current()
+                    raise _job_lease().JobLeaseLost(f"job lease lost: {lease.why if lease else ''}")
                 if id(result) in member_order and result_step == step:
                     # Kept in frontier order by insertion — O(log n) search per finished member, never a re-sort.
                     bisect.insort(finished_members, result, key=lambda r: member_order[id(r)])
@@ -6490,3 +6495,10 @@ def _settle_default_on_decision_gate(
     logger.info("decision gate settled by its default %r → %s", raw, shaped["answer"])
     value["answer"] = shaped["answer"]
     return value
+
+
+def _job_lease():
+    """``matrx_graph.workers.job_lease``, imported on use (the workers package imports the executor)."""
+    from matrx_graph.workers import job_lease
+
+    return job_lease

matrx_graph/workers/FEATURE.md

@@ -17,7 +17,7 @@ and `workflow.job`'s post-reorg baseline schema on 2026-08-09.
 - **Jobs run in parallel, bounded, each in its own task (2026-09-17).** `WorkerRunner` holds an `asyncio.Semaphore(max_concurrent)`: it keeps claiming while slots are free, waits on the *clock* (`poll_interval`) only when the queue is empty, and waits on a *slot* when every one is busy. Each claimed job runs as its own `asyncio.Task` with its own lease renewer, and `complete`/`fail`/cancel semantics are per job and unchanged. **`max_concurrent` defaults to 1** — a host that has not thought about the bound keeps exactly the behavior it had; aidream passes `workflow_worker.max_concurrent_jobs` (a `platform.feature_knob` row, currently 8, read once at worker start). Why: the loop was strictly serial, so ONE slow job held every other workflow on the platform. Measured in production 2026-09-17, five simultaneous Cloud Browser activations (each a workflow job that waits on a container coming up) came up as a 5.5-second-per-run ladder, and for those 22 seconds nothing else in the queue moved; at the 50-run fleet ceiling that is a ~4.5-minute tail. **Per-job isolation is structural, not conventional**: a job task is created from the claim loop's context, so `contextvars` it sets (an ambient `AppContext`, an ORM connection) belong to that task alone — they cannot reach a sibling job and cannot outlive the job in the loop. Under the old serial `await` they did: aidream's `_run_one` installs an ambient AppContext at the tail of every job, and that context stayed installed in the claim loop afterwards. Proof: `tests/test_workers.py` — the wall-clock test, the peak-concurrency test, the neighbour-isolation test, the two drain tests, and the context-isolation test (each demonstrated failing against the serial loop before the change).
 - **Stopping is a bounded drain.** `stop()` takes no new claims; `run_forever()` then waits out the jobs already running for `stop_grace_seconds` (aidream passes its share of THE DEPLOY WINDOW) and cancels whatever is left. A canceled job fails retryable and its run resumes from its last checkpoint — the same outcome a serial worker's single job always had when the container was replaced.
 - **A worker only claims what it can run (2026-10-12, `capability.py`).** Every server — deployed fleet and developer laptops — shares one database, so a job may belong to a graph whose node types this build has not shipped (live: a local build's `brand.setup.land_logo` run was resumed by production and errored "Unknown node type"). `enqueue()` / `enqueue_claimed()` stamp `workflow.job.metadata.required_node_types` (canonical spellings, removed types excluded) from the run's stored definition; `claim_one(capabilities=...)` adds `COALESCE(j.metadata->'required_node_types','[]') <@ caps` (`matrx_orm.JsonbContainedBy`), so an unstamped legacy job stays claimable by anyone. `WorkerRunner` and `run_single_job` pass `worker_capabilities()` (the default registry) on every claim. Belt two: a host job runner that finds a gap at run time raises `JobNeedsCapability`; the runner calls `defer_for_capability()` (un-claim, attempt refunded, requirement stamped, immediately visible) — never `fail()`, never `complete()`, never a run-status write. The run says what it waits for in `workflow.run.metadata._awaiting_capability` (`missing`, `message`, `since`, `checked_at`, `checked_by`) plus one `workflow.run_log` line (`run_awaiting_capability`); every incapable worker refreshes it via `stamp_runs_awaiting_capability()` and `checked_at` is a lifecycle-watchdog liveness signal, so a waiting run is not force-failed as stranded while a fleet is watching it. The first capable worker clears the marker. Proof: `tests/test_capability_aware_claim.py` (incl. the predicate evaluated in live Postgres; fails against the pre-change queue/runner).
-- **A copy that loses its lease stops itself (lane K1, 2026-10-21).** A renewal that answers "not ours" (reaped — including a FALSE "dead" after a database outage longer than the lease, live job c1d89868 — or settled elsewhere) marks the copy's `job_lease.JobLease` lost and cancels its task; the host skips its terminal write (the copy that holds the job owns the run); many-item work calls `job_lease.check()` before each item (it also reads the job row, at most every tenth of a renewal beat). The reaper does not try to tell an outage from a death: a false "dead" is harmless by construction. Proof: aidream `aidream/services/tests/test_a_reaped_run_never_waits_on_itself.py`.
+- **A copy that loses its lease stops itself (lane K1, 2026-10-21).** A renewal that answers "not ours" (reaped — including a FALSE "dead" after a database outage longer than the lease, live job c1d89868 — or settled elsewhere) marks the copy's `job_lease.JobLease` lost and cancels its task; the host skips its terminal write (the copy that holds the job owns the run); many-item work calls `job_lease.check()` before each item (it also reads the job row, at most every tenth of a renewal beat). The reaper does not try to tell an outage from a death: a false "dead" is harmless by construction. Proof: aidream `aidream/services/tests/test_a_reaped_run_never_waits_on_itself.py`. **Lane K1b:** whoever notices the loss first (the renewal or the work's own `check`) cancels the copy's owner task, and run-ahead re-raises a member that ended cancelled on a lost lease (it used to swallow it and route on). A job the reaper only DEAD-LETTERED (a false "dead" on its last attempt) is taken back by the live copy (`queue.take_back_false_dead_letter`, compare-and-set) and finished — stopping would strand the run, since nothing claims a dead-lettered job; a requeued or re-claimed job still stops the old copy. A renewal landing after `complete()` is not a loss (`JobLease.settling`). Each copy's `JobLease.generation` (the job's attempt) fences what it holds (import slots). A platform-wide outage longer than the lease plus grace cancels EVERY in-flight job at once and they all restart from their checkpoints together: the same spend per job as before, but as one burst. Proof: `tests/test_a_lost_lease_never_strands_or_lingers.py`.
 - **Recovery starts before work.** `WorkerRunner.run_forever()` calls `reap_expired_leases()` before its first claim. The host's recurring reaper remains a backstop.
 - **A transient claim outage never kills the worker.** A database timeout or
   connection failure while claiming `workflow.job` is captured durably and

matrx_graph/workers/job_lease.py

@@ -41,15 +41,31 @@ class JobLease:
     #: fewest seconds between two reads of the job row by ``check`` (a many-item run checks before every item); the
     #: runner sets it from its lease (a tenth of the renewal beat), and the renewal's own answer is used in between
     look_every: float = 0.0
+    #: the task running this copy of the job: whoever notices the loss first (the renewal, or the work's own
+    #: ``check``) cancels it, so the WHOLE copy stops at once (lane K1b, B2: run-ahead swallowed a ``JobLeaseLost``
+    #: raised inside one node, and the copy kept checkpointing and routing until the next renewal)
+    owner: asyncio.Task | None = field(default=None, repr=False)
+    #: the copy finished its work and is settling the job (``complete``): a renewal that lands after the settle
+    #: commits answers "not ours", and that is not a loss (K1b, M1)
+    settling: bool = False
+    #: per-copy fencing token for anything this copy holds (an import slot): a newer copy's is always greater
+    generation: float = field(default_factory=time.time)
+    visibility_timeout: int = 600
     lost: bool = False
     why: str = ""
     _looked: float = field(default=0.0, repr=False)
 
     def mark_lost(self, why: str) -> None:
+        if self.settling:
+            log.info("job %s: settled while a renewal was in flight; not a lost lease", self.job_id[:8])
+            return
         if not self.lost:
             log.error("job %s (run %s) lost its lease (%s): this copy stops and writes nothing more",
                       self.job_id[:8], self.run_id[:8], why)
         self.lost, self.why = True, why
+        owner = self.owner
+        if owner is not None and not owner.done() and owner is not _running_task():
+            owner.cancel(f"job {self.job_id[:8]} lease lost: {why}")
 
     async def still_held(self) -> bool | None:
         """The job row's answer: True while this claim holds it, False once it does not, None when unreadable."""
@@ -66,7 +82,28 @@ class JobLease:
         get = row.get if isinstance(row, dict) else lambda k: getattr(row, k, None)
         if get("claimed_at") is None:
             return False
-        return self.claimed_by is None or get("claimed_by") == self.claimed_by
+        # every claim stamps a token: a token that is not this copy's (or one where this copy had none) is another
+        # claim (K1b, M4: a token-less copy used to read every later claim as its own)
+        return get("claimed_by") == self.claimed_by
+
+    async def not_ours(self, why: str) -> bool:
+        """The queue says this copy no longer holds its job. When the reaper only DEAD-LETTERED it (a false "dead" on
+        the last attempt), no other copy can ever own the run: take the job back and keep going, so the run is
+        finished and its terminal status written (K1b, B1). Otherwise mark the lease lost (which stops the copy).
+        Returns True when the copy keeps the job."""
+        from matrx_graph.workers.queue import take_back_false_dead_letter
+
+        if self.settling:
+            return True
+        try:
+            kept = await take_back_false_dead_letter(
+                self.job_id, claimed_by=self.claimed_by, visibility_timeout_seconds=self.visibility_timeout)
+        except Exception:  # noqa: BLE001 — unknown: the copy stops (the safe side for a run another copy may own)
+            log.warning("job %s: could not try to take back a dead-lettered job", self.job_id[:8], exc_info=True)
+            kept = False
+        if not kept:
+            self.mark_lost(why)
+        return kept
 
     async def check(self) -> None:
         """Raise :class:`JobLeaseLost` when this copy no longer holds its job; read the row at most every
@@ -74,11 +111,18 @@ class JobLease:
         if not self.lost and time.monotonic() - self._looked >= self.look_every:
             self._looked = time.monotonic()
             if await self.still_held() is False:
-                self.mark_lost("the job row is claimed by another copy or was requeued")
+                await self.not_ours("the job row is claimed by another copy or was requeued")
         if self.lost:
             raise JobLeaseLost(f"job {self.job_id[:8]} lease lost: {self.why}")
 
 
+def _running_task() -> asyncio.Task | None:
+    try:
+        return asyncio.current_task()
+    except RuntimeError:
+        return None
+
+
 _current: ContextVar[JobLease | None] = ContextVar("matrx_graph_job_lease", default=None)
 
 

matrx_graph/workers/queue.py

@@ -707,6 +707,62 @@ async def renew_lease(
     return result.rows_affected > 0
 
 
+async def take_back_false_dead_letter(
+    job_id: str,
+    *,
+    claimed_by: str | None,
+    visibility_timeout_seconds: int = 600,
+) -> bool:
+    """A live copy whose job the reaper DEAD-LETTERED (a false "dead": an outage outlasted the lease on the job's last
+    attempt) takes the job back, so it finishes the run and writes its terminal status (lane K1b, B1, 2026-10-21).
+
+    Without this the live copy read "not ours", cancelled itself and skipped its terminal write, and no other copy would
+    ever claim a dead-lettered job: the run stayed non-terminal (stranded) for every job kind. A dead-lettered job has
+    no other owner, so taking it back can never run two copies. Compare-and-set: only a row that is still unclaimed,
+    still dead-lettered BY THE REAPER (``last_error.type == 'lease_expired'``) and still parked in the far future
+    (an operator's retry moves ``visible_at`` back) is taken; anything else answers False and the copy stops.
+    """
+    from matrx_graph.db._registry import get_model
+
+    model = get_model("WfJob")
+    try:
+        rows = await model.filter(id=str(UUID(job_id))).all(use_cache=False)
+    except Exception:  # noqa: BLE001 — unreadable: not taken back (the caller decides on the renewal's answer)
+        log.warning("could not read job %s to see whether the reaper dead-lettered it", job_id[:8], exc_info=True)
+        return False
+    if not rows:
+        return False
+    row = rows[0]
+    get = row.get if isinstance(row, dict) else lambda k: getattr(row, k, None)
+    note = _parse_json(get("last_error")) or {}
+    if get("claimed_at") is not None or not isinstance(note, dict):
+        return False
+    if note.get("queue_state") != "dead_letter" or note.get("type") != "lease_expired":
+        return False
+    now = datetime.now(UTC)
+    taken = (
+        await model.filter(
+            id=str(UUID(job_id)),
+            claimed_at__isnull=True,
+            visible_at__gt=now + timedelta(days=_FAR_FUTURE_DAYS // 2),
+        ).update(
+            claimed_at=now,
+            claimed_by=claimed_by,
+            visible_at=now + timedelta(seconds=visibility_timeout_seconds),
+            last_error={
+                "queue_state": "in_progress",
+                "type": "false_dead_taken_back",
+                "message": "the reaper dead-lettered a job its worker was still running (an outage outlasted the "
+                "lease); the live copy took it back to finish the run",
+            },
+        )
+    ).updated_rows
+    if taken:
+        log.warning("job %s: the reaper dead-lettered it while this worker still ran it; taken back to finish the run",
+                    job_id[:8])
+    return bool(taken)
+
+
 async def reap_expired_leases(*, grace_seconds: int = 120) -> int:
     """Requeue jobs whose lease expired ≥ ``grace_seconds`` ago (audit P1-14).
 
@@ -731,7 +787,9 @@ async def reap_expired_leases(*, grace_seconds: int = 120) -> int:
     new copy at once, so the run never waits on its own earlier copy.
 
     Jobs with attempts left are requeued (claim_one re-claims and bumps
-    ``attempt``); exhausted jobs are dead-lettered with the same far-future
+    ``attempt``); exhausted jobs are dead-lettered (a live copy whose job was
+    falsely dead-lettered takes it back, ``take_back_false_dead_letter``, lane
+    K1b, so its run is never stranded) with the same far-future
     visible_at shape ``fail()`` uses. Returns how many rows were touched.
     """
     from matrx_graph.db._registry import get_model

matrx_graph/workers/runner.py

@@ -328,6 +328,10 @@ class WorkerRunner:
             run_id=job.run_id,
             claimed_by=job.claimed_by,
             look_every=max(self._visibility_timeout // 3, 10) / 10,
+            owner=asyncio.current_task(),
+            visibility_timeout=self._visibility_timeout,
+            # the job's attempt: every re-claim bumps it, so a later copy's fencing token is always greater (K1b, B5)
+            generation=float(job.attempt or 0),
         )
         renewer = asyncio.create_task(
             self._renew_loop(job.job_id, claimed_by=job.claimed_by, lease=lease, owner=asyncio.current_task())
@@ -336,6 +340,7 @@ class WorkerRunner:
             app = await self._app_factory(job)
             with job_lease.holding(lease):
                 await self._run_job(job, app)
+            lease.settling = True  # a renewal answering "not ours" after this settles is not a loss (K1b, M1)
             await complete(job.job_id, claimed_by=job.claimed_by)
         except JobNeedsCapability as need:
             await hand_back_for_capability(job, need)
@@ -452,13 +457,19 @@ class WorkerRunner:
                 # slot and the restarted copy waited on itself (run 8749d414). Another copy may own the run now, so
                 # this one stops: the lease is marked lost (many-item work checks it before each item), the job
                 # task is cancelled, and the host skips its terminal write.
+                # Lane K1b, B1: when the reaper only DEAD-LETTERED the job (a false "dead" on its last attempt), no
+                # other copy can ever own the run, so this copy takes the job back and finishes it (stopping would
+                # strand the run: nothing would ever write its terminal status).
+                if lease is not None:
+                    if lease.settling:
+                        return
+                    if await lease.not_ours("the queue reaped or settled the job"):
+                        continue
                 log.error(
                     "job %s lease is no longer ours (reaped or deleted) — "
                     "this copy stops; the current holder owns the run",
                     job_id,
                 )
-                if lease is not None:
-                    lease.mark_lost("the queue reaped or settled the job")
                 if owner is not None and not owner.done():
                     owner.cancel()
                 return

{matrx_graph-0.2.331.dist-info → matrx_graph-0.2.332.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: matrx-graph
-Version: 0.2.331
+Version: 0.2.332
 Summary: Channel-based, super-step workflow orchestration engine for the Matrx ecosystem
 Project-URL: Homepage, https://github.com/AI-Matrix-Engine/aidream-current
 Project-URL: Repository, https://github.com/AI-Matrix-Engine/aidream-current

{matrx_graph-0.2.331.dist-info → matrx_graph-0.2.332.dist-info}/RECORD

@@ -147,7 +147,7 @@ matrx_graph/executor/resume_chain.py,sha256=YJv8P8Ca1wSAsUEjeii3Tg6Fgoy5N9Tf2Gs4
 matrx_graph/executor/resume_preflight.py,sha256=vz9_01QINvgwK9u2y7NcF7B9J2Yv7K1jHalRuZD7siw,11977
 matrx_graph/executor/review_flags.py,sha256=hMR8tPrSd_-vgsLiTc82MmEaIx-sMkC5yf53zIbhh8M,5542
 matrx_graph/executor/run_stop.py,sha256=HpdGE83MabnU_4rnf_Vn8zRyk5LBF3qbZQ2Ouvc36hY,3004
-matrx_graph/executor/scheduler.py,sha256=gTVz-B-N8CfDhZpEWeliHvCpmU-geQCDTzMmWo2nO5Q,322574
+matrx_graph/executor/scheduler.py,sha256=1ezi7TYpY-tLSk-dsGjuk38WIK3ivdPw8wUDFVCdLfw,323193
 matrx_graph/executor/schema_validation.py,sha256=X2CbDX7_vZGBtcw_L0BQcjDD-KEAxnGA8ruhm9Qp5CI,5250
 matrx_graph/nodes/__init__.py,sha256=ikOsSIu8aaSYAZXa_Uks-RW_BkIjwiO1TwTptLom0bY,10118
 matrx_graph/nodes/_sandbox.py,sha256=Uizb2QIbW9MgZ_4MtWoJlmjR1mMXZNb9oMYGhXNXeQ0,10517
@@ -246,15 +246,15 @@ matrx_graph/types/send.py,sha256=sKE_moS4uTB4P3L02Q7uKj0WhTzABm3vTk2o54EpoPA,254
 matrx_graph/types/usl.py,sha256=-zXXiHZpxpv7V7n_xgdcnY2l8XlWoM9ZUQNkwHYmqiQ,3376
 matrx_graph/types/value_objects.py,sha256=ylVpyj-_YQ2xhMAHVW2AdRo5yqzgXLajGUuaIZhJFQY,1295
 matrx_graph/types/variables.py,sha256=pCECEaDP3y2qwZkIlhpICRucFTMHRLW8wU3zXQBooFY,10896
-matrx_graph/workers/FEATURE.md,sha256=EQUOHh4UGkSsi2-7sdtxYivjHxYcwT3Ne84HJjFCM2I,17549
+matrx_graph/workers/FEATURE.md,sha256=yZXybUsAbUkYDwXBNnzoJXGCedQmVK_BQCZ3AXDlogk,18502
 matrx_graph/workers/__init__.py,sha256=G8e24ryKLafH_NggiiGIb9EZ7YparOJ5DaBOw0bbPv0,2888
 matrx_graph/workers/capability.py,sha256=9kFjZCfuie5GkevLLTBJVJ3m9NeeS36RJhbjK0r86fw,11058
 matrx_graph/workers/cron_watcher.py,sha256=JVzUJ7Wmtz9PPwUtkxnSyqkheK3nEM-7lHUlecdazYY,3988
 matrx_graph/workers/event_watcher.py,sha256=DYk4RgapiJJDyKY8zirBpp3HAlqi7BWExf4rWW97FHY,6263
-matrx_graph/workers/job_lease.py,sha256=WZfzKVoHKgywXPODJNQJl2RQL03l-JGBjdnfAKOthr8,4693
-matrx_graph/workers/queue.py,sha256=R31UGRb4LDOvjxVACYyP1oBOe6_0LYRhZ8lg2EZHy18,29586
-matrx_graph/workers/runner.py,sha256=574Ch6fMcZun00pQKT5rTjYFBHVeh-zJFOMDJD-VO1Y,23146
+matrx_graph/workers/job_lease.py,sha256=pYZidqPB4hgHa8KKZqSJDGetsW7Wm_aVwSXM0ycrrfY,7213
+matrx_graph/workers/queue.py,sha256=GGzuJju1DvrZFPbfvGQEMuWay-2m18NzZPNlWligS4Q,32413
+matrx_graph/workers/runner.py,sha256=6ALoNst6LY9pK-8I7r0QPLjdKpSfyimp5JwFb1PFdmI,23946
 matrx_graph/workers/sleep_wake.py,sha256=1V-opM2zYtuN3n4ez85C4trYHoEapVwz8F7_MAex8HI,5970
-matrx_graph-0.2.331.dist-info/METADATA,sha256=eVmqsKOWrcK1FIknWrxajPcHoH8q_8Zd5v6qu46Qhng,5480
-matrx_graph-0.2.331.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
-matrx_graph-0.2.331.dist-info/RECORD,,
+matrx_graph-0.2.332.dist-info/METADATA,sha256=Xtv8QD6KJcjVl9GHAwEdySTwocg45udFubgG2r2VWYk,5480
+matrx_graph-0.2.332.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
+matrx_graph-0.2.332.dist-info/RECORD,,