AbstractRuntime 0.15.2__tar.gz → 0.15.3__tar.gz

{abstractruntime-0.15.2 → abstractruntime-0.15.3}/CHANGELOG.md

@@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.15.3] - 2026-10-29
+
+### Fixed
+
+- **An index build no longer announces its journal replay as changes** (`JsonFileRunStore`; a 0.15.2 regression).
+  0.15.2's first sync after a build re-applies the journal tail before the build's mark (backlog 0863) and noted every
+  replayed record on the change ring: a plain read (`list_run_index`, `warm_session_index`) moved `change_token` and
+  `changed_run_ids` then named every run in the tail as changed, so a change-driven consumer re-read every run after
+  each build (the gateway's mail notification check loaded all 400 finished runs of its index test on every full
+  pass: AbstractGateway backlog 0965, red CI since 0.15.2). Replayed records are still filed into the session,
+  children and active indexes (the 0863 guarantee stands). A replay is quiet only when it says nothing new: its file
+  was known to the store at the build's mark (the scan memo, the sidecar included) and is unchanged since; a file
+  that landed or changed during the build (the record precedes the file, J53-1), a known run now gone, and every
+  record after the mark are announced as before, so a consumer whose token already covers the record still learns
+  of it; a store with no sidecar announces its first replay (a bounded one-off). Tests:
+  `tests/test_json_file_run_store_change_token_after_build.py` (red on 0.15.2).
+
 ## [0.15.2] - 2026-10-28
 
 ### Fixed

{abstractruntime-0.15.2 → abstractruntime-0.15.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: AbstractRuntime
-Version: 0.15.2
+Version: 0.15.3
 Summary: AbstractRuntime: a durable graph runner designed to pair with AbstractCore.
 Project-URL: AbstractCore (website), https://www.abstractcore.ai/
 Project-URL: AbstractRuntime (GitHub), https://github.com/lpalbou/abstractruntime
@@ -55,10 +55,17 @@ Description-Content-Type: text/markdown
 
 It is designed for long-running workflows that must survive restarts and explicitly model blocking (human input, timers, external events, subworkflows) without keeping Python stacks alive.
 
-**Version:** 0.15.2 • **Python:** 3.10+
+**Version:** 0.15.3 • **Python:** 3.10+
 
 **Status:** pre-1.0 (API may evolve). For production use, pin versions and follow `CHANGELOG.md`.
 
+## What's new in 0.15.3
+
+- **A run-store index build no longer announces its journal replay as changes**: a plain read (`list_run_index`,
+  `warm_session_index`) leaves `change_token` and `changed_run_ids` untouched, so a change-driven consumer (the
+  gateway's mail notification check) no longer re-reads every finished run after each build (a 0.15.2 regression;
+  AbstractGateway backlog 0965). A file that landed during the build and a known run that vanished are still announced.
+
 ## What's new in 0.15.2
 
 - **Run index builds never lose a run created during them**: a run another process (or store object) wrote while

{abstractruntime-0.15.2 → abstractruntime-0.15.3}/README.md

@@ -4,10 +4,17 @@
 
 It is designed for long-running workflows that must survive restarts and explicitly model blocking (human input, timers, external events, subworkflows) without keeping Python stacks alive.
 
-**Version:** 0.15.2 • **Python:** 3.10+
+**Version:** 0.15.3 • **Python:** 3.10+
 
 **Status:** pre-1.0 (API may evolve). For production use, pin versions and follow `CHANGELOG.md`.
 
+## What's new in 0.15.3
+
+- **A run-store index build no longer announces its journal replay as changes**: a plain read (`list_run_index`,
+  `warm_session_index`) leaves `change_token` and `changed_run_ids` untouched, so a change-driven consumer (the
+  gateway's mail notification check) no longer re-reads every finished run after each build (a 0.15.2 regression;
+  AbstractGateway backlog 0965). A file that landed during the build and a known run that vanished are still announced.
+
 ## What's new in 0.15.2
 
 - **Run index builds never lose a run created during them**: a run another process (or store object) wrote while

{abstractruntime-0.15.2 → abstractruntime-0.15.3}/llms-full.txt

@@ -55,10 +55,17 @@ See llms.txt for the linked index.
 
 It is designed for long-running workflows that must survive restarts and explicitly model blocking (human input, timers, external events, subworkflows) without keeping Python stacks alive.
 
-**Version:** 0.15.2 • **Python:** 3.10+
+**Version:** 0.15.3 • **Python:** 3.10+
 
 **Status:** pre-1.0 (API may evolve). For production use, pin versions and follow `CHANGELOG.md`.
 
+## What's new in 0.15.3
+
+- **A run-store index build no longer announces its journal replay as changes**: a plain read (`list_run_index`,
+  `warm_session_index`) leaves `change_token` and `changed_run_ids` untouched, so a change-driven consumer (the
+  gateway's mail notification check) no longer re-reads every finished run after each build (a 0.15.2 regression;
+  AbstractGateway backlog 0965). A file that landed during the build and a known run that vanished are still announced.
+
 ## What's new in 0.15.2
 
 - **Run index builds never lose a run created during them**: a run another process (or store object) wrote while

{abstractruntime-0.15.2 → abstractruntime-0.15.3}/llms.txt

@@ -3,7 +3,7 @@
 > Durable workflow runtime (interrupt → checkpoint → resume) with an append-only execution ledger.
 
 - Python: 3.10+ (see `pyproject.toml`)
-- Version: 0.15.2
+- Version: 0.15.3
 - Ecosystem: [AbstractFramework](https://github.com/lpalbou/AbstractFramework) umbrella, integrates with [AbstractCore](https://github.com/lpalbou/abstractcore)
 - Hard invariant: `RunState.vars` must be JSON-serializable (`src/abstractruntime/core/models.py`)
 - Current AbstractCore baseline: base `abstractruntime` requires AbstractCore 2.30.0 or newer with its light install (remote providers, tools and the MCP worker, media routing); `abstractruntime[apple]` and `abstractruntime[gpu]` require AbstractCore's `apple` and `gpu` settings (`pyproject.toml`)

{abstractruntime-0.15.2 → abstractruntime-0.15.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "AbstractRuntime"
-version = "0.15.2"
+version = "0.15.3"
 description = "AbstractRuntime: a durable graph runner designed to pair with AbstractCore."
 readme = "README.md"
 requires-python = ">=3.10"

abstractruntime-0.15.3/release-notes.md

@@ -0,0 +1,14 @@
+### Fixed
+
+- **An index build no longer announces its journal replay as changes** (`JsonFileRunStore`; a 0.15.2 regression).
+  0.15.2's first sync after a build re-applies the journal tail before the build's mark (backlog 0863) and noted every
+  replayed record on the change ring: a plain read (`list_run_index`, `warm_session_index`) moved `change_token` and
+  `changed_run_ids` then named every run in the tail as changed, so a change-driven consumer re-read every run after
+  each build (the gateway's mail notification check loaded all 400 finished runs of its index test on every full
+  pass: AbstractGateway backlog 0965, red CI since 0.15.2). Replayed records are still filed into the session,
+  children and active indexes (the 0863 guarantee stands). A replay is quiet only when it says nothing new: its file
+  was known to the store at the build's mark (the scan memo, the sidecar included) and is unchanged since; a file
+  that landed or changed during the build (the record precedes the file, J53-1), a known run now gone, and every
+  record after the mark are announced as before, so a consumer whose token already covers the record still learns
+  of it; a store with no sidecar announces its first replay (a bounded one-off). Tests:
+  `tests/test_json_file_run_store_change_token_after_build.py` (red on 0.15.2).

{abstractruntime-0.15.2 → abstractruntime-0.15.3}/src/abstractruntime/storage/json_files.py

@@ -188,6 +188,16 @@ class JsonFileRunStore(RunStore):
         self._journal_path = self._base / ".runs_created.log"
         self._journal_lock = threading.Lock()
         self._journal_offset: Optional[int] = None
+        # The journal's size when an index build marked it (`_mark_journal_start`): records
+        # before it are REPLAYED by the first sync (filed into the indexes, nothing announced:
+        # they predate the build's scan, which already read their files, or are this store's own
+        # saves, announced when they happened). Only records after it are changes.
+        self._journal_replay_mark: int = 0
+        # What this store knew when it marked: {run id: file identity token} from the scan memo
+        # (the sidecar included). A replayed record is quiet only for a file known then and
+        # unchanged since; anything else (a file that landed or changed during the build, a known
+        # run now gone) is announced like any change.
+        self._journal_replay_known: Dict[str, tuple] = {}
         from collections import deque as _deque
 
         self._journal_own_appends: Any = _deque(maxlen=8192)
@@ -747,10 +757,17 @@ class JsonFileRunStore(RunStore):
         """Called BEFORE an index's full scan: entries appended from here on
         are re-applied by `_sync_indexes` (idempotent), so nothing created
         during the scan is lost."""
+        if self._journal_offset is not None:
+            return
+        self._load_scan_sidecar_once()  # the memo as a restarted store knows it, before the snapshot
+        with self._scan_memo_lock:
+            known = {rid: tuple(v[0]) for rid, v in self._scan_memo.items()}
         with self._journal_lock:
             if self._journal_offset is None:
                 size = self._journal_size()
                 self._journal_offset = self._journal_replay_start(size) if size is not None else 0
+                self._journal_replay_mark = int(size) if size is not None else 0
+                self._journal_replay_known = known
 
     def _journal_replay_start(self, size: int) -> int:
         """Where the first sync after an index build starts reading: the first record boundary
@@ -824,24 +841,57 @@ class JsonFileRunStore(RunStore):
                 return
             complete = chunk.rfind(b"\n") + 1  # a line still being written waits for the next sync
             self._journal_offset = offset + complete
+            mark = self._journal_replay_mark
             pending = dict(self._journal_pending)
         entries = list(pending.items())
         now = time.time()
-        for raw in chunk[:complete].decode("utf-8", "replace").splitlines():
+        # A record wholly before the build's mark is a replay (0.15.2): it is filed, never
+        # announced as a change (`change_token` / `changed_run_ids` stay put: a read that moved
+        # them made every change-driven consumer re-read every run after each build, 0.15.3).
+        replayed: set = set()
+        announced: set = set()
+        pos = offset
+        for raw_line in chunk[:complete].split(b"\n"):
+            line_end = pos + len(raw_line) + 1
+            raw = raw_line.decode("utf-8", "replace")
             op, rid = raw[:1], raw[1:].strip()
             if op in ("+", "-", "~", "=") and rid:
                 entries.append((rid, (op, now)))
-        self._apply_journal_entries(entries)
-
-    def _apply_journal_entries(self, entries: Any) -> None:
+                (replayed if line_end <= mark else announced).add(rid)
+            pos = line_end
+        self._apply_journal_entries(entries, quiet=replayed - announced)
+        if offset + complete >= mark:
+            self._journal_replay_known = {}  # the replay is over: later records are all new
+
+    def _apply_journal_entries(self, entries: Any, *, quiet: Any = frozenset()) -> None:
+        """`quiet`: run ids whose records are replays (an index build's tail). A replay is filed
+        without a change-ring note only when it says nothing new: its file was known at the mark
+        (`_journal_replay_known`) and is unchanged since; a known run now gone IS announced. A file
+        that landed or changed during the build (J53-1: the record precedes the file) is announced
+        like any change, so a consumer whose token already covers the record still learns of it."""
         cutoff = time.time() - float(self.STALE_TEMP_AGE_S)
         for rid, (op, since) in entries:
+            note = rid not in quiet
             p = self._path(rid)
             exists = p.exists()
+            if not note:
+                known = self._journal_replay_known.get(rid)
+                if exists:
+                    try:
+                        unchanged = known is not None and tuple(known) == tuple(self._stat_token(p.stat()))
+                    except OSError:
+                        unchanged = False
+                    note = not unchanged
+                else:
+                    with self._scan_memo_lock:
+                        in_memo = rid in self._scan_memo
+                    with self._active_lock:
+                        in_active = rid in self._active
+                    note = known is not None or in_memo or in_active
             if op in ("+", "~", "=") and exists:
                 fields = self._scan_fields(p)
                 if fields is not None:
-                    self._index_fields_into_indexes(rid, fields)
+                    self._index_fields_into_indexes(rid, fields, note=note)
                 # `~` announces a flip TO active: until the file says so it is
                 # not resolved (the writer's replace may land after its line).
                 # `=` is written AFTER its file was published: one read resolves it.
@@ -849,7 +899,7 @@ class JsonFileRunStore(RunStore):
             elif op == "-" and not exists:
                 self._drop_from_children_index(rid)
                 self._drop_from_session_index(rid)
-                self._forget_active(rid)
+                self._forget_active(rid, note=note)
                 self._memo_pop(rid)
                 resolved = True
             elif op == "=":
@@ -862,7 +912,7 @@ class JsonFileRunStore(RunStore):
                 else:
                     self._journal_pending[rid] = (op, since)
 
-    def _index_fields_into_indexes(self, rid: str, fields: Dict[str, Any]) -> None:
+    def _index_fields_into_indexes(self, rid: str, fields: Dict[str, Any], *, note: bool = True) -> None:
         parent = fields.get("parent_run_id")
         parent = str(parent) if parent else None
         with self._index_lock:
@@ -875,7 +925,7 @@ class JsonFileRunStore(RunStore):
             if self._session_index is not None and sid:
                 self._session_index.setdefault(sid, {})[rid] = fields.get("session_kind")
                 self._run_session[rid] = sid
-        self._note_active_status(rid, fields.get("status"))
+        self._note_active_status(rid, fields.get("status"), note=note)
 
     def warm_session_index(self) -> None:
         """Build the session and children indexes now (host boot), so the
@@ -1267,24 +1317,37 @@ class JsonFileRunStore(RunStore):
             e = self._active[rid] = _ActiveEntry()
         return e
 
-    def _note_active_status(self, rid: str, status: Any) -> None:
+    def _note_active_status(self, rid: str, status: Any, *, note: bool = True) -> None:
         """Record what a save, a journal line or a verified read says about a
         run. Always recorded (before the first build too): a scan in progress
         merges by generation, a build not yet started reads disk truth and
-        reconciles these notes with it."""
+        reconciles these notes with it.
+
+        `note=False` (a replayed journal record, `_apply_journal_entries`): the
+        membership is kept right, but nothing is announced: no sequence bump,
+        no change-ring entry, so `change_token` does not move for a read."""
         if not rid:
             return
         with self._active_lock:
+            existed = rid in self._active
             e = self._active_entry(rid)
+            active = self._is_active_status(status)
+            if not note and (not existed or e.active == active):
+                # A replay of a file known at the mark and unchanged since: filed, not announced
+                # (an existing entry whose active flag FLIPS is a change, announced below).
+                e.active = active
+                return
+            e.active = active
             self._active_seq += 1
             e.gen = self._active_seq
-            e.active = self._is_active_status(status)
             self._change_ring.note(self._active_seq, rid)
 
-    def _forget_active(self, rid: str) -> None:
+    def _forget_active(self, rid: str, *, note: bool = True) -> None:
         """The run is gone (deleted here, or a `-id` line): drop its record."""
         with self._active_lock:
             self._active.pop(rid, None)
+            if not note:
+                return
             self._active_seq += 1
             self._change_ring.note(self._active_seq, rid)
 

abstractruntime-0.15.3/tests/test_json_file_run_store_change_token_after_build.py

@@ -0,0 +1,131 @@
+"""An index build (the first `list_run_index` / `warm_session_index` of a store) replays the
+journal tail before its mark (0.15.2, backlog 0863) WITHOUT announcing those records as changes:
+`change_token` does not move for a read and `changed_run_ids` names nothing. In 0.15.2 the replay
+noted every replayed `+id` on the change ring, so after each build a change-driven consumer (the
+gateway's mail notification check) saw every finished run as changed and loaded them all
+(AbstractGateway backlog 0965: `test_400_finished_runs_a_change_to_one_loads_exactly_one`)."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from abstractruntime.core.models import RunState, RunStatus
+from abstractruntime.storage.json_files import JsonFileRunStore
+
+
+def _run(rid: str, status: RunStatus, i: int) -> RunState:
+    return RunState(run_id=rid, workflow_id="w", status=status, current_node="n", vars={}, actor_id="t",
+                    updated_at=f"2020-01-01T00:{i // 60:02d}:{i % 60:02d}+00:00")
+
+
+def test_an_index_build_does_not_move_the_change_token(tmp_path: Path) -> None:
+    store = JsonFileRunStore(tmp_path / "runs")
+    for i in range(400):
+        store.save(_run(f"done-{i:03d}", RunStatus.COMPLETED if i % 2 else RunStatus.FAILED, i))
+    token = store.change_token()
+    assert len(store.list_run_index(status=RunStatus.COMPLETED, limit=200)) == 200
+    assert len(store.list_run_index(status=RunStatus.FAILED, limit=200)) == 200
+    store.warm_session_index()
+    assert store.change_token() == token, "a read moved the change token"
+    assert store.changed_run_ids(token) == set(), "a read announced runs as changed"
+
+    # A real change after the build is still announced (this store), or unknown (another process).
+    store.save(_run("done-400", RunStatus.COMPLETED, 400))
+    assert store.changed_run_ids(token) == {"done-400"}
+    token2 = store.change_token()
+    JsonFileRunStore(tmp_path / "runs").save(_run("foreign", RunStatus.COMPLETED, 401))
+    assert store.changed_run_ids(token2) is None
+
+
+def test_a_run_another_store_created_before_the_mark_is_still_filed_by_the_replay(tmp_path: Path) -> None:
+    """The 0.15.2 guarantee (backlog 0863) survives: the replay files a run whose `+` line
+    landed before the mark; it only stops announcing it."""
+    writer = JsonFileRunStore(tmp_path / "runs")
+    for i in range(20):
+        writer.save(_run(f"r-{i:02d}", RunStatus.COMPLETED, i))
+    reader = JsonFileRunStore(tmp_path / "runs")
+    token = reader.change_token()
+    writer.save(RunState(run_id="late", workflow_id="w", status=RunStatus.WAITING, current_node="n", vars={},
+                         actor_id="t", session_id="s1"))
+    reader.warm_session_index()
+    rows = reader.list_run_index(session_id="s1", limit=10)
+    assert [r["run_id"] for r in rows] == ["late"]
+    assert "late" in {r["run_id"] for r in reader.list_run_index(status=RunStatus.WAITING, limit=10)}
+    # Another process wrote the journal: the change set is unknown, never silently empty.
+    assert reader.changed_run_ids(token) is None
+
+
+def _publish_without_a_journal_line(src_dir: Path, dest_store: JsonFileRunStore, run: RunState) -> None:
+    """A run file landing in the folder with no new journal record: its `+id` was appended earlier
+    (J53-1: the line goes first, the file after), here by a store in another folder."""
+    import shutil
+
+    JsonFileRunStore(src_dir).save(run)
+    shutil.copyfile(src_dir / f"run_{run.run_id}.json", dest_store._base / f"run_{run.run_id}.json.tmp")
+    (dest_store._base / f"run_{run.run_id}.json.tmp").replace(dest_store._base / f"run_{run.run_id}.json")
+
+
+def test_a_run_published_during_the_scan_is_announced_by_the_replay(tmp_path: Path) -> None:
+    """Review (DESK1): the replay is quiet only for what the store knew before the build's mark. A
+    `+X` appended before the mark whose file lands DURING the scan (the J53-1 window: line first,
+    file after) must still be announced: a consumer whose token already covers that line would
+    otherwise never learn of X. The file lands on the scan's first file read, so the scan's own
+    sync replays `+X` with the file present (the pending re-check never sees it)."""
+    writer = JsonFileRunStore(tmp_path / "runs")
+    for i in range(20):
+        writer.save(_run(f"r-{i:02d}", RunStatus.COMPLETED, i))
+    writer._persist_scan_memo()  # the sidecar: what a restarted store knows at its mark
+    reader = JsonFileRunStore(tmp_path / "runs")
+    writer._journal_append("+late")  # the line, before the file (another process mid-save)
+    token = reader.change_token()  # covers the `+late` line already
+    real = reader._scan_fields
+    landed = {"done": False}
+
+    def scan_fields(p):
+        if not landed["done"]:
+            landed["done"] = True
+            _publish_without_a_journal_line(tmp_path / "other", reader, _run("late", RunStatus.COMPLETED, 99))
+        return real(p)
+
+    reader._scan_fields = scan_fields  # type: ignore[method-assign]
+    reader.warm_session_index()  # the build: mark, scan (late lands on the first read), sync
+    assert landed["done"]
+    assert "late" in {r["run_id"] for r in reader.list_run_index(status=RunStatus.COMPLETED, limit=50)}
+    assert reader.changed_run_ids(token) == {"late"}, "a run the store did not know before the mark must be announced"
+
+
+def test_a_store_without_a_sidecar_announces_its_first_replay(tmp_path: Path) -> None:
+    """No sidecar, nothing saved by this store: at its mark it knows no file, so the replayed
+    records are announced (a bounded one-off: the 16 KiB tail), never silently dropped."""
+    writer = JsonFileRunStore(tmp_path / "runs")
+    for i in range(20):
+        writer.save(_run(f"r-{i:02d}", RunStatus.COMPLETED, i))
+    reader = JsonFileRunStore(tmp_path / "runs")
+    token = reader.change_token()
+    reader.warm_session_index()
+    assert reader.changed_run_ids(token) == {f"r-{i:02d}" for i in range(20)}
+
+
+def test_a_run_unlinked_during_the_scan_is_announced_by_the_replay(tmp_path: Path) -> None:
+    """A `-X` appended before the mark whose unlink lands right after the scan recorded X: the
+    store knew X (its scan), so the replay announces that it is gone."""
+    writer = JsonFileRunStore(tmp_path / "runs")
+    for i in range(20):
+        writer.save(_run(f"r-{i:02d}", RunStatus.COMPLETED, i))
+    writer._persist_scan_memo()
+    reader = JsonFileRunStore(tmp_path / "runs")
+    writer._journal_append("-r-07")  # the line, before the unlink
+    token = reader.change_token()
+    real = reader._scan_fields
+
+    def scan_fields(p):
+        out = real(p)
+        if p.name == "run_r-07.json" and p.exists():
+            p.unlink()  # gone right after the scan recorded it
+        return out
+
+    reader._scan_fields = scan_fields  # type: ignore[method-assign]
+    reader.warm_session_index()  # the build: mark, scan (r-07 recorded, then gone), sync
+    assert not (tmp_path / "runs" / "run_r-07.json").exists()
+    assert "r-07" not in {r["run_id"] for r in reader.list_run_index(status=RunStatus.COMPLETED, limit=50)}
+    assert reader.changed_run_ids(token) == {"r-07"}, "a run gone since the store knew it must be announced"

abstractruntime-0.15.2/release-notes.md

@@ -1,11 +0,0 @@
-### Fixed
-
-- **An index build no longer loses a run another store object creates during it** (`JsonFileRunStore`, backlog 0863).
-  A new run's `+` journal line is written before its file is published; a session/children/active index build that
-  marked the journal between the two and scanned the folder before the file landed missed the run: the listing until
-  the next reconciliation, the session and children indexes for the life of the store object (the reconciliation
-  repaired only the listing). Seen as a gateway that answered 404 `session_not_found` when archiving a session written
-  while it booted (AbstractGateway backlog 0963); a split-mode runner creating a run during the API process's boot
-  hit the same window. The first sync after a build now re-applies the journal's last 16 KiB before the build's mark
-  (`JOURNAL_REPLAY_TAIL_BYTES`; idempotent, once per build), and the reconciliation re-files a run it repairs into
-  the session and children indexes too (and drops a vanished one from them).