AbstractRuntime 0.7.0__tar.gz → 0.7.1__tar.gz

{abstractruntime-0.7.0 → abstractruntime-0.7.1}/CHANGELOG.md

@@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.7.1] - 2026-09-28
+
+### Fixed
+
+- **Images in replayed history stay images.** When the history window drops older turns, it writes a
+  `[#TRUNCATION: ...]` notice at the start of the oldest kept message. If that message held a list of content parts
+  (text plus an image, `[{"type": "text", ...}, {"type": "image_url", ...}]`), the notice turned the whole list into
+  one string, so the model received the image's base64 as text. The notice is now added as one text part and every
+  other part is left exactly as it was. A message that starts with its `<runtime_metadata>` envelope keeps it first.
+  This affects `announce_dropped` and `window_transcript`, and so the gateway's `/runs/start` client-context window and
+  any replay that keeps images. Content of any other type is left untouched and a warning is logged.
+- **An image no longer counts as tens of thousands of tokens in the window.** `estimate_message_tokens` turned a
+  content-part list into a string, so a 150 KB inline image counted about 50,000 tokens and pushed every older turn out
+  of the history window. Text parts now count their text, and each media part (image, audio, file) counts a flat 512
+  tokens (`MEDIA_PART_TOKEN_ESTIMATE`, AbstractCore's per-image fallback). The input-budget trim
+  (`trim_messages_to_max_input_tokens`) uses the same estimate.
+
 ## [0.7.0] - 2026-09-28
 
 Replayed session history is one window: the most recent 50,000 tokens of whole turns, recorded in the run.

{abstractruntime-0.7.0 → abstractruntime-0.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: AbstractRuntime
-Version: 0.7.0
+Version: 0.7.1
 Summary: AbstractRuntime: a durable graph runner designed to pair with AbstractCore.
 Project-URL: AbstractCore (website), https://www.abstractcore.ai/
 Project-URL: AbstractRuntime (GitHub), https://github.com/lpalbou/abstractruntime
@@ -54,7 +54,7 @@ Description-Content-Type: text/markdown
 
 It is designed for long-running workflows that must survive restarts and explicitly model blocking (human input, timers, external events, subworkflows) without keeping Python stacks alive.
 
-**Version:** 0.7.0 • **Python:** 3.10+
+**Version:** 0.7.1 • **Python:** 3.10+
 
 **Status:** pre-1.0 (API may evolve). For production use, pin versions and follow `CHANGELOG.md`.
 
@@ -146,7 +146,7 @@ state = rt.resume(
 assert state.status.value == "completed"
 ```
 
-## What’s included (v0.7.0)
+## What’s included (v0.7.1)
 
 Kernel (import-light):
 - workflow graphs: `WorkflowSpec` (`src/abstractruntime/core/spec.py`)

{abstractruntime-0.7.0 → abstractruntime-0.7.1}/README.md

@@ -4,7 +4,7 @@
 
 It is designed for long-running workflows that must survive restarts and explicitly model blocking (human input, timers, external events, subworkflows) without keeping Python stacks alive.
 
-**Version:** 0.7.0 • **Python:** 3.10+
+**Version:** 0.7.1 • **Python:** 3.10+
 
 **Status:** pre-1.0 (API may evolve). For production use, pin versions and follow `CHANGELOG.md`.
 
@@ -96,7 +96,7 @@ state = rt.resume(
 assert state.status.value == "completed"
 ```
 
-## What’s included (v0.7.0)
+## What’s included (v0.7.1)
 
 Kernel (import-light):
 - workflow graphs: `WorkflowSpec` (`src/abstractruntime/core/spec.py`)

{abstractruntime-0.7.0 → abstractruntime-0.7.1}/docs/api.md

@@ -252,10 +252,10 @@ Implementation: `src/abstractruntime/session_turns.py`, `src/abstractruntime/ses
 
 - `select_session_turns(run_store, session_id, *, include_occurrences=True, until_ms=None, automation_id=None, through_occurrence=None, include_drafts=False, limit=50)` (package root): a session's turns, oldest first. Turns are parent-less runs except automation controllers, plus automation occurrences (a retried occurrence counts once, as its newest attempt); child runs, runtime-internal runs, legacy scheduled wrappers and draft-test runs are left out
 - `session_chat_messages(run_store=, ledger_store=, artifact_store=, session_id=, max_tokens=HISTORY_REPLAY_MAX_TOKENS, until_ms=None, exclude_run_ids=None, automation_id=None, through_occurrence=None, strict=False)` (package root): the session's completed turns as user/assistant message pairs; in a discussion session the discussion's seed comes first. Returns a `ReplayedHistory` (a list of messages) whose `.report` says what was replayed and dropped. `strict=True` raises `SessionHistoryError` (`reason_code = "history_unavailable"`) instead of returning a partial history. `max_messages`, `max_total_chars` and `max_chars_per_message` (keyword-only, default `None`) are the caps retired in 0.7.0: they are accepted so hosts built against 0.6 (AbstractGateway 0.6.0) still get their history, and ignored. Passing any of them logs one warning and names them in `report["ignored_inputs"]`
-- The history window: `HISTORY_REPLAY_MAX_TOKENS = 50_000` (package root). Replay keeps the most recent turns that fit 50,000 estimated tokens, newest first, as whole messages. There is no message-count cap and no character cap, and no message is ever cut. The fold stops at the first older turn that does not fit, so the window has no gaps. A total of exactly 50,000 tokens fits. One exception: when the newest turn alone is larger than the window, it is kept whole (`oversize_turn_kept: true`), because cutting it would lose content and dropping it would replay nothing. The model can use the rest of its context window; if a turn is too large for the model, the provider reports the error. Tokens are counted with `abstractruntime.memory.token_budget.estimate_message_tokens`. `max_tokens` must be a positive int
+- The history window: `HISTORY_REPLAY_MAX_TOKENS = 50_000` (package root). Replay keeps the most recent turns that fit 50,000 estimated tokens, newest first, as whole messages. There is no message-count cap and no character cap, and no message is ever cut. The fold stops at the first older turn that does not fit, so the window has no gaps. A total of exactly 50,000 tokens fits. One exception: when the newest turn alone is larger than the window, it is kept whole (`oversize_turn_kept: true`), because cutting it would lose content and dropping it would replay nothing. The model can use the rest of its context window; if a turn is too large for the model, the provider reports the error. Tokens are counted with `abstractruntime.memory.token_budget.estimate_message_tokens`: for a content-part list it counts the text parts' text plus `MEDIA_PART_TOKEN_ESTIMATE` (512) for each media part (`MEDIA_PART_TYPES`: image, audio, file), never the part's base64; any other part (a tool result, a thinking block) counts by its text (0.7.1). `max_tokens` must be a positive int
 - `ReplayedHistory.report`: `{policy: "most_recent_whole_turns", max_tokens, token_estimator, replayed_messages, replayed_tokens, dropped_messages, dropped_tokens, dropped_counts_complete, oversize_turn_kept}`. Replay stops reading once the window is full, so in a long session `dropped_counts_complete` is `false` and the dropped counts cover only the turns it read. When turns were dropped, the oldest replayed message starts with a `[#TRUNCATION: N earlier message(s) ... were dropped from replay by the history window ...]` line (ADR-0026). Hosts record the report in the run as `vars._runtime.session_history`
 - `fold_history_window(pairs, *, max_tokens=HISTORY_REPLAY_MAX_TOKENS)` (package root): the window itself, applied to chronological turns (lists of whole messages); returns `(kept, report)`
-- `announce_dropped(messages, report, *, stamp_metadata=True)` (package root): writes the window's `[#TRUNCATION: ...]` notice at the start of the oldest kept message, in place, when `report` says turns were dropped. Use it after your own `fold_history_window` call (for example on client-sent history) so the notice matches session replay. `stamp_metadata=True` also adds `metadata.replay_truncated` and `metadata.history_window` to that message; pass `False` for plain role/content messages. `_announce_dropped` is kept as an alias of the pre-0.7.0 private name
+- `announce_dropped(messages, report, *, stamp_metadata=True)` (package root): writes the window's `[#TRUNCATION: ...]` notice at the start of the oldest kept message, in place, when `report` says turns were dropped. Use it after your own `fold_history_window` call (for example on client-sent history) so the notice matches session replay. `stamp_metadata=True` also adds `metadata.replay_truncated` and `metadata.history_window` to that message; pass `False` for plain role/content messages. When the message's content is a content-part list (`[{"type": "text", ...}, {"type": "image_url", ...}]`), the notice is added as one `{"type": "text"}` part and every other part is left as it is, so images stay images (0.7.1); a stamped head keeps its `<runtime_metadata>` envelope first. Content of any other type is left untouched and a warning is logged. `_announce_dropped` is kept as an alias of the pre-0.7.0 private name
 - `window_transcript(messages, *, max_tokens=HISTORY_REPLAY_MAX_TOKENS, current_turn_start=None)` (package root): the same window over a transcript you already hold (a turn is a user message and the messages after it). `current_turn_start` is the index of the message that opened the turn in progress: from there on everything is one turn, the newest, always kept whole (`oversize_turn_kept` when it alone exceeds the window), whatever user-role messages a loop adds inside it; an index out of range raises `ValueError`. Returns a `ReplayedHistory` of copies with the notice written by `announce_dropped(..., stamp_metadata=False)`. The entity chat driver and the entity visit workflow use it for their prompts. The visit records the report in `vars._runtime.session_history`; the chat driver puts it on `TurnReport.history_window`. On the react visit arm, BRIDGE sets `_runtime.history_window_tokens` (50,000) and `_runtime.history_window_turn_start` (the visitor's message) and the AbstractAgent react loop (0.3.17 or newer) sends `window_transcript(context.messages)` on each call and records the report; the stored transcript stays whole. The report carries `window_applied: true`; with an older AbstractAgent that records no window the turn still completes, a warning is logged and the report is `{"window_applied": false, "reason": "agent_too_old", ...}` (the whole transcript was sent). When the oldest kept message carries a `<runtime_metadata>` envelope, the notice is written after it
 - `session_attribution(run_store, session_id)` (`abstractruntime.core.run_attribution`): `None` or `{"kind": "chat" | "automation" | "occurrence" | "discussion", ...}`; a discussion adds its validated root, automation, occurrence and workspace. Raises `SessionAttributionError` when the lookup cannot be completed
 - `is_draft_lifecycle(run_lifecycle)` (package root): whether a run's lifecycle value marks a draft test run

{abstractruntime-0.7.0 → abstractruntime-0.7.1}/docs/architecture.md

@@ -1,7 +1,7 @@
 # AbstractRuntime — Architecture
 
 > Updated: 2026-09-28
-> Version: 0.7.0
+> Version: 0.7.1
 > Scope: this describes **what is implemented in this repository**.
 
 AbstractRuntime is a **durable workflow runtime**: it executes workflow graphs as a persisted state machine with explicit waits (user, time, events, jobs, subworkflows). A run can pause for hours/days and resume **without** keeping Python stacks/coroutines alive.

{abstractruntime-0.7.0 → abstractruntime-0.7.1}/llms-full.txt

@@ -52,7 +52,7 @@ See llms.txt for the linked index.
 
 It is designed for long-running workflows that must survive restarts and explicitly model blocking (human input, timers, external events, subworkflows) without keeping Python stacks alive.
 
-**Version:** 0.7.0 • **Python:** 3.10+
+**Version:** 0.7.1 • **Python:** 3.10+
 
 **Status:** pre-1.0 (API may evolve). For production use, pin versions and follow `CHANGELOG.md`.
 
@@ -144,7 +144,7 @@ state = rt.resume(
 assert state.status.value == "completed"
 ```
 
-## What’s included (v0.7.0)
+## What’s included (v0.7.1)
 
 Kernel (import-light):
 - workflow graphs: `WorkflowSpec` (`src/abstractruntime/core/spec.py`)
@@ -430,7 +430,7 @@ rt = create_local_runtime(provider="ollama", model="qwen3:4b")
 # AbstractRuntime — Architecture
 
 > Updated: 2026-09-28
-> Version: 0.7.0
+> Version: 0.7.1
 > Scope: this describes **what is implemented in this repository**.
 
 AbstractRuntime is a **durable workflow runtime**: it executes workflow graphs as a persisted state machine with explicit waits (user, time, events, jobs, subworkflows). A run can pause for hours/days and resume **without** keeping Python stacks/coroutines alive.
@@ -1107,10 +1107,10 @@ Implementation: `src/abstractruntime/session_turns.py`, `src/abstractruntime/ses
 
 - `select_session_turns(run_store, session_id, *, include_occurrences=True, until_ms=None, automation_id=None, through_occurrence=None, include_drafts=False, limit=50)` (package root): a session's turns, oldest first. Turns are parent-less runs except automation controllers, plus automation occurrences (a retried occurrence counts once, as its newest attempt); child runs, runtime-internal runs, legacy scheduled wrappers and draft-test runs are left out
 - `session_chat_messages(run_store=, ledger_store=, artifact_store=, session_id=, max_tokens=HISTORY_REPLAY_MAX_TOKENS, until_ms=None, exclude_run_ids=None, automation_id=None, through_occurrence=None, strict=False)` (package root): the session's completed turns as user/assistant message pairs; in a discussion session the discussion's seed comes first. Returns a `ReplayedHistory` (a list of messages) whose `.report` says what was replayed and dropped. `strict=True` raises `SessionHistoryError` (`reason_code = "history_unavailable"`) instead of returning a partial history. `max_messages`, `max_total_chars` and `max_chars_per_message` (keyword-only, default `None`) are the caps retired in 0.7.0: they are accepted so hosts built against 0.6 (AbstractGateway 0.6.0) still get their history, and ignored. Passing any of them logs one warning and names them in `report["ignored_inputs"]`
-- The history window: `HISTORY_REPLAY_MAX_TOKENS = 50_000` (package root). Replay keeps the most recent turns that fit 50,000 estimated tokens, newest first, as whole messages. There is no message-count cap and no character cap, and no message is ever cut. The fold stops at the first older turn that does not fit, so the window has no gaps. A total of exactly 50,000 tokens fits. One exception: when the newest turn alone is larger than the window, it is kept whole (`oversize_turn_kept: true`), because cutting it would lose content and dropping it would replay nothing. The model can use the rest of its context window; if a turn is too large for the model, the provider reports the error. Tokens are counted with `abstractruntime.memory.token_budget.estimate_message_tokens`. `max_tokens` must be a positive int
+- The history window: `HISTORY_REPLAY_MAX_TOKENS = 50_000` (package root). Replay keeps the most recent turns that fit 50,000 estimated tokens, newest first, as whole messages. There is no message-count cap and no character cap, and no message is ever cut. The fold stops at the first older turn that does not fit, so the window has no gaps. A total of exactly 50,000 tokens fits. One exception: when the newest turn alone is larger than the window, it is kept whole (`oversize_turn_kept: true`), because cutting it would lose content and dropping it would replay nothing. The model can use the rest of its context window; if a turn is too large for the model, the provider reports the error. Tokens are counted with `abstractruntime.memory.token_budget.estimate_message_tokens`: for a content-part list it counts the text parts' text plus `MEDIA_PART_TOKEN_ESTIMATE` (512) for each media part (`MEDIA_PART_TYPES`: image, audio, file), never the part's base64; any other part (a tool result, a thinking block) counts by its text (0.7.1). `max_tokens` must be a positive int
 - `ReplayedHistory.report`: `{policy: "most_recent_whole_turns", max_tokens, token_estimator, replayed_messages, replayed_tokens, dropped_messages, dropped_tokens, dropped_counts_complete, oversize_turn_kept}`. Replay stops reading once the window is full, so in a long session `dropped_counts_complete` is `false` and the dropped counts cover only the turns it read. When turns were dropped, the oldest replayed message starts with a `[#TRUNCATION: N earlier message(s) ... were dropped from replay by the history window ...]` line (ADR-0026). Hosts record the report in the run as `vars._runtime.session_history`
 - `fold_history_window(pairs, *, max_tokens=HISTORY_REPLAY_MAX_TOKENS)` (package root): the window itself, applied to chronological turns (lists of whole messages); returns `(kept, report)`
-- `announce_dropped(messages, report, *, stamp_metadata=True)` (package root): writes the window's `[#TRUNCATION: ...]` notice at the start of the oldest kept message, in place, when `report` says turns were dropped. Use it after your own `fold_history_window` call (for example on client-sent history) so the notice matches session replay. `stamp_metadata=True` also adds `metadata.replay_truncated` and `metadata.history_window` to that message; pass `False` for plain role/content messages. `_announce_dropped` is kept as an alias of the pre-0.7.0 private name
+- `announce_dropped(messages, report, *, stamp_metadata=True)` (package root): writes the window's `[#TRUNCATION: ...]` notice at the start of the oldest kept message, in place, when `report` says turns were dropped. Use it after your own `fold_history_window` call (for example on client-sent history) so the notice matches session replay. `stamp_metadata=True` also adds `metadata.replay_truncated` and `metadata.history_window` to that message; pass `False` for plain role/content messages. When the message's content is a content-part list (`[{"type": "text", ...}, {"type": "image_url", ...}]`), the notice is added as one `{"type": "text"}` part and every other part is left as it is, so images stay images (0.7.1); a stamped head keeps its `<runtime_metadata>` envelope first. Content of any other type is left untouched and a warning is logged. `_announce_dropped` is kept as an alias of the pre-0.7.0 private name
 - `window_transcript(messages, *, max_tokens=HISTORY_REPLAY_MAX_TOKENS, current_turn_start=None)` (package root): the same window over a transcript you already hold (a turn is a user message and the messages after it). `current_turn_start` is the index of the message that opened the turn in progress: from there on everything is one turn, the newest, always kept whole (`oversize_turn_kept` when it alone exceeds the window), whatever user-role messages a loop adds inside it; an index out of range raises `ValueError`. Returns a `ReplayedHistory` of copies with the notice written by `announce_dropped(..., stamp_metadata=False)`. The entity chat driver and the entity visit workflow use it for their prompts. The visit records the report in `vars._runtime.session_history`; the chat driver puts it on `TurnReport.history_window`. On the react visit arm, BRIDGE sets `_runtime.history_window_tokens` (50,000) and `_runtime.history_window_turn_start` (the visitor's message) and the AbstractAgent react loop (0.3.17 or newer) sends `window_transcript(context.messages)` on each call and records the report; the stored transcript stays whole. The report carries `window_applied: true`; with an older AbstractAgent that records no window the turn still completes, a warning is logged and the report is `{"window_applied": false, "reason": "agent_too_old", ...}` (the whole transcript was sent). When the oldest kept message carries a `<runtime_metadata>` envelope, the notice is written after it
 - `session_attribution(run_store, session_id)` (`abstractruntime.core.run_attribution`): `None` or `{"kind": "chat" | "automation" | "occurrence" | "discussion", ...}`; a discussion adds its validated root, automation, occurrence and workspace. Raises `SessionAttributionError` when the lookup cannot be completed
 - `is_draft_lifecycle(run_lifecycle)` (package root): whether a run's lifecycle value marks a draft test run

{abstractruntime-0.7.0 → abstractruntime-0.7.1}/llms.txt

@@ -3,7 +3,7 @@
 > Durable workflow runtime (interrupt → checkpoint → resume) with an append-only execution ledger.
 
 - Python: 3.10+ (see `pyproject.toml`)
-- Version: 0.7.0
+- Version: 0.7.1
 - Ecosystem: [AbstractFramework](https://github.com/lpalbou/AbstractFramework) umbrella, integrates with [AbstractCore](https://github.com/lpalbou/abstractcore)
 - Hard invariant: `RunState.vars` must be JSON-serializable (`src/abstractruntime/core/models.py`)
 - Current AbstractCore baseline: base `abstractruntime` installs `abstractcore[remote,tools,vision,voice,audio,music]>=2.18.0`

{abstractruntime-0.7.0 → abstractruntime-0.7.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "AbstractRuntime"
-version = "0.7.0"
+version = "0.7.1"
 description = "AbstractRuntime: a durable graph runner designed to pair with AbstractCore."
 readme = "README.md"
 requires-python = ">=3.10"

abstractruntime-0.7.1/release-notes.md

@@ -0,0 +1,14 @@
+### Fixed
+
+- **Images in replayed history stay images.** When the history window drops older turns, it writes a
+  `[#TRUNCATION: ...]` notice at the start of the oldest kept message. If that message held a list of content parts
+  (text plus an image, `[{"type": "text", ...}, {"type": "image_url", ...}]`), the notice turned the whole list into
+  one string, so the model received the image's base64 as text. The notice is now added as one text part and every
+  other part is left exactly as it was. A message that starts with its `<runtime_metadata>` envelope keeps it first.
+  This affects `announce_dropped` and `window_transcript`, and so the gateway's `/runs/start` client-context window and
+  any replay that keeps images. Content of any other type is left untouched and a warning is logged.
+- **An image no longer counts as tens of thousands of tokens in the window.** `estimate_message_tokens` turned a
+  content-part list into a string, so a 150 KB inline image counted about 50,000 tokens and pushed every older turn out
+  of the history window. Text parts now count their text, and each media part (image, audio, file) counts a flat 512
+  tokens (`MEDIA_PART_TOKEN_ESTIMATE`, AbstractCore's per-image fallback). The input-budget trim
+  (`trim_messages_to_max_input_tokens`) uses the same estimate.

{abstractruntime-0.7.0 → abstractruntime-0.7.1}/src/abstractruntime/memory/token_budget.py

@@ -10,6 +10,7 @@ budget (ADR-0008) even when the underlying model supports much larger contexts.
 
 from __future__ import annotations
 
+import json
 import logging
 from typing import Any, Dict, Iterable, List, Optional
 
@@ -35,15 +36,50 @@ def estimate_tokens(text: str, *, model: Optional[str] = None) -> int:
         return max(1, int(len(s) / 4))
 
 
+# Flat estimate for one non-text content part (an image, a file) — AbstractCore's
+# own fallback per image (`VLMTokenCalculator` base_tokens). The real cost depends
+# on the model and the image size; what matters here is that it is a bounded
+# per-part cost, not the length of the part's base64 payload.
+MEDIA_PART_TOKEN_ESTIMATE = 512
+# Content-part types that carry media (an image, audio or a file), counted at the flat estimate.
+# Every other part (tool_result, thinking, ...) is counted by its text, never flat.
+MEDIA_PART_TYPES = frozenset({"image_url", "image", "input_image", "input_audio", "audio", "file"})
+
+
 def estimate_message_tokens(message: Dict[str, Any], *, model: Optional[str] = None) -> int:
-    """Estimate tokens for a chat message dict (role+content)."""
+    """Estimate tokens for a chat message dict (role+content).
+
+    A content-part list (OpenAI-style `[{"type": "text", ...}, {"type":
+    "image_url", ...}]`) counts the text of its text parts, plus
+    `MEDIA_PART_TOKEN_ESTIMATE` for each media part (`MEDIA_PART_TYPES`); any
+    other part is counted by its serialized text. Before 0.7.1 the list
+    was stringified, so an inline image counted as its base64 text (~50,000
+    tokens for a 150 KB image) and pushed every older turn out of the window.
+    """
     if not isinstance(message, dict):
         return 0
     role = str(message.get("role") or "").strip()
-    content = "" if message.get("content") is None else str(message.get("content"))
+    raw = message.get("content")
+    media_parts = 0
+    if isinstance(raw, list):
+        texts: List[str] = []
+        for part in raw:
+            if isinstance(part, dict) and isinstance(part.get("text"), str) and "text" in str(part.get("type") or "text"):
+                texts.append(part["text"])
+            elif isinstance(part, str):
+                texts.append(part)
+            elif isinstance(part, dict) and str(part.get("type") or "") in MEDIA_PART_TYPES:
+                media_parts += 1
+            elif isinstance(part, dict) and part.get("type") == "text":
+                continue  # an empty or non-string text part carries nothing
+            else:
+                texts.append(json.dumps(part, ensure_ascii=False, default=str))
+        content = "\n".join(texts)
+    else:
+        content = "" if raw is None else str(raw)
     # Include a small role prefix so token estimation reflects chat formatting overhead.
     text = f"{role}: {content}" if role else content
-    return estimate_tokens(text, model=model)
+    return estimate_tokens(text, model=model) + media_parts * MEDIA_PART_TOKEN_ESTIMATE
 
 
 def trim_messages_to_max_input_tokens(

{abstractruntime-0.7.0 → abstractruntime-0.7.1}/src/abstractruntime/session_history.py

@@ -236,6 +236,23 @@ def announce_dropped(messages: List[Dict[str, Any]], report: Dict[str, Any], *,
     `stamp_metadata=False` leaves the message's keys as they were (plain
     role/content transcripts that go straight to a model client); the notice
     in the content is the same.
+
+    Content shapes (0.7.1):
+    - a string (or empty/None): the notice is a prefix line, after the
+      message's own `<runtime_metadata>` envelope when it has one;
+    - a content-part list (OpenAI-style `[{"type": "text", ...},
+      {"type": "image_url", ...}]`, the shape AbstractCore accepts): the
+      notice is inserted as ONE `{"type": "text"}` part and every other part
+      is kept as is (images byte-identical). Prefixing it to the list used to
+      turn the whole content into a string, so the model read the image's
+      base64 as text. Without an envelope the notice part goes first. With
+      one (it heads the first text part, where the payload boundary looks
+      for it), that part is split: the envelope stays alone in it, the
+      notice part follows, then the part's remaining text as its own text
+      part — joined with newlines, the text reads exactly as the string path;
+    - anything else: the content is left untouched (rewriting an unknown
+      shape could corrupt it) and a warning is logged; with
+      `stamp_metadata=True` the metadata still records the drop.
     """
     if not messages or int(report.get("dropped_messages") or 0) <= 0:
         return
@@ -257,7 +274,19 @@ def announce_dropped(messages: List[Dict[str, Any]], report: Dict[str, Any], *,
     # payload boundary would otherwise stack a second one in front).
     from .turn_grounding import split_head_grounding  # lazy: the grounding helpers load AbstractCore
 
-    envelope, rest = split_head_grounding(head.get("content") or "")
+    content = head.get("content")
+    if isinstance(content, list):
+        head["content"] = _notice_into_parts(content, notice, split_head_grounding)
+        return
+    if content is not None and not isinstance(content, str):
+        logger.warning(
+            "history window: the oldest kept message has %s content; the #TRUNCATION notice "
+            "was not written into it (%s message(s) dropped)",
+            type(content).__name__,
+            report["dropped_messages"],
+        )
+        return
+    envelope, rest = split_head_grounding(content or "")
     if envelope:
         sep = "" if envelope.endswith("\n") else "\n"
         head["content"] = f"{envelope}{sep}{notice}\n{rest.lstrip(chr(10))}"
@@ -265,6 +294,25 @@ def announce_dropped(messages: List[Dict[str, Any]], report: Dict[str, Any], *,
         head["content"] = f"{notice}\n{rest}"
 
 
+def _is_text_part(part: Any) -> bool:
+    return isinstance(part, dict) and str(part.get("type") or "").strip().lower() == "text"
+
+
+def _notice_into_parts(parts: List[Any], notice: str, split_head_grounding: Any) -> List[Any]:
+    """A new part list with the notice as one text part (see `announce_dropped`)."""
+    out = list(parts)
+    notice_part = {"type": "text", "text": notice}
+    first = next((i for i, part in enumerate(out) if _is_text_part(part)), None)
+    if first is not None:
+        envelope, rest = split_head_grounding(out[first].get("text"))
+        if envelope:
+            head_part = {**out[first], "text": envelope.rstrip("\n")}
+            tail = rest.lstrip("\n")
+            replacement = [head_part, notice_part] + ([{**out[first], "text": tail}] if tail else [])
+            return out[:first] + replacement + out[first + 1 :]
+    return [notice_part] + out
+
+
 # The pre-0.7.0 private name, kept for callers that imported it.
 _announce_dropped = announce_dropped
 

abstractruntime-0.7.1/tests/test_history_window_multimodal.py

@@ -0,0 +1,142 @@
+"""The history window over multimodal messages (0.7.1).
+
+A message's content may be an OpenAI-style part list (`[{"type": "text", ...},
+{"type": "image_url", ...}]`, the shape AbstractCore's providers and the
+runtime's own media path build). The window's drop notice must go in as a TEXT
+PART — prefixing it to the content used to turn the whole list into a string,
+so the model received the image's base64 as text. And the window's token
+estimate must not count that base64 as text either.
+"""
+
+from __future__ import annotations
+
+import copy
+
+from abstractruntime.memory.token_budget import MEDIA_PART_TOKEN_ESTIMATE, estimate_message_tokens
+from abstractruntime.session_history import announce_dropped, fold_history_window, window_transcript
+
+_B64 = "iVBORw0KGgo" + "A" * 200_000 + "=="
+_IMAGE = {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{_B64}", "detail": "high"}}
+
+
+def _report(dropped: int = 2) -> dict:
+    return {**fold_history_window([], max_tokens=10)[1], "dropped_messages": dropped, "dropped_tokens": 9}
+
+
+def _notice_of(report: dict) -> str:
+    probe = [{"role": "user", "content": ""}]
+    announce_dropped(probe, report, stamp_metadata=False)
+    return probe[0]["content"].rstrip("\n")
+
+
+def test_list_content_keeps_its_image_and_gains_one_notice_text_part() -> None:
+    image = copy.deepcopy(_IMAGE)
+    messages = [{"role": "user", "content": [{"type": "text", "text": "what is this?"}, image]}]
+    announce_dropped(messages, _report(), stamp_metadata=False)
+
+    content = messages[0]["content"]
+    assert isinstance(content, list) and len(content) == 3
+    notice, text, img = content
+    assert notice["type"] == "text" and notice["text"].startswith("[#TRUNCATION: 2 earlier message(s)")
+    assert set(notice) == {"type", "text"}
+    assert text == {"type": "text", "text": "what is this?"}
+    assert img == _IMAGE and img["image_url"]["url"].endswith(_B64)
+    assert all(_B64 not in p.get("text", "") for p in content)
+
+
+def test_list_content_with_a_stamped_head_keeps_the_envelope_first() -> None:
+    from abstractruntime.turn_grounding import message_carries_grounding_envelope, stamp_user_turn_grounding
+
+    stamped = [{"role": "user", "content": "hello"}]
+    assert stamp_user_turn_grounding(stamped)
+    head_text = stamped[0]["content"]
+    envelope = head_text[: head_text.index("hello")]
+
+    messages = [{"role": "user", "content": [{"type": "text", "text": head_text}, copy.deepcopy(_IMAGE)]}]
+    report = _report()
+    announce_dropped(messages, report, stamp_metadata=False)
+
+    content = messages[0]["content"]
+    assert isinstance(content, list)
+    assert content[-1] == _IMAGE
+    texts = [p["text"] for p in content if p.get("type") == "text"]
+    # The envelope still heads the first text part, exactly once in the message.
+    assert texts[0].startswith(envelope.rstrip("\n")) and "hello" not in texts[0]
+    assert sum(t.count("<runtime_metadata>") for t in texts) == 1
+    assert message_carries_grounding_envelope(messages[0])
+    # One text part carries the notice, then the turn's own words.
+    assert texts[1] == _notice_of(report)
+    assert texts[2] == "hello"
+    # Joined the way providers join text parts, it reads as the string path does.
+    as_string = [{"role": "user", "content": head_text}]
+    announce_dropped(as_string, report, stamp_metadata=False)
+    assert "\n".join(texts) == as_string[0]["content"]
+
+
+def test_string_content_is_unchanged() -> None:
+    report = _report()
+    messages = [{"role": "user", "content": "hello"}]
+    announce_dropped(messages, report, stamp_metadata=False)
+    assert messages[0]["content"] == (
+        "[#TRUNCATION: 2 earlier message(s) of this session (~9 tokens) were dropped from replay by the "
+        "history window (the most recent 10 tokens, whole turns; abstractruntime.session_history); "
+        "this history starts mid-conversation]\nhello"
+    )
+
+
+def test_list_content_without_a_text_part_gains_the_notice_first() -> None:
+    messages = [{"role": "user", "content": [copy.deepcopy(_IMAGE)]}]
+    announce_dropped(messages, _report(), stamp_metadata=False)
+    content = messages[0]["content"]
+    assert len(content) == 2 and content[0]["type"] == "text" and content[1] == _IMAGE
+
+
+def test_unknown_content_type_is_left_untouched() -> None:
+    odd = {"custom": "shape"}
+    messages = [{"role": "user", "content": odd}]
+    announce_dropped(messages, _report())
+    assert messages[0]["content"] is odd
+    assert messages[0]["metadata"]["replay_truncated"] is True
+
+
+def test_window_transcript_keeps_an_image_turn_as_parts() -> None:
+    messages = [
+        {"role": "user", "content": "old " * 2000},
+        {"role": "assistant", "content": "old answer"},
+        {"role": "user", "content": [{"type": "text", "text": "look"}, copy.deepcopy(_IMAGE)]},
+        {"role": "assistant", "content": "a cat"},
+    ]
+    out = window_transcript(messages, max_tokens=2000)
+    assert out.report["dropped_messages"] == 2
+    head = out[0]["content"]
+    assert isinstance(head, list) and head[-1] == _IMAGE
+    assert head[0]["text"].startswith("[#TRUNCATION:")
+    assert messages[2]["content"][0] == {"type": "text", "text": "look"}  # input not mutated
+    assert len(messages[2]["content"]) == 2
+
+
+def test_token_estimate_counts_an_image_part_once_not_its_base64() -> None:
+    message = {"role": "user", "content": [{"type": "text", "text": "look"}, copy.deepcopy(_IMAGE)]}
+    tokens = estimate_message_tokens(message)
+    text_only = estimate_message_tokens({"role": "user", "content": "look"})
+    assert tokens == text_only + MEDIA_PART_TOKEN_ESTIMATE
+    # The base64 alone is ~50k tokens as text: it would push every older turn out.
+    assert tokens < 1000
+
+
+def test_token_estimate_counts_every_text_part() -> None:
+    parts = [{"type": "text", "text": "alpha " * 100}, {"type": "text", "text": "beta " * 100}]
+    many = estimate_message_tokens({"role": "user", "content": parts})
+    one = estimate_message_tokens({"role": "user", "content": "alpha " * 100})
+    assert many > one
+
+
+def test_token_estimate_counts_a_non_media_part_by_its_text_not_flat() -> None:
+    # An Anthropic-style tool_result or thinking part is text, however it is typed:
+    # it must never be counted as a flat media part.
+    big = "x " * 50_000
+    tool_part = {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "t1", "content": big}]}
+    as_text = estimate_message_tokens({"role": "user", "content": big})
+    assert estimate_message_tokens(tool_part) >= as_text
+    empty_text = {"role": "user", "content": [{"type": "text"}]}
+    assert estimate_message_tokens(empty_text) < MEDIA_PART_TOKEN_ESTIMATE

abstractruntime-0.7.0/release-notes.md

@@ -1,94 +0,0 @@
-Replayed session history is one window: the most recent 50,000 tokens of whole turns, recorded in the run.
-Library callers of `session_chat_messages` and `automation_timeline_messages` should read the **Breaking** note.
-
-### Changed
-
-- The base install and the `apple` / `gpu` profiles require AbstractCore 2.18.0 or newer. Its voice extra brings
-  AbstractVoice 0.13.0, the first release that honours the `voice_openai_api_key` setting described below.
-
-- **Session history replay is one window: the most recent 50,000 tokens.** `session_chat_messages` keeps the newest
-  turns that fit `HISTORY_REPLAY_MAX_TOKENS` (50,000) estimated tokens, as whole messages, newest first. The window has
-  no gaps, and a total of exactly 50,000 tokens fits. The 40-message cap, the 24,000-character total budget and the
-  8,000-character cut per message are removed, so no replayed message is ever cut. When the newest turn alone is larger
-  than the window, it is kept whole and the report says so (`oversize_turn_kept`). Tokens are counted with the existing
-  estimator, `memory.token_budget.estimate_message_tokens`. This window applies to automation growing mode (it
-  replaces the 40-message / 24,000-character limit of contract D), to discussion seeds (`automation_timeline_messages`)
-  and to host session seeding. The window is the only limit on replayed history; the model can use the rest of its
-  context window (operator ruling 2026-09-28, ADR-0026 §3: budgets are met by choosing whole turns, never by cutting
-  content).
-- **Library API:** `session_chat_messages` still accepts `max_messages`, `max_chars_per_message` and
-  `max_total_chars` (keyword-only, default `None`) but IGNORES them: they no longer limit anything. Hosts built against
-  0.6 (AbstractGateway 0.6.0 passes `max_messages=40, max_total_chars=24000` and treats any error as "no history") keep
-  replaying their sessions under the window. Passing any of them logs one warning and names them in
-  `report["ignored_inputs"]`. Pass `max_tokens` (a positive int) to use a different window.
-- **Breaking (library API):** `automation_timeline_messages` takes `max_tokens` instead of `max_messages` /
-  `max_chars_per_message` / `max_total_chars`. `GROWING_MAX_MESSAGES` and `GROWING_MAX_TOTAL_CHARS` are removed from
-  `abstractruntime.automations.models`.
-- **The entity chat driver and the entity visit workflow use the same window.** They used to keep only the last 10
-  turns in the prompt, without saying so. Now every turn is kept, and each prompt carries the most recent 50,000 tokens
-  of whole turns, with the `[#TRUNCATION: ...]` line when older turns were dropped. The visit records the window's
-  report in `vars._runtime.session_history`; `ChatSession` puts it on `TurnReport.history_window` (and
-  `ChatSession.history_window`).
-- **The react visit arm (the one AbstractGateway uses) sends the window too.** Its transcript is the durable
-  `context.messages`, sent by the AbstractAgent react loop on every call; before, nothing bounded it (after 12 turns of
-  ~10k tokens a request was ~117k tokens, and a poisoned tool result rode every later turn). BRIDGE now sets
-  `_runtime.history_window_tokens` (50,000); the adapter sends `window_transcript(context.messages)` on every call —
-  a tool result stays with its turn, so a 495k-character tool result drops out of the request once a newer turn
-  exists — and records the report in `vars._runtime.session_history`. The stored transcript stays whole. BRIDGE also
-  records `_runtime.history_window_turn_start` (the visitor's message), and the window keeps the turn in progress whole
-  from there (`window_transcript(..., current_turn_start=)`): an `ask_user` answer or operator guidance inside the turn
-  no longer splits off the turn's own question and tool results; a turn larger than the window is kept whole and
-  reported as `oversize_turn_kept`. HARVEST
-  keeps a turn whose adapter recorded no window (AbstractAgent older than 0.3.17 sends the whole transcript) but
-  logs a warning ("abstractagent < 0.3.17 does not apply the history window; upgrade abstractagent") and records
-  `{"window_applied": false, "reason": "agent_too_old", ...}` in `vars._runtime.session_history` (counting the messages
-  the last request carried, not the reply stored after it); a windowed turn
-  records `window_applied: true`. **The window on the react arm needs AbstractAgent 0.3.17 or newer.**
-- The `[#TRUNCATION: ...]` line is written after a stamped message's own `<runtime_metadata>` envelope, which stays at
-  the head (written in front, it made the payload boundary stack a second envelope).
-- **Breaking (library API):** the `history_turns` parameter of `build_visit_workflow` and `ChatSession` is removed,
-  with `DEFAULT_HISTORY_TURNS` (no caller in the framework passed it). The entity visit no longer sets the
-  per-message character limits `_limits.max_tool_message_chars` (32,000) and `_limits.max_message_chars` (80,000) on
-  re-sent history, and `VISIT_HISTORY_TOOL_RESULT_CAP_CHARS` / `VISIT_HISTORY_MESSAGE_CAP_CHARS` are removed. A
-  single oversized tool result is still limited by AbstractAgent's own guard for oversized messages (200,000 characters).
-- `trim_messages_to_max_input_tokens` (used only when a caller sets a positive `max_input_tokens`) keeps each tool
-  result together with the assistant tool call before it. It used to drop the call and keep the result, which
-  OpenAI-style APIs reject.
-
-- **The context mode decides whether an automation's target reads history.** At admission the runtime sets
-  `use_context` (and `include_context` when present) to `true` for growing occurrences and `false` for independent
-  ones, and to `true` for discussion forks. A `use_context: false` frozen in an older definition no longer makes a
-  growing automation or a discussion replay nothing (basic-agent's `use_context` pin defaults to `false`). The run
-  records `_runtime.automation_context = {mode, use_context, target_use_context}`. Occurrences that were already
-  admitted before the upgrade keep their frozen inputs.
-
-### Added
-
-- Voice discovery (`local_get_voice_catalog`, `local_list_tts_models`, `local_list_stt_models`) accepts a
-  host-supplied `voice_openai_api_key` and passes it to AbstractVoice as the plugin setting of the same name (the
-  OpenAI credential for its `openai` engines; AbstractVoice reads no environment variable for it). The discovery
-  facade and the local clients accept and forward it (`AbstractCoreDiscoveryFacade.get_voice_catalog` /
-  `list_tts_models` / `list_stt_models(voice_openai_api_key=...)`); the remote client accepts it and never sends it.
-  In `llm_kwargs` it reaches the voice plugin through the provider's `config` and changes no text request; tests
-  prove this on real OpenAI, LM Studio, Ollama and MLX provider classes, with the network refused.
-- **The window is recorded.** `session_chat_messages` and `automation_timeline_messages` return a `ReplayedHistory`
-  (a list of messages) with a `.report`: `policy`, `max_tokens`, `token_estimator`, `replayed_messages`,
-  `replayed_tokens`, `dropped_messages`, `dropped_tokens`, `dropped_counts_complete`, `oversize_turn_kept`. A growing
-  occurrence records it in `vars._runtime.session_history`, so the `automation.admitted` record's frozen inputs carry
-  it too. A discussion root records its seed window the same way. When older turns are dropped, the oldest replayed
-  message starts with a `[#TRUNCATION: ...]` line that gives the number of dropped messages and tokens and the window.
-  New exports at the package root: `HISTORY_REPLAY_MAX_TOKENS`, `ReplayedHistory`, `fold_history_window`,
-  `announce_dropped` (writes that `[#TRUNCATION: ...]` line after a host's own `fold_history_window` call, for example
-  on client-sent history; `session_history._announce_dropped` stays as an alias) and `window_transcript` (the window
-  over a transcript the caller already holds).
-- `local_list_tts_models` reports why a provider cannot be listed: when the listing is unavailable, `error` carries
-  AbstractVoice's `unavailable_reason` (for example "no OpenAI API key is configured ..." or an unknown provider id).
-  It used to return `available: false, error: null`.
-- Replay reads turns newest-first in batches that double in size, and stops once the window is full. A long session
-  therefore costs about as much as its window holds.
-
-### Tests and CI
-
-- CI installs the extras that exist (`.[test]`) and runs the history-window, entity window, automation timeline,
-  input-trim and voice-key tests. The five tests that need the sibling `abstractflow/` checkout or the
-  `abstractagent` package skip when those are absent (a standalone checkout or an sdist).