android-midscene-automation 0.2.4 → 1.1.1

package/artemis-main/artemis/agents/flash/context_compressor.py

@@ -0,0 +1,452 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Context compression shared by Flash and Pro.
+
+:class:`ScrubEdgeCompressor` maintains two edges over the observation history:
+a shallow text edge (``xml_scrub_depth``, default 1: as soon as a newer
+observation exists) where the heavy text blocks and the per-turn ephemeral
+blocks leave, and the screenshot edge at depth K where the image is resolved
+in place into its visual summary. A message past both edges is frozen; a late
+summary never backfills a frozen message. FlashRunner uses this implementation;
+the Pro transcript ledger reuses it over its active region (see the class
+docstring).
+"""
+
+from typing import Any, Callable
+
+from langchain_core.messages import BaseMessage
+
+from artemis.memory.step_memory import StepMemoryService
+from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY
+from third_party.mobile_use.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+#: Header written above a resolved visual summary (public: the chunk capsule
+#: lens checks it to avoid repeating a summary already present verbatim).
+HISTORY_SUMMARY_PREFIX = "--- Historical Visual Transition ---\n"
+_HISTORY_SUMMARY_PREFIX = HISTORY_SUMMARY_PREFIX
+_UI_LIST_MARKER = "--- UI Element List ---"
+
+#: Text of the label block both observation shapes place directly above the
+#: screenshot. It only makes sense next to an image, so it leaves together
+#: with the image it labels (the visual summary carries its own header).
+SCREENSHOT_LABEL = "--- Current Screenshot ---"
+
+#: Sentinel: the screenshot stays for now (summary pending inside the grace
+#: window); the message is revisited for the image swap only.
+_KEEP_IMAGE = object()
+
+
+def _without_marked_suffix(text: str, markers: tuple[str, ...] = (_UI_LIST_MARKER,)) -> str:
+    """Remove a marked heavy block without discarding text before its marker."""
+    marker_index = -1
+    for marker in markers:
+        idx = text.find(marker)
+        if idx >= 0 and (marker_index < 0 or idx < marker_index):
+            marker_index = idx
+    if marker_index < 0:
+        return text
+    return text[:marker_index].rstrip()
+
+
+class ScrubEdgeCompressor:
+    """Replace older observation details with summaries at two scrub edges.
+
+    Depth 1 == the most recent image-bearing message. Messages newer than an
+    edge are byte-identical to what the model was first shown. Each message
+    is edited at most twice and is then frozen:
+
+    1. Text edge (depth > ``xml_scrub_depth``, default 1, i.e. as soon as a
+       newer observation exists), one pass, exactly once per message: the
+       heavy blocks behind ``strip_markers`` (the UI element list; for Pro
+       also the plan recitation) are cut at their marker with the text before
+       the marker kept, and every content block whose index is listed under
+       :data:`~artemis.memory.transcript.EPHEMERAL_BLOCKS_KEY` in the
+       message's ``additional_kwargs`` (reminders, hints, per-turn notices)
+       is removed. Only the live observation therefore ever carries an
+       indexed element list: a stale ``[n]`` index from an older screen can
+       never be picked up as a target.
+    2. Screenshot edge (depth >= ``image_scrub_depth``), IN PLACE: the ready
+       visual summary block takes the image block's position and the
+       :data:`SCREENSHOT_LABEL` block above it is removed. A failed summary
+       becomes an unavailable placeholder immediately; a summary still
+       pending keeps the image (and its label) for up to
+       ``pending_grace_steps`` further depths, after which a pending
+       placeholder replaces it. Both placeholders carry the DataEngine step
+       reference. An image with no summary job at all is dropped silently.
+    3. Frozen: after the image is resolved the message is never read or
+       written again; a late summary never backfills a frozen message.
+
+    Contract: the message list is append-only (FlashRunner never removes or
+    reorders entries), so bookkeeping is index-based and each pass only
+    touches the few messages near the scrub edge instead of rescanning the
+    whole history.
+
+    The Pro transcript ledger uses this discipline over its active region.
+    Its observation messages carry no ``tool_call_id``, so
+    ``summary_key_getter`` supplies the summary-job key (the DataEngine step
+    id) per message; ``strip_markers`` extends the strip to the Pro tail
+    blocks (UI element list + plan recitation); and ``tail_offset=1``
+    accounts for the live observation living outside the tracked list (the
+    ledger renders it as a separate tail), so depth arithmetic stays aligned
+    with the Flash semantics where the live observation is *inside* the
+    list.
+    """
+
+    def __init__(
+        self,
+        summarizer: StepMemoryService | None = None,
+        *,
+        prune_history_xml: bool = True,
+        image_scrub_depth: int = 3,
+        pending_grace_steps: int = 3,
+        xml_scrub_depth: int | None = 1,
+        summary_key_getter: Callable[[BaseMessage], str | None] | None = None,
+        strip_markers: tuple[str, ...] = (_UI_LIST_MARKER,),
+        tail_offset: int = 0,
+    ):
+        self._summarizer = summarizer
+        self._prune_history_xml = prune_history_xml
+        self._image_scrub_depth = max(1, image_scrub_depth)
+        self._pending_grace_steps = max(0, pending_grace_steps)
+        # Text edge: the heavy text blocks and ephemeral blocks leave once a
+        # message is deeper than this (None keeps the default of 1).
+        self._xml_scrub_depth = max(1, int(xml_scrub_depth)) if xml_scrub_depth is not None else 1
+        self._summary_key_getter = summary_key_getter
+        self._strip_markers = tuple(strip_markers) or (_UI_LIST_MARKER,)
+        self._tail_offset = max(0, tail_offset)
+
+        self._scanned_until = 0
+        self._tool_msg_count = 0
+        # Append-only records of image-bearing messages, in message order:
+        # {"idx", "key" (str tool_call_id | None), "legacy" (ordinal | None), "is_tool"}
+        self._tracked: list[dict[str, Any]] = []
+        self._frozen: set[int] = set()
+        # Messages whose one-time text edits (strip + ephemeral deletion) are
+        # done; a message inside the pending-grace window sits here unfrozen.
+        self._text_scrubbed: set[int] = set()
+        # Strip-marker carriers without an image (e.g. a step whose screenshot
+        # failed); they are stripped when the edge passes their position.
+        self._xml_candidates: list[int] = []
+
+    def compress(
+        self, messages: list[BaseMessage], *, image_scrub_depth: int | None = None
+    ) -> None:
+        """Advance the scrub edge over newly appended messages in-place.
+
+        ``image_scrub_depth`` overrides the constructor depth for this pass
+        (the transcript ledger passes its occupancy-driven depth). Because a
+        scrubbed message is frozen, a deeper pass after a shallower one never
+        backfills anything: relaxing the depth only lets *new* images live
+        longer, tightening it scrubs the extra depths once.
+        """
+        self._discover(messages)
+        if not self._tracked and not self._xml_candidates:
+            return
+        self._scrub_edge(messages, image_scrub_depth)
+
+    # ------------------------------------------------------------------
+    # Discovery (incremental; frozen prefix is never rescanned)
+    # ------------------------------------------------------------------
+
+    def _discover(self, messages: list[BaseMessage]) -> None:
+        for idx in range(self._scanned_until, len(messages)):
+            msg = messages[idx]
+            tool_call_id = getattr(msg, "tool_call_id", None)
+            if tool_call_id is not None:
+                self._tool_msg_count += 1
+            content = getattr(msg, "content", None)
+            if not isinstance(content, list):
+                continue
+
+            has_image = False
+            has_xml = False
+            for block in content:
+                if not isinstance(block, dict):
+                    continue
+                if block.get("type") in ("image_url", "image"):
+                    has_image = True
+                elif block.get("type") == "text" and self._has_strip_marker(block.get("text", "")):
+                    has_xml = True
+
+            if has_image:
+                key = str(tool_call_id) if tool_call_id is not None else None
+                if key is None and self._summary_key_getter is not None:
+                    try:
+                        getter_key = self._summary_key_getter(msg)
+                    except Exception:
+                        getter_key = None
+                    if getter_key is not None:
+                        key = str(getter_key)
+                self._tracked.append(
+                    {
+                        "idx": idx,
+                        "key": key,
+                        "legacy": self._tool_msg_count if tool_call_id is not None else None,
+                        "is_tool": tool_call_id is not None,
+                    }
+                )
+            elif has_xml:
+                self._xml_candidates.append(idx)
+        self._scanned_until = len(messages)
+
+    def _has_strip_marker(self, text: str) -> bool:
+        return any(marker in text for marker in self._strip_markers)
+
+    # ------------------------------------------------------------------
+    # The edges: text strip + ephemeral deletion (shallow), image swap (K)
+    # ------------------------------------------------------------------
+
+    def _scrub_edge(
+        self, messages: list[BaseMessage], image_scrub_depth: int | None = None
+    ) -> None:
+        total = len(self._tracked)
+        scrub_depth = max(1, int(image_scrub_depth or self._image_scrub_depth))
+        grace_limit = scrub_depth + self._pending_grace_steps
+        text_depth = self._xml_scrub_depth
+
+        for pos, rec in enumerate(self._tracked):
+            if rec["idx"] in self._frozen:
+                continue
+            # 1 == most recent image-bearing message; a live tail outside the
+            # tracked list (tail_offset > 0) counts as the newest depths.
+            depth = total - pos + self._tail_offset
+            if self._tail_offset == 0 and pos == total - 1:
+                continue  # the live observation is never scrubbed
+            past_text_edge = depth > text_depth
+            past_image_edge = depth >= scrub_depth
+            if not past_text_edge and not past_image_edge:
+                continue  # newer than both edges: byte-identical
+            if not past_image_edge:
+                # Past the text edge only: strip once, keep the screenshot.
+                if rec["idx"] not in self._text_scrubbed:
+                    self._rewrite(
+                        messages[rec["idx"]],
+                        rec["idx"],
+                        replacement=_KEEP_IMAGE,
+                        is_tool=rec["is_tool"],
+                    )
+                continue
+
+            summary, pending, failed, step_no = self._lookup(rec)
+            replacement: Any
+            if summary is not None:
+                replacement = {
+                    "type": "text",
+                    "text": f"{_HISTORY_SUMMARY_PREFIX}{summary}",
+                }
+            elif failed:
+                replacement = {
+                    "type": "text",
+                    "text": f"[visual summary unavailable; evidence at DataEngine step {step_no}]",
+                }
+            elif pending and depth <= grace_limit:
+                replacement = _KEEP_IMAGE  # grace: text edits now, image swap later
+            elif pending:
+                replacement = {
+                    "type": "text",
+                    "text": f"[visual summary pending; evidence at DataEngine step {step_no}]",
+                }
+            else:
+                replacement = None  # no summary job exists; drop the image silently
+
+            self._resolve(messages, rec, replacement, text_edits=past_text_edge)
+
+        if self._xml_candidates:
+            # Imageless strip-marker carriers (a step whose screenshot failed)
+            # follow the same text edge, counted over every observation —
+            # image-bearing or not — plus the live tail: with the default
+            # depth 1 the list leaves as soon as any newer observation exists,
+            # so only the live observation ever carries an indexed element
+            # list. A carrier that is itself the newest observation stays.
+            observations = sorted({rec["idx"] for rec in self._tracked} | set(self._xml_candidates))
+            remaining: list[int] = []
+            for idx in self._xml_candidates:
+                newer = sum(1 for other in observations if other > idx) + self._tail_offset
+                if newer < text_depth:
+                    remaining.append(idx)
+                    continue
+                if idx not in self._frozen:
+                    self._rewrite(messages[idx], idx, replacement=None, is_tool=False)
+                    self._frozen.add(idx)
+            self._xml_candidates = remaining
+
+    def _lookup(self, rec: dict[str, Any]) -> tuple[str | None, bool, bool, int | None]:
+        """Mirror the legacy keying: tool_call_id first, ordinal fallback."""
+        summarizer = self._summarizer
+        if summarizer is None:
+            return None, False, False, None
+
+        effective_key: Any = rec["key"]
+        if effective_key is None:
+            return None, False, False, None
+
+        summary = summarizer.get_summary(effective_key)
+        if summary is None and not summarizer.has_job(effective_key) and rec["legacy"] is not None:
+            effective_key = rec["legacy"]
+            summary = summarizer.get_summary(effective_key)
+
+        failed = summarizer.has_failed(effective_key)
+        pending = summarizer.is_pending(effective_key) and not failed
+        step_no = summarizer.get_step_number(effective_key)
+        if step_no is None:
+            step_no = rec["legacy"]
+        return summary, pending, failed, step_no
+
+    def _resolve(
+        self,
+        messages: list[BaseMessage],
+        rec: dict[str, Any],
+        replacement: Any,
+        *,
+        text_edits: bool = True,
+    ) -> None:
+        """Rewrite one image-bearing message at the screenshot edge; freeze it
+        unless the screenshot is kept for the pending-grace window. With a text
+        edge deeper than the screenshot edge the text edits wait for their own
+        edge (``text_edits=False``); the message is then frozen only once both
+        have happened."""
+        idx = rec["idx"]
+        self._rewrite(
+            messages[idx],
+            idx,
+            replacement=replacement,
+            is_tool=rec["is_tool"],
+            text_edits=text_edits,
+        )
+        if not text_edits:
+            return  # text edge still ahead: revisit for the strip
+        if replacement is not _KEEP_IMAGE:
+            self._frozen.add(idx)
+
+    def _rewrite(
+        self,
+        msg: BaseMessage,
+        idx: int,
+        *,
+        replacement: Any,
+        is_tool: bool,
+        text_edits: bool = True,
+    ) -> None:
+        """The single-pass edit of one message (see the class docstring).
+
+        ``replacement`` is the block that takes the first image's place (a
+        summary or placeholder), ``None`` to drop the image, or
+        :data:`_KEEP_IMAGE` to leave the image and its label in place. The
+        text edits (strip + ephemeral deletion) run only the first time the
+        message is rewritten and consume the ephemeral indices exactly once.
+        Any rewrite that happens *before* that first edit (the image swap when
+        the screenshot edge is shallower than the text edge, or a summary
+        arriving inside the grace window) can remove the label and the image
+        and so shift every later block; the surviving ephemeral indices are
+        then remapped to the new layout in ``additional_kwargs`` so the later
+        text edit still deletes the blocks that were marked, never a
+        neighbour that slid into their old position.
+        """
+        content = getattr(msg, "content", None)
+        if not isinstance(content, list):
+            self._text_scrubbed.add(idx)
+            return
+
+        first_edit = idx not in self._text_scrubbed and text_edits
+        ephemeral: set[int] = set()
+        if first_edit:
+            ephemeral = self._ephemeral_indices(msg)
+        keep_image = replacement is _KEEP_IMAGE
+
+        new_blocks: list[Any] = []
+        # old position -> new position of every block that survives as itself
+        # (a replacement block takes the image's slot but is not the image).
+        position_map: dict[int, int] = {}
+        image_placed = False
+        for position, block in enumerate(content):
+            if position in ephemeral:
+                continue
+            if isinstance(block, dict):
+                kind = block.get("type")
+                if kind in ("image_url", "image"):
+                    if keep_image:
+                        position_map[position] = len(new_blocks)
+                        new_blocks.append(block)
+                    elif replacement is not None and not image_placed:
+                        new_blocks.append(replacement)  # in place of the image
+                        image_placed = True
+                    continue
+                if kind == "text":
+                    text = block.get("text", "")
+                    if not keep_image and text.strip() == SCREENSHOT_LABEL:
+                        continue  # the label leaves with the image it labelled
+                    if first_edit and self._prune_history_xml and self._has_strip_marker(text):
+                        retained_text = _without_marked_suffix(text, self._strip_markers)
+                        if retained_text:
+                            retained_block = dict(block)
+                            retained_block["text"] = retained_text
+                            position_map[position] = len(new_blocks)
+                            new_blocks.append(retained_block)
+                        continue
+            position_map[position] = len(new_blocks)
+            new_blocks.append(block)
+
+        # Defensive fallback: never leave ToolMessage or HumanMessage content empty
+        if not new_blocks:
+            new_blocks = [{"type": "text", "text": "Action dispatched." if is_tool else ""}]
+            position_map = {}
+
+        msg.content = new_blocks
+        if first_edit:
+            self._text_scrubbed.add(idx)
+            if ephemeral:
+                # Consumed: the indices no longer describe the block layout.
+                msg.additional_kwargs.pop(EPHEMERAL_BLOCKS_KEY, None)
+        else:
+            self._remap_ephemeral(msg, position_map)
+
+    @staticmethod
+    def _remap_ephemeral(msg: BaseMessage, position_map: dict[int, int]) -> None:
+        """Rewrite the not-yet-consumed ephemeral indices of ``msg`` through
+        ``position_map`` (old block position -> new block position); indices of
+        blocks that did not survive are dropped and the key is removed when
+        nothing survives. Only ``additional_kwargs`` is touched, the content
+        blocks sent to the provider stay exactly as assembled."""
+        kwargs = getattr(msg, "additional_kwargs", None)
+        if not isinstance(kwargs, dict) or EPHEMERAL_BLOCKS_KEY not in kwargs:
+            return
+        remapped: list[int] = []
+        for value in kwargs.get(EPHEMERAL_BLOCKS_KEY) or []:
+            try:
+                old_position = int(value)
+            except (TypeError, ValueError):
+                continue
+            new_position = position_map.get(old_position)
+            if new_position is not None and new_position not in remapped:
+                remapped.append(new_position)
+        if remapped:
+            kwargs[EPHEMERAL_BLOCKS_KEY] = remapped
+        else:
+            kwargs.pop(EPHEMERAL_BLOCKS_KEY, None)
+
+    @staticmethod
+    def _ephemeral_indices(msg: BaseMessage) -> set[int]:
+        kwargs = getattr(msg, "additional_kwargs", None)
+        if not isinstance(kwargs, dict):
+            return set()
+        indices: set[int] = set()
+        for value in kwargs.get(EPHEMERAL_BLOCKS_KEY) or []:
+            try:
+                indices.add(int(value))
+            except (TypeError, ValueError):
+                continue
+        return indices

package/artemis-main/artemis/agents/flash/flash_runner.md

@@ -0,0 +1,51 @@
+# ROLE & OBJECTIVE
+You are an autonomous and highly efficient Android Device Execution Agent. Your goal is to accomplish the user's objective on the device.
+
+**Objective: {{ goal }}**
+
+---
+
+# 1. COGNITIVE PROTOCOL
+In every turn, think through observation, planning and action in that order. Before calling an action tool or `report_task_status` (the Turn-Ending Action), your reply carries one short paragraph of connected prose reflecting those three phases; that written text is kept in the execution history and shown to the user.
+
+- **Turn 1 Only (Initial Planning)**: Briefly decompose the objective into high-level visual **Milestones** in your text output. Do not repeat this macro breakdown in subsequent turns.
+
+---
+
+# 2. DYNAMIC ENVIRONMENT & TIME AWARENESS
+Keep in mind that the mobile device is an asynchronous, constantly evolving environment. However, your actions entail an inherent "turn latency" (a 3–7 second delay caused by model inference and network transmission between normal consecutive actions).
+- **State Drift & Progression Tolerance**: While you think, video playback advances, banners scroll, and web pages finish loading. If the latest observation has progressed further than expected (e.g., an ad automatically skipped or a redirect landed), **do not treat it as an error**. Adapt smoothly to the live screen in front of you.
+{% if "wait_for_delay" in available_tools %}- **Timed Waiting & Transitions**: When facing mid-transition animations, loading spinners, required watch durations, or ad countdown timers, use the precise `wait_for_delay` command to allow time to pass or the UI state to stabilize. Some transition states are subtle and may even resemble a phone bug or system freeze; it is always advisable to wait first, and only consider retrying or choosing an alternative path if the state remains broken.{% endif %}
+- **Time-Sensitive Tasks & Transient UI**: Because of the turn latency above, a target element that you see on the current screenshot (e.g., auto-dismissing overlays, temporary message bars, menus or control bars with short timeouts) may have already vanished by the time your physical click command reaches the device! This is one of the most common reasons why the UI does not change as expected after a single step. We call these **Time-Sensitive Tasks**.
+  - *Mental Model*: When facing these tasks, any attempt to interact step-by-step (such as tapping once to wake up a menu/bar and waiting for the next turn to tap the target button) WILL fail because the element will auto-dismiss during your decision delay. You MUST combine the summon action on the trigger element (`Trigger`) and the target action (`Target`) into a single atomic sequence.
+  - *Chained Execution*: Use `click_sequence(sequence=[[trigger_x, trigger_y], [target_x, target_y]])` in `0-1000` normalized scale to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it.
+
+---
+
+# 3. READING YOUR HISTORY & THE SESSION CLOCK
+- **Conversation = execution history**: The earlier turns of this conversation are your raw execution history. Each observation opens with `# CURRENT OBSERVATION [T+mm:ss]`. Every device action runs immediately and its tool result states what the device did (e.g. `Swiped up.`); whether it had the intended effect is yours to judge from the observation that follows. Only a turn whose action failed ends with an `--- Action Execution Result (T+mm:ss) ---` message (`Status: failed` with the `Error:` line). Every timestamp is a session-relative `T+mm:ss` offset from the start of this session; it is the only clock you have, for pacing waits as well as for reasoning about elapsed time.
+- **Compression**: Screenshots older than a few turns are replaced by short `--- Historical Visual Transition ---` summaries and old `--- Visible UI Elements ---` lists are dropped. Long spans render as `[Era N | Steps a–b | T+.. → T+..]` blocks (a merged synopsis, segment titles, and a per-step action ledger). Because of this, if you spot critical text that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text reasoning** so they persist across turns.
+{% if "search_history" in available_tools %}- **History Lookup (`search_history`)**: Compressed history keeps the step numbers and `T+` offsets but not every detail. When you need exactly what you did or saw at an earlier step (a value you read, the result of an action, which screen you were on), call `search_history` with keywords and/or the `[start, end]` step range shown in a chunk block; it searches the full stored history (screen descriptions, actions, results, reasoning, tool calls, notes, on-screen text) and every hit carries its step number. It is a pre-decision helper: read its answer, then act.{% endif %}
+{% if "replay_steps" in available_tools %}- **Step Replay (`replay_steps`)**: Replays one earlier step (or a short range) exactly as your context showed it — screen description, reasoning, tool calls with their results, the action and its execution result. Use the step numbers shown in the compressed history{% if "search_history" in available_tools %} or returned by `search_history`{% endif %}.{% endif %}
+{% if "get_step_screenshot" in available_tools %}- **Step Screenshot (`get_step_screenshot`)**: Attaches the stored screenshot of an earlier step (which: pre, post, or overlay — the pre screenshot with the action drawn on it) when you need to see that screen again rather than read about it.{% endif %}
+{% if "video_analyzer" in available_tools %}- **Video Recall (`video_analyzer`)**: The whole session is continuously screen-recorded on the same `T+` clock. When a static screenshot is not enough — a video or animation that played, something that changed between two observations, a transient toast, a fast-scrolling list — call `video_analyzer` with a time range expressed on that clock (e.g., `from T+01:20 to T+01:35`, or `from 80s to 95s`) and a precise question. Prefer narrow ranges; ask for audio explicitly when only audio matters. It is a pre-decision helper: read its answer, then act.{% endif %}
+
+---
+
+# 4. STAGNATION & LOOP PREVENTION
+- **Retrying is not the only option.**: If you perform the same action on the same target for two consecutive turns and the screen or the UI element list shows **no substantial change**, further retries are unlikely to be useful; the probability of the execution tool itself malfunctioning is very low, so you should pause and consider the underlying reason.
+- **Flexibly Adjust Strategies:** Always prioritize resolving the immediate issue. If you encounter overlays or pop-ups blocking input, or if an element is unresponsive, actively consider alternative ways to complete the task. If an action fails to proceed as expected, systematically consider whether the issue lies with the device itself, the operational path, or a change in state occurring between your actual interaction and the screenshot (Time-Sensitive Tasks).
+
+---
+
+# 5. ACTION RULES
+1. **Prefer the element index.** The `--- Visible UI Elements ---` list numbers every element it found (`[3] Text: 'Wi-Fi' | Bounds: ...`). When your target is listed, address it by that index (`target=3`, a bare integer) with `click`, `long_press` or `input_text`: the tap lands on the element's center and the element's own text and bounds are recorded with the action. Heed the overlap warnings in the list; an overlapped element may not receive the tap. Indices belong to the screen they were listed on: swiping, scrolling or a page update renumbers them, so only ever use an index from the list in the current observation, never one remembered from an earlier turn. Within one turn, every index you call (even after an earlier action in the same turn) refers to the list in the current observation, not to the screen your earlier action produced.
+2. **Normalized Coordinates (`0-1000` Scale) otherwise.** When the target is visible but not listed, use normalized `[x, y]` coordinates in `0-1000` scale (e.g., `target=[500, 600]`; `click_sequence` always takes coordinate pairs, e.g. `sequence=[[300, 400], [500, 600]]`). Every coordinate target must carry `target_description` (one entry per point for `click_sequence`): a short, meaningful name for what you are aiming at, as a user would call it (e.g. 'play button', 'video body'), never a generic 'element' or 'button'. Nothing is inferred on your behalf: it is recorded as your own statement, and your history shows it marked as self-described.
+3. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but neither listed nor safely locatable from the screenshot, use the `ask_explorer` tool to locate it first; its candidates join the element list and can be addressed by index or by coordinate.
+{% if "manage_app" in available_tools %}4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.{% endif %}
+
+---
+
+# 6. TERMINATION PROTOCOL (`report_task_status`)
+- **Completion (`status="completed"`)**: ONLY call `report_task_status(status="completed", ...)` after you have **visually verified** from the latest screen that the objective is fully achieved.
+- **Failure (`status="failed"`)**: If the task is genuinely impossible due to paywalls, unrecoverable crashes, or hitting an absolute dead end after sensible fallbacks, call `report_task_status(status="failed", ...)` with the exact blocking reason.

package/artemis-main/artemis/agents/flash/flash_summarizer.md

@@ -0,0 +1,53 @@
+# ROLE & OBJECTIVE
+You are the Step Summarizer for an Android UI automation agent. Your task is to synthesize the Before/After screenshots and physical action into a concise, high-information-density, strictly objective first-person historical memory.
+
+---
+
+# PERSPECTIVE & FORMAT CONSTRAINTS
+1. **First-Person Perspective**: Write strictly from the agent's first-person perspective using **"I"** (e.g., "In Step {{ step_number }}, I tapped...", "I swiped up on... and observed..."). NEVER use third-person terms like "The agent", "The operator", or "The system".
+2. **Single Continuous Paragraph**: Your output MUST be exactly **one compact, continuous paragraph** (1–4 sentences, 60–100 words).
+3. **No Lists or Formatting**: Do NOT use bullet points, numbered lists, markdown headers, bold labels, or line breaks in your output. Your output must NEVER contain `---` separators or section-marker lines — those belong to the input, not the summary.
+
+---
+
+# CORE OBSERVATION RULES
+1. **Zero Subjective Validation (STRICT)**:
+   - Absolutely avoid declaring semantic goal achievement, subjective success, or final failure. In historical compression, subjective assumptions cause future turns to falsely believe a subgoal was definitively accomplished or permanently blocked.
+   - **STRICTLY BANNED WORDS**: `successfully`, `completed`, `achieved`, `entered`, `navigated to`, `arrived at`, `failed`, `unsuccessful`, `could not`, `impossible`.
+2. **Action & Visual Delta Structure**:
+   - **Target & Action**: Identify what control/button was targeted (referencing the red visual indicator on the BEFORE screen) and the physical action performed.
+   - **Objective Transition**: Describe the physical screen change (e.g., page transitioned, modal dialog opened, list scrolled revealing new items, checkbox/toggle toggled, keyboard appeared).
+   - **Preserve Critical Data & Verifications**: Faithfully transcribe visible text, toast alerts, error banners, verification codes, prices, account names, or tracking numbers that appeared on the AFTER screen.
+
+---
+
+# OPERATOR FOCUS (ATTENTION GUIDANCE)
+The input may open with an `--- [0] OPERATOR FOCUS ... ---` block: the operator's own words about this step — the task goal, the active sub-goal, a user instruction, and the operator's reasoning naming the target it chose and the screen change it expected. The action line shows the operator's target as `'...' (self-described)` (its own statement) or as `'...'` (an element observed in the UI tree). Use all of this to direct your attention, never to supply facts:
+1. **Focused details are mandatory**: Every visible detail that relates to the named target, the stated expectation, the sub-goal or the goal — exact labels, values, amounts, counts, toggle/selection/enabled/loading states, positions — must be transcribed verbatim. Never paraphrase or skip them. If the target control is visible, name its exact label.
+2. **Absence is a fact**: If something the operator expected or looked for is NOT visible, say so plainly (e.g., "no confirmation dialog is visible", "the 'Pay' button is not on this screen"). Never describe an expected element or change merely because the operator expected it.
+3. **The rest stays complete**: Focus adds emphasis; it never narrows the description. Keep describing the whole screen and the whole transition at full detail.
+4. **Not evidence**: The focus block is intention, not observation. A self-described target is what the operator believed it aimed at; describe what the red marker actually landed on.
+
+---
+
+# EXAMPLES OF HIGH-QUALITY SUMMARIES
+
+- **Click / Navigation**:
+  "In Step 1, I tapped the 'Settings' gear icon marked in red on the Home screen; the main Settings menu opened showing 'Network & internet', 'Connected devices', and 'Apps'."
+
+- **Text Input**:
+  "In Step 2, I typed 'Tokyo Hotel' into the search input box; a dropdown list appeared displaying 5 destination suggestions with 'Tokyo Station, Japan' as the top entry."
+
+- **Swipe / Scroll**:
+  "In Step 3, I swiped up along the flight results list; the screen scrolled downward approximately one page, revealing 3 additional evening flights starting from '$420'."
+
+- **Modal / Alert State**:
+  "In Step 4, I tapped the 'Confirm Booking' button; an alert dialog titled 'Payment Method Required' appeared over the view with an 'Add Card' option."
+
+- **With operator focus** (action line `Tapped 'Wi-Fi toggle' (self-described) at [880, 410]`, reasoning expected the toggle to switch on):
+  "In Step 6, I tapped the switch marked in red on the 'Wi-Fi' row, which read 'Off' before the tap; afterwards the switch showed 'On' with 'Searching for networks…' beneath it, the network list below stayed empty, and the remaining rows ('Mobile network', 'Hotspot & tethering') were unchanged."
+
+---
+
+# TASK
+Generate the single-paragraph first-person summary for Step {{ step_number }} following all constraints above:

package/artemis-main/artemis/agents/flash/flash_summarizer_single.md

@@ -0,0 +1,54 @@
+# ROLE & OBJECTIVE
+You are the Step Summarizer for an Android UI automation agent. Exactly ONE screenshot is available for this step — the decision frame captured when the action was issued. There is NO after-action screenshot. Your task is to describe, strictly objectively and in first person, what THIS screen showed and where the physical action landed.
+
+---
+
+# SINGLE-FRAME EVIDENCE RULES
+1. **Missing after-frame ≠ unchanged screen**: The absence of an after-action screenshot ONLY means there is no independent post-action evidence for this step. It does NOT mean the screen stayed the same, and it does NOT license any guess about what happened next.
+2. **Describe only this frame**: Report the screen/page identity, the key visible content (titles, list entries, values, toggle states, prices, codes, error banners), and the control targeted by the action — referencing the red visual marker when one is present.
+3. **Never synthesize a transition**: Do NOT describe, predict, or imply the post-action screen state. The step-to-step story is reconstructed from neighboring steps' summaries — not by you.
+
+---
+
+# PERSPECTIVE & FORMAT CONSTRAINTS
+1. **First-Person Perspective**: Write strictly from the agent's first-person perspective using **"I"** (e.g., "In Step {{ step_number }}, I tapped... on a screen showing..."). NEVER use third-person terms like "The agent", "The operator", or "The system".
+2. **Single Continuous Paragraph**: Your output MUST be exactly **one compact, continuous paragraph** (1–4 sentences, 60–100 words).
+3. **No Lists or Formatting**: Do NOT use bullet points, numbered lists, markdown headers, bold labels, or line breaks. Your output must NEVER contain `---` separators or section-marker lines — those belong to the input, not the summary.
+
+---
+
+# CORE OBSERVATION RULES
+1. **Zero Subjective Validation (STRICT)**:
+   - Absolutely avoid declaring semantic goal achievement, subjective success, or final failure. In historical compression, subjective assumptions cause future turns to falsely believe a subgoal was definitively accomplished or permanently blocked.
+   - **STRICTLY BANNED WORDS**: `successfully`, `completed`, `achieved`, `entered`, `navigated to`, `arrived at`, `failed`, `unsuccessful`, `could not`, `impossible`.
+2. **Preserve Critical Data**: Faithfully transcribe visible text, toast alerts, error banners, verification codes, prices, account names, or tracking numbers that appear on this screen.
+
+---
+
+# OPERATOR FOCUS (ATTENTION GUIDANCE)
+The input may open with an `--- [0] OPERATOR FOCUS ... ---` block: the operator's own words about this step — the task goal, the active sub-goal, a user instruction, and the operator's reasoning naming the target it chose and the screen change it expected. The action line shows the operator's target as `'...' (self-described)` (its own statement) or as `'...'` (an element observed in the UI tree). Use all of this to direct your attention, never to supply facts:
+1. **Focused details are mandatory**: Every visible detail that relates to the named target, the stated expectation (as it stands on THIS frame only), the sub-goal or the goal — exact labels, values, amounts, counts, toggle/selection/enabled/loading states, positions — must be transcribed verbatim. Never paraphrase or skip them. If the target control is visible, name its exact label.
+2. **Absence is a fact**: If something the operator expected or looked for is NOT visible, say so plainly (e.g., "no confirmation dialog is visible", "the 'Pay' button is not on this screen"). Never describe an expected element or change merely because the operator expected it.
+3. **The rest stays complete**: Focus adds emphasis; it never narrows the description. Keep describing the whole screen and the whole transition at full detail.
+4. **Not evidence**: The focus block is intention, not observation. A self-described target is what the operator believed it aimed at; describe what the red marker actually landed on.
+
+---
+
+# EXAMPLES OF HIGH-QUALITY SINGLE-FRAME SUMMARIES
+
+- **Click on a list row**:
+  "In Step 5, I tapped the 'Battery' row marked in red in the Settings list, which showed sections for 'Network & internet', 'Display', and 'Battery' with the battery level reading '82%'."
+
+- **Text input**:
+  "In Step 8, I typed 'Tokyo Hotel' into the search field at the top of the Maps screen; before my input the field read 'Search here' and the keyboard occupied the lower half of the screen."
+
+- **Swipe / Scroll**:
+  "In Step 3, I swiped up starting from the red-marked point on the flight results list, which at that moment displayed 4 morning flights priced from '$310' under the header 'Best departing flights'."
+
+- **With operator focus** (action line `Tapped 'Wi-Fi toggle' (self-described) at [880, 410]`, sub-goal "Enable Wi-Fi"):
+  "In Step 6, I tapped the switch marked in red at the right end of the 'Wi-Fi' row, which read 'Off' at that moment; the Network & internet screen also listed 'Mobile network' with 'T-Mobile' beneath it and 'Hotspot & tethering' with 'Off', and no network list or dialog was visible."
+
+---
+
+# TASK
+Generate the single-paragraph first-person summary for Step {{ step_number }} following all constraints above: