android-midscene-automation 0.2.4 → 1.1.0

package/artemis-main/artemis/agents/outputter/outputter.md

@@ -0,0 +1,36 @@
+# Context & Inputs
+
+## User's Initial Goal
+> {{ initial_goal }}
+
+{% if plan_and_history %}
+## Execution History & Plan
+```markdown
+{{ plan_and_history }}
+```
+{% endif %}
+
+{% if output_description and not structured_output %}
+## Target Information to Extract
+The user explicitly requested to extract the following information:
+> "{{ output_description }}"
+{% endif %}
+
+{% if target_schema %}
+## Target Output Fields
+Your final answer will afterwards be formatted into the JSON schema below. Gather evidence for every field, and make sure your final answer states each value you found (or that it could not be verified). You do not need to write JSON yourself.
+```json
+{{ target_schema }}
+```
+{% endif %}
+
+---
+
+# Instruction
+
+1. **Assess Confidence & Verify**: Based on the execution history, final state, and the target information to extract, determine if you can confidently formulate the final answer. If the current evidence is insufficient or you require more granular proof, you **MUST** actively gather information and details using your tools first. If your tools fail (e.g., return errors or time out), or if the steps were skipped/not observed, you **MUST** report this lack of evidence honestly. Do not assume, guess, or fabricate any unobserved steps.
+2. **Write Persistent Report**: Once your investigation is complete, you **MUST** synthesize your findings into a Markdown summary report (detailing Status [SUCCESS/FAILED/PARTIAL], Evidence, and a clear explanation). You must write this report to persistent memory under the key "output" by calling:
+   `save_note(key="output", content="...your markdown report...")`
+3. **Final Response**: Only after saving the report, write a direct, helpful, and jargon-free final summary response to the user. Do not use any XML tags, code block wrappers, or special delimiters—just state the conclusion clearly and naturally, ensuring any extracted target information is clearly presented.
+
+

package/artemis-main/artemis/agents/outputter/outputter.py

@@ -0,0 +1,552 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import asyncio
+import json
+from functools import lru_cache
+from pathlib import Path
+from typing import Any
+
+from jinja2 import Template
+from langchain_core.messages import BaseMessage, HumanMessage, SystemMessage, ToolMessage
+from langchain_core.tools import BaseTool
+from artemis.core.tool_failure import ToolFailure, is_tool_failure
+from artemis.config import OutputConfig
+from artemis.context import ArtemisContext
+from artemis.data_engine.trace import trace
+from artemis.graph.state import State
+from artemis.llm.structured import ParseFailure, content_to_text, parse_structured
+from artemis.services.llm import acomplete, get_llm, invoke_llm_with_timeout_message, with_fallback
+from artemis.tools.scratchpad import (
+    get_append_note_tool_pure,
+    get_list_notes_tool_pure,
+    get_read_note_tool_pure,
+    get_save_note_tool_pure,
+    get_update_note_tool_pure,
+)
+from artemis.tools.video_tool import get_video_analyzer_tool_pure
+from artemis.tools.history import get_history_tools
+from artemis.tools.tool_wrapper import (
+    invoke_tool_with_injection,
+    tool_result_messages,
+)
+from artemis.memory.context_policy import build_history_for
+from artemis.utils.notes import read_note_content
+from artemis.utils.task_tree import get_active_subgoal_hashes
+from third_party.mobile_use.utils.logger import get_logger
+from pydantic import BaseModel
+
+logger = get_logger(__name__)
+
+MAX_TURNS = 20
+
+# Note key of the report the prompt asks for. The UI Task Report card and the
+# MCP docs read it from notes/output.md.
+OUTPUT_NOTE_KEY = "output"
+
+# Note writes run one by one in the order issued; all other tools only read
+# and can run in parallel.
+_WRITE_TOOLS = frozenset({"save_note", "update_note", "append_note"})
+
+_MAX_TURNS_ERROR = "Error: Outputter failed to resolve the query within maximum turns."
+
+_SYSTEM_MESSAGE = (
+    "You are the Output Synthesis Agent for an Android UI automation"
+    " system. Your sole objective is to verify whether the user's initial"
+    " goal was achieved and synthesize the final report.\n\n## Core"
+    " Principles\n1. STRICTLY EVIDENCE-BASED & NO HALLUCINATION: Rely ONLY"
+    " on the actual observed execution history and successful tool outputs."
+    " Do not assume, reconstruct, or hallucinate any actions, elements, or"
+    " steps that did not actually occur during this specific run. If a step"
+    " was skipped or not observed (e.g., because the device was already in"
+    " the target state), you must explicitly state that it was 'Not"
+    " Observed / Already in State' rather than fabricating the interaction."
+    " If a tool call (like `video_analyzer`) fails or returns an error, you"
+    " must report the failure and the resulting lack of evidence, rather"
+    " than assuming success or guessing the details. If the user requests a"
+    " step-by-step guide, you must only include steps that were actually"
+    " executed and observed; do not invent 'standard' steps to fill in"
+    " gaps.\n2. NO JARGON IN FINAL ANSWER: The 'Final Answer' must be"
+    " written in clear, user-friendly language, completely free of"
+    " agent-specific jargon (e.g., do not mention 'nodes', 'XPath',"
+    " 'selectors', 'ReAct', or 'tools').\n\n## Tool Usage Guidelines\n-"
+    " LAZY VERIFICATION: If the execution history and the final screenshot"
+    " already in your context provide undeniable proof of the outcome, do"
+    " not call tools. Directly output your conclusion.\n- ACTIVE RETRIEVAL:"
+    " You should only call tools (e.g., `search_history` to locate the"
+    " step, `replay_steps` for its full record, `get_step_screenshot` for"
+    " its image) if you need to extract specific hidden data (like"
+    " verification codes, tracking numbers) or if the final state is"
+    " ambiguous.\n- BATCH INDEPENDENT LOOKUPS: When you need several"
+    " lookups that do not depend on each other, request them together in"
+    " the same turn instead of one per turn.\n- VIDEO ANALYSIS COST:"
+    " The `video_analyzer` tool is expensive. Use it only when necessary"
+    " (e.g., to verify video playback or motion).\n- PERSISTENT WRITE: If"
+    " you extract important information requested by the user (such as a"
+    " verification code, tracking number, or a text summary), you should"
+    " use `save_note` or `update_note` to write it directly into persistent"
+    " notes so it can be retrieved by other agents or the system."
+)
+
+_FINAL_TURN_NOTICE = (
+    "This is your final turn and tools are no longer available. Write your"
+    " final answer to the user now, using only the evidence gathered so far."
+    " State plainly anything you could not verify."
+)
+
+_EMPTY_REPLY_NUDGE = (
+    "Your previous reply was empty. If you still need evidence, call the"
+    " relevant tools; otherwise write your final answer to the user now."
+)
+
+_FORMAT_SYSTEM_MESSAGE = (
+    "You are a helpful assistant. Your task is to take the"
+    " raw execution summary and format it into the"
+    " requested structured output schema. Do not invent any"
+    " information. If the raw summary does not contain the"
+    " required info, leave those fields empty or null."
+)
+
+
+@lru_cache(maxsize=1)
+def _prompt_template() -> Template:
+    return Template(Path(__file__).with_name("outputter.md").read_text(encoding="utf-8"))
+
+
+def _notes_base_dir(ctx: ArtemisContext) -> str | Path | None:
+    base_dir = getattr(ctx.data_engine, "base_dir", None) if ctx.data_engine else None
+    return base_dir if isinstance(base_dir, (str, Path)) else None
+
+
+def _resolve_plan_and_history(ctx: ArtemisContext) -> str | None:
+    """Build the plan and history text passed to the outputter prompt."""
+    try:
+        history = ctx.data_engine.get_agent_friendly_steps()
+
+        # Read current plan if it exists and base_dir is valid
+        current_plan = ""
+        base_dir = _notes_base_dir(ctx)
+        if base_dir:
+            current_path = Path(base_dir) / "notes" / "task_plan.md"
+            if current_path.exists():
+                try:
+                    current_plan = current_path.read_text(encoding="utf-8")
+                except Exception as e:
+                    logger.error(f"Failed to read current plan for outputter: {e}")
+
+        if not history:
+            return None
+
+        active_subgoal_hash = "default"
+        if current_plan:
+            try:
+                active_subgoal_hash, _ = get_active_subgoal_hashes(current_plan)
+            except Exception as e:
+                logger.error(f"Failed to parse active subgoal in outputter: {e}")
+
+        return build_history_for(
+            "outputter",
+            current_plan,
+            history,
+            active_subgoal_hash,
+            engine=ctx.data_engine,
+        )
+    except Exception as e:
+        logger.error(f"Failed to resolve plan and history in outputter: {e}")
+        return None
+
+
+def _verification_block(ctx: ArtemisContext, graph_output: State) -> str | None:
+    """Format the verdict ledger and test summary, or None if the run had no check items."""
+    try:
+        from artemis.graph.checkpoints import read_ledger, read_run_outcome
+
+        ledger_records = read_ledger(ctx.data_engine.base_dir)
+        run_outcome = read_run_outcome(ctx.data_engine.base_dir) or getattr(
+            graph_output, "run_outcome", None
+        )
+        if not (ledger_records or run_outcome):
+            return None
+        lines = ["--- Verification Verdict Ledger (append-only) ---"]
+        for r in ledger_records:
+            lines.append(
+                f"- attempt {r.get('attempt_id')} [{r.get('kind')}]"
+                f" '{r.get('item_text')}' -> {r.get('status')}:"
+                f" {r.get('evidence', '')}"
+            )
+        if run_outcome:
+            tests = run_outcome.get("tests") or {}
+            lines.append(
+                "--- Test Summary ---\n"
+                f"task_status={run_outcome.get('task_status')},"
+                f" passed={tests.get('passed', 0)},"
+                f" failed={tests.get('failed', 0)},"
+                f" inconclusive={tests.get('inconclusive', 0)},"
+                f" unchecked={tests.get('unchecked', 0)}"
+            )
+        lines.append(
+            "Report every declared check item with its final state"
+            " (passed / failed / inconclusive / unchecked) and include"
+            " the complete verdict sequence above. A failed assertion"
+            " is a test result to report verbatim, not something to"
+            " explain away. Items reported unchecked were never"
+            " judged — never present them as passed."
+        )
+        return "\n".join(lines)
+    except Exception as e:
+        logger.error(f"Failed to attach verdict ledger to outputter: {e}")
+        return None
+
+
+def _environment_block(ctx: ArtemisContext) -> str | None:
+    """Describe which UI hierarchy backend served the run.
+
+    Mentions a mid-run switch too: falling back to UIAutomator2 explains slower
+    steps and conflicts with other automation tools.
+    """
+    from artemis.clients.screen_client_factory import hierarchy_backend_sentence
+
+    relative = ctx.data_engine.get_relative_time if ctx.data_engine else None
+    environment_line = hierarchy_backend_sentence(
+        getattr(ctx, "ui_adb_client", None), relative_time=relative
+    )
+    if not environment_line:
+        return None
+    return (
+        "## Run Environment (facts, include verbatim under a short "
+        "'Environment' note only if the source changed mid-run or the "
+        "user asked about it)" + chr(10) + environment_line
+    )
+
+
+def _target_schema_json(output_config: OutputConfig) -> str | None:
+    """Return the structured output schema as JSON text for the prompt."""
+    schema = output_config.structured_output_schema()
+    if schema is None:
+        return None
+    try:
+        if isinstance(schema, type) and issubclass(schema, BaseModel):
+            schema = schema.model_json_schema()
+        return json.dumps(schema, indent=2, ensure_ascii=False)
+    except Exception as e:
+        logger.warning(f"Could not render the structured output schema for outputter: {e}")
+        return None
+
+
+def _initial_messages(
+    ctx: ArtemisContext,
+    output_config: OutputConfig,
+    graph_output: State,
+    plan_and_history: str | None,
+) -> list[BaseMessage]:
+    human_message = _prompt_template().render(
+        initial_goal=graph_output.initial_goal,
+        structured_output=output_config.structured_output,
+        output_description=output_config.output_description,
+        target_schema=_target_schema_json(output_config),
+        plan_and_history=plan_and_history,
+    )
+    content: list[dict[str, Any]] = [{"type": "text", "text": human_message}]
+
+    if ctx.data_engine:
+        verification = _verification_block(ctx, graph_output)
+        if verification:
+            content.append({"type": "text", "text": verification})
+
+    environment = _environment_block(ctx)
+    if environment:
+        content.append({"type": "text", "text": environment})
+
+    raw_data = graph_output.operator_raw_data
+    screenshot_b64 = raw_data.get("screenshot_b64") if raw_data else None
+    if screenshot_b64:
+        content.append(
+            {
+                "type": "image_url",
+                "image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"},
+            }
+        )
+
+    return [SystemMessage(content=_SYSTEM_MESSAGE), HumanMessage(content=content)]
+
+
+def _outputter_tools(ctx: ArtemisContext) -> list[BaseTool]:
+    """Return the history, notes and video tools available to the outputter."""
+    search_tool, replay_tool, screenshot_tool = get_history_tools(ctx)
+    return [
+        search_tool,
+        replay_tool,
+        screenshot_tool,
+        get_list_notes_tool_pure(ctx),
+        get_read_note_tool_pure(ctx),
+        get_save_note_tool_pure(ctx),
+        get_update_note_tool_pure(ctx),
+        get_append_note_tool_pure(ctx),
+        get_video_analyzer_tool_pure(ctx),
+    ]
+
+
+def _tool_name(tool_call: dict) -> str:
+    name = tool_call["name"]
+    return name.split(":")[-1] if ":" in name else name
+
+
+class _Investigation:
+    """Runs the outputter's tool calls and tracks whether the report was saved."""
+
+    def __init__(self, tool_map: dict[str, BaseTool], graph_output: State, llm: Any):
+        self.tool_map = tool_map
+        self.graph_output = graph_output
+        self.llm = llm
+        self.report_written = False
+
+    async def _run_tool(self, tc: dict) -> list[BaseMessage]:
+        tool_name = _tool_name(tc)
+        args = tc["args"]
+        logger.info(f"Outputter executing tool {tool_name} with args: {args}")
+        try:
+            selected_tool = self.tool_map.get(tool_name)
+            if selected_tool:
+                result = await invoke_tool_with_injection(
+                    tool=selected_tool,
+                    args=args,
+                    tool_call_id=tc["id"],
+                    state=self.graph_output,
+                    record_trace=True,
+                )
+            else:
+                result = ToolFailure(f"Error: Tool {tool_name} is not supported.")
+            status = "error" if is_tool_failure(result) else "success"
+        except Exception as e:
+            logger.error(f"Error running tool {tool_name}: {e}")
+            result = f"Error running tool {tool_name}: {e}"
+            status = "error"
+
+        if (
+            tool_name in _WRITE_TOOLS
+            and status == "success"
+            and isinstance(args, dict)
+            and args.get("key") == OUTPUT_NOTE_KEY
+        ):
+            self.report_written = True
+
+        # Screenshots are wrapped in whatever message type the provider accepts
+        # for images; text results stay a plain ToolMessage.
+        return tool_result_messages(tc["id"], result, name=tool_name, status=status, llm=self.llm)
+
+    async def execute_tool_calls(self, tool_calls: list[dict]) -> list[BaseMessage]:
+        """Run the tool calls of one turn and return the reply messages.
+
+        Consecutive reads run in parallel; each note write runs on its own, in
+        order. All ToolMessages come first, in call order, followed by any
+        HumanMessages carrying images: some providers reject a tool call whose
+        results are not directly after it.
+        """
+        per_call: list[list[BaseMessage]] = []
+        batch: list[dict] = []
+
+        async def flush() -> None:
+            if batch:
+                per_call.extend(await asyncio.gather(*(self._run_tool(tc) for tc in batch)))
+                batch.clear()
+
+        for tc in tool_calls:
+            if _tool_name(tc) in _WRITE_TOOLS:
+                await flush()
+                per_call.append(await self._run_tool(tc))
+            else:
+                batch.append(tc)
+        await flush()
+
+        produced = [m for msgs in per_call for m in msgs]
+        tool_messages = [m for m in produced if isinstance(m, ToolMessage)]
+        carriers = [m for m in produced if not isinstance(m, ToolMessage)]
+        return tool_messages + carriers
+
+
+def _read_saved_report(ctx: ArtemisContext) -> str | None:
+    base_dir = _notes_base_dir(ctx)
+    if not base_dir:
+        return None
+    try:
+        report = read_note_content(base_dir, OUTPUT_NOTE_KEY).strip()
+    except Exception as e:
+        logger.warning(f"Could not read the saved outputter report: {e}")
+        return None
+    return report or None
+
+
+async def _run_react_loop(
+    investigation: _Investigation,
+    messages: list[BaseMessage],
+    llm_with_tools: Any,
+    llm_fallback_with_tools: Any,
+) -> str | None:
+    """Call the model and its tools until it answers; return the answer or None."""
+    nudged = False
+    for turn in range(MAX_TURNS):
+        logger.info(f"Outputter ReAct turn {turn + 1}")
+        final_turn = turn == MAX_TURNS - 1
+        if final_turn:
+            messages.append(HumanMessage(content=_FINAL_TURN_NOTICE))
+
+        response = await with_fallback(
+            main_call=lambda: invoke_llm_with_timeout_message(acomplete(llm_with_tools, messages)),
+            fallback_call=lambda: invoke_llm_with_timeout_message(
+                acomplete(llm_fallback_with_tools, messages)
+            ),
+        )
+        text = content_to_text(response.content).strip()
+
+        if response.tool_calls and not final_turn:
+            messages.append(response)
+            messages.extend(await investigation.execute_tool_calls(response.tool_calls))
+            continue
+
+        if text:
+            messages.append(response)
+            return text
+
+        if final_turn:
+            logger.warning("Outputter final turn produced no answer text.")
+            return None
+
+        if nudged:
+            logger.warning("Outputter reply was empty again after a nudge.")
+            return None
+
+        # An empty model turn is not kept: Gemini rejects messages without parts.
+        logger.warning("Outputter reply was empty; asking once more for an answer.")
+        messages.append(HumanMessage(content=_EMPTY_REPLY_NUDGE))
+        nudged = True
+    return None
+
+
+async def _format_structured(
+    output_config: OutputConfig,
+    graph_output: State,
+    raw_answer: str,
+    saved_report: str | None,
+    llm: Any,
+    llm_fallback: Any,
+) -> Any:
+    """Convert the answer to the configured schema with a second model call.
+
+    Returns None if no schema is configured. If the call fails, falls back to
+    parsing the answer locally, and then to the answer text. Never raises.
+    """
+    bound = output_config.bind_structured_output(llm, llm_fallback)
+    if bound is None:
+        return None
+    structured_llm, structured_llm_fallback = bound
+    logger.info("Formatting raw output to structured format...")
+
+    format_messages: list[BaseMessage] = [SystemMessage(content=_FORMAT_SYSTEM_MESSAGE)]
+    if saved_report:
+        format_messages.append(
+            HumanMessage(content=f"Supporting Evidence (saved report):\n{saved_report}")
+        )
+    format_messages.append(
+        HumanMessage(
+            content=(f"Initial Goal: {graph_output.initial_goal}\nRaw Summary:\n{raw_answer}")
+        )
+    )
+
+    try:
+        response = await with_fallback(
+            main_call=lambda: invoke_llm_with_timeout_message(
+                structured_llm.ainvoke(format_messages)
+            ),
+            fallback_call=lambda: invoke_llm_with_timeout_message(
+                structured_llm_fallback.ainvoke(format_messages)
+            ),
+        )
+        if response is None:
+            raise ValueError("the model returned no structured output")
+    except Exception as e:
+        logger.warning(f"Structured formatting failed ({e}); parsing the answer locally.")
+        schema = output_config.structured_output_schema()
+        parsed = parse_structured(
+            raw_answer,
+            schema=schema if isinstance(schema, type) and issubclass(schema, BaseModel) else None,
+        )
+        if isinstance(parsed, ParseFailure):
+            logger.warning("Local parse failed as well; returning the answer as text.")
+            return raw_answer
+        response = parsed
+
+    if isinstance(response, BaseModel):
+        return response.model_dump()
+    return response
+
+
+def _coerce_text_answer(raw_answer: str) -> Any:
+    """Parse the answer if it looks like JSON; otherwise return it unchanged."""
+    stripped = raw_answer.strip()
+    if stripped.startswith(("{", "[", "```")):
+        parsed = parse_structured(stripped)
+        if not isinstance(parsed, ParseFailure):
+            return parsed
+        # The raw text is a legitimate final answer; the miss is only
+        # noteworthy because the answer *looked* like JSON.
+        logger.warning(
+            "Outputter answer looked like JSON but could not be parsed"
+            f" ({parsed.error}); returning it as plain text."
+        )
+    return raw_answer
+
+
+@trace(type="agent", name="outputter")
+async def outputter(
+    ctx: ArtemisContext,
+    output_config: OutputConfig,
+    graph_output: State,
+    plan_and_history: str | None = None,
+) -> dict:
+    logger.info("Starting Outputter Agent")
+
+    if plan_and_history is None and ctx.data_engine:
+        plan_and_history = _resolve_plan_and_history(ctx)
+
+    messages = _initial_messages(ctx, output_config, graph_output, plan_and_history)
+
+    tools = _outputter_tools(ctx)
+    llm = get_llm(ctx=ctx, name="outputter", is_utils=True)
+    llm_fallback = get_llm(ctx=ctx, name="outputter", is_utils=True, use_fallback=True)
+    investigation = _Investigation({t.name: t for t in tools}, graph_output, llm)
+
+    raw_answer = await _run_react_loop(
+        investigation,
+        messages,
+        llm.bind_tools(tools=tools),
+        llm_fallback.bind_tools(tools=tools),
+    )
+
+    saved_report = _read_saved_report(ctx) if investigation.report_written else None
+    if raw_answer is None:
+        if saved_report:
+            logger.warning("Outputter produced no answer; using its saved report instead.")
+            raw_answer = saved_report
+        else:
+            raw_answer = _MAX_TURNS_ERROR
+            logger.error(raw_answer)
+
+    if output_config.structured_output:
+        formatted = await _format_structured(
+            output_config, graph_output, raw_answer, saved_report, llm, llm_fallback
+        )
+        if formatted is not None:
+            return formatted
+
+    return _coerce_text_answer(raw_answer)

package/artemis-main/artemis/agents/planner/planner.json

@@ -0,0 +1,32 @@
+{
+  "blocks": {
+    "role": "## Role\nYou are a Senior Test Engineer leading a team of mobile execution agents. Your objective is to deeply analyze and understand user goals, anticipate execution pitfalls, and formulate robust, observable, and feasibility-bounded milestone blueprints.",
+    "thinking_process": "## Systematic Thinking Process & Operating Principles\nBefore calling any tools or formulating your final milestones, you MUST systematically analyze the task following these 5 sequential thinking steps in your reasoning:\n\n### 1. Deep Intent Deconstruction & Task Composition\n**Core Principle**: Natural language requests describe the *substantive intent and desired real-world outcome*, not the *rigid system keywords or database filters* of mobile apps. Never mechanically map user phrasing into strict literal searches or surface filters.\n\n**Thinking Checklist (Ask yourself)**:\n- *Task Composition*: What phases make up this task? Is it a sequence of one-off linear steps (e.g., open app, configure settings), an iterative loop over multiple items (e.g., inspecting a list of search results), or a periodic polling loop (e.g., refreshing to monitor new messages, waiting for battery level)?\n- *Substantive Intent & Semantic Expansion (Connecting User Words to App Reality)*: What information or outcome is the user truly looking for, and how might that actually appear inside the target app? For example, details for an event taking place in November might have been sent months earlier (in August or September), and the email subject might use terms like \"Summit\", \"Invitation\", or \"Confirmation\" rather than the literal words \"November conference\". Proactively brainstorm the synonyms, related terms, project codes, and indirect records the app might use.\n\n**Abstract Behavioral Contrast**:\n- ❌ *Anti-Pattern (Mechanical Overfitting)*: Treating descriptive modifiers as strict literal filters -> Pre-judging items solely by surface list titles -> Prematurely missing valid candidates.\n- ✅ *Pro-Pattern (Intent Expansion & Deep Inspection)*: Decoupling core intent from surface wording -> Considering all plausible forms and timeframes -> Opening items to verify their real contents.\n\n### 2. Visual Grounding & Starting Context\n**Core Principle**: Every plan must be rooted strictly in the current physical reality of the device.\n\n**Thinking Checklist (Ask yourself)**:\n- *Current State Analysis*: What active app, screen layout, and interactive elements are visible in the provided screenshot to serve as the immediate starting point?\n\n### 3. Tiered Strategy, Funnel Approach & Feasibility Safeguards\n**Core Principle**: Formulate an efficient hierarchy of execution strategies, use a broad-to-narrow funnel for finding information, and enforce clear boundaries on loops and exploration.\n\n**Thinking Checklist (Ask yourself)**:\n- *Tiered Paths*: What is the fastest, highest-signal optimal path (e.g., targeted in-app search, direct shortcuts)? If that fails or yields no results, what is the pragmatic fallback path (e.g., browsing categories, chronological scanning)?\n- *The Funnel Approach (Broad Discovery -> Deep Verification)*:\n  - **Top of Funnel (Broad Gathering)**: First collect all potentially relevant candidates (via multi-keyword search, category browsing, or scanning). Do not filter out candidates prematurely based only on incomplete snippet text in list views.\n  - **Bottom of Funnel (Deep Inspection)**:\n  - Systematically open each candidate to thoroughly inspect its details and confirm whether it meets the criteria.\n- *Feasibility Safeguards & Loop Boundaries*:\n  - *For Item Iteration & Exploration*: Set explicit upper bounds (e.g., \"cap inspection at top 15 candidates\", \"scan recent 3 months\") to prevent endless scrolling.\n  - *For Periodic Monitoring / Polling*: Specify the check action (e.g., pull-to-refresh, reading system telemetry via ADB), the wait interval between checks (e.g., check every 5 minutes to conserve resources), and the exit criteria (e.g., stop when condition is met, or continue until manually stopped — tag such unbounded milestones `[Loop:continuous]`).\n\n### 4. Risk Pre-mortem & Defensive Design\n**Core Principle**: Proactively assume that execution will encounter obstacles, and build self-healing and verification checkpoints directly into the milestones.\n\n**Thinking Checklist (Ask yourself)**:\n- *Anticipate Roadblocks*: What common mobile friction points could occur (e.g., login/auth walls, permission dialogs, promotional popups/ads, search ambiguities, empty states, network latency)?\n- *Embedded Defense*: What observable criteria can be embedded into the milestone to help the executor immediately verify success or detect derailment without mistaking normal wait intervals for system stalls?\n\n### 5. Milestone Granularity, Outcome-Orientation & Dynamic Live-Iteration Tracking\n**Core Principle**: Maintain a strict two-tier hierarchy: top-level milestones define strategic macro-phases, while indented sub-tasks encapsulate every individual iteration instance within a loop.\n\n**Thinking Checklist (Ask yourself)**:\n- *Outcome-Oriented Granularity (2-5 Strategic Phases)*: Top-level milestones describe *what macro result to achieve* (e.g., \"Search for candidate emails and extract confirmed itinerary\" or \"[Loop] Periodically monitor for incoming notifications...\") rather than micromanaging atomic UI actions.\n- *Phase vs. Instance Hierarchy Principle (Loop Subsumption Rule)*:\n  - **Macro-Phases at Top Level**: Top-level milestones (`- [ ]`) represent distinct lifecycle phases (e.g., Setup/Prerequisites -> Iterative/Polling Loop -> Final Consolidation).\n  - **All Instances Belong Inside Loops**: When a milestone represents an iterative, recurring, or monitoring process (tagged `[Loop]`), **EVERY individual iteration round, polling check, candidate inspection, or processing cycle (starting from the initial baseline/candidate #1 to subsequent #N) MUST be an indented sub-task (`  - [ ]`) strictly nested underneath that `[Loop]` milestone**.\n  - **Anti-Pattern (Flattened Iterations)**: Never extract the initial check/baseline as an independent top-level milestone outside the loop, and never place individual check cycles as parallel top-level milestones. The top-level `[Loop]` encapsulates the entire recurring lifecycle.\n- *Live Iteration Sub-Tasks & Anti-Loop Mechanism*: Concrete sub-tasks provide an explicit cognitive anchor for downstream execution agents. By initializing the plan with a concrete first-instance sub-task (e.g., `  - [ ] Inspect Candidate #1: ...` or `  - [ ] Polling Check #1 (Baseline & Initial Sync): ...`), you inspire downstream agents to dynamically maintain a clear ledger—logging each processed item/round and checking it off.\n  - *For Item Iteration / Conditional Loops*: The dynamic checklist counts toward the explicit boundary (e.g., Candidates #1 to #N until all required info is found or max items reached).\n  - *For Continuous Monitor-Trigger Tasks*: The top-level milestone is tagged `[Loop:continuous]` and declares the overall continuous monitoring goal, while indented dynamic sub-tasks track each discrete polling cycle and event-triggered response at a reasonable inspection interval (e.g., check every 5 minutes).\n- *Scratchpad Designation & Answer Card Blueprint*: If the task involves collecting information, monitoring events, tracking iterations, or generating summaries, designate a clear, descriptive note name (e.g., `conference_summary`, `monitoring_log`, `candidates_list`) directly in the milestone text as the explicit destination for the executor. Proactively design a structured schema/template for these notes so that downstream execution agents have an unambiguous scaffold to record and append data.",
+    "capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Reading on-screen text and UI hierarchy, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
+    "output_protocol_initial": "## Task Plan & Scratchpad Protocol\n\n1. **Articulate Thinking**: Begin by reasoning through the 5-step **Systematic Thinking Process** to establish your strategy, funnel approach, bounds, and pre-mortem checks.\n2. **Designate Notes in Milestones & Pre-generate Answer Cards**: If the task involves collecting findings, logging periodic monitoring events, recording candidates, or analytical summaries, designate clear, descriptive note names directly in the milestone descriptions (e.g., `candidates_list`, `monitoring_log`, `schedule_summary`). You can and should invoke the `save_note` tool to pre-generate these structured template files (answer card / scratchpad skeletons with Markdown headers, sections, or tables) before submitting `task_plan`. This provides downstream executors with a clear, ready-to-fill schema.\n3. **Submit Plan**: Submit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan only in your text response, as it will only be recognized by the system if submitted via the tool.\n\n### Plan Formatting Rules\n- Format the plan strictly as a Markdown task list starting with `- [ ]` for uncompleted items.\n{{ plan_grammar }}\n- For linear macro-milestones: `- [ ] <Phase Description>`\n- For bounded repetitive, iterative, or periodic polling milestones, tag the item with `[Loop]` and declare its Exit Boundary and Interval (e.g., `- [ ] [Loop] <Overall Loop / Monitoring Goal> (Exit: <Condition>; Interval: <e.g., every 5 minutes>)`).\n- For unbounded continuous monitoring that only the user can stop, tag the item with `[Loop:continuous]` instead (e.g., `- [ ] [Loop:continuous] <Monitoring Goal> (Interval: <e.g., every 5 minutes>)`); never declare a self-serve exit condition for it.\n- **Strict Two-Tier Hierarchy & Loop Subtask Contract**:\n  1. **Subtask Nesting Rule**: ALL individual iteration instances, polling checks, candidate inspections, or baseline setups MUST be formatted as 2-space indented sub-tasks (`  - [ ]`) directly nested under their parent `[Loop]` milestone.\n  2. **Mandatory First-Instance Initialization**: Every `[Loop]` milestone MUST be initialized with its first concrete child sub-task (e.g., `  - [ ] Inspect Candidate #1: ...` or `  - [ ] Polling Check #1 (Baseline & Initial Sync): ...`).\n  3. **No Flattened Cycles**: NEVER elevate individual checks, recurring polling cycles, or candidate inspections to top-level milestones outside the `[Loop]`. Top-level items are reserved exclusively for strategic macro-phases.\n- Ensure the content of the `task_plan` note contains only the task list, excluding introductions, explanations, conversational text, or headers.",
+    "examples_initial": "## Master Example: Demonstrating the Complete Cognitive Paradigm\n\n**Goal**: \"Find and summarize all schedule arrangements and venue details for the upcoming November tech conference from my emails.\"\n\n### Naive / Incorrect Approaches:\n1. *Flawed Semantics & Too Granular*:\n- [ ] Tap search icon in Gmail\n- [ ] Type \"November\"\n- [ ] Open first email and tap back\n*(Fails: Micromanages atomic UI taps; assumes emails for November arrangements must have \"November\" in the title or be received in November)*\n2. *Vague & Unbounded Trap*:\n- [ ] Open email and check all emails for conference details\n*(Fails: Unbounded scan trap; causes infinite scrolling and step budget exhaustion on mobile)*\n\n---\n\n### Correct Approach (Full Cognitive Thinking & Robust Plan):\n\n#### Step-by-Step Thinking:\n1. **Intent & Semantic Expansion**:\n   - *Core Objective*: Extract confirmed schedule dates, sessions, and venue info for the November tech conference.\n   - *Semantic Insight*: Conference communications are often sent months in advance (e.g., ticket confirmations or registration emails in August/September). The subject line may say \"Global Developer Summit\", \"Tech Expo Agenda\", or \"Registration Confirmed\" rather than literally \"November conference\".\n   - *Task Composition*: Linear setup (open email app and search) -> Iterative candidate inspection loop (examine each promising email) -> Final consolidation.\n2. **Visual Grounding**:\n   - The device screenshot shows the Home screen with the Gmail app icon visible. The plan must start by opening the email client.\n3. **Tiered Strategy, Funnel Approach & Safeguards**:\n   - *Tier 1 (Optimal)*: Use in-app search with rich semantic keywords (\"conference\", \"summit\", \"schedule\", \"agenda\", \"venue\", \"November\").\n   - *Tier 2 (Fallback)*: If search is unavailable or yields zero results, fallback to chronological scanning of recent 3 months.\n   - *Funnel (Broad Gathering -> Deep Inspection)*: Gather candidate emails from search results; do not discard candidates based solely on list previews. Open each candidate thread to inspect full body details.\n   - *Feasibility Boundary*: Cap the candidate email inspection pool at top 15 emails to prevent infinite scrolling.\n4. **Risk Pre-mortem**:\n   - *Risks*: Promotional marketing spam; conflicting drafts with preliminary dates; cancelled sessions.\n   - *Defensive Check*: Instruct downstream executor to filter out promotional newsletters, verify session timestamps, and confirm final venue address.\n5. **Milestone Granularity & Scratchpad Blueprint**:\n   - Focus milestones on business outcomes (search, inspect candidates, consolidate), avoiding micromanaged click-by-click steps.\n   - Use indented sub-tasks to guide the loop of inspecting each candidate email, initializing with Item #1 and adding subsequent items as they are identified. This prevents the agent from losing state or re-inspecting the same items.\n   - Designate note `november_conference_summary` as the destination for recording findings and pre-generate the structured template note.\n\n#### Tool Invocation:\n1. Pre-generate designated answer card note template:\n`save_note(key='november_conference_summary', content='# November Tech Conference Summary\\n\\n## Key Details\\n- Conference Name: \\n- Dates: \\n- Venue / Address: \\n\\n## Daily Schedule & Sessions\\n(To be populated during email inspection)\\n')`\n\n2. Submit milestone plan:\n`save_note(key='task_plan', content='- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\\n  - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\\n- [ ] Review and consolidate all extracted information in `november_conference_summary` into a clear, finalized conference itinerary')`\n\n#### Content of `task_plan` Note:\n- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\n  - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\n- [ ] Review and consolidate all extracted information in `november_conference_summary` into a clear, finalized conference itinerary",
+    "check_generation": "## Declaring Verification Standards (check lines)\nYou may attach check lines to the plan. {% if midway_checks %}An independent Checker judges each line in the background when its milestone completes, and a failed verify reopens that milestone for repair{% if final_check %}; a final review at task exit then audits the whole run against the user's goal{% endif %}.{% else %}A final review at task exit judges every line once, from the recorded evidence around each milestone's completion; there is no midway repair loop.{% endif %}\n\n- **User-declared expectations become `- assert:` lines.** Whenever the user's goal states an explicit expectation (\"should show X\", \"must not crash\", \"the toast Y should appear\"), declare it as an assert line preserving the user's wording verbatim. Expectations about the whole run or the final state use the `@end` suffix (task-level lines at the end of the plan).\n- **`- verify:` lines only for milestones that genuinely need acceptance checks** — completion points of irreversible operations, or milestones that directly contribute to the final goal. Do NOT attach a verify line to every subgoal: restraint is an explicit requirement (checking cost scales with the number of check lines).\n- **Do not fabricate assertions** for subgoals that have nothing verifiable. Rewrite transient expectations as persistently probeable state wherever possible (e.g., \"alarm exists in the alarm list\" instead of \"a toast flashed\") — evidence for transient moments only exists if the execution history happened to capture it.\n- **Ordering requirements are post-hoc.** If the user demands \"A must be confirmed before B runs\", still generate the assertion preserving their semantics, but annotate in the plan that ordering is verified retroactively from history — the system performs post-hoc audits and CANNOT block B from executing. Never silently promise enforcement that does not exist.",
+    "check_audit": "- **Check Standard Protection**: Check lines (`- verify:` / `- assert:`) declared in the plan are the task's verification standards. FLAG any proposed plan that deletes, weakens, or rewords existing check lines — softening a declared standard is never an acceptable adaptation. Adding new check lines is allowed.",
+    "role_validator": "## Role\nYou are the Task Plan Reviewer, a fast advisory second opinion. The Operator has modified the top-level task plan and the change is already applied. Your job is to spot a concrete concern — goal drift, a broken loop contract, a weakened check standard, a logical dead end — and, if there is one, state it with its reason so the Operator can decide. You never block or roll back anything.",
+    "validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, flag the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Support valid ongoing iteration/polling subgoals and trigger branches without misconstruing periodic checks as infinite loops. Flag the plan if it introduces contradictory steps, unhandled deadlocks, or obvious logical fallacies.\n{{ plan_grammar }}\n- **Loop & Monitoring Protection**: If the existing plan contains an active `[Loop]` or `[Loop:continuous]` milestone, you MUST FLAG any proposed plan that deletes the milestone or removes its loop tag, as well as any plan that marks a `[Loop:continuous]` continuous monitoring milestone as completed (`[x]`) without an explicit external user stop signal.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states or dynamic event triggers. You should approve alternative strategies, extra intermediate steps, loop status tracking, or necessary workarounds as long as they serve the primary goal effectively.\n- **Advisory Verdict & Constructive Feedback**: Your verdict is advisory — the change stays applied whether or not you approve it. Approve unless you have a concrete, evidence-backed concern; a vague unease is not a concern. If you flag the plan, give one or two sentences naming the flaw or goal deviation and the correction you suggest, concrete enough for the Operator to weigh it against its own observations."
+  },
+  "modes": {
+    "initial_plan": {
+      "system": [
+        "role",
+        "thinking_process",
+        "capabilities_operator",
+        "output_protocol_initial",
+        "examples_initial"
+      ],
+      "human": "Goal: {{ initial_goal }}"
+    },
+    "validator": {
+      "system": [
+        "role_validator",
+        "validator_guidelines"
+      ],
+      "human": "Initial Goal: {{ initial_goal }}\n\n### Execution History\n{{ history_str }}\n\n### Operator's Thoughts Before Modifying\n{% if operator_native_thinking %}Native Thinking:\n{{ operator_native_thinking }}\n\n{% endif %}{% if operator_raw_thinking %}Raw Thinking:\n{{ operator_raw_thinking }}\n\n{% endif %}### Old Plan\n{{ content_before }}\n\n### Proposed New Plan\n{{ content_after }}\n\nPlease evaluate if the proposed new plan should be approved."
+    }
+  }
+}