@loadmill/droid-cua 2.29.0 → 2.30.0

package/skill/droid-cua/references/running-and-healing.md

@@ -2,38 +2,38 @@
 
 ## Construct the run
 
-Prefer a single test before a directory or scenario run. Preserve an existing provider/model choice in `ci/droid-cua.json`; otherwise use the Loadmill provider. Use Loadmill Smart as the recommended default, or Loadmill Pulse when the user prioritizes faster, lower-cost execution:
+Prefer a single test before a directory or scenario run. Preserve an existing provider/model choice in `ci/droid-cua.json`; otherwise use the Loadmill provider with Loadmill Pulse as the recommended model for daily authoring, local runs, and CI. Escalate to Loadmill Smart when stability is a demonstrated problem and run evidence shows Smart improves it:
 
 ```sh
-# Recommended default
---llm-provider loadmill --cua-model loadmill-smart
-
-# Faster, lower-cost option
+# Recommended daily model
 --llm-provider loadmill --cua-model loadmill-pulse
+
+# Evidence-driven stability escalation
+--llm-provider loadmill --cua-model loadmill-smart
 ```
 
-All Loadmill models require `LOADMILL_API_TOKEN`. Do not switch between Smart, Pulse, and Beacon silently; preserve the project's model or explain the capability, speed, and experimental tradeoffs and follow the user's choice. If the user chooses direct OpenAI access instead, require `OPENAI_API_KEY` and use `--llm-provider openai` with one of `gpt-5.6-terra`, `gpt-5.6-luna`, or `gpt-5.4`. Do not mix a provider with another provider's model, silently switch providers, or override an existing project choice without explaining it.
+All Loadmill models require `LOADMILL_API_TOKEN`. Do not switch from Pulse to Smart merely because a test failed or was slow; first identify a model-related stability issue, then keep Smart only when comparison evidence shows an improvement. Do not switch between Smart, Pulse, and Beacon silently; preserve the project's model or explain the capability, speed, and experimental tradeoffs and follow the user's choice. If the user chooses direct OpenAI access instead, require `OPENAI_API_KEY` and use `--llm-provider openai` with one of `gpt-5.6-terra`, `gpt-5.6-luna`, or `gpt-5.4`. Do not mix a provider with another provider's model, silently switch providers, or override an existing project choice without explaining it.
 
 Include `--report reports/<test-name>.html` for normal test runs unless the user opts out. Prefer an explicit filename over a directory so the result is easy to locate. Add `--debug` for every agent-observed run: it writes the screenshots needed for live progress updates as well as diagnosis.
 
 ```sh
 # Attached Android device or running emulator
-droid-cua --llm-provider loadmill --cua-model loadmill-smart --avd adb:<serial> --instructions tests/example.dcua --context tests/context.md --debug --report reports/example.html
+droid-cua --llm-provider loadmill --cua-model loadmill-pulse --avd adb:<serial> --instructions tests/example.dcua --context tests/context.md --debug --report reports/example.html
 
 # Launchable Android AVD
 droid-cua --avd avd:<avd-name> --instructions tests/example.dcua --config ci/droid-cua.json --debug --report reports/example.html
 
 # iOS Simulator
-droid-cua --llm-provider loadmill --cua-model loadmill-smart --platform ios --avd "<simulator-name>" --instructions tests/example.dcua --debug --report reports/example.html
+droid-cua --llm-provider loadmill --cua-model loadmill-pulse --platform ios --avd "<simulator-name>" --instructions tests/example.dcua --debug --report reports/example.html
 
 # Local web
-droid-cua --llm-provider loadmill --cua-model loadmill-smart --target web --browser chrome --session-mode fresh --instructions tests/example.dcua --debug --report reports/example.html
+droid-cua --llm-provider loadmill --cua-model loadmill-pulse --target web --browser chrome --session-mode fresh --instructions tests/example.dcua --debug --report reports/example.html
 
 # LambdaTest
-droid-cua --llm-provider loadmill --cua-model loadmill-smart --device-source lambdatest --platform android --device-name "<name>" --os-version "<version>" --app <app.apk> --instructions tests/example.dcua --debug --report reports/example.html
+droid-cua --llm-provider loadmill --cua-model loadmill-pulse --device-source lambdatest --platform android --device-name "<name>" --os-version "<version>" --app <app.apk> --instructions tests/example.dcua --debug --report reports/example.html
 
 # AWS Device Farm
-droid-cua --llm-provider loadmill --cua-model loadmill-smart --device-source aws-device-farm --platform android --device-name "<name>" --os-version "<version>" --app <app.apk> --instructions tests/example.dcua --debug --report reports/example.html
+droid-cua --llm-provider loadmill --cua-model loadmill-pulse --device-source aws-device-farm --platform android --device-name "<name>" --os-version "<version>" --app <app.apk> --instructions tests/example.dcua --debug --report reports/example.html
 
 # Direct OpenAI alternative
 droid-cua --llm-provider openai --cua-model gpt-5.4 --avd adb:<serial> --instructions tests/example.dcua --context tests/context.md --debug --report reports/example.html
@@ -43,7 +43,7 @@ Use `.ipa` and `--platform ios` for supported cloud iOS runs. Preserve existing
 
 ## Confirm the run with the user
 
-Before starting any test command, send the user a short, plain-language run plan and wait for explicit confirmation. Treat this as a product conversation, not a command preview. Do not infer that a request to run a test also confirms an automatically selected device, browser, or batch of tests.
+Before starting any test command, send the user a short, plain-language run plan and wait for explicit confirmation. Treat this as a product conversation, not a command preview. Do not infer that a request to run a test also confirms an automatically selected device, browser, or batch of tests. For an explicitly requested author-run-improve workflow, the plan may ask approval for the baseline and the evidence-driven edit/rerun and confidence runs needed to reach a readiness decision; follow [iterative-improvement.md](iterative-improvement.md).
 
 The run plan should briefly state:
 
@@ -54,7 +54,7 @@ The run plan should briefly state:
 
 Do not include CLI flags, commands, provider or model names, credential/key names, config/context filenames, or internal log/report paths unless the user asks or one changes a decision they need to make. Do not paste the test source into the confirmation.
 
-Ask the user about every unresolved or ambiguous choice before running. If the plan is complete, ask for a clear go-ahead (for example, `Ready to run this?`). Start only after they confirm. If they change the tests, target, or other material run detail, show the revised plan and confirm it again.
+Ask the user about every unresolved or ambiguous choice before running. If the plan is complete, ask for a clear go-ahead (for example, `Ready to run this?`). Start only after they confirm. If they change the target, app build, test scope, provider/model, or another user-visible consequence, show the revised plan and confirm it again. Wording, context, config experiments, targeted reruns, and final confidence runs within an approved iterative workflow do not require separate confirmation when the target and side effects stay unchanged.
 
 For example: `I’m ready to run the Chrome smoke test in a fresh local Chrome window. It will check that Chrome opens and then closes. I’ll share screenshots as it progresses and provide a report at the end. Ready to run this?`
 
@@ -62,6 +62,14 @@ For example: `I’m ready to run the Chrome smoke test in a fresh local Chrome w
 
 Remain attached until the command exits and cloud cleanup finishes. Run the command through an interactive/pollable terminal mechanism so progress and new artifacts can be observed before it exits. Read terminal progress as it appears; do not treat a long model step as success or failure before completion.
 
+## Diagnose Loadmill flow failures
+
+When an instruction that invokes a saved Loadmill flow fails, inspect the `[Loadmill] FAILED` block first. Capture the command, the `Error:` text, and the suite run ID. Droid asks Loadmill for a failure explanation automatically, so the `Error:` text contains Loadmill's reason when one is available.
+
+If that reason is insufficient, open the suite run from the Loadmill Flow card in the desktop app and inspect the failed flow step and its available request, response, and assertion evidence. For a headless run, use the reported suite run ID and Loadmill site base URL to locate the same run at `/app/api-tests/test-suite-runs/<suite-run-id>`. Do not expose secrets from Loadmill evidence in chat, logs, or reports.
+
+Classify a flow that started and then failed inside Loadmill as a **Loadmill flow failure**, not a Droid UI-test failure. Do not rewrite later UI instructions or assertions to hide it. Correct the `.dcua` instruction only when the evidence shows Droid selected the wrong flow or passed the wrong parameters because the request was ambiguous. Retry the unchanged instruction once only for a transient Loadmill service or network failure; otherwise stop and report the Loadmill explanation, failing flow step, and suite-run link or ID.
+
 ## Show live visual progress
 
 Use the screenshots already captured by `--debug` to make the run understandable while it is happening. This is a required user-facing behavior, not merely an artifact to inspect after the run. After the CLI prints `Debug logging enabled: <logs/execution-...>.jsonl`, new PNGs appear in that log file's sibling directory. Keep the CLI running in a pollable/background session and poll that directory while the test is running.
@@ -95,6 +103,16 @@ After the run, verify that the report exists and resolve its absolute path. Open
 
 Report the outcome, failing instruction and first divergence when applicable, relevant error text, screenshot/debug/report paths, target, provider/model, and command with secret values omitted.
 
+After all requested running and healing work is complete, end with a short plain-language retrospective. Summarize the overall journey rather than listing every action or rerun:
+
+- What behavior was tested and whether it is ready.
+- The main ambiguity, failure, slow step, recovery loop, or other bottleneck found.
+- What changed in the test, context, data, or configuration and why it addressed that evidence.
+- The most meaningful before/after timing and stability results, such as total or hotspot duration, retries, recoveries, repeated actions, or consistency. Use exact numbers only when the artifacts support them; otherwise describe the observed improvement without inventing precision.
+- Any remaining risk or blocker, followed by links to the final report and relevant evidence.
+
+Keep this retrospective concise and lead with the result and improvement. Do not make the user reconstruct what happened from terminal output, screenshots, reports, or a chronological transcript.
+
 ## Classify the first divergence
 
 1. **Setup or authentication**: missing/invalid provider credential, dependency, license, browser, or executable. Return to onboarding and rerun only the failed check.
@@ -107,4 +125,4 @@ Report the outcome, failing instruction and first divergence when applicable, re
 
 Before editing, state the classification and evidence. Change only `.dcua`, `context.md`, `test-data.md`, or Droid CUA config. Never edit app code, delete coverage, loosen the expected outcome, or add broad waits simply to obtain green output.
 
-Make one minimal change, explain why it corrects the test rather than hides a product defect, and rerun only the affected test. Allow at most two edit/rerun cycles. If it still fails, stop with the original and final evidence, changes attempted, and recommended owner/action.
+Apply the smallest evidence-supported batch, explain why each change corrects the test rather than hides a product defect, and rerun only the affected test. Continue analyzing, editing, and rerunning until the final representative evidence supports readiness or a genuine blocker remains. Do not stop based on a fixed rerun count. Stop when the evidence indicates an application regression, an environment or product dependency prevents progress, or no evidence-supported test, context, data, or configuration improvement remains; report the original and final evidence, changes attempted, and recommended owner/action.