@loadmill/droid-cua 2.38.0 → 2.40.0

package/skill/droid-cua/references/authoring-tests.md

@@ -4,32 +4,91 @@
 
 Find nearby `.dcua` tests and project files before writing. Match their directory layout, naming, vocabulary, starting state, and config conventions. Do not open `.env` or `.secrets`.
 
+## Establish intended coverage
+
+Before drafting or healing, identify the requested behavior, starting state, scenario inputs, business outcome, and cleanup from the user request and existing test. Keep this brief in the working plan; no separate specification file is required. Distinguish user-required checks from incidental checks introduced during earlier agent iterations. If an existing assertion's purpose is unclear, preserve it until its scope is resolved.
+
+Preserve scenario choices during simplification: a full-day booking and a three-hour booking exercise different behavior. Adding UI cancellation also changes coverage; do it when requested, not automatically for every creation test.
+
+Prefer predictable setup and data over conditional recovery. Use known cleanup and simple relative dates when appropriate for the project. A date such as next Monday assumes a working day and available account state; establish those assumptions rather than inventing a search for the earliest conflict-free date. Restarting the app is a project-specific starting condition, not a universal requirement. Include only setup needed by this scenario and supported by the request, project knowledge, or observed behavior. Do not import dates, reset flows, or other prerequisites from a nearby example just because they made that example runnable. If an unknown prerequisite matters, identify the gap rather than silently inventing a mandatory step.
+
 ## Write `.dcua` instructions
 
-- Put one clear user goal or assertion on each non-empty line. One instruction does not need to equal one UI action.
-- Write concise, human-readable business or journey instructions that describe what the user wants to accomplish. Prefer `Select the desired date` or `Log out` over separate instructions for opening a picker or menu, clicking controls, handling an expected confirmation, and submitting.
-- Do not prescribe coordinates, selectors, implementation details, model strategy, or every tap and keystroke. These micro-steps couple the test to incidental UI structure, add model rounds, and make harmless UI changes more likely to break it.
-- Break a business-level instruction into smaller actions only when run evidence shows that the agent cannot complete it reliably or the intermediate state is itself behavior the test must verify.
-- State the starting condition when it is not guaranteed by the environment.
-- Use explicit assertions for important outcomes. Prefer stable visible facts such as a screen title, confirmation message, changed value, or navigation result.
-- Use exact visible labels when naming controls or outcomes.
+- Put one readable micro-goal on each non-empty line: a small user task with a recognizable completion point, such as choosing a room, selecting dates, or submitting a request. One goal can require several UI actions.
+- Keep useful task boundaries. Do not compress login, navigation, data entry, and verification into a single instruction to reduce the line count.
+- Specify the scenario choices precisely while leaving ordinary UI mechanics to Droid. Avoid coordinates, selectors, icon appearance, and instructions for every tap or keystroke.
+- Use visible labels when they identify a meaningful choice or resolve ambiguity. Short navigation hints are welcome when useful, such as `Custom duration (scroll right in the duration options)`; do not require exact layout or incidental copy.
+- Add lower-level guidance only when evidence shows it is necessary to complete the goal reliably, or the intermediate behavior is explicitly under test. Prefer clarifying the goal over appending a transcript of a failed run.
 - Keep each new test focused on one behavior. Separate unrelated flows only when each resulting test is fully independent and does not rely on execution order or state created by another test.
-- When updating or optimizing an existing test, preserve its complete flow in one `.dcua` file. Do not split it into sequential or dependent tests unless the user explicitly requests that structure and the project has a verified orchestration mechanism that guarantees ordering and shared state.
-- Use `//` comments sparingly for human context; comments are not executed.
-- Never include secret values in the test.
+- When updating or optimizing an existing test, preserve its complete intended flow in one independently runnable `.dcua` file. Do not split it into sequential or dependent tests without explicit user direction and verified orchestration.
+- Use `//` comments sparingly for human context; comments are not executed. Never include secret values.
+
+## Choose assertions by coverage
+
+Ask: **If this check failed while the requested business outcome remained correct, would that be a defect this scenario is intended to catch?** If not, it generally should not be a standalone assertion here. Droid must observe intermediate states to perform actions; those observations do not all need to become test assertions.
+
+Verify meaningful outcomes and identify the created or changed object by the relevant scenario inputs. Checking that the request for the selected room, dates, and duration is confirmed prevents an unrelated approved request from satisfying the test. Do not automatically assert every dialog closure, available option, summary label, or absence of a warning when the final outcome already covers the intended behavior.
+
+Preserve explicit user requirements and genuine product failures. Remove an agent-added assertion only when it is demonstrably outside intended coverage or redundant with a retained check; explain that coverage-based reason, not merely that it failed or cost time. An intermediate control or exact message still needs an assertion when that behavior is the requested subject of the test.
+
+## Example: readable booking journeys
+
+These examples use a fictional room-booking app, generic test accounts, and invented flow and control names. They do not describe a customer's product. Assume the project maps the test account to configured secret keys and the named cleanup flow makes the sample room available. This hypothetical user explicitly wants a fresh app start and UI cancellation coverage. Adapt those choices to the actual project; neither is a universal requirement.
+
+Full-day booking:
+
+```text
+Run the Loadmill "Reset sample room bookings" flow.
+Open the booking app; if already open, fully close it and reopen it.
+Log in as the test member, logging out first if another user is signed in.
+Go to Home.
+Start a room booking.
+Choose "Sample room".
+Select next Monday–Tuesday as full days.
+Submit the booking.
+Go to My bookings.
+Verify the new booking for Sample room and the selected dates is confirmed for 2 full days.
+Open that booking and cancel it through its menu.
+Verify that booking no longer appears in My bookings.
+```
+
+Custom-duration booking:
+
+```text
+Run the Loadmill "Reset sample room bookings" flow.
+Open the booking app; if already open, fully close it and reopen it.
+Log in as the test member, logging out first if another user is signed in.
+Go to Home.
+Start a room booking.
+Choose "Sample room".
+Select next Monday at 09:00.
+Choose "Custom duration" (scroll right in the duration options) and set 3 hours.
+Submit the booking.
+Go to My bookings.
+Verify the new booking for Sample room on the selected date at 09:00 is confirmed for 3 hours.
+Open that booking and cancel it through its menu.
+Verify that booking no longer appears in My bookings.
+```
+
+The final assertions identify the requested result and cleanup. Neither test checks unrelated duration options or how the calendar collapses. The short scrolling hint helps find a relevant choice without scripting the entire interaction.
+
+If a run leaves the calendar selection unconfirmed, first clarify `Select next Monday–Tuesday as full days` to `Select next Monday–Tuesday as full days and confirm the date selection.` Avoid accumulating instructions such as “tap Apply now even if the total updated” plus assertions that the grid and button disappear. Verify that the clearer goal resolves the observed failure; retain a specific control hint only if it is still needed. Do not hide an unresolved Droid or product defect in test prose.
+
+## Example: account preferences
 
-Example:
+This fictional settings app illustrates the same approach without booking dates, cleanup flows, or app restarts. The requested behavior is saving a notification preference and retaining it when settings are reopened; assume the test account initially uses Daily notifications.
 
 ```text
-Sign in using the USER_EMAIL and USER_PASSWORD secrets.
-Assert that the Home screen is visible and shows the signed-in account.
-Select December 15 as the requested date.
-Assert that December 15 is shown as the requested date.
-Log out.
-Assert that the sign-in screen is visible.
+Log in to the settings app as the test member.
+Open Notification settings.
+Change the email digest frequency to Weekly.
+Save the preferences.
+Leave Notification settings and reopen it.
+Verify the email digest frequency is still Weekly.
+Restore the email digest frequency to Daily and save.
 ```
 
-Keep meaningful outcomes explicit even when actions are high-level. For example, follow `Log out` with an assertion that the sign-in screen is visible when logout behavior is under test.
+The persistence check protects the requested behavior. Checking a toast, button color, or menu animation would add unrelated requirements. Restore the known starting value for repeatability; do not invent cleanup when the scenario or environment does not need it.
 
 Use the secret key names that exist in `.secrets`; do not inspect their values. If key discovery is necessary, filter the file so only keys reach output, for example `sed -n 's/^\([^#][^=]*\)=.*/\1/p' .secrets`. If a required key is absent, ask the user to add it without sharing its value.
 

package/skill/droid-cua/references/iterative-improvement.md

@@ -10,7 +10,7 @@ One confirmation may cover that iterative workflow. Continue without asking agai
 
 ## 2. Establish a reproducible baseline
 
-Run the narrowest test with its existing project model and config. Keep these fixed for the baseline, then change any of them only as part of a documented, evidence-supported optimization batch:
+Record the intended coverage using [authoring-tests.md](authoring-tests.md) before editing, including which assertions are user requirements. Run the narrowest test with its existing project model and config. Keep these fixed for the baseline, then change any of them only as part of a documented, evidence-supported optimization batch:
 
 - App build and initial account/data state.
 - Device, OS, device source, or browser session mode.
@@ -49,18 +49,18 @@ Stop healing when evidence indicates an application regression. Preserve the fai
 For long-running tests, fix all well-supported bottlenecks that can be addressed safely before paying for another full run. Use this order to build the batch unless evidence points elsewhere:
 
 1. **`.dcua` wording and scope**
-   - Express one human-readable business goal per instruction, not one click, menu, field, dialog, or submit action per line.
+   - Express one readable micro-goal per instruction. Choosing a room, selecting dates, and submitting a booking are useful separate goals; opening each menu or tapping each control usually is not.
    - Collapse consecutive micro-steps into the user outcome they jointly accomplish, such as `Select the desired date` or `Log out`, while keeping meaningful assertions explicit.
    - Use exact visible labels only when they disambiguate the goal or identify an outcome the test must verify.
-   - Add the missing starting condition or assertion at the first ambiguous transition.
+   - Resolve an ambiguous transition by checking the starting state, data, and wording first. Add an assertion only when it protects intended coverage; observing progress does not require a new test assertion.
    - Remove scenario steps that do not protect the behavior under test.
-   - Add lower-level actions only when the evidence shows the business-level instruction is unreliable or an intermediate state is part of the coverage.
+   - Rewrite the unclear goal before adding lower-level actions. Add a short hint only when the evidence shows it is still necessary or an intermediate state is part of the coverage; do not append emphatic commands and UI-state checks from each failed run.
    - Keep the existing scenario in one independently runnable `.dcua` file. Splitting a long test into ordered or state-dependent tests is not an available optimization because separate execution order and shared state are not guaranteed.
-   - Never remove or weaken a meaningful assertion to obtain a pass.
+   - Preserve required assertions and scenario choices. Remove agent-added, out-of-scope or redundant assertions only with the coverage-based explanation required by the authoring guide; preserve checks whose purpose is unresolved.
 
 2. **`context.md`**
    - Update the project's existing canonical context file; never create a test-specific context file for the optimization.
-   - Add durable product navigation, terminology, stable interruptions, and narrowly observed recovery behavior.
+   - Add durable product navigation, terminology, and stable interruptions. Do not move accumulated scenario recovery instructions into shared context to make the test appear simpler.
    - Move reusable mechanics out of the test, but keep scenario-specific values and expectations in the test.
    - Remove redundant, speculative, or conflicting instructions. Do not restate Droid's base behavior or encode a workaround for a Droid/product bug as product knowledge.
 
@@ -114,11 +114,14 @@ For every included change, record the evidence, rationale, expected effect, and
 
 ## 6. Compare and iterate
 
+After each edit, reread the whole test against its intended coverage. Remove superseded hints and redundant checks, keep one micro-goal per instruction, and ensure setup or data problems have not become speculative recovery branches. Keep diagnostic history in the report, not the test or shared context. Validate the simplified version in the next representative run.
+
 After each rerun, compare:
 
 - Did the first divergence disappear for the expected reason?
 - Did the hotspot's duration, rounds, actions, previews, retries, or recoveries improve?
 - Did another step regress or did coverage change?
+- Did the test become harder to read or more dependent on incidental UI details? Explain any retained complexity by coverage or execution evidence; fewer lines alone are not an improvement.
 - Did the combined change set improve the intended hotspots without obscuring a regression?
 
 Keep the batch when the evidence supports the overall result. If it regresses or remains ambiguous, use the timeline to isolate, revise, or revert only the suspect subset instead of automatically rerunning every change separately. Continue while the evidence supports another useful test, context, data, or configuration improvement. Stop and report the strongest remaining explanation only when a genuine blocker remains or no evidence-supported improvement is left to try.
@@ -129,6 +132,7 @@ After the optimized test passes, decide whether the accumulated evidence is stro
 
 Call the test ship-ready only when:
 
+- The final test still expresses the intended coverage as readable micro-goals, with no unnecessary assertions, stale debugging instructions, or speculative recovery branches.
 - All intended instructions and meaningful assertions pass in the final representative evidence.
 - No unexplained recovery, repeated-action loop, or recurring high-risk hotspot remains.
 - Timing and action counts are reasonably consistent, with transient provider retries identified separately.

package/skill/droid-cua/references/onboarding-and-troubleshooting.md

@@ -4,6 +4,12 @@ Follow the Desktop Device Setup Wizard sequence: choose a platform, run checks,
 
 Do not create or modify user test files merely to perform setup. Get approval before installing software, changing system configuration, running `sudo`, or accepting a license. After remediation, rerun only the failed check.
 
+## Identify the installed skill
+
+When diagnosing skill behavior, report the installed skill version and bundled skill version from `droid-cua skill status <agent>` (`cursor`, `claude-code`, or `codex`). Use `--local` for a project installation. The skill's own version is `metadata.version` in `SKILL.md`; it is independent of the CLI package version. If an older CLI does not display versions, read this field from the active skill copy instead.
+
+`unknown` means version metadata is unavailable, as with older copies. `modified` means the installed files differ from the bundled skill; it can indicate an older release or local edits, including edits that left the version unchanged. Report that state alongside the version. Updating the CLI does not update existing skill copies; explain any mismatch before using `droid-cua skill install <agent> --force`, which overwrites local changes.
+
 ## Droid CUA CLI compatibility
 
 If the CLI rejects a model, option, command, or capability that this skill explicitly documents, treat an outdated global `@loadmill/droid-cua` package as a likely setup issue. Do not suggest an update for an ordinary test, target, provider, or application failure.

package/skill/droid-cua/references/project-and-auth.md

@@ -40,4 +40,4 @@ Offer exactly these two choices, then wait for the user's selection:
 
 Use a secret-safe mechanism that does not print or expose the value in commands, logs, or responses. Verify only that the key is present. Never copy a token from another file, a shell profile, another project, an agent skill/configuration, or any other location. Do not inspect inherited environment variables to discover a token. When the active project's `.env` contains the key, run `droid-cua` normally from that project; the CLI loads that file automatically.
 
-Only when the user explicitly selects direct OpenAI access, use `--llm-provider openai` with one of the supported OpenAI CUA models: `gpt-5.6-terra`, `gpt-5.6-luna`, or `gpt-5.4`. Check only the active project's `.env` for `OPENAI_API_KEY`; do not use one found elsewhere. If it is missing, guide the user to create and configure their own key in the same project using the same secret-safe process. Do not silently switch providers to bypass missing Loadmill authentication.
+Only when the user explicitly selects direct OpenAI access, use `--llm-provider openai` with `gpt-6-sol` or `gpt-6-luna`. Saved direct GPT-5.6 Terra and GPT-5.4 selections migrate to GPT-6 Sol; saved GPT-5.6 Luna selections migrate to GPT-6 Luna. Check only the active project's `.env` for `OPENAI_API_KEY`; do not use one found elsewhere. If it is missing, guide the user to create and configure their own key in the same project using the same secret-safe process. Do not silently switch providers to bypass missing Loadmill authentication.

package/skill/droid-cua/references/running-and-healing.md

@@ -16,10 +16,16 @@ Preserve an existing provider/model choice in `ci/droid-cua.json`; otherwise use
 --llm-provider loadmill --cua-model loadmill-smart
 ```
 
-All Loadmill models require `LOADMILL_API_TOKEN`. Do not switch from Pulse to Smart merely because a test failed or was slow; first identify a model-related stability issue, then keep Smart only when comparison evidence shows an improvement. Do not switch between Smart, Pulse, and Beacon silently; preserve the project's model or explain the capability, speed, and experimental tradeoffs and follow the user's choice. If the user chooses direct OpenAI access instead, require `OPENAI_API_KEY` and use `--llm-provider openai` with one of `gpt-5.6-terra`, `gpt-5.6-luna`, or `gpt-5.4`. Do not mix a provider with another provider's model, silently switch providers, or override an existing project choice without explaining it.
+All Loadmill models require `LOADMILL_API_TOKEN`. Smart and Pulse now route through the Loadmill gateway to GPT-6 Sol and GPT-6 Luna. Do not switch from Pulse to Smart merely because a test failed or was slow; first identify a model-related stability issue, then keep Smart only when comparison evidence shows an improvement. Do not switch between Smart, Pulse, and Beacon silently; preserve the project's model or explain the capability, speed, and experimental tradeoffs and follow the user's choice. If the user chooses direct OpenAI access instead, require `OPENAI_API_KEY` and use `--llm-provider openai` with `gpt-6-sol` or `gpt-6-luna`. Do not mix a provider with another provider's model, silently switch providers, or override an existing project choice without explaining it.
 
 Include `--report reports/<test-name>.html` for normal test runs unless the user opts out. Prefer an explicit filename over a directory so the result is easy to locate. Add `--debug` for every agent-observed run: it writes the screenshots needed for live progress updates as well as diagnosis.
 
+For macOS and Windows desktop tests, the desktop app shows a system notice for three seconds before Droid takes control. The three-second delay is fixed; once the test starts, use Stop in Droid to end it. Avoid using the mouse or keyboard during the run. Droid brings the selected target window to the foreground before the first capture. When the run ends, the target stays in front and a second system notice says Droid has finished controlling the desktop. Open Droid to review the result. Some Windows apps return an all-black window capture; Droid brings the target back to the foreground before cropping the visible desktop to its visible frame, excluding the invisible Windows resize border. The fallback logs an all-black capture warning and crops before resizing to preserve detail. A visible-frame lookup warning means Droid is using the outer window bounds, which may include invisible resize borders. Keep the target visible and unobstructed on the primary display. In accessibility input mode, bring the target to the foreground yourself if the fallback requests it; Droid does not focus it for screenshots. If a screenshot shows another app or black off-screen edges, check whether the target could be focused and fully displayed before changing the `.dcua` instructions.
+
+When downstream automation needs an exact Loadmill handoff, add `--report-metadata reports/<test-name>.json`. Read it only after Droid exits. Droid clears the destination at run start and writes the saved run ID plus the last uploaded screenshot object name only after cloud report publication succeeds, including when a failed test's report was published. Do not infer the run or screenshot by searching recent Loadmill runs when this file was requested but absent.
+
+For desktop runs, treat the test result and dashboard report as separate outcomes. A run can appear in Droid Runs while its report is still uploading. A `report.upload.skipped` entry in the app's `desktop-events.jsonl` explains when sharing was disabled or authentication was unavailable. For failures, use the run warning and `report.upload.failed` entry, which includes the upload stage and HTTP status when available. The absence of a failure entry does not prove that a still-unavailable report was published; it may still be uploading. Direct the user to the local report in Droid when sharing did not complete. Do not infer the cause from an unclickable dashboard row alone.
+
 ```sh
 # Attached Android device or running emulator
 droid-cua run tests/example.dcua --llm-provider loadmill --cua-model loadmill-pulse --avd adb:<serial> --context tests/context.md --debug --report reports/example.html
@@ -40,7 +46,7 @@ droid-cua run tests/example.dcua --llm-provider loadmill --cua-model loadmill-pu
 droid-cua run tests/example.dcua --llm-provider loadmill --cua-model loadmill-pulse --device-source aws-device-farm --platform android --device-name "<name>" --os-version "<version>" --app <app.apk> --debug --report reports/example.html
 
 # Direct OpenAI alternative
-droid-cua run tests/example.dcua --llm-provider openai --cua-model gpt-5.4 --avd adb:<serial> --context tests/context.md --debug --report reports/example.html
+droid-cua run tests/example.dcua --llm-provider openai --cua-model gpt-6-sol --avd adb:<serial> --context tests/context.md --debug --report reports/example.html
 
 # Selected tests from different folders
 droid-cua run tests/auth/login.dcua tests/checkout/purchase.dcua admin/refund.dcua --avd adb:<serial> --config ci/droid-cua.json --debug --report reports/selected-tests.html
@@ -105,6 +111,7 @@ When the command completes, show the final safe screenshot in addition to the re
 - Exit `130`: interrupted run.
 - `--debug`: writes structured execution JSONL and adjacent screenshots under `logs/`.
 - `--report`: writes a self-contained HTML report. Prefer an explicit `.html` file path such as `reports/example.html` so the evidence location is unambiguous.
+- `--report-metadata`: writes the exact published Loadmill run and last screenshot handoff for downstream automation. Treat an absent file as no successful publication for the current invocation.
 
 After the run, verify that the report exists and resolve its absolute path. Open it for the user when the agent environment supports local-file opening and policy permits; otherwise provide a clickable local-file link or the absolute path so the user can open it directly.
 
@@ -130,6 +137,6 @@ Keep this retrospective concise and lead with the result and improvement. Do not
 
 ## Guarded healing
 
-Before editing, state the classification and evidence. Change only `.dcua`, `context.md`, `test-data.md`, or Droid CUA config. Never edit app code, delete coverage, loosen the expected outcome, or add broad waits simply to obtain green output.
+Before editing, state the classification and evidence. Change only `.dcua`, `context.md`, `test-data.md`, or Droid CUA config. Never edit app code, delete required coverage, loosen the expected outcome, or add broad waits simply to obtain green output. Use the coverage rules in [authoring-tests.md](authoring-tests.md) to distinguish required assertions from agent-added, out-of-scope or redundant checks; retain assertions whose purpose is unclear. A failing business outcome remains a failure.
 
-Apply the smallest evidence-supported batch, explain why each change corrects the test rather than hides a product defect, and rerun only the affected test. Continue analyzing, editing, and rerunning until the final representative evidence supports readiness or a genuine blocker remains. Do not stop based on a fixed rerun count. Stop when the evidence indicates an application regression, an environment or product dependency prevents progress, or no evidence-supported test, context, data, or configuration improvement remains; report the original and final evidence, changes attempted, and recommended owner/action.
+Simplify unclear goals and check setup/data before adding UI detail. Replace obsolete guidance rather than accumulating commands from each failed run, and review the whole test for readability after editing. Apply the smallest evidence-supported batch, explain why each change corrects the test rather than hides a product defect, and rerun only the affected test. Continue analyzing, editing, and rerunning until the final representative evidence supports readiness or a genuine blocker remains. Do not stop based on a fixed rerun count. Stop when the evidence indicates an application regression, an environment or product dependency prevents progress, or no evidence-supported test, context, data, or configuration improvement remains; report the original and final evidence, changes attempted, and recommended owner/action.