@validate.qa/runner 1.0.33 → 1.0.34
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli.js +261 -32
- package/dist/cli.mjs +261 -32
- package/package.json +1 -1
package/dist/cli.js
CHANGED
|
@@ -65163,7 +65163,7 @@ Return ONLY one valid JSON object matching the schema below. Do NOT include <thi
|
|
|
65163
65163
|
Example - BAD happy path: \`["Navigate to /feature", "Click Create", "Verify dialog opens"]\` \u2192 only tests the trigger.
|
|
65164
65164
|
Example - GOOD happy path (generic, async): \`["Navigate to landing and log in", "Navigate to the feature page", "Note the current count of items on the list", "Click the primary action button", "Fill required inputs and submit", "Wait for completion signal (up to 60s)", "Verify the item count increased by 1", "Open the newly created item", "Verify its content is non-empty and contains the input I provided"]\`
|
|
65165
65165
|
Example - GOOD happy path (sync CRUD): \`["Navigate to landing and log in", "Navigate to the list page", "Click New", "Fill required fields", "Click Save", "Verify the new record appears in the list", "Click the record and verify its detail view shows the values I entered"]\`
|
|
65166
|
-
12. **Auth happy paths require concrete session proof.** A login/auth HAPPY_PATH is incomplete if it only proves "user reaches dashboard" or "authenticated page is visible." It MUST verify at least one durable signed-in signal that was observed in the evidence: user/account menu opens, Logout/Sign out is available, the observed display name/email/avatar/account item is present, authenticated navigation remains available after a reload, or another app-specific session control is usable. Prefer concrete observed labels from the evidence (for example "Logout", "
|
|
65166
|
+
12. **Auth happy paths require concrete session proof.** A login/auth HAPPY_PATH is incomplete if it only proves "user reaches dashboard" or "authenticated page is visible." It MUST verify at least one durable signed-in signal that was observed in the evidence: user/account menu opens, Logout/Sign out is available, the observed display name/email/avatar/account item is present, authenticated navigation remains available after a reload, or another app-specific session control is usable. Prefer concrete observed labels from the evidence (for example "Logout", "Sign out", "Account", "Profile", or the user's own displayed name/email from evidence) over generic phrases like "personalized content." If the explorer only observed a redirect and no signed-in signal, keep the route outcome but mark the session-proof outcome reliability "low" and state what concrete signal was missing.
|
|
65167
65167
|
13. **Independent execution contract.** Every scenario MUST pass when run alone, first, last, in random order, and in parallel against an environment with no relevant feature data except explicitly guaranteed immutable fixtures. A scenario that depends on a sibling scenario's side effects (e.g. "delete the conversation that another test created") is invalid. If two flows are inherently coupled, bundle them as a single USER_JOURNEY scenario.
|
|
65168
65168
|
14. **Scenario-owned state.** If a scenario reads, searches, opens, updates, messages, cancels, or deletes an entity, that scenario MUST create or seed the entity in its own setup steps. "Use an existing record" / "the conversation in the demo account" / "the existing user named X" are forbidden phrasings \u2014 they assume mutable pre-existing app data. Carve-out: baseline observations (current count, list length, "the dashboard heading is visible") are allowed as prerequisite reads ONLY when the durable assertion targets a self-created entity. Phrase setup as: "Click New, fill required fields with run-scoped values, submit and verify the new record appears, THEN proceed to <feature under test>."
|
|
65169
65169
|
15. **Run-scoped test data.** Generated entities MUST use unique, run-scoped identifiers \u2014 phrase as "test message <Date.now()>" or "QA-<unique id>" in steps. Subsequent assertions MUST target that unique identifier, not "the first item", "the latest message", "the last row", or any visible display name observed during exploration. Hardcoding any specific display name, username, email, conversation title, or other identifier observed in the explorer's snapshot is forbidden \u2014 that data is incidental, not a stable contract. The same identifiers will not exist (or will look different) when the test runs in a fresh environment or against a different account.
|
|
@@ -66123,7 +66123,7 @@ Return ONLY one valid JSON object matching the schema below. Do NOT include <thi
|
|
|
66123
66123
|
9. **No duplicates** - check existing test names before adding
|
|
66124
66124
|
10. **All tests start from the app launch entry** - every generated test starts by launching the target app. The runner clears app data when the device/reset policy safely supports it; otherwise it performs a terminate + activate relaunch, which can preserve app session data. Therefore scenarios MUST navigate through the app's own UI from whatever launch state is valid, and MUST NOT rely on mutable data from a prior scenario disappearing. Steps must reflect this: say "Launch the app and tap Sign In" not "deep-link to the login screen". Do NOT navigate by deep-link; the app's launch + tab/drawer navigation is the only entry path. \`targetPages\` lists the screen-ids involved in the flow.
|
|
66125
66125
|
11. **targetPages must be copied CHARACTER-FOR-CHARACTER from the screen-ids in the evidence** (they look like \`com.example.app.MainActivity#ab12cd34\`). NEVER invent, prettify, or abbreviate them \u2014 no "Landing", "AUTH", "Home Screen", or any name that is not literally in the provided screens. A scenario whose targetPages contains even ONE id not present in the evidence is REJECTED WHOLESALE by the validator, so a single made-up route silently destroys the entire scenario. If you are unsure which screen-id a step touches, use the screen-id of the screen where the flow STARTS.
|
|
66126
|
-
12. **Happy paths must verify the DURABLE OUTCOME, not just the trigger.** A HAPPY_PATH is NOT complete until it verifies the feature's real output. "Button tapped" / "Sheet opened" / "Spinner appeared" is NOT a happy path - that is a UI interaction test. Apply this to any feature where an action produces a result the user then sees or uses: CRUD creates, photo/file uploads, searches that return results, imports, exports, payments, messages sent, AI/ML outputs, generated documents, sync runs, analysis runs, conversion, data processing. The scenario MUST include:
|
|
66126
|
+
12. **Happy paths must verify the DURABLE OUTCOME, not just the trigger.** A HAPPY_PATH is NOT complete until it verifies the feature's real output. "Button tapped" / "Sheet opened" / "Spinner appeared" / "Settings screen opened" is NOT a happy path - that is a UI interaction test. Form-mutation happy paths (change password, create/edit/save a record, send a message, checkout, etc.) MUST include typing into the relevant fields AND committing the form (submit/save/send or keyboard Enter/Done) \u2014 navigating to the form screen alone is insufficient. Apply this to any feature where an action produces a result the user then sees or uses: CRUD creates, photo/file uploads, searches that return results, imports, exports, payments, messages sent, AI/ML outputs, generated documents, sync runs, analysis runs, conversion, data processing. The scenario MUST include:
|
|
66127
66127
|
- The trigger step (Tap Submit / Generate / Send / Upload / Run / Analyze / Save / Pay / etc.)
|
|
66128
66128
|
- A wait step for the operation to complete. Phrase it by SIGNAL, not wall-clock ("Wait for the result to appear - up to 60 seconds"). Prefer signals the app actually uses: spinner disappears, status badge flips to a success state, a new row appears, an item count increments, a success toast/snackbar shows, the screen transitions to a result screen.
|
|
66129
66129
|
- A verification step on the DURABLE RESULT - the thing that outlives the transient UI: a new list row, a persisted record, a received message, a committed state change, visible output text. Do NOT assert on the spinner disappearing alone; assert on what replaced it.
|
|
@@ -66794,6 +66794,9 @@ var require_mobile_generator = __commonJS({
|
|
|
66794
66794
|
"../runner-core/dist/services/mobile/mobile-generator.js"(exports2) {
|
|
66795
66795
|
"use strict";
|
|
66796
66796
|
Object.defineProperty(exports2, "__esModule", { value: true });
|
|
66797
|
+
exports2.collapseConsecutiveBacks = collapseConsecutiveBacks;
|
|
66798
|
+
exports2.scenarioIntentMismatchReason = scenarioIntentMismatchReason;
|
|
66799
|
+
exports2.dedupeConsecutiveIdenticalActions = dedupeConsecutiveIdenticalActions;
|
|
66797
66800
|
exports2.collapseRetryCycles = collapseRetryCycles;
|
|
66798
66801
|
exports2.runMobileGeneratePhase = runMobileGeneratePhase;
|
|
66799
66802
|
var crypto_1 = require("crypto");
|
|
@@ -67064,7 +67067,7 @@ var require_mobile_generator = __commonJS({
|
|
|
67064
67067
|
## Your job
|
|
67065
67068
|
1. mobile_snapshot the current screen.
|
|
67066
67069
|
2. Execute the scenario's steps using the real UI: tap buttons, fill fields (mobile_type), submit, navigate.
|
|
67067
|
-
3. COMPLETE THE ACTION the scenario is about \u2014 do not stop half-way. If the scenario submits a form (sign in, sign up, search, create), you MUST tap the submit/primary button
|
|
67070
|
+
3. COMPLETE THE ACTION the scenario is about \u2014 do not stop half-way. If the scenario submits a form (sign in, sign up, search, create, change password, edit profile), you MUST commit it \u2014 tap the submit/primary button OR press the keyboard's Enter/Done action on the last field (mobile_type with a trailing newline on the focused field when no button is visible). Filling fields alone is NOT the behavior under test. Navigating to a form screen without changing its fields is also NOT enough for mutation scenarios (change/create/update/edit). If it navigates, you MUST land on the destination. A "happy path" that fills a form but never commits \u2014 or only opens the form without typing \u2014 is WRONG.
|
|
67068
67071
|
4. When you reach the target screen / complete the flow, PROVE the outcome with an assert tool:
|
|
67069
67072
|
- mobile_assert_visible({ref}) \u2014 the target screen's stable element is on screen.
|
|
67070
67073
|
- mobile_assert_text({ref, expected}) \u2014 an element contains expected text (only durable when the element has a stable accessibility-id / resource-id).
|
|
@@ -67079,6 +67082,8 @@ var require_mobile_generator = __commonJS({
|
|
|
67079
67082
|
- Stay inside this app.${targetLine} Do not open or use app stores, browsers, email apps, settings, or any other app. If an action leaves the target app, the runner will relaunch the target app and reject that action.
|
|
67080
67083
|
- NEVER tap the phone's home screen / launcher (app icons, "Predicted app" chips, search bars, widgets). Those are NOT part of the app under test \u2014 tapping them is junk that poisons the test. If the app ever backgrounds or you find yourself on the launcher, use mobile_relaunch to cold-start the app; do NOT tap a launcher icon or app-prediction to get back in.
|
|
67081
67084
|
- Do NOT re-do a sequence you already completed. If you filled a form then navigated away, do NOT come back and fill it again \u2014 the scenario needs ONE clean pass. Filling the login form, going back, and re-filling it is exactly the kind of wandering that produces a bloated, broken test.
|
|
67085
|
+
- Do NOT thrash the system Back button. One intentional back/navigation is fine; a chain of backs while lost is not a test \u2014 re-snapshot, re-orient, and take the productive path once.
|
|
67086
|
+
- Prefer product identity locators (accessibility-id, resource-id, visible label). Never assert on bare hierarchy roots or unlabeled full-screen containers \u2014 those prove nothing.
|
|
67082
67087
|
- Do ONLY the actions this scenario needs. Do NOT re-open a menu you already opened, re-select an option that is already selected, toggle a setting back and forth, or repeat a step that already succeeded \u2014 EVERY action you take is baked into the generated test verbatim, so each extra action becomes a permanent junk step.
|
|
67083
67088
|
- Each action must make NEW progress toward the goal. If an action leaves the screen unchanged, do NOT repeat it \u2014 pick a different element or move on.
|
|
67084
67089
|
- Finish promptly once the outcome is proven: the instant the expected outcome is visible, assert it and call finish_generate. Do not keep exploring after the goal is proven.
|
|
@@ -67097,7 +67102,7 @@ This app has stored login credentials under these keys:
|
|
|
67097
67102
|
${keyList}
|
|
67098
67103
|
|
|
67099
67104
|
When a scenario needs a signed-in session, complete the FULL login sequence \u2014 filling the fields alone does NOT log you in:
|
|
67100
|
-
1. Reach the email/password form. 2. Fill each credential field with \`mobile_fill_credential\` (pass the field ref and the KEY, e.g. \`{ ref: "e5", credentialKey: "TEST_USER_EMAIL" }\`) \u2014 the runner types the real value on the device and the generated test stores a \`{{KEY}}\` placeholder that replay resolves; you never see the secret. 3.
|
|
67105
|
+
1. Reach the email/password form. 2. Fill each credential field with \`mobile_fill_credential\` (pass the field ref and the KEY, e.g. \`{ ref: "e5", credentialKey: "TEST_USER_EMAIL" }\`) \u2014 the runner types the real value on the device and the generated test stores a \`{{KEY}}\` placeholder that replay resolves; you never see the secret. 3. COMMIT the form \u2014 tap the submit/primary button (Sign In / Log In / Continue) OR, if no button is visible, focus the last field and \`mobile_type\` a trailing newline (\`\\n\`) to trigger the keyboard's Enter/Done action. 4. Re-snapshot and VERIFY an authenticated screen appeared. If login fails twice, do NOT keep retrying or wander: call finish_generate with verified:false and say the login could not be completed (the credentials may be invalid for this app).
|
|
67101
67106
|
|
|
67102
67107
|
SECURITY RULES:
|
|
67103
67108
|
- NEVER call \`mobile_type\` with a real email, password, or token value \u2014 the runner will REJECT the call and you lose an iteration. Use \`mobile_fill_credential\` for every credential field.
|
|
@@ -67447,6 +67452,7 @@ ${statePacket}` },
|
|
|
67447
67452
|
if (action.type === "ASSERT_VISIBLE" || action.type === "ASSERT_TEXT")
|
|
67448
67453
|
assertCount++;
|
|
67449
67454
|
},
|
|
67455
|
+
recordedActions: actions,
|
|
67450
67456
|
log: log2
|
|
67451
67457
|
});
|
|
67452
67458
|
pushResult(dispatchResult);
|
|
@@ -67462,7 +67468,7 @@ ${statePacket}` },
|
|
|
67462
67468
|
return { actions, assertCount, finish, lastSnapshot: getLatest(), lastScreenName };
|
|
67463
67469
|
}
|
|
67464
67470
|
async function dispatchGenerateTool(args) {
|
|
67465
|
-
const { toolName, toolArgs, driver, platform: platform3, latest, absorbObservation, resolveRef, credentials, screenName, recordAction, log: log2 } = args;
|
|
67471
|
+
const { toolName, toolArgs, driver, platform: platform3, latest, absorbObservation, resolveRef, credentials, screenName, recordAction, recordedActions, log: log2 } = args;
|
|
67466
67472
|
const unknownRef = (ref) => ({
|
|
67467
67473
|
ok: false,
|
|
67468
67474
|
error: `Unknown element ref "${String(ref)}". Call mobile_snapshot first; refs are only valid for the latest snapshot.`
|
|
@@ -67527,14 +67533,23 @@ ${statePacket}` },
|
|
|
67527
67533
|
const screenChanged = changedScreen(before, absorbed);
|
|
67528
67534
|
const identityTap = isIdentityBearingTapTarget(element);
|
|
67529
67535
|
if (shouldRecordAction(result, absorbed) && (screenChanged || identityTap)) {
|
|
67530
|
-
|
|
67531
|
-
log2(` tap left content signature unchanged but target has identity \u2014 recording commit control "${labelForElement(element)}".`);
|
|
67532
|
-
}
|
|
67533
|
-
recordAction({
|
|
67536
|
+
const tapAction = {
|
|
67534
67537
|
type: "TAP",
|
|
67535
67538
|
selector: (0, mobile_source_parser_js_2.preferredLocator)(element),
|
|
67536
|
-
metadata: metadataForElement(element, beforeScreenName, platform3)
|
|
67537
|
-
|
|
67539
|
+
metadata: metadataForElement(element, beforeScreenName, platform3),
|
|
67540
|
+
innerText: null,
|
|
67541
|
+
url: "",
|
|
67542
|
+
value: null
|
|
67543
|
+
};
|
|
67544
|
+
const lastRecorded = recordedActions[recordedActions.length - 1];
|
|
67545
|
+
if (lastRecorded && actionCycleKey(lastRecorded) === actionCycleKey(tapAction)) {
|
|
67546
|
+
log2(` tap left content signature unchanged but duplicates the prior step \u2014 not recording again.`);
|
|
67547
|
+
} else if (!screenChanged && identityTap) {
|
|
67548
|
+
log2(` tap left content signature unchanged but target has identity \u2014 recording commit control "${labelForElement(element)}".`);
|
|
67549
|
+
recordAction(tapAction);
|
|
67550
|
+
} else {
|
|
67551
|
+
recordAction(tapAction);
|
|
67552
|
+
}
|
|
67538
67553
|
} else if (shouldRecordAction(result, absorbed)) {
|
|
67539
67554
|
log2(" tap changed nothing on screen \u2014 not recording it as a step (dead action).");
|
|
67540
67555
|
}
|
|
@@ -67878,6 +67893,105 @@ ${statePacket}` },
|
|
|
67878
67893
|
...scenario.expectedOutcomes.flatMap((outcome) => [outcome.description, outcome.hint])
|
|
67879
67894
|
].filter((value) => typeof value === "string" && value.trim().length > 0).join(" ").toLowerCase();
|
|
67880
67895
|
}
|
|
67896
|
+
var INSPECTION_OUTCOME_TYPES = /* @__PURE__ */ new Set([
|
|
67897
|
+
"element-visible",
|
|
67898
|
+
"text-visible",
|
|
67899
|
+
"element-hidden",
|
|
67900
|
+
"screen-visible"
|
|
67901
|
+
]);
|
|
67902
|
+
function isInspectionOnlyScenario(scenario, actions) {
|
|
67903
|
+
if (actions.length === 0 || scenario.expectedOutcomes.length === 0)
|
|
67904
|
+
return false;
|
|
67905
|
+
if (!scenario.expectedOutcomes.every((o) => INSPECTION_OUTCOME_TYPES.has(o.type))) {
|
|
67906
|
+
return false;
|
|
67907
|
+
}
|
|
67908
|
+
if (actions.some((a) => a.type === "TYPE" || a.type === "CLEAR")) {
|
|
67909
|
+
return false;
|
|
67910
|
+
}
|
|
67911
|
+
const text = scenarioText(scenario);
|
|
67912
|
+
const visibilityOnlyGoal = (/\b(entry\s+points?|providers?|options?|provider\s+buttons?)\b/.test(text) || /\b(show|display)\s+(all|the)?\s*(provider|option|button|link|entry)/.test(text) || /\b(all\s+(three\s+)?providers?)\b/.test(text)) && !/\b(change|update|create|save|send|register|edit|with\s+valid|successful\s+(sign|log)|complete\s+(the\s+)?(login|sign[- ]?in)|submit\s+(the\s+)?(form|credentials)|enter\s+(an?\s+)?(email|password))\b/.test(text);
|
|
67913
|
+
if (visibilityOnlyGoal) {
|
|
67914
|
+
const assertCount2 = actions.filter((a) => a.type === "ASSERT_VISIBLE" || a.type === "ASSERT_TEXT").length;
|
|
67915
|
+
return assertCount2 > 0;
|
|
67916
|
+
}
|
|
67917
|
+
const completionIntent = scenario.type === "HAPPY_PATH" || /\b(happy\s*path|successful|with\s+valid|complete\s+(the\s+)?(login|sign[- ]?in|registration|checkout|purchase))\b/.test(text);
|
|
67918
|
+
const formIntent = /\b(sign\s*in|log\s*in|login|authenticate|register|sign\s*up|checkout|create|save|submit|send|change|update|edit)\b/.test(text);
|
|
67919
|
+
if (completionIntent && formIntent)
|
|
67920
|
+
return false;
|
|
67921
|
+
const assertCount = actions.filter((a) => a.type === "ASSERT_VISIBLE" || a.type === "ASSERT_TEXT").length;
|
|
67922
|
+
return assertCount > 0;
|
|
67923
|
+
}
|
|
67924
|
+
function isNavigationFocusedScenario(scenario) {
|
|
67925
|
+
const text = scenarioText(scenario);
|
|
67926
|
+
const navigates = /\b(back|return|go\s+home|navigate\s+back|leave|exit|cancel\s+out|dismiss|previous\s+screen)\b/.test(text);
|
|
67927
|
+
const completesFlow = /\b(happy\s*path|successful|complete\s+(the\s+)?(login|sign[- ]?in|registration|checkout)|with\s+valid)\b/.test(text);
|
|
67928
|
+
return navigates && !completesFlow;
|
|
67929
|
+
}
|
|
67930
|
+
function isFormValidationScenario(scenario) {
|
|
67931
|
+
if (scenario.variant === "validation")
|
|
67932
|
+
return true;
|
|
67933
|
+
if (scenario.type !== "EDGE_CASE")
|
|
67934
|
+
return false;
|
|
67935
|
+
const text = scenarioText(scenario);
|
|
67936
|
+
return /\b(empty|required|missing|blank|invalid|reject|error|validation|wrong|incorrect|mismatch)\b/.test(text);
|
|
67937
|
+
}
|
|
67938
|
+
function isFormMutationHappyPath(scenario) {
|
|
67939
|
+
if (isNavigationFocusedScenario(scenario))
|
|
67940
|
+
return false;
|
|
67941
|
+
if (isFormValidationScenario(scenario))
|
|
67942
|
+
return false;
|
|
67943
|
+
const text = scenarioText(scenario);
|
|
67944
|
+
const isHappy = scenario.type === "HAPPY_PATH" || /\b(happy\s*path|successful|success)\b/.test(text);
|
|
67945
|
+
if (!isHappy)
|
|
67946
|
+
return false;
|
|
67947
|
+
if (/\b(sign\s*in|log\s*in|login|authenticate|sign\s*out|log\s*out)\b/.test(text) && !/\b(change|update|reset)\s+(the\s+)?password\b/.test(text)) {
|
|
67948
|
+
return false;
|
|
67949
|
+
}
|
|
67950
|
+
if (/\b(show|display|visible|options?|providers?|links?)\b/.test(text) && !/\b(change|update|create|save|send|register|edit|post|write|compose|delete|add|set)\b/.test(text)) {
|
|
67951
|
+
return false;
|
|
67952
|
+
}
|
|
67953
|
+
const mutates = /\b(change|update|create|save|send|register|sign\s*up|checkout|edit|post|write|compose|set|add|delete|remove)\b/.test(text);
|
|
67954
|
+
const hasObject = /\b(password|email|profile|message|post|item|record|form|settings|account|name|title|description|comment|bio|phone|address|payment|card|user|note|task|entry)\b/.test(text);
|
|
67955
|
+
return mutates && hasObject;
|
|
67956
|
+
}
|
|
67957
|
+
function isNavigationChromeTap(action) {
|
|
67958
|
+
if (action.type !== "TAP")
|
|
67959
|
+
return false;
|
|
67960
|
+
const textForAction = actionText(action);
|
|
67961
|
+
return /\b(go\s+back|back\s+to|←|return\s+to|cancel|close|dismiss|not\s+now|skip|maybe\s+later)\b/.test(textForAction);
|
|
67962
|
+
}
|
|
67963
|
+
function isAuthMethodChooserTap(action) {
|
|
67964
|
+
if (action.type !== "TAP")
|
|
67965
|
+
return false;
|
|
67966
|
+
const textForAction = actionText(action);
|
|
67967
|
+
if (/\b(submit|confirm|sign\s*in|log\s*in|login)\b/.test(textForAction) && !/\b(with|using|via)\b/.test(textForAction)) {
|
|
67968
|
+
return false;
|
|
67969
|
+
}
|
|
67970
|
+
return /\b(continue\s+with|sign\s+in\s+with|log\s+in\s+with|use\s+(email|phone)|choose|pick|select)\b/.test(textForAction);
|
|
67971
|
+
}
|
|
67972
|
+
function isCredentialSubmitTap(action) {
|
|
67973
|
+
if (action.type !== "TAP")
|
|
67974
|
+
return false;
|
|
67975
|
+
if (isNavigationChromeTap(action) || isAuthMethodChooserTap(action))
|
|
67976
|
+
return false;
|
|
67977
|
+
const textForAction = actionText(action);
|
|
67978
|
+
return /\b(sign\s*in|log\s*in|login|continue|submit|done|next|confirm)\b/.test(textForAction);
|
|
67979
|
+
}
|
|
67980
|
+
function isKeyboardFormCommit(action) {
|
|
67981
|
+
return action.type === "TYPE" && (action.value ?? "").endsWith("\n");
|
|
67982
|
+
}
|
|
67983
|
+
function isFormCommitAction(action) {
|
|
67984
|
+
if (isKeyboardFormCommit(action))
|
|
67985
|
+
return true;
|
|
67986
|
+
if (isCredentialSubmitTap(action))
|
|
67987
|
+
return true;
|
|
67988
|
+
if (action.type !== "TAP")
|
|
67989
|
+
return false;
|
|
67990
|
+
if (isNavigationChromeTap(action) || isAuthMethodChooserTap(action))
|
|
67991
|
+
return false;
|
|
67992
|
+
const textForAction = actionText(action);
|
|
67993
|
+
return /\b(save|create|send|confirm|next|apply|post|checkout|pay|finish|update|register|sign\s*up|done|continue|submit)\b/i.test(textForAction);
|
|
67994
|
+
}
|
|
67881
67995
|
function actionText(action) {
|
|
67882
67996
|
const bundle = action.metadata.locatorBundle;
|
|
67883
67997
|
return [
|
|
@@ -67894,9 +68008,80 @@ ${statePacket}` },
|
|
|
67894
68008
|
bundle?.xpath
|
|
67895
68009
|
].filter((value) => typeof value === "string" && value.trim().length > 0).join(" ").toLowerCase();
|
|
67896
68010
|
}
|
|
68011
|
+
function isStructuralOnlyAssert(action) {
|
|
68012
|
+
if (action.type !== "ASSERT_VISIBLE" && action.type !== "ASSERT_TEXT")
|
|
68013
|
+
return false;
|
|
68014
|
+
const sel = (action.selector ?? "").trim();
|
|
68015
|
+
if (/^\/hierarchy(\[\d+\])?(\/android\.widget\.FrameLayout\[\d+\])*$/i.test(sel))
|
|
68016
|
+
return true;
|
|
68017
|
+
if (isStructuralAssertionIdentity(sel) || isStructuralAssertionIdentity(action.metadata.targetLabel)) {
|
|
68018
|
+
const bundle = action.metadata.locatorBundle;
|
|
68019
|
+
const human = Boolean(nonEmpty2(action.innerText) || nonEmpty2(action.metadata.targetLabel) || nonEmpty2(bundle?.text) || nonEmpty2(bundle?.contentDesc));
|
|
68020
|
+
if (!human)
|
|
68021
|
+
return true;
|
|
68022
|
+
}
|
|
68023
|
+
if (/^\/hierarchy/i.test(sel)) {
|
|
68024
|
+
const bundle = action.metadata.locatorBundle;
|
|
68025
|
+
const hasIdentity = Boolean(nonEmpty2(action.metadata.targetLabel) || nonEmpty2(action.innerText) || nonEmpty2(bundle?.text) || nonEmpty2(bundle?.contentDesc) || nonEmpty2(bundle?.accessibilityId) || nonEmpty2(bundle?.resourceId) && !isStructuralAssertionIdentity(bundle?.resourceId));
|
|
68026
|
+
if (!hasIdentity)
|
|
68027
|
+
return true;
|
|
68028
|
+
}
|
|
68029
|
+
return false;
|
|
68030
|
+
}
|
|
68031
|
+
function collapseConsecutiveBacks(actions) {
|
|
68032
|
+
if (actions.length <= 1)
|
|
68033
|
+
return actions;
|
|
68034
|
+
const out = [actions[0]];
|
|
68035
|
+
for (let i = 1; i < actions.length; i++) {
|
|
68036
|
+
if (actions[i].type === "BACK" && out[out.length - 1].type === "BACK")
|
|
68037
|
+
continue;
|
|
68038
|
+
out.push(actions[i]);
|
|
68039
|
+
}
|
|
68040
|
+
return out;
|
|
68041
|
+
}
|
|
67897
68042
|
function scenarioIntentMismatchReason(scenario, actions) {
|
|
67898
68043
|
const text = scenarioText(scenario);
|
|
67899
|
-
|
|
68044
|
+
if (isInspectionOnlyScenario(scenario, actions)) {
|
|
68045
|
+
return null;
|
|
68046
|
+
}
|
|
68047
|
+
if (isNavigationFocusedScenario(scenario)) {
|
|
68048
|
+
return null;
|
|
68049
|
+
}
|
|
68050
|
+
if (isFormValidationScenario(scenario)) {
|
|
68051
|
+
const typedFields = actions.some((a) => a.type === "TYPE");
|
|
68052
|
+
const hasCommit = actions.some((a) => isFormCommitAction(a));
|
|
68053
|
+
const hasValidationAssert = actions.some((a) => {
|
|
68054
|
+
if (a.type !== "ASSERT_VISIBLE" && a.type !== "ASSERT_TEXT")
|
|
68055
|
+
return false;
|
|
68056
|
+
return /\b(error|required|invalid|empty|missing|incorrect|wrong|try again|mismatch|does not match|must match)\b/.test(actionText(a));
|
|
68057
|
+
});
|
|
68058
|
+
const emptyFieldIntent = /\b(empty|required|missing|blank)\b/.test(text);
|
|
68059
|
+
const invalidValueIntent = /\b(wrong|incorrect|invalid|mismatch|reject)\b/.test(text);
|
|
68060
|
+
if (emptyFieldIntent && !invalidValueIntent) {
|
|
68061
|
+
if (!hasCommit && !hasValidationAssert) {
|
|
68062
|
+
return "validation scenario did not record a form commit or validation outcome";
|
|
68063
|
+
}
|
|
68064
|
+
return null;
|
|
68065
|
+
}
|
|
68066
|
+
if (typedFields && !hasCommit) {
|
|
68067
|
+
return "validation scenario filled fields but never committed the form";
|
|
68068
|
+
}
|
|
68069
|
+
if (!hasCommit && !hasValidationAssert) {
|
|
68070
|
+
return "validation scenario did not record a form commit or validation outcome";
|
|
68071
|
+
}
|
|
68072
|
+
return null;
|
|
68073
|
+
}
|
|
68074
|
+
if (isFormMutationHappyPath(scenario)) {
|
|
68075
|
+
const hasType = actions.some((a) => a.type === "TYPE");
|
|
68076
|
+
const hasCommit = actions.some((a) => isFormCommitAction(a));
|
|
68077
|
+
if (!hasType) {
|
|
68078
|
+
return "form-mutation happy path did not record any field entry";
|
|
68079
|
+
}
|
|
68080
|
+
if (!hasCommit) {
|
|
68081
|
+
return "form-mutation happy path did not record a form commit";
|
|
68082
|
+
}
|
|
68083
|
+
}
|
|
68084
|
+
const isCredentialSignIn = /\b(sign\s*in|log\s*in|login|authenticate|auth)\b/.test(text) && /\b(email|e-mail|password|credential|valid credentials?)\b/.test(text) && !/\b(reset|forgot|recover|logout|log\s*out|sign\s*out|change\s+password|update\s+password)\b/.test(text);
|
|
67900
68085
|
const typedEmailIndexes = [];
|
|
67901
68086
|
const typedPasswordIndexes = [];
|
|
67902
68087
|
actions.forEach((action, index) => {
|
|
@@ -67913,19 +68098,19 @@ ${statePacket}` },
|
|
|
67913
68098
|
const lastCredentialIndex = Math.max(...typedEmailIndexes, ...typedPasswordIndexes);
|
|
67914
68099
|
const hasAnyCredentialEntry = Number.isFinite(lastCredentialIndex);
|
|
67915
68100
|
const submitAfterCredentials = actions.some((action, index) => {
|
|
67916
|
-
if (
|
|
68101
|
+
if (index < lastCredentialIndex)
|
|
67917
68102
|
return false;
|
|
67918
|
-
|
|
67919
|
-
if (!/\b(sign\s*in|log\s*in|login|continue|submit)\b/.test(textForAction))
|
|
68103
|
+
if (!isFormCommitAction(action))
|
|
67920
68104
|
return false;
|
|
67921
|
-
|
|
68105
|
+
if (action.type === "TYPE" && index === lastCredentialIndex)
|
|
68106
|
+
return true;
|
|
68107
|
+
return index > lastCredentialIndex;
|
|
67922
68108
|
});
|
|
67923
68109
|
const isAuthLikeScenario = /\b(sign(?:ed)?\s*in|log(?:ged)?\s*in|login|authenticate|auth|credential|email|e-mail|password)\b/.test(text);
|
|
67924
68110
|
if (isAuthLikeScenario && hasAnyCredentialEntry && !submitAfterCredentials) {
|
|
67925
|
-
return "credential fields were filled but no
|
|
68111
|
+
return "credential fields were filled but no form commit was recorded after them";
|
|
67926
68112
|
}
|
|
67927
68113
|
const uiOnlyCredentialScenario = /\b(mask|masked|visibility|show password|hide password|toggle|keyboard|keypad|max(?:imum)?\s*length|field focus|autofill)\b/.test(text);
|
|
67928
|
-
const emptyFieldValidationScenario = /\b(empty|required|missing|blank)\b/.test(text);
|
|
67929
68114
|
const sessionLifecycleScenario = /\b(logout|log\s*out|sign\s*out|session persistence|remain(?:s)? signed in|remember me)\b/.test(text);
|
|
67930
68115
|
const loginTapWithoutCredentialPath = actions.some((action) => {
|
|
67931
68116
|
if (action.type !== "TAP")
|
|
@@ -67933,9 +68118,11 @@ ${statePacket}` },
|
|
|
67933
68118
|
const textForAction = actionText(action);
|
|
67934
68119
|
if (/\b(logout|log\s*out|sign\s*out)\b/.test(textForAction))
|
|
67935
68120
|
return false;
|
|
68121
|
+
if (isAuthMethodChooserTap(action) || isNavigationChromeTap(action))
|
|
68122
|
+
return false;
|
|
67936
68123
|
return /\b(sign\s*in|log\s*in|login)\b/.test(textForAction);
|
|
67937
68124
|
});
|
|
67938
|
-
if (loginTapWithoutCredentialPath && !hasAnyCredentialEntry && !uiOnlyCredentialScenario &&
|
|
68125
|
+
if (loginTapWithoutCredentialPath && !hasAnyCredentialEntry && !uiOnlyCredentialScenario && (sessionLifecycleScenario || /\b(valid credentials?|happy path|successful sign\s*in|successful login)\b/.test(text))) {
|
|
67939
68126
|
return "auth/session scenario tapped a login or sign-in control without recording a credential path";
|
|
67940
68127
|
}
|
|
67941
68128
|
if (isCredentialSignIn) {
|
|
@@ -67947,7 +68134,7 @@ ${statePacket}` },
|
|
|
67947
68134
|
const missing = [
|
|
67948
68135
|
typedEmail ? null : "email/username entry",
|
|
67949
68136
|
typedPassword ? null : "password entry",
|
|
67950
|
-
tappedSubmit ? null : "
|
|
68137
|
+
tappedSubmit ? null : "form commit (submit tap or keyboard Enter)"
|
|
67951
68138
|
].filter((value) => value !== null);
|
|
67952
68139
|
return `credential sign-in scenario did not record ${missing.join(", ")}`;
|
|
67953
68140
|
}
|
|
@@ -67968,11 +68155,12 @@ ${statePacket}` },
|
|
|
67968
68155
|
const typeScreen = actions[lastTypeIndex]?.metadata?.screenName?.trim() ?? "";
|
|
67969
68156
|
const assertScreen = actions[lastAssertIndex]?.metadata?.screenName?.trim() ?? "";
|
|
67970
68157
|
const screensDiffer = typeScreen !== "" && assertScreen !== "" && typeScreen !== assertScreen;
|
|
67971
|
-
const
|
|
67972
|
-
|
|
68158
|
+
const lastTypeAction = actions[lastTypeIndex];
|
|
68159
|
+
const keyboardCommitOnLastType = lastTypeAction != null && isKeyboardFormCommit(lastTypeAction);
|
|
68160
|
+
const commitAfterType = keyboardCommitOnLastType || actions.some((action, index) => {
|
|
68161
|
+
if (index <= lastTypeIndex || index >= lastAssertIndex)
|
|
67973
68162
|
return false;
|
|
67974
|
-
|
|
67975
|
-
return /\b(sign\s*in|log\s*in|login|continue|submit|save|create|send|confirm|done|next|apply|post|checkout|pay|finish|update|register|sign\s*up)\b/i.test(textForAction);
|
|
68163
|
+
return isFormCommitAction(action);
|
|
67976
68164
|
});
|
|
67977
68165
|
if (screensDiffer && !commitAfterType) {
|
|
67978
68166
|
return "form fields were filled then a success assert was recorded without a commit/submit tap between them";
|
|
@@ -67983,6 +68171,17 @@ ${statePacket}` },
|
|
|
67983
68171
|
return `${action.type}|${action.selector}|${action.value ?? ""}`;
|
|
67984
68172
|
}
|
|
67985
68173
|
var MAX_RETRY_CYCLE_LENGTH = 8;
|
|
68174
|
+
function dedupeConsecutiveIdenticalActions(actions) {
|
|
68175
|
+
if (actions.length <= 1)
|
|
68176
|
+
return actions;
|
|
68177
|
+
const out = [actions[0]];
|
|
68178
|
+
for (let i = 1; i < actions.length; i++) {
|
|
68179
|
+
if (actionCycleKey(actions[i]) === actionCycleKey(out[out.length - 1]))
|
|
68180
|
+
continue;
|
|
68181
|
+
out.push(actions[i]);
|
|
68182
|
+
}
|
|
68183
|
+
return out;
|
|
68184
|
+
}
|
|
67986
68185
|
function collapseRetryCycles(actions) {
|
|
67987
68186
|
const out = [...actions];
|
|
67988
68187
|
let changed = true;
|
|
@@ -68023,15 +68222,30 @@ ${statePacket}` },
|
|
|
68023
68222
|
}
|
|
68024
68223
|
function assembleTest(args) {
|
|
68025
68224
|
const { scenario, recording, ctx, platform: platform3, appDataCleared, log: log2 } = args;
|
|
68026
|
-
let actions =
|
|
68225
|
+
let actions = dedupeConsecutiveIdenticalActions(recording.actions);
|
|
68027
68226
|
if (actions.length < recording.actions.length) {
|
|
68028
|
-
log2(`
|
|
68227
|
+
log2(` deduped ${recording.actions.length - actions.length} consecutive identical step(s) (${recording.actions.length} \u2192 ${actions.length}).`);
|
|
68228
|
+
}
|
|
68229
|
+
const beforeBackCollapse = actions.length;
|
|
68230
|
+
actions = collapseConsecutiveBacks(actions);
|
|
68231
|
+
if (actions.length < beforeBackCollapse) {
|
|
68232
|
+
log2(` collapsed ${beforeBackCollapse - actions.length} consecutive BACK step(s) (${beforeBackCollapse} \u2192 ${actions.length}).`);
|
|
68233
|
+
}
|
|
68234
|
+
const beforeCycleCollapse = actions.length;
|
|
68235
|
+
actions = collapseRetryCycles(actions);
|
|
68236
|
+
if (actions.length < beforeCycleCollapse) {
|
|
68237
|
+
log2(` collapsed ${beforeCycleCollapse - actions.length} retry-cycle step(s) out of the recorded trace (${beforeCycleCollapse} \u2192 ${actions.length}).`);
|
|
68029
68238
|
}
|
|
68030
68239
|
const beforeContainerDrop = actions.length;
|
|
68031
68240
|
actions = actions.filter((a) => !isBareContainerTap(a));
|
|
68032
68241
|
if (actions.length < beforeContainerDrop) {
|
|
68033
68242
|
log2(` dropped ${beforeContainerDrop - actions.length} bare-container tap step(s) (no identity \u2014 un-replayable ambient taps).`);
|
|
68034
68243
|
}
|
|
68244
|
+
const beforeWeakAssertDrop = actions.length;
|
|
68245
|
+
actions = actions.filter((a) => !isStructuralOnlyAssert(a));
|
|
68246
|
+
if (actions.length < beforeWeakAssertDrop) {
|
|
68247
|
+
log2(` dropped ${beforeWeakAssertDrop - actions.length} structural-only assert step(s) (no product identity).`);
|
|
68248
|
+
}
|
|
68035
68249
|
const intentMismatch = scenarioIntentMismatchReason(scenario, actions);
|
|
68036
68250
|
if (intentMismatch) {
|
|
68037
68251
|
log2(` skipped scenario "${scenario.name}": ${intentMismatch}.`);
|
|
@@ -68843,10 +69057,22 @@ For a fix, return the COMPLETE steps array (not a diff).`;
|
|
|
68843
69057
|
if (!/\b(tap|click|press)\b/i.test(step.action))
|
|
68844
69058
|
return false;
|
|
68845
69059
|
const text = mobileStepSearchText(step);
|
|
68846
|
-
if (/\b(logout|log\s*out|sign\s*out|reset|forgot|
|
|
69060
|
+
if (/\b(logout|log\s*out|sign\s*out|reset|forgot|go\s+back|back\s+to|cancel|close|dismiss)\b/.test(text)) {
|
|
68847
69061
|
return false;
|
|
68848
69062
|
}
|
|
68849
|
-
|
|
69063
|
+
if (/\b(continue\s+with|sign\s+in\s+with|log\s+in\s+with|use\s+(email|phone)|choose|pick|select)\b/.test(text) && !/\b(submit|confirm)\b/.test(text)) {
|
|
69064
|
+
return false;
|
|
69065
|
+
}
|
|
69066
|
+
return /\b(sign\s*in|signin|log\s*in|login|continue|submit|done|next|confirm)\b/.test(text);
|
|
69067
|
+
}
|
|
69068
|
+
function isKeyboardFormCommitStep(step) {
|
|
69069
|
+
if (!/\b(type|fill|input|enter)\b/i.test(step.action))
|
|
69070
|
+
return false;
|
|
69071
|
+
const value = typeof step.value === "string" ? step.value : "";
|
|
69072
|
+
return value.endsWith("\n");
|
|
69073
|
+
}
|
|
69074
|
+
function isFormCommitStep(step) {
|
|
69075
|
+
return isCredentialSubmitTap(step) || isKeyboardFormCommitStep(step);
|
|
68850
69076
|
}
|
|
68851
69077
|
function detectInvalidAuthValidationSetup(steps, error2) {
|
|
68852
69078
|
if (!/\b(element not found|no such element|not visible|timeout|assert|could not be located|unexpected screen|screen drift)\b/i.test(error2 ?? "")) {
|
|
@@ -68871,20 +69097,23 @@ For a fix, return the COMPLETE steps array (not a diff).`;
|
|
|
68871
69097
|
if ((step.order ?? -1) >= lastCredentialEditOrder)
|
|
68872
69098
|
return false;
|
|
68873
69099
|
const text = mobileStepSearchText(step);
|
|
68874
|
-
return /\b(
|
|
69100
|
+
return /\b(sign\s+in\s+with\s+email|log\s+in\s+with\s+email|email\s+and\s+password|login\s+form|credential\s+form|email|e-mail|username|user\s*name)\b/.test(text);
|
|
68875
69101
|
});
|
|
68876
69102
|
if (hasCredentialFormEvidence)
|
|
68877
69103
|
return null;
|
|
68878
69104
|
return "credential-control test edits a credential field without recording a path to the email/password form; stale auth-control test is invalid until generation records a complete form path";
|
|
68879
69105
|
}
|
|
68880
|
-
const hasSubmitAfterCredentialEdit = steps.some((step) => (step.order ?? -1) > lastCredentialEditOrder &&
|
|
69106
|
+
const hasSubmitAfterCredentialEdit = steps.some((step) => (step.order ?? -1) > lastCredentialEditOrder && isFormCommitStep(step));
|
|
68881
69107
|
if (hasSubmitAfterCredentialEdit)
|
|
68882
69108
|
return null;
|
|
69109
|
+
const hasKeyboardCommitOnCredential = steps.some((step) => (step.order ?? -1) === lastCredentialEditOrder && isKeyboardFormCommitStep(step));
|
|
69110
|
+
if (hasKeyboardCommitOnCredential)
|
|
69111
|
+
return null;
|
|
68883
69112
|
const hasValidationAssertion = steps.some((step) => {
|
|
68884
69113
|
if (!/\b(assert|wait)\b/i.test(step.action))
|
|
68885
69114
|
return false;
|
|
68886
69115
|
const text = mobileStepSearchText(step);
|
|
68887
|
-
return /\b(invalid|required|error|empty|missing|incorrect|wrong|try again|validation|must enter|enter (an )?(email|password)|email is required|password is required)\b/.test(text);
|
|
69116
|
+
return /\b(invalid|required|error|empty|missing|incorrect|wrong|try again|validation|must enter|enter (an )?(email|password)|email is required|password is required|mismatch|does not match)\b/.test(text);
|
|
68888
69117
|
});
|
|
68889
69118
|
if (hasValidationAssertion)
|
|
68890
69119
|
return null;
|
|
@@ -330644,7 +330873,7 @@ var require_package4 = __commonJS({
|
|
|
330644
330873
|
"package.json"(exports2, module2) {
|
|
330645
330874
|
module2.exports = {
|
|
330646
330875
|
name: "@validate.qa/runner",
|
|
330647
|
-
version: "1.0.
|
|
330876
|
+
version: "1.0.34",
|
|
330648
330877
|
description: "validate local test runner - execute Playwright tests on your machine, connected to validate.qa cloud.",
|
|
330649
330878
|
bin: {
|
|
330650
330879
|
"validate-runner": "dist/cli.js"
|
package/dist/cli.mjs
CHANGED
|
@@ -65168,7 +65168,7 @@ Return ONLY one valid JSON object matching the schema below. Do NOT include <thi
|
|
|
65168
65168
|
Example - BAD happy path: \`["Navigate to /feature", "Click Create", "Verify dialog opens"]\` \u2192 only tests the trigger.
|
|
65169
65169
|
Example - GOOD happy path (generic, async): \`["Navigate to landing and log in", "Navigate to the feature page", "Note the current count of items on the list", "Click the primary action button", "Fill required inputs and submit", "Wait for completion signal (up to 60s)", "Verify the item count increased by 1", "Open the newly created item", "Verify its content is non-empty and contains the input I provided"]\`
|
|
65170
65170
|
Example - GOOD happy path (sync CRUD): \`["Navigate to landing and log in", "Navigate to the list page", "Click New", "Fill required fields", "Click Save", "Verify the new record appears in the list", "Click the record and verify its detail view shows the values I entered"]\`
|
|
65171
|
-
12. **Auth happy paths require concrete session proof.** A login/auth HAPPY_PATH is incomplete if it only proves "user reaches dashboard" or "authenticated page is visible." It MUST verify at least one durable signed-in signal that was observed in the evidence: user/account menu opens, Logout/Sign out is available, the observed display name/email/avatar/account item is present, authenticated navigation remains available after a reload, or another app-specific session control is usable. Prefer concrete observed labels from the evidence (for example "Logout", "
|
|
65171
|
+
12. **Auth happy paths require concrete session proof.** A login/auth HAPPY_PATH is incomplete if it only proves "user reaches dashboard" or "authenticated page is visible." It MUST verify at least one durable signed-in signal that was observed in the evidence: user/account menu opens, Logout/Sign out is available, the observed display name/email/avatar/account item is present, authenticated navigation remains available after a reload, or another app-specific session control is usable. Prefer concrete observed labels from the evidence (for example "Logout", "Sign out", "Account", "Profile", or the user's own displayed name/email from evidence) over generic phrases like "personalized content." If the explorer only observed a redirect and no signed-in signal, keep the route outcome but mark the session-proof outcome reliability "low" and state what concrete signal was missing.
|
|
65172
65172
|
13. **Independent execution contract.** Every scenario MUST pass when run alone, first, last, in random order, and in parallel against an environment with no relevant feature data except explicitly guaranteed immutable fixtures. A scenario that depends on a sibling scenario's side effects (e.g. "delete the conversation that another test created") is invalid. If two flows are inherently coupled, bundle them as a single USER_JOURNEY scenario.
|
|
65173
65173
|
14. **Scenario-owned state.** If a scenario reads, searches, opens, updates, messages, cancels, or deletes an entity, that scenario MUST create or seed the entity in its own setup steps. "Use an existing record" / "the conversation in the demo account" / "the existing user named X" are forbidden phrasings \u2014 they assume mutable pre-existing app data. Carve-out: baseline observations (current count, list length, "the dashboard heading is visible") are allowed as prerequisite reads ONLY when the durable assertion targets a self-created entity. Phrase setup as: "Click New, fill required fields with run-scoped values, submit and verify the new record appears, THEN proceed to <feature under test>."
|
|
65174
65174
|
15. **Run-scoped test data.** Generated entities MUST use unique, run-scoped identifiers \u2014 phrase as "test message <Date.now()>" or "QA-<unique id>" in steps. Subsequent assertions MUST target that unique identifier, not "the first item", "the latest message", "the last row", or any visible display name observed during exploration. Hardcoding any specific display name, username, email, conversation title, or other identifier observed in the explorer's snapshot is forbidden \u2014 that data is incidental, not a stable contract. The same identifiers will not exist (or will look different) when the test runs in a fresh environment or against a different account.
|
|
@@ -66128,7 +66128,7 @@ Return ONLY one valid JSON object matching the schema below. Do NOT include <thi
|
|
|
66128
66128
|
9. **No duplicates** - check existing test names before adding
|
|
66129
66129
|
10. **All tests start from the app launch entry** - every generated test starts by launching the target app. The runner clears app data when the device/reset policy safely supports it; otherwise it performs a terminate + activate relaunch, which can preserve app session data. Therefore scenarios MUST navigate through the app's own UI from whatever launch state is valid, and MUST NOT rely on mutable data from a prior scenario disappearing. Steps must reflect this: say "Launch the app and tap Sign In" not "deep-link to the login screen". Do NOT navigate by deep-link; the app's launch + tab/drawer navigation is the only entry path. \`targetPages\` lists the screen-ids involved in the flow.
|
|
66130
66130
|
11. **targetPages must be copied CHARACTER-FOR-CHARACTER from the screen-ids in the evidence** (they look like \`com.example.app.MainActivity#ab12cd34\`). NEVER invent, prettify, or abbreviate them \u2014 no "Landing", "AUTH", "Home Screen", or any name that is not literally in the provided screens. A scenario whose targetPages contains even ONE id not present in the evidence is REJECTED WHOLESALE by the validator, so a single made-up route silently destroys the entire scenario. If you are unsure which screen-id a step touches, use the screen-id of the screen where the flow STARTS.
|
|
66131
|
-
12. **Happy paths must verify the DURABLE OUTCOME, not just the trigger.** A HAPPY_PATH is NOT complete until it verifies the feature's real output. "Button tapped" / "Sheet opened" / "Spinner appeared" is NOT a happy path - that is a UI interaction test. Apply this to any feature where an action produces a result the user then sees or uses: CRUD creates, photo/file uploads, searches that return results, imports, exports, payments, messages sent, AI/ML outputs, generated documents, sync runs, analysis runs, conversion, data processing. The scenario MUST include:
|
|
66131
|
+
12. **Happy paths must verify the DURABLE OUTCOME, not just the trigger.** A HAPPY_PATH is NOT complete until it verifies the feature's real output. "Button tapped" / "Sheet opened" / "Spinner appeared" / "Settings screen opened" is NOT a happy path - that is a UI interaction test. Form-mutation happy paths (change password, create/edit/save a record, send a message, checkout, etc.) MUST include typing into the relevant fields AND committing the form (submit/save/send or keyboard Enter/Done) \u2014 navigating to the form screen alone is insufficient. Apply this to any feature where an action produces a result the user then sees or uses: CRUD creates, photo/file uploads, searches that return results, imports, exports, payments, messages sent, AI/ML outputs, generated documents, sync runs, analysis runs, conversion, data processing. The scenario MUST include:
|
|
66132
66132
|
- The trigger step (Tap Submit / Generate / Send / Upload / Run / Analyze / Save / Pay / etc.)
|
|
66133
66133
|
- A wait step for the operation to complete. Phrase it by SIGNAL, not wall-clock ("Wait for the result to appear - up to 60 seconds"). Prefer signals the app actually uses: spinner disappears, status badge flips to a success state, a new row appears, an item count increments, a success toast/snackbar shows, the screen transitions to a result screen.
|
|
66134
66134
|
- A verification step on the DURABLE RESULT - the thing that outlives the transient UI: a new list row, a persisted record, a received message, a committed state change, visible output text. Do NOT assert on the spinner disappearing alone; assert on what replaced it.
|
|
@@ -66799,6 +66799,9 @@ var require_mobile_generator = __commonJS({
|
|
|
66799
66799
|
"../runner-core/dist/services/mobile/mobile-generator.js"(exports) {
|
|
66800
66800
|
"use strict";
|
|
66801
66801
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
66802
|
+
exports.collapseConsecutiveBacks = collapseConsecutiveBacks;
|
|
66803
|
+
exports.scenarioIntentMismatchReason = scenarioIntentMismatchReason;
|
|
66804
|
+
exports.dedupeConsecutiveIdenticalActions = dedupeConsecutiveIdenticalActions;
|
|
66802
66805
|
exports.collapseRetryCycles = collapseRetryCycles;
|
|
66803
66806
|
exports.runMobileGeneratePhase = runMobileGeneratePhase;
|
|
66804
66807
|
var crypto_1 = __require("crypto");
|
|
@@ -67069,7 +67072,7 @@ var require_mobile_generator = __commonJS({
|
|
|
67069
67072
|
## Your job
|
|
67070
67073
|
1. mobile_snapshot the current screen.
|
|
67071
67074
|
2. Execute the scenario's steps using the real UI: tap buttons, fill fields (mobile_type), submit, navigate.
|
|
67072
|
-
3. COMPLETE THE ACTION the scenario is about \u2014 do not stop half-way. If the scenario submits a form (sign in, sign up, search, create), you MUST tap the submit/primary button
|
|
67075
|
+
3. COMPLETE THE ACTION the scenario is about \u2014 do not stop half-way. If the scenario submits a form (sign in, sign up, search, create, change password, edit profile), you MUST commit it \u2014 tap the submit/primary button OR press the keyboard's Enter/Done action on the last field (mobile_type with a trailing newline on the focused field when no button is visible). Filling fields alone is NOT the behavior under test. Navigating to a form screen without changing its fields is also NOT enough for mutation scenarios (change/create/update/edit). If it navigates, you MUST land on the destination. A "happy path" that fills a form but never commits \u2014 or only opens the form without typing \u2014 is WRONG.
|
|
67073
67076
|
4. When you reach the target screen / complete the flow, PROVE the outcome with an assert tool:
|
|
67074
67077
|
- mobile_assert_visible({ref}) \u2014 the target screen's stable element is on screen.
|
|
67075
67078
|
- mobile_assert_text({ref, expected}) \u2014 an element contains expected text (only durable when the element has a stable accessibility-id / resource-id).
|
|
@@ -67084,6 +67087,8 @@ var require_mobile_generator = __commonJS({
|
|
|
67084
67087
|
- Stay inside this app.${targetLine} Do not open or use app stores, browsers, email apps, settings, or any other app. If an action leaves the target app, the runner will relaunch the target app and reject that action.
|
|
67085
67088
|
- NEVER tap the phone's home screen / launcher (app icons, "Predicted app" chips, search bars, widgets). Those are NOT part of the app under test \u2014 tapping them is junk that poisons the test. If the app ever backgrounds or you find yourself on the launcher, use mobile_relaunch to cold-start the app; do NOT tap a launcher icon or app-prediction to get back in.
|
|
67086
67089
|
- Do NOT re-do a sequence you already completed. If you filled a form then navigated away, do NOT come back and fill it again \u2014 the scenario needs ONE clean pass. Filling the login form, going back, and re-filling it is exactly the kind of wandering that produces a bloated, broken test.
|
|
67090
|
+
- Do NOT thrash the system Back button. One intentional back/navigation is fine; a chain of backs while lost is not a test \u2014 re-snapshot, re-orient, and take the productive path once.
|
|
67091
|
+
- Prefer product identity locators (accessibility-id, resource-id, visible label). Never assert on bare hierarchy roots or unlabeled full-screen containers \u2014 those prove nothing.
|
|
67087
67092
|
- Do ONLY the actions this scenario needs. Do NOT re-open a menu you already opened, re-select an option that is already selected, toggle a setting back and forth, or repeat a step that already succeeded \u2014 EVERY action you take is baked into the generated test verbatim, so each extra action becomes a permanent junk step.
|
|
67088
67093
|
- Each action must make NEW progress toward the goal. If an action leaves the screen unchanged, do NOT repeat it \u2014 pick a different element or move on.
|
|
67089
67094
|
- Finish promptly once the outcome is proven: the instant the expected outcome is visible, assert it and call finish_generate. Do not keep exploring after the goal is proven.
|
|
@@ -67102,7 +67107,7 @@ This app has stored login credentials under these keys:
|
|
|
67102
67107
|
${keyList}
|
|
67103
67108
|
|
|
67104
67109
|
When a scenario needs a signed-in session, complete the FULL login sequence \u2014 filling the fields alone does NOT log you in:
|
|
67105
|
-
1. Reach the email/password form. 2. Fill each credential field with \`mobile_fill_credential\` (pass the field ref and the KEY, e.g. \`{ ref: "e5", credentialKey: "TEST_USER_EMAIL" }\`) \u2014 the runner types the real value on the device and the generated test stores a \`{{KEY}}\` placeholder that replay resolves; you never see the secret. 3.
|
|
67110
|
+
1. Reach the email/password form. 2. Fill each credential field with \`mobile_fill_credential\` (pass the field ref and the KEY, e.g. \`{ ref: "e5", credentialKey: "TEST_USER_EMAIL" }\`) \u2014 the runner types the real value on the device and the generated test stores a \`{{KEY}}\` placeholder that replay resolves; you never see the secret. 3. COMMIT the form \u2014 tap the submit/primary button (Sign In / Log In / Continue) OR, if no button is visible, focus the last field and \`mobile_type\` a trailing newline (\`\\n\`) to trigger the keyboard's Enter/Done action. 4. Re-snapshot and VERIFY an authenticated screen appeared. If login fails twice, do NOT keep retrying or wander: call finish_generate with verified:false and say the login could not be completed (the credentials may be invalid for this app).
|
|
67106
67111
|
|
|
67107
67112
|
SECURITY RULES:
|
|
67108
67113
|
- NEVER call \`mobile_type\` with a real email, password, or token value \u2014 the runner will REJECT the call and you lose an iteration. Use \`mobile_fill_credential\` for every credential field.
|
|
@@ -67452,6 +67457,7 @@ ${statePacket}` },
|
|
|
67452
67457
|
if (action.type === "ASSERT_VISIBLE" || action.type === "ASSERT_TEXT")
|
|
67453
67458
|
assertCount++;
|
|
67454
67459
|
},
|
|
67460
|
+
recordedActions: actions,
|
|
67455
67461
|
log: log2
|
|
67456
67462
|
});
|
|
67457
67463
|
pushResult(dispatchResult);
|
|
@@ -67467,7 +67473,7 @@ ${statePacket}` },
|
|
|
67467
67473
|
return { actions, assertCount, finish, lastSnapshot: getLatest(), lastScreenName };
|
|
67468
67474
|
}
|
|
67469
67475
|
async function dispatchGenerateTool(args) {
|
|
67470
|
-
const { toolName, toolArgs, driver, platform: platform3, latest, absorbObservation, resolveRef, credentials, screenName, recordAction, log: log2 } = args;
|
|
67476
|
+
const { toolName, toolArgs, driver, platform: platform3, latest, absorbObservation, resolveRef, credentials, screenName, recordAction, recordedActions, log: log2 } = args;
|
|
67471
67477
|
const unknownRef = (ref) => ({
|
|
67472
67478
|
ok: false,
|
|
67473
67479
|
error: `Unknown element ref "${String(ref)}". Call mobile_snapshot first; refs are only valid for the latest snapshot.`
|
|
@@ -67532,14 +67538,23 @@ ${statePacket}` },
|
|
|
67532
67538
|
const screenChanged = changedScreen(before, absorbed);
|
|
67533
67539
|
const identityTap = isIdentityBearingTapTarget(element);
|
|
67534
67540
|
if (shouldRecordAction(result, absorbed) && (screenChanged || identityTap)) {
|
|
67535
|
-
|
|
67536
|
-
log2(` tap left content signature unchanged but target has identity \u2014 recording commit control "${labelForElement(element)}".`);
|
|
67537
|
-
}
|
|
67538
|
-
recordAction({
|
|
67541
|
+
const tapAction = {
|
|
67539
67542
|
type: "TAP",
|
|
67540
67543
|
selector: (0, mobile_source_parser_js_2.preferredLocator)(element),
|
|
67541
|
-
metadata: metadataForElement(element, beforeScreenName, platform3)
|
|
67542
|
-
|
|
67544
|
+
metadata: metadataForElement(element, beforeScreenName, platform3),
|
|
67545
|
+
innerText: null,
|
|
67546
|
+
url: "",
|
|
67547
|
+
value: null
|
|
67548
|
+
};
|
|
67549
|
+
const lastRecorded = recordedActions[recordedActions.length - 1];
|
|
67550
|
+
if (lastRecorded && actionCycleKey(lastRecorded) === actionCycleKey(tapAction)) {
|
|
67551
|
+
log2(` tap left content signature unchanged but duplicates the prior step \u2014 not recording again.`);
|
|
67552
|
+
} else if (!screenChanged && identityTap) {
|
|
67553
|
+
log2(` tap left content signature unchanged but target has identity \u2014 recording commit control "${labelForElement(element)}".`);
|
|
67554
|
+
recordAction(tapAction);
|
|
67555
|
+
} else {
|
|
67556
|
+
recordAction(tapAction);
|
|
67557
|
+
}
|
|
67543
67558
|
} else if (shouldRecordAction(result, absorbed)) {
|
|
67544
67559
|
log2(" tap changed nothing on screen \u2014 not recording it as a step (dead action).");
|
|
67545
67560
|
}
|
|
@@ -67883,6 +67898,105 @@ ${statePacket}` },
|
|
|
67883
67898
|
...scenario.expectedOutcomes.flatMap((outcome) => [outcome.description, outcome.hint])
|
|
67884
67899
|
].filter((value) => typeof value === "string" && value.trim().length > 0).join(" ").toLowerCase();
|
|
67885
67900
|
}
|
|
67901
|
+
var INSPECTION_OUTCOME_TYPES = /* @__PURE__ */ new Set([
|
|
67902
|
+
"element-visible",
|
|
67903
|
+
"text-visible",
|
|
67904
|
+
"element-hidden",
|
|
67905
|
+
"screen-visible"
|
|
67906
|
+
]);
|
|
67907
|
+
function isInspectionOnlyScenario(scenario, actions) {
|
|
67908
|
+
if (actions.length === 0 || scenario.expectedOutcomes.length === 0)
|
|
67909
|
+
return false;
|
|
67910
|
+
if (!scenario.expectedOutcomes.every((o) => INSPECTION_OUTCOME_TYPES.has(o.type))) {
|
|
67911
|
+
return false;
|
|
67912
|
+
}
|
|
67913
|
+
if (actions.some((a) => a.type === "TYPE" || a.type === "CLEAR")) {
|
|
67914
|
+
return false;
|
|
67915
|
+
}
|
|
67916
|
+
const text = scenarioText(scenario);
|
|
67917
|
+
const visibilityOnlyGoal = (/\b(entry\s+points?|providers?|options?|provider\s+buttons?)\b/.test(text) || /\b(show|display)\s+(all|the)?\s*(provider|option|button|link|entry)/.test(text) || /\b(all\s+(three\s+)?providers?)\b/.test(text)) && !/\b(change|update|create|save|send|register|edit|with\s+valid|successful\s+(sign|log)|complete\s+(the\s+)?(login|sign[- ]?in)|submit\s+(the\s+)?(form|credentials)|enter\s+(an?\s+)?(email|password))\b/.test(text);
|
|
67918
|
+
if (visibilityOnlyGoal) {
|
|
67919
|
+
const assertCount2 = actions.filter((a) => a.type === "ASSERT_VISIBLE" || a.type === "ASSERT_TEXT").length;
|
|
67920
|
+
return assertCount2 > 0;
|
|
67921
|
+
}
|
|
67922
|
+
const completionIntent = scenario.type === "HAPPY_PATH" || /\b(happy\s*path|successful|with\s+valid|complete\s+(the\s+)?(login|sign[- ]?in|registration|checkout|purchase))\b/.test(text);
|
|
67923
|
+
const formIntent = /\b(sign\s*in|log\s*in|login|authenticate|register|sign\s*up|checkout|create|save|submit|send|change|update|edit)\b/.test(text);
|
|
67924
|
+
if (completionIntent && formIntent)
|
|
67925
|
+
return false;
|
|
67926
|
+
const assertCount = actions.filter((a) => a.type === "ASSERT_VISIBLE" || a.type === "ASSERT_TEXT").length;
|
|
67927
|
+
return assertCount > 0;
|
|
67928
|
+
}
|
|
67929
|
+
function isNavigationFocusedScenario(scenario) {
|
|
67930
|
+
const text = scenarioText(scenario);
|
|
67931
|
+
const navigates = /\b(back|return|go\s+home|navigate\s+back|leave|exit|cancel\s+out|dismiss|previous\s+screen)\b/.test(text);
|
|
67932
|
+
const completesFlow = /\b(happy\s*path|successful|complete\s+(the\s+)?(login|sign[- ]?in|registration|checkout)|with\s+valid)\b/.test(text);
|
|
67933
|
+
return navigates && !completesFlow;
|
|
67934
|
+
}
|
|
67935
|
+
function isFormValidationScenario(scenario) {
|
|
67936
|
+
if (scenario.variant === "validation")
|
|
67937
|
+
return true;
|
|
67938
|
+
if (scenario.type !== "EDGE_CASE")
|
|
67939
|
+
return false;
|
|
67940
|
+
const text = scenarioText(scenario);
|
|
67941
|
+
return /\b(empty|required|missing|blank|invalid|reject|error|validation|wrong|incorrect|mismatch)\b/.test(text);
|
|
67942
|
+
}
|
|
67943
|
+
function isFormMutationHappyPath(scenario) {
|
|
67944
|
+
if (isNavigationFocusedScenario(scenario))
|
|
67945
|
+
return false;
|
|
67946
|
+
if (isFormValidationScenario(scenario))
|
|
67947
|
+
return false;
|
|
67948
|
+
const text = scenarioText(scenario);
|
|
67949
|
+
const isHappy = scenario.type === "HAPPY_PATH" || /\b(happy\s*path|successful|success)\b/.test(text);
|
|
67950
|
+
if (!isHappy)
|
|
67951
|
+
return false;
|
|
67952
|
+
if (/\b(sign\s*in|log\s*in|login|authenticate|sign\s*out|log\s*out)\b/.test(text) && !/\b(change|update|reset)\s+(the\s+)?password\b/.test(text)) {
|
|
67953
|
+
return false;
|
|
67954
|
+
}
|
|
67955
|
+
if (/\b(show|display|visible|options?|providers?|links?)\b/.test(text) && !/\b(change|update|create|save|send|register|edit|post|write|compose|delete|add|set)\b/.test(text)) {
|
|
67956
|
+
return false;
|
|
67957
|
+
}
|
|
67958
|
+
const mutates = /\b(change|update|create|save|send|register|sign\s*up|checkout|edit|post|write|compose|set|add|delete|remove)\b/.test(text);
|
|
67959
|
+
const hasObject = /\b(password|email|profile|message|post|item|record|form|settings|account|name|title|description|comment|bio|phone|address|payment|card|user|note|task|entry)\b/.test(text);
|
|
67960
|
+
return mutates && hasObject;
|
|
67961
|
+
}
|
|
67962
|
+
function isNavigationChromeTap(action) {
|
|
67963
|
+
if (action.type !== "TAP")
|
|
67964
|
+
return false;
|
|
67965
|
+
const textForAction = actionText(action);
|
|
67966
|
+
return /\b(go\s+back|back\s+to|←|return\s+to|cancel|close|dismiss|not\s+now|skip|maybe\s+later)\b/.test(textForAction);
|
|
67967
|
+
}
|
|
67968
|
+
function isAuthMethodChooserTap(action) {
|
|
67969
|
+
if (action.type !== "TAP")
|
|
67970
|
+
return false;
|
|
67971
|
+
const textForAction = actionText(action);
|
|
67972
|
+
if (/\b(submit|confirm|sign\s*in|log\s*in|login)\b/.test(textForAction) && !/\b(with|using|via)\b/.test(textForAction)) {
|
|
67973
|
+
return false;
|
|
67974
|
+
}
|
|
67975
|
+
return /\b(continue\s+with|sign\s+in\s+with|log\s+in\s+with|use\s+(email|phone)|choose|pick|select)\b/.test(textForAction);
|
|
67976
|
+
}
|
|
67977
|
+
function isCredentialSubmitTap(action) {
|
|
67978
|
+
if (action.type !== "TAP")
|
|
67979
|
+
return false;
|
|
67980
|
+
if (isNavigationChromeTap(action) || isAuthMethodChooserTap(action))
|
|
67981
|
+
return false;
|
|
67982
|
+
const textForAction = actionText(action);
|
|
67983
|
+
return /\b(sign\s*in|log\s*in|login|continue|submit|done|next|confirm)\b/.test(textForAction);
|
|
67984
|
+
}
|
|
67985
|
+
function isKeyboardFormCommit(action) {
|
|
67986
|
+
return action.type === "TYPE" && (action.value ?? "").endsWith("\n");
|
|
67987
|
+
}
|
|
67988
|
+
function isFormCommitAction(action) {
|
|
67989
|
+
if (isKeyboardFormCommit(action))
|
|
67990
|
+
return true;
|
|
67991
|
+
if (isCredentialSubmitTap(action))
|
|
67992
|
+
return true;
|
|
67993
|
+
if (action.type !== "TAP")
|
|
67994
|
+
return false;
|
|
67995
|
+
if (isNavigationChromeTap(action) || isAuthMethodChooserTap(action))
|
|
67996
|
+
return false;
|
|
67997
|
+
const textForAction = actionText(action);
|
|
67998
|
+
return /\b(save|create|send|confirm|next|apply|post|checkout|pay|finish|update|register|sign\s*up|done|continue|submit)\b/i.test(textForAction);
|
|
67999
|
+
}
|
|
67886
68000
|
function actionText(action) {
|
|
67887
68001
|
const bundle = action.metadata.locatorBundle;
|
|
67888
68002
|
return [
|
|
@@ -67899,9 +68013,80 @@ ${statePacket}` },
|
|
|
67899
68013
|
bundle?.xpath
|
|
67900
68014
|
].filter((value) => typeof value === "string" && value.trim().length > 0).join(" ").toLowerCase();
|
|
67901
68015
|
}
|
|
68016
|
+
function isStructuralOnlyAssert(action) {
|
|
68017
|
+
if (action.type !== "ASSERT_VISIBLE" && action.type !== "ASSERT_TEXT")
|
|
68018
|
+
return false;
|
|
68019
|
+
const sel = (action.selector ?? "").trim();
|
|
68020
|
+
if (/^\/hierarchy(\[\d+\])?(\/android\.widget\.FrameLayout\[\d+\])*$/i.test(sel))
|
|
68021
|
+
return true;
|
|
68022
|
+
if (isStructuralAssertionIdentity(sel) || isStructuralAssertionIdentity(action.metadata.targetLabel)) {
|
|
68023
|
+
const bundle = action.metadata.locatorBundle;
|
|
68024
|
+
const human = Boolean(nonEmpty2(action.innerText) || nonEmpty2(action.metadata.targetLabel) || nonEmpty2(bundle?.text) || nonEmpty2(bundle?.contentDesc));
|
|
68025
|
+
if (!human)
|
|
68026
|
+
return true;
|
|
68027
|
+
}
|
|
68028
|
+
if (/^\/hierarchy/i.test(sel)) {
|
|
68029
|
+
const bundle = action.metadata.locatorBundle;
|
|
68030
|
+
const hasIdentity = Boolean(nonEmpty2(action.metadata.targetLabel) || nonEmpty2(action.innerText) || nonEmpty2(bundle?.text) || nonEmpty2(bundle?.contentDesc) || nonEmpty2(bundle?.accessibilityId) || nonEmpty2(bundle?.resourceId) && !isStructuralAssertionIdentity(bundle?.resourceId));
|
|
68031
|
+
if (!hasIdentity)
|
|
68032
|
+
return true;
|
|
68033
|
+
}
|
|
68034
|
+
return false;
|
|
68035
|
+
}
|
|
68036
|
+
function collapseConsecutiveBacks(actions) {
|
|
68037
|
+
if (actions.length <= 1)
|
|
68038
|
+
return actions;
|
|
68039
|
+
const out = [actions[0]];
|
|
68040
|
+
for (let i = 1; i < actions.length; i++) {
|
|
68041
|
+
if (actions[i].type === "BACK" && out[out.length - 1].type === "BACK")
|
|
68042
|
+
continue;
|
|
68043
|
+
out.push(actions[i]);
|
|
68044
|
+
}
|
|
68045
|
+
return out;
|
|
68046
|
+
}
|
|
67902
68047
|
function scenarioIntentMismatchReason(scenario, actions) {
|
|
67903
68048
|
const text = scenarioText(scenario);
|
|
67904
|
-
|
|
68049
|
+
if (isInspectionOnlyScenario(scenario, actions)) {
|
|
68050
|
+
return null;
|
|
68051
|
+
}
|
|
68052
|
+
if (isNavigationFocusedScenario(scenario)) {
|
|
68053
|
+
return null;
|
|
68054
|
+
}
|
|
68055
|
+
if (isFormValidationScenario(scenario)) {
|
|
68056
|
+
const typedFields = actions.some((a) => a.type === "TYPE");
|
|
68057
|
+
const hasCommit = actions.some((a) => isFormCommitAction(a));
|
|
68058
|
+
const hasValidationAssert = actions.some((a) => {
|
|
68059
|
+
if (a.type !== "ASSERT_VISIBLE" && a.type !== "ASSERT_TEXT")
|
|
68060
|
+
return false;
|
|
68061
|
+
return /\b(error|required|invalid|empty|missing|incorrect|wrong|try again|mismatch|does not match|must match)\b/.test(actionText(a));
|
|
68062
|
+
});
|
|
68063
|
+
const emptyFieldIntent = /\b(empty|required|missing|blank)\b/.test(text);
|
|
68064
|
+
const invalidValueIntent = /\b(wrong|incorrect|invalid|mismatch|reject)\b/.test(text);
|
|
68065
|
+
if (emptyFieldIntent && !invalidValueIntent) {
|
|
68066
|
+
if (!hasCommit && !hasValidationAssert) {
|
|
68067
|
+
return "validation scenario did not record a form commit or validation outcome";
|
|
68068
|
+
}
|
|
68069
|
+
return null;
|
|
68070
|
+
}
|
|
68071
|
+
if (typedFields && !hasCommit) {
|
|
68072
|
+
return "validation scenario filled fields but never committed the form";
|
|
68073
|
+
}
|
|
68074
|
+
if (!hasCommit && !hasValidationAssert) {
|
|
68075
|
+
return "validation scenario did not record a form commit or validation outcome";
|
|
68076
|
+
}
|
|
68077
|
+
return null;
|
|
68078
|
+
}
|
|
68079
|
+
if (isFormMutationHappyPath(scenario)) {
|
|
68080
|
+
const hasType = actions.some((a) => a.type === "TYPE");
|
|
68081
|
+
const hasCommit = actions.some((a) => isFormCommitAction(a));
|
|
68082
|
+
if (!hasType) {
|
|
68083
|
+
return "form-mutation happy path did not record any field entry";
|
|
68084
|
+
}
|
|
68085
|
+
if (!hasCommit) {
|
|
68086
|
+
return "form-mutation happy path did not record a form commit";
|
|
68087
|
+
}
|
|
68088
|
+
}
|
|
68089
|
+
const isCredentialSignIn = /\b(sign\s*in|log\s*in|login|authenticate|auth)\b/.test(text) && /\b(email|e-mail|password|credential|valid credentials?)\b/.test(text) && !/\b(reset|forgot|recover|logout|log\s*out|sign\s*out|change\s+password|update\s+password)\b/.test(text);
|
|
67905
68090
|
const typedEmailIndexes = [];
|
|
67906
68091
|
const typedPasswordIndexes = [];
|
|
67907
68092
|
actions.forEach((action, index) => {
|
|
@@ -67918,19 +68103,19 @@ ${statePacket}` },
|
|
|
67918
68103
|
const lastCredentialIndex = Math.max(...typedEmailIndexes, ...typedPasswordIndexes);
|
|
67919
68104
|
const hasAnyCredentialEntry = Number.isFinite(lastCredentialIndex);
|
|
67920
68105
|
const submitAfterCredentials = actions.some((action, index) => {
|
|
67921
|
-
if (
|
|
68106
|
+
if (index < lastCredentialIndex)
|
|
67922
68107
|
return false;
|
|
67923
|
-
|
|
67924
|
-
if (!/\b(sign\s*in|log\s*in|login|continue|submit)\b/.test(textForAction))
|
|
68108
|
+
if (!isFormCommitAction(action))
|
|
67925
68109
|
return false;
|
|
67926
|
-
|
|
68110
|
+
if (action.type === "TYPE" && index === lastCredentialIndex)
|
|
68111
|
+
return true;
|
|
68112
|
+
return index > lastCredentialIndex;
|
|
67927
68113
|
});
|
|
67928
68114
|
const isAuthLikeScenario = /\b(sign(?:ed)?\s*in|log(?:ged)?\s*in|login|authenticate|auth|credential|email|e-mail|password)\b/.test(text);
|
|
67929
68115
|
if (isAuthLikeScenario && hasAnyCredentialEntry && !submitAfterCredentials) {
|
|
67930
|
-
return "credential fields were filled but no
|
|
68116
|
+
return "credential fields were filled but no form commit was recorded after them";
|
|
67931
68117
|
}
|
|
67932
68118
|
const uiOnlyCredentialScenario = /\b(mask|masked|visibility|show password|hide password|toggle|keyboard|keypad|max(?:imum)?\s*length|field focus|autofill)\b/.test(text);
|
|
67933
|
-
const emptyFieldValidationScenario = /\b(empty|required|missing|blank)\b/.test(text);
|
|
67934
68119
|
const sessionLifecycleScenario = /\b(logout|log\s*out|sign\s*out|session persistence|remain(?:s)? signed in|remember me)\b/.test(text);
|
|
67935
68120
|
const loginTapWithoutCredentialPath = actions.some((action) => {
|
|
67936
68121
|
if (action.type !== "TAP")
|
|
@@ -67938,9 +68123,11 @@ ${statePacket}` },
|
|
|
67938
68123
|
const textForAction = actionText(action);
|
|
67939
68124
|
if (/\b(logout|log\s*out|sign\s*out)\b/.test(textForAction))
|
|
67940
68125
|
return false;
|
|
68126
|
+
if (isAuthMethodChooserTap(action) || isNavigationChromeTap(action))
|
|
68127
|
+
return false;
|
|
67941
68128
|
return /\b(sign\s*in|log\s*in|login)\b/.test(textForAction);
|
|
67942
68129
|
});
|
|
67943
|
-
if (loginTapWithoutCredentialPath && !hasAnyCredentialEntry && !uiOnlyCredentialScenario &&
|
|
68130
|
+
if (loginTapWithoutCredentialPath && !hasAnyCredentialEntry && !uiOnlyCredentialScenario && (sessionLifecycleScenario || /\b(valid credentials?|happy path|successful sign\s*in|successful login)\b/.test(text))) {
|
|
67944
68131
|
return "auth/session scenario tapped a login or sign-in control without recording a credential path";
|
|
67945
68132
|
}
|
|
67946
68133
|
if (isCredentialSignIn) {
|
|
@@ -67952,7 +68139,7 @@ ${statePacket}` },
|
|
|
67952
68139
|
const missing = [
|
|
67953
68140
|
typedEmail ? null : "email/username entry",
|
|
67954
68141
|
typedPassword ? null : "password entry",
|
|
67955
|
-
tappedSubmit ? null : "
|
|
68142
|
+
tappedSubmit ? null : "form commit (submit tap or keyboard Enter)"
|
|
67956
68143
|
].filter((value) => value !== null);
|
|
67957
68144
|
return `credential sign-in scenario did not record ${missing.join(", ")}`;
|
|
67958
68145
|
}
|
|
@@ -67973,11 +68160,12 @@ ${statePacket}` },
|
|
|
67973
68160
|
const typeScreen = actions[lastTypeIndex]?.metadata?.screenName?.trim() ?? "";
|
|
67974
68161
|
const assertScreen = actions[lastAssertIndex]?.metadata?.screenName?.trim() ?? "";
|
|
67975
68162
|
const screensDiffer = typeScreen !== "" && assertScreen !== "" && typeScreen !== assertScreen;
|
|
67976
|
-
const
|
|
67977
|
-
|
|
68163
|
+
const lastTypeAction = actions[lastTypeIndex];
|
|
68164
|
+
const keyboardCommitOnLastType = lastTypeAction != null && isKeyboardFormCommit(lastTypeAction);
|
|
68165
|
+
const commitAfterType = keyboardCommitOnLastType || actions.some((action, index) => {
|
|
68166
|
+
if (index <= lastTypeIndex || index >= lastAssertIndex)
|
|
67978
68167
|
return false;
|
|
67979
|
-
|
|
67980
|
-
return /\b(sign\s*in|log\s*in|login|continue|submit|save|create|send|confirm|done|next|apply|post|checkout|pay|finish|update|register|sign\s*up)\b/i.test(textForAction);
|
|
68168
|
+
return isFormCommitAction(action);
|
|
67981
68169
|
});
|
|
67982
68170
|
if (screensDiffer && !commitAfterType) {
|
|
67983
68171
|
return "form fields were filled then a success assert was recorded without a commit/submit tap between them";
|
|
@@ -67988,6 +68176,17 @@ ${statePacket}` },
|
|
|
67988
68176
|
return `${action.type}|${action.selector}|${action.value ?? ""}`;
|
|
67989
68177
|
}
|
|
67990
68178
|
var MAX_RETRY_CYCLE_LENGTH = 8;
|
|
68179
|
+
function dedupeConsecutiveIdenticalActions(actions) {
|
|
68180
|
+
if (actions.length <= 1)
|
|
68181
|
+
return actions;
|
|
68182
|
+
const out = [actions[0]];
|
|
68183
|
+
for (let i = 1; i < actions.length; i++) {
|
|
68184
|
+
if (actionCycleKey(actions[i]) === actionCycleKey(out[out.length - 1]))
|
|
68185
|
+
continue;
|
|
68186
|
+
out.push(actions[i]);
|
|
68187
|
+
}
|
|
68188
|
+
return out;
|
|
68189
|
+
}
|
|
67991
68190
|
function collapseRetryCycles(actions) {
|
|
67992
68191
|
const out = [...actions];
|
|
67993
68192
|
let changed = true;
|
|
@@ -68028,15 +68227,30 @@ ${statePacket}` },
|
|
|
68028
68227
|
}
|
|
68029
68228
|
function assembleTest(args) {
|
|
68030
68229
|
const { scenario, recording, ctx, platform: platform3, appDataCleared, log: log2 } = args;
|
|
68031
|
-
let actions =
|
|
68230
|
+
let actions = dedupeConsecutiveIdenticalActions(recording.actions);
|
|
68032
68231
|
if (actions.length < recording.actions.length) {
|
|
68033
|
-
log2(`
|
|
68232
|
+
log2(` deduped ${recording.actions.length - actions.length} consecutive identical step(s) (${recording.actions.length} \u2192 ${actions.length}).`);
|
|
68233
|
+
}
|
|
68234
|
+
const beforeBackCollapse = actions.length;
|
|
68235
|
+
actions = collapseConsecutiveBacks(actions);
|
|
68236
|
+
if (actions.length < beforeBackCollapse) {
|
|
68237
|
+
log2(` collapsed ${beforeBackCollapse - actions.length} consecutive BACK step(s) (${beforeBackCollapse} \u2192 ${actions.length}).`);
|
|
68238
|
+
}
|
|
68239
|
+
const beforeCycleCollapse = actions.length;
|
|
68240
|
+
actions = collapseRetryCycles(actions);
|
|
68241
|
+
if (actions.length < beforeCycleCollapse) {
|
|
68242
|
+
log2(` collapsed ${beforeCycleCollapse - actions.length} retry-cycle step(s) out of the recorded trace (${beforeCycleCollapse} \u2192 ${actions.length}).`);
|
|
68034
68243
|
}
|
|
68035
68244
|
const beforeContainerDrop = actions.length;
|
|
68036
68245
|
actions = actions.filter((a) => !isBareContainerTap(a));
|
|
68037
68246
|
if (actions.length < beforeContainerDrop) {
|
|
68038
68247
|
log2(` dropped ${beforeContainerDrop - actions.length} bare-container tap step(s) (no identity \u2014 un-replayable ambient taps).`);
|
|
68039
68248
|
}
|
|
68249
|
+
const beforeWeakAssertDrop = actions.length;
|
|
68250
|
+
actions = actions.filter((a) => !isStructuralOnlyAssert(a));
|
|
68251
|
+
if (actions.length < beforeWeakAssertDrop) {
|
|
68252
|
+
log2(` dropped ${beforeWeakAssertDrop - actions.length} structural-only assert step(s) (no product identity).`);
|
|
68253
|
+
}
|
|
68040
68254
|
const intentMismatch = scenarioIntentMismatchReason(scenario, actions);
|
|
68041
68255
|
if (intentMismatch) {
|
|
68042
68256
|
log2(` skipped scenario "${scenario.name}": ${intentMismatch}.`);
|
|
@@ -68848,10 +69062,22 @@ For a fix, return the COMPLETE steps array (not a diff).`;
|
|
|
68848
69062
|
if (!/\b(tap|click|press)\b/i.test(step.action))
|
|
68849
69063
|
return false;
|
|
68850
69064
|
const text = mobileStepSearchText(step);
|
|
68851
|
-
if (/\b(logout|log\s*out|sign\s*out|reset|forgot|
|
|
69065
|
+
if (/\b(logout|log\s*out|sign\s*out|reset|forgot|go\s+back|back\s+to|cancel|close|dismiss)\b/.test(text)) {
|
|
68852
69066
|
return false;
|
|
68853
69067
|
}
|
|
68854
|
-
|
|
69068
|
+
if (/\b(continue\s+with|sign\s+in\s+with|log\s+in\s+with|use\s+(email|phone)|choose|pick|select)\b/.test(text) && !/\b(submit|confirm)\b/.test(text)) {
|
|
69069
|
+
return false;
|
|
69070
|
+
}
|
|
69071
|
+
return /\b(sign\s*in|signin|log\s*in|login|continue|submit|done|next|confirm)\b/.test(text);
|
|
69072
|
+
}
|
|
69073
|
+
function isKeyboardFormCommitStep(step) {
|
|
69074
|
+
if (!/\b(type|fill|input|enter)\b/i.test(step.action))
|
|
69075
|
+
return false;
|
|
69076
|
+
const value = typeof step.value === "string" ? step.value : "";
|
|
69077
|
+
return value.endsWith("\n");
|
|
69078
|
+
}
|
|
69079
|
+
function isFormCommitStep(step) {
|
|
69080
|
+
return isCredentialSubmitTap(step) || isKeyboardFormCommitStep(step);
|
|
68855
69081
|
}
|
|
68856
69082
|
function detectInvalidAuthValidationSetup(steps, error2) {
|
|
68857
69083
|
if (!/\b(element not found|no such element|not visible|timeout|assert|could not be located|unexpected screen|screen drift)\b/i.test(error2 ?? "")) {
|
|
@@ -68876,20 +69102,23 @@ For a fix, return the COMPLETE steps array (not a diff).`;
|
|
|
68876
69102
|
if ((step.order ?? -1) >= lastCredentialEditOrder)
|
|
68877
69103
|
return false;
|
|
68878
69104
|
const text = mobileStepSearchText(step);
|
|
68879
|
-
return /\b(
|
|
69105
|
+
return /\b(sign\s+in\s+with\s+email|log\s+in\s+with\s+email|email\s+and\s+password|login\s+form|credential\s+form|email|e-mail|username|user\s*name)\b/.test(text);
|
|
68880
69106
|
});
|
|
68881
69107
|
if (hasCredentialFormEvidence)
|
|
68882
69108
|
return null;
|
|
68883
69109
|
return "credential-control test edits a credential field without recording a path to the email/password form; stale auth-control test is invalid until generation records a complete form path";
|
|
68884
69110
|
}
|
|
68885
|
-
const hasSubmitAfterCredentialEdit = steps.some((step) => (step.order ?? -1) > lastCredentialEditOrder &&
|
|
69111
|
+
const hasSubmitAfterCredentialEdit = steps.some((step) => (step.order ?? -1) > lastCredentialEditOrder && isFormCommitStep(step));
|
|
68886
69112
|
if (hasSubmitAfterCredentialEdit)
|
|
68887
69113
|
return null;
|
|
69114
|
+
const hasKeyboardCommitOnCredential = steps.some((step) => (step.order ?? -1) === lastCredentialEditOrder && isKeyboardFormCommitStep(step));
|
|
69115
|
+
if (hasKeyboardCommitOnCredential)
|
|
69116
|
+
return null;
|
|
68888
69117
|
const hasValidationAssertion = steps.some((step) => {
|
|
68889
69118
|
if (!/\b(assert|wait)\b/i.test(step.action))
|
|
68890
69119
|
return false;
|
|
68891
69120
|
const text = mobileStepSearchText(step);
|
|
68892
|
-
return /\b(invalid|required|error|empty|missing|incorrect|wrong|try again|validation|must enter|enter (an )?(email|password)|email is required|password is required)\b/.test(text);
|
|
69121
|
+
return /\b(invalid|required|error|empty|missing|incorrect|wrong|try again|validation|must enter|enter (an )?(email|password)|email is required|password is required|mismatch|does not match)\b/.test(text);
|
|
68893
69122
|
});
|
|
68894
69123
|
if (hasValidationAssertion)
|
|
68895
69124
|
return null;
|
|
@@ -330649,7 +330878,7 @@ var require_package4 = __commonJS({
|
|
|
330649
330878
|
"package.json"(exports, module) {
|
|
330650
330879
|
module.exports = {
|
|
330651
330880
|
name: "@validate.qa/runner",
|
|
330652
|
-
version: "1.0.
|
|
330881
|
+
version: "1.0.34",
|
|
330653
330882
|
description: "validate local test runner - execute Playwright tests on your machine, connected to validate.qa cloud.",
|
|
330654
330883
|
bin: {
|
|
330655
330884
|
"validate-runner": "dist/cli.js"
|
package/package.json
CHANGED