testeiya 0.3.9 → 0.3.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +353 -27
- package/dist/prompt/index.js +6 -2
- package/dist/prompt/index.js.map +1 -1
- package/dist/prompt/print.js +8 -0
- package/dist/prompt/print.js.map +1 -1
- package/dist/prompt/system-prompt.js +14 -4
- package/dist/prompt/system-prompt.js.map +1 -1
- package/dist/prompt/testomatio.js +3 -3
- package/dist/prompt/tools.js +28 -7
- package/dist/prompt/tools.js.map +1 -1
- package/dist/src/args.js +198 -46
- package/dist/src/args.js.map +1 -1
- package/dist/src/cli.js +64 -20
- package/dist/src/cli.js.map +1 -1
- package/dist/src/doctor.js +171 -0
- package/dist/src/doctor.js.map +1 -0
- package/dist/src/env.js +39 -5
- package/dist/src/env.js.map +1 -1
- package/dist/src/mcp.js +3 -3
- package/dist/src/mcp.js.map +1 -1
- package/dist/src/model.js +15 -10
- package/dist/src/model.js.map +1 -1
- package/dist/src/models.js +38 -0
- package/dist/src/models.js.map +1 -0
- package/dist/src/output.js +216 -0
- package/dist/src/output.js.map +1 -0
- package/dist/src/run.js +141 -27
- package/dist/src/run.js.map +1 -1
- package/dist/src/session.js +46 -25
- package/dist/src/session.js.map +1 -1
- package/dist/src/sessions.js +72 -0
- package/dist/src/sessions.js.map +1 -0
- package/dist/src/skills.js +130 -0
- package/dist/src/skills.js.map +1 -0
- package/package.json +7 -3
- package/prompt/index.ts +11 -2
- package/prompt/print.ts +9 -0
- package/prompt/system-prompt.ts +18 -4
- package/prompt/testomatio.ts +3 -3
- package/prompt/tools.ts +30 -7
- package/skills/codeceptjs/ci-fix-tests/SKILL.md +119 -0
- package/skills/codeceptjs/codeceptjs-auth/SKILL.md +141 -0
- package/skills/codeceptjs/codeceptjs-exploration/SKILL.md +100 -0
- package/skills/codeceptjs/codeceptjs-fundamentals/SKILL.md +196 -0
- package/skills/codeceptjs/codeceptjs-run-analysis/SKILL.md +83 -0
- package/skills/codeceptjs/debugging-codeceptjs-tests/SKILL.md +126 -0
- package/skills/codeceptjs/migrate-codeceptjs-4/SKILL.md +109 -0
- package/skills/codeceptjs/migrate-cypress-to-codeceptjs/SKILL.md +295 -0
- package/skills/codeceptjs/migrate-protractor-to-codeceptjs/SKILL.md +339 -0
- package/skills/codeceptjs/migrate-selenium-java-to-codeceptjs/SKILL.md +459 -0
- package/skills/codeceptjs/migrate-testcafe-to-codeceptjs/SKILL.md +338 -0
- package/skills/codeceptjs/refactoring-codeceptjs-tests/SKILL.md +78 -0
- package/skills/codeceptjs/writing-codeceptjs-tests/SKILL.md +102 -0
- package/skills/playwright/playwright-best-practices/SKILL.md +303 -0
- package/skills/playwright/playwright-best-practices/advanced/authentication-flows.md +360 -0
- package/skills/playwright/playwright-best-practices/advanced/authentication.md +871 -0
- package/skills/playwright/playwright-best-practices/advanced/clock-mocking.md +364 -0
- package/skills/playwright/playwright-best-practices/advanced/mobile-testing.md +409 -0
- package/skills/playwright/playwright-best-practices/advanced/multi-context.md +288 -0
- package/skills/playwright/playwright-best-practices/advanced/multi-user.md +393 -0
- package/skills/playwright/playwright-best-practices/advanced/network-advanced.md +452 -0
- package/skills/playwright/playwright-best-practices/advanced/third-party.md +464 -0
- package/skills/playwright/playwright-best-practices/architecture/pom-vs-fixtures.md +363 -0
- package/skills/playwright/playwright-best-practices/architecture/test-architecture.md +369 -0
- package/skills/playwright/playwright-best-practices/architecture/when-to-mock.md +383 -0
- package/skills/playwright/playwright-best-practices/browser-apis/browser-apis.md +391 -0
- package/skills/playwright/playwright-best-practices/browser-apis/iframes.md +403 -0
- package/skills/playwright/playwright-best-practices/browser-apis/service-workers.md +504 -0
- package/skills/playwright/playwright-best-practices/browser-apis/websockets.md +403 -0
- package/skills/playwright/playwright-best-practices/core/annotations.md +424 -0
- package/skills/playwright/playwright-best-practices/core/assertions-waiting.md +361 -0
- package/skills/playwright/playwright-best-practices/core/configuration.md +452 -0
- package/skills/playwright/playwright-best-practices/core/fixtures-hooks.md +417 -0
- package/skills/playwright/playwright-best-practices/core/global-setup.md +434 -0
- package/skills/playwright/playwright-best-practices/core/locators.md +242 -0
- package/skills/playwright/playwright-best-practices/core/page-object-model.md +315 -0
- package/skills/playwright/playwright-best-practices/core/projects-dependencies.md +453 -0
- package/skills/playwright/playwright-best-practices/core/test-data.md +492 -0
- package/skills/playwright/playwright-best-practices/core/test-suite-structure.md +361 -0
- package/skills/playwright/playwright-best-practices/core/test-tags.md +298 -0
- package/skills/playwright/playwright-best-practices/debugging/console-errors.md +420 -0
- package/skills/playwright/playwright-best-practices/debugging/debugging.md +504 -0
- package/skills/playwright/playwright-best-practices/debugging/error-testing.md +360 -0
- package/skills/playwright/playwright-best-practices/debugging/flaky-tests.md +496 -0
- package/skills/playwright/playwright-best-practices/frameworks/angular.md +530 -0
- package/skills/playwright/playwright-best-practices/frameworks/nextjs.md +469 -0
- package/skills/playwright/playwright-best-practices/frameworks/react.md +531 -0
- package/skills/playwright/playwright-best-practices/frameworks/vue.md +574 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/ci-cd.md +468 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/docker.md +283 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/github-actions.md +546 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/gitlab.md +397 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/other-providers.md +521 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/parallel-sharding.md +371 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/performance.md +453 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/reporting.md +424 -0
- package/skills/playwright/playwright-best-practices/infrastructure-ci-cd/test-coverage.md +497 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/accessibility.md +359 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/api-testing.md +719 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/browser-extensions.md +506 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/canvas-webgl.md +493 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/component-testing.md +500 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/drag-drop.md +576 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/electron.md +509 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/file-operations.md +377 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/file-upload-download.md +562 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/forms-validation.md +561 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/graphql-testing.md +331 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/i18n.md +508 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/performance-testing.md +476 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/security-testing.md +430 -0
- package/skills/playwright/playwright-best-practices/testing-patterns/visual-regression.md +634 -0
- package/skills/playwright/playwright-cli/SKILL.md +420 -0
- package/skills/playwright/playwright-cli/references/element-attributes.md +23 -0
- package/skills/playwright/playwright-cli/references/playwright-tests.md +39 -0
- package/skills/playwright/playwright-cli/references/request-mocking.md +87 -0
- package/skills/playwright/playwright-cli/references/running-code.md +241 -0
- package/skills/playwright/playwright-cli/references/session-management.md +225 -0
- package/skills/playwright/playwright-cli/references/storage-state.md +275 -0
- package/skills/playwright/playwright-cli/references/test-generation.md +433 -0
- package/skills/playwright/playwright-cli/references/tracing.md +139 -0
- package/skills/playwright/playwright-cli/references/video-recording.md +143 -0
- package/skills/skills.lock.json +41 -37
- package/skills/skills.yaml +1 -0
- package/skills/testomatio/explorbot/explorbot-fundamentals/SKILL.md +76 -0
- package/skills/testomatio/explorbot/explorbot-fundamentals/references/no-install.md +36 -0
- package/skills/testomatio/explorbot/explorbot-plan/SKILL.md +108 -0
- package/skills/testomatio/explorbot/explorbot-setup/SKILL.md +86 -0
- package/skills/testomatio/explorbot/prima/SKILL.md +87 -0
- package/skills/testomatio/qa-process/qa-explain-behavior/SKILL.md +111 -0
- package/skills/testomatio/qa-process/qa-explain-behavior/references/answer-shapes.md +116 -0
- package/skills/testomatio/qa-process/qa-explain-behavior/references/examples.md +157 -0
- package/skills/testomatio/qa-process/qa-lead-strategy-advisor/SKILL.md +91 -0
- package/skills/testomatio/qa-process/qa-lead-strategy-advisor/references/output-format.md +71 -0
- package/skills/testomatio/qa-process/testing-workflow/SKILL.md +227 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/CLAUDE.md +56 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/SKILL.md +242 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/references/CODECEPTJS_BEST_PRACTICES.md +182 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/references/FINAL_SUMMARY_TEMPLATE.md +22 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/references/PLAYWRIGHT_BEST_PRACTICES.md +90 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/references/POM_BEST_PRACTICES.md +53 -0
- package/skills/testomatio/test-automation/automate-manual-test-cases/references/TEST_DATA_MANAGEMENT.md +52 -0
- package/skills/testomatio/test-automation/debug-fix-failed-flaky-autotests/SKILL.md +107 -0
- package/skills/testomatio/test-automation/debug-fix-failed-flaky-autotests/references/DEBUGGING_QUICK_REFERENCE.md +113 -0
- package/skills/testomatio/test-automation/qa-automation-test-consolidation/SKILL.md +29 -0
- package/skills/testomatio/test-automation/qa-data-seeder/SKILL.md +83 -0
- package/skills/testomatio/test-automation/run-tests-with-testomatio-reporter/SKILL.md +153 -0
- package/skills/testomatio/test-automation/setup-change-aware-pr-testing/SKILL.md +190 -0
- package/skills/testomatio/test-automation/setup-ci-automation/SKILL.md +76 -0
- package/skills/testomatio/test-automation/testomat-allure-adapter/SKILL.md +289 -0
- package/skills/testomatio/test-management/detect-duplicate-test-cases/SKILL.md +86 -0
- package/skills/testomatio/test-management/detect-duplicate-test-cases/references/DUPLICATE_INSTRUCTIONS.md +54 -0
- package/skills/testomatio/test-management/improve-test-cases/SKILL.md +114 -0
- package/skills/testomatio/test-management/improve-test-cases/references/TESTOMAT_MARKDOWN_EXAMPLE.md +66 -0
- package/skills/testomatio/test-management/pull-request-diff-analyzer/SKILL.md +141 -0
- package/skills/testomatio/test-management/qa-e2e-tests-reporting/SKILL.md +354 -0
- package/skills/testomatio/test-management/qa-e2e-tests-reporting/references/TESTOMATIO_ARTIFACTS.md +82 -0
- package/skills/testomatio/test-management/qa-e2e-tests-reporting/references/TESTOMATIO_HTML_REPORT.md +37 -0
- package/skills/testomatio/test-management/qa-e2e-tests-reporting/references/TESTOMATIO_REPORTERS_CONFIG.md +118 -0
- package/skills/testomatio/test-management/qa-pr-requirements-analyzer/SKILL.md +149 -0
- package/skills/testomatio/test-management/qa-pr-requirements-analyzer/references/summary-example.md +54 -0
- package/skills/testomatio/test-management/qa-requirement-reviewer/SKILL.md +126 -0
- package/skills/testomatio/test-management/qa-requirement-reviewer/references/requirements_reviewer_examples.md +128 -0
- package/skills/testomatio/test-management/qa-split-testing-levels-pyramid/SKILL.md +55 -0
- package/skills/testomatio/test-management/qa-sprint-report-by-testomatio/SKILL.md +220 -0
- package/skills/testomatio/test-management/qa-sprint-report-by-testomatio/references/qa-sprint-report.md +126 -0
- package/skills/testomatio/test-management/qa-test-code-coverage/SKILL.md +174 -0
- package/skills/testomatio/test-management/qa-test-code-coverage/references/COVERAGE_FILE_FORMAT.md +138 -0
- package/skills/testomatio/test-management/qa-test-code-coverage/references/E2E_FRAMEWORKS.md +65 -0
- package/skills/testomatio/test-management/qa-test-code-coverage/scripts/check-coverage.mjs +40 -0
- package/skills/testomatio/test-management/qa-thinking/SKILL.md +36 -0
- package/skills/testomatio/test-management/qa-write-test-cases/SKILL.md +267 -0
- package/skills/testomatio/test-management/qa-write-test-cases/references/test-case-format.md +387 -0
- package/skills/testomatio/test-management/qa-write-test-cases/references/testomat-tms-guide.md +72 -0
- package/skills/testomatio/test-management/qa-write-test-cases/references/writing-rule.md +159 -0
- package/skills/testomatio/test-management/scan-automation-project/SKILL.md +152 -0
- package/skills/testomatio/test-management/sync-test-cases-with-tms/SKILL.md +107 -0
- package/skills/testomatio/test-management/sync-test-cases-with-tms/references/TESTOMATIO_CLI.md +140 -0
- package/skills/testomatio/test-management/testomatio-mcp/SKILL.md +195 -0
- package/skills/testomatio/test-management/testomatio-mcp/references/MCP_SETUP.md +177 -0
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "testeiya",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.14",
|
|
4
4
|
"description": "AI testing agent — QA-focused coding agent for manual and automated tests",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -28,10 +28,12 @@
|
|
|
28
28
|
},
|
|
29
29
|
"scripts": {
|
|
30
30
|
"start": "npm run build && node dist/src/cli.js",
|
|
31
|
+
"dev": "./testeiya",
|
|
31
32
|
"build": "tsc -p tsconfig.json && npm run build:vendor",
|
|
32
33
|
"build:vendor": "esbuild scripts/vendor-entry.ts --bundle --format=esm --platform=node --packages=external --outfile=dist/vendor/mcp.js",
|
|
33
34
|
"typecheck": "tsc -p tsconfig.json --noEmit",
|
|
34
|
-
"prepublishOnly": "npm run build"
|
|
35
|
+
"prepublishOnly": "npm run build",
|
|
36
|
+
"test": "npm run build && node --test test/*.test.js"
|
|
35
37
|
},
|
|
36
38
|
"keywords": [
|
|
37
39
|
"testing",
|
|
@@ -54,6 +56,8 @@
|
|
|
54
56
|
"devDependencies": {
|
|
55
57
|
"@types/node": "^22.19.19",
|
|
56
58
|
"esbuild": "^0.28.1",
|
|
57
|
-
"
|
|
59
|
+
"tsx": "^4.23.12",
|
|
60
|
+
"typescript": "^5.9.3",
|
|
61
|
+
"yaml": "^2.9.0"
|
|
58
62
|
}
|
|
59
63
|
}
|
package/prompt/index.ts
CHANGED
|
@@ -6,7 +6,7 @@ import {
|
|
|
6
6
|
projectSettings,
|
|
7
7
|
type TmsAccess,
|
|
8
8
|
} from "./testomatio.js";
|
|
9
|
-
import { nonInteractive, reportOutput } from "./print.js";
|
|
9
|
+
import { briefAnswer, nonInteractive, reportOutput } from "./print.js";
|
|
10
10
|
import {
|
|
11
11
|
contextPromptSection,
|
|
12
12
|
type ContextEntry,
|
|
@@ -28,6 +28,8 @@ export function buildSystemPrompt(options?: SystemPromptOptions): string {
|
|
|
28
28
|
interactive,
|
|
29
29
|
toolBullets: options?.toolBullets,
|
|
30
30
|
rules: options?.rules,
|
|
31
|
+
connectedClis: options?.connectedClis,
|
|
32
|
+
connectedMcps: options?.connectedMcps,
|
|
31
33
|
}),
|
|
32
34
|
];
|
|
33
35
|
|
|
@@ -62,7 +64,8 @@ export function buildSystemPrompt(options?: SystemPromptOptions): string {
|
|
|
62
64
|
parts.push(testomatioNotConnected());
|
|
63
65
|
}
|
|
64
66
|
|
|
65
|
-
// The
|
|
67
|
+
// The answer contract goes last so it is the final instruction the model reads.
|
|
68
|
+
if (options?.brief) parts.push(briefAnswer);
|
|
66
69
|
if (options?.outputFile) parts.push(reportOutput(options.outputFile));
|
|
67
70
|
|
|
68
71
|
return parts.join("\n\n");
|
|
@@ -85,9 +88,15 @@ export interface SystemPromptOptions {
|
|
|
85
88
|
tms?: TmsAccess;
|
|
86
89
|
/** Absolute path the agent must write its final report to (`--output`). */
|
|
87
90
|
outputFile?: string;
|
|
91
|
+
/** Answer a question instead of doing a task and reporting (`testeiya ask`). */
|
|
92
|
+
brief?: boolean;
|
|
88
93
|
projectInfo?: TestomatioProjectInfo | null;
|
|
89
94
|
/** User-added context (linked folders, cloned repos, uploaded docs). */
|
|
90
95
|
contextEntries?: ContextEntry[];
|
|
91
96
|
/** Non-empty predefined `.testeiya` context folders (manual-tests, code, …). */
|
|
92
97
|
contextFolders?: ContextFolder[];
|
|
98
|
+
/** CLI tools the user has connected and signed in (e.g. `gh`, `acli`). */
|
|
99
|
+
connectedClis?: string[];
|
|
100
|
+
/** MCP servers connected for this session (the enabled `mcp.json` set). */
|
|
101
|
+
connectedMcps?: string[];
|
|
93
102
|
}
|
package/prompt/print.ts
CHANGED
|
@@ -21,3 +21,12 @@ export function reportOutput(path: string): string {
|
|
|
21
21
|
</final-report>
|
|
22
22
|
`;
|
|
23
23
|
}
|
|
24
|
+
|
|
25
|
+
export const briefAnswer = dedent`
|
|
26
|
+
<answer>
|
|
27
|
+
* You were asked a question, not given a task. Answer it.
|
|
28
|
+
* Lead with the answer in one line, then the evidence you checked.
|
|
29
|
+
* A few sentences. No report file, no headings, no plan.
|
|
30
|
+
* Say plainly when what you found does not settle the question.
|
|
31
|
+
</answer>
|
|
32
|
+
`;
|
package/prompt/system-prompt.ts
CHANGED
|
@@ -9,6 +9,8 @@ export function getSystemPrompt(cwd?: string, options?: PromptSurface): string {
|
|
|
9
9
|
if (!interactive) missingSecretAction = "report it as a blocker in your output";
|
|
10
10
|
let extraRules = "";
|
|
11
11
|
for (const rule of options?.rules ?? []) extraRules += ` * ${rule}\n`;
|
|
12
|
+
const cliList = (options?.connectedClis ?? []).join(", ") || "none connected yet";
|
|
13
|
+
const mcpList = (options?.connectedMcps ?? []).join(", ") || "none connected yet";
|
|
12
14
|
return dedent`
|
|
13
15
|
<role>
|
|
14
16
|
You are Testeiya, an AI agent that helps with QA tasks.
|
|
@@ -87,6 +89,16 @@ export function getSystemPrompt(cwd?: string, options?: PromptSurface): string {
|
|
|
87
89
|
|
|
88
90
|
${tools({ extra: options?.toolBullets })}
|
|
89
91
|
|
|
92
|
+
<connections>
|
|
93
|
+
Testeiya connects external tools for you — CLI tools and MCP servers. This session has:
|
|
94
|
+
|
|
95
|
+
* **Connected CLIs:** ${cliList}
|
|
96
|
+
* **Connected MCP servers:** ${mcpList}
|
|
97
|
+
|
|
98
|
+
* **Missing Connection — Ask, Never Improvise:** When a task needs a tool that is not in the lists above, STOP and ask the user to connect it in Testeiya (Settings → Connections). Never reach the service sideways: no raw REST/GraphQL calls against its API, no scraping credentials from dotfiles or env dumps, no installing binaries on your own.
|
|
99
|
+
* If a task needs both a CLI and an MCP server, ask once for both and say that Testeiya supports them both as connections — the user installs them side by side there.
|
|
100
|
+
</connections>
|
|
101
|
+
|
|
90
102
|
<goals>
|
|
91
103
|
You help in variety of tasks related to software testing, including writing test cases, analyzing test results, and providing feedback.
|
|
92
104
|
You have skills to perform QA tasks on user demand.
|
|
@@ -98,10 +110,8 @@ export function getSystemPrompt(cwd?: string, options?: PromptSurface): string {
|
|
|
98
110
|
* Assisting with manual test execution
|
|
99
111
|
* Manage tests with TMS Testomat.io
|
|
100
112
|
* Setting up CI pipelines for continuous testing
|
|
101
|
-
* Analyzing requirements
|
|
102
|
-
*
|
|
103
|
-
* Launching automated exploratory tests via explorbot
|
|
104
|
-
* Analyzing test results and providing feedback
|
|
113
|
+
* Analyzing requirements and issues
|
|
114
|
+
* Analyzing test results
|
|
105
115
|
|
|
106
116
|
This is what you can do.
|
|
107
117
|
</goals>
|
|
@@ -148,6 +158,10 @@ export interface PromptSurface {
|
|
|
148
158
|
toolBullets?: string[];
|
|
149
159
|
/** Extra `<rules>` bullets, for rules that only hold in this harness. */
|
|
150
160
|
rules?: string[];
|
|
161
|
+
/** CLI tools the user has connected and signed in (e.g. `gh`, `acli`). */
|
|
162
|
+
connectedClis?: string[];
|
|
163
|
+
/** MCP servers connected for this session (the enabled `mcp.json` set). */
|
|
164
|
+
connectedMcps?: string[];
|
|
151
165
|
}
|
|
152
166
|
|
|
153
167
|
/*
|
package/prompt/testomatio.ts
CHANGED
|
@@ -35,7 +35,7 @@ const tmsViaDirectTools = dedent`
|
|
|
35
35
|
| Runs, testruns, plans, labels, issues, analytics | MCP \`testomatio-*\` tools |
|
|
36
36
|
| Creating/updating tests or suites | edit the markdown file, then \`npx check-tests push\` |
|
|
37
37
|
| Creating/updating runs, linking issues | MCP create/update tools |
|
|
38
|
-
| Launching automated or mixed Run remotely on CI | Use npx @testomatio/reporter run --remote <profile>, raise error if no CI profiles configured
|
|
38
|
+
| Launching automated or mixed Run remotely on CI | Use npx -y @testomatio/reporter@latest run --remote <profile>, raise error if no CI profiles configured
|
|
39
39
|
| Running automated Run | Use local test runner with testomatio reporter attached to report to current project
|
|
40
40
|
|
|
41
41
|
---
|
|
@@ -73,7 +73,7 @@ const tmsViaProxy = dedent`
|
|
|
73
73
|
| Runs, testruns, plans, labels, issues, analytics | \`mcp({ search: … })\`, then \`mcp({ tool: … })\` |
|
|
74
74
|
| Creating/updating tests or suites | edit the markdown file, then \`npx check-tests push\` |
|
|
75
75
|
| Creating/updating runs, linking issues | the matching \`mcp\` create/update operation |
|
|
76
|
-
| Launching automated or mixed Run remotely on CI | Use npx @testomatio/reporter run --remote <profile>, raise error if no CI profiles configured |
|
|
76
|
+
| Launching automated or mixed Run remotely on CI | Use npx -y @testomatio/reporter@latest run --remote <profile>, raise error if no CI profiles configured |
|
|
77
77
|
| Running automated Run | Use local test runner with testomatio reporter attached to report to current project |
|
|
78
78
|
|
|
79
79
|
---
|
|
@@ -106,7 +106,7 @@ const tmsViaCli = dedent`
|
|
|
106
106
|
| Suite hierarchy / structure / which files exist | \`ls\` / \`find\` |
|
|
107
107
|
| Runs, testruns, plans, labels, issues, analytics | \`curl\` against \`/api/v2\` |
|
|
108
108
|
| Creating/updating tests or suites | edit the markdown file, then \`npx check-tests push\` |
|
|
109
|
-
| Launching automated or mixed Run remotely on CI | Use npx @testomatio/reporter run --remote <profile>, raise error if no CI profiles configured |
|
|
109
|
+
| Launching automated or mixed Run remotely on CI | Use npx -y @testomatio/reporter@latest run --remote <profile>, raise error if no CI profiles configured |
|
|
110
110
|
| Running automated Run | Use local test runner with testomatio reporter attached to report to current project |
|
|
111
111
|
|
|
112
112
|
---
|
package/prompt/tools.ts
CHANGED
|
@@ -11,11 +11,19 @@ import dedent from 'dedent';
|
|
|
11
11
|
*/
|
|
12
12
|
export function tools(options?: { extra?: string[] }): string {
|
|
13
13
|
const extra = (options?.extra ?? []).join('');
|
|
14
|
-
let
|
|
15
|
-
if (
|
|
16
|
-
|
|
14
|
+
let routing = ' - GitHub: issues and projects through MCP.';
|
|
15
|
+
if (onPath('gh')) {
|
|
16
|
+
routing +=
|
|
17
17
|
'\n - Pull requests and repositories: `gh` in `bash`. When a PR question needs issue detail, query the issue through MCP — never infer it from a PR title.';
|
|
18
18
|
}
|
|
19
|
+
if (onPath('acli')) {
|
|
20
|
+
routing +=
|
|
21
|
+
'\n - Jira in bulk (many issues at once), boards, sprints and filters: `acli` in `bash`, per the `atlassian` skill. Confluence and attachments have no `acli` command and stay on MCP.';
|
|
22
|
+
}
|
|
23
|
+
if (onPath('glab')) {
|
|
24
|
+
routing +=
|
|
25
|
+
'\n - GitLab releases, labels, milestones, snippets, CI artifacts, and any endpoint with no MCP tool: `glab` in `bash`. Issues, merge requests, pipelines and code search stay on MCP, which covers them well.';
|
|
26
|
+
}
|
|
19
27
|
return dedent`
|
|
20
28
|
<available-tools>
|
|
21
29
|
You have these tools available:
|
|
@@ -27,7 +35,7 @@ ${extra} * **Internal Skills:** For QA-related tasks, check available skills fi
|
|
|
27
35
|
* **External Integrations:** Use MCP tools when they provide superior data or specialized integrations.
|
|
28
36
|
- Primary MCP (Testomat.io): Use Testomat.io MCP tools to extend functionality.
|
|
29
37
|
- Secondary MCP (Jira, GitHub, etc.): Invoke only when user explicitly asks or when remote context exploration is required.
|
|
30
|
-
${
|
|
38
|
+
${routing}
|
|
31
39
|
|
|
32
40
|
<tool-governance>
|
|
33
41
|
* **Prioritize Specificity:** Use \`read\` instead of \`cat\`, and \`edit\` instead of \`sed\`. Dedicated tools provide better error tracking and safety.
|
|
@@ -39,11 +47,26 @@ ${github}
|
|
|
39
47
|
`;
|
|
40
48
|
}
|
|
41
49
|
|
|
42
|
-
/** Is
|
|
43
|
-
function
|
|
50
|
+
/** Is this CLI on PATH? Never tell the agent to use a binary this machine lacks. */
|
|
51
|
+
function onPath(bin: string): boolean {
|
|
44
52
|
for (const dir of (process.env.PATH ?? '').split(delimiter)) {
|
|
45
53
|
if (!dir) continue;
|
|
46
|
-
if (existsSync(join(dir,
|
|
54
|
+
if (existsSync(join(dir, bin)) || existsSync(join(dir, `${bin}.exe`))) return true;
|
|
47
55
|
}
|
|
48
56
|
return false;
|
|
49
57
|
}
|
|
58
|
+
|
|
59
|
+
/**
|
|
60
|
+
* CLIs Testeiya knows how to connect, probed on PATH — the same trio routed in
|
|
61
|
+
* `<available-tools>` above. Shared with the prompt's connections section so
|
|
62
|
+
* the two can never disagree about whether `gh` etc. exist.
|
|
63
|
+
*/
|
|
64
|
+
export const KNOWN_CLIS: { bin: string; label: string }[] = [
|
|
65
|
+
{ bin: 'gh', label: 'gh (GitHub CLI)' },
|
|
66
|
+
{ bin: 'acli', label: 'acli (Atlassian CLI)' },
|
|
67
|
+
{ bin: 'glab', label: 'glab (GitLab CLI)' },
|
|
68
|
+
];
|
|
69
|
+
|
|
70
|
+
export function pathClis(): string[] {
|
|
71
|
+
return KNOWN_CLIS.filter((c) => onPath(c.bin)).map((c) => c.label);
|
|
72
|
+
}
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ci-fix-tests
|
|
3
|
+
description: "Use on CI in non-interactive mode after a failed CodeceptJS run — attempts safe fixes only (locator drift, missing waits), reruns failing scenarios, rolls back edits that didn't help, reports at `output/ci-fix.md`. No refactors, no config edits, no flake-masking. Trigger on \"ci fix tests\", \"auto-fix failing tests\", \"attempt repair on CI\", or as a CI step after a failed run."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Auto-fix CodeceptJS Tests on CI
|
|
7
|
+
|
|
8
|
+
A non-interactive auto-repair loop. Reads the failed-tests baseline from `output/` (where `aiTrace` left trace artifacts), attempts a small set of safe fixes, reruns only the failing scenarios, compares, and writes a report at `output/ci-fix.md`. Rolls back any edit that didn't help.
|
|
9
|
+
|
|
10
|
+
**Conservative by design.** CI is not the place to restructure tests, change abstractions, or fix anything that needs human judgement. The goal is "pass the run". Anything riskier is escalated through the report.
|
|
11
|
+
|
|
12
|
+
This skill assumes a working **git** checkout — rollback uses `git checkout -- <file>` to restore originals.
|
|
13
|
+
|
|
14
|
+
## Workflow
|
|
15
|
+
|
|
16
|
+
### 1. Read the baseline failures
|
|
17
|
+
Use the **codeceptjs-run-analysis** skill against the existing `output/trace_*/` artifacts to enumerate every failing scenario. Record:
|
|
18
|
+
|
|
19
|
+
- the **set** of failing tests (file path + scenario name)
|
|
20
|
+
- the failed step in each, and the locator / wait / assertion involved
|
|
21
|
+
- a cluster signature per failure (so unique vs systemic is clear)
|
|
22
|
+
|
|
23
|
+
Save this as the **baseline** — `count`, `failing_set`, `clusters`.
|
|
24
|
+
|
|
25
|
+
### 2. Pick safe fix candidates
|
|
26
|
+
Only attempt fixes from this allowed list:
|
|
27
|
+
|
|
28
|
+
- **Locator drift** — the failed locator no longer matches anything but a similar element exists. Use the **codeceptjs-exploration** skill (headless) to find candidates; pick a short locator scoped to a stable region — `I.click('Save', '.toolbar')` — in this order: visible text / accessible name → ARIA `{ role, name }` → `$name` (`customLocator`) → composed CSS. Replace the locator at the failing step only.
|
|
29
|
+
- **Missing wait for a spinner / loader / modal** — the failed step's ARIA snapshot shows a spinner or skeleton present, or the target element appears later. Add a single matching `I.waitFor*` immediately before the failing step.
|
|
30
|
+
- **`I.wait(N)` replacement** — when a hardcoded sleep is the only thing between a failing assertion and a passing one and the gating element is identifiable, replace the sleep with a specific `waitFor*`.
|
|
31
|
+
|
|
32
|
+
**Do not attempt:**
|
|
33
|
+
- Auth / credential / env-var fixes — those are CI configuration, not test code.
|
|
34
|
+
- Refactoring (extracting page objects, custom helpers, splitting scenarios).
|
|
35
|
+
- Renaming Scenarios or tags — CI's `--grep` and pipelines reference them.
|
|
36
|
+
- Adding `retries: N` to a Scenario — masking flake is a human decision.
|
|
37
|
+
- Anything outside the failing test file (no `codecept.conf.*`, no helper, no PO edits).
|
|
38
|
+
|
|
39
|
+
If a failure doesn't match any allowed-fix pattern, **skip it** and record it as `unresolved` for the report.
|
|
40
|
+
|
|
41
|
+
### 3. Apply edits and rerun
|
|
42
|
+
Track every modification: file path, line number, before/after snippet, fix kind. Then rerun **only the failing scenarios** under the same config CI used:
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
npx codeceptjs run --grep '<scenario>|<scenario>|...'
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Find the right config via `package.json` scripts or the CI workflow file. Match the worker count if `run-workers` was used originally.
|
|
49
|
+
|
|
50
|
+
### 4. Compare against the baseline
|
|
51
|
+
Use **codeceptjs-run-analysis** again to enumerate the failing set after the fix.
|
|
52
|
+
|
|
53
|
+
| New failing-set vs baseline | Status |
|
|
54
|
+
|---|---|
|
|
55
|
+
| Empty (zero failures) | `SUCCESS_FULL` |
|
|
56
|
+
| Strict subset of the baseline | `SUCCESS_PARTIAL` |
|
|
57
|
+
| Identical set | `NO_IMPROVEMENT` |
|
|
58
|
+
| Larger, or contains a test not in baseline | `REGRESSED` |
|
|
59
|
+
|
|
60
|
+
### 5. Decide: keep or roll back
|
|
61
|
+
|
|
62
|
+
- `SUCCESS_FULL` / `SUCCESS_PARTIAL` → **keep** the edits.
|
|
63
|
+
- `NO_IMPROVEMENT` / `REGRESSED` → **roll back every edit** with `git checkout -- <files>`. The on-disk state at the end of the skill must be identical to what it was at the start.
|
|
64
|
+
|
|
65
|
+
A regression is treated the same as no improvement — the attempt didn't work, the working state is what was on disk before. Don't try to "undo only the bad ones" — the whole batch goes.
|
|
66
|
+
|
|
67
|
+
### 6. Write the report (always)
|
|
68
|
+
Write `output/ci-fix.md` regardless of outcome — CI reads it to surface results.
|
|
69
|
+
|
|
70
|
+
## Report format
|
|
71
|
+
|
|
72
|
+
```markdown
|
|
73
|
+
# CI Fix Report — <ISO timestamp>
|
|
74
|
+
|
|
75
|
+
**Status:** SUCCESS_FULL | SUCCESS_PARTIAL | NO_IMPROVEMENT | REGRESSED
|
|
76
|
+
|
|
77
|
+
## Baseline
|
|
78
|
+
Failing scenarios: N
|
|
79
|
+
- `tests/foo_test.js` — "user can log in"
|
|
80
|
+
- `tests/bar_test.js` — "checkout flow"
|
|
81
|
+
- ...
|
|
82
|
+
|
|
83
|
+
## Attempted fixes
|
|
84
|
+
- `tests/foo_test.js:42` — locator drift: `I.click('Save')` → `I.click('Save', '.toolbar')`
|
|
85
|
+
- `tests/bar_test.js:15` — added `I.waitForInvisible('.spinner')` before checkout click
|
|
86
|
+
- `tests/baz_test.js:7` — replaced `I.wait(3)` with `I.waitForVisible('.confirmation-dialog', 10)`
|
|
87
|
+
|
|
88
|
+
## Result
|
|
89
|
+
Failing scenarios after fix: M (was N)
|
|
90
|
+
- Resolved: `tests/foo_test.js` — "user can log in"
|
|
91
|
+
- Still failing: `tests/bar_test.js` — "checkout flow"
|
|
92
|
+
- New failures (REGRESSED only): `tests/baz_test.js` — "edit profile"
|
|
93
|
+
|
|
94
|
+
## Failure clusters (NO_IMPROVEMENT / REGRESSED only)
|
|
95
|
+
- **Cluster A (3 scenarios)** — `Element not found: button.btn-primary`. Likely a deploy regression or design change.
|
|
96
|
+
- **Cluster B (1 scenario)** — `Navigation timeout to /checkout`. Possible backend issue.
|
|
97
|
+
|
|
98
|
+
## Discarded changes (NO_IMPROVEMENT / REGRESSED only)
|
|
99
|
+
- `tests/foo_test.js` — restored
|
|
100
|
+
- `tests/bar_test.js` — restored
|
|
101
|
+
- `tests/baz_test.js` — restored
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
The first line of `Status:` is the machine-parseable signal. The rest is for the human reading the failed CI build.
|
|
105
|
+
|
|
106
|
+
## Things to avoid
|
|
107
|
+
|
|
108
|
+
- Editing files outside the failing test files. Config, helpers, page objects, and CI workflows are off-limits in this mode.
|
|
109
|
+
- Attempting more than one fix kind on the same scenario in one pass. Fix one thing, observe, then try another.
|
|
110
|
+
- `retries: N` to mask flakiness — human decision.
|
|
111
|
+
- Skipping rollback on `NO_IMPROVEMENT`. Even no-op edits can confuse the next run or the next reviewer.
|
|
112
|
+
- Writing a partial or missing `output/ci-fix.md`. CI depends on it; the absence of a report is itself a failure mode.
|
|
113
|
+
- Running the **whole** suite for the verification step — only the originally-failing scenarios, to keep CI time bounded.
|
|
114
|
+
|
|
115
|
+
## Related skills
|
|
116
|
+
|
|
117
|
+
- `codeceptjs-run-analysis` — baseline, clustering, post-fix comparison
|
|
118
|
+
- `codeceptjs-exploration` — replacement locators for drift
|
|
119
|
+
- `codeceptjs-fundamentals` — config, helper, CI environment
|
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: codeceptjs-auth
|
|
3
|
+
description: >
|
|
4
|
+
Use when a CodeceptJS test needs login, user roles, or session reuse — or
|
|
5
|
+
when another CodeceptJS skill identifies authorization is required. Also on
|
|
6
|
+
auth-related test failures. Trigger on login, sign-in, sign-up,
|
|
7
|
+
authentication, sessions, "logged in", admin/editor/user roles.
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# CodeceptJS Auth Plugin
|
|
11
|
+
|
|
12
|
+
The `auth` plugin logs each user in once, captures cookies (or localStorage/token via overrides), and restores the session for subsequent tests. Stale sessions trigger a fresh login automatically.
|
|
13
|
+
|
|
14
|
+
If the project already has `auth` configured (fundamentals' discovery output), **reuse** its existing inject name and user keys — don't reconfigure.
|
|
15
|
+
|
|
16
|
+
## Decide first — ask the user
|
|
17
|
+
|
|
18
|
+
Four answers shape the plugin. Don't guess any.
|
|
19
|
+
|
|
20
|
+
1. **Is a session needed at all?** Public flows (landing, signup) → skip the plugin.
|
|
21
|
+
2. **One user or many?** Default one; add more only when actually exercised.
|
|
22
|
+
3. **If many — what splits them?** Role, workspace/tenant, plan tier, sign-in provider, per-test fixture — real systems vary. Ask; use the answer to name `users.<key>` entries.
|
|
23
|
+
4. **What's the auth type?**
|
|
24
|
+
- **Form** — default; canonical shape below
|
|
25
|
+
- **OAuth / SSO** — click provider button, drive the IdP page (often separate origin)
|
|
26
|
+
- **Magic link / passwordless** — UI flow rarely worth automating; prefer an API mint or reading the link from a test mailbox
|
|
27
|
+
- **API token** — skip the form; write the token into `localStorage` via `executeScript`, or `I.setCookie(...)`
|
|
28
|
+
- **2FA / OTP** — async `login`; fetch the code from a test mailbox / backdoor before submitting
|
|
29
|
+
|
|
30
|
+
## Rules
|
|
31
|
+
|
|
32
|
+
1. **Login flow never lives in the conf.** Put it in `steps_file.js` (if included) or a page object; conf only references it: `login: (I) => I.login()`.
|
|
33
|
+
2. **Credentials from env only** — `.env` loaded via `process.loadEnvFile()` (no dotenv dependency). Passwords wrapped with `secret(...)`. No literals anywhere — conf, steps file, test, git history.
|
|
34
|
+
3. **`.env` is gitignored; `.env.example` is committed** with names, no values. Gitignore `output/*_session.json` too.
|
|
35
|
+
|
|
36
|
+
## Canonical shape
|
|
37
|
+
|
|
38
|
+
```js
|
|
39
|
+
// codecept.conf.js — first line of the file
|
|
40
|
+
process.loadEnvFile()
|
|
41
|
+
|
|
42
|
+
export const config = {
|
|
43
|
+
include: { I: './steps_file.js' },
|
|
44
|
+
plugins: {
|
|
45
|
+
auth: {
|
|
46
|
+
enabled: true,
|
|
47
|
+
saveToFile: true,
|
|
48
|
+
users: {
|
|
49
|
+
admin: {
|
|
50
|
+
login: (I) => I.login(),
|
|
51
|
+
check: (I) => I.see('Welcome, User', '.navbar'),
|
|
52
|
+
},
|
|
53
|
+
},
|
|
54
|
+
},
|
|
55
|
+
},
|
|
56
|
+
}
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
```js
|
|
60
|
+
// steps_file.js
|
|
61
|
+
import { secret } from 'codeceptjs'
|
|
62
|
+
const { I } = inject()
|
|
63
|
+
|
|
64
|
+
export default function () {
|
|
65
|
+
return actor({
|
|
66
|
+
login() {
|
|
67
|
+
I.amOnPage('/login')
|
|
68
|
+
I.fillField('Email', process.env.ADMIN_EMAIL)
|
|
69
|
+
I.fillField('Password', secret(process.env.ADMIN_PASSWORD))
|
|
70
|
+
I.click('Sign in')
|
|
71
|
+
},
|
|
72
|
+
})
|
|
73
|
+
}
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
```sh
|
|
77
|
+
# .env (gitignored) # .env.example (committed)
|
|
78
|
+
USER_EMAIL=... USER_EMAIL=
|
|
79
|
+
USER_PASSWORD=<secret> USER_PASSWORD=
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
## Pre-flight (before writing config)
|
|
83
|
+
|
|
84
|
+
1. **Read the real login page** — MCP `run_code` to `/login`, inspect the ARIA snapshot (`codeceptjs-exploration`). Field labels / `name` / `id` / submit control from the actual page, not guesses. Unclear authorization mechanism → ask the user.
|
|
85
|
+
2. **Pick a role-specific post-login marker** — something rendered only for *this* user (navbar username, `data-user-role`).
|
|
86
|
+
3. **Confirm session storage** — cookies (default) for server-rendered apps; localStorage/sessionStorage for SPAs. Verify after a manual login with `I.executeScript(() => Object.keys(localStorage))`. Cookie fetch/restore silently no-op against token storage.
|
|
87
|
+
|
|
88
|
+
## Verify
|
|
89
|
+
|
|
90
|
+
Run a one-Scenario file that calls `login(<role>)` then asserts on the post-login marker:
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
npx codeceptjs run --grep '<scenario>' --debug # real run, not dry — dry-run doesn't init plugins
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Enable `saveToFile: true` only after this round-trip succeeds — a bad saved session masks a broken `login`.
|
|
97
|
+
|
|
98
|
+
Then wire into hooks/tests: `Before(({ login }) => login())` for suite-wide, or per-test as needed.
|
|
99
|
+
|
|
100
|
+
## Multi-role shape
|
|
101
|
+
|
|
102
|
+
Only after question 3 is answered. Keys named after whatever splits users *in this system*; one matching actor method per key:
|
|
103
|
+
|
|
104
|
+
```js
|
|
105
|
+
users: {
|
|
106
|
+
admin: { login: (I) => I.loginAsAdmin() },
|
|
107
|
+
workspaceB: { login: (I) => I.loginToWorkspaceB() },
|
|
108
|
+
}
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Don't parameterise into a single `login(key)` — the plugin keys sessions by name, explicit methods read better. Switch mid-Scenario: `session('<key>')` opens a parallel browser context (fundamentals § Writing tests).
|
|
112
|
+
|
|
113
|
+
## Token / localStorage auth
|
|
114
|
+
|
|
115
|
+
Override `fetch` / `restore` when sessions live outside cookies:
|
|
116
|
+
|
|
117
|
+
```js
|
|
118
|
+
admin: {
|
|
119
|
+
login: (I) => I.loginAsAdmin(),
|
|
120
|
+
check: (I) => I.see('Admin', '.navbar'),
|
|
121
|
+
fetch: (I) => I.executeScript(() => localStorage.getItem('session_id')),
|
|
122
|
+
restore: (I, session) => {
|
|
123
|
+
I.amOnPage('/')
|
|
124
|
+
I.executeScript((s) => localStorage.setItem('session_id', s), session)
|
|
125
|
+
},
|
|
126
|
+
}
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
`check(I, session)` receives whatever `fetch` returned — throw inside `check` to force fresh login (e.g. `/me` endpoint shows wrong user).
|
|
130
|
+
|
|
131
|
+
## Pitfalls
|
|
132
|
+
|
|
133
|
+
- Credentials inlined in conf/test — always env-driven + `secret()`.
|
|
134
|
+
- Forgetting to gitignore `.env` and `output/*_session.json` — both leak credentials.
|
|
135
|
+
|
|
136
|
+
## Related skills
|
|
137
|
+
|
|
138
|
+
- `codeceptjs-fundamentals` — secrets rule, sessions, config mutation trap
|
|
139
|
+
- `codeceptjs-exploration` — reading the live login page
|
|
140
|
+
- `writing-codeceptjs-tests` / `refactoring-codeceptjs-tests` — invoke this skill when auth is identified
|
|
141
|
+
- `debugging-codeceptjs-tests` — auth-related failure patterns
|
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: codeceptjs-exploration
|
|
3
|
+
description: >
|
|
4
|
+
Use when an agent needs to learn what's on a page in CodeceptJS — read the
|
|
5
|
+
ARIA tree, inspect candidate elements, pick or disambiguate a stable locator.
|
|
6
|
+
Drives the live browser via MCP `run_code` / `snapshot`. Invoked by
|
|
7
|
+
writing-codeceptjs-tests, debugging-codeceptjs-tests, and
|
|
8
|
+
refactoring-codeceptjs-tests whenever page inspection is needed.
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# CodeceptJS Page Exploration
|
|
12
|
+
|
|
13
|
+
Authoring a test, debugging a failure, and refactoring a stale locator share one task: open a page, find the right element, pick a stable locator. This is that playbook.
|
|
14
|
+
|
|
15
|
+
## Tools
|
|
16
|
+
|
|
17
|
+
- **`run_code`** — runs CodeceptJS code, returns produced values, captures `console.*`, saves a final-state snapshot. For *do something and look at the result*.
|
|
18
|
+
- **`snapshot`** — captures state without acting (URL, cookies, localStorage, HTML, ARIA, screenshot, console). For "what's on the page right now".
|
|
19
|
+
|
|
20
|
+
Artifact sources, in preference order:
|
|
21
|
+
|
|
22
|
+
1. **ARIA snapshot** — structured, no styling noise, easy duplicate/accessibility-name scanning
|
|
23
|
+
2. **Screenshot** — visual confirmation; catches layout breaks ARIA can't show
|
|
24
|
+
3. **HTML** — only when ARIA lacks context (custom widgets without accessible names, attribute-driven behaviour)
|
|
25
|
+
|
|
26
|
+
## Inspect an element
|
|
27
|
+
|
|
28
|
+
`I.grabWebElement(locator)` → one WebElement; `I.grabWebElements(locator)` → array. Same cross-helper API on Playwright / Puppeteer / WebDriver.
|
|
29
|
+
|
|
30
|
+
| You want to … | Method |
|
|
31
|
+
|---|---|
|
|
32
|
+
| Confirm rendered / visible / enabled | `exists()`, `isVisible()`, `isEnabled()` |
|
|
33
|
+
| Read text / value / attribute / property | `getText()`, `getValue()`, `getAttribute(n)`, `getProperty(n)` |
|
|
34
|
+
| Position on page | `getBoundingBox()` — flags offscreen / zero-sized |
|
|
35
|
+
| Rendered markup | `toOuterHTML()`, `toSimplifiedHTML(300)` (truncated, MCP-friendly) |
|
|
36
|
+
| Stable selector for a fix | `toAbsoluteXPath()` |
|
|
37
|
+
| Inside an iframe | `inIframe(async (body) => { ... })` |
|
|
38
|
+
| Drill into children | `$(loc)`, `$$(loc)` |
|
|
39
|
+
| Browser-side function | `evaluate(fn, ...args)` |
|
|
40
|
+
|
|
41
|
+
## Discover candidates when the obvious locator misses
|
|
42
|
+
|
|
43
|
+
When `Edit` matches nothing, the control may say "Change", carry `aria-label="Edit user"`, or live in `.btn-edit`. Cast a wide net with a permissive XPath via `I.grabWebElements`, then disambiguate.
|
|
44
|
+
|
|
45
|
+
OR together in the XPath:
|
|
46
|
+
|
|
47
|
+
- visible text — `text()` (or `.` for descendants)
|
|
48
|
+
- attributes — `@class`, `@aria-label`, `@title`, `@data-action`, `@id`
|
|
49
|
+
- **synonyms** — edit/change/modify; delete/remove/trash; submit/send/save
|
|
50
|
+
|
|
51
|
+
Case-insensitive via `translate(...)`:
|
|
52
|
+
|
|
53
|
+
```
|
|
54
|
+
//*[contains(translate(., 'EDIT', 'edit'), 'edit')
|
|
55
|
+
or contains(translate(@class, 'EDIT', 'edit'), 'edit')
|
|
56
|
+
or contains(translate(@aria-label, 'EDIT', 'edit'), 'edit')
|
|
57
|
+
or contains(translate(., 'CHANGE', 'change'), 'change')]
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Then iterate `toSimplifiedHTML(150)` over the results, pick the right candidate, commit a stable locator from its discriminating attribute or text.
|
|
61
|
+
|
|
62
|
+
## Pick a stable locator
|
|
63
|
+
|
|
64
|
+
Two decisions in order: **which region scopes the lookup** (context), **what identifies the element inside it**. Region first keeps the identifier short and semantic — the discriminator found during disambiguation belongs in the context argument:
|
|
65
|
+
|
|
66
|
+
```js
|
|
67
|
+
I.click('Edit user', '.user-row') // ✅ region + what the user sees
|
|
68
|
+
I.click('#user-row-42 button.edit') // ❌ same element, brittle, unreadable
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Stable regions: landmarks (`nav`, `main`, `{ role: 'dialog' }`), app-shell containers (`.sidebar`, `.toolbar`, `.modal`), rows/cards identified by data via `locate(...)`. Identifier priority (full rationale: `codeceptjs-fundamentals` § Locators):
|
|
72
|
+
|
|
73
|
+
1. Visible label / accessible name — plain string already matches `aria-label`; don't expand to `{ css: '[aria-label="..."]' }`
|
|
74
|
+
2. ARIA role when ambiguous within context or role is part of the check
|
|
75
|
+
3. `$name` via `customLocator` when team test attributes exist
|
|
76
|
+
4. Composed CSS, still scoped: `I.click('button.edit', '#user-row-42')`
|
|
77
|
+
5. `toAbsoluteXPath()` — last resort; flag the team to add a `data-testid`
|
|
78
|
+
|
|
79
|
+
**Never commit an unverified locator** — confirm via `run_code` (`I.seeElement(loc, context)` or `grabWebElement(loc)`) that it matches exactly one element.
|
|
80
|
+
|
|
81
|
+
## Common patterns
|
|
82
|
+
|
|
83
|
+
- Strict mode 2+ matches → `grabWebElements('Save')` + `toSimplifiedHTML(200)` each, find discriminator, pass as context: `I.click('Save', '.modal')`
|
|
84
|
+
- Button rendered but doesn't act → `grabWebElement('Submit')` + `isEnabled()` + `getBoundingBox()` — disabled? offscreen? zero-sized?
|
|
85
|
+
- Wrong row in a list → `grabWebElements('.user-row')`, `getText()` per row to identify, `getAttribute('data-id')` for stable hook
|
|
86
|
+
- Inside iframe → `(await I.grabWebElement('iframe.editor')).inIframe(async (body) => body.$('button'))`
|
|
87
|
+
|
|
88
|
+
## Things to avoid
|
|
89
|
+
|
|
90
|
+
- Choosing a locator without seeing candidates first.
|
|
91
|
+
- Committing `toAbsoluteXPath()` when a semantic locator is available.
|
|
92
|
+
- Committing unscoped locators where a context keeps them short.
|
|
93
|
+
- Ignoring the screenshot — "exists in HTML" ≠ "user can see it".
|
|
94
|
+
- `usePlaywrightTo` / `useWebDriverTo` when WebElement methods cover it.
|
|
95
|
+
|
|
96
|
+
## Related skills
|
|
97
|
+
|
|
98
|
+
- `codeceptjs-fundamentals` — locator priority, await rule
|
|
99
|
+
- `writing-codeceptjs-tests` — invokes this during Mode B exploration
|
|
100
|
+
- `debugging-codeceptjs-tests` — invokes this for live inspection; offline variant via `codeceptq`
|